Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If you ask the service to send too many notifications in a quick succession it sends infinite notifications #10

Open
mxcl opened this issue Aug 4, 2018 · 7 comments

Comments

@mxcl
Copy link

mxcl commented Aug 4, 2018

Pretty sure this is not my code doing this.

If you ask the service to send like 50 notifications in quick succession it repeatedly sends the same batch of notifications forever.

I have seen this happen twice now.

@mxcl
Copy link
Author

mxcl commented Aug 4, 2018

This is deployed to AWS on a fairly constrained Ubuntu server.

My send is within a router callback for PerfectHTTPServer, there is no loop in my code and I am logging when the callback happens, it only happens once. So unless Apple are doing this it does seem to be within PerfectNotifications or one of its dependencies.

@mxcl
Copy link
Author

mxcl commented Aug 4, 2018

However notably the callback is called like 50 times in rapid succession. separate instantiations on different threads. Possibly a concurrency violation?

@kjessup
Copy link
Member

kjessup commented Aug 7, 2018

Thank you for opening the issue. I just wanted to let you know that I'm digging into this one today and will report back ASAP.

@mxcl
Copy link
Author

mxcl commented Aug 7, 2018

Great thanks x. When I said callback only called once I meant it’s called like 50 times, but I expected 50, after that the notifications keep getting resent until I kill the server.

@mxcl
Copy link
Author

mxcl commented Aug 15, 2018

The recent commit fixes this, though an awful lot of notifications seem to get dropped instead, with a lot of internalServerError errors and ~HTTP2Client.startReadThread messages.

So thanks, I'll look into whether or not I can help fix that.

 APNs: 500 Internal Server Error: Connection dropped 90a6<snip>
 APNs: 500 Internal Server Error: Connection dropped bcee82<snip>

I get 20 or so of that. Followed by 40 or so HTTP2Client.startReadThread.

So I guess too many connections to APNs are being opened, I'm going to investigate.

@mxcl
Copy link
Author

mxcl commented Aug 15, 2018

Actually maybe not 100% fixed, looking at my notifications log some were doubled up. But it does stop now at least rather than sending them until I kill the server.

@mxcl
Copy link
Author

mxcl commented Aug 15, 2018

Yeah ok, this seems to possibly be the issue, I put a print counter into the code when HTTP2 streams are created and when I use my test case I get up to 66 open connections to APNs.

A google suggests 15 is the undocumented limit per IP that Apple allow before they start closing the connections.

Also my reading suggests that we are preemptively creating new streams, Apple say you should be able to do 9000 payload writes per second to one connection, I'm just trying to write 66 here within a few seconds.

Anyway, if there's any feedback I can take it into account as I attempt to fix this. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants