Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GetModes randomly hangs #95

Open
clorteau opened this issue Nov 5, 2018 · 37 comments
Open

GetModes randomly hangs #95

clorteau opened this issue Nov 5, 2018 · 37 comments

Comments

@clorteau
Copy link

clorteau commented Nov 5, 2018

What version of Python are you using (python -V)?

2.7.13

What operating system and processor architecture are you using (python -c 'import platform; print(platform.uname());')?

('Linux', 'raspberrypi.lan', '4.14.71-v7+', '#1145 SMP Fri Sep 21 15:38:35 BST 2018', 'armv7l', '')
Running Raspbian on a raspberry pi 3 b+.

Which Python packages do you have installed (run the pip freeze or pip3 freeze command and paste output)?

arlo==1.0.9
arrow==0.12.1
backports-abc==0.5
backports.functools-lru-cache==1.5
bitstring==3.1.5
certifi==2018.4.16
chardet==3.0.4
cryptography==1.7.1
enum34==1.1.6
evdev==1.1.2
futures==3.2.0
gpiozero==1.4.1
idna==2.6
ipaddress==1.0.17
keyring==10.1
keyrings.alt==1.3
monotonic==1.4
numpy==1.12.1
picamera==1.13
Pillow==4.0.0
praw==5.4.0
prawcore==0.14.0
pyasn1==0.1.9
pycrypto==2.6.1
pygobject==3.22.0
pymongo==3.4.0
pyOpenSSL==16.2.0
PySocks==1.6.8
python-dateutil==2.7.2
pyxdg==0.25
PyYAML==3.12
requests==2.18.4
RPi.GPIO==0.6.4
SecretStorage==2.3.1
singledispatch==3.4.0.3
six==1.11.0
speedtest-cli==1.0.0
spidev==3.3
sseclient==0.0.19
tornado==5.0.2
update-checker==0.16
urllib3==1.22

Which Arlo hardware do you have (camera types - [Arlo, Pro, Q, etc.], basestation model, etc.)?

Arlo

What did you do?

I'm running the following script: https://github.com/clorteau/armdisarmarlo/blob/master/monitor.py
On lines 153 and 67 you can see a call to Arlo.GetModes. It sometimes hang for apparently forever (I have let it stay stuck for hours). I tried adding a timer that refreshes the connection token every 10 minutes but that doesn't help.

What did you expect to see?

I would expect the call to return or at least throw an exception or an error message. Instead, literally nothing happens. That call sometimes simply never returns.

What did you see instead?

Nothing; sometimes the call simply doesn't return.

Does this issue reproduce with the latest release?

I'm using the latest release.
Edit: I just updated to 1.1.8. I'll report back.

@clorteau
Copy link
Author

clorteau commented Nov 6, 2018

I just upgraded my Arlo library to 1.1.8. I'll report back with results.

@jsm174
Copy link

jsm174 commented Nov 6, 2018

This seems similar to my issue #93.

@jeffreydwalter
Copy link
Owner

@clorteau this is most certainly the same type of issue as #93. Most likely, I won't have time to look into this until the weekend, but I will definitely dig in.

@clorteau
Copy link
Author

clorteau commented Nov 6, 2018

Great, thanks for looking into it. For what it's worth, I have let the script run all night after upgrading to 1.1.8 and it still hasn't froze.

@jeffreydwalter
Copy link
Owner

Great! Glad to hear that. Keep me posted.

@clorteau
Copy link
Author

clorteau commented Nov 7, 2018

Well, it's been running flawlessly for 2 nights and a day now. So false alarm I guess, my bad!

If I just may venture a suggestion maybe in the readme at https://github.com/jeffreydwalter/arlo state that "pip install Arlo" gets you an outdated version and that "pip install git+https://github.com/jeffreydwalter/arlo" will give you the latest.

Anyway thank you for the support and the library that's very useful.

@clorteau clorteau closed this as completed Nov 7, 2018
@jeffreydwalter
Copy link
Owner

Glad to hear it's working for you.

pip install arlo does NOT get you an outdated version. It gets the latest stable release version from pypi.

@clorteau
Copy link
Author

clorteau commented Nov 10, 2018

Actually I take that back; it has been happening several times today. So I can now answer "Does this issue reproduce with the latest release?" as "Yes".

Anything I can do to help? I don't mind editing the source to add some debugging outputs.

Is that of any use? It appeared 26 seconds before another freeze:

Exception in thread EventStream: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/Arlo.py" , line 295, in QueueEvents self.event_streams[basestation_id].Disconnect() File "/usr/local/lib/python2.7/dist-packages/Arlo.py" , line 100, in Disconnect self.Unregister() File "/usr/local/lib/python2.7/dist-packages/Arlo.py" , line 120, in Unregister if self.heartbeat_thread != threading.current_threa d(): AttributeError: 'EventStream' object has no attribute ' heartbeat_thread'

@jeffreydwalter
Copy link
Owner

@clorteau just pushed a fix for this issue. Please give it a try when you can and let me know how it goes. Thanks!

@clorteau
Copy link
Author

clorteau commented Jan 24, 2019 via email

@clorteau
Copy link
Author

clorteau commented Jan 24, 2019

I keep getting this:

Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "./monitor.py", line 143, in monitorAndUpdate
arlo_armed = arlo.getArmed()
File "./monitor.py", line 66, in getArmed
status = self.arlo.GetModes(self.basestation)
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 505, in GetModes
return self.NotifyAndGetResponse(basestation, {"action":"get","resource":"modes","publishResponse":False})
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 391, in NotifyAndGetResponse
self.Subscribe(basestation)
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 317, in Subscribe
self.event_streams[basestation_id].Start()
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 94, in Start
self.event_stream_thread.start()
AttributeError: 'EventStream' object has no attribute 'event_stream_thread'

This is in a large try that catches all Exception yet I don't catch it so my thread crashes.

@jeffreydwalter
Copy link
Owner

I don't have my Arlo basestation hooked up right now, so it's impossible for me to test. I increased the message timeout to 10 minutes from 2 minutes. Give the latest release a shot. Thanks.

@clorteau
Copy link
Author

Same result. Note that I'm getting the exception about immediately.

@jeffreydwalter
Copy link
Owner

jeffreydwalter commented Jan 25, 2019

Seems like you might be getting an exception when attempting to connect to the event stream. Just pushed another with a print debugging statement. Can you tell me what you see. Thanks! (Sorry for the shitty dev practices. :) )

@clorteau
Copy link
Author

Lol no worries. But same result i'm afraid.

@jeffreydwalter
Copy link
Owner

jeffreydwalter commented Jan 25, 2019 via email

@clorteau
Copy link
Author

No really, same thing; this is the entirety of my output - all the prints are from me:

PID: 24721
Started
Input PIN then to arm/disarm; '/' then to quit; '.' then to force refresh
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "./monitor.py", line 144, in monitorAndUpdate
arlo_armed = arlo.getArmed()
File "./monitor.py", line 67, in getArmed
status = self.arlo.GetModes(self.basestation)
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 505, in GetModes
return self.NotifyAndGetResponse(basestation, {"action":"get","resource":"modes","publishResponse":False})
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 391, in NotifyAndGetResponse
self.Subscribe(basestation)
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 317, in Subscribe
self.event_streams[basestation_id].Start()
File "/usr/local/lib/python2.7/dist-packages/Arlo.py", line 94, in Start
self.event_stream_thread.start()
AttributeError: 'EventStream' object has no attribute 'event_stream_thread'

And I do have the latest version:

$ pip freeze | grep arlo
arlo==1.2.7

@clorteau
Copy link
Author

clorteau commented Jan 25, 2019

@jeffreydwalter If you'd like I created an account with access to my base station so you can run tests. You can email me at [redacted] for the user name and password if you want it.

Edit: Btw I don't see any changes in the last 2 commits besides version number changes:
a9f9bac
55c3ac5

@jeffreydwalter
Copy link
Owner

@clorteau thanks for the offer! I REALLY appreciate it. I ended up just hooking my Arlo basestation back up. :) Did some testing and pushed a fix. Please give the latest a try!

@clorteau
Copy link
Author

Damn, same result again...

Sorry to insist but I still don't see any other change besides a version number change in the last commit: a655bc9 .

@jeffreydwalter
Copy link
Owner

Ha! I forgot to git add the file. lol Sorry! Try now.

@clorteau
Copy link
Author

Cool. Now I get:

Exception: Failed to subscribe to eventstream: init() got an unexpected keyword argument 'session'

@jeffreydwalter
Copy link
Owner

Okay, great.That exception leads me to believe that you have the wrong version of some package or some conflicting package. Can you paste the output of pip freeze?

@clorteau
Copy link
Author

Here you go:

$ pip freeze
arlo==1.2.9
arrow==0.12.1
backports-abc==0.5
backports.functools-lru-cache==1.5
bitstring==3.1.5
certifi==2018.11.29
chardet==3.0.4
cryptography==1.7.1
enum34==1.1.6
evdev==1.1.2
futures==3.2.0
Glances==3.0.2
gpiozero==1.4.1
idna==2.8
ipaddress==1.0.17
keyring==10.1
keyrings.alt==1.3
monotonic==1.4
numpy==1.12.1
picamera==1.13
Pillow==4.0.0
praw==5.4.0
prawcore==0.14.0
psutil==5.4.8
pyarlo==0.2.3
pyasn1==0.1.9
pycrypto==2.6.1
pygobject==3.22.0
pymongo==3.4.0
pyOpenSSL==16.2.0
PySocks==1.6.8
python-dateutil==2.7.2
pyxdg==0.25
PyYAML==3.12
requests==2.21.0
RPi.GPIO==0.6.5
SecretStorage==2.3.1
singledispatch==3.4.0.3
six==1.11.0
speedtest-cli==1.0.0
spidev==3.3
sseclient==0.0.19
sseclient-py==1.7
tornado==5.0.2
update-checker==0.16
urllib3==1.24.1

@jeffreydwalter
Copy link
Owner

Yep, here's your problem:

sseclient==0.0.19
sseclient-py==1.7

Please uninstall sseclient-py and try again.

@clorteau
Copy link
Author

Aaahhh... it even warns in the homepage about it! I thought I checked that but well, evidently no. I don't even know how I got sseclient-py, i'll look at getting rid of it.

@jeffreydwalter
Copy link
Owner

pip uninstall sseclient-py should to the trick. I might have to add a check for that in the installer.

@clorteau
Copy link
Author

Hey much better now! :) So far so good. I'll let it run now to see if the original issue is gone. (According to my first post I did not have it installed originally when I reported the issue)

@jeffreydwalter
Copy link
Owner

Awesome! I don't see sseclient-py installed in your original pip freeze output. The behavior now should be that the function will return None if it times out. I might change that to have it raise an exception instead...

@clorteau
Copy link
Author

Cool, my code is already checking for a return of None to refresh the token and re-try, so i'll be able to see if it runs as intended.

Thanks a lot for all the support.

@jeffreydwalter
Copy link
Owner

You got it. Thanks for all your help!

@clorteau
Copy link
Author

Even though I do get a number of None and can refresh the session token, it still ends up hanging, apparently 30 minutes after the last session refresh. Here's a log of my last run:

Jan 25 14:54:01 raspberrypi numpad arming: PID: 18309
Jan 25 14:54:01 raspberrypi numpad arming: Started
Jan 25 16:26:20 raspberrypi numpad arming: Connection error
Jan 25 16:26:22 raspberrypi numpad arming: Refreshed Arlo connection token
Jan 25 17:12:18 raspberrypi numpad arming: Connection error
Jan 25 17:12:21 raspberrypi numpad arming: Refreshed Arlo connection token
Jan 25 17:34:22 raspberrypi numpad arming: Correct PIN entered
Jan 25 18:44:37 raspberrypi numpad arming: Connection error
Jan 25 18:44:39 raspberrypi numpad arming: Refreshed Arlo connection token
Jan 25 18:57:46 raspberrypi numpad arming: Correct PIN entered
Jan 25 19:16:47 raspberrypi numpad arming: Connection error
Jan 25 19:16:49 raspberrypi numpad arming: Refreshed Arlo connection token

Then at 19:46:55 my script hangs seemingly forever on the GetArmed and doesn't show any exception or print statement.

@jeffreydwalter
Copy link
Owner

Bummer... I think the problem is that the queue timeout never really happens because of ping messages and other junk in the queue that are getting read. I'll take another stab at it tomorrow.

What is that Connection error?

@clorteau
Copy link
Author

clorteau commented Jan 26, 2019 via email

@alistairg
Copy link

I'm working through the "'EventStream' object has no attribute 'event_stream_thread'" error when I get it elsewhere. In my case it's caused by Arlo returning a 'logout' event immediately upon subscription, so bad things happen.

I suspect it's because a prior run of the API crashed, we're retrying quickly, so Arlo tries to terminate the prior connection, but the termination message is sent to the new session. That'd then cause subscription "hangs" as the event subscription was (incorrectly) terminated during "Register"...

I think we can probably disregard that "logout" message if registration has not successfully completed. I'll do more tests, and submit a PR if that fixes it.

@jeffreydwalter
Copy link
Owner

That sounds reasonable.

@jeffreydwalter
Copy link
Owner

@alistairg @clorteau don't know if you guys are still using this library, but I did a pretty big rewrite of the eventstream code a while back. Would be interested to know if the latest build fixes your issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants