USPS Component Broken -- myusps.USPSError: authentication failed -- Debugging

[brianjking]

Make sure you are running the latest version of Home Assistant before reporting an issue.

You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:

Home Assistant release (hass --version):

0.52

Python release (python3 --version):

Python 3.4.2

Component/platform:

USPS

Description of problem:

USPS fails to init/login and subsequently displays persistent notification in UI. Shows errors in log too (see log info below).

Expected:

USPS is displayed in UI.

Problem-relevant configuration.yaml entries and steps to reproduce:

Traceback (if applicable):

2017-08-26 09:55:41 INFO (MainThread) [homeassistant.setup] Setting up usps
2017-08-26 09:55:41 ERROR (MainThread) [homeassistant.setup] Error during setup of component usps
Traceback (most recent call last):
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/homeassistant/setup.py", line 194, in _async_setup_component
    component.setup, hass, processed_config)
  File "/usr/lib/python3.4/asyncio/futures.py", line 388, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.4/asyncio/tasks.py", line 286, in _wakeup
    value = future.result()
  File "/usr/lib/python3.4/asyncio/futures.py", line 277, in result
    raise self._exception
  File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/homeassistant/components/usps.py", line 40, in setup
    conf = config[DOMAIN]
KeyError: 'usps'
2017-08-26 09:55:41 INFO (MainThread) [homeassistant.core] Bus:Handling <Event call_service[L]: service_data=message=The following components and platforms could not be set up:
* [usps](https://home-assistant.io/components/usps/)
Please check your config, notification_id=invalid_config, title=Invalid config, service=create, domain=persistent_notification, service_call_id=1974245168-5>
2017-08-26 09:55:41 INFO (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: new_state=<state persistent_notification.invalid_config=The following components and platforms could not be set up:
* [usps](https://home-assistant.io/components/usps/)
Please check your config; title=Invalid config @ 2017-08-26T09:55:41.967178-05:00>, old_state=None, entity_id=persistent_notification.invalid_config>
2017-08-26 09:55:41 INFO (MainThread) [homeassistant.core] Bus:Handling <Event service_executed[L]: service_call_id=1974245168-5>
2017-08-26 09:55:41 ERROR (MainThread) [homeassistant.setup] Unable to setup dependencies of sensor.usps. Setup failed for dependencies: usps
2017-08-26 09:55:41 ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform sensor.usps: Could not setup all dependencies.
2017-08-26 09:55:41 ERROR (MainThread) [homeassistant.setup] Unable to setup dependencies of camera.usps. Setup failed for dependencies: usps
2017-08-26 09:55:41 ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform camera.usps: Could not setup all dependencies.
2017-08-26 09:55:41 INFO (MainThread) [homeassistant.core] Bus:Handling <Event call_service[L]: service_data=message=The following components and platforms could not be set up:
* sensor.usps
* [usps](https://home-assistant.io/components/usps/)
Please check your config, notification_id=invalid_config, title=Invalid config, service=create, domain=persistent_notification, service_call_id=1974245168-6>
2017-08-26 09:55:41 INFO (MainThread) [homeassistant.core] Bus:Handling <Event call_service[L]: service_data=message=The following components and platforms could not be set up:
* camera.usps
* sensor.usps
* [usps](https://home-assistant.io/components/usps/)
Please check your config, notification_id=invalid_config, title=Invalid config, service=create, domain=persistent_notification, service_call_id=1974245168-7>
2017-08-26 09:55:42 INFO (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: new_state=<state persistent_notification.invalid_config=The following components and platforms could not be set up:
* sensor.usps
* [usps](https://home-assistant.io/components/usps/)
Please check your config; title=Invalid config @ 2017-08-26T09:55:42.007400-05:00>, old_state=<state persistent_notification.invalid_config=The following components and platforms could not be set up:
* [usps](https://home-assistant.io/components/usps/)
Please check your config; title=Invalid config @ 2017-08-26T09:55:41.967178-05:00>, entity_id=persistent_notification.invalid_config>
2017-08-26 09:55:42 INFO (MainThread) [homeassistant.core] Bus:Handling <Event service_executed[L]: service_call_id=1974245168-6>
2017-08-26 09:55:42 INFO (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: new_state=<state persistent_notification.invalid_config=The following components and platforms could not be set up:
* camera.usps
* sensor.usps
* [usps](https://home-assistant.io/components/usps/)
Please check your config; title=Invalid config @ 2017-08-26T09:55:42.019753-05:00>, old_state=<state persistent_notification.invalid_config=The following components and platforms could not be set up:
* sensor.usps
* [usps](https://home-assistant.io/components/usps/)
Please check your config; title=Invalid config @ 2017-08-26T09:55:42.007400-05:00>, entity_id=persistent_notification.invalid_config>

Additional info:

• Appears that Docker version also has same issue see USPS failing in 0.52 docker build #9142
• My YAML config https://github.com/brianjking/homeassistant-config/blob/master/sensor/usps.yaml

Ping @mezz64 @happyleavesaoc Any guidance, please? <3 Thank you!

[happyleavesaoc]

usps had a breaking config change:

https://home-assistant.io/blog/2017/08/26/release-0-52/#read-more

[brianjking]

@happyleavesaoc I saw that in release notes, but I don't see any changes I need to make.

[happyleavesaoc]

@brianjking there's an example in the release notes, and the docs are updated as well: https://home-assistant.io/components/usps/

[mezz64]

As @happyleavesaoc mentioned updated info is in the docs. To summarize, remove your sensor specific config and add a new config for the USPS platform. You can also define the scan internal under the camera platform if you'd like to change the mail image rotation interval.

[danodemano]

This is nowhere near clear in the release notes what needs done. For those that end up here:

Remove the:

- platform: usps
  name: "USPS Home"
  username: !secret usps_username
  password: !secret usps_password

From your sensor config/yaml.

Add in:

usps:
    username: !secret usps_username
    password: !secret usps_password

To your main config.

[JSCSJSCS]

I've tried changing the USPS Camera scan interval without success.
https://community.home-assistant.io/t/new-usps-component-options-for-scan-interval-not-working/25565
Running Hass.io 0.52

[mezz64]

@JSCSJSCS this is a known issue with the camera platform. The plan during the PR process was to utilize the correct path (scan_interval) for implementing that option and correct the larger problem in the platform at a later date. I'd urge you to create a separate issue more specific to the camera platform rather than USPS and hopefully someone can figure it out.

[brianjking]

Sorry, didn't read info about breaking changes closely enough.

@mezz64 As long as I switch to the format listed below this can be it's own standalone file instead of configuration.yaml, right?

usps:
    username: !secret usps_username
    password: !secret usps_password

Thanks!

[JSCSJSCS]

@mezz64 Thank you for the reply. I looked for the USPS Camera component issue queue on github and was unsuccessful. If you would please drop me the link to it, I will post my issue there. Thank!

[mezz64]

@brianjking You can move the usps config into a separate file the same way you would for any other platform with an include directive.

@JSCSJSCS I was suggesting that you open a new issue to describe your problem.

[brianjking]

@mezz64

Yesterday my mail camera was working, today it switched to the error seen above until I restarted Home Assistant. Is this a known issue at this time?

Thanks!

[mezz64]

As currently presented, the camera error is a bit misleading. What you posted just means there are no mail images available. You will see this every day after midnight when your mail count resets until the new mail is made available the next morning and all day Sunday.

I was going to submit a PR to the front end to change the error message to "No image available" so it doesn't seem like there was a problem.

[JSCSJSCS]

@brianjking I can confirm that my UPSP Camera has the error. On your post, I tried to restart Hass.io and the images returned without error. Hmm.

[mezz64]

Anyone not seeing camera images, please verify you have images available for the current day/time on the usps site corresponding to them not being visible in hass. See my reply above about situations where you won't have images available. Also, please also keep in mind we only check for updates every 30 minutes so there can be a 30min lag from when you get your informed delivery digest email in the morning.

[JSCSJSCS]

@mezz Thank you for the reply. I should have mentioned in my last post that I did check the USPS site to see that I did indeed have images. Today is the same. See images below.

I tried simply to refresh the page, but that did not help. But after Hass.io restart it worked.

[mezz64]

This is definitely odd behavior. Lets focus on the USPS Mail sensor a bit to make sure data is being updated correctly.

Is the value changing to 0 after midnight and then correctly updating to the number of mail pieces for the current day sometime in the morning?

Last thing is to enable debug logging for the usps components so we can see when new images are being fetched. Add the following lines under your logger/logs directive in your configuration.yaml:

    homeassistant.components.camera.usps: debug
    homeassistant.components.usps: debug
    homeassistant.components.sensor.usps: debug

[JSCSJSCS]

I'm glad we are going to get to the bottom of this, but I can't help today - no mail is due! :) I will add the lines to my config for tomorrow though.

[fronzbot]

I can confirm that I'm seeing the same issue as @JSCSJSCS that the after one day the usps camera displays "Error loading image" despite new mail being visible in the usps dashboard. I can retrieve the images again with a restart of homeassistant. The packages and mail information are updating properly, so it seems to be an issue with fetching the images themselves.

I'm adding debug logging for the camera and will report back tomorrow.

[JSCSJSCS]

Since I starter debug in logger, I haven't had an issue. HAHA
Since this component is a webpage scrapper, it may have to be reworked soon in any event. I just got this announcement.

Dear Informed Delivery User,

The United States Postal Service® is excited to bring you a new streamlined look and feel for Informed Delivery® in the coming weeks!

[happyleavesaoc]

Myself or someone else will update the dependency once those changes roll out.

[CCOSTAN]

Today is labor day in the states (Holiday) so no mail delivery. I seem to be getting camera errors in the logs every 5 seconds (default scan interval) .

result
Sep 04 21:16:03 Carlo-Pi hass[2107]: raise self._exception
Sep 04 21:16:03 Carlo-Pi hass[2107]: File "/usr/lib/python3.4/concurrent/futures/thread.py", lin
e 54, in run
Sep 04 21:16:03 Carlo-Pi hass[2107]: result = self.fn(*self.args, **self.kwargs)
Sep 04 21:16:03 Carlo-Pi hass[2107]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homea
ssistant/components/camera/usps.py", line 49, in camera_image
Sep 04 21:16:03 Carlo-Pi hass[2107]: self._usps.update()
Sep 04 21:16:03 Carlo-Pi hass[2107]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homea
ssistant/util/__init__.py", line 306, in wrapper
Sep 04 21:16:03 Carlo-Pi hass[2107]: result = method(*args, **kwargs)
Sep 04 21:16:03 Carlo-Pi hass[2107]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homea
ssistant/components/usps.py", line 80, in update
Sep 04 21:16:03 Carlo-Pi hass[2107]: self.packages = myusps.get_packages(self.session)
Sep 04 21:16:03 Carlo-Pi hass[2107]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/myusp
s/__init__.py", line 177, in wrapped
Sep 04 21:16:03 Carlo-Pi hass[2107]: return function(*args)
Sep 04 21:16:04 Carlo-Pi hass[2107]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/myusp
s/__init__.py", line 208, in get_packages
Sep 04 21:16:04 Carlo-Pi hass[2107]: dashboard = _require_elem(response, DASHBOARD_TAG, DASHBOAR
D_ATTRS)
Sep 04 21:16:04 Carlo-Pi hass[2107]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/myusp
s/__init__.py", line 89, in _require_elem
Sep 04 21:16:04 Carlo-Pi hass[2107]: raise ValueError('No element found')
Sep 04 21:16:04 Carlo-Pi hass[2107]: ValueError: No element found
Sep 04 21:16:11 Carlo-Pi hass[2107]: 2017-09-04 21:16:11 INFO (Thread-8) [myusps] attempting to
get package data

[mezz64]

@CCOSTAN Default camera scan interval is 10s and that only sets how often the images are cycled through, not when they are fetched. The update calls to the actual usps website are only done every 30 min, or at least they should be. Are you sure you're seeing usps related errors every 5 seconds?

[CCOSTAN]

Yes. It's the camera object barfing I believe. @mezz64

[CCOSTAN]

Here is the full log. Every 5 seconds, this message appears in the log.

2017-09-05 15:56:13 INFO (Thread-8) [myusps] attempting to get package data
2017-09-05 15:56:17 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.usps_mail fails
Traceback (most recent call last):
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/helpers/entity.py", line 225, in async_update_ha_state
    yield from self.hass.async_add_job(self.update)
  File "/usr/lib/python3.4/asyncio/futures.py", line 388, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.4/asyncio/tasks.py", line 286, in _wakeup
    value = future.result()
  File "/usr/lib/python3.4/asyncio/futures.py", line 277, in result
    raise self._exception
  File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/sensor/usps.py", line 107, in update
    self._usps.update()
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/util/__init__.py", line 306, in wrapper
    result = method(*args, **kwargs)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/usps.py", line 80, in update
    self.packages = myusps.get_packages(self.session)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/myusps/__init__.py", line 177, in wrapped
    return function(*args)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/myusps/__init__.py", line 208, in get_packages
    dashboard = _require_elem(response, DASHBOARD_TAG, DASHBOARD_ATTRS)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/myusps/__init__.py", line 89, in _require_elem
    raise ValueError('No element found')
ValueError: No element found

[happyleavesaoc]

Is your My USPS dashboard normal looking?

[CCOSTAN]

Yes. 3 pieces of mail. @happyleavesaoc

[happyleavesaoc]

@CCOSTAN Sorry, I meant your package dashboard. The error is originating from the package request, not the mail request.

[CCOSTAN]

That was it @happyleavesaoc - I didn't realize that was a requirement and I was not opted in. I opted in and everything is working PERFECTLY now! Thanks!

[fronzbot]

OK, a bit of a delay but here's a follow up on the "Error Loading Image" after about a day. The scraper is properly grabbing the mail and packages I expect, but does not display an image. Looking in the log with debug turned on for camera only (hard to do much more since I have to wait a full 24 hours from restart before the problem crops up- my log is already at 9MB....) I see the following for today's mail. Note that the image link is correct (though redacted) but the image never displays in home assistant. Also, that traceback repeats over and over and over again in my log, if that means anything to someone.

Now, full disclosure, I have a ton of other stuff in my hass install so it's possible that the traceback is an error for another platform that coincidentally happens seconds after the usps image request, but... I think that's unlikely 😄

2017-09-07 15:18:19 DEBUG (Thread-11) [homeassistant.components.camera.usps] Fetching article image: {'date': '2017-09-07', 'id': '<REDACTED>', 'image': '<REDACTED>'}
2017-09-07 15:18:21 ERROR (MainThread) [aiohttp.server] Unhandled exception
Traceback (most recent call last):
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/aiohttp/web_protocol.py", line 435, in start
    yield from resp.write_eof()
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/aiohttp/web_response.py", line 583, in write_eof
    yield from super().write_eof(body)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/aiohttp/web_response.py", line 420, in write_eof
    yield from self._payload_writer.write_eof(data)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/aiohttp/http_writer.py", line 282, in write_eof
    yield from self.drain(True)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/aiohttp/http_writer.py", line 295, in drain
    yield from self._stream.drain()
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/aiohttp/http_writer.py", line 128, in drain
    yield from self._protocol._drain_helper()
  File "/usr/lib/python3.4/asyncio/streams.py", line 197, in _drain_helper
    yield from waiter
  File "/usr/lib/python3.4/asyncio/futures.py", line 388, in __iter__
    yield self  # This tells Task to wait for completion.
concurrent.futures._base.CancelledError

[arsaboo]

@fronzbot Those errors have already been logged #8603

[brianjking]

Yeah @jcconnell it looks like it's broken as mentioned by @happyleavesaoc in #9143 (comment) and happyleavesaoc/python-myusps#9.

Hopefully @happyleavesaoc can find some time to try and figure this out or someone can step in and help him diagnose/fix the issue. Unfortunately, this surpasses my abilities to fix. 😭

[happyleavesaoc]

I tried a couple times now. I couldn't fix it. Either needs a fresh set of eyes, or we should remove this component.

[jcconnell]

@happyleavesaoc It does appear that their URL structure has changed. How did you identify all of the URL strings in init.py?

[brianjking]

I certainly hope this component isn't removed. Definitely willing to help in whatever way I can to get this working.

[happyleavesaoc]

@jcconnell some have changed, but I accounted for that. I used the browser's developer tools to examine the traffic.

[mezz64]

I also took a few whacks at fixing this last week and couldn't get things working. I'll give it another go this weekend.

[raccettura]

Maybe long term it's better to look at the web tools api usps offers. Looks like it's free but requires signup. I doubt they offer informed delivery, but I didn't dig to deep.

https://www.usps.com/business/web-tools-apis/welcome.htm#GeneralWebToolsFAQs

[happyleavesaoc]

Last I checked, their official API does not offer the functionality that Informed Delivery does. That includes packages and mailpieces. It's aimed at ecommerce shops rather than customers.

[txNgineer]

That is right. Currently they show:

Once registered, you will automatically receive an email containing your assigned Web Tools User ID. After you receive this email, you can use the following APIs:

Price Calculators
Package Tracking
Service Standards & Commitments
Address Information
Package Pickup

It looks like scraping https://informeddelivery.usps.com/box/pages/secure/DashboardAction_input.action?restart=1 is the only way to get the data. That approach will always be subject to any whims of change they have.

While that might be ok for a dev or very advanced user, it seems that it would not be good for the "typical" code base.

[twist2k]

Finally get informed delivery setup and now its broken, how sad haha.

[mezz64]

I made another go at fixing this over the weekend and had no luck. The site code seems to indicate it's just a simple ajax post call on the initial auth URL, but there is clearly something else going on that I can't figure out.

Unless someone else has better luck i'm afraid I agree we should remove it for now.

[brianjking]

Decided to try to disable the USPS Camera option to see if the package tracking would output another error. It looks like it does output a different error, perhaps this one can be resolved and we can at least keep the package monitoring aspect of Informed Delivery in Home Assistant.

@happyleavesaoc @mezz64 -- Any ideas?

I did notice that I do not have a usps_cookies.pickle file created and that the usps_cache.sqlite hasn't been updated since 31 October 2017.

2017-11-22 18:10:25 ERROR (MainThread) [homeassistant.setup] Error during setup of component usps
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/setup.py", line 193, in _async_setup_component
    component.setup, hass, processed_config)
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/usps.py", line 46, in setup
    import myusps
  File "/srv/homeassistant/lib/python3.5/site-packages/myusps/__init__.py", line 12, in <module>
    import requests_cache
  File "/srv/homeassistant/lib/python3.5/site-packages/requests_cache/__init__.py", line 28, in <module>
    from .core import(
  File "/srv/homeassistant/lib/python3.5/site-packages/requests_cache/core.py", line 21, in <module>
    ver = tuple(map(int, requests.__version__.split(".")))
AttributeError: module 'pip._vendor.requests.__version__' has no attribute 'split'

[mezz64]

I'm not sure why that error is generated, but it's not related to the authentication issue. The camera component didn't change anything in the underlying myusps library so I'm afraid disabling it won't make a difference.

[brianjking]

😢 That sucks, thanks for the quick reply. How are the Android & iOS apps getting tracking details for USPS packages, or even services such as 17track.net?

[mezz64]

The official api allows for package lookups if you supply the tracking #, I'd assume that's what 17track is using. I'll look into doing a capture of the official app calls to see if that's an alternate way in, looks like it requires phone # verification though so it may not be practical.

[happyleavesaoc]

@brianjking Neither the mailpieces or packages portion will work, since they both rely on login.

Which mobile apps?

Generally there are two methods:

1. You have to enter the tracking numbers -- the USPS API supports that -- this is not discovering tracking numbers based on address, unfortunately.
2. Other services monitor your email address for emails related to packages, once your turn on email notifications.

[mezz64]

@happyleavesaoc There is an Informed Delivery android app. I installed it on an older pre-Nougat phone so I could trust the Charles SSL cert and ran all the network traffic through the proxy. Unfortunately, I don't see anything obvious relating to the login procedure. It doesn't show the same calls as on the web, but the phone number verification seems to be adding in a layer that I can't capture.

[raccettura]

Poking around, it looks like it's a POST to:
https://reg.usps.com/entreg/json/AuthenticateAction

With a payload of:

struts.token.name:token
token: [string]
route:
username:[username]
password:[password]
newPassword:
retypeNewPassword:

the newpassword fields are empty during a login. That returns a 200 if successful.

There's also this:
https://reg.usps.com/entreg/LoginAction

Which seems to happen after, but doesn't seem to do anything other than redirect to usps.com.

[happyleavesaoc]

@raccettura When I do that, I get a 200 response, but an error in the response JSON, and the expected cookies are not set. Does this differ from your experience?

[brianjking]

@raccettura @happyleavesaoc any updates or anything I can do to help here? Thanks & Happy Holidays!

[raccettura]

So now I'm seeing this when debugging in a browser:
https://reg.usps.com/entreg/LoginAction
accepts form data. and returns a 302 -> www.usps.com with cookies.

[brianjking]

Thanks @raccettura Maybe this will help @happyleavesaoc or someone else figure this out. One of my favorite components/automations in Home Assistant, unfortunately debugging this further is way out of my expertise.

Thanks & Happy Holidays!

[raccettura]

Playing around with this a little more, I think what USPS really cares about is the request to AuthenticateAction

I haven't identified which, but it seems like it only returns 200 success with login cookies when cookies are sent as part of the request (along with user/password). Could they be using cookies as another token perhaps to prevent csrf?

[happyleavesaoc]

@raccettura Yes, that's likely the issue. See more discussion here happyleavesaoc/python-myusps#9

[happyleavesaoc]

Fixed in #12465