Skip to content

Conversation

@davida72
Copy link
Contributor

@davida72 davida72 commented May 19, 2025

Overview

This PR changes the Home Assistant setup process, which I hope makes for a more intuitive onboarding process while maintaining full compatibility with existing configurations. The changes include more checks, pre-population of fields and clear error messages.

The setup is now a four step process:

  1. Name and council
  2. Council specific information
  3. Selenium configuration (where required)
  4. Advanced settings

Name and Council

  • Automatic detection of property using Home Assistant lat/lon allows us to pre-populate name and council. Uses Google Geocode API for street, admin ward and postcode. Postcodes.io API for the LAD24CD code.
  • Dynamic description will tell the user their council has been detected, or ask for their council to be added.
  • If no council is found, then the user can select the council as required.

Council Specific Information

  • wiki_note from input.json is used as a description, so no need to refer to the wiki.
  • If a URL is required, then it will offer up the sample URL as defined in input.json.
  • Checks that URL has been changed from the sample URL before continuing.
  • Only asks for a URL where necessary.
  • If a postcode is required, then it will pre-populate with the postcode auto-detected from Home Assistant lat/lon.

Selenium

  • Searches for working Selenium URLs and presents a working one by default.
  • Checks user-entered Selenium URL before continuing.
  • Checks Chromium when checked by the user.

Advanced

  • Automatically refresh brought to this section.
  • JSON settings moved to this section.

Configure

  • Used a similar flow to first-time setup.
  • Added a feature to use test settings, taken from input.json, for troubleshooting purposes.

Housekeeping

  • Updated Readme.
  • Help button now goes to the Readme.
  • Cleaned up input.json to remove unnecessary wiki_command_url_override.
  • Built new tests and fixed some sensor and calendar tests. 100% pass.
  • Used an input.json that will always be from the master version.
  • Split into more manageable files.

Notes

  • The code and readme has confusion around manual_refresh vs automatic_refresh, but I left them as is.
  • I've used the presence of wiki_command_url_override in the council data as the method of establishing whether the user needs to enter a URL, but perhaps this isn't the best method.
  • We use the presence of web_driver in the council data as the method of establishing whether the council script needs Selenium, but there may be a better method.
  • I think there's a better way of handling councils that use original_parser, but I'll leave that for another occasion.
  • input.json would benefit from some more work.

Council is auto-detected in Step 1, using auto-detected street name as default name.

ha-auto-select-council

Or reverts to a fallback if no lat/lon or no council is found.

ha-council-not-listed

In Step 2, only necessary information is asked from the user.

ha-uprn

It will pre-populate the auto-detected postcode.

ha-postcode

It will offer up a placeholder URL taken from input.json if a URL is required.

ha-url

But it will check that it's been modified before moving on.

ha-url-error

Step 3 are the Selenium checks, if the council requires it. It pre-populates with the auto-detected Selenium server.

ha-selenium

It won't let the user progress without a working Selenium server.

ha-selenium-error

Or a working Chromium.

ha-chromium-error

Step 4 is Advanced Settings.

ha-advanced

Doing checks on timeout value.

ha-timeout-error

And update frequency value.

ha-update-error

On Configure, the user has the option of using test settings taken from input.json

ha-test-settings

if latitude == 0 and longitude == 0:
_LOGGER.warning("Home location not set in Home Assistant configuration")
else:
_LOGGER.debug("Fetching property info for coordinates: (%s, %s)", latitude, longitude)

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (private)
as clear text.
if latitude == 0 and longitude == 0:
_LOGGER.warning("Home location not set in Home Assistant configuration")
else:
_LOGGER.debug("Fetching property info for coordinates: (%s, %s)", latitude, longitude)

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (private)
as clear text.

# Make get a regular method that returns a response object, not a coroutine
def get(self, url, **kwargs):
if "maps.googleapis.com" in url:

Check failure

Code scanning / CodeQL

Incomplete URL substring sanitization High test

The string
maps.googleapis.com
may be at an arbitrary position in the sanitized URL.
@codecov
Copy link

codecov bot commented May 19, 2025

Codecov Report

Attention: Patch coverage is 71.26866% with 154 lines in your changes missing coverage. Please review.

Project coverage is 79.30%. Comparing base (771dcae) to head (db5e3d8).
Report is 18 commits behind head on master.

Files with missing lines Patch % Lines
custom_components/uk_bin_collection/utils.py 77.27% 45 Missing ⚠️
custom_components/uk_bin_collection/config_flow.py 68.36% 31 Missing ⚠️
...ustom_components/uk_bin_collection/options_flow.py 76.92% 27 Missing ⚠️
...stom_components/uk_bin_collection/property_info.py 62.12% 25 Missing ⚠️
...tom_components/uk_bin_collection/initialisation.py 48.88% 23 Missing ⚠️
custom_components/uk_bin_collection/__init__.py 70.00% 3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (771dcae) and HEAD (db5e3d8). Click for more details.

HEAD has 5 uploads less than BASE
Flag BASE (771dcae) HEAD (db5e3d8)
6 1
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1450      +/-   ##
==========================================
- Coverage   86.82%   79.30%   -7.53%     
==========================================
  Files           9       13       +4     
  Lines        1131     1324     +193     
==========================================
+ Hits          982     1050      +68     
- Misses        149      274     +125     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@robbrad
Copy link
Owner

robbrad commented May 19, 2025

thanks @davida72

Couple of questions-

1/ what’s the upgrade/ reconfigure path for existing users? Any breaking changes here?

2/ Are we are increasing our dependency on 3rd party apis with this approach ? Is selecting the council from a drop down an issue that needs to be fixed? Flip side of this approach is people can’t select a council they are forced to choose the one their lat long dictates.

I’ll do some more checking when I have some more time

@davida72
Copy link
Contributor Author

What’s the upgrade/ reconfigure path for existing users? Any breaking changes here?

Nothing to upgrade or reconfigure. I was careful to be consistent.

Are we are increasing our dependency on 3rd party apis with this approach ?

Good question. If the API calls fail for any reason then we revert back to the previous system automatically. If it's available we'll use it. If it isn't we won't.

Is selecting the council from a drop down an issue that needs to be fixed? Flip side of this approach is people can’t select a council they are forced to choose the one their lat long dictates.

I should have made clear that it just defaults to the auto-detected council, but the user is free to choose another. My hope is that it makes it easy for a user to find their council, whatever it's called.

@robbrad
Copy link
Owner

robbrad commented May 19, 2025

What’s the upgrade/ reconfigure path for existing users? Any breaking changes here?

Nothing to upgrade or reconfigure. I was careful to be consistent.

Are we are increasing our dependency on 3rd party apis with this approach ?

Good question. If the API calls fail for any reason then we revert back to the previous system automatically. If it's available we'll use it. If it isn't we won't.

Is selecting the council from a drop down an issue that needs to be fixed? Flip side of this approach is people can’t select a council they are forced to choose the one their lat long dictates.

I should have made clear that it just defaults to the auto-detected council, but the user is free to choose another. My hope is that it makes it easy for a user to find their council, whatever it's called.

I can't see how the user won't need to upgrade? Do all the fields apply the same names as before?

@davida72
Copy link
Contributor Author

I can't see how the user won't need to upgrade? Do all the fields apply the same names as before?

Yes. What's loaded and saved is exactly the same. It's just the flow that's different.

@davida72
Copy link
Contributor Author

davida72 commented May 20, 2025

Here's a comparison of what's saved to core.config_entries in Home Assistant. Four different types of councils. In the case of the Selenium URL, the difference is just down to my personal setup.

JSON Data Comparison: Brighton Old vs Brighton New

Field Brighton - Old Brighton - New Match
created_at 2025-05-20T07:21:05.928960+00:00 2025-05-20T07:19:03.526159+00:00
data.council BrightonandHoveCityCouncil BrightonandHoveCityCouncil
data.headless true true
data.icon_color_mapping "" (empty string) "" (empty string)
data.local_browser false false
data.manual_refresh_only true true
data.name Brighton - Old Brighton - New
data.number "44" "44"
data.postcode "BN1 8NE" "BN1 8NE"
data.skip_get_url true true
data.timeout 60 60
data.update_interval 12 12
data.url https://cityclean.brighton-hove.gov.uk/link/collections https://cityclean.brighton-hove.gov.uk/link/collections
data.web_driver http://192.168.200.97:4444 http://selenium:4444/
disabled_by null null
discovery_keys {} (empty object) {} (empty object)
domain uk_bin_collection uk_bin_collection
entry_id 01JVP92HM8VJBW8FBY5SPBQN6J 01JVP8YT35XYC1P6AK2BJ4HAPM
minor_version 1 1
modified_at 2025-05-20T07:21:05.928989+00:00 2025-05-20T07:19:03.526167+00:00
options {} (empty object) {} (empty object)
pref_disable_new_entities false false
pref_disable_polling false false
source user user
subentries field not present [] (empty array)
title Brighton - Old Brighton - New
unique_id null null
version 3 3

Summary of Differences

6 fields differ between the two objects:

  1. created_at: Brighton - Old was created about 2 minutes later than Brighton - New
  2. data.name: "Brighton - Old" vs "Brighton - New"
  3. data.web_driver: Different endpoints - IP address vs hostname ("selenium:4444/")
  4. entry_id: Different unique identifiers
  5. modified_at: Different modification timestamps (correlate with creation times)
  6. subentries: Only present in Brighton - New as an empty array

19 fields are identical between both objects.


JSON Data Comparison: Clackmannanshire Old vs Clackmannanshire New

Field Clackmannanshire - Old Clackmannanshire - New Match
created_at 2025-05-20T07:22:43.562227+00:00 2025-05-20T07:22:03.088756+00:00
data.council Clackmannanshire Clackmannanshire
data.icon_color_mapping "" (empty string) "" (empty string)
data.manual_refresh_only true true
data.name Clackmannanshire - Old Clackmannanshire - New
data.original_parser GooglePublicCalendarCouncil GooglePublicCalendarCouncil
data.timeout 60 60
data.update_interval 12 12
data.url https://calendar.google.com/calendar/ical/0d775884b4db6a7bae5204f06dae113c1a36e505b25991ebc27c6bd42edf5b5e%40group.calendar.google.com/public/basic.ics https://calendar.google.com/calendar/ical/0d775884b4db6a7bae5204f06dae113c1a36e505b25991ebc27c6bd42edf5b5e%40group.calendar.google.com/public/basic.ics
disabled_by null null
discovery_keys {} (empty object) {} (empty object)
domain uk_bin_collection uk_bin_collection
entry_id 01JVP95GZA8754KQJGMJNN7VA6 01JVP949EGQY80R8T4SEM7RGNS
minor_version 1 1
modified_at 2025-05-20T07:22:43.562251+00:00 2025-05-20T07:22:03.088758+00:00
options {} (empty object) {} (empty object)
pref_disable_new_entities false false
pref_disable_polling false false
source user user
subentries field not present [] (empty array)
title Clackmannanshire - Old Clackmannanshire - New
unique_id null null
version 3 3

Summary of Differences

6 fields differ between the two objects:

  1. created_at: Clackmannanshire - Old was created about 40 seconds later than Clackmannanshire - New
  2. data.name: "Clackmannanshire - Old" vs "Clackmannanshire - New"
  3. entry_id: Different unique identifiers
  4. modified_at: Different modification timestamps (correlate with creation times)
  5. subentries: Only present in Clackmannanshire - New as an empty array
  6. title: "Clackmannanshire - Old" vs "Clackmannanshire - New"

16 fields are identical between both objects.

Note: Unlike the Brighton comparison, these Clackmannanshire configurations use Google Calendar as the data source (GooglePublicCalendarCouncil parser) rather than web scraping, and don't include web driver or browser-related settings.


JSON Data Comparison: Huntingdonshire Old vs Huntingdonshire New

Field Huntingdonshire - Old Huntingdonshire - New Match
created_at 2025-05-20T07:25:35.546116+00:00 2025-05-20T07:26:16.652458+00:00
data.council HuntingdonDistrictCouncil HuntingdonDistrictCouncil
data.icon_color_mapping "" (empty string) "" (empty string)
data.manual_refresh_only true true
data.name Huntingdonshire - Old Huntingdonshire - New
data.skip_get_url true true
data.timeout 60 60
data.update_interval 12 12
data.uprn "10012048679" "10012048679"
data.url http://www.huntingdonshire.gov.uk/refuse-calendar/ http://www.huntingdonshire.gov.uk/refuse-calendar/
disabled_by null null
discovery_keys {} (empty object) {} (empty object)
domain uk_bin_collection uk_bin_collection
entry_id 01JVP9ARXTQD15DR1SBKB4PK9V 01JVP9C12CT2D8PHQ4KZS25ABG
minor_version 1 1
modified_at 2025-05-20T07:25:35.546140+00:00 2025-05-20T07:26:16.652460+00:00
options {} (empty object) {} (empty object)
pref_disable_new_entities false false
pref_disable_polling false false
source user user
subentries field not present [] (empty array)
title Huntingdonshire - Old Huntingdonshire - New
unique_id null null
version 3 3

Summary of Differences

6 fields differ between the two objects:

  1. created_at: Huntingdonshire - New was created about 41 seconds later than Huntingdonshire - Old
  2. data.name: "Huntingdonshire - Old" vs "Huntingdonshire - New"
  3. entry_id: Different unique identifiers
  4. modified_at: Different modification timestamps (correlate with creation times)
  5. subentries: Only present in Huntingdonshire - New as an empty array
  6. title: "Huntingdonshire - Old" vs "Huntingdonshire - New"

16 fields are identical between both objects.

Note: These configurations include a UPRN (Unique Property Reference Number) "10012048679" for property identification.


JSON Data Comparison: Ipswich Old vs Ipswich New

Field Ipswich - Old Ipswich - New Match
created_at 2025-05-20T07:28:22.726984+00:00 2025-05-20T07:27:47.267565+00:00
data.council IpswichBoroughCouncil IpswichBoroughCouncil
data.icon_color_mapping JSON object with Recycling, Rubbish, Garden Waste icons/colors JSON object with Recycling, Rubbish, Garden Waste icons/colors
data.manual_refresh_only true true
data.name Ipswich - Old Ipswich - New
data.number "Siloam Place" "Siloam Place"
data.timeout 60 60
data.update_interval 12 12
data.url https://app.ipswich.gov.uk/bin-collection/ https://app.ipswich.gov.uk/bin-collection/
disabled_by null null
discovery_keys {} (empty object) {} (empty object)
domain uk_bin_collection uk_bin_collection
entry_id 01JVP9FW6696437DSAE0G9CF6Z 01JVP9ESJ3HKK51S86K5T7E7V6
minor_version 1 1
modified_at 2025-05-20T07:28:22.727009+00:00 2025-05-20T07:27:47.267568+00:00
options {} (empty object) {} (empty object)
pref_disable_new_entities false false
pref_disable_polling false false
source user user
subentries field not present [] (empty array)
title Ipswich - Old Ipswich - New
unique_id null null
version 3 3

Summary of Differences

6 fields differ between the two objects:

  1. created_at: Ipswich - Old was created about 35 seconds later than Ipswich - New
  2. data.name: "Ipswich - Old" vs "Ipswich - New"
  3. entry_id: Different unique identifiers
  4. modified_at: Different modification timestamps (correlate with creation times)
  5. subentries: Only present in Ipswich - New as an empty array
  6. title: "Ipswich - Old" vs "Ipswich - New"

15 fields are identical between both objects.

Note: These configurations include custom icon color mappings for different bin types (Recycling=black, Rubbish=green, Garden Waste=brown) and use "Siloam Place" as the address identifier.

@davida72
Copy link
Contributor Author

Three other points:

  • To test the lat/lon lookup, you can un-remark the lat/lon numbers at the start of property_info.py.
  • The github-advanced-security points out that there's a debug log of lat/lon which should be removed.
  • I've used this link as the source of truth for input.json, which will always be the master. My thought was that it avoids having to update the URL each time there's a new version, but perhaps this isn't desirable.

The URL is stored in const.py
COUNCIL_DATA_URL = "https://raw.githubusercontent.com/robbrad/UKBinCollectionData/refs/heads/master/uk_bin_collection/tests/input.json"

@robbrad
Copy link
Owner

robbrad commented May 21, 2025

@davida72 I had a bit of time to test this, but I need more time to fully test it.

Initial feedback (some opinionated sorry)

1/ The language translations are missing in custom_components/uk_bin_collection/translations.
2/ Where have the checks for Selenium and local Chrome gone?
3/ I don’t want to create a dependency on a location lookup—selecting a council from a drop-down is easy enough
4/ If I’m being honest, I don’t find it much simpler than the original approach. A simpler approach would be to have fewer forms: one to fetch and select the council list, and a final form to fill out. I find the step-by-step (form-by-form) approach more annoying than seeing everything I need in one place - but thats just me
5/Please when raising changes like this raise an issue first - and reference it . eg feature X or Bug y

Ill do some more testing and can see you have put in some SERIOUS work on this so it is appreciated

@davida72
Copy link
Contributor Author

Appreciate you taking a look. I know you're busy.

My motivation for the changes was that I found initial setup as a user too complicated. It asked for information it didn't need, the flow was confusing in places, and the documentation (in my case) was wrong. I felt that it undersold the huge body of work underneath it. So my efforts in re-working the flow were with those considerations in mind: asking only for the information it needs, putting dynamic instructions in the flow, pre-populating where possible.

1/ The language translations are missing in custom_components/uk_bin_collection/translations.

The translation files could be created fairly easily, although in places I'm using text from input.json which is in English only. The existing setup uses hard-coded English strings in places, I think, e.g. Selenium and Chromium checks.

2/ Where have the checks for Selenium and local Chrome gone?

check_selenium_server and check_chromium_installed in utils.py. They are used during initialisation and on the Selenium step.

3/ I don’t want to create a dependency on a location lookup—selecting a council from a drop-down is easy enough

The API calls get postcode, lad24cd, street name, town and admin ward, which can be used to create a degree of automation and avoid errors during the setup. It's not just used in the drop-down. To me this is a notable improvement, but I sense that you don't want to use an API. I'd query the word 'dependency' since the flow works fine if either of the APIs fail, but maybe there's just something about using a third-party service by default that you don't like. Would you feel more comfortable if it was a choice made by the user? e.g. a checkbox that asked Automatically detect my location.

On the point of the drop-down, some councils use an official name that might be different from the one that a user expects, e.g. Hull City Council is officially called Kingston upon Hull.

4/ If I’m being honest, I don’t find it much simpler than the original approach. A simpler approach would be to have fewer forms: one to fetch and select the council list, and a final form to fill out. I find the step-by-step (form-by-form) approach more annoying than seeing everything I need in one place - but thats just me

My feeling is that if you understand what it's asking and why, then the form is fine. I found it overwhelming as a first-time-user and it was a frustrating experience trying to get it to work. It asked for a URL it didn't need; it told me to enter information that was incorrect; it didn't check my Selenium server. I agree that it's personal preference, but I prefer the step-by-step approach of getting one set of information correct before moving on to the next step; besides - it didn't all fit on my screen!

I'm ambivalent about this though. Most of my frustrations could be resolved with better messaging and better checks.

5/Please when raising changes like this raise an issue first - and reference it . eg feature X or Bug y

Got it.

@robbrad
Copy link
Owner

robbrad commented May 22, 2025

@davida72 - just to validate - when I tried a Selenium based council the checks wernt there in the form - sorry to confirm but were these removed?

@davida72
Copy link
Contributor Author

I see what you mean. You're asking where they are in the UI, not in the code.

If a known, working Selenium is found then it will pre-populate it in the input box. If a user tries to progress without one then it notifies them, as per the screenshot earlier in this thread.

My motivation for this was my experience in setting this step up for the first time.

❌ Selenium not found at http://localhost:4444
❌ Selenium not found at http://selenium:4444
❌ Chromium not found

This was all true, but unhelpful. I knew I didn't have Selenium on those URLs and I knew I didn't have Chromium. Selenium was set up at http://mini:4444 for me. I entered that in and continued, but there was no check to see if it worked. After a lot of trial and error I found out that on my Home Assistant machine mini wasn't resolving to its IP address.

So in the revised flow, there's an additional check going out of the form to see if the user-entered Selenium URL works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants