Skip to content

Ensure package_create resources trigger XLoader submission for datastore and views#265

Open
TomeCirun wants to merge 3 commits intockan:masterfrom
TomeCirun:handle-tabular-resources-in-package-create
Open

Ensure package_create resources trigger XLoader submission for datastore and views#265
TomeCirun wants to merge 3 commits intockan:masterfrom
TomeCirun:handle-tabular-resources-in-package-create

Conversation

@TomeCirun
Copy link
Contributor

@TomeCirun TomeCirun commented Nov 4, 2025

When we harvest a dataset, we call the create_or_update_package function or use package_create with resources. All resources are included inside package_dict, so they aren’t created separately — resource_create is never called. As a result, the xloader hook isn’t triggered, resources aren’t pushed to the datastore, and their default views aren’t created.

To resolve this, I removed the after_resource_create hook and now handle everything within notify.

"resource did not pass validation yet.", resource_dict.get('id'))
return
# Only submit if URL has changed
if not getattr(entity, 'url_changed', False):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about changing the Data Dictionary? Or the resource format? Or removing a schema?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ThrawnCA , thanks for taking the time to review this PR — I really appreciate it. Could you please clarify what the expected output should be? Just to note, this line was copied from the previous logic, so it remains as originally implemented.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. You're right that the logic did include this before. But are you also going to replace after_resource_update, which doesn't include that restriction and applies XLoader to any update?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot about that — sorry, and thanks for pointing it out! I’ll update the PR soon.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ThrawnCA

I moved the logic into a separate function sync_datastore_flag and we now call it inside this check:

if not getattr(entity, 'url_changed', False):

This reduces redundant calls. If the URL has changed, _submit_to_xloader will set the flag on the resource. I also removed the after_resource_update hook.

Please let me know if there’s anything else I should address.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are other fields that should potentially trigger XLoader on change, such as the resource format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants