Skip to content

feat: subscribe to the open data registry TDE-1778 & TDE-1860#1142

Open
blacha wants to merge 7 commits intomasterfrom
feat/open-data-queue
Open

feat: subscribe to the open data registry TDE-1778 & TDE-1860#1142
blacha wants to merge 7 commits intomasterfrom
feat/open-data-queue

Conversation

@blacha
Copy link
Member

@blacha blacha commented Feb 26, 2026

Motivation

We have a large number of buckets that all have object_created sns topics on, I would like to trigger argo workflows when STAC documents are modified in each of these buckets

image

Modifications

Swap to a single SQS queue for all bucket notifications
subscribe a few open data registry buckets linz/imagery linz/elevation

Include SQS creation in the EKS stack to ease deployment after modifications

image

NOTE: the SqsQueues stack will have to be destroyed after deployment.

Verification

Will deploy and see what happens! 🎉

@paulfouquet
Copy link
Collaborator

To deploy the changes (this has been verified on a dev environment), it needs a manual deployment before merging the PR to avoid issues with the CI/CD deployment:

  1. Remove SqsQueues stack dependency to the LinzEksCluster stack:
    in infra/cdk.ts, comment //cluster.addDependency(sqsQueuesStack);
    and in infra/eks/cluster.ts comment:
    //const sqsPublishQueueArn = Fn.importValue(CfnOutputKeys.ObjectCreatedSqsQueueArn);
...
    // argoEventsSa.role.addToPrincipalPolicy(
    //   new PolicyStatement({
    //     actions: ['sqs:GetQueueUrl', 'sqs:ReceiveMessage', 'sqs:DeleteMessage', 'sqs:GetQueueAttributes'],
    //     resources: [sqsPublishQueueArn],
    //   }),
    // );
  1. Deploy Workflows
  2. Deploy SqsQueues
  3. Put dependency back: uncomment what has been commented in 1.
  4. Deploy Workflows

@paulfouquet paulfouquet changed the title feat: subscribe to the open data registry feat: subscribe to the open data registry TDE-1778 & TDE-1860 Mar 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants