Manually updated/deleted Pulsar resources should be reconciled if a CRD resource exists for them

[mattsre]

As of v0.14.0 of the Pulsar resources operator, when a Pulsar resource which is created via a CustomResource is manually updated or deleted outside of k8s (say from the pulsar-admin CLI), the resource is not reconciled back to the desired state as defined in the CustomResource. I'd expect the Pulsar resources operator to be continuously driving these resources towards the intended state, as defined by the CRs.

These cases should be fairly easy to reproduce:

Update Reconciliation

1. Create a Pulsar namespace using the PulsarNamespace CRD and set messageTTL to 6h
2. Update the Pulsar namespace outside of k8s (via pulsar-admin, pulsarctl, console, etc) to have a 12h message TTL

Expected Behavior
message TTL is reconciled back to 6h
Current Behavior
Pulsar namespace is not changed back. Message TTL stays at 12h until the PulsarNamespace CR is updated, after which the message TTL is reset to 6h

Delete Reconciliation

1. Create a Pulsar topic using the PulsarTopic CRD
2. Delete the Pulsar topic manually

Expected Behavior
Pulsar topic is recreated, it has a CR defined for it and should exist
Current Behavior
Pulsar topic is never re-created until the PulsarTopic CR is updated

[david-streamlio]

Can you please share the logs from the Resource operator? It will help us debug the issue. Thanks!

[mattsre]

I've gathered logs from the resources operator with --zap-devel enabled for debug logging. Apologies for the redactions, hopefully it's clear enough to follow. These are taken from a private cloud streamnative cluster, after applying the following resources with kubectl apply -f example.yaml -n pulsar-resources-operator:

apiVersion: resource.streamnative.io/v1alpha1
kind: PulsarConnection
metadata:
  name: redacted
spec:
  adminServiceURL: http://private-cloud-broker.pulsar.svc.cluster.local:8080  
  brokerServiceURL: pulsar://private-cloud-broker.pulsar.svc.cluster.local:6651
---
apiVersion: resource.streamnative.io/v1alpha1
kind: PulsarTenant
metadata:
  name: sn-private-cloud
spec:
  name: redacted
  connectionRef:
    name: redacted
  lifecyclePolicy: CleanUpAfterDeletion
  adminRoles:
  - admin
  - ops
---
apiVersion: resource.streamnative.io/v1alpha1
kind: PulsarNamespace
metadata:
  name: sn-redacted
spec:
  name: redacted/redacted
  connectionRef:
    name: redacted
  lifecyclePolicy: CleanUpAfterDeletion
  backlogQuotaLimitSize: 10Gi
  backlogQuotaLimitTime: 24h
  messageTTL: 6h
---
apiVersion: resource.streamnative.io/v1alpha1
kind: PulsarTopic
metadata:
  name: redacted
spec:
  name: persistent://redacted/redacted/redacted
  connectionRef:
    name: redacted
  inactiveTopicPolicies:
    deleteWhileInactive: false

The logs are attached below. After resources were created I deleted a topic and changed the namespace message TTL from 6h to 12h. The attached logs are from ~1 hour after I made those changes, and it doesn't appear the operator has logged anything since the creation of the initial resources. The topic is still deleted and namespace still changed.

logs.txt