How to use Kamaji with ETCD-shim via Kine (AWS RDS - Postgres)?

[Rezkmike]

I'm migrating the current Kamaji setup from using the default datastore to Kine-Postgres datastore

Since the Kamaji is deployed on AWS EKS, AWS RDS is a good choice.

But having problems retrieving their ca.key file and generating my client certs based on the ca. crt is not working.

AWS RDS CA Cert : https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html

Probably my implementation was not correct. Anyone can help?

[Rezkmike]

found this article, but not cover AWS RDS.

if you are using your own Postgres DB can use this

https://kamaji.clastix.io/guides/alternative-datastore/

[prometherion]

I know @jwitko was giving a try to AWS RDS, not sure if he can share some insights.

[jwitko]

I was supposed to create a bug (and many other issues) but I still haven't had the chance to circle back. I do still intend to though.

[prometherion]

Thanks for the detailed bug report and for sharing all your findings.

I'm not an expert on AWS RDS, if we could have more insights from SMEs we can work collaboratively on a fix.

• There is a misleading 'requirement' to present a CA key to RDS. This is actually not required. It is required in the CRD

CA is required to ensure the connection is secured.

[Rezkmike]

my workaround for this issue.

run the below command to give ownership to the created rds-user

ALTER DATABASE <TENANT-DB>
OWNER TO <RDS-USER>;

then it can delete the tenant-cluster successfully

[prometherion]

Thanks for the quick report! I'll try to give it a spin on regular Pgsql backed by CloudnativePG, if no side-effects are occurring, we can release a patch.

[prometherion]

If someone is up for a quick test, #461 should be able to support AWS RDS.

[log1cb0mb]

Just noticed this discussion. Seems similar requirement proposed here: #422

So yes, this cannot work currently as Datastore tlsConfig is fixed and tries to perform client cert-based authentication against DB provider which will not work since not supported. The current implementation is basically expecting a "self managed" database/cluster
where its configured with signed TLS certs (self signed or otherwise) by the administrator.

For one, in order to perform cert based auth, a client certificate will need to be signed by AWS RDS public CAs and secondly, it just does not support cert-based auth at all:

[Rezkmike]

you may create the datastore (postgres) manually without ca.key, and set the datastore in the Kamaji control plane to use Postgres.

[log1cb0mb]

I dont think that is possible, tlsConfig specs keys under datastore CRD , certificateAuthority and clientCertificate are required fields. CR admission and the Kamaji webhook wouldnt accept the resource without it.

[prometherion]

This has been addressed with the v0.6.0 release: tlsConfig is no more mandatory, as well as clientCertificate

[log1cb0mb]

as @jwitko confirmed over slack that this can work by providing RDS public CA cert under Datastore tlsConfig.CA and feeding bogus tlsConfig.clientCertificate information. Since RDS or any managed DB endpoint is not expecting client's cert-based authentication, simple CA verification should do the job.

[prometherion]

This has been addressed with the v0.6.0 release! 🚀

[zazathomas]

@prometherion
I know this is a closed discussion but I'm trying to use a managed postgresql db as my preferred datastore but I'm running into issues.
This is the sample datastore manifest I'm trying to apply:

apiVersion: kamaji.clastix.io/v1alpha1
kind: DataStore
metadata:
  name: neon-pg-datastore
spec:
  driver: PostgreSQL
  endpoints:
    - host:5432
  basicAuth:
    username:
      secretReference:
        name: neon-pg-secret
        namespace: kamaji-system
        keyPath: username
    password:
      secretReference:
        name: neon-pg-secret
        namespace: kamaji-system
        keyPath: password
  tlsConfig:
    certificateAuthority:
      certificate:
        secretReference:
          name: postgres-default-ca
          namespace: kamaji-system
          keyPath: ca.crt

This is the error I'm getting:
Error from server: error when creating "pg-datastore.yaml": admission webhook "vdatastore.kb.io" denied the request: panic: runtime error: invalid memory address or nil pointer dereference [recovered]

[prometherion]

We had a similar issue a few days ago, may I ask you to give latest a try?

If issue persists, let's open a bug report along with Kamaji's logs and version.

[zazathomas]

@prometherion Sure will do

[zazathomas]

@prometherion
Created an issue to track this here
#744