retain canary-by-header* nginx annotations when creating the canary ingress

[sding3]

Describe the feature

During a weighted canary rollout, it's useful for debugging efforts to be able to specifically target the canary deployments using http headers. Without this, the debugging effort would have to resort to sending large number of requests (given the current canary weight) to land some requests on new version of the service being rolled out.

Nginx supports nginx.ingress.kubernetes.io/canary-weight and nginx.ingress.kubernetes.io/canary-by-header on the same ingress (link to doc), but this does not work if two ingresses are at play where one ingress is doing weight while the other is using header, which precludes the possibility of me creating another separate canary ingress with canary-by-header.

Flagger, when it copies the application's existing ingress to the <ingress>-canary ingress, it specifically ignores the nginx.ingress.kubernetes.io/canary-by-header* annotations, which means any nginx.ingress.kubernetes.io/canary-by-header* annotations placed by me on the original ingress would get lost when flagger creates the canary ingress.

I've also tried to use analysis.match.headers to get flagger to create the header match conditions on the ingress that it creates, but it appears that analysis.match.headers is mutually exclusive with weighted deployments.

Proposed solution

For Nginx weighted canary deployments, when creating/updating the canary ingress, flagger should include the following annotations:

nginx.ingress.kubernetes.io/canary-by-header
nginx.ingress.kubernetes.io/canary-by-header-value
nginx.ingress.kubernetes.io/canary-by-header-pattern

Any alternatives you've considered?

I understand that I could instead create a separate ingress that target the canary service using different hostnames or HTTP path prefixes (as well as targeting the canary service from inside of the cluster), but it would be more convenient for us to target the canary via headers given our current setup/conventions.