[FEATURE] Allow Response Objects and Exceptions to Access Http Response Headers

[fddattal]

Is your feature request related to a problem?

A clear and concise description of what the problem is, e.g. I'm always frustrated when [...].

I am using the OpenSearchClient with the AwsSdk2Transport, and I cannot retrieve the http headers in the response. This makes it very hard for me to debug issues with my service provider as I cannot provide them request ids.

What solution would you like?

A clear and concise description of what you want to happen.

I would like for two changes:

1. All response headers should be available on response objects exposed via a method such as getResponseHeaders().
2. All exceptions thrown because of a response error should also include the http headers from that response. I propose a class: OpenSearchResponseException to encapsulate this behavior.

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

I could not figure out a way to accomplish this by subclassing and overriding code in this package. I would need to migrate to a RestClient or ApacheHttpClient, implement Sigv4 signing, and do my own response parsing.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

Here looks to be where the response is de-serialized and where headers are lost:

https://github.com/opensearch-project/opensearch-java/blob/main/java-client/src/main/java/org/opensearch/client/transport/aws/AwsSdk2Transport.java#L619-L636

[Xtansia]

Thanks for the feature suggestion @fddattal, I can certainly see the value.

Is this something you'd be interested in helping to contribute?

[fddattal]

@Xtansia I'm currently swamped, so I may not be able to get to it immediately. I'll check back in next week and let you know.

[Xtansia]

@fddattal No problem, completely understand that, there's no pressure

[Xtansia]

I can think of a couple more immediately accessible options, though neither ideal:

1. If you need true programmatic access to the response headers, you could utilize the generic client which gives access to them, but isn't as clean to work with and requires changing each location you call the client.
2. If only logging is required, you could PR a minor change to make AwsSdkTransport::parseResponse have protected visibility rather than private then in your own codebase extend from AwsSdk2Transport and add the logging in an override.

[georgereuben]

Hi @Xtansia ! I'm interested in working on this feature. Can I take up this issue?

[Xtansia]

@georgereuben Certainly feel free to work on this

[georgereuben]

Hi @Xtansia @fddattal , I had been working on a few changes that might make it possible to access the response headers. I tried using a dynamic proxy that would handle getResponseHeaders() calls but that was not compatible since the response classes are generated.
Two other options I could think of are to update the codegen templates to include a ResponseHeaders interface, or to use wrapper classes with response data and headers. Please let me know which one seems feasible or if there are any other approaches.

[fddattal]

I'm more include towards updating the codegen. I think this option we yield more debuggable and maintainable code.

[georgereuben]

Yeah sounds good in the long run. Let me see what I can do with the current templates