Response parsing breaks with 'grpc-encoding: gzip'

[ctoomey]

When the gRPC response is gzipped (indicated by the grpc-encoding: gzip response header), the grpc-web stream parser fails with this error (including the first few bytes for reference):

Uncaught Error: The stream is broken @0/0. Error: invalid frame byte. With input:
1,0,0,2,126,31,...

And the grpc-web filter in Envoy unfortunately automatically adds this request header when proxying to the gRPC server: grpc-accept-encoding:identity,deflate,gzip, which in the case of akka-grpc at least causes the server to gzip. The envoy source code has a comment stating that this header 'is required for gRPC'.

When I hacked akka-grpc to ignore the above header and not gzip, the response is properly parsed.

I'm a backend engr., but if someone can give me some pointers I could take a stab at fixing this.

[rodrig0-lima]

I have the same issue ... I had to hack the akka server to override grpc-accept-enconding so the server doesn't gzip the content

[seroperson]

According to this, seems like such compression is redundant.
With envoy you can use request_headers_to_remove (doc) or this to completely remove or modify this header.

[ctoomey]

@seroperson thanks for the pointers. Per this the grpc web client implements the grpc-web protocol spec described in that document, which says nothing about not using the grpc-accept-encoding and grpc-encoding headers. Is that an oversight in the spec that should be fixed? Otherwise Envoy is following the spec and the client library isn't and should be fixed.

[otto-dev]

This cost me some hours. Here is a solution, a bit of code to get others started that run into this issue with Akka HTTP:

    val service = NetworkWorkersHandler(new NetworkWorkersService())
    val handler: HttpRequest => Future[HttpResponse] = { request =>
      val withoutEncoding = request.copy(headers = request.headers.filterNot(_.name == "grpc-accept-encoding"))
      service(withoutEncoding)
    }
    val binding = Http().bindAndHandleAsync(
      handler,
      interface = "127.0.0.1",
      port = 8000,
      connectionContext = HttpConnectionContext(),
    )

[pranaypratyush]

I am getting this exact same problem with a python server when I use a bytes Type in my proto. Is that internally being streamed because my proto is unary only? How do I fix?

[otto-dev]

How do I fix?

A workaround is to remove the grpc-accept-encoding header in the akka-received HTTP request prior to passing it to the generated service. An example of how such a wrapper for a gRPC service can look is my post above.

[pranaypratyush]

How do I fix?

A workaround is to remove the grpc-accept-encoding header in the akka-received HTTP request prior to passing it to the generated service. An example of how such a wrapper for a gRPC service can look is my post above.

But I am using python grpc server with envoy. And not akka.

[otto-dev]

Oh sorry, I remembered this as an akka-related issue and didn't see the repo in the notification. I guess the solution may be similar, the implementation different.

[akan]

now my grpc serice server set context->set_compression_algorithm(GRPC_COMPRESS_GZIP);
I got the same exception: Uncaught Error: The stream is broken @0/0.
How to solve this problem?
I cannot change the grpc service server. how to revise envoy?

[ctoomey]

@akan , @seroperson's comment above explains how.