wolfjsse netty-tests

[JeremiahM37]

Add Netty SSL test image for wolfJSSE FIPS
Docker image that runs upstream Netty SSL tests with wolfJSSE in FIPS mode.
-Clones Netty 4.1.115.Final and applies FIPS compatibility patches via shell script
-Runs handler, handler-proxy, and testsuite modules (~800 tests after unsupported fips tests removed)
-Skips OpenSSL-specific tests and FIPS-incompatible algorithms (MD5, 3DES, etc)
-Uses wolfSSL example certs (fetched from GitHub during build)

[Copilot]

Pull request overview

This PR adds a Docker test image for running upstream Netty SSL tests with wolfJSSE in FIPS mode. The implementation clones Netty 4.1.115.Final, applies FIPS compatibility patches, and configures the environment to run approximately 800 SSL tests using wolfSSL example certificates.

Key changes:

• Adds build infrastructure and FIPS compatibility patch script for Netty testing
• Replaces Netty's self-signed certificates with wolfSSL example certificates
• Skips OpenSSL-specific tests and FIPS-incompatible algorithms (MD5, 3DES, weak ciphers)

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 13 comments.

File	Description
build.sh	Build script with argument parsing, Docker image validation, and build orchestration for the Netty test container
apply_netty_fips_fixes.sh	Comprehensive patching script that modifies Netty source for wolfJSSE FIPS compatibility - replaces certificates, reorders cipher suites, disables incompatible tests
Dockerfile	Multi-stage Docker build that compiles patched Netty, installs wolfSSL certificates, and creates test runner script
README.md	Documentation with build instructions and usage examples for running the Netty test suite

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cconlon]

Did we confirm that this CloseNotify behavior was not a bug in wolfJSSE?

[JeremiahM37]

The CloseNotifyTest failure is FIPS-specific - the test passes on non-FIPS wolfJSSE. The close_notify event sequence/timing differs in FIPS mode due to stricter handshake requirements.

[cconlon]

Did we verify that these session handling test failures are not things we need to fix in wolfJSSE?

[JeremiahM37]

These session handling tests (testSessionCache, testSessionAfterHandshake, etc.) pass on non-FIPS wolfJSSE and only fail in fips mode.

[cconlon]

When I run this following the README steps, I see the following which looks like two tests have errors:

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   SocketSslGreetingTest.testSslGreeting » Timeout testSslGreeting(io.netty.handl...
[ERROR]   SocketSslGreetingTest.testSslGreeting » Timeout testSslGreeting(io.netty.handl...
[INFO]
[ERROR] Tests run: 167, Failures: 0, Errors: 2, Skipped: 9
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:29 min
[INFO] Finished at: 2026-01-07T00:18:25Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project netty-testsuite: There are test failures.
[ERROR]
[ERROR] Please refer to /app/netty/testsuite/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

============================================================
=== AGGREGATE TEST SUMMARY (all mvn runs) ===
============================================================
- handler: [WARNING] Tests run: 497, Failures: 0, Errors: 0, Skipped: 158
- handler-proxy: [INFO] Tests run: 55, Failures: 0, Errors: 0, Skipped: 0
- testsuite: [ERROR] Tests run: 167, Failures: 0, Errors: 2, Skipped: 9

TOTAL: Tests run: 719, Failures: 0, Errors: 2, Skipped: 167

========================================
=== ALL TESTS COMPLETE ===
========================================
Some tests failed. Check logs above.

Earlier in the test log I also saw this OutOfMemoryError show up. Is this expected?

00:02:13.410 [main] DEBUG io.netty.handler.ssl.SslHandler - [id: 0xembedded, L:embedded - R:embedded] HANDSHAKEN: protocol:NONE cipher suite:NONE
00:02:15.414 [nioEventLoopGroup-176-3] DEBUG io.netty.buffer.PoolThreadCache - Freed 13 thread-local buffer(s) from thread: nioEventLoopGroup-176-3
00:02:15.414 [nioEventLoopGroup-176-2] DEBUG io.netty.buffer.PoolThreadCache - Freed 10 thread-local buffer(s) from thread: nioEventLoopGroup-176-2
Jan 07, 2026 12:09:04 AM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-jupiter' failed to execute tests
java.lang.OutOfMemoryError: Java heap space

[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 497, Failures: 0, Errors: 0, Skipped: 158
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12:31 min
[INFO] Finished at: 2026-01-07T00:09:34Z
[INFO] ------------------------------------------------------------------------

[JeremiahM37]

Fixed the 2 failing tests (cert issue) and fixed the java heap space issue. At the end of the handler test I still get "There was a timeout or other error in the fork" but I have confirmed that it is running all of the tests we haven't disabled, and this only happens during the shutdown, and it does not happen on the non fips mode.