Jedis performance degrades with large connection pool size (under high concurrency)

[yangty89]

Hi, I'am using JMH to test the performance of Jedis under high concurrency (200 concurrent threads). The test is performed on a localhost Redis with a 8-processor CPU. The version of Jedis is 4.1.1 and The Version of Redis is 6.2.6. The test code is shown below:

@State(Scope.Benchmark)
public class CommonClientBenchmark {

    private JedisPool jedisPool;
    private final String host = "127.0.0.1";
    private final int port = 6379;
    private final int poolSize = 50

    @Setup
    public void setup() {
        JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
        jedisPoolConfig.setMaxTotal(poolSize);
        jedisPoolConfig.setMaxIdle(poolSize);
        jedisPool = new JedisPool(jedisPoolConfig, host, port, 30000);
    }

    @TearDown
    public void tearDown() {
        jedisPool.close();
    }

    @Threads(200)
    @Fork(1)
    @Benchmark
    @BenchmarkMode(Mode.Throughput)
    @Warmup(iterations = 1, time = 30, timeUnit = TimeUnit.SECONDS)
    @Measurement(iterations = 2, time = 30, timeUnit = TimeUnit.SECONDS)
    public void jedisSet() {
        try (Jedis jedis = jedisPool.getResource()) {
            jedis.set("jedis", "jedis");
        }
    }

    public static void main(String[] args) throws IOException, RunnerException {
        CommonClientBenchmark commonClientBenchmark = new CommonClientBenchmark();
        commonClientBenchmark.setup();
        org.openjdk.jmh.Main.main(args);
    }
}

When I set poolSize to 50, I got a QPS of about 70k. However, When I increase the poolSize to 200 (which matches the concurrent thread number), the QPS dropped heavily to about only 30k. I've also run a redis-benchmark with 200 connections, and I got a QPS of about 80k. To find out the reason of performance loss of Jedis with large pool size under high concurrency, I've also tried using top -o cpu to observer the overall CPU usage and the CPU usage of each process. And I got the stats below:

=> 200 thread, jedis 50 conn, qps about 70000
Load Avg: 6.41, 5.72, 5.21  CPU usage: 18.89% user, 51.0% sys, 30.10% idle
ID   COMMAND      %CPU      TIME     #TH     #WQ  #PORT MEM    PURG   CMPRS
8082  java         258.1     00:55.99 236/4   1    511   211M-  0B     0B
0     kernel_task  100.9     33:45.69 212/9   0    0     264M   0B     0B
1514  redis-server 86.8      16:31.14 4/1     0    17    20M    0B     4268K


=> 200 thread, jedis pool 200 conn, qps about 30000
Load Avg: 8.94, 12.30, 8.44  CPU usage: 5.75% user, 85.42% sys, 8.81% idle
PID   COMMAND      %CPU  TIME     #TH     #WQ  #PORT MEM    PURG   CMPRS  PGRP
4556  java         599.7 07:06.31 236/15  1    511   202M   0B     0B     4255
0     kernel_task  56.8  12:42.50 212/9   0    0     323M   0B     0B     0
1514  redis-server 29.6  06:10.48 4/1     0    17    20M    0B     2488K  1514

I found that Jedis consumes much more CPU time when the connection pool size augments to 200, and, according to the overall CPU usage data, a big portion (about 85%) of CPU time was spent in kernel mode. I wonder if this means that with large connection pool size and under high concurrency, Jedis will make much more socket syscall (write/read) in parallel, and the resulting frequent context switch finally impact the performance heavily? Many thanks!

[sazzad16]

@yangty89 Thank you. Hopefully we / anyone from the community will be able to work on it.

[yangty89]

thanks a lot

[gordonhjko]

Redis is single thread single command system. Better to use multiple redis instances with small connection pool