Sentinel Mode instance role mismatch

[jacobstorms]

Hello,

We are sometimes seeing this error while trying to configure the client to connect to the master node in a 3 node sentinel setup:

Redis::ConnectionError (Instance role mismatch. Expected master, got slave.)

For example:

module DataCache
  class << self
    def redis
      @redis ||= Redis.new(redis_opts)
    end

    def redis_opts
      svc = 'redis-headless'
     
      # query headless svc to get all sentinel addresses
      sentinels = Resolv.getaddresses(svc).map do |address|
        { host: address, port: 26379, password: ENV["REDIS_PASSWORD"] }
      end

      # connect to master for read/write
      {
        host: "mymaster",
        sentinels: sentinels,
        password: ENV["REDIS_PASSWORD"],
        role: :master
      }
    end
  end
end

irb(main):017:0> DataCache.redis
=> #<Redis client v4.6.0 for redis://redis-node-1.redis-headless.redis-namespace.svc.cluster.local:6379/0>
irb(main):018:0> DataCache.redis_opts
=> {:host=>"mymaster", :sentinels=>[{:host=>"10.0.3.247", :port=>26379, :password=>"###"}, {:host=>"10.0.2.40", :port=>26379, :password=>"###"}, {:host=>"10.0.3.36", :port=>26379, :password=>"###"}], :password=>"###", :role=>:master}
irb(main):019:0> DataCache.redis.get('testkey')
Traceback (most recent call last):
        2: from (irb):19
        1: from (eval):5:in `call'
Redis::ConnectionError (Instance role mismatch. Expected master, got slave.)
irb(main):020:0> 

Our setup (mostly aligned with this one) includes 1 master and 2 slave nodes. It is unclear to me how the client ever gets connected to a slave node given we are configuring it with role :master? But it appears to be selecting any of the 3 nodes indiscriminately (there is no error on the times when the client correctly connects to the master node)

127.0.0.1:26379> sentinel master mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) "redis-node-2.redis-headless.redis-namespace.svc.cluster.local"

I'm hoping I just have something misconfigured and someone might kindly point me in the right direction

[branweb1]

I saw this same error, and it wound up being a redis config issue similar to bitnami/charts#10745 (we're running redis in kubernetes). The misconfiguration caused the sentinels to disagree about which node was the master, and that broke failover. When the service rolled, master would get demoted to slave but the client wouldn't be informed, hence the error. After all the nodes had rolled, the error would change from the one you're seeing to "No sentinels found".

[vfiset]

@branweb1 are you using istio too or plain k8s ? We face the same issue where a node gets demoted from master to slave but the redis client does not get informed or does not try to get the new master.