Redis Sentinel

Introduction

Redis Sentinel provides high availability for Redis. In practical terms this means that using Sentinel you can create a Redis deployment that resists without human intervention certain kinds of failures.

Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.

This is the full list of Sentinel capabilities at a macroscopical level (i.e. the big picture):

  • Monitoring. Sentinel constantly checks if your master and replica instances are working as expected.
  • Notification. Sentinel can notify the system administrator, or other computer programs, via an API, that something is wrong with one of the monitored Redis instances.
  • Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a replica is promoted to master, the other additional replicas are reconfigured to use the new master, and the applications using the Redis server are informed about the new address to use when connecting.
  • Configuration provider. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.

How sentinel works

Sentinels handle the failover by re-writing config files of the Redis instances that are running. Let’s go through a scenario:

Say we have a master “A” replicating to slaves “B” and “C”. We have three Sentinels (s1, s2, s3) running on our application servers, which write to Redis. At this point “A”, our current master, goes offline. Our sentinels all see “A” as offline, and send SDOWN messages to each other. Then they all agree that “A” is down, so “A” is set to be in ODOWN status. From here, an election happens to see who is most ahead, and in this case “B” is chosen as the new master.

The config file for “B” is set so that it is no longer the slave of anyone. Meanwhile, the config file for “C” is rewritten so that it is no longer the slave of “A” but rather “B.” From here, everything continues on as normal. Should “A” come back online, the Sentinels will recognize this, and rewrite the configuration file for “A” to be the slave of “B,” since “B” is the current master.

Installation

Before installation, redis must be install into the system. In this example, we’ll create one master node of redis and one slave which will be managed by one sentinel called as s1.

Redis master configuration

Just create a master.conf file which is a duplicate file of redis.conf which is in redis installation directory. After create master.conf just change the following things:

# Run as background service
daemonize yes

# for daemonized pid file path
pidfile "/var/run/redis_6379.pid"

# specify log file path
logfile "/var/log/6379.log"

# Set the number of databases, from 0 to 16
databases 16

# specify db dump path
dbfilename dump.rdb

# db will written into this directory
dir "/var/dump/"

# specify redis instance port
port 6379

# bind your public/local ip, to let you access by your ip address
bind ip1 ip2 ip

After the settings of master.conf file just start the redis server with this configuration file as below:

Start Master node

redis-cli -h 127.0.0.1 -p 6379 info replication

After executing this command you will see following output:

# Replication
role:master
connected_slaves:0
master_replid:049098811534d348dd88d82df0672295ee261482
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:179479
master_repl_meaningful_offset:179479
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:179479

As you can see that there is no connected_slaves as of now with this master redis. But this has been configured as master in the environment.

Now let’s configure slave node of redis as below:

Slave Node

For configuring slave node, you need to change the properites like bind ip, port, log directory, dump file etc as you did in master.conf. Apart from that below configuration needs to change in order to make redis node as slave of master node.

# Specify master instance location
replicaOf 127.0.0.1 6379

Now its time to start slave as below:

Start Slave Node

redis-server slave.conf

Suppose you given slave port as 6380 so it will start on this port.

Verify master and slave node

Let’s verify whether it has been configured with master as slave or not by executing following command again:

redis-cli -h 127.0.0.1 -p 6379 info replication

It will show us following output:

# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=6380,state=online,offset=199429,lag=1
master_replid:049098811534d348dd88d82df0672295ee261482
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:199429
master_repl_meaningful_offset:199429
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:199429

As you can see, connected_slaves is 1 which means that slave node has been configured successfully.

Configure Sentinel

It’s time to configure sentinel to manage both the nodes. Just open sentinel.conf file and change following things:

# port to run sentinel
port 26379
dir /tmp

# Consider master objectively down only if at least number of sentinels agree
sentinel monitor master 127.0.0.1 6379 1

# How many seconds master not responding will consider it as down
sentinel down-after-milliseconds master 3000

# How many slave could sentinel reconfigure simultaneously, use low number to prevent all instance unreachable at about the same time
sentinel parallel-syncs master 1

# Sentinel failover timeout
sentinel failover-timeout master 18000

Start sentinel

redis-server sentinel.conf --sentinel

Verify sentinel

redis-cli -h 127.0.0.1 -p 26379 info sentinel

It will show you following output:

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=127.0.0.1:6379,slaves=1,sentinels=1

As you can see the information about slaves and masters.

It’s time to verify the sentinel and replication of things:

Just Execute the following command:

redis-cli -h 127.0.0.1 -p 6379

127.0.0.1:6379> hset first-key hello world
integer 1
127.0.0.1:6379>keys *
1) "first-key"

### After that just connect with the slave node in order to check the replication of data into slave node

redis-cli -h 127.0.0.1 -p 6380
127.0.0.1:6380> keys *
1) "first-key"

As you can see that we create a set into master node and it is automatically gets created or replicated into slave node.

To know more about sentinel click here.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s