Approx 4 minutes read

I was using a Amazon Elasticache cache.t2.medium Redis instance as a cache store for one of the Rails application. Over the time I observed that connections to Redis instance are increasing day by day and once it hit ~1500, the Rails application started throwing connection timeout exceptions:

1 Error connecting to Redis on redis-host:6379 (Redis::TimeoutError)

On searching over internet, I found that the default parameter group has timeout value set to 0 which means, any connection that has been initialized to redis will be kept by redis even if the connection initialized by the client is down. Well the first thing I did was to create a new parameter group and set its value to a non-zero integer. However I couldn’t apply the parameter group changes to live production application and hence started looking for alternatives.

On further investigation, I found that Redis provides CLIENT KILL command to kill the client connections. To see all client connections to your Redis instance, you can use CLIENT LIST command:

1 redis 127.0.0.1:6379> CLIENT LIST
2 id=9115408 addr=10.0.1.252:56348 fd=95 name= age=23 idle=23 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
3 id=9115336 addr=10.0.2.151:52018 fd=37 name= age=107 idle=107 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

To kill a connection, you can use CLIENT KILL command as below:

1 redis 127.0.0.1:6379> CLIENT KILL 10.0.1.252:56348
2 (integer) 1

On successful completion, the command will return number of client connections killed. Well this was all good but I can’t kill ~1500 connections one by one. Hence I wrote a little python script to achieve this task.

 1 import redis
 2 import re
 3 
 4 idle_max = 300
 5 
 6 r = redis.Redis(host="localhost", port=6379, password=None)
 7 cl = r.execute_command("client", "list")
 8 
 9 pattern = r"addr=(.*?) .*? idle=(\d*)"
10 regex = re.compile(pattern)
11 for match in regex.finditer(cl):
12     if int(match.group(2)) > idle_max:
13         r.execute_command("client", "kill", match.group(1))

Above script will kill all redis client connections which have been idle for 300 seconds. You can adjust the value of idle_max variable as per your need. Save this script as killredisconn.py and execute as below:

1 $ python killredisconn.py

You need to install redis python package before executing this command. You can install the same by issuing following command:

1 $ sudo pip install redis

Once I executed above script, number of client connection decreased dramatically and connection timeout exceptions from our Rails application disappeared.

comments powered by Disqus