I was using a Amazon Elasticache cache.t2.medium Redis instance as a cache store for one of the Rails application. Over the time I observed that connections to Redis instance are increasing day by day and once it hit ~1500, the Rails application started throwing connection timeout exceptions:

1
Error connecting to Redis on redis-host:6379 (Redis::TimeoutError)

On searching over internet, I found that the default parameter group has timeout value set to 0 which means, any connection that has been initialized to redis will be kept by redis even if the connection initialized by the client is down. Well the first thing I did was to create a new parameter group and set its value to a non-zero integer. However I couldn’t apply the parameter group changes to live production application and hence started looking for alternatives.

On further investigation, I found that Redis provides CLIENT KILL command to kill the client connections. To see all client connections to your Redis instance, you can use CLIENT LIST command:

1
2
3
redis 127.0.0.1:6379> CLIENT LIST
id=9115408 addr=10.0.1.252:56348 fd=95 name= age=23 idle=23 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=9115336 addr=10.0.2.151:52018 fd=37 name= age=107 idle=107 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

To kill a connection, you can use CLIENT KILL command as below:

1
2
redis 127.0.0.1:6379> CLIENT KILL 10.0.1.252:56348
(integer) 1

On successful completion, the command will return number of client connections killed. Well this was all good but I can’t kill ~1500 connections one by one. Hence I wrote a little python script to achieve this task.

1
2
3
4
5
6
7
8
9
10
11
12
13
import redis
import re

idle_max = 300

r = redis.Redis(host="localhost", port=6379, password=None)
cl = r.execute_command("client", "list")

pattern = r"addr=(.*?) .*? idle=(\d*)"
regex = re.compile(pattern)
for match in regex.finditer(cl):
    if int(match.group(2)) > idle_max:
        r.execute_command("client", "kill", match.group(1))

Above script will kill all redis client connections which have been idle for 300 seconds. You can adjust the value of idle_max variable as per your need. Save this script as killredisconn.py and execute as below:

1
$ python killredisconn.py

You need to install redis python package before executing this command. You can install the same by issuing following command:

1
$ sudo pip install redis

Once I executed above script, number of client connection decreased dramatically and connection timeout exceptions from our Rails application disappeared.