I was using a Amazon Elasticache
cache.t2.medium Redis instance as a cache store for one of the Rails application. Over the time I observed that connections to Redis instance are increasing day by day and once it hit ~1500, the Rails application started throwing connection timeout exceptions:
1 Error connecting to Redis on redis-host:6379 (Redis::TimeoutError)
On searching over internet, I found that the default parameter group has
timeout value set to 0 which means, any connection that has been initialized to redis will be kept by redis even if the connection initialized by the client is down. Well the first thing I did was to create a new parameter group and set its value to a non-zero integer. However I couldn’t apply the parameter group changes to live production application and hence started looking for alternatives.
On further investigation, I found that Redis provides
CLIENT KILL command to kill the client connections. To see all client connections to your Redis instance, you can use
CLIENT LIST command:
1 2 3 redis 127.0.0.1:6379> CLIENT LIST id=9115408 addr=10.0.1.252:56348 fd=95 name= age=23 idle=23 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=9115336 addr=10.0.2.151:52018 fd=37 name= age=107 idle=107 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
To kill a connection, you can use
CLIENT KILL command as below:
1 2 redis 127.0.0.1:6379> CLIENT KILL 10.0.1.252:56348 (integer) 1
On successful completion, the command will return number of client connections killed. Well this was all good but I can’t kill ~1500 connections one by one. Hence I wrote a little python script to achieve this task.
1 2 3 4 5 6 7 8 9 10 11 12 13 import redis import re idle_max = 300 r = redis.Redis(host="localhost", port=6379, password=None) cl = r.execute_command("client", "list") pattern = r"addr=(.*?) .*? idle=(\d*)" regex = re.compile(pattern) for match in regex.finditer(cl): if int(match.group(2)) > idle_max: r.execute_command("client", "kill", match.group(1))
Above script will kill all redis client connections which have been idle for 300 seconds. You can adjust the value of
idle_max variable as per your need. Save this script as
killredisconn.py and execute as below:
1 $ python killredisconn.py
You need to install
redis python package before executing this command. You can install the same by issuing following command:
1 $ sudo pip install redis
Once I executed above script, number of client connection decreased dramatically and connection timeout exceptions from our Rails application disappeared.