Setting up Multiple Redis Instances on a Single Magento Server

I had a bit of trouble figuring out a scalable way to set up multiple Redis instances on a single Magento server. There are a few posts (1, 2, 3) that were helpful, but included too much manual setup for my purposes. They all also required starting/stopping all redis instances individually.

Note: I have not found a good way to make this work with Upstart. If your Redis installation is using Upstart and you would like to use this method, just move your /etc/init/redis-server.conf file to /etc/init/redis-server.conf.bak. When I figure out a good solution for Upstart, I’ll update this post.

I am using Ubuntu 12.04 LTS, but most of this applies to any distro. This setup shares a Redis default config among all servers. Each new server only requires that a server specific configuration file be created with only four required settings. All redis instances may be started at once with

service redis-server start

or each server may be started individually with

service redis-server start server1

. This works will all service commands.

Install Redis and PHP Redis Client

apt-get update;
apt-get install -y php-pear php5-dev make redis-server;
pecl install redis;
echo 'extension=redis.so' > /etc/php5/conf.d/redis.ini;
service php5-fpm restart; #If you are running PHP-FPM

Set Up Your Init Script

cd /etc/init.d;
mv redis-server redis-server.bak;
touch redis-server;
chmod 0755 redis-server;
nano redis-server;

Paste these contents:

#!/bin/bash
### BEGIN INIT INFO
# Provides:             redis-server
# Required-Start:       $syslog $remote_fs
# Required-Stop:        $syslog $remote_fs
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    redis-server - Persistent key-value db
# Description:          redis-server - Persistent key-value db
### END INIT INFO

if [ -n "$2" ]
then
NAME=$2

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/redis-server
DAEMON_ARGS=/etc/redis/servers/$NAME.conf
DESC=redis-server

RUNDIR=/var/run/redis
PIDFILE=$RUNDIR/$NAME.pid

test -x $DAEMON || exit 0

set -e

case "$1" in
  start)
        echo -n "Starting $DESC: "
        mkdir -p $RUNDIR
        touch $PIDFILE
        chown redis:redis $RUNDIR $PIDFILE
        chmod 755 $RUNDIR
        if start-stop-daemon --start --quiet --umask 007 --pidfile $PIDFILE --chuid redis:redis --exec $DAEMON -- $DAEMON_ARGS
        then
                echo "$NAME."
        else
                echo "failed"
        fi
        ;;
  stop)
        echo -n "Stopping $DESC: "
        if start-stop-daemon --stop --retry forever/QUIT/1 --quiet --oknodo --pidfile $PIDFILE --exec $DAEMON
        then
                echo "$NAME."
        else
                echo "failed"
        fi
        rm -f $PIDFILE
        ;;

  restart|force-reload)
        ${0} stop $2
        ${0} start $2
        ;;

  status)
        echo -n "$DESC is "
        if start-stop-daemon --stop --quiet --signal 0 --name ${NAME} --pidfile ${PIDFILE}
        then
                echo "running"
        else
                echo "not running"
                exit 1
        fi
        ;;

  *)
        echo "Usage: /etc/init.d/$NAME {start|stop|restart|force-reload}" >&2
        exit 1
        ;;
esac

else

FILES=/etc/redis/servers/*
for f in $FILES
do
SERVERNAME=$(sed 's|/etc/redis/servers/||g' <<< $f)
SERVERNAME=$(sed 's|.conf||g' <<< $SERVERNAME)
/etc/init.d/redis-server "$1" "$SERVERNAME"
done

fi

exit 0

Set Up Your Config

Default config file

cd /etc/redis;
mkdir servers;
cp redis.conf servers/server1.conf;
cd servers;
nano server1.conf;

Strip out all non server specific configuration from server1.conf. The remaining lines should include pidfile, port, logfile, and dbfilename. pidfile, logfile, and dbfilename should all match the server conf file name “server1″. The resulting file should look something like this:

# Redis server config
include /etc/redis/redis.conf

# When running daemonized, Redis writes a pid file in /var/run/redis.pid by
# default. You can specify a custom pid file location here.
pidfile /var/run/redis/server1.pid

# Accept connections on the specified port, default is 6379.
# If port 0 is specified Redis will not listen on a TCP socket.
port 6379

# Specify the log file name. Also 'stdout' can be used to force
# Redis to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /prod/null
logfile /var/log/redis/server1.log

# The filename where to dump the DB
dbfilename dump-server1.rdb

Restart Redis instances:

service redis-server restart;

Additional Servers

To create new servers, simply copy the server1.conf file to server2.conf and update all relevant strings to your new server name (in this case, server2). If your server name is unique enough, you can just run the file through sed. You’ll also need to update the port manually to a new port.

cd /etc/redis/servers;
cp server1.conf server2.conf;
sed -i s/server1/server2/g server2.conf;
grep -hR ^port . | sort; # Determine the highest used port number
nano server2.conf; #Update the port number

Resulting file:

# Redis server config
include /etc/redis/redis.conf

# When running daemonized, Redis writes a pid file in /var/run/redis.pid by
# default. You can specify a custom pid file location here.
pidfile /var/run/redis/server2.pid

# Accept connections on the specified port, default is 6379.
# If port 0 is specified Redis will not listen on a TCP socket.
port 6380

# Specify the log file name. Also 'stdout' can be used to force
# Redis to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /prod/null
logfile /var/log/redis/server2.log

# The filename where to dump the DB
dbfilename dump-server2.rdb

Start the new Redis instance(s):

service redis-server start;

Please let me know if you see any issues with this or have any suggestions. This seems to work for me very well so far.

Kirk Madera

Menu