+++
date = "2012-03-21"
title = "Redis using 2GB of memory on 70MB data set - the fix"
tags = ["redis", "memory", "malloc", "jemalloc"]
slug = "redis-using-2gb-of-memory-on-70mb-data-set-the-fix"
+++
For [Ariejan.net](http://ariejan.net) I use [redis](http://redis.io) to cache pages and shards. This works great and all, but today I noticed something alarming:

    redis	Running	3d 12h 4m 	0.0%	45.2% [1829556 kB]

Yes, that's about 1.7 GB of RAM. That's way too much for what I cache. Let's see what redis has to say for itself:

    $ redis-cli
    redis 127.0.0.1:6379> info
    redis_version:2.2.12
    ...
    connected_clients:5
    connected_slaves:0
    used_memory:71626608
    used_memory_human:68.31M
    used_memory_rss:1873465344
    mem_fragmentation_ratio:26.16

Well, that's awkward. The OS is reporting 1.7GB memory usage, while Redis claims to store a mere 68MB. What's happening here!

You may have noticed that I included `mem_fragmentation_ratio` in the snippet above as well. It's at a whopping 26.16, meaning that for every byte I store, 26.16 bytes of memory are used. This explains the 1.7GB memory usage.

But, how do I get rid of this? My system has enough RAM to cope redis as is, but it's not a comforting thought to leave Redis running like this.

As it turns out, there isn't a lot you can do about this. Redis 2.2 uses _malloc_ which causes the fragmentation. One alternative is to add a slave redis-server, migrate your data and then switch the slave to master. Although this is reported to work well, it's not a good solution to the problem.

Fortunately, Redis 2.4 on Linux by default does not use _malloc_ anymore. Instead it uses _jemalloc_. From the redis README:

> Redis is compiled and linked against libc malloc by default, with the exception of jemalloc being the default on Linux systems. This default was picked because jemalloc has proven to have fewer fragmentation problems than libc malloc.

The only logical step to take is to upgrade to Redis 2.4.

If you're already running (or just upgraded to) 2.4 you can easily check if your redis is using _jemalloc_:

    $ redis-cli
    redis 127.0.0.1:6379> info
    redis_version:2.4.9
    mem_fragmentation_ratio:1.11
    mem_allocator:jemalloc-2.2.5

I've taken this step and the `mem_fragmentation_ratio` samples I've measured have all been in the 1.1-1.4 regions.