Search
Categories
- Development
- Fun
- General
- Graphics
- Mac
- Personal
- Software
Membase cache item expiration issue
Posted on February 21, 2011 by Mikkel Ovesen
Membase can be used as a distributed caching platform, and best of all… it is free to use. This is from their website:
Unlimited use in development and up to two nodes may be deployed for free in a production cluster, with modest additional prices for additional nodes.
When I started to develope and test Membase… I first wondered. Why does the total number of caching items continue to increase. Why do the number never decrease?
I expected items to be removed when their expiration time passed, but they didn’t and I started to wonder whether I was doing something wrong.
It turns out that Membase does not have an active expiration cache clean-up functionality. If an item has expired, it is not removed until the next Get, or when the cache is full and items need to be evicted.
Here is a very good description of the logic (Original here):
Your understanding of evictions is correct. When a new item is added and there is no space for it, an older item must be thrown away. One key thing to understand is that expiration and eviction have nothing to do with one another.
When an item expires, nothing actually happens. It is only upon the next access of that item that the server will notice it has expired and remove it from memory. There is also no guarantee that already expired items will be evicted first, though it practicality they are usually old enough that they tend to be among the first to get evicted (but nothing in the server controls this).
Now, onto the nuances of evictions. Within memcached there is a slab allocator which handles memory allocation for items. Basically, the whole memory space is broken up into 1mb pages which are then broken up into slabs of varying sizes. There are configuration options to control this, but that’s the default mechanism. Each size is called a “slab class” and has its own eviction logic which means that if one slab class fills up, you can’t reclaim memory from other slab classes. Depending on the variation in size of the objects that you are putting in, this could become more or less of a problem for you. Can you telnet to your servers and run ‘stats’ and ‘stats slabs’ and paste the output? That will let me understand exactly where memory is being taken up in your server.
Another key point to understand is that evictions are not always a bad thing. With changing workloads you can end up with lots of stale data still stored in cache that just needs to take some time to be pushed out. If everything seems to be running fine (with the exception of the evictions) then you probably don’t really need to take any action. Are you seeing a large number of misses as well? High misses and high evictions can usually be correlated to mean that the application is expecting data to be there when it’s not, and that would be a problem. If the application is receiving the data that it’s asking for, your cache is working correctly.
Your understanding of evictions is correct. When a new item is added and there is no space for it, an older item must be thrown away. One key thing to understand is that expiration and eviction have nothing to do with one another.When an item expires, nothing actually happens. It is only upon the next access of that item that the server will notice it has expired and remove it from memory. There is also no guarantee that already expired items will be evicted first, though it practicality they are usually old enough that they tend to be among the first to get evicted (but nothing in the server controls this).Now, onto the nuances of evictions. Within memcached there is a slab allocator which handles memory allocation for items. Basically, the whole memory space is broken up into 1mb pages which are then broken up into slabs of varying sizes. There are configuration options to control this, but that’s the default mechanism. Each size is called a “slab class” and has its own eviction logic which means that if one slab class fills up, you can’t reclaim memory from other slab classes. Depending on the variation in size of the objects that you are putting in, this could become more or less of a problem for you. Can you telnet to your servers and run ‘stats’ and ‘stats slabs’ and paste the output? That will let me understand exactly where memory is being taken up in your server.
Another key point to understand is that evictions are not always a bad thing. With changing workloads you can end up with lots of stale data still stored in cache that just needs to take some time to be pushed out. If everything seems to be running fine (with the exception of the evictions) then you probably don’t really need to take any action. Are you seeing a large number of misses as well? High misses and high evictions can usually be correlated to mean that the application is expecting data to be there when it’s not, and that would be a problem. If the application is receiving the data that it’s asking for, your cache is working correctly.
So if it seems like the total number of your Membase cache items just increases and increases, don’t worry it is actually normal behavior. Just make sure you have configured the server instance with the recommended amount of server memory and disk resources. Take a look in the best practises.
Share|
This entry was posted in Development, Software and tagged cache, caching, membase, memcache. Bookmark the permalink.