Ruby leaks memory

I’ve spent a considerable amount of time with various tools attempting to figure out why it is that our thin processes (and mongrels before them) grow so egregiously. Typically they reach about 450Mb in a day, after which we restart them via monit.What makes them grow? Well, we are fetching a lot of stuff from the DB all the time – meaning that thousands of small strings are being instantiated – so perhaps we can attribute some growth to heap fragmentation. But we tried changing to ptmalloc3 – it didn’t help; in fact I think in our case this is rather a red herring.In an effort to get the problem under control, I wrote a plugin to reduce the number of strings that are made, changing the implementation of the mysql library so that all_hashes actually returns fake hashes that are implemented as arrays – to prevent all those column names being saved as frozen strings (for the hash keys) for every row that is fetched from the DB. But that didn’t help much, if at all, either.But whilst I was playing with ruby with valgrind, I noticed some memory going missing. At first I thought it was probably me. But with further investigation I found a simple expression that makes ruby leak.

a = eval "b=0"

It’s actually the eval that leaks – the a = is not really needed, but it makes the leak show as a definite leak as opposed to a possible one in this simple one liner. If you want to leak a lot of memory, this is the way:

def grow  for i in 1..100    eval "b#{i}=1"  endend15000.times {grow}

You can fiddle with the numbers to make it grow as much as you like.Valgrind reports the leak like this (this one made by running the loop 5000.times):

==18706== 217,988,864 bytes in 499,985 blocks aredefinitely lost in loss record 6 of 6==18706==    at 0x4A05AF7: realloc (vg_replace_malloc.c:306)==18706==    by 0x432398: ruby_xrealloc (gc.c:151)==18706==    by 0x465E9C: local_append (parse.y:5649)==18706==    by 0x465F64: local_cnt (parse.y:5667)==18706==    by 0x4646AC: assignable (parse.y:4902)==18706==    by 0x458E80: ruby_yyparse (parse.y:844)==18706==    by 0x45E5F4: yycompile (parse.y:2606)==18706==    by 0x45E8F4: rb_compile_string (parse.y:2676)==18706==    by 0x41DDF3: compile (eval.c:6412)==18706==    by 0x41E289: eval (eval.c:6493)==18706==    by 0x41E817: rb_f_eval (eval.c:6611)==18706==    by 0x41C765: call_cfunc (eval.c:5700)==18706==    by 0x41BB04: rb_call0 (eval.c:5856)==18706==    by 0x41D291: rb_call (eval.c:6103)==18706==    by 0x415182: rb_eval (eval.c:3494)

The memory is allocated when ruby is expanding its local variable table in the parser. But what I don’t know yet is exactly where to add a call to free to release that memory. I’m hoping that someone over at ruby-core can help. Interestingly, it appears that Rubinius leaks too, which is surprising given that it is a completely new implementation.I’m not the only one to have found a leak in Ruby lately – I wonder if the issues with god are related to this?Fixing this leak may not completely cure our Rails memory growth problem (probably won’t), but at least it will help.

Advertisements

Changing to thin from mongrel

Thin is getting some attention, so I thought we would give it a try.Installation is just a matter of gem install thin.Run it with something likethin -e production -s 6That’s 6 servers running on 0.0.0.0:3000 to 0.0.0.0:3005Look at the examples if you need to make a monit recipe.One thing we have is some code to make individual log files for each server instance. This is how it was with mongrel – we put this code in environment.rb inside the Rails::Initializer block:

if ENV['RAILS_ENV'] == 'production'  if defined?(Mongrel::HttpServer)    ObjectSpace.each_object(Mongrel::HttpServer) {|i| @port = i.port}    @port = "unknown" unless @port && @port.to_i > 0    config.logger = Logger.new(File.expand_path(      RAILS_ROOT+"/log/#{ENV['RAILS_ENV']}.#{@port}.log"), 2, 25000000)  endend

Somthing very similar will work with thin:

if ENV['RAILS_ENV'] == 'production'  if defined?(Thin::Server)    ObjectSpace.each_object(Thin::Server) {|i| @port = i.backend.port}    @port = "unknown" unless @port && @port.to_i > 0    config.logger = Logger.new(File.expand_path(      RAILS_ROOT+"/log/#{ENV['RAILS_ENV']}.#{@port}.log"), 2, 25000000)  endend

With this code in place you will get individual log files named production.3000.log, production.3001.log etc.Finally we were seeing these errors:

terminate called after throwing an instance of 'std::runtime_error'  what():  unable to delete epoll event: Bad file descriptor

This is a known problem not with thin, but with EventMachine. Grab an updated gem like this:

gem install eventmachine --source http://code.macournoyer.com