I just released a new version of slim-attributes. There are some small speed gains and some other minor changes from 0.4.1, but there are no big changes.Read more about slim-attributes at the slim-attributes homepage, or read on below.IntroductionSlim-attributes is a small patch to the ActiveRecord Mysql adaptor that stops rails from immediately making ruby strings from the column names and data from your database queries. Because you probably don’t need them all!So ruby strings are lazily created on demand – it’s faster and uses less memory. And it drops directly in, requiring only the installation of a gem and adding 1 line to environment.rb.Measuring with just ActiveRecord code – fetching stuff from the database – we see anything up to a 50% (or more) speed increase, but it really depends on your system and environment, and what you are doing with the results from the database. The more columns your tables have, the better the improvement will likely be. Measure your own system and send me the results!InstallationTry:
gem install slim-attributes -- --with-mysql-config
or:
gem install slim-attributes
then add this to environment.rb:
require 'slim_attributes'
DescriptionNormally the mysql adaptor in Rails returns a hash of the data returned from the database, one hash per active record object returned by the query. The routine that generates these hashes is called all_hashes, and this is what we replace. The reason for overriding all_hashes is threefold:
- making a hash of each and every row returned from the database is slow
- rails makes frozen copies of each column name string (for the keys) which results in a great many strings which are not really needed
- we observe that it’s not often that all the fields of rows fetched from the database are actually used
So this is an alternative implementation of all_hashes that returns a ‘fake hash’ which contains a hash of the column names (the same hash of names is used for every row), and also contains the row data in an area memcpy’d directly from the mysql API (which is much faster than creating ruby strings).The field contents are then instantiated into Ruby strings on demand – ruby strings are only made if you need them – when you ask for a particular attribute from the model object.Note that if you always look at all the columns when you fetch data from the database then this won’t necessarily be faster that the unpatched mysql adapter. But it won’t be much slower either, and we do expect that most times not all the columns from a result set are accessed.Future developmentI speculate that further speed gains might be had through keeping the mysql result objects from mysql-ruby around, and not copying the data from them at all until it is needed. However, mysql-ruby limits the non freed result sets to just 20 before calling GC.start, so surgery inside mysql-ruby would be required to achieve this.
Pennysmalls 
Great work with this!The forced GC of mysql-ruby is a royal pain, performance wise and I imagine it being a killer for intense Rails actions, which basically implies a forced GC every 2 requests should the action render with 10 odd queries. We’re playing with disabling the 20 result forced GC functionality with this branch of mysqlplus :http://github.com/oldmoe/mysqlplus/tree/with_async_validationBenchmark script available here :http://github.com/oldmoe/mysqlplus/tree/e81e145c15e693850dd6180ae6547ae74068a6eb/test/gc_benchmark.rbThe pattern has been a 100%+ performance increase on 1.8 and little to no difference on 1.9- Lourens
Yeah lourens’ branch of mysqlplus has an on off switch for the auto-gc.re: further speculation–that’s an interesting idea. I’m not sure how often you end up never examining a row of a query, though.One interesting idea would be to keep the all_hashes result around and re-use it if the DB hasn’t changed [kind of like http://www.inwebwetrust.net/post/2008/09/08/query-memcached without the memcache storage part–just the memcached ‘last updated at per table’ algorithm].That said, as you mentioned previously, I fear that DB retrieval of objects is not the largest bottleneck in AR [though this plugin alleviates it greatly]. Haven’t done much profiling of it, though–I could be wrong.That being said if that’s not the bottleneck then what is?Cheers!-=R
The idea would be to save on copying the data out of the mysql result object, and to instantiate model attributes only when you need them directly from the result. I’ll try to get a benchmark from my experimental code to see if there’s much benefit (it currently segfaults 😦 ).By the way, it doesn’t matter if you examine a row or not – the point is that only those attributes you need from any row are ever copied out of the mysql result data.I have looked at Query memcached – as it stands it is incompatible with slim_attributes because it needs to serialise all the data to memcached which ends up instantiating all the attributes. Might be worth looking at inventing a compatible query cache though.Perhaps there are other areas that can be improved in AR, and profiling it may be worthwhile. I suspect that the speed improvement we have seen in slim-attributes by not instantiating all those ruby objects is probably the biggest improvement that can be made in a fairly simple way.
Do you know what the proper config options are to install this under Windows? I know you got extconf.rb from the mysql gem, and it seems the best solution for that gem for Windows is to get a pre-compiled version.The error I’m getting is checking for main() in libmysql.lib… noI’d appreciate any ideas on how to solve this.
bgates: Sorry I do not know enough about compiling ruby and extensions for it under windows.If there is anyone who can successfully compile slim_attributes for windows then I would happily host it / link to it.