Mongoid is not designed for extensibility; if you need to modify its behaviour in a slight way, you will probably have no choice but to monkey-patch it.
A while ago I was working on a feature that required displaying information from several mongodb collections. The performance was pretty bad, since for each item being displayed it was necessary to traverse nested and polymorphic associations to get the rest of the data.
Although Mongoid provides eager loading support out of the box, it has a few limitations:
- No Nested: Only direct relations can be eager loaded.
- No Polymorphic: Polymorphic relations can't be included.
- Criteria-only: It's only possible to use eager loading with a
Mongoid::Criteriaobject. We can't leverage the functionality if we have a list of objects.
Unfortunately, one of those traversed associations was both nested & polymorphic, so in the beginning the only available solution was to eager load the relations manually.
After thinking about it for a while, I decided to give it a shot and come up with an extension to eager load polymorphic and nested associations, and do away with all the boilerplate that is necessary to perform eager loading.
Mongoid::Includes 💎
When writing the library, I picked a few constraints in order to give the project a clear direction:
- Reuse Mongoid's eager loading functionality as building blocks.
- Fail-fast, or as early as possible.
- Cover only the most common use cases.
- Allow to override eager loading for the not so common ones.
Choosing these constraints allowed me to keep the library fairly small, without compromising its usefulness in more complex scenarios.
The result is mongoid_includes, a gem that enhances support for eager loading in Mongoid, allowing to include polymorphic and nested associations, and modify eager loading queries on the fly.
Album.includes(:songs).includes(:musicians, from: :band)
Band.includes(:albums, with: ->(albums) { albums.gt(release: 1970).limit(2) })Mongoid::Includes extends the includes method to support polymorphic associations without any syntax change. For nested includes, it expects a :from option, indicating from which relation the include is going to be performed, eager loading it as well.
While those are the most typical cases, it also supports a :with option which conveniently allows to modify the default query, and a :loader option which receives the foreign keys of the documents to include.
Polymorphic or nested includes might be a sign of a poorly designed schema.
mongoid_includesis very easy to use, but it should only be used if it's truly necessary.
Extending Mongoid
Although it was possible to reuse the eager loading logic in Mongoid, doing so required a lot of fiddling and monkey-patching (using prepend), since the library does not provide any point of extension.
Mongoid's eager loading was written to work with queries, and assumes that the included documents will match an association on the model, so it relies on the association metadata to perform the includes. There is no simple way to reuse the logic without using relation metadata.
The biggest downside though, is that there is no way to perform eager loading for a set of documents, since the code relies on the contract of Mongoid::Criteria. We can't use eager loading if we triggered the query by using any Enumerable method, or got the models by aggregation or any in-memory operation 😥
It would be a lot easier to extend Mongoid's functionality if it had a more modular design. Adding support for plugins that can be attached to the query lifecycle would be a huge step in that direction—less patching means more and better extensions.
A Better Way
Some ORMs take a very different approach when it comes to eager loading. Ecto, a popular database wrapper for the Elixir language, has a different philosophy:
NOTE: Ecto does not lazy load associations. While lazily loading associations may sound convenient at first, in the long run it becomes a source of confusion and performance issues.
As a long time Mongoid user, I can painfully relate to this statement. N+1 queries are one of the fastest ways to degrade performance, and lazy loading associations makes it a lot easier to introduce them by accident. By not implementing lazy loading, the library becomes a lot simpler, and it encourages good practices and efficient data access patterns 🍹
Ecto also allows you to modify which models will be included for an association—like the :with option in mongoid_includes—and you can also preload associations on a given model or models after they have been fetched from the database using the Repo.preload/2 method. So much win!
Playing with Ecto inspired me to keep looking for a better solution for eager loading in Mongoid. Mongoid::Includes solves the first two limitations, but wouldn't it be great if we could preload documents without a query? 😉
