Microgroove & Getmusic.com.au
Microgroove builds Getmusic.com.au with RavenDB
Getmusic.com.au is a direct-to-consumer store, offering CDs, MP3s, DVDs, apparel and merchandise from Universal Music Group, Warner, EMI, Sony and a range of independent labels. At close to two million SKUs, hundreds of thousands of MP3 streams and multiple real-time mobile APIs, Getmusic provided some unique data challenges.
The site was built and hosted by Microgroove, a software development company based in Seattle. At its foundation, Getmusic is powered by the Microgroove Music Platform, although a substantial amount of custom development was required for project specific functionality.
Microgroove has long relied on SQL Server for general purpose data storage. However, during the initial development phases of Getmusic a certain category of functional was proving difficult to implement in traditional SQL. After investigating alternatives, Map / Reduce was quickly identified as the perfect solution. That led us to investigate hosts for Map / Reduce jobs and two were selected; MongoDB and RavenDB.
There were many things to like about MongoDB including ease of setup, write speed & mindshare. But, it lacked ACID-compliance and at the time there were three competing .NET client libraries and it was clear only one would be a long term winner. The trouble was, no-one could agree on which one.
RavenDB was equally as easy to setup, fast and just felt right in a .NET environment. In addition, it came packaged with a management studio out of the box. We were aware of Hibernating Rhinos great work on NHibernate, Profiler and of Ayende in general.
Map / Reduce
Getmusic sells direct to consumer and receives near real-time sales data from the commerce and fulfillment provider. This data is compiled into daily, weekly and monthly publically viewable sales charts for each of the nine media types (e.g. CD, Download, Vinyl, Apparel, etc.) on offer.
At first, this sounded like a straightforward set of SQL “group by” queries. However, given the large number of SKUs, date range parameters and continuous updates of sales data, even highly optimized SQL queries were timing out. Offline processing was considered, but felt like a lot of work to setup and maintain.
Map / Reduce was a perfect fit for this task as it allowed us to categorize the media types to operate over (the map) and a way to aggregate their sales data (the reduce). Offline processing was essentially given to us for free, as whenever new data is stored in RavenDB, related Map / Reduce indexes are re-indexed automatically.
Public charts pages query RavenDB’s Map / Reduce indexes directly, using no more than the default caching built into the RavenDB client libraries and server. Far from the original SQL-driven performance issues, these charts pages are now some of the fastest sections of the site.
Beyond Map / Reduce
Once we’d tasted success with RavenDB and its Map / Reduce functionality, we found ourselves turning to RavenDB to solve other problems, too. Whenever new functionality was added to the site, it would be developed with RavenDB as the sole backend data store, instead of SQL Server. No more fluent NHibernate mappings to declare, no more schemas to generate and deploy.
As the core Microgroove Music Platform was developed long before Raven, SQL Server is still part of the Getmusic picture. However, our architecture was changed to accommodate multiple backend stores, allowing the developer to choose the most appropriate for a particular function. For example, new product data entering the system is first stored in SQL Server and a CREATED event is published on a NServiceBus managed bus. Several subscribers are responsible for storing the data in RavenDB, activity logs, etc.
Our Release model type requires 12 tables in SQL Server (an N+1 nightmare), but is stored as a single document type in RavenDB.
Eventually, and as new functionality is implemented using RavenDB only, we expect this scheme to be less relevant, at least in terms of SQL Server and RavenDB.
Advice to other developers
RavenDB is extremely easy to get started with and feels like a natural extension to the .NET environment. But, old SQL habits die hard, especially the bad ones. Therefore, you’ll get the most out of RavenDB if you approach a problem from a non-relational point of view from the beginning. Think about your use cases and what a typical unit of work looks likes.
The same is true for a production deployment. Learn the capabilities of RavenDB’s management studio early and get a feel for what managing production installations on a day-to-day basis feels like.