To shard, or not to shard: that is the question

When dealing with very high volumes of data, one usually needs to decide how the system is going to scale when the data grows beyond a reasonable size. In my current project we’re working with tables that reach 1B records.  I’m talking about MySQL 5.5.

After initial load-testing and profiling we had our first “oh, shit” moment. When tables are in hundreds of millions of records, regular “laws of physics” don’t always apply. So, we desperately started looking at ways to alleviate these problems.

Clearly, the most scalable method is sharding. Sharding is usually implemented at the application level in such a way that the application knows on which node a particular piece of data resides. A common example is placing users 1-1M on shard 1, users 1M-2M on shard 2, etc. This method allows for theoretically unlimited amounts of data.

Now, the bad part. Implementing sharding is pretty complex and time consuming. Most importantly, sharding forces breaking up the data by certain criteria (like users_id). All queries that specify the user are OK, because we know where to go. But, queries that don’t specify user IDs must be executed on ALL shards. That is a major point of pain. Of course, you can come up with a smart parallel execution strategy, but then you’re effectively entering a realm of programming of distributed databases.

There are some ready alternatives: ScaleDB and Gizzard are examples of NewSQL, and are layers that reside between the application and mulitiple MySQL nodes, and know where to execute the queries. Then, of course, there is a plethora of NoSQL solutions that are capable of distributing the data: Cassandra, MongoDB, etc. In our case, increased complexity associated with introducing new systems into the project, would outweigh benefits of these systems.

So, what’s a desperate developer to do in such a case? Well, in some cases it’s possible to fall back on time-tested ways of reducing table size by means of archival. Archival does not mean that the archived data is not accessible. It might just be accessed from another table. Also, sometimes it helps to partition the table on MySQL level. While it’s far short of the benefits that complete sharding would yield, it does help with performance of queries with partitioned column.

The conclusion is that while sharding is very powerful and scalable technique for dealing with large data volumes, it involves a lot of complexity and side-effects. A careful examination of alternatives is in order. Be it NewSQL, NoSQL, or plain OldSQL (archival, DB-level partitioning), it’s worthwhile to carefully analyze tradeoffs of each solution. 

Agile Change and Rework Waste

One the main benefits of Agile methodologies is adaptability to change. Changes occur in all projects, Agile or not, but Agile projects are better equipped to handle the changes through short iterations and ongoing adaptation to new requirements. The customer does not need to specify everything upfront. As the system is being developed, he can refine the system with the knowledge that was obtained from previous iterations.

Now, there is a difference between refining the product, re-prioritizing certain features, redefining the unimplemented features on the one hand, and discarding an implementation of already developed features on the other hand. Discarding often occurs because “now i see it differently” or because “that’s not what I wanted”. Granted, as people discover more about the product, their needs and requirements change.  However, the proper way to address it is to implement the minimal and agreed agreed functionality first, and then “grow” it according to evolving understanding of the system.

Unfortunately, people often confuse iterative and incremental development. Iterative development means that a team develops bare bones of a feature and then grows it. Incremental development means that one whole feature is implemented after another whole feature.  In such case, if the implementation is wrong, then most of the effort is wasted, and the feature must be reworked.  This is not the Agile change management – this is waste, and it should be rigorously eliminated (think Lean). 

An alternative to wasting effort on implementation of misunderstood, or incorrectly specified functionality, is not to request the customers to sign the requirements in blood. It is to structure the implementation in such a way that least debatable or more clearly understood and agreed upon functionality is implemented first. If there are areas that are likely to undergo significant amount of change, it's worthwhile to prototype them before committing to a full-fledged implementation. Changing prototypes is a lot cheaper than changing actual implementation.

For example, say you need to implement user management functionality in a back office part of your application.  You could try to specify all fields that are required, you could make guesses, or you could torture your customer to divulge the secrets of user’s information. Or, alternatively, you could just implement a very basic set of properties (first/last name, address, birth date) in initial iteration, and later “grow” the user data as requested by the customer.

Furthermore,  it is in customer’s and provider’s best interests to think the requirements through before starting their implementation. Incomplete and incorrect specs cannot be miraculously cured by Agile methodologies. What’s incorrectly specified will be incorrectly implemented. So spend the time analyzing and planning what and how things will be implemented during the upcoming iteration or release.

Agile Business Discovery

One of the most important advantages of Agile methodologies is the rapid feedback loop. This feedback loop is intended for refining and optimizing the development process. It also helps to ensure that final product matches customer’s expectations and solves the actual business problem.

In all this, it is assumed that the business problem is known. But what if it’s not? What if the business model itself is being discovered? Shouldn’t we apply the same Agile principles in discovering the business model?

You might ask: “What kind of business is that, if it does not understand its own business model?” Many examples come to mind: start-ups, big organizations trying to expand, etc. In all such cases, the business model or at least the required feature-set needs to be discovered through a rapid feedback loop with potential customers.

Describing Innovation At Google, Patrick Copeland put it extremely concise: “Before building it right, build the right IT”. He recommends what he calls prEtotyping (as opposed to prOtotyping) to get a real-life customer feedback about a potentially great business idea. Whereas prototyping focuses on proving that the product can be build and refining its technical structure, prEtotyping is aimed at proving the business model itself, and helps screening ideas before any real investments are made into the development.

Many people believe that their ideas or business models worth millions. However, from the experience we know that not every company becomes the next Google, Facebook, Tweeter, etc. The only way to get any confidence in a product is to elicit early feedback from potential customers. Essentially, it means applying empirical, scientific technique to discovery of viable business ideas. As with any empirical studies, the method requires actual data, preferably in a numeric form. This data can include page hits, returning customers, etc. Often, even the most primitive pseudo-implementation (CRUD?) is enough to gather such data. Sometimes, just a study of customer preferences and simulated sessions allow vetting of ideas without any implementation or prototypes.

In conclusion, if you’re trying to implement a new business idea or to augment an existing one, use the same Agile feedback methods to validate or screen out the idea before investing in it.

Economics of Amazon Outage

As if the world needs another rant about Amazon EC2 outage, I’m going to risk adding my 2 cents to the debate.

On the one hand you have apocalyptic visions on the cloud future focusing on inherent unreliability of public clouds. On the other hand you have people explaining, reasonably enough, that there is nothing new about hardware failures, and talking about necessity of preparation for it by means of redundancy and other fail-safe methods.

I, obviously, see logic in the latter view, and understand that “outage happens:)”, and you need to prepare for it. However, often the missing ingredient in this line of reasoning is the cost of the preparation. In the end, everything boils down to economics. There is a very good description of estimating the cost of the preparation in this article: “The most straightforward approach is to estimate the cost of a failure and then multiply by the probability it will occur.”

Now, if the probability of having such failure is higher in the cloud than in private data-center, and cost of insuring against such failure is also higher, then overall costs, compared to private data center, rise significantly. Many cloud deployments do not include the cost of redundant infrastructure in their economic models. Adding that cost, which often doubles the numbers, can make it more difficult to justify the cloud strategy.

In addition to the cost of insurance against widespread outages, there is the cost of bullet-proofing your software against local “mishaps”. People working with the clouds surely experienced disappearing instances, diminishing processing resources resulting from heavy processing loads of other tenants, etc. Dealing with all this takes time and effort, i.e. money, to address. That money also goes into financial models used to estimate cloud worthiness.

Of course, elasticity of the cloud and its on-demand availability, have virtually no replacement, and will therefore be always needed by small or growing businesses. But for stable and predictable IT needs, it remains to be seen how the economics play out and what makes more sense from the financial perspective.

How much code coverage do you really need?

This post was prompted by reading a number of categorical tweets from @unclebobmartin. In case you’re not familiar with Uncle Bob – he’s one of the most prominent Software Industry experts, author of Clean Code, signatory to Agile Manifesto. In late nineties he did a profound work on documenting best OO practices (SRP, open/closed, interface segregation, etc). So when he speaks – it’s worth at least consideration.

He takes a maximalistic approach to TDD and unit testing in general. It can be clearly seen from his tweets:

 “Two things. Repeatability and cost. Manual tests are horrifically expensive compared to automated tests.”

“Manual tests aren't tests; they are trials. And since there are humans involved, they are suspect.”

“What you are telling me is that I should be open to the possibility that some code shouldn't be tested. Hmmm..”
“100% code coverage isn't an achievement, it's a minimum requirement. If you write a line of code, you'd better test it.”

He goes on to compare software testing with other mundane but critical activities that are considered mandatory in other fields:

“A surgeon on the battlefield may not have time to wash thoroughly, but the risk of death and cost of treatment will be high.”

“Do accountants cover only 80% of their spreadsheets with double entry bookkeeping?”

“How many times have you seen major outages that were due to some silly code that some silly programmer thought wasn't worth testing?”

 

While all these points certainly have merits, they show only one side of the picture. The reality is that not all applications require such a meticulous testing. Not all application are of the same importance as surgeries on a battlefield or accounting of big $$$. (not to mention the “creative” accounting employed in many cases:).

An even more important point is that thorough code coverage does not guarantee absence of bugs. Even Uncle Bob admits that:

“Tests cannot prove the absence of bugs. But tests can prove that code behaves as expected.”

This is obvious considering that same misconceptions and logical mistakes that were put in the code by the developer, are not likely to be discovered by the same developer when testing his own code.

In the end it all boils down to ROI and pragmatism. Some apps need more testing than others. Some modules need more testing than others. Some bugs need more fixing than others. There will always be a judgment call about whether additional time and money spent on automated testing and coverage are justified or are just a premature optimization.

Layers Are Not Tiers

I know what you're thinking: well duh! And yet often the distinction gets lost somewhere between design of the software and deployment topology.

Layering is an important technique that allows to manage complexity, increase cohesion of the components and to isolate inevitable changes from impacting rest of the system.

Tiers usually refer to the deployment structure of the systems. Tiers are separated from each other for various operational and security reasons. My point is that multiple layers can often be deployed into the same application tier and should not be equated with them.

A lot of the misconception, at least in Java world, comes from the traditional JEE stack: presentation layer, business layer, DB layer. Those are valid distinctions that should exist at the code level, but not necessarily in deployment. For example, it often makes sense to deploy presentation layer and business layer together for the sake of performance.

The standard deployment topology looks something like this:

But sometimes there is not need to separate Presentation Layer from Business Layer into different deployment tiers. Combining them allows us to avoid overhead of serializing and transporting data between the tiers. So the resulting topology would look like this:

Furthermore, sometimes it even makes sense to access DB directly from presentation layer, entirely skipping the business layer for fast reads of DB data. This does not mean having SQL code mixed in the pages, it just means that data extraction does not need to be done in fat EJB-like components. It can come straight from DB views and be wrapped by a light data access layer used by the MVC controllers. This might sound like a heresy but do we really need this useless layer of empty services that just pass data from DB to presentation?

Of course “writes” need to be processed by the business components and have all business rules applied before landing in DB. Updates are usually done on a per-object level and benefit from rich domain model. But “reads” often just need highly optimized and joined data from DB, especially if it’s a read-only data.

For all these reasons it’s important to distinguish layers from tiers. Multiple layers can be combined into the same deployment tier and sometimes even have some of the layers skipped.

Sonar: feel the pulse of your code

Regular code reviews are like physical exercise: everybody agrees how important it is but only few have time and energy for it. Yet, code reviews are one of the best ways to reduce defects and associated waste in debugging time, not to mention the overall product reliability.

There is a distinction between low-level tedious review of syntax of programs and review of their business functions. It is the low-level syntax and structure that are the ones that take most time and are also least interesting.

Reviews for technical components of the program have been automated for quite some time. Tools like PMD, FindBugs and Checkstyle are very popular Eclipse plugins and represent the first line of defense against simple but dangerous bugs. The problem with those tools is that they are intended for separate individuals and their reporting is done at a relatively low-level of detail.

Enter Sonar (see screenshots below).

Sonar is a web-based tool that analyzes your code using countless different metrics. It checks violations of coding rules similar to PMD/FindBugs/CheckStyle. It also presents an extremely useful dashboard of aggregated stats such as LOC, classes, duplications %. It even keeps track of how those metrics were changing in time. Best of all it’s easily integratable with Maven and Hudson and works seamlessly behind the scenes on each build.

Having high-level overview of code IMO is extremely important especially when the code base is large. Stats like coding-rules violations, test-coverage, complexity-per-method or per-class, cohesion are conveniently presented in numerical form. And these are just few of dozens (if not more) relevant stats.

If you want to see thorniest places in the code – go to Hotspots view. It ranks parts of your code using various metrics of “badness”. These include most violated rules, most untested classes, most complex classes, etc. Hotspots view is linked with Violation Drilldown views where you can expand areas of interest. In addition to that, Design view can be used to find cyclical dependencies between packages (a big no-no in proper OO design).

Ability to observe how these metrics change in time is very important because it allows to trace meaningful degradation in code quality down to individual code modification (i.e. when someone submits a bunch of spaghetti).

Overall Sonar is a great tool that exposes previously hidden data about code and presents it in a very convenient, interactive and intuitive format.

If you want to keep you code clean (no pun intended on Uncle Bob’s book) – Sonar is your tool.

Dashboard

Hotspots

Timeline