2016, Mesos and Kubernetes

28 Dec 2015

As my last post of the year, I wanted to write few words on Mesos and Kubernetes. The two have reached a great level of maturity and a lot of blogposts have been written during this year. I’m confident that, during the next year, the adoption will grow even more, so it’s time for a recap to prepare ourselves for 2016. In this blogpost I want to discuss the current status of Mesos and Kubernetes and give a little list of PROs and CONs to have a rough idea to start the year with a cluster manager.

Mesos and Kubernetes

Before starting, I must spend some words writing a rough comparison of Mesos and Kubernetes. This isn’t an easy task as Mesos doesn’t exactly do the same job of Kubernetes, at least not alone. To understand why, we need to quickly dive into how Mesos and Kubernetes work.

Mesos is a cluster manager based on the idea of two-level scheduling. To highly simplify this, Mesos is used to manage the resources of a cluster and to offer them to higher level “frameworks” that can accept these resources to launch tasks.

Kubernetes instead is a cluster manager designed on the concepts behind Borg, Google’s cluster manager with no concept of two level scheduling.

When talking about Mesos, I usually talk about Mesos “stack” as I find extremely not correct to only talk about Mesos: as we said before, due to the two-level scheduling, you need a “Mesos framework” (i.e. Marathon, Aurora, Singularity) to be able to use Mesos as you would do with the Kubernetes scheduler, for example. A popular (paid) solution to manage a cluster using Mesos with enterprises support is DCOS.

Apart from the two level scheduling, there are many differences, such as their dependencies (Zookeper, Etcd, etc.) and how they are used. It’s not my objective here to discuss the details or the internals of Mesos and/or Kubernetes, I would rather go straight to discuss some PROs and CONs about the two.

Current state, PROs and CONs

What’s clear when having a look at Mesos and Kubernetes is that the two are in a very different stage of their lives. Mesos is being pushed by Mesosphere and targeted many successful milestones during this year, while Kubernetes reached the first stable release in July. This means dealing with a widely used product with a relatively long history project (Mesos) vs a shiny, nicely designed new project (Kubernetes).

I could talk for hours about the two, but I really think that currently there’s no goto solution to adopt in case you are thinking about using them for your cluster(s), it’s thus important to highlight some PROs and CONs to give you a quick idea before getting your hands dirt.

Mesos stack

PROs

• Proven to scale, used at Apple with 75.000 nodes
• Extremely flexible. It is currently used in production by many enterprises and there are many Mesos frameworks available that can fit different needs.
• Relatively old, easier to find production stories and best practices to use it.
• It can use different formats than docker containers.

CONs

• Born for a different era and adapted to the age of docker containers. This is still evident in some design decisions, among which the way in which some of the cluster information are exposed, via a Json API, which is a nightmare to use.
• Too many languages. As devops I want to know what I’m running on my cluster and I want to be able to fix minor issues in case I found them in the codebase. To have a running Mesos stack, many components are involved: Mesos (C++), Marathon (Scala), Mesos-DNS (Golang), etc. It is not so common to find developers that can be proficient in so many languages at the same time.
• All the scheduler miss better abstractions for microservices like Pods, Services, Namespaces. These could be easily implemented, but are not there yet.

Kubernetes

PROs

• Extremely well designed API.
• Pods, Services, Namespaces are the right abstractions for microservices.
• Vibrant, fast moving community.
• One language, Golang.

CONs

• Very young, bugs are there waiting for you.
• No known big company using it at scale. Google itself is still using BORG and while many companies are migrating to Kubernetes there are no big known production clusters at the time of writing.
• Docker only (at the moment).

Conclusion

There’s a lot more to say about the two and there’s no clear winner in the fight for the best cluster manager. If you want to use a cluster manager I strongly recommend to try both Mesos and Kubernetes as there are a lot of things to learn from both and you will probably want to know both before using something in production. Let’s see if 2016 will really be the year in which Kubernetes will be mature and used in production and how the Mesos ecosystem will react to this competition.