1

I think about installing an as small as possible kubernetes setup on scaleway. The idea is to prepare myself with a kind of MVP that is able to run my applications components and turn it into a full blown redundant setup when usage grows.

The tutorial at https://www.tauceti.blog/post/kubernetes-the-not-so-hard-way-with-ansible-at-scaleway-part-1/ mentions that

  • etcd should not be installed on controller nodes but on separate VMs
  • etcd needs at least three nodes

What's the reason for recommending separate installs and could I run only one etcd on a controller ?

Please consider that I only search for a functional setup and not a highly available.

Marged
  • 701

1 Answers1

3

etcd needs at least three nodes

This is true of any distributed system that has a leader/master to maintain consistency if you require it to be fault tolerant. In fact, you need an odd number of nodes to ensure that if the cluster cannot get evenly split in two (due to network outage for example) as when this happens neither side can elect a leader and the whole cluster will become unavailable. Three happens to be the minimum number that is able to tolerate a single node going down without affecting the uptime of the cluster.

If you do not require a highly available system you can get away with a single node but non highly available solutions are not recommended for production use for obvious reasons - you are always free to ignore this advice if your system is smaller enough and you understand the risks of the system falling over or cannot justify the expense of the extra nodes.

etcd should not be installed on controller nodes but on separate VMs

This is also a stability/scalability issue - you are free to mix controller nodes with compute nodes in most distributed systems but they can struggle in this setup when under high load. If you don't have enough nodes to create a three node cluster then you don't have enough nodes to stress the systems to a point where this will matter.

Both of these issues can be addressed when your system grows to a point where the nodes start to struggle or you can warrant the cost of setting up the extra nodes.

You could start off with one node for a MVP but both kubernetes and etcds guides are geared towards a distributed setup and you only really benefit from them being setup in a cluster. You may also encounter issues trying to grow it from one node to three nodes. If you can afford a three node setup then I would start off with that and just have all the nodes start with all of the services splitting them out when you want to grow the setup further.

Michael Daffin
  • 3,948
  • Thanks for the clarification. I understand that a "good" setup would be (n controller nodes) + (3 etcd nodes) + (several workers). Reading I would ... just have all the nodes start with all of the services splitting them out I will go for a total of 3 VMs, put etcd and the controller stuff on all of them and make them workers too (or perhaps only one of them and give that one more CPU and RAM). Does this sound like a good plan ? – Marged Oct 28 '17 at 06:39
  • I would make all three of them identical and each run all of the services - then you can lose any of the nodes without loss of functionality and with only three nodes the controllers will not be doing enough to really warrant more cpu/memory. Then they are all the same and you do not have to worry about treating the controller node differently. You can add extra worker nodes to this setup and then when you get enough nodes remove the workers from the controllers - if you grow further you can consider splitting the kubernetes and etcd controllers up onto their own machines. – Michael Daffin Oct 28 '17 at 11:10