How I reduced 48% of my cloud cost by using Google Cloud preemptible instances

Google Cloud Platform has an amazing feature that few people use, partially because it is unknown, but mainly because it is very difficult to set up a system architecture that allows you to use. This feature is preemptible instances. How does it work? Simple: you have a virtual machine like any other, except that this VM will shutdown unexpectedly within 24 hours and be eventually unavailable for short periods. The advantage: this preemptive instances cost less than 50% compared to the ordinary machine.

Usually, people use this kind of machine for servers that run workers or asynchronous jobs, a kind of application that does not need 24/7 availability. In my case, I could use the preemptible instances for my internal API, an application that do need 24/7 availability. This internal API can’t stay offline, so the way I solved the unavailability problem was by running many servers in parallel  behind a haproxy load balancer. So, in basically 3 steps I could reduce my API infrastructure cost by 50%.

Step 1 – Setup the client to be fault tolerant

My code is in Scala language. Basically, I made the client to repeat a request when it eventually failed. This is necessary because, even if the API machines are behind the load balancer, the load balancer takes some time (seconds) to realize that a specific machine is down, so eventually it sends some requests to unavailable machines. The client code snippet is:

def query(params, retries = 0) {
  val response = api.query(params)
  response.onSuccess {
    codeForSuccess()
  }
  response.onFailure {
    case x => {
      LOG.error(s"Failure on $retries try of API request: " + x.getMessage)
      Thread.sleep(retries * 3000) //this sleep is optional
      query(params, retries + 1) //the could be a maximum number of retries here
    }
  }
}

Step 2 – put all servers behind a load balancer

I created a haproxy config file that I can auto-update based on a list of servers that I get from the gcloud command line. Here is the script that re-writes the haproxy config file with a list of all servers that has a specific substring in their names:

#!/bin/bash
SERVER_SUBSTRING=playax-fingerprint
EMPTY_FILE=`cat /etc/haproxy/haproxy.cfg |grep -v $SERVER_SUBSTRING`
NEW_LINES=`gcloud compute instances list |grep $SERVER_SUBSTRING | sed 's/true//g' |sed 's/ [ ]*/ /g'|cut -d" " -f4|awk '{print " server playax-fingerprint" $NF " " $NF ":9000 check inter 5s rise 1 fall 1 weight 1"}'`
echo "$EMPTY_FILE" >new_config
echo "$NEW_LINES" >>new_config
sudo cp new_config /etc/haproxy/haproxy.cfg
sudo ./restart.sh

The restart script reloads the haproxy configuration without any outage.

Step 3 – create an instance group for these servers

By creating an instance template and an instance group, I can easily add or remove servers to the infrastructure. The preemptible configuration is inside the instance template page in google cloud panel.

  1. Create an instance template with preemptible option checked
  2. Create an instance group that uses that template

Screen Shot 2016-05-04 at 10.40.58 PM

 

Screen Shot 2016-05-04 at 10.41.18 PM

One very important warning is that you need to plan your capacity to allow 20% of your servers to be down (remember that preemptible instances eventually are out). In my case, I had 20 servers before using the preemptible option. With the preemptible on, I changed the group to 25 servers.

Before After
Servers 20 24
Cost per server $0.07 $0.03
Total cost per hour $1.4 $0.72
Total cost per month $1,008 $518

Price reduction:  $490 or 48.6%

Graphs of server usage along 1 day (observe how many outages there are, but application ran perfectly ):

Screen Shot 2016-05-04 at 11.12.36 PM

Migrating from RubyOnRails to Scala Play 2.0 (part 1)

play_full_colorThe last week I was on vacation and I used the time to study a new technology (things that nowadays I don’t have time to do in daily life).

I decided to migrate my wife’s website (for those who don’t know, my wife is the very famous brazilian singer Daniella Alcarpe, known also as @cantora).

The website is quite simple and easy to develop. Actually, there is no rocket science at all in everything I did. But I had a lot of fun and I learn a lot of things. This post cannot be used as a complete migration plan for those who want to migrate from Rails to Play, but it will give you an idea of the work you will have. Basically, you will have to re-write everything from scratch… :-) or almost this…
Continue reading

Software Developers Retreat (dia 4)

O quarto dia do nosso retiro de desenvolvedores acabou!

Mesão de trabalho

Mesão de trabalho

Recebemos a visita do nosso CEO. Ele veio junto com um programador do time que não pôde participar do retiro por causa de compromissos pessoais. Esse programador era um especialista em churrasco, pois nasceu no sul do Brasil e morou muitos anos “no meio do mato”. Logo que chegou, ele se prontificou a tomar conta do nosso almoço. Foi até a cidade, no mercado central, e explicou pro açougueiro a maneira certa de cortar a carne :-) Depois voltou para a casa e comandou a churrasqueira, enquanto o resto do pessoal trabalhava. Continue reading

Software Developers Retreat (dia 3)

O terceiro dia do nosso retiro de desenvolvedores acabou!

Standup Meeting

Standup Meeting

O sol estava brilhando e bateu forte na nossa janela. Como o pessoal tinha ido dormir um pouco mais tarde, também acabamos acordando um pouco mais tarde. Como de costume, fizemos nosso standup meeting as 11h. A moça que contratamos para limpar a casa e cozinhar não veio nesse dia. Acabamos tendo que dar um jeito com as tarefas da casa. Continue reading

Software Developers Retreat (dia 2)

O segundo dia do nosso retiro de desenvolvedores acabou!

Trabalhando Juntos

Trabalhando Juntos

Acordamos lá pelas 8 e meia, 9 horas. Também teve gente que acordou mais cedo para fazer um cooper. Como de costume, fizemos nosso Standup Meeting as 11h da manhã. Recebemos a visita do nosso diretor de produtos. Ele veio de São Paulo para cá, passar o dia com a gente. Foi muito importante sua vinda, pois pudemos conversar em mais detalhes sobre algumas histórias que estávamos implementando.
Continue reading

Software Developers Retreat (dia 0)

Já faz alguns meses que eu estou planejando esse evento, que finalmente tomou forma e terá início a partir de amanhã, dia 26/nov/2012.

A ideia não é nova, embora eu tenha visto poucas organizações, empresas ou pessoas executando ela de fato. Tudo surgiu de uma conversa com meu grande amigo Alexandre Freire, que já tinha vivido a experiência de levar programadores para a sua casa no bonete na Ilhabela em São Paulo. A aventura dele está documentada no blog do estúdio livre.
Continue reading

10 SEO tips to improve your website ranking


Reaching the first place in search engines race is a hard and long term job. Since I started my work in Elo7, we’ve been doing a lot of improvements in our platform to boost our position in search engines. I’d like to share some of these techniques with everyone. Before doing it, just a small disclaimer: if your website has a content that is adjusted to the search keywords you want be found, these tips are useful. I think this is the most important advice in SEO: you need to have good content. Content that is interesting for the people who search for specific terms.
Continue reading

SPLASH (antiga OOPSLA)

OOPSLA

Em 1985, um grupo de 4 pioneiros em programação orientada a objetos decidiu organizar nos EUA uma conferência sobre programação de sistemas orientados a objetos. No grupo estavam Adele Goldberg, Tom Love, David Smith, and Allen Wirfs-Brock, e a conferência foi chamada de OOPSLA (Object-Oriented Programming, Systems, Languages, and Applications). A primeira OOPSLA aconteceu no Hotel Marriott, em Portland, Oregon, em novembro de 1986. Cerca de 600 pessoas participaram, 50 artigos foram apresentados e os participantes ouviram sobre Smalltalk, Lisp, Flavors, CommonLoops, Emerald, Trellis/Owl, Mach, Prolog, ABCL/1, prototypes, e programação concorrente e distribuída de pessoas como Danny Bobrow, Gregor Kiczales, Rick Rashid, Andrew Black, Dave Ungar, Henry Lieberman, Ralph Johnson, Dan Ingalls, Ward Cunningham, Kent Beck, Ivar Jacobson e Bertrand Meyer.

Continue reading