How to use Google Cloud AutoML Vision to identify when an audio file is a song or a voice conversation
Post Category → Programming
How to create a Pokemon Twitter chatbot on Google Cloud in 20 minutes
How I reduced 48% of my cloud cost by using Google Cloud preemptible instances
Google Cloud Platform has an amazing feature that few people use, partially because it is unknown, but mainly because it is very difficult to set up a system architecture that allows you to use. This feature is preemptible instances. How does it work? Simple: you have a virtual machine like any other, except that this VM will shutdown unexpectedly within 24 hours and be eventually unavailable for short periods. The advantage: this preemptive instances cost less than 50% compared to the ordinary machine.
Usually, people use this kind of machine for servers that run workers or asynchronous jobs, a kind of application that does not need 24/7 availability. In my case, I could use the preemptible instances for my internal API, an application that do need 24/7 availability. This internal API can’t stay offline, so the way I solved the unavailability problem was by running many servers in parallel behind a haproxy load balancer. So, in basically 3 steps I could reduce my API infrastructure cost by 50%.
Step 1 – Setup the client to be fault tolerant
My code is in Scala language. Basically, I made the client to repeat a request when it eventually failed. This is necessary because, even if the API machines are behind the load balancer, the load balancer takes some time (seconds) to realize that a specific machine is down, so eventually it sends some requests to unavailable machines. The client code snippet is:
def query(params, retries = 0) { val response = api.query(params) response.onSuccess { codeForSuccess() } response.onFailure { case x => { LOG.error(s"Failure on $retries try of API request: " + x.getMessage) Thread.sleep(retries * 3000) //this sleep is optional query(params, retries + 1) //the could be a maximum number of retries here } } }
Step 2 – put all servers behind a load balancer
I created a haproxy config file that I can auto-update based on a list of servers that I get from the gcloud command line. Here is the script that re-writes the haproxy config file with a list of all servers that has a specific substring in their names:
#!/bin/bash SERVER_SUBSTRING=playax-fingerprint EMPTY_FILE=`cat /etc/haproxy/haproxy.cfg |grep -v $SERVER_SUBSTRING` NEW_LINES=`gcloud compute instances list |grep $SERVER_SUBSTRING | sed 's/true//g' |sed 's/ [ ]*/ /g'|cut -d" " -f4|awk '{print " server playax-fingerprint" $NF " " $NF ":9000 check inter 5s rise 1 fall 1 weight 1"}'` echo "$EMPTY_FILE" >new_config echo "$NEW_LINES" >>new_config sudo cp new_config /etc/haproxy/haproxy.cfg sudo ./restart.sh
The restart script reloads the haproxy configuration without any outage.
Step 3 – create an instance group for these servers
By creating an instance template and an instance group, I can easily add or remove servers to the infrastructure. The preemptible configuration is inside the instance template page in google cloud panel.
- Create an instance template with preemptible option checked
- Create an instance group that uses that template
One very important warning is that you need to plan your capacity to allow 20% of your servers to be down (remember that preemptible instances eventually are out). In my case, I had 20 servers before using the preemptible option. With the preemptible on, I changed the group to 25 servers.
Before | After | |
Servers | 20 | 24 |
Cost per server | $0.07 | $0.03 |
Total cost per hour | $1.4 | $0.72 |
Total cost per month | $1,008 | $518 |
Price reduction: $490 or 48.6%
Graphs of server usage along 1 day (observe how many outages there are, but application ran perfectly ):
Migrating from RubyOnRails to Scala Play 2.0 (part 1)
The last week I was on vacation and I used the time to study a new technology (things that nowadays I don’t have time to do in daily life).
I decided to migrate my wife’s website (for those who don’t know, my wife is the very famous brazilian singer Daniella Alcarpe, known also as @cantora).
The website is quite simple and easy to develop. Actually, there is no rocket science at all in everything I did. But I had a lot of fun and I learn a lot of things. This post cannot be used as a complete migration plan for those who want to migrate from Rails to Play, but it will give you an idea of the work you will have. Basically, you will have to re-write everything from scratch… 🙂 or almost this…
Continue reading
Software Developers Retreat (dia 4)
O quarto dia do nosso retiro de desenvolvedores acabou!
Recebemos a visita do nosso CEO. Ele veio junto com um programador do time que não pôde participar do retiro por causa de compromissos pessoais. Esse programador era um especialista em churrasco, pois nasceu no sul do Brasil e morou muitos anos “no meio do mato”. Logo que chegou, ele se prontificou a tomar conta do nosso almoço. Foi até a cidade, no mercado central, e explicou pro açougueiro a maneira certa de cortar a carne 🙂 Depois voltou para a casa e comandou a churrasqueira, enquanto o resto do pessoal trabalhava. Continue reading
Software Developers Retreat (dia 3)
O terceiro dia do nosso retiro de desenvolvedores acabou!
O sol estava brilhando e bateu forte na nossa janela. Como o pessoal tinha ido dormir um pouco mais tarde, também acabamos acordando um pouco mais tarde. Como de costume, fizemos nosso standup meeting as 11h. A moça que contratamos para limpar a casa e cozinhar não veio nesse dia. Acabamos tendo que dar um jeito com as tarefas da casa. Continue reading
Software Developers Retreat (dia 2)
O segundo dia do nosso retiro de desenvolvedores acabou!
Acordamos lá pelas 8 e meia, 9 horas. Também teve gente que acordou mais cedo para fazer um cooper. Como de costume, fizemos nosso Standup Meeting as 11h da manhã. Recebemos a visita do nosso diretor de produtos. Ele veio de São Paulo para cá, passar o dia com a gente. Foi muito importante sua vinda, pois pudemos conversar em mais detalhes sobre algumas histórias que estávamos implementando.
Continue reading
Software Developers Retreat (dia 1)
Software Developers Retreat (dia 0)
Já faz alguns meses que eu estou planejando esse evento, que finalmente tomou forma e terá início a partir de amanhã, dia 26/nov/2012.
A ideia não é nova, embora eu tenha visto poucas organizações, empresas ou pessoas executando ela de fato. Tudo surgiu de uma conversa com meu grande amigo Alexandre Freire, que já tinha vivido a experiência de levar programadores para a sua casa no bonete na Ilhabela em São Paulo. A aventura dele está documentada no blog do estúdio livre.
Continue reading
10 SEO tips to improve your website ranking
Reaching the first place in search engines race is a hard and long term job. Since I started my work in Elo7, we’ve been doing a lot of improvements in our platform to boost our position in search engines. I’d like to share some of these techniques with everyone. Before doing it, just a small disclaimer: if your website has a content that is adjusted to the search keywords you want be found, these tips are useful. I think this is the most important advice in SEO: you need to have good content. Content that is interesting for the people who search for specific terms.
Continue reading