Apache Spark 2.3 brought initial native support for Kubernetes. With the recent release of Spark 2.4 the integration has been improved and client mode is now supported. Time to go through step-by-step and run some primitive Spark jobs.…
Read more »Having a jump host, bastian host (on AWS, on GCP) or sometimes called edge node is a common way to access computing resources which are not accessible otherwise. In this post I’ll explore how to create a Kubernetes pod acting as such a jump server within the cluster, which is then used to create more pods. I’ll need this later when running Spark on Kubernetes, but you can use it to be able to check the cluster’s internal network, too.…
Read more »This sweet cloud promise, it sounds so simple - log into your cloud account, select a machine type and hit start. But as soon as you are running a dozen machines in all parts of the world for different projects and teams, you will start struggling with pulling apart your bill and granting rights to different people knowing which is what. Then you will find: this becomes unmanageable quite fast. Sure enough, every major cloud provider offers some kind of solution for this organisational mess, and this blog post will try to do a comparision. It is separated into two parts: the simple way, and the enterprise way.…
Hetzner Cloud: Die kleinste mögliche Cloud ist Anfang 2018 an den Start gegangen. Man bekommt schon ganz viel, obwohl fast alles noch fehlt.…
Read more »Recently, Apache Hadoop 3 has been released. Really, who cares? Everyone and his aunt is jumping onto the Kubernetes train. While maybe it seems that Hadoop and Kubernetes are very different beasts, in reality, they are not. It’s kind of the same thing, but in different flavors. Yet, Kubernetes looks like the overall winner.…
Read more »Joel Spolsky’s essays are legend. His collection of posts on his blog “Joel on Software” in the years from 2000 onwards have been very influential in the development community, not only because of their engaging writing style, but because they set clear directions for tech companies. One specific dogma he successfully implanted in my head was: “Never ever do a complete software rewrite” (You can read it here, please return afterwards). This probably is still very true today, because
“when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time”.
Let’s stress this dogma.
… Read more »I’m an IT consultant. Never did I suggest to any of my clients they should interrupt a well-running application in production. Maybe that was not as professional as it might seem. Let’s see why.…
Read more »Early February 2017, I got email from Google saying I had passed the Cloud Architect certification exam. Yay! This post reflects what helped me preparing for the test.…
Read more »Recently, I’ve been involved with the Clouds. Not so much everybody’s darling AWS, but more the runner-ups in the race: Google Cloud and Microsoft Azure. Both cool kids, but everyone with his its very different style and philosophy of what it considers “cool”.…
Read more »