Add Safety to your Ansible Automations Using Ansible for automation really makes the work of systems administrators scalable as it gives to the sysadmin the power to launch sets of commands on groups of machines. While some other automation tools such as Chef, Puppet or Saltstack also solve the same problem of scalability and saving…
Details
Applying Data Mesh for Fintech Industry (Part1) If you are working in the area of data and specially if you are a data engineer or data architect you probably have heard about Data Mesh as a paradigm shift or improvement over the well know data hub or data lake paradigms. Data mesh it is certainly…
Details
In the context of streaming applications Kafka has been a very used technology to solve real time and exactly-once-processing problems by many big and medium companies. It also has been used as a temporary logs in which all applications could benefit from brokerage and read from multiple destinations the same incoming data. With the introduction…
Details
In this post you’ll learn how to ingest data files in near real time using Spark Structured Streaming framework and storing them in Cassandra database. Spark Structured Streaming allows the programmer to implement the streaming solutions by means of dataframes instead of RDDs. This makes the development much more agile while preserving the performance thanks…
Details
Dear reader, in this post you’ll learn how to deploy a Kafka cluster and a Flink cluster. You’ll also learn how to make a simple application in Flink to process a stream of event coming from a Kafka producer. The tutorial is intended to prepare the reader to have this first experience with both technologies…
Details
Introduction It is very common that the machine learning (ML) models are developed locally. Specially when de task is done within a small data scientists group. However, to allow those ML models to be highly used by other services it is needed to put them in a large infrastructure so they can be highly available…
Details
Architecture diagram In this tutorial Kubernetes will be used to create a Spark cluster from which parallel jobs will be launched. The launch of the jobs won’t be done directly through the master node of the Spark cluster but from another node running an instance of Airflow. This provides more control over the executed jobs…
Details
This tutorial is the continuation of the tutorial Data Federation with Presto on Docker. After having acquired some understanding and experience of Docker now it is turn to deploy these Docker images into a managed cluster with Kubernetes. Kubernetes allows to orchestrate a cluster of microservices in which each service corresponds to one container.…
Details
Data Federation with Presto on Docker Introduction This post is about describing how to set a Presto installation in the form of a cluster on top of Docker containers. On the process of setting up the cluster several useful features of Docker will be presented. However, the main purpose is to show how Presto can…
Details
Online Platform for Clinical and Molecular data Retrieval Introduction This post is about describing a technologies stack used in a project aimed to create an online platform for clinical and molecular data retrieval. The online platform also includes a functionality to analyse molecular data stored in the cloud. The context of the project is in…
Details