{ "generatedAt": "2026-10-20T14:02:44.182Z", "posts": [ { "id": "datamesh-from-the-bottom-up-what-swe-already-knew", "title": "Data Mesh From the Bottom Up: What Software Engineering Already Knew", "date": "2026-10-18", "category": "Data Engineering", "tags": [ "data-mesh", "ddd", "data-contracts", "team-topologies" ], "summary": "Data Mesh starts with DDD's strategic design applied to data: domains, bounded contexts and context maps that software engineers already know, unpacked from the bottom up.", "related": [], "banner": { "image": "content/images/data-mesh-30000-foot-view.svg", "fit": "contain" }, "readingTime": 11, "bodySnippet": "Data mesh was introduced by Zhamak Dehghani in 2019 [@dehghani2019] as an answer to the complexity of data projects at scale. The term is young, but not the ideas behind it: software engineers working in operational systems already know bounded contexts, domains, subdomains and explicit interfaces. As Dehghani argues, these concepts are essential to make teams responsible (and accountable) for wha", "bodyUrl": "data/bodies/posts/datamesh-from-the-bottom-up-what-swe-already-knew" }, { "id": "stream-processing-the-key-ideas", "title": "Stream Processing: The Key Ideas", "date": "2025-05-08", "category": "Data Engineering", "tags": [ "data-streaming" ], "summary": "The core concepts behind stream processing: unbounded data, motion through the system, and why it never truly rests.", "related": [], "readingTime": 7, "bodySnippet": "**Stream processing** is the discipline focused on processes and techniques used to extract information and value from *unbounded data.* This kind of data has undefined, theoretically infinite size, and often arrives in no particular order in the processing system. Even worse, it must be handled on limited, physical hardware. How could something infinite fit into something finite?\n\nWell, it doesn'", "bodyUrl": "data/bodies/posts/stream-processing-the-key-ideas" }, { "id": "sql-gaps-and-islands", "title": "The Gaps And Islands Problem", "date": "2025-04-06", "category": "Tutorials", "tags": [ "sql" ], "summary": "Solving the classic gaps-and-islands SQL problem with a worked, real-world example.", "related": [], "readingTime": 15, "bodySnippet": "The gaps and islands is a classical data analysis problem where we aim to identify gaps or contiguous ranges of values (islands). This problem is relevant to applications where we need to identify the interruption of a sequence. To illustrate the problem, let’s supose we have cards as ilustrated in the image below:\n\n![Eight cards sorted by their order number. Each card has an id at the top, an ord", "bodyUrl": "data/bodies/posts/sql-gaps-and-islands" }, { "id": "spark-spill-problem", "title": "The Spill Problem in Apache Spark", "date": "2025-03-22", "category": "Data Engineering", "tags": [ "spark" ], "summary": "Why Spark spills partitions to disk when executor memory runs out, and what that costs you.", "math": true, "readingTime": 13, "bodySnippet": "In the post *[Spark Executor and its memory](https://leandroasaservice.com/posts/spark-executor-memory-1/),* we explored tasks, their partitions, and how executor memory is divided into different regions, each with its own responsibilities [@spark-executor-memory]. We also mentioned that problems may arise when partitions are too large. That is what we are going to discuss.\n\nAs we have seen, data ", "bodyUrl": "data/bodies/posts/spark-spill-problem" }, { "id": "deploying-password-generator-application-on-aws-eks-pt2", "title": "Deploying a Password Generator Application in AWS EKS - Part 2", "date": "2025-02-18", "category": "Tutorials", "tags": [ "kubernetes" ], "summary": "Giving the EKS password generator a custom Cloudflare domain, a Let's Encrypt certificate via cert-manager, and sticky sessions with cookies or upstream hashing.", "related": [ "deploying-password-generator-application-on-aws-eks-pt1", "kubernetes-basic-architecture" ], "readingTime": 8, "bodySnippet": "In [the previous post](/posts/deploying-password-generator-application-on-aws-eks-pt1/) we started to deploy a password generator application in AWS EKS. By the end of it, two problems came to light:\n\n- There is no SSL certificate assuring that our domain is ours. It could be someone else, faking a connection to sniff data from your computer.\n- We need a ✨**remarkable**✨ address.\n\nWe are going to ", "bodyUrl": "data/bodies/posts/deploying-password-generator-application-on-aws-eks-pt2" }, { "id": "deploying-password-generator-application-on-aws-eks-pt1", "title": "Deploying a password generator application in AWS EKS - Part 1", "date": "2025-02-09", "category": "Tutorials", "tags": [ "kubernetes" ], "summary": "Deploying a distroless password generator app to AWS EKS, from cluster setup to a running Kubernetes deployment.", "related": [ "kubernetes-basic-architecture", "deploying-password-generator-application-on-aws-eks-pt2" ], "readingTime": 7, "bodySnippet": "In the post [Creating a distroless signed docker image](https://leandroasaservice.com/posts/docker-distroless-signed-docker-image-application/), a password generator application was shipped in a distroless signed docker image. In this tutorial, we are going to use that image and deploy the same application in AWS EKS.\n\nWe are not going to show how to install local dependencies or how to set an AWS", "bodyUrl": "data/bodies/posts/deploying-password-generator-application-on-aws-eks-pt1" }, { "id": "developing-testable-spark-batch-application", "title": "Developing a Testable Batch Spark Application", "date": "2024-12-29", "category": "Data Engineering", "tags": [ "spark" ], "summary": "Patterns for structuring Spark batch jobs so their business logic can actually be unit tested.", "math": true, "readingTime": 24, "bodySnippet": "## Introduction\n\nIn my experience, developing testable Spark applications code is not an easy task for data practitioners. I am not going to discuss the underlying reasons.\n\nIn this post, I present my reasoning while developing a testable batch Spark Application. The text is presented in two sections. The first section , *TDD - Developing code from the tests*, I show an example of how to develop a", "bodyUrl": "data/bodies/posts/developing-testable-spark-batch-application" }, { "id": "kubernetes-basic-architecture", "title": "Kubernetes Architecture - The Basics", "date": "2024-12-14", "category": "Notes", "tags": [ "kubernetes", "notes" ], "summary": "Study notes covering the core building blocks of Kubernetes architecture.", "readingTime": 6, "bodySnippet": "Kubernetes is an open-source container orchestrator popular among Software Engineers, DevOps Engineers, and it's gaining momentum in Data. In this post, I'm sharing the notes I took while studying Kubernetes Architecture. Before starting, I'd like to summarize some key-words.\n\n## The Jargon\n\n- **Agent:** it's a software that acts in behalf of an user or other software, which can also be an agent. ", "bodyUrl": "data/bodies/posts/kubernetes-basic-architecture" }, { "id": "docker-distroless-signed-docker-image-application", "title": "Creating a Distroless Signed Docker Image", "date": "2024-10-12", "category": "Tutorials", "tags": [ "docker" ], "summary": "Building a minimal distroless Docker image and signing it to reduce attack surface and verify provenance.", "readingTime": 6, "bodySnippet": "## Distroless Images\n\nDistroless container images, unlike the traditional ones, does not include software that are common in distro-based images, such as package managers and shells [@chainguard-distroless]. This approach aims to minimize the image size and reduce vulnerabilities by removing unnecessary components to run an application [@chainguard-distroless].\n\nThese types of images are suitable ", "bodyUrl": "data/bodies/posts/docker-distroless-signed-docker-image-application" }, { "id": "spark-why-should-we-be-careful-with-distinct", "title": "Why Should You Be Careful With DISTINCT?", "date": "2024-10-08", "category": "Data Engineering", "tags": [ "spark" ], "summary": "Careless DISTINCT clauses can silently tank Spark query performance -- here's why, and what to do instead.", "readingTime": 6, "bodySnippet": "If there's a chance a `DataFrame` contains duplicated rows, it's a good idea to deduplicate it before loading into the table. Better to be safe than sorry, right? Absolutely. But sometimes using `DISTINCT` clauses carelessly lead to serious performance issues. I think every data practitioner has made this mistake of adding `DISTINCT` clauses to every query and `DataFrame` to ensure no duplicated r", "bodyUrl": "data/bodies/posts/spark-why-should-we-be-careful-with-distinct" }, { "id": "spark-application-adding-job-description", "title": "Adding Job Descriptions Details to an Apache Spark Application", "date": "2024-09-30", "category": "Data Engineering", "tags": [ "spark" ], "summary": "Using setJobGroup to label Spark jobs so they're easy to trace in the Spark UI.", "readingTime": 6, "bodySnippet": "Having a clear job description in an Apache Spark application makes it easy to spot optimization opportunities. By using the `setJobGroup` method properly, you can quickly link code issues with what shows up in the Spark UI [@spark-setjobgroup]. In this short post, I'll show how to do just that.\n\n## The Problem\n\nLet's suppose we are running a benchmark for a simple application that only reads and", "bodyUrl": "data/bodies/posts/spark-application-adding-job-description" }, { "id": "spark-executor-memory", "title": "Spark Executor And Its Memory", "date": "2024-09-24", "category": "Data Engineering", "tags": [ "spark" ], "summary": "How Spark divides executor memory between on-heap and off-heap, and what the unified memory region actually does.", "math": true, "readingTime": 8, "bodySnippet": "In the *[Spark Application Architecture](https://leandroasaservice.com/posts/spark-application-architecture-en-us/)* post, we discussed Apache Spark architecture concepts. As we could see, tasks are the fundamental unity of work in Spark, and we are going to use it here to talk about Spark Executor and its memory.\n\nIn the section *\"Tasks and Partitions\"*, we are going to see the relation among tas", "bodyUrl": "data/bodies/posts/spark-executor-memory" }, { "id": "spark-application-architecture", "title": "Apache Spark Application Architecture", "date": "2024-09-22", "category": "Data Engineering", "tags": [ "spark" ], "summary": "A tour of Apache Spark's components, and how actions, transformations and lazy evaluation fit together.", "readingTime": 13, "bodySnippet": "In this post, I'd like to show some concepts for better understanding of Apache Spark applications. Most of the content here is available in many books, blog posts, paid courses and free YouTube videos. Here, I just compiled these materials and added some important details regarding my experience. This text is divided in three sections.\n\nThe first section, \"*Apache Spark Components Overview*\", I p", "bodyUrl": "data/bodies/posts/spark-application-architecture" } ] }