L̶u̵m̶i̵n̷o̴u̶s̶m̶e̵n̵B̶l̵o̵g̵

HTTP/2 standard not being adopted yet fully, it may be too early to make a widespread push for HTTP/3. The major difference between HTTP/2 and HTTP/3 is which transport protocol they use. Instead of TCP, HTTP/3 uses a new protocol called QUIC. QUIC is a general purpose transport protocol meant to address the head-of-line blocking issues HTTP/2 has with TCP. It allows you to create a series of stateful streams (similar to TCP) over UDP. It gives ​new performance and security benefits that new HTTP/3 protocol offers. https://portswigger.net/daily-swig/http-3-everything-you-need-to-know-about-the-next-generation-web-protocol

Introduction to Change Data Capture pattern and it's modern design approaches https://luminousmen.com/post/change-data-capture

When you look around you come to the conclusion that many people pay too little attention to what they say and how they say it https://luminousmen.com/post/be-the-idiot

Post about hidden gem in CPython - interning mechanism. https://luminousmen.com/post/interning-in-cpython

NiFi and Apache Airflow are the best ETL tools. What separates them, what are their greatest strengths and weaknesses, which is suitable for your needs? https://www.astronomer.io/blog/apache-nifi-vs-airflow

Apache Hudi pioneered the serverless transactional layer for event logs that significantly shape the data infrastructure. The article gives an in-depth overview of Apache Hudi's building blocks and future roadmap aligning with its founding principle.

Don't go into management. Or go, it's up to you. My job is to show the "what if" picture. https://luminousmen.com/post/management-is-not-a-promotion

Interviews are thoroughly broken and I have no idea how to actually make them better.

Caching, as trivial as it may seem, is a difficult task for engineers. https://luminousmen.com/post/spark-tips-caching

Hive meta store is a critical component in the interception of all query engines path provides a virtualization layer between the storage and compute. What makes Hive Metastore so special? This article attempts to describe some of them and says that the Hive metastore is not going anywhere in the near future.

I remember being 10 I was thinking the teachers knew what was going on. Now I’m there I realize they were still figuring this shit out too.

Apache Hadoop provides the following filesystem clients for reading from and writing to Amazon S3: S3N (URI scheme: s3n) - A native filesystem for reading and writing regular files on S3. S3N allows Hadoop to access files on S3 that were written with other tools, and conversely, other tools can access files written to S3N using Hadoop. S3N is stable and widely used, but it is not being updated with any new features. S3N requires a suitable version of the jets3t JAR on the classpath. S3A (URI scheme: s3a) - Hadoop’s successor to the S3N filesystem. S3A uses Amazon’s libraries to interact with S3. S3A supports accessing files larger than 5 GB, and it provides performance enhancements and other improvements. For Apache Hadoop, S3A is the successor to S3N and is backward compatible with S3N. Using Apache Hadoop, all objects accessible from s3n:// URLs should also be accessible from S3A by replacing the URL scheme. Note Amazon EMR does not currently support use of the Apache Hadoop S3A file system. S3 (URI scheme: s3) - Apache Hadoop implementation of a block-based filesystem backed by S3. Apache Hadoop has deprecated use of this filesystem as of May 2016. Just want to save it here. Original in web archive #aws

Accidental complexity is stuff you have to do, like builds and infrastructure configuration and so forth, whereas essential complexity is things like the code your customers want you to deliver. We as a developers want to do essential 100% of the time and throw everything else over the fence to other teams. Now with microservices it become a harder thing to do, but service mesh can solve some of the problems. "Service mesh" is an umbrella term for products that seek to solve the problems that microservices’ architectures create. These challenges include security, network traffic control, and application telemetry. Service mesh is typically implemented as a scalable set of network proxies deployed alongside application code (a pattern sometimes called a sidecar). These proxies handle the communication between the microservices and also act as a point at which the service mesh features can be introduced. They trying to decouple the platform-level features from the application itself. For those who's interested this is a good article on the evolution and current problems that service mesh is solving. But yeah, you got it — it's another layer of problems, probably more complex, but who cares? DevOps can handle that, right? Right?!

Good digging into optimizing Apache Spark join operations with or conditions: https://sujithjay.com/spark/or-within-joins #spark

«There's this one quiet guy who's the absolute beast of excel spreadsheets analytics.» Cool introductory video about data engineering for those who are not in the field yet. https://youtu.be/qWru-b6m030 Btw, we have this one book to help new or not that new engineer, just FYI #big_data

No one knows how to manage programmers — and everyone comes up with workarounds instead of solutions. One reality is the product, and the real engineers who improve it, because they want to and because they can. And guess what — Jira board is bullshit. I've seen horrible products that had a Kanban board in perfect condition, and great products with several developers that told everyone to go fuck themselves. And the only person who can synchronize the real state of the project with the Jira tickets is a Team Lead. We live in a horrible world where Jira ticket circulation is considered a positive thing, while the real improvement of the product remains unnoticed. I think I'll get another drink. #soft_skills #leadership

I'm sorry, but I think we need to sort out the terms in Data Engineering - I'm starting to get confused myself. Let me give you some thinking on what I usually mean: Pipeline - a sequence of stages Stage/Job - includes several tasks Task - a single atomic process to be executed (script, command, utility) Workflow - automation technology of a business process The difference between pipeline and workflow: Pipeline a clearly described process in which tasks are executed sequentially, or processes executed in parallel. Workflow is usually non-linear and has an often abstract human description, processes may not run in parallel. Workflow has branches and loops. #big_data

While there are a lot of advantages of using Cloud Dataproc, there are a lot of pitfalls that inexperienced engineers can run into. And in this post, we'll take a look at some things you should take into account when you're building Cloud Dataproc-based pipelines or architecture using it. https://luminousmen.com/post/things-to-consider-while-running-google-cloud-dataproc

Who is responsible for the engineering team failures? If you're a Team Leader of a team then it should be your moto: Failures are always mine, and successes are always teams. I think this is a good principle to build a good team and good company. Take care of the engineers. Customers, management, other departments often pour heaps of shit on them when the team makes mistakes. If you are a Team leader, you need to filter the negative and convert it to a positive. Take the hit for team mistakes. If the team failed, it means you build up bad processes. If the team succeed, it is their success and it doesn't matter if it is true or not. When I first tried myself as a team lead I was angry at one of my collegues who made a mistake, and since I'm a fairly straightforward person — I said it directly. Quite quickly I started to understand that it's a dead-end road and does not solve the problem. I began to understand that it is not engineers who work for a leader, but the leader exists to serve people. The goal of a leader is to serve. The goal of a team lead is to help the team. Do not play a blame game, you need to look for the root cause of the problem. And most likely, it lies in the organization of work. Just discuss these problems quietly in retrospective meeting, find the root cause, create action items and do them. #soft_skills #leadership