L̶u̵m̶i̵n̷o̴u̶s̶m̶e̵n̵B̶l̵o̵g̵

Conflicts are inevitable, especially at work. If you find yourself in one, you can take this general approach to how to resolve it, hopefully successfully https://luminousmen.com/post/handling-conflicts

Apache Spark supports loading data from tables using JDBC, but sometimes engineers use this interface blindly without thinking that it is not optimized by default https://luminousmen.com/post/spark-tips-optimizing-jdbc-data-source-reads/

Amazon EMR Serverless is a new deployment option in Amazon EMR that makes it easy and cost-effective for data engineers and analysts to run petabyte-scale data analytics in the cloud. With EMR Serverless, you can run applications built using open-source frameworks such as Apache Spark, Hive, and Presto without having to configure, manage, optimize, or secure clusters. https://youtu.be/i6jH-QjFuGA

In this article Uber writes about the details of current Apache Spark shuffle implementation. And its presenting their own implementation of Spark shuffle service with Remote Shuffle Service (RSS). The blog narrates how RSS works and its efficiency in solving reliability & scalability issues. Recommended read! https://eng.uber.com/ubers-highly-scalable-and-distributed-shuffle-as-a-service/

Sometimes, it is difficult to determine what the original dependencies of a Python application (pip-based) are, a new pip feature - constraints file is a way to solve this problem. https://luminousmen.com/post/pip-constraints-files

Databricks MLflow 2.0 with MLPipeline that can orchestrate ML pipelines with a YAML file to configure used algorithm, evaluation criteria, and column you write the prediction https://databricks.com/blog/2022/06/29/introducing-mlflow-pipelines-with-mlflow-2-0.html

The hardware component of concurrency is always left untouched when explained in any of the books on the subject. In this post we will overview the levels of parallel hardware. An excerpt from Grokking Concurrency by Kirill Bobrov https://luminousmen.com/post/multiple-levels-of-parallel-hardware

Making CI great again with Github Actions https://luminousmen.com/post/making-ci-workflow-faster-with-github-actions

I believe that there needs to be a paradigm shift in thinking from quality assurance to quality assistance in the entire team involved in product development https://luminousmen.com/post/quality-is-the-responsibility-of-the-team

At the last Re:Invent AWS announced a new EMR Serverless service. It's not really a new service, it's just another deployment option for the old EMR service which lets you not have to think about configuring a cluster. But it's not that simple, more in the post: https://luminousmen.com/post/emr-serverless-a-400level-guide

For the past 6 months or so, I've been busy working with the folks at Manning Publications to create a cool little survival guide that will help engineers pass the concurrency jungle. I am proud to announce my book is now available for early access! Save 50% by using the code 'mlbobrov' Link: http://mng.bz/06eW

A little bit of rumbling about TDD https://luminousmen.com/post/rumbling-about-test-driven-development

A challenge of running the data analytics org is the help desk style request known as can you pull the data real quick! The author explains why one can't simply pull the data quickly, highlighting real-world data problems. https://motifanalytics.medium.com/why-cant-you-pull-data-real-quick-318f90024712

How to properly do pair programming with Bitbucket Spooning

Have you ever committed an SSH private key, a password file or any sensitive data to your repository before? If yes, this post maybe useful to you https://luminousmen.com/post/delete-sensitive-data-from-git

A little about CloudFormation to brighten your day https://luminousmen.com/post/a-very-quick-introduction-to-the-pain-of-aws-cloudformation

"Before you are a leader, success is all about growing yourself. When you become a leader, success is all about growing others." — Jack Welch https://luminousmen.com/post/first-rule-of-the-leader

Streamlit for rapid prototyping is awesome for data apps. In a few lines of code you can create quite a complex UI, with support for loading files and, it seems, all the currently existing libraries for rendering graphs. No, just look how beautiful it is! My app