L̶u̵m̶i̵n̷o̴u̶s̶m̶e̵n̵B̶l̵o̵g̵

Let's start the week on a positive note, shall we? The smallest word on the slide is "nuanced" - that's probably the right answer.

I miss writing code https://luminousmen.com/post/i-miss-writing-code/

Friday vibes: https://youtu.be/FG8sUgjBGXs

The vibe in big tech around the push toward AI is basically: as long as we haven't been laid off and there's still work to do, we're enjoying the ride - we'll figure out the rest later

What's the last benchmark you actually re-ran yourself? https://luminousmen.com/post/benchmaxxing/

Turns out almost nobody actually picks their isolation level. https://luminousmen.com/post/write-skew-the-anomaly-the-sql-standard-forgot/

Updated post today: what actually moves partitioning, how to fix skew starting with the cheapest option, and why AQE sometimes throws your repartition away and never tells you: https://luminousmen.com/post/spark-tips-partition-tuning/

Trust is one specific thing: the moment people stop checking your work. Your teammate merges a 2,000-line refactor and nobody reads all 2,000 lines. What got reviewed was the person. You earn it slowly and lose it fast. Today we gonna talk about every place that shows up - team, manager, tools, dependencies, data: https://luminousmen.com/post/fundamentals-of-trust-in-engineering/

Most of your backfills didn't have to happen. A team I worked with once caught a bug that had been quietly corrupting data for a month. Two-line fix. They re-ran the pipeline over 30 days, went to lunch, and came back to a number that had doubled. Because of append mode. 🔗 Link: https://luminousmen.com/post/most-of-your-backfills-didnt-have-to-happen/

I always thought I could feel the difference between something a human made and something a machine made. Then I found out a million people cried at an AI band, and stopped the second they saw the label, same songs. New post: why my empathy turned out to be a sticker detector, and what actually survives when the sticker gets faked. 🔗Link: https://luminousmen.substack.com/p/my-empathy-is-a-sticker-detector

I'm about to lose some followers. Design patterns suck. 🔗Link: https://luminousmen.com/post/design-patterns-suck/

Yikes Google: https://www.thatprivacyguy.com/blog/chrome-silent-nano-install/

I've built an AI whose only job is to tell me that I'm full of shit. Not write code or summarize my meetings. Just find the hole in my reasoning before a coworker finds it for me in a Slack thread two days later. I built a Claude Code skill that plays devil's advocate on my own work. I paste in my analysis, and before I send anything it runs a structured challenge: What am I assuming? What evidence did I skip? What's the strongest case for the opposite conclusion? How confident should I actually be? We've got endless AI to write faster, generate more, ship sooner. Almost none of it pushes back. But if your work is incidents, root cause, architecture calls - speed was never the bottleneck. The quality of your thinking was. 🔗Link: https://luminousmen.substack.com/p/stop-building-ai-that-agrees-with

Everyone asks how the vibe coding's going. Nobody asks how the vibe's doing. Source: https://www.reddit.com/r/ClaudeAI/comments/1tdyrdv/claudes_first_day_at_dunder_mifflin/

AI isn't replacing data engineers. It's replacing the work that never should have been called data engineering in the first place. Link: https://luminousmen.com/post/if-your-data-engineer-can-be-replaced-by-an-agent-you-already-hired-badly/

A data engineer on Reddit got drunk and wrote down everything he learned in 10 years. The post lives on. I agree with almost all of it. Preserved it here: https://luminousmen.com/post/drunk-post-things-ive-learned-as-a-sr-engineer

"How many unique users did X AND Y?" - simple question, brutal to answer at scale. Theta sketches solve this. But the gap between "here's the algorithm" and "here's how to not get burned in production" is enormous. I wrote the guide I wish existed when I started working with them: https://luminousmen.substack.com/p/mastering-theta-sketches-for-cardinality

New post: Data Sketches - how to count unique users across billions of events without melting your cluster. Why COUNT DISTINCT doesn't scale, how sketches actually work under the hood, and which one to pick for your problem. https://luminousmen.substack.com/p/the-power-of-data-sketches-a-comprehensive

Built a tycoon game about data engineering. You start at the ground level. Goal: reach AGI before you go bankrupt. Free, browser, no signup: https://game.luminousmen.com/