How to DWH with Python

Wow, have you known about this awesome hardware benchmarks page for ClickHouse? 🤩 See it: https://benchmark.clickhouse.com/hardware/ Results are contributed by ClickHouse users with various setups: from local laptops and bare metal VMs to cloud filesystems like AWS EFS. In particular, I was interested in AWS EFS/EBS comparison: both are quite bad when compared to bare metal (which is no surprise 🤓) but with a huge advantage of EBS on cold runs 👍🏻 Hot runs EFS/EBS performance is comparable: both are about 6 times worse than bare metal. Thus, both options are good for a quick MVP. EC2+EBS is a simpler setup while EFS can be attached to a disposable serveless ClickHouse container run as an ECS task.

#youtube #briefly #ethereum Ethereum Data Analysis and Ingestion in AWS 🎥 YouTube talk + slides with transcription What's inside: — Building a realtime API for calculating tokens balances. — Public vs own Ethereum nodes comparison. — Support for other EVM and non-EVM blockchains. Pick the most suitable format, same story: — 🗒 Slides with a transcription for the video. — ⚡️ A super-quick summary (2 minutes read). — 📰 The original article, Medium.

#briefly #ethereum How to export a full Ethereum history into S3, efficiently https://blockchain.works-hub.com/learn/how-to-export-a-full-ethereum-history-into-s3-efficiently-f37df A brief summary of my original article about building a Data Platform for Ethereum: — Which node to use: a free public one, a node provider, or run your own? — Start querying right away: public BigQuery datasets with Ethereum data. — How large is the dataset and how to process it cost-efficiently? — Implementing a real-time Ethereum data ingestion. Give it a chance if the original article is too long for you but you are interested in the best practices for Ethereum data engineering 😉

#article #ethereum Exporting the full history of Ethereum into S3 https://medium.com/@tony.bryzgaloff/how-to-dump-full-ethereum-history-to-s3-296fb3ad175 (author: @bryzgaloff) What's inside: — BigQuery public datasets with Ethereum data: how to transfer to S3 quickly. — Alternative approach: exporting data from a public Ethereum node. No need to run your own node! — Processing uint256 with AWS Athena. — Processing realtime updates from Ethereum. — Best Data Engineering practices to process Ethereum data. A short summary inside 👇

Подготовил конспект статьи от Shopify о сетапе Airflow на 10 тысяч DAG'ов со 150 тысячами запусков в день. Сэкономит вам время на прочтении и поможет освежить в памяти в будущем. #briefly #airflow Airflow: scaling out recommendations by Shopify https://telegra.ph/Airflow-scaling-out-recommendations-by-Shopify-06-03 What's inside: — Cloud Storage vs Network File System. — Metadata retention policy. — Manifest file. — Consistent distribution of load. — Concurrency management. — Using different execution environments. Origin: Lessons Learned From Running Apache Airflow at Scale