How to DWH with Python
前往频道在 Telegram
Data Engineering practices, cases and implementation hints 👨💻 by @bryzgaloff
显示更多461
订阅者
无数据24 小时
无数据7 天
无数据30 天
吸引订阅者
四月 '23
四月 '23
+3
在0个频道中
三月 '23
+4
在0个频道中
Get PRO
二月 '23
+3
在0个频道中
Get PRO
一月 '23
+3
在0个频道中
Get PRO
十二月 '22
+6
在0个频道中
Get PRO
十一月 '22
+11
在0个频道中
Get PRO
十月 '22
+23
在0个频道中
Get PRO
九月 '22
+449
在0个频道中
| 日期 | 订阅者增长 | 提及 | 频道 | |
| 29 四月 | 0 | |||
| 28 四月 | 0 | |||
| 27 四月 | 0 | |||
| 26 四月 | 0 | |||
| 25 四月 | +1 | |||
| 24 四月 | 0 | |||
| 23 四月 | 0 | |||
| 22 四月 | +1 | |||
| 21 四月 | 0 | |||
| 20 四月 | 0 | |||
| 19 四月 | 0 | |||
| 18 四月 | 0 | |||
| 17 四月 | 0 | |||
| 16 四月 | 0 | |||
| 15 四月 | 0 | |||
| 14 四月 | 0 | |||
| 13 四月 | 0 | |||
| 12 四月 | 0 | |||
| 11 四月 | 0 | |||
| 10 四月 | 0 | |||
| 09 四月 | 0 | |||
| 08 四月 | 0 | |||
| 07 四月 | 0 | |||
| 06 四月 | 0 | |||
| 05 四月 | +1 | |||
| 04 四月 | 0 | |||
| 03 四月 | 0 | |||
| 02 四月 | 0 | |||
| 01 四月 | 0 |
频道帖子
Wow, have you known about this awesome hardware benchmarks page for ClickHouse? 🤩 See it: https://benchmark.clickhouse.com/hardware/
Results are contributed by ClickHouse users with various setups: from local laptops and bare metal VMs to cloud filesystems like AWS EFS.
In particular, I was interested in AWS EFS/EBS comparison: both are quite bad when compared to bare metal (which is no surprise 🤓) but with a huge advantage of EBS on cold runs 👍🏻
Hot runs EFS/EBS performance is comparable: both are about 6 times worse than bare metal.
Thus, both options are good for a quick MVP. EC2+EBS is a simpler setup while EFS can be attached to a disposable serveless ClickHouse container run as an ECS task.
| 2 | #youtube #briefly #ethereum Ethereum Data Analysis and Ingestion in AWS
🎥 YouTube talk + slides with transcription
What's inside:
— Building a realtime API for calculating tokens balances.
— Public vs own Ethereum nodes comparison.
— Support for other EVM and non-EVM blockchains.
Pick the most suitable format, same story:
— 🗒 Slides with a transcription for the video.
— ⚡️ A super-quick summary (2 minutes read).
— 📰 The original article, Medium. | 1 065 |
| 3 | #briefly #ethereum How to export a full Ethereum history into S3, efficiently
https://blockchain.works-hub.com/learn/how-to-export-a-full-ethereum-history-into-s3-efficiently-f37df
A brief summary of my original article about building a Data Platform for Ethereum:
— Which node to use: a free public one, a node provider, or run your own?
— Start querying right away: public BigQuery datasets with Ethereum data.
— How large is the dataset and how to process it cost-efficiently?
— Implementing a real-time Ethereum data ingestion.
Give it a chance if the original article is too long for you but you are interested in the best practices for Ethereum data engineering 😉 | 692 |
| 4 | #article #ethereum Exporting the full history of Ethereum into S3
https://medium.com/@tony.bryzgaloff/how-to-dump-full-ethereum-history-to-s3-296fb3ad175 (author: @bryzgaloff)
What's inside:
— BigQuery public datasets with Ethereum data: how to transfer to S3 quickly.
— Alternative approach: exporting data from a public Ethereum node. No need to run your own node!
— Processing uint256 with AWS Athena.
— Processing realtime updates from Ethereum.
— Best Data Engineering practices to process Ethereum data.
A short summary inside 👇 | 1 970 |
| 5 | Подготовил конспект статьи от Shopify о сетапе Airflow на 10 тысяч DAG'ов со 150 тысячами запусков в день. Сэкономит вам время на прочтении и поможет освежить в памяти в будущем.
#briefly #airflow Airflow: scaling out recommendations by Shopify
https://telegra.ph/Airflow-scaling-out-recommendations-by-Shopify-06-03
What's inside:
— Cloud Storage vs Network File System.
— Metadata retention policy.
— Manifest file.
— Consistent distribution of load.
— Concurrency management.
— Using different execution environments.
Origin: Lessons Learned From Running Apache Airflow at Scale | 0 |
