Data Engineering / Инженерия данных / Data Engineer / DWH
الذهاب إلى القناة على Telegram
Data Engineering: ETL / DWH / Data Pipelines based on Open-Source software. Инженерия данных. ✔ DWH / SQL ✔ Airflow / Python / ETL / dbt / Spark ✔ AI Agents Рекламу не размещаю Вопросы: @iv_shamaev | datatalks.ru
إظهار المزيد2 694
المشتركون
+124 ساعات
+57 أيام
+2130 أيام
أرشيف المشاركات
Modern Data Stack - Ranking | OSS Insight
https://ossinsight.io/collections/modern-data-stack/
Play with Docker
▫️Docker 101 Tutorial - Self-paced tutorials to increase your Docker knowledge.
▫️Lab Environment - Complete a workshop without installing anything using this Docker playground.
▫️Community Training - Free and paid learning materials from Docker Captains.
https://www.docker.com/play-with-docker/
GitHub - TolstikovIgor/ETL: GeekBrains: Построение хранилища данных и основы ETL
https://github.com/TolstikovIgor/ETL
TelegramOperator — apache-airflow-providers-telegram Documentation
Оператор Airflow для отправки уведомлений в Telegram
https://airflow.apache.org/docs/apache-airflow-providers-telegram/stable/operators.html
Docker Swarm для самых маленьких / Хабр
https://habr.com/ru/post/659813/
Создание современной платформы для работы с данными с помощью Open-Source-решений / Хабр
https://habr.com/ru/company/vk/blog/671642/
Complete Data Engineer’s Vocabulary | by Kovid Rathee | Towards Data Science
https://towardsdatascience.com/complete-data-engineers-vocabulary-87967e374fad
Data Engineering on GCP Cheatsheet
https://github.com/ml874/Data-Engineering-on-GCP-Cheatsheet
Repost from Data-comics
Читала отчёт по DevOps Setups benchmarking 2022 от Luca G и humanitec.
В целом, есть интересные моменты про разные типы команд разработчиков, ребята провели большую работу.
Но результаты преподнесли немного дезинформирующе.
Пример - на приложенной картинке.
Что не так? 😁
Ссылка на отчёт тут: https://humanitec.com/whitepapers/2021-devops-setups-benchmarking-report
Файл, кому интересно, приложу в комменты.
What is the Parquet File Format and Why You Should Use It
https://www.upsolver.com/blog/apache-parquet-why-use
Practical Real-time Data Processing and Analytics: Distributed Computing and Event Processing using Apache Spark, Flink, Storm, and Kafka
What You Will Learn
▫️Get an introduction to the established real-time stack
▫️Understand the key integration of all the components
▫️Get a thorough understanding of the basic building blocks for real-time solution designing
▫️Garnish the search and visualization aspects for your real-time solution
▫️Get conceptually and practically acquainted with real-time analytics
▫️Be well equipped to apply the knowledge and create your own solutions
Apache Hive Essentials: Essential techniques to help you process, and get unique insights from, big data
What you will learn
▫️Create and set up the Hive environment
▫️Discover how to use Hive's definition language to describe data
▫️Discover interesting data by joining and filtering datasets in Hive
▫️Transform data by using Hive sorting, ordering, and functions
▫️Aggregate and sample data in different ways
▫️Boost Hive query performance and enhance data security in Hive
▫️Customize Hive to your needs by using user-defined functions and integrate it with other tools
