Data1984

LinkedIn remains one of the coolest place in term of data engineering where many popular open-source technologies emerge. https://engineering.linkedin.com/blog/2023/declarative-data-pipelines-with-hoptimator

This reminds me of a solution designed and implemented a couple of years ago. But back then we used DynamoDB streams to capture item-level changes with exactly-one semantics, Lambda to modify data and Kinesis Firehose to deliver data to Redshift. Looks like now things are simpler.

While AWS tries to support all three formats, and helps to choose right one for your use-case, Databricks introduces unified format, so you will not need to pick 😎

AWS Glue Data Quality is Generally Available | AWS Big Data Blog https://aws.amazon.com/blogs/big-data/aws-glue-data-quality-is-generally-available/

Choosing an open table format for your transactional data lake on AWS | AWS Big Data Blog https://aws.amazon.com/blogs/big-data/choosing-an-open-table-format-for-your-transactional-data-lake-on-aws/

Empower every BI professional to do more with Microsoft Fabric https://build.microsoft.com/en-US/sessions/8b23c96e-7c35-463d-88b4-564d23dc14a5

This is huge! PowerBI now has Git integration and Power BI Desktop ‘Developer Mode' which means you can edit files in code editor like VSCode and deploy them using PowerBI deployment pipelines. In other words "dashboards as a code".

The State of Data Engineering 2023 https://lakefs.io/blog/the-state-of-data-engineering-2023/

Microsoft introduced Fabric, which is a combination of Power BI, Azure Synapse, Data Factory and Data Explorer on top of ADLS gen2 using Delta (Parquet) as data lake format. A new component is Data Activator which seems to be a no-code rule engine. https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/

dbt Guide | GitLab https://about.gitlab.com/handbook/business-technology/data-team/platform/dbt-guide/