Cloud Data Platform: Chapter 4: Getting data into the platform (Part 3)

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications Ingesting data from files There are two primary ways to deliver files into a cloud data platform. The first one is to …

Cloud Data Platform: Chapter 4: Getting data into the platform (Part 2)

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications Ingesting data from relational databases Ingesting data from RDBMSs using a SQL interface Different levels of an ingestion process On level 1, …

Cloud Data Platform: Chapter 4: Getting data into the platform (Part 1)

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications In this chapter, we’ll focus on the ingestion layer. Databases, files, APIs, and streams The complexity comes from the fact that a …

Cloud Data Platform: Chapter 3: Part 5: Six-layer data platform architecture – GCP

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications Google Cloud services for a cloud data platform BATCH DATA INGESTION Google Cloud offers several services to perform batch data ingestion - …

Cloud Data Platform: Chapter 3: Part 4: Six-layer data platform architecture – AWS

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications AWS services for cloud data platforms BATCH DATA INGESTION For batch data ingestion, AWS offers two fully managed services. AWS Glue can …

Cloud Data Platform: Chapter 3: Part 3: Six-layer data platform architecture – Open source and commercial alternatives

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications Batch data ingestion Data ingestion is a major component of any data platform, whether it’s a cloud data platform or a traditional …

Cloud Data Platform: Chapter 3: Part 2: Six-layer data platform architecture – Azure

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications Data platform on Azure Azure services for a cloud data platform BATCH DATA INGESTION Azure Data Factory is an ETL overlay service …

Cloud Data Platform: Chapter 3: Part 1: Six-layer data platform architecture – Overview

Cloud data platform layered architecture This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications In chapter 1 we introduced a very high-level architecture of a data platform. It had four layers …

Cloud Data Platform: Chapter 2: Why a data platform and not just a data warehouse

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms – Manning Publications Cloud data platforms and cloud data warehouses: The practical aspects Imagine that we’ve been tasked with building a small reporting solution for …

Cloud Data Platform: Chapter 1: Introducing the data platform

This is my learning note from the book Designing Cloud Data Platforms written by Danil Zburivsky and Lynda Partner. Support the authors by buying the book from Designing Cloud Data Platforms - Manning Publications Data warehouses struggle with data variety, volume, and velocity Variety Variety is indeed the spice of life when it comes to …