Cloud Analytics and Big Data Services: AWS vs Azure vs Google Cloud
Warehouses, Spark, streaming, ETL and data lakes: what each analytics service does and how Redshift, Synapse/Fabric, BigQuery and friends compare.
Every company eventually asks the same question: “Can we get all our data in one place and actually answer questions with it?” The cloud answer is a stack of five or six services (storage, ingestion, processing, warehouse, BI), and each provider names and slices them differently. On top of that, the industry has spent years moving from “data warehouse vs data lake” to the lakehouse. This post lays out the analytics building blocks, explains what each one is for, and compares the AWS, Azure and Google Cloud options.
What the analytics category covers
Analytics services turn raw operational data (database rows, events, logs, files) into something people can query and models can learn from. Most platforms follow the same flow:
%%{init: {"themeVariables": {"fontSize": "14px"}, "flowchart": {"useMaxWidth": false, "nodeSpacing": 30, "rankSpacing": 40, "padding": 8, "wrappingWidth": 150}}}%%
flowchart TB
subgraph IN["Ingest and store"]
direction LR
S[Sources:<br/>DBs, apps, events] --> I[Ingest:<br/>ETL / streaming] --> L[Lake:<br/>object storage]
end
subgraph OUT["Process and serve"]
direction LR
P[Process:<br/>Spark / SQL] --> W[Serve:<br/>warehouse] --> B[BI, ML,<br/>apps]
end
IN --> OUT
- Ingest. Batch ETL/ELT tools and streaming services move data in.
- Store. A data lake on object storage holds raw and curated data in open formats (Parquet, Delta, Iceberg).
- Process. Spark, SQL engines and stream processors clean, join and aggregate it.
- Serve. A warehouse or lakehouse SQL endpoint powers dashboards, ad-hoc queries and ML feature pipelines.
The modern twist is the lakehouse: open table formats on object storage give lake data warehouse-like transactions and performance, so the line between “lake” and “warehouse” keeps fading.
Data warehouses
A cloud data warehouse is a columnar, massively parallel SQL engine built for scanning billions of rows. It’s where your BI tool points and where analysts live.
BigQuery set the template for serverless warehousing: no clusters, separate storage and compute, pay per query or per reserved slot. Redshift started as a cluster you sized and has moved steadily towards serverless. On Azure, the picture changed. Synapse Analytics still works and has no retirement date, but Microsoft is putting its new investment into Microsoft Fabric, a SaaS analytics platform built on OneLake. For a greenfield Azure analytics platform, evaluate Fabric first.
Amazon Redshift Soon
Columnar MPP data warehouse, available as provisioned RA3 clusters or Redshift Serverless (billed per RPU-hour). Queries data in S3 through Redshift Spectrum, supports zero-ETL integrations from Aurora and DynamoDB, and data sharing across accounts.
Dedicated SQL pools (MPP warehouse), serverless SQL over the data lake, Spark pools and pipelines in one workspace. Still supported, but Microsoft’s new investment goes to Microsoft Fabric, which is the default choice for new analytics platforms on Azure.
BigQuery Soon
Serverless warehouse with no clusters to size; pay per TB scanned or reserve slots. Separation of storage and compute, BigQuery ML for SQL-based models, BI Engine for fast dashboards, and BigLake tables over open formats such as Apache Iceberg.
On serverless, per-TB-scanned pricing, a careless
SELECT *on a huge table is a real bill. Partition and cluster tables, select only the columns you need, and set per-user or per-project query quotas.
Spark and Hadoop processing
Apache Spark is the workhorse for large-scale transformation, data engineering and ML feature preparation. Each cloud runs it as a managed service, either as clusters you size or serverless.
On Azure, Databricks is the dominant choice. It’s a first-party Azure service, and its lakehouse platform (Delta Lake, Unity Catalog) is also available on AWS and Google Cloud, which makes it a popular portability layer. EMR and Dataproc are excellent at running open-source Spark cheaply, especially on Spot capacity.
Amazon EMR Soon
Managed Spark, Hive, Presto/Trino, Flink and HBase on EC2, EKS or EMR Serverless. Spot-friendly and cost-effective for large batch jobs; the EMR runtime for Spark is tuned for performance on AWS.
First-party Azure service from Databricks, providing the lakehouse platform with Delta Lake, Unity Catalog, collaborative notebooks, Databricks SQL and MLflow. The most common Spark choice on Azure.
Managed open-source clusters (Spark, Hadoop, Kafka, HBase). Still available, but most new Spark work on Azure goes to Databricks or Fabric.
Dataproc Soon
Managed Spark and Hadoop clusters that start in about 90 seconds, plus Serverless for Apache Spark (formerly Dataproc Serverless) for batch and interactive Spark without clusters.
Real-time streaming analytics
Batch pipelines answer “what happened yesterday”. Streaming answers “what’s happening now”: fraud detection, live dashboards, IoT telemetry, clickstream personalisation. A streaming stack needs an ingestion log (Kinesis Data Streams, Event Hubs, Pub/Sub or Kafka) and a stream processor that does windowed aggregations, joins and alerts.
My advice: only stream what genuinely needs low latency. Streaming pipelines cost more to build, run and debug than batch. Many “real-time” requirements turn out to be “every 15 minutes”, which micro-batch handles fine.
Amazon Kinesis Soon
Kinesis Data Streams for ingesting and replaying event streams, Amazon Data Firehose for loading streams into S3, Redshift or OpenSearch, and Amazon Managed Service for Apache Flink for stateful stream processing. Amazon MSK offers managed Kafka.
Serverless stream processing with a SQL-like query language over Event Hubs, IoT Hub or Blob inputs, including windowing and reference-data joins. Outputs to SQL, Cosmos DB, Power BI, Data Lake and more.
Fast analytics on high-volume telemetry, logs and time series with KQL. The same engine powers Log Analytics and Fabric Real-Time Intelligence.
Dataflow Soon
Fully managed Apache Beam runner for unified streaming and batch pipelines, with autoscaling, exactly-once processing and templates for common Pub/Sub-to-BigQuery patterns.
The event ingestion services themselves are compared in the application integration post.
ETL and data integration
Integration services pull data from SaaS apps, databases and files, transform it, and land it in the lake or warehouse. They’re also the orchestration layer that schedules everything else.
The trend is towards ELT: load raw data first, transform with SQL inside the warehouse (often with dbt), and keep visual ETL tools for connectivity and orchestration. Zero-ETL features (Aurora to Redshift, Cosmos DB and SQL mirroring into Fabric, Datastream to BigQuery) remove pipelines entirely for some sources.
AWS Glue Soon
Serverless Spark-based ETL jobs, crawlers that infer schemas, and the Glue Data Catalog used by Athena, EMR and Redshift Spectrum. Glue DataBrew offers no-code data preparation.
Orchestrates data movement and transformation with 90+ connectors, visual data flows and a self-hosted integration runtime for on-premises sources. Data Factory in Microsoft Fabric is its next generation.
Cloud Data Fusion Soon
Visual, code-free pipeline builder based on open-source CDAP, with a large connector library. For orchestration, Google Cloud also offers Cloud Composer (managed Apache Airflow).
Data lake storage and interactive query
The lake is object storage organised into zones (raw, cleaned, curated), stored in open columnar formats, and governed by a catalog. Interactive query engines let analysts run SQL directly against those files without loading them into a warehouse first. That’s great for exploration, infrequent queries and keeping one copy of the data.
Amazon Athena Soon
Serverless SQL (Trino-based) over data in S3 in Parquet, ORC, JSON, CSV or Iceberg, billed per TB scanned. Pairs with the Glue Data Catalog and Lake Formation for fine-grained access control.
Blob Storage with a hierarchical namespace, POSIX-like ACLs and Hadoop-compatible access, used as the lake layer for Synapse, Databricks and HDInsight. Fabric’s OneLake builds on the same technology.
BigLake tables let BigQuery query Iceberg, Parquet and other files in Cloud Storage with the same fine-grained security as native tables, and Dataplex governs the lake.
Comparing the providers
- Google Cloud is often the favourite for analytics because of BigQuery’s simplicity and performance. The serverless-first design means little to tune.
- AWS offers the most components (Redshift, EMR, Glue, Athena, Kinesis, MSK, Lake Formation). That’s a lot of flexibility, but more assembly.
- Azure is consolidating onto Microsoft Fabric, with Databricks as the main Spark platform. Tight Power BI integration is a real advantage for Microsoft-centric organisations.
Common pitfalls
- Building a swamp. A lake with no catalog, ownership or quality checks turns into a dumping ground. Invest in governance (Lake Formation, Unity Catalog, Purview, Dataplex) from the start.
- Too many small files. Streaming and frequent loads create millions of tiny files that cripple query performance. Schedule compaction, or use table formats that handle it.
- Copying data everywhere. Each copy (lake, warehouse, marts, extracts) adds cost and drift. Open table formats and query-in-place let several engines share one copy.
- Forgetting cost controls. Serverless engines scale beautifully, including your bill. Set budgets, quotas and alerts per team.
- Ignoring Fabric and lakehouse roadmaps. On Azure especially, new projects on older services may face a migration later. Check the product’s direction before committing.
Conclusion
Analytics on any cloud comes down to a lake on object storage, a processing engine, a SQL serving layer and pipelines to feed them. BigQuery is the simplest end-to-end option, AWS gives you every component to assemble, and Azure’s future is Fabric plus Databricks. Start small: one source, one curated dataset, one dashboard people actually use. Then expand. Every service side by side is in the Analytics comparison.