Cloud Application Integration Services: AWS vs Azure vs Google Cloud
Queues, pub/sub, event routing, streaming, workflows and API gateways: what each integration service does and how SQS, Service Bus, Pub/Sub and friends compare.
Two services that call each other directly over HTTP work fine until one of them is slow, down or being deployed. Then the other one fails too, retries pile up, and a small blip becomes an outage. Application integration services exist to break that coupling. They let components talk through queues, topics, events and workflows, so each part can fail, scale and deploy on its own. The problem is that every cloud has five or six of these services with overlapping features. This post explains what each pattern is for and maps it to the AWS, Azure and Google Cloud services.
What application integration covers
Integration services move messages and coordinate work between components. The patterns:
| Pattern | Shape | Example |
|---|---|---|
| Queue | One producer, one consumer group, each message processed once | Order processing workers |
| Pub/sub | One message, many independent subscribers | “Order placed” goes to billing, email and analytics |
| Event routing | Events filtered and routed by content to targets | Blob uploaded triggers a thumbnail function |
| Event streaming | Ordered, replayable, high-volume log | Clickstream, telemetry, change data capture |
| Workflow | Multi-step process with state, retries and branching | Loan approval, data pipeline, saga |
| API gateway | Managed front door for synchronous APIs | Auth, throttling and routing for public APIs |
A single application often uses several of them together:
%%{init: {"themeVariables": {"fontSize": "14px"}, "flowchart": {"useMaxWidth": false, "nodeSpacing": 30, "rankSpacing": 40, "padding": 8, "wrappingWidth": 150}}}%%
flowchart LR
C[Client] --> G[API gateway] --> O[Order service]
O --> T[Topic:<br/>order placed]
T --> Q1[Billing queue]
T --> Q2[Email queue]
T --> W[Workflow:<br/>fulfilment]
Message queues
A queue decouples a producer from a consumer. The producer drops a message and moves on. Consumers pull messages at their own pace and delete them once processed. If consumers fall behind, the queue absorbs the backlog. If one crashes mid-message, the message becomes visible again and another worker picks it up.
Design consumers to be idempotent. Most queues deliver at least once, so the same message will occasionally arrive twice. Use dead-letter queues for messages that keep failing, and alert on them. A dead-letter queue nobody watches is just a slower way to lose data.
Amazon SQS Soon
Fully managed queues with no capacity to provision. Standard queues give near-unlimited throughput with at-least-once delivery; FIFO queues give ordering and exactly-once processing. Dead-letter queues, delay queues and long polling built in.
Enterprise message broker with queues and topics, sessions for ordered processing, transactions, duplicate detection, scheduled messages and dead-lettering. Supports AMQP 1.0 and JMS 2.0 on Premium.
Simple, very cheap queues inside a storage account for basic work distribution. Up to 64 KB messages and no ordering guarantees; use Service Bus when you need broker features.
Pub/Sub Soon
Pub/Sub covers queue-style work distribution with pull subscriptions, ordering keys, exactly-once delivery and dead-letter topics. Cloud Tasks handles explicit task dispatch with rate limiting and scheduling.
Pub/sub and notifications
Pub/sub delivers one message to many subscribers, each getting its own copy. It’s how you let new consumers react to an event without changing the producer: add a subscription and you’re done. A very common pattern on AWS is SNS fanning out to several SQS queues, giving every consumer its own buffered, retryable copy.
Amazon SNS Soon
Topics that fan messages out to SQS queues, Lambda functions, HTTP endpoints, email, SMS and mobile push. FIFO topics preserve order into FIFO queues; message filtering routes subsets to subscribers.
Topics with subscriptions and SQL-like filter rules for durable enterprise fan-out, with all of Service Bus’s reliability features.
Push notifications at scale to iOS, Android, Windows and web, with tagging and templates to target user segments.
Pub/Sub Soon
A global topic with many subscriptions (push or pull), message filtering, replay with seek, and direct BigQuery and Cloud Storage subscriptions that write messages without code. Firebase Cloud Messaging handles mobile push.
Event routing
Event routers sit between event sources and handlers and route by content. They’re ideal for reacting to things that happen in the cloud platform itself (a file uploaded, a VM deallocated, a resource tagged) and for loosely coupled, event-driven architectures where producers shouldn’t know who consumes their events.
The difference from pub/sub is subtle. Routers are built around filtering rules and many target types, often with schemas, archives and replay, while pub/sub is a simpler topic-subscription model. EventBridge, Event Grid and Eventarc all use or support the CloudEvents standard, which helps portability.
Amazon EventBridge Soon
Serverless event bus receiving events from AWS services, SaaS partners and your apps, routed by content-based rules to 20+ targets. Pipes connects sources to targets with filtering and enrichment; Scheduler runs cron-style jobs at scale.
Push-based event routing for Azure resource events (blob created, resource changed) and custom topics, using CloudEvents. Namespaces add MQTT broker support and pull delivery.
Eventarc Soon
Routes events from Google Cloud services (via Cloud Audit Logs or direct sources), Pub/Sub and third parties to Cloud Run, GKE, Workflows and Cloud Run functions, using CloudEvents.
Event streaming ingestion
Streams look like queues but behave differently. Messages aren’t deleted when read; they stay in an ordered, partitioned log for a retention period, and many consumers read at their own offsets. That makes streams the right tool for high-volume telemetry, clickstreams, change data capture, and anything you might want to replay.
If your team already knows Kafka, all three providers offer either managed Kafka or a Kafka-compatible endpoint.
Kinesis Data Streams Soon
Ordered, replayable streams of records sharded for throughput, retained for 24 hours up to a year. On-demand mode scales automatically. Amazon MSK provides managed Apache Kafka for Kafka-native workloads.
Big-data streaming platform that ingests millions of events per second, with partitions, consumer groups, capture to Data Lake, and a Kafka-compatible endpoint so Kafka clients work unchanged.
Pub/Sub Soon
Pub/Sub handles high-volume event ingestion with automatic scaling and no partitions to manage. Google Cloud Managed Service for Apache Kafka is available for Kafka-native workloads.
Pick by semantics, not by name. If you need “process each job once, then forget it”, that’s a queue. If you need “keep an ordered history many systems can replay”, that’s a stream. Using a stream as a work queue (or the reverse) causes most of the pain I see in event-driven designs.
Workflow orchestration
When a business process has several steps with branching, retries, timeouts, waits for humans and compensation when something fails, encoding it as a chain of queues and functions gets messy fast. Workflow services hold the state for you and make the process visible, which is invaluable when someone asks “where is order 4711 stuck?”
Step Functions and Workflows are developer-focused state machines. Logic Apps leans towards low-code integration with a huge connector library, making it as much an iPaaS as an orchestrator.
AWS Step Functions Soon
Visual state machines that coordinate Lambda, ECS, Glue and 200+ AWS services with retries, error handling, parallel branches and human approval steps. Standard workflows run up to a year; Express workflows suit high-volume, short runs.
Low-code workflows with 1,400+ connectors for SaaS, on-premises and Azure services. Consumption (multi-tenant, pay per action) or Standard (single-tenant, VNet-integrated, runs locally). Durable Functions is the code-first alternative.
Workflows Soon
Serverless orchestration in YAML or JSON that calls Google Cloud APIs and any HTTP service, with retries, callbacks and parallel steps. Billed per step; Cloud Composer (Airflow) covers data-pipeline orchestration.
API gateways and management
API gateways front synchronous APIs: authentication, rate limiting, request transformation, routing to backends, caching and analytics. Full API management adds a developer portal, subscription keys, versioning and lifecycle governance, which you need once external partners consume your APIs.
A newer use case is the AI gateway: putting a gateway in front of LLM endpoints to enforce token quotas, load balance across model deployments, cache responses and log usage per team.
Amazon API Gateway Soon
Create REST, HTTP and WebSocket APIs in front of Lambda, containers or any HTTP backend, with authorisers (IAM, Cognito, Lambda), throttling, usage plans and API keys. HTTP APIs are cheaper and faster for simple proxies.
Full API management with a gateway, developer portal, policies (transform, rate limit, validate JWT), versioning and a self-hosted gateway for hybrid. Consumption and v2 tiers cover small to enterprise use, including AI gateway policies for LLM APIs.
API Gateway Soon
Lightweight managed gateway for Cloud Run, Cloud Run functions and App Engine using OpenAPI specs, with API keys and JWT auth. Apigee is Google’s full enterprise API management platform.
Comparing the providers
- Azure has the richest enterprise messaging (Service Bus) and the most connectors (Logic Apps), which suits enterprise integration and hybrid scenarios.
- AWS offers clean, single-purpose building blocks (SQS, SNS, EventBridge, Step Functions) that compose well and scale with no tuning.
- Google Cloud consolidates much of this into Pub/Sub, which covers queues, fan-out and streaming in one global service, and Apigee is a leading API management platform.
Common pitfalls
- Assuming exactly-once. Unless the service explicitly guarantees it and you configure it, expect duplicates. Make handlers idempotent.
- Unbounded retries. A poison message retried forever blocks a queue or burns money. Cap retries and use dead-letter queues.
- Large payloads in messages. Most services cap message size (often 256 KB to 1 MB). Store the payload in object storage and send a reference.
- Ordering assumptions. Global ordering is rare and expensive. Order per key (customer, device) using FIFO queues, sessions or ordering keys only where it matters.
- No tracing. Asynchronous flows are hard to debug. Propagate correlation IDs and use distributed tracing (see the monitoring post).
Conclusion
Integration services make systems resilient by letting components fail and scale independently. Use queues for work distribution, pub/sub for fan-out, event routers for reacting to platform and domain events, streams for high-volume replayable data, workflows for multi-step processes, and an API gateway for synchronous entry points. A good next step: find one synchronous call chain in your system where a downstream failure takes everything down, and put a queue in the middle. See all the services side by side in the Application Integration comparison.