Loosely Coupled Architectures
Amazon SQS — Key Facts:
- Unlimited throughput, unlimited messages
- Default retention: 4 days, max 14 days
- Max message size: 1,024 KB (1 MB)
- SQS: Standard: At-least-once delivery, best-effort ordering (Standard queue)
- SQS Visibility Timeout: Crucial (default 30s): after polling, message invisible to other consumers. If not deleted within timeout, message reappears. Use ChangeMessageVisibility API for more time.
- SQS Long Polling: Preferred (preferred over short polling): consumer waits for messages if queue empty. 1-20 second wait. Reduces API calls and cost.
- SQS FIFO: SQS FIFO: Exactly-once processing, strict ordering. 300 msg/s without batching, 3000 msg/s with. Uses Deduplication ID (to prevent duplicates) and Message Group ID (for ordering within a group).
- Dead Letter Queue (DLQ): After N failed processing attempts, message goes to DLQ.
SQS + ASG pattern: CloudWatch alarm on ApproximateNumberOfMessages metric → trigger ASG scaling.
Amazon SNS — Key Facts:
- Pub/Sub: one message fans out to all subscribers
- Up to 12.5 million subscriptions per topic, 100,000 topics
- Subscribers: SQS, Lambda, Kinesis Data Firehose, email, SMS, HTTP endpoints
- FIFO Topic: Ordering by Message Group ID, deduplication. Limited throughput (same as SQS FIFO).
- Message Filtering: JSON filter policy on subscriptions to only receive relevant messages.
SNS + SQS Fan-Out Pattern: SNS topic sends to multiple SQS queues. Fully decoupled, no data loss. Great for delivering one S3 event to multiple queues (S3 only supports one event rule per event type/prefix combination).
Amazon Kinesis Data Streams:
- Real-time streaming. Data retained up to 365 days (can replay).
- Data up to 10 MB per put (typically small real-time records).
- Kinesis Data Streams: Ordering is guaranteed within a shard (same Partition ID = same shard).
- Provisioned mode: choose shards manually. 1 MB/s in, 2 MB/s out per shard. Pay per shard/hour.
- On-Demand mode: auto-scales. Pay per stream/hour + data per GB.
Amazon Data Firehose (formerly Kinesis Data Firehose):
- Fully managed, Kinesis Data Firehose: Near real-time (60s buffer min). NOT truly real-time.
- Destinations: S3, Redshift, OpenSearch, Splunk, custom HTTP. No data storage. No replay.
- Can transform data with Lambda before delivery.
Kinesis Data Streams vs Firehose vs SQS:
- Real-time, replay, custom consumers → Kinesis Data Streams
- Load streaming data into storage → Firehose
- Decouple microservices, task queues → SQS
- Fan-out notifications → SNS
Amazon MSK (Managed Streaming for Apache Kafka): Alternative to Kinesis. Supports larger message sizes (1 MB default, up to 10 MB). Data stored on EBS. MSK Serverless available.
Amazon MQ: Managed ActiveMQ/RabbitMQ. Use when migrating apps that use AMQP, STOMP, MQTT (open protocols). New apps should use SQS/SNS. Multi-AZ with failover using EFS.
AWS Step Functions: Orchestrate Lambda functions and other services as state machines. Features: sequence, parallel, conditions, timeouts, error handling, human approval. Max execution time: 1 year.