Data Ingestion & Analytics

Amazon Athena — serverless SQL on S3. Uses Presto. $5/TB scanned. Performance tips:

• Use columnar data (Parquet or ORC) — use Glue to convert
• Compress data (bzip2, gzip, snappy, etc.)
• Partition datasets in S3 by date/region/etc.
• Use larger files (>128 MB) to minimize overhead

Federated Query: Athena can query across RDS, ElastiCache, DynamoDB, etc. using Data Source Connectors (Lambda).

AWS Glue — serverless ETL service. Extract, Transform, Load.

• Glue Data Catalog: Metadata repository for all datasets. Works with Athena, Redshift Spectrum, EMR.
• Glue Crawler: Auto-discovers schema from S3, RDS, DynamoDB, etc.
• Glue Job Bookmarks: Prevents reprocessing old data.
• Glue Streaming ETL: Built on Spark Structured Streaming. Compatible with Kinesis, Kafka, MSK.
• Convert data to Parquet: S3 → EventBridge → Lambda → Glue ETL Job → output Parquet to S3 → Athena

AWS Lake Formation — set up a data lake in days. Built on Glue. Fine-grained access control (row + column level). De-duplication using ML Transforms.

Amazon OpenSearch Service (formerly Elasticsearch):

• Search any field, even partial matches. Use as complement to another DB (e.g., DynamoDB → Lambda → OpenSearch for search; DynamoDB for CRUD).
• Two modes: managed cluster or serverless.
• Ingestion: Kinesis Data Firehose, IoT, CloudWatch Logs.

Amazon EMR (Elastic MapReduce): Managed Hadoop/Spark cluster. For big data processing, ML, web indexing. Node types: Master (long-running), Core (tasks + storage, long-running), Task (tasks only, Spot-friendly). Use Spot for task nodes to save cost.

Amazon QuickSight — serverless BI/visualization tool. SPICE in-memory computation engine. Enterprise edition supports Column-Level Security. Users/Groups only exist within QuickSight (not IAM).

Amazon MSK vs Kinesis:

References

• Amazon Kinesis Data Streams
• Kinesis Data Firehose
• AWS Glue
• Amazon Athena