# Changelog

All notable changes to the AWS IoT Wireless Device Bulk Management Solution will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.0.0] - 2025-10-22

### Added
- **Complete CDK infrastructure** for serverless AWS IoT Wireless device management
- **Dual operation support** - Bulk device creation and updates via JSON files
- **S3-triggered processing pipeline** with Step Functions orchestration
- **Operation-specific concurrency control** - Separate flows for Create and Update operations with independent TPS limits
- **Large dataset support** - Handles 100,000+ devices using S3 payload pattern to avoid Step Functions size limits
- **Aurora Serverless v2 database** (PostgreSQL 16.1) with RDS Data API integration for device tracking
- **Comprehensive validation** for device data and operations with detailed error reporting
- **SMSN validation** - Requires exactly 64 characters for consistent device identification
- **Validation error reports** - Complete validation failures trigger report generation and notifications with detailed error information
- **Real-time CloudWatch monitoring** with custom metrics and comprehensive dashboard
- **Configurable notification system** supporting SQS, SNS, or no notifications
- **Intelligent batch processing** that maximizes concurrency utilization while respecting Lambda timeout constraints
- **Comprehensive retry mechanisms** with exponential backoff for AWS IoT Wireless API throttling
- **Automatic cleanup** of temporary batch files with dual mechanisms (immediate Lambda + S3 lifecycle)
- **TPS validation and verification** system with detailed logging and metrics
- **Rate limiting and retry logic** for all AWS IoT Wireless API calls
- **Detailed reporting** with success/failure analysis and S3 storage
- **Comprehensive build system** with Lambda validation and testing during brazil-build
- **S3-based result storage** - API processor stores detailed results in S3 to avoid Step Functions payload limits
- **Concurrent S3 operations** - Parallel uploads/downloads for batch files and results (up to 20 concurrent)
- **Main batch name extraction** - Consistent batch naming across database tables
- **Comprehensive IAM documentation** - Detailed permissions for deployment and runtime operations
- **Sample database queries** - 50+ SQL examples for Aurora PostgreSQL database
- **Enhanced debugging** - Comprehensive logging for troubleshooting large-scale operations

### Features
- **Operation-Specific TPS Configuration**: Create operations use `createDeviceApiTps`, Update operations use `min(updateDeviceApiTps, getDeviceApiTps)`
- **Dual Step Functions Flows**: `CreateDevicesMap` and `UpdateDevicesMap` with independent concurrency limits
- **S3 Payload Pattern**: Batch data and detailed results stored in S3 with lightweight references passed through Step Functions
- **Intelligent Batch Creation**: Maximizes concurrency utilization by creating batches equal to max_concurrency when possible
- **Multi-Level Retry Strategy**: Boto3 adaptive retry + application-level exponential backoff + Step Functions retry
- **Automatic Operation Detection**: Determines Create vs Update based on device data structure
- **Comprehensive Error Handling**: Distinguishes between retryable and permanent errors
- **Wave Processing**: Large jobs processed in multiple waves when needed
- **Database Integration**: Complete device lifecycle tracking with SMSN to AWS ID mapping
- **CloudWatch Integration**: Custom metrics, alarms, and comprehensive monitoring dashboard
- **Intelligent Batch Cleanup**: Automatic deletion of temporary batch files and result files with fallback S3 lifecycle rules
- **Strict SMSN Validation**: Enforces exactly 64-character SMSN format for consistency
- **Build System Integration**: Comprehensive Lambda validation and testing during brazil-build process
- **100K+ Device Support**: Optimized for processing 100,000+ devices with increased Lambda timeouts and concurrent operations
- **Scalable Result Processing**: Parallel loading of batch results with ThreadPoolExecutor for efficient report generation

### Architecture
- **Input Bucket**: S3 bucket with EventBridge integration for file upload triggers
- **Report Bucket**: S3 bucket for batch files, reports, temporary data storage, and detailed results
- **Step Functions**: Dual-flow state machine (`AWSIoTWirelessDeviceBulkManagementStateMachine`) with operation-specific routing
- **Lambda Functions**: Input processor (10-min timeout), API processor (15-min timeout), report/notify (10-min timeout), and database initialization
- **Aurora Database**: Serverless v2 cluster (PostgreSQL 16.1) with Data API for scalable device tracking
- **Notification Services**: SQS queue or SNS topic for completion notifications
- **CloudWatch**: Comprehensive monitoring with custom metrics and dashboard
- Support for Amazon Sidewalk device positioning configuration
- Parallel processing with configurable concurrency limits
- Error classification and retry mechanisms
- Comprehensive CloudWatch dashboard for monitoring

### Device Operations
- **Create Operations**: Full device creation with validation and positioning support
- **Update Operations**: Device updates with comprehensive error handling
- **Batch Processing**: Parallel processing of large device batches with intelligent sizing that maximizes concurrency
- **Database Integration**: Complete device state tracking with Aurora PostgreSQL using main batch names
- **Monitoring**: Real-time metrics and comprehensive dashboard
- **Notifications**: Flexible notification options for batch completion
- **Error Handling**: Robust error handling with detailed reporting
- **Rate Limiting**: Configurable TPS limits to prevent API throttling
- **Positioning Support**: Full support for Sidewalk device positioning
- **Validation**: Comprehensive input validation with detailed error messages
- **Reporting**: Detailed success/failure reports stored in S3 with complete device information

### Technical Highlights
- **Event-Driven**: S3 upload triggers automated processing
- **Serverless**: Complete serverless architecture with Lambda and Step Functions
- **Scalable**: Auto-scaling with configurable concurrency limits, supports 100K+ devices
- **Resilient**: Built-in retry logic and error handling
- **Observable**: Comprehensive monitoring and alerting capabilities
- **Efficient**: Concurrent S3 operations and optimized batch processing
- **Consistent**: Main batch name extraction ensures database consistency

### Configuration
- Configurable notification types (SQS, SNS, or None)
- Adjustable TPS limits for API rate limiting
- Customizable batch sizes and concurrency settings
- Flexible deployment options with CDK parameters
- Comprehensive Makefile targets for development and deployment
- Lambda validation during build process
- S3 lifecycle rules for automatic cleanup (7-day retention)
- Increased Lambda timeouts for large-scale processing (10-15 minutes)

### Database
- **Database Name**: `iot_devices`
- **Engine**: Aurora PostgreSQL 16.1 (Serverless v2)
- **Capacity**: 0.5 - 4 ACUs (auto-scaling)
- **Tables**: `devices` (device tracking with main batch names), `batches` (batch processing status)
- **Access**: RDS Data API (HTTPS, no VPC required for Lambda functions)
- **Features**: Automatic schema creation, triggers for updated_at timestamps, comprehensive indexes
- **Sample Queries**: 50+ SQL examples for common operations and analytics

### Critical Fixes Included
- **Step Functions JSONPath** - Correct Map state parameter mapping using `$.Map.Item.Value.fieldName` syntax
- **TPS Configuration** - Fixed CDK parameter access to use actual config values instead of unresolved references
- **Batch File Cleanup** - Enhanced cleanup logic to handle both main and individual batch names with detailed logging
- **Step Functions Payload Limit** - S3-based result storage eliminates 256KB payload size limit for 100K+ devices
- **Report-Notify S3 Permissions** - Added missing `grantRead` permission for loading detailed results from S3
- **Batch Concurrency Optimization** - Maximizes concurrency utilization by creating batches equal to max_concurrency
- **Main Batch Name Consistency** - Extracts main batch name from individual batch names for consistent database storage
- **Aurora Version Compatibility** - Updated to PostgreSQL 16.1 for wider regional availability

### Documentation
- **README.md**: Comprehensive deployment and usage guide with IAM requirements
- **SAMPLE_QUERIES.md**: 50+ SQL query examples for Aurora database
- **CHANGELOG.md**: Complete version history and feature documentation
- **IAM Requirements**: Detailed permissions for deployment and runtime operations
- **Sample Files**: Example JSON files for device creation and updates

### Performance
- **100K Devices**: ~3 hours processing time (well within 4-hour Step Functions limit)
- **Concurrent Processing**: Up to 10 concurrent Lambda executions for create operations
- **Batch Optimization**: Intelligent sizing maximizes parallelism while respecting timeout constraints
- **S3 Operations**: Up to 20 concurrent uploads, 10 concurrent downloads
- **Database**: Auto-scaling Aurora Serverless v2 handles high-volume operations

### Cost Optimization
- Aurora Serverless v2 auto-scales based on load (0.5-4 ACUs)
- S3 lifecycle rules for automatic cleanup (7-day retention)
- Efficient Lambda execution with optimized batch sizes
- CloudWatch log retention (2 weeks)

### Security
- IAM least privilege for all Lambda functions
- VPC isolation for Aurora database
- Encryption at rest for S3 and Aurora
- Secrets Manager for database credentials
- Comprehensive IAM documentation for deployment and runtime

---

## Version History

- **1.0.0** (2025-10-22) - Initial production release with complete feature set, 100K+ device support, and comprehensive documentation
