S3 Cost Analyzer

A utility for downloading and analyzing cost data from S3 files. This tool efficiently handles data extraction, parsing, and aggregation, with special support for recovering deleted files using S3 versioning.

Features

• Modular design separating downloading and analysis
• Efficient caching of downloaded files to avoid redundant downloads
• Support for recovering deleted files via S3 versioning
• Extensible analysis framework for custom cost metrics
• Calculation of idle vs. non-idle cost ratios
• Hourly cost averaging
• Command-line interface for easy usage

Installation

Clone the repository:

git clone https://github.com/yourusername/s3_cost_analyzer.git
cd s3_cost_analyzer

Install the package:

pip install -e .

Usage

Command Line Interface

The package provides a command-line interface for common operations:

Downloading files:

s3-cost-analyze download /path/to/query_file.json --aws-profile myprofile

Analyzing costs:

s3-cost-analyze analyze /path/to/query_file.json --aws-profile myprofile

Managing configuration:

s3-cost-analyze config --show
s3-cost-analyze config --set aws_profile myprofile

Custom Analysis Scripts

You can create custom analysis scripts by importing the package:

from s3_cost_analyzer.analyzer import S3CostAnalyzer

# Initialize analyzer
analyzer = S3CostAnalyzer(aws_profile='myprofile')

# Download and analyze files
results = analyzer.analyze_query('/path/to/query_file.json')

# Process results
print(f"Total cost: ${results['total_cost']:.2f}")

Idle Cost Analysis

To run the specialized idle cost analysis:

python -m analysis.idle_cost_analysis /path/to/query_file.json

This will:

1. Download all the required files (if not already cached)
2. Extract cost data from each file
3. Calculate idle vs. non-idle cost ratios
4. Generate a detailed report

Architecture

The package is organized into several modules:

• downloader.py: Handles downloading and caching of S3 files
• processor.py: Extracts and aggregates cost data from files
• analyzer.py: Provides the main interface for analysis
• config.py: Manages configuration settings
• cli.py: Implements the command-line interface

Custom analysis scripts can be created in the analysis/ directory to extend the functionality.

Configuration

Configuration is managed through a JSON file (default: ~/.s3_cost_analyzer.json). Available settings:

• aws_profile: AWS profile name to use for authentication
• data_dir: Directory to store downloaded data
• output_dir: Directory to store analysis results
• log_level: Logging level (INFO, DEBUG, etc.)

Requirements

• Python 3.8+
• boto3
• requests

License

MIT