feat: Add AWS cloud infrastructure logs integration#858
Open
bhaskarvilles wants to merge 45 commits intocisagov:mainfrom
Open
feat: Add AWS cloud infrastructure logs integration#858bhaskarvilles wants to merge 45 commits intocisagov:mainfrom
bhaskarvilles wants to merge 45 commits intocisagov:mainfrom
Conversation
…5.07.0_merge_cisagov
…5.07.0_merge_cisagov
…5.07.0_merge_cisagov
…5.07.0_merge_cisagov
- Apply multiple enhancements to `clean-processed-folder.py` so that it runs fast
enough to keep up with the generation of log files in pipeline capture mode. These
changes increased the file processing rate by a factor of 100.
- Preprocess the filebeat registry into a format for checking file presence using the
`in` operator.
- Replace regular expression pattern matching for mime file types with list searching.
- Refactor running `fuser` with the subprocess module for increased speed.
- Treat zero-length files, which have no mime type, as eligible log files.
- Update logging to improve the ability to monitor script performance.
- Fix the search for Suricata log files which was excluding log files created in the
pipeline mode.
- Run `clean-processed-folder.py` every minute in order to minimize the risk of
overflowing the partition where the log files are stored especially in the pipeline
capture mode.
Fix log files not removed quickly enough
…5.08.1_merge_cisagov
…5.08.1_merge_cisagov
…5.09.0_merge_cisagov
…5.11.0_merge_cisagov
Malcolm v25.11.0
…5.12.0_merge_cisagov
…5.12.0_merge_cisagov
…5.12.0_merge_cisagov
…5.12.0_merge_cisagov
…5.12.0_merge_cisagov
…5.12.1_merge_cisagov
Malcolm v25.12.1
- Implement Logstash parser for VPC Flow Logs (v2 format) - Implement Logstash parser for CloudTrail API activity logs - Add Python script for automated S3 log collection - Follow Malcolm's ECS field mapping patterns - Integrate with existing GeoIP/ASN enrichment pipeline - Add security event tagging (unauthorized access, high-risk actions) Addresses cisagov#232
- Add comprehensive user documentation for AWS cloud logs - Include setup instructions for VPC Flow Logs and CloudTrail - Add troubleshooting guide and best practices - Create environment configuration template - Update documentation table of contents - Add Python requirements for log collector Related: cisagov#232
mmguero
added
logstash
Collaborator
|
Thanks, I will review this in the next week or so and get it merged for probably a February release. |
- Add comprehensive unit tests for VPC Flow Logs parser (6 tests) - Add comprehensive unit tests for CloudTrail parser (6 tests) - Create AWS VPC Flow Logs Overview dashboard with 6 visualizations - Create AWS CloudTrail Activity dashboard with 6 visualizations - Add dashboard import guide with instructions Tests cover: - Successful parsing of log formats - Security event detection (unauthorized access, high-risk actions) - Protocol mapping and field extraction - Cloud metadata validation Dashboards include: - Traffic/API activity timelines - Success/failure ratios - Top talkers/users/actions - Protocol/security event breakdowns - Geographic distribution maps Related: cisagov#232
- Add AWS ELB/ALB access logs parser (Classic ELB and ALB support) - Add AWS S3 access logs parser with sensitive file detection - Add AWS Route 53 query logs parser with DNS tunneling/DGA detection - Add Azure NSG Flow Logs parser with port scan detection - Add Azure Activity Logs parser with high-risk operation monitoring All parsers include: - ECS field mapping - Security event detection and tagging - Integration with Malcolm's GeoIP/ASN enrichment - Comprehensive threat detection (SQL injection, DGA, tunneling, etc.) Updated log collector to support all new log types. Related: cisagov#232
Collaborator
|
Just wanted to let you know I saw your updates. It will probably be next week sometime before I'm able to carve out the time to review this, but know it's appreciated and we'll get them reviewed. I may end up moving the logstash filters to a separate (new) parse pipeline rather than putting them in "enrichment" but don't worry about it for now, that's easy to adjust. Cheers. |
Author
|
@mmguero Thanks for your reply, Just adding more enhancements, i will wait for your review and for the next release, planning to integrate Azure, Akamai and other cloud provider logs as well. |
Parsers: - AWS RDS: error, slow query (DDL/DML/SELECT classify), audit, PostgreSQL - Azure App Gateway: access logs + WAF (OWASP rule group detection) Dashboards (7 new): - aws-elb-overview: requests, status codes, response time, top clients, SQLi events - aws-s3-access-overview: ops over time, bytes, sensitive file/security events - aws-route53-overview: queries, response codes, DNS threat events (tunneling/DGA) - azure-nsg-overview: traffic timeline, allow/deny, protocols, port scans - azure-activity-overview: operations, outcomes, top users, high-risk events - azure-appgw-overview: (via cloud-unified) WAF blocks, OWASP rules - cloud-unified-security: unified cross-provider overview with geo attack map Alerting Rules (10 monitors): - Mass unauthorized access attempts - DNS tunneling / DGA - S3 sensitive file access - SQL injection (ELB + AppGW WAF) - Azure high-risk resource deletion - CloudTrail root account usage - Azure WAF RCE/LFI attack - RDS mass authentication failures - Large data exfiltration (>1GB) - Azure NSG mass port scanning Related: cisagov#232
Collaborator
|
I'm on site at one of our funding sources this week, but I plan on looking at this and getting it merged next week. Thanks for your patience! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds comprehensive cloud infrastructure log ingestion and analysis capabilities to Malcolm, enabling monitoring of hybrid cloud + on-prem environments.
Related Issue
Closes #232
What's New
Features
✅ VPC Flow Logs parsing (v2 format)
✅ CloudTrail API activity parsing
✅ Automated S3 log collection
✅ ECS field mapping
✅ GeoIP/ASN enrichment integration
✅ Security tagging (unauthorized access, high-risk actions, etc.)
Security Event Detection
VPC Flow Logs
rejected_traffic- Blocked connectionshigh_volume_transfer- Large data transfers (>10MB)potential_port_scan- Multiple connection rejectionsCloudTrail
unauthorized_access_attempt- AccessDenied errorshigh_risk_action- Destructive operations (DeleteBucket, TerminateInstances, etc.)root_account_usage- Root user activityfailed_authentication- Failed console loginsTesting
Documentation
Files Changed
Summary
Impact
This contribution enables Malcolm to:
Checklist
Next Steps
Author: @bhaskarvilles
Branch:
Cloud-Infrastructure-Logs-Integration