Feature Request: Custom Metrics Collection and Registration

[laplaque]

Overview

Extend the Instana plugins architecture to support custom metrics collection beyond the default process monitoring metrics, with TOML-based configuration and multiple OpenTelemetry metric types.

Current Limitations

Fixed Metric Set: All services currently collect the same predefined process metrics:

• CPU/memory usage, process count, disk I/O, file descriptors, thread counts, context switches

Single Metric Type: All metrics are implemented as OpenTelemetry Observable Gauges, regardless of their semantic meaning

No Extensibility: Services cannot register application-specific or business metrics

No Configuration: The metric_types field in plugin.toml is not actively used for customization

Proposed Solution

1. TOML-Based Custom Metrics Configuration

Extend plugin.toml to support custom metric definitions:

[custom_metrics]
enabled = true

[[custom_metrics.sources]]
name = "jmx_collector"
type = "jmx"
endpoint = "localhost:9999"
metrics = [
    {name = "heap_memory", otel_type = "gauge", unit = "bytes"},
    {name = "thread_pool_size", otel_type = "updowncounter", unit = "1"}
]

[[custom_metrics.sources]]
name = "log_parser"
type = "log_file" 
path = "/var/log/app.log"
metrics = [
    {name = "error_count", otel_type = "counter", pattern = "ERROR"},
    {name = "response_time", otel_type = "histogram", pattern = "Transaction.*completed in (\\d+)ms", buckets = [10, 50, 100, 500, 1000]}
]

[[custom_metrics.sources]]
name = "http_endpoint"
type = "http"
url = "http://localhost:8080/metrics"
format = "json"
interval = 30
metrics = [
    {name = "active_sessions", otel_type = "gauge", json_path = "$.sessions.active"},
    {name = "requests_per_second", otel_type = "counter", json_path = "$.requests.total"}
]

2. Multiple OpenTelemetry Metric Types

Support all OpenTelemetry metric instruments:

Counter (monotonically increasing):

counter = self.meter.create_counter("requests_total")
counter.add(1, {"method": "GET"})

UpDownCounter (can increase/decrease):

updown = self.meter.create_up_down_counter("active_connections") 
updown.add(1)  # connection opened

Histogram (distributions/latencies):

histogram = self.meter.create_histogram("request_duration_ms")
histogram.record(125.3, {"endpoint": "/api/users"})

Observable Gauge (current implementation):

gauge = self.meter.create_observable_gauge("cpu_usage")

3. Pluggable Metric Source System

Built-in Source Types:

• jmx: Java Management Extensions
• log_file: Log file parsing with regex patterns
• http: HTTP/REST endpoint polling
• database: SQL query execution
• file: File-based metrics (JSON, CSV, etc.)
• command: Execute shell commands

Custom Source Plugins: Allow third-party metric collectors

4. Architecture Changes Required

New Components:

• MetricSourceRegistry: Manage metric source plugins
• CustomMetricCollector: Orchestrate custom metric collection
• MetricSourceBase: Abstract base class for metric sources
• ConfigValidator: Validate TOML metric configurations

Modified Components:

• base_sensor.py: Integrate custom metrics into monitoring loop
• otel_connector.py: Support multiple OpenTelemetry metric types
• toml_utils.py: Parse and validate custom metrics configuration
• metadata_store.py: Store custom metric metadata

5. Implementation Phases

Phase 1: Core Infrastructure

• Metric source plugin system
• TOML configuration parsing
• Multiple OTel metric type support

Phase 2: Basic Source Types

• HTTP endpoint polling
• Log file parsing
• Command execution

Phase 3: Advanced Sources

• JMX integration
• Database connectivity
• File-based metrics

Phase 4: Advanced Features

• Custom aggregation functions
• Metric transformations
• Conditional collection

Impact Assessment

Complexity: Major architectural change requiring significant development and testing effort

Backward Compatibility: Must maintain existing functionality without breaking changes

Performance: Custom metric collection could impact monitoring overhead

Dependencies: May require additional Python packages (JMX, database drivers, etc.)

Success Criteria

• Services can define custom metrics via TOML configuration
• Support for all OpenTelemetry metric types
• Pluggable architecture for metric sources
• Zero impact on existing services
• Comprehensive test coverage for all source types
• Performance benchmarks showing acceptable overhead

Target Release

Proposed for v1.0.0 as a major feature release after psutil migration stabilization.

Dependencies

• Completion of psutil migration (recently completed)
• Stable OpenTelemetry integration
• TOML configuration framework enhancements

Related Work

This feature request emerged from discussions about extending the current process monitoring capabilities to support application-specific metrics and business KPIs that go beyond standard system metrics.

[laplaque]

Implementation Plan: Custom Metrics Collection and Registration

After analyzing the current codebase against the requirements in this issue, I've developed an implementation plan to address all identified limitations.

Current State Assessment

Foundation Elements Present:

• TOML configuration infrastructure (toml_utils.py)
• OpenTelemetry integration (otel_connector.py)
• Metadata management system (metadata_store.py)
• Plugin architecture foundation

Issues Outstanding:

• Fixed metric set across all services (hardcoded process metrics only)
• Single metric type limitation (Observable Gauges only)
• No extensibility for custom/application-specific metrics
• Configuration not used for metric customization
• Missing components: MetricSourceRegistry, CustomMetricCollector, MetricSourceBase, ConfigValidator

Implementation Status: 0% Complete - None of the core custom metrics features have been implemented yet.

The implementation will be broken down into 5 incremental releases (v0.2.00 → v1.0.00), each building on the previous milestone. Technical specifications for each release follow in separate comments.

[laplaque]

Release v0.2.00: Foundation & Multiple OpenTelemetry Metric Types

Overview

Establish the core infrastructure and support for all OpenTelemetry metric types beyond the current Observable Gauge limitation.

Deliverables

1. Enhanced OTel Connector (common/otel_connector.py)

• Add support for Counter, UpDownCounter, Histogram, and Observable Gauge
• Implement metric type validation and appropriate creation methods
• Add error handling for unsupported metric configurations

2. Metric Source Base Classes (common/metric_sources/)

• MetricSourceBase - Abstract base class for all metric sources
• StaticMetricSource - For simple value-based metrics
• CallbackMetricSource - For function-based metric collection

3. Basic Metric Source Registry (common/metric_registry.py)

• Registration and discovery system for metric sources
• Type checking and validation framework
• Basic lifecycle management (register/unregister)

4. Enhanced Configuration Support

• Extend toml_utils.py to parse [custom_metrics] sections
• Add validation for metric type and source configurations
• Implement configuration merging (defaults + custom)

5. Updated Base Sensor Integration

• Modify base_sensor.py to use registry-based metric collection
• Maintain backward compatibility with existing process metrics
• Add hooks for custom metric integration

Configuration Example

[custom_metrics]
enabled = true

[[custom_metrics.metrics]]
name = "heap_memory"
source_type = "static"
otel_type = "gauge"
unit = "bytes"
value_callback = "get_heap_memory"

[[custom_metrics.metrics]]
name = "request_count"
source_type = "static"
otel_type = "counter"
unit = "1"
value_callback = "get_request_count"

Success Criteria

• All 4 OpenTelemetry metric types supported and tested
• Plugin architecture foundation established
• Backward compatibility maintained
• 80%+ test coverage for new components
• Performance impact < 5% vs current implementation

[laplaque]

Release v0.3.00: Basic Built-in Metric Sources

Overview

Implement fundamental metric source types for immediate value: HTTP endpoints, command execution, and log file parsing.

Deliverables

1. HTTP Endpoint Source (common/metric_sources/http_source.py)

• Poll REST/HTTP endpoints for metrics data
• Support JSON response parsing with JSONPath expressions
• Configurable intervals, timeouts, and retry logic
• Authentication support (Bearer tokens, basic auth)
• Response validation and error handling

2. Command Execution Source (common/metric_sources/command_source.py)

• Execute shell commands and parse output for metrics
• Support for various output formats (JSON, CSV, plain text)
• Regex-based value extraction from command output
• Command timeout and error handling
• Security considerations for command execution

3. Log File Parsing Source (common/metric_sources/log_source.py)

• Monitor log files for metric-relevant patterns
• Regex-based pattern matching and value extraction
• File rotation handling and position tracking
• Rate-based metrics (errors per minute, transactions per second)
• Configurable file paths and update intervals

4. Enhanced Configuration

• Extended TOML configuration for new source types
• Validation for source-specific parameters
• Configuration templates and examples

5. Comprehensive Testing

• Unit tests for each source type
• Integration tests with mock services
• Performance and reliability testing
• Documentation and usage examples

Configuration Example

[[custom_metrics.sources]]
name = "api_health"
type = "http"
url = "http://localhost:8080/health"
interval = 30
metrics = [
    {name = "response_time", otel_type = "histogram", json_path = "$.response_time"},
    {name = "active_connections", otel_type = "gauge", json_path = "$.connections.active"}
]

[[custom_metrics.sources]]
name = "error_monitor"
type = "log_file"
path = "/var/log/app.log"
interval = 10
metrics = [
    {name = "error_count", otel_type = "counter", pattern = "ERROR", unit = "1"}
]

Success Criteria

• 3 production-ready built-in source types
• Real-world integration examples
• Performance benchmarks established
• 85%+ test coverage
• Comprehensive documentation

[laplaque]

Release v0.4.00: Dynamic Plugin System & Advanced Features

Overview

Implement dynamic loading capabilities, advanced metric aggregation, and enhanced configuration management for production readiness.

Deliverables

1. Dynamic Plugin Loader (common/plugin_loader.py)

• Runtime discovery and loading of custom metric source plugins
• Plugin validation and compatibility checking
• Secure plugin sandboxing and error isolation
• Plugin lifecycle management (load/unload/reload)
• Support for both Python modules and external executables

2. Advanced Metric Aggregation (common/aggregators/)

• Time-window aggregation (sliding windows, tumbling windows)
• Statistical aggregations (percentiles, moving averages)
• Cross-metric calculations and derived metrics
• Memory-efficient aggregation algorithms
• Configurable aggregation rules via TOML

3. Enhanced Configuration Management

• Configuration hot-reloading without restart
• Environment variable interpolation
• Configuration inheritance and composition
• Validation with detailed error reporting
• Configuration migration utilities

4. Database Integration Source (common/metric_sources/database_source.py)

• SQL query execution for metric collection
• Support for PostgreSQL, MySQL, SQLite, SQL Server
• Connection pooling and query optimization
• Parameterized queries for security
• Query result transformation and mapping

5. Production Features

• Circuit breaker pattern for failing sources
• Graceful degradation and fallback mechanisms
• Comprehensive logging and debugging tools
• Performance monitoring and bottleneck detection
• Memory usage optimization

Configuration Example

[custom_metrics]
plugin_directories = ["/opt/plugins", "./local_plugins"]
aggregation_window = "5m"

[[custom_metrics.sources]]
name = "database_metrics"
type = "database"
connection_string = "postgresql://user:pass@localhost/db"
interval = 60
queries = [
    {name = "active_users", sql = "SELECT COUNT(*) FROM users WHERE last_login > NOW() - INTERVAL '1 hour'", otel_type = "gauge"},
    {name = "transaction_volume", sql = "SELECT COUNT(*) FROM transactions WHERE created_at > $1", params = ["1h_ago"], otel_type = "counter"}
]

[[custom_metrics.aggregators]]
source = "api_health.response_time"
type = "percentile"
percentiles = [50, 95, 99]
window = "5m"

Success Criteria

• Dynamic plugin system fully functional
• Production-grade reliability features
• Advanced aggregation capabilities
• 90%+ test coverage
• Performance optimizations documented
• Migration path from v0.3.00 established

[laplaque]

Release v1.0.00: Enterprise Features & Official Release

Overview

Complete the custom metrics system with enterprise-grade features, comprehensive tooling, and production deployment capabilities.

Deliverables

1. Advanced Monitoring & Observability

• Built-in metrics for the metrics system itself (meta-metrics)
• Custom dashboard templates for Grafana/Kibana
• Health check endpoints for load balancer integration
• Performance profiling and optimization tools
• SLA monitoring and alerting integration

2. Enterprise Integration Features

• LDAP/Active Directory authentication for sensitive sources
• Role-based access control for metric configuration
• Audit logging for configuration changes
• Enterprise service discovery integration
• Multi-tenant configuration isolation

3. Advanced Metric Sources

• Message queue integration (RabbitMQ, Apache Kafka, Redis)
• Cloud provider APIs (AWS CloudWatch, Azure Monitor, GCP)
• Container orchestration metrics (Kubernetes, Docker Swarm)
• Network monitoring integration (SNMP, network latency probes)
• Custom protocol handlers (TCP, UDP, gRPC health checks)

4. Production Deployment Tools

• Docker containerization with multi-stage builds
• Kubernetes Helm charts and operators
• Configuration management via HashiCorp Vault
• Blue-green deployment support
• Automated backup and disaster recovery

5. Developer Experience

• Plugin development SDK with templates and examples
• Configuration validation CLI tool
• Migration utilities for version upgrades
• Comprehensive API documentation
• Integration test framework for custom plugins

6. Documentation & Support

• Complete user and administrator guides
• Plugin development tutorials
• Troubleshooting guides and FAQ
• Performance tuning recommendations
• Integration examples for common scenarios

Configuration Example

[custom_metrics]
version = "1.0.0"
enterprise_features = true

[custom_metrics.auth]
type = "ldap"
server = "ldap://company.com"
base_dn = "ou=users,dc=company,dc=com"

[[custom_metrics.sources]]
name = "kubernetes_cluster"
type = "kubernetes"
namespace = "production"
metrics = [
    {name = "pod_cpu_usage", resource = "pods", aggregation = "avg", otel_type = "gauge"},
    {name = "deployment_replicas", resource = "deployments", field = "replicas", otel_type = "gauge"}
]

[custom_metrics.high_availability]
enabled = true
backup_interval = "1h"
failover_timeout = "30s"

Success Criteria

• Production-ready for enterprise deployment
• Comprehensive security audit passed
• Performance benchmarks documented
• 95%+ test coverage across all components
• Complete documentation suite
• Backward compatibility guaranteed
• Migration paths from all previous versions
• Official support channels established

Post-Release

• Community plugin repository
• Regular security updates
• Long-term support (LTS) commitments
• Integration with major monitoring platforms