Merge pull request #41 from njfio/feature/advanced-search-and-logging

feat: Advanced Search Scoring & Logging Standardization

Author: njfio

README.md

@@ -7,7 +7,7 @@ An AI agent memory system implemented in Rust with intelligent memory management
 
 ## Project Status
 
-- **Library**: Rust library with 189 passing tests
+- **Library**: Rust library with 191 passing tests
 - **Version**: 0.1.0 (development)
 - **Architecture**: Modular design with optional feature flags
 - **Storage**: In-memory, file-based, and SQL backends (PostgreSQL)
@@ -16,22 +16,25 @@ An AI agent memory system implemented in Rust with intelligent memory management
 
 ## Recent Improvements
 
-- ✅ Standardized logging system with structured tracing
-- ✅ Fixed compilation errors in memory management tests
-- ✅ Enhanced automatic summarization with fallback handling
-- ✅ Improved test coverage for advanced memory operations
-- ✅ Comprehensive test suite with 189 passing tests
+- ✅ **Advanced Search Scoring**: Implemented sophisticated content type scoring algorithm with multi-factor analysis
+- ✅ **Logging Standardization**: Comprehensive structured logging system with tracing integration
+- ✅ **Search Intelligence**: Production-ready content classification for documentation, code, and knowledge content
+- ✅ **Test Coverage**: Enhanced test suite with 191 passing tests including search algorithm validation
+- ✅ **Memory Management**: Fixed compilation errors and improved automatic summarization
+- ✅ **Documentation**: Added logging standards and search scoring documentation
 
 ## Core Features
 
 ### Memory Management
 
 - Memory operations (store, retrieve, update, delete)
+- **Advanced Search Engine**: Sophisticated content type scoring with multi-factor analysis
 - Intelligent memory consolidation and summarization
 - Automatic summarization with multiple strategies
 - Lifecycle management with archival and cleanup
 - Memory optimization and compression
 - Advanced analytics and performance monitoring
+- **Structured Logging**: Comprehensive tracing and monitoring system
 
 ### Knowledge Graph
 
@@ -161,7 +164,7 @@ cargo run --example phase5b_document_demo --features "document-processing"
 ## Testing
 
 ```bash
-# Run all tests (189 tests)
+# Run all tests (191 tests)
 cargo test
 
 # Run with specific features
@@ -231,6 +234,7 @@ src/
 - **[Deployment Guide](docs/deployment.md)** - Production deployment instructions
 - **[Testing Guide](docs/testing_guide.md)** - Testing strategies and best practices
 - **[Error Handling Guide](docs/error_handling_guide.md)** - Comprehensive error handling
+- **[Logging Standards](docs/logging_standards.md)** - Structured logging and tracing guidelines
 
 ### API Documentation
 

docs/logging_standards.md

@@ -0,0 +1,321 @@
+# Logging Standards for Synaptic
+
+## Overview
+
+This document defines the logging standards and best practices for the Synaptic memory system. Consistent logging is crucial for production monitoring, debugging, and observability.
+
+## Logging Framework
+
+We use the `tracing` crate for structured logging throughout the codebase. This provides:
+- Structured, machine-readable logs
+- Hierarchical spans for request tracing
+- Configurable log levels
+- Rich context information
+
+## Log Levels
+
+### ERROR
+Use for errors that require immediate attention or indicate system failures:
+```rust
+tracing::error!(
+    error = %err,
+    operation = "memory_store",
+    key = %memory_key,
+    "Failed to store memory entry"
+);
+```
+
+**When to use:**
+- Database connection failures
+- Critical system errors
+- Security violations
+- Data corruption
+
+### WARN
+Use for potentially problematic situations that don't stop execution:
+```rust
+tracing::warn!(
+    threshold = %threshold,
+    current_usage = %usage,
+    "Memory usage approaching threshold"
+);
+```
+
+**When to use:**
+- Resource usage warnings
+- Deprecated API usage
+- Configuration issues
+- Performance degradation
+
+### INFO
+Use for general operational information:
+```rust
+tracing::info!(
+    component = "memory_manager",
+    operation = "consolidation",
+    memories_processed = %count,
+    duration_ms = %duration.as_millis(),
+    "Memory consolidation completed"
+);
+```
+
+**When to use:**
+- System startup/shutdown
+- Major operation completion
+- Configuration changes
+- User actions
+
+### DEBUG
+Use for detailed diagnostic information:
+```rust
+tracing::debug!(
+    query = %query_text,
+    results_count = %results.len(),
+    execution_time_ms = %execution_time.as_millis(),
+    "Search query executed"
+);
+```
+
+**When to use:**
+- Algorithm execution details
+- Internal state changes
+- Performance metrics
+- Detailed operation flow
+
+### TRACE
+Use for very detailed diagnostic information:
+```rust
+tracing::trace!(
+    step = "similarity_calculation",
+    vector_a_len = %vector_a.len(),
+    vector_b_len = %vector_b.len(),
+    similarity_score = %score,
+    "Calculated vector similarity"
+);
+```
+
+**When to use:**
+- Fine-grained execution flow
+- Loop iterations
+- Mathematical calculations
+- Low-level operations
+
+## Structured Logging Patterns
+
+### Standard Fields
+
+Always include these fields when relevant:
+
+```rust
+tracing::info!(
+    component = "memory_manager",     // Module/component name
+    operation = "store",              // Operation being performed
+    user_id = %user_id,              // User context (if applicable)
+    session_id = %session_id,        // Session context (if applicable)
+    duration_ms = %duration.as_millis(), // Operation duration
+    "Operation completed successfully"
+);
+```
+
+### Error Logging Pattern
+
+```rust
+tracing::error!(
+    error = %err,                    // Error details
+    error_type = "ValidationError",  // Error classification
+    component = "input_validator",   // Where error occurred
+    operation = "validate_memory",   // What was being done
+    input_size = %input.len(),       // Relevant context
+    "Input validation failed"
+);
+```
+
+### Performance Logging Pattern
+
+```rust
+tracing::info!(
+    component = "search_engine",
+    operation = "semantic_search",
+    query_complexity = %complexity,
+    results_count = %results.len(),
+    cache_hit = %cache_hit,
+    duration_ms = %duration.as_millis(),
+    memory_usage_mb = %memory_usage / 1024 / 1024,
+    "Search operation completed"
+);
+```
+
+## Spans for Request Tracing
+
+Use spans to trace operations across multiple functions:
+
+```rust
+#[tracing::instrument(
+    name = "memory_consolidation",
+    fields(
+        memory_count = %memories.len(),
+        strategy = %strategy_name
+    )
+)]
+async fn consolidate_memories(
+    memories: &[MemoryEntry],
+    strategy_name: &str
+) -> Result<ConsolidationResult> {
+    tracing::info!("Starting memory consolidation");
+    
+    // Operation implementation
+    
+    tracing::info!(
+        consolidated_count = %result.consolidated_count,
+        "Consolidation completed successfully"
+    );
+    
+    Ok(result)
+}
+```
+
+## Context Propagation
+
+Use the `tracing::Span::current()` to add context to existing spans:
+
+```rust
+async fn process_memory(&self, memory: &MemoryEntry) -> Result<()> {
+    let span = tracing::Span::current();
+    span.record("memory_id", &memory.id);
+    span.record("memory_type", &format!("{:?}", memory.memory_type));
+    
+    // Processing logic
+}
+```
+
+## What NOT to Log
+
+### Avoid println!/eprintln!
+Never use `println!` or `eprintln!` in production code. Use tracing instead:
+
+```rust
+// ❌ Don't do this
+println!("Processing memory: {}", memory_id);
+
+// ✅ Do this instead
+tracing::info!(
+    memory_id = %memory_id,
+    "Processing memory entry"
+);
+```
+
+### Sensitive Information
+Never log sensitive data:
+- Passwords or API keys
+- Personal information
+- Encryption keys
+- Session tokens
+
+```rust
+// ❌ Don't do this
+tracing::info!("User password: {}", password);
+
+// ✅ Do this instead
+tracing::info!(
+    user_id = %user_id,
+    "User authentication successful"
+);
+```
+
+### High-Frequency Logs
+Avoid logging in tight loops or high-frequency operations without rate limiting:
+
+```rust
+// ❌ Don't do this
+for item in large_collection {
+    tracing::debug!("Processing item: {:?}", item);
+}
+
+// ✅ Do this instead
+tracing::debug!(
+    item_count = %large_collection.len(),
+    "Processing collection"
+);
+for (index, item) in large_collection.iter().enumerate() {
+    if index % 1000 == 0 {
+        tracing::debug!(
+            processed = %index,
+            total = %large_collection.len(),
+            "Processing progress"
+        );
+    }
+}
+```
+
+## Configuration
+
+### Log Levels by Environment
+
+- **Development**: `TRACE` or `DEBUG`
+- **Testing**: `INFO` or `WARN`
+- **Production**: `INFO` or `WARN`
+- **Critical Systems**: `WARN` or `ERROR`
+
+### Environment Variables
+
+```bash
+# Set log level
+RUST_LOG=synaptic=info
+
+# Component-specific levels
+RUST_LOG=synaptic::memory=debug,synaptic::search=info
+
+# JSON output for production
+SYNAPTIC_LOG_FORMAT=json
+```
+
+## Testing Logging
+
+Use `tracing-test` for testing log output:
+
+```rust
+#[cfg(test)]
+mod tests {
+    use tracing_test::traced_test;
+    
+    #[traced_test]
+    #[tokio::test]
+    async fn test_memory_operation() {
+        // Test implementation
+        
+        // Verify logs were emitted
+        assert!(logs_contain("Memory operation completed"));
+    }
+}
+```
+
+## Monitoring Integration
+
+Logs should integrate with monitoring systems:
+
+```rust
+// Add correlation IDs for distributed tracing
+tracing::info!(
+    trace_id = %trace_id,
+    span_id = %span_id,
+    component = "memory_manager",
+    "Operation started"
+);
+```
+
+## Performance Considerations
+
+- Use lazy evaluation for expensive log formatting
+- Consider log sampling for high-volume operations
+- Use appropriate log levels to control verbosity
+- Avoid string allocation in hot paths
+
+```rust
+// ❌ Expensive formatting always executed
+tracing::debug!("Complex data: {}", expensive_format(&data));
+
+// ✅ Lazy evaluation
+tracing::debug!(data = ?data, "Complex data processed");
+```
+
+This logging standard ensures consistent, structured, and useful logging throughout the Synaptic codebase.

src/cli/syql/parser.rs

@@ -198,7 +198,9 @@ impl SyQLParser {
             return Ok(CompletionContext::Keyword);
         }
 
-        let last_token = tokens.last().unwrap().to_uppercase();
+        let last_token = tokens.last()
+            .ok_or_else(|| MemoryError::InvalidQuery { message: "Empty token list".to_string() })?
+            .to_uppercase();
 
         match last_token.as_str() {
             "SELECT" | "MATCH" | "WHERE" | "HAVING" | "ORDER" | "GROUP" => Ok(CompletionContext::Keyword),

src/memory/knowledge_graph/mod.rs

@@ -148,7 +148,7 @@ impl MemoryKnowledgeGraph {
         }
 
         // Sort by relationship strength
-        related_memories.sort_by(|a, b| b.relationship_strength.partial_cmp(&a.relationship_strength).unwrap());
+        related_memories.sort_by(|a, b| b.relationship_strength.partial_cmp(&a.relationship_strength).unwrap_or(std::cmp::Ordering::Equal));
 
         Ok(related_memories)
     }