feat: Dynamic Library Sync System for autogenlib, serena, and graph-sitter

[codegen-sh]

🚀 Overview

This PR implements a dynamic library synchronization system that automatically keeps external libraries (autogenlib, serena, graph-sitter) up-to-date in the analyzer repository.

✨ What's New

1. Automated Library Sync (sync_libraries.py)

• Hash-based change detection for efficient updates
• Clones libraries to temporary directory (.lib_sync_temp/, git ignored)
• Copies only Python source files, filtering out tests and build artifacts
• Supports offline mode after initial clone
• State tracking in Libraries/.sync_state.json

2. Module Validation (validate_modules.py)

• Validates all library imports
• Tests adapter modules and classes
• Verifies function availability
• Provides detailed validation reports

3. Comprehensive Documentation (LIBRARY_SYNC.md)

• Quick start guide
• Automated sync setup (hooks, CI/CD, cron)
• Troubleshooting guide
• Best practices and FAQs

📊 Libraries Synced

Library	Files Synced	Source
autogenlib	8 files	Zeeeepa/autogenlib
serena	37 files	Zeeeepa/serena
graph_sitter_lib	650 files	Zeeeepa/graph-sitter

🎯 Key Features

• ✅ Auto-detects changes using MD5 hashing
• ✅ Syncs only what's needed (Python source files)
• ✅ Filters out cruft (tests, pycache, build artifacts)
• ✅ Maintains sync state for efficient updates
• ✅ Works offline once initially cloned
• ✅ Version controlled in Libraries/ directory

📁 Directory Structure

analyzer/
├── Libraries/                    # Synced libraries
│   ├── autogenlib/              # From Zeeeepa/autogenlib
│   ├── serena/                  # From Zeeeepa/serena
│   ├── graph_sitter_lib/        # From Zeeeepa/graph-sitter
│   └── .sync_state.json         # Sync state tracking
├── .lib_sync_temp/              # Temp clones (gitignored)
├── sync_libraries.py            # Sync script
├── validate_modules.py          # Validation script
├── LIBRARY_SYNC.md              # Documentation
└── .gitignore                   # Excludes temp directory

🔧 Usage

Sync All Libraries

python sync_libraries.py

Sync Specific Library

python sync_libraries.py --library autogenlib

Check Status

python sync_libraries.py --check

Validate Modules

python validate_modules.py

🤖 Automated Sync Options

Git Hook (Recommended)

Auto-sync after git pull by adding to .git/hooks/post-merge:

#!/bin/bash
python3 sync_libraries.py

GitHub Actions

Daily automated sync with CI/CD integration

Cron Job

Server-side periodic sync every 6 hours

📝 Changes Made

• Added sync_libraries.py - Dynamic library sync script
• Added validate_modules.py - Module validation script
• Added LIBRARY_SYNC.md - Comprehensive documentation
• Added .gitignore - Excludes temporary sync directory
• Updated Libraries/autogenlib/ - Synced from source (8 files)
• Added Libraries/serena/ - Synced from source (37 files)
• Added Libraries/graph_sitter_lib/ - Synced from source (650 files)
• Added Libraries/.sync_state.json - Tracks last sync

🎉 Benefits

1. Always Up-to-Date: Libraries auto-sync from source repos
2. Clean Integration: No git submodules complexity
3. Minimal Size: Only source files, no tests/docs/build artifacts
4. Offline Support: Works after initial clone
5. Version Control: All synced files tracked in git
6. Easy Rollback: Git history allows reverting library updates

⚠️ Note

The validation script currently shows some import errors due to missing dependencies (openai, networkx, etc.) and a syntax issue in static_libs.py that existed prior to this PR. These will need to be addressed separately.

🔗 Related Documentation

• LIBRARY_SYNC.md - Full documentation
• sync_libraries.py - Sync script
• validate_modules.py - Validation script

---

Ready for Review! ✅

---

💻 View my work • 👤 Initiated by @Zeeeepa • About Codegen
⛔ Remove Codegen from PR • 🚫 Ban action checks

---

Summary by cubic

Adds a dynamic library sync system that auto-updates autogenlib, serena, and graph-sitter in the repo, plus a validator and docs. This removes manual copying and keeps Libraries/ current.

• New Features

  • Automated sync (sync_libraries.py) with hash-based change detection, offline mode, and state in Libraries/.sync_state.json.
  • Clones to .lib_sync_temp (gitignored) and copies only Python sources, skipping tests/build artifacts.
  • Validation (validate_modules.py) to check imports, adapters, and function availability with a report.
  • LIBRARY_SYNC.md with quick start and CI/cron setup.
  • Libraries synced: autogenlib (8 files), serena (37), graph_sitter_lib (~650).

• Migration

  • Run: python sync_libraries.py after pulling; optionally add a post-merge hook or CI job.
  • For validation, install needed deps (e.g., openai, networkx) to avoid import errors.

[sourcery-ai]

The pull request #1 has too many files changed.

The GitHub API will only let us fetch up to 300 changed files, and this pull request has 708.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}