From a023125db7a83a1e3d0fe8311c83d576acf8eb32 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 21 Aug 2025 15:06:08 -0500 Subject: [PATCH 1/3] feat(plugins): add automated plugin documentation sync workflow - Add GitHub Actions workflow for syncing plugin docs from influxdb3_plugins - Create issue template for triggering sync requests - Add Node.js transformation script (port_to_docs.js) with ES modules - Add mapping configuration (docs_mapping.yaml) for all official plugins - Add npm scripts for plugin sync operations - Include comprehensive documentation in helper-scripts/influxdb3-plugins/README.md The workflow provides: - Issue-triggered automation with no cross-repo secrets required - Validation of source READMEs against template requirements - Content transformation with Hugo shortcodes and GitHub URLs - Screenshot generation for visual validation - Automatic PR creation with detailed change summaries --- .github/ISSUE_TEMPLATE/sync-plugin-docs.yml | 89 +++ .github/workflows/sync-plugins.yml | 353 +++++++++ helper-scripts/influxdb3-plugins/README.md | 669 ++++++++++++++++++ .../influxdb3-plugins/docs_mapping.yaml | 122 ++++ .../influxdb3-plugins/port_to_docs.js | 486 +++++++++++++ package.json | 5 +- 6 files changed, 1723 insertions(+), 1 deletion(-) create mode 100644 .github/ISSUE_TEMPLATE/sync-plugin-docs.yml create mode 100644 .github/workflows/sync-plugins.yml create mode 100644 helper-scripts/influxdb3-plugins/README.md create mode 100644 helper-scripts/influxdb3-plugins/docs_mapping.yaml create mode 100644 helper-scripts/influxdb3-plugins/port_to_docs.js diff --git a/.github/ISSUE_TEMPLATE/sync-plugin-docs.yml b/.github/ISSUE_TEMPLATE/sync-plugin-docs.yml new file mode 100644 index 0000000000..1bafbdbb30 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/sync-plugin-docs.yml @@ -0,0 +1,89 @@ +name: Sync Plugin Documentation +description: Request synchronization of plugin documentation from influxdb3_plugins repository +title: "Sync plugin docs: [PLUGIN_NAMES]" +labels: ["sync-plugin-docs", "documentation", "automation"] +assignees: [] +body: + - type: markdown + attributes: + value: | + ## Plugin Documentation Sync Request + + This will trigger an automated workflow to sync plugin documentation from the influxdb3_plugins repository to docs-v2. + + The workflow will: + 1. āœ… Validate source READMEs against template requirements + 2. šŸ”„ Transform content for docs-v2 compatibility + 3. šŸ—ļø Build Hugo site and generate screenshots + 4. šŸ“ Create a pull request with the changes + + **Note**: All plugin READMEs must pass validation before sync can proceed. + + - type: input + id: plugins + attributes: + label: Plugin Names + description: | + Comma-separated list of plugins to sync (e.g., `basic_transformation, downsampler`) or `all` to sync all plugins. + Plugin names should match the directory names in `influxdata/` (without the `influxdata/` prefix). + placeholder: "basic_transformation, downsampler" + value: "all" + validations: + required: true + + - type: input + id: source_commit + attributes: + label: Source Commit/Branch + description: | + Commit SHA, branch name, or tag from influxdb3_plugins repository to sync from. + Leave as `master` to sync from the latest main branch. + placeholder: "master" + value: "master" + validations: + required: true + + - type: textarea + id: context + attributes: + label: Additional Context + description: | + Optional: Provide any additional context about this sync request, such as: + - What changes were made to the plugins + - Any special considerations for review + - Links to related issues or PRs + placeholder: "Updated basic_transformation plugin to support new data types..." + validations: + required: false + + - type: checkboxes + id: checklist + attributes: + label: Pre-sync Checklist + description: Please confirm the following before submitting + options: + - label: Plugin READMEs in influxdb3_plugins follow the [README_TEMPLATE.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README_TEMPLATE.md) structure + required: true + - label: All required sections are present (Description, Configuration, Installation steps, Example usage, Troubleshooting) + required: true + - label: Plugin metadata includes proper emoji indicators (āš” triggers, šŸ·ļø tags, šŸ”§ compatibility) + required: true + - label: Code examples include expected outputs where applicable + required: true + + - type: markdown + attributes: + value: | + --- + + **What happens next?** + 1. Creating this issue will automatically trigger the sync workflow + 2. The workflow will validate all specified plugin READMEs + 3. If validation passes, content will be transformed and a PR will be created + 4. If validation fails, you'll receive feedback on what needs to be fixed + 5. This issue will be automatically closed when the process completes + + **Need help?** + - See the [plugin documentation workflow guide](https://github.com/influxdata/docs-v2/blob/master/helper-scripts/influxdb3-plugins/README.md) + - Check the [plugin template requirements](https://github.com/influxdata/influxdb3_plugins/blob/master/README_TEMPLATE.md) + - Review existing [plugin documentation](https://docs.influxdata.com/influxdb3/core/reference/plugins/) \ No newline at end of file diff --git a/.github/workflows/sync-plugins.yml b/.github/workflows/sync-plugins.yml new file mode 100644 index 0000000000..581b3ab062 --- /dev/null +++ b/.github/workflows/sync-plugins.yml @@ -0,0 +1,353 @@ +name: Sync Plugin Documentation + +on: + issues: + types: [opened] + workflow_dispatch: + inputs: + plugins: + description: 'Plugin names to sync (comma-separated, or "all")' + required: true + default: 'all' + source_commit: + description: 'influxdb3_plugins commit SHA or branch' + required: false + default: 'master' + +permissions: + contents: write + pull-requests: write + issues: write + +jobs: + sync-plugins: + runs-on: ubuntu-latest + # Only run on issues with sync-plugin-docs label or manual dispatch + if: | + github.event_name == 'workflow_dispatch' || + (github.event_name == 'issues' && contains(github.event.issue.labels.*.name, 'sync-plugin-docs')) + + steps: + - name: Parse issue inputs + id: parse-inputs + if: github.event_name == 'issues' + uses: actions/github-script@v7 + with: + script: | + const issue = context.payload.issue; + const body = issue.body || ''; + + // Extract plugins and source_commit from issue body + const pluginsMatch = body.match(/### Plugin Names\s*\n\s*(.+)/); + const commitMatch = body.match(/### Source Commit\/Branch\s*\n\s*(.+)/); + + const plugins = pluginsMatch ? pluginsMatch[1].trim() : 'all'; + const sourceCommit = commitMatch ? commitMatch[1].trim() : 'master'; + + core.setOutput('plugins', plugins); + core.setOutput('source_commit', sourceCommit); + core.setOutput('issue_number', issue.number); + + - name: Set workflow inputs + id: inputs + run: | + if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then + echo "plugins=${{ github.event.inputs.plugins }}" >> $GITHUB_OUTPUT + echo "source_commit=${{ github.event.inputs.source_commit }}" >> $GITHUB_OUTPUT + echo "issue_number=" >> $GITHUB_OUTPUT + else + echo "plugins=${{ steps.parse-inputs.outputs.plugins }}" >> $GITHUB_OUTPUT + echo "source_commit=${{ steps.parse-inputs.outputs.source_commit }}" >> $GITHUB_OUTPUT + echo "issue_number=${{ steps.parse-inputs.outputs.issue_number }}" >> $GITHUB_OUTPUT + fi + + - name: Update issue status + if: steps.inputs.outputs.issue_number != '' + uses: actions/github-script@v7 + with: + script: | + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: ${{ steps.inputs.outputs.issue_number }}, + body: 'šŸ”„ Plugin sync started...\n\nPlugins: `${{ steps.inputs.outputs.plugins }}`\nSource: `${{ steps.inputs.outputs.source_commit }}`' + }); + + - name: Checkout docs-v2 + uses: actions/checkout@v4 + with: + path: docs-v2 + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Checkout influxdb3_plugins (sparse) + run: | + git clone --filter=blob:none --sparse https://github.com/influxdata/influxdb3_plugins.git influxdb3_plugins + cd influxdb3_plugins + git sparse-checkout set influxdata/ validate_readme.py + git checkout ${{ steps.inputs.outputs.source_commit }} + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'yarn' + cache-dependency-path: docs-v2/yarn.lock + + - name: Install dependencies + run: | + cd docs-v2 + CYPRESS_INSTALL_BINARY=0 yarn install + + - name: Setup Python + uses: actions/setup-python@v4 + with: + python-version: '3.9' + + - name: Validate plugin READMEs + id: validate + run: | + cd influxdb3_plugins + + # Determine which plugins to validate + if [[ "${{ steps.inputs.outputs.plugins }}" == "all" ]]; then + PLUGINS="" + else + PLUGINS="${{ steps.inputs.outputs.plugins }}" + fi + + # Run validation + if [[ -n "$PLUGINS" ]]; then + echo "Validating specific plugins: $PLUGINS" + python validate_readme.py --plugins "$PLUGINS" 2>&1 | tee validation.log + else + echo "Validating all plugins" + python validate_readme.py 2>&1 | tee validation.log + fi + + # Check if validation passed + if [[ ${PIPESTATUS[0]} -eq 0 ]]; then + echo "validation_passed=true" >> $GITHUB_OUTPUT + else + echo "validation_passed=false" >> $GITHUB_OUTPUT + fi + + - name: Report validation failure + if: steps.validate.outputs.validation_passed == 'false' + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + let validationLog = ''; + try { + validationLog = fs.readFileSync('influxdb3_plugins/validation.log', 'utf8'); + } catch (e) { + validationLog = 'Could not read validation log'; + } + + const issueNumber = '${{ steps.inputs.outputs.issue_number }}'; + if (issueNumber) { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: parseInt(issueNumber), + body: `āŒ **Validation Failed** + + Plugin READMEs do not meet template requirements. Please fix the following issues in influxdb3_plugins: + + \`\`\` + ${validationLog} + \`\`\` + + See [README_TEMPLATE.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README_TEMPLATE.md) for requirements.` + }); + + await github.rest.issues.update({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: parseInt(issueNumber), + state: 'closed', + labels: ['sync-plugin-docs', 'validation-failed'] + }); + } + + core.setFailed('Plugin validation failed'); + + - name: Transform plugin documentation + if: steps.validate.outputs.validation_passed == 'true' + run: | + cd docs-v2 + + # Set PLUGIN_DIR for the transformation script + export INFLUXDB3_PLUGINS_PATH="../influxdb3_plugins" + + # Run the transformation + if [[ "${{ steps.inputs.outputs.plugins }}" == "all" ]]; then + node helper-scripts/influxdb3-plugins/port_to_docs.js + else + # Transform specific plugins + IFS=',' read -ra PLUGIN_ARRAY <<< "${{ steps.inputs.outputs.plugins }}" + for plugin in "${PLUGIN_ARRAY[@]}"; do + plugin=$(echo "$plugin" | xargs) # trim whitespace + echo "Transforming plugin: $plugin" + node helper-scripts/influxdb3-plugins/port_to_docs.js --plugin "$plugin" + done + fi + + - name: Build Hugo site + run: | + cd docs-v2 + npx hugo --quiet --minify + + - name: Setup Playwright + run: | + cd docs-v2 + npx playwright install chromium + + - name: Generate screenshots + id: screenshots + run: | + cd docs-v2 + + # Start Hugo server in background + npx hugo server --bind 0.0.0.0 --port 1313 --quiet & + HUGO_PID=$! + + # Wait for server to start + sleep 10 + + # Create screenshots directory + mkdir -p plugin-screenshots + + # Generate screenshots for changed plugin pages + node -e " + const { chromium } = require('playwright'); + const fs = require('fs'); + const path = require('path'); + + async function captureScreenshots() { + const browser = await chromium.launch(); + const page = await browser.newPage({ viewport: { width: 1200, height: 800 } }); + + // Find changed plugin files + const glob = require('glob'); + const pluginFiles = glob.sync('content/shared/influxdb3-plugins/plugins-library/**/*.md'); + + for (const file of pluginFiles) { + const pluginName = path.basename(file, '.md'); + const url = \`http://localhost:1313/influxdb3/core/reference/plugins/\${pluginName}/\`; + + try { + console.log(\`Capturing screenshot for \${pluginName}\`); + await page.goto(url, { waitUntil: 'networkidle' }); + await page.screenshot({ + path: \`plugin-screenshots/\${pluginName}.png\`, + fullPage: true + }); + } catch (error) { + console.log(\`Could not capture \${pluginName}: \${error.message}\`); + } + } + + await browser.close(); + } + + captureScreenshots().catch(console.error); + " + + # Stop Hugo server + kill $HUGO_PID + + # List generated screenshots + ls -la plugin-screenshots/ || echo "No screenshots generated" + echo "screenshots_generated=$(ls plugin-screenshots/ 2>/dev/null | wc -l)" >> $GITHUB_OUTPUT + + - name: Create Pull Request + if: steps.validate.outputs.validation_passed == 'true' + id: create-pr + uses: peter-evans/create-pull-request@v5 + with: + path: docs-v2 + token: ${{ secrets.GITHUB_TOKEN }} + commit-message: | + sync: update plugin documentation from influxdb3_plugins@${{ steps.inputs.outputs.source_commit }} + + Plugins: ${{ steps.inputs.outputs.plugins }} + branch: sync-plugins-${{ github.run_number }} + title: "Sync plugin documentation: ${{ steps.inputs.outputs.plugins }}" + body: | + ## Plugin Documentation Sync + + **Source**: influxdb3_plugins@${{ steps.inputs.outputs.source_commit }} + **Plugins**: ${{ steps.inputs.outputs.plugins }} + **Triggered by**: ${{ github.event_name == 'issues' && format('Issue #{0}', steps.inputs.outputs.issue_number) || 'Manual workflow dispatch' }} + + ### Changes Made + - āœ… Validated source READMEs against template requirements + - šŸ”„ Transformed content for docs-v2 compatibility + - šŸ–¼ļø Generated ${{ steps.screenshots.outputs.screenshots_generated }} plugin page screenshots + + ### Transformations Applied + - Removed emoji metadata (already in plugin JSON metadata) + - Converted relative links to GitHub URLs + - Added Hugo product shortcodes (`{{% product-name %}}`) + - Added standard logging section + - Updated support sections with docs-v2 format + + ### Screenshots + Plugin page previews are available in the `plugin-screenshots/` directory (if any were generated). + + ### Review Checklist + - [ ] All plugin pages render correctly + - [ ] GitHub links resolve properly + - [ ] Product shortcodes display correctly + - [ ] Code examples are properly formatted + - [ ] Support sections link correctly + + --- + *This PR was automatically generated by the plugin sync workflow.* + + - name: Update issue with success + if: steps.validate.outputs.validation_passed == 'true' && steps.inputs.outputs.issue_number != '' + uses: actions/github-script@v7 + with: + script: | + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: ${{ steps.inputs.outputs.issue_number }}, + body: `āœ… **Plugin sync completed successfully!** + + **Pull Request**: #${{ steps.create-pr.outputs.pull-request-number }} + **Plugins synced**: ${{ steps.inputs.outputs.plugins }} + **Source commit**: ${{ steps.inputs.outputs.source_commit }} + **Screenshots generated**: ${{ steps.screenshots.outputs.screenshots_generated }} + + The PR is ready for review and includes all necessary transformations and validations.` + }); + + await github.rest.issues.update({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: ${{ steps.inputs.outputs.issue_number }}, + state: 'closed', + labels: ['sync-plugin-docs', 'completed'] + }); + + - name: Report failure + if: failure() && steps.inputs.outputs.issue_number != '' + uses: actions/github-script@v7 + with: + script: | + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: ${{ steps.inputs.outputs.issue_number }}, + body: `āŒ **Plugin sync failed** + + Please check the [workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details. + + You may need to: + - Check that the source commit exists + - Verify plugin names are correct + - Ensure all validation requirements are met` + }); \ No newline at end of file diff --git a/helper-scripts/influxdb3-plugins/README.md b/helper-scripts/influxdb3-plugins/README.md new file mode 100644 index 0000000000..86487bc251 --- /dev/null +++ b/helper-scripts/influxdb3-plugins/README.md @@ -0,0 +1,669 @@ +# InfluxDB 3 Plugin Documentation Automation + +This directory contains scripts and configuration for automating plugin documentation sync between `influxdata/influxdb3_plugins` (source) and `influxdata/docs-v2` (documentation site). + +## Files in This Directory + +- **`port_to_docs.js`** - Transforms plugin READMEs from influxdb3_plugins to docs-v2 format +- **`docs_mapping.yaml`** - Maps source plugin READMEs to destination documentation files +- **`README.md`** - This file: complete workflow documentation + +## Quick Start + +### Automated Sync (Recommended) + +When you update plugin READMEs in influxdb3_plugins: + +1. **Commit your changes** - The reminder workflow will automatically detect changes +2. **Click the sync link** - A comment will appear with a pre-filled link to create a sync request +3. **Review the PR** - The workflow creates a pull request with transformed content and screenshots + +### Manual Sync (Alternative) + +To manually trigger synchronization: + +1. **Create sync request**: [Open sync request form](https://github.com/influxdata/docs-v2/issues/new?template=sync-plugin-docs.yml) +2. **Fill in plugin names** and source commit +3. **Submit issue** - This automatically triggers the sync workflow + +### Local Development + +For local testing and development: + +1. **Validate source content** (run from influxdb3_plugins directory): + ```bash + cd /path/to/influxdb3_plugins + python validate_readme.py + ``` + +2. **Transform content** (run from docs-v2 root directory): + ```bash + cd /path/to/docs-v2 + npm run sync-plugins + ``` + +3. **Review changes**: + ```bash + git diff content/shared/influxdb3-plugins/ + ``` + +## NPM Scripts + +From the docs-v2 root directory, you can use these convenient npm scripts: + +- `npm run sync-plugins` - Transform all plugin documentation +- `npm run sync-plugins:dry-run` - Preview changes without writing files +- `npm run validate-plugin-config` - Validate the mapping configuration + +## Overview + +This workflow maintains consistent plugin documentation between repositories while ensuring bidirectional consistency and enabling future automation. + +## Key Principles + +1. **Single Source of Truth**: Plugin READMEs in `influxdb3_plugins` are the canonical source +2. **Consistency**: All plugin READMEs follow a standardized template +3. **Automation**: Documentation updates flow automatically from source to docs +4. **Validation**: Both repositories validate documentation quality +5. **No Divergence**: Content differences are managed through transformation rules + +## Repository Responsibilities + +### influxdb3_plugins (Source) +- Contains canonical plugin documentation in README files +- Follows strict template structure +- Includes all technical details and examples +- Validates README compliance + +### docs-v2 (Documentation Site) +- Transforms and enhances source documentation +- Adds product-specific formatting and shortcodes +- Includes additional support sections +- Never directly edits plugin content + +## Phase 1: Standardize Source Documentation + +### 1.1 README Template Structure + +All plugin READMEs in `influxdb3_plugins` must follow this structure: + +```markdown +# [Plugin Name] + +āš” [trigger-type1, trigger-type2] šŸ·ļø [tag1, tag2, tag3] šŸ”§ InfluxDB 3 Core, InfluxDB 3 Enterprise + +## Description + +[2-4 sentences describing what the plugin does, its main capabilities, supported trigger types, and any special features] + +## Configuration + +Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin's `config_file_path` parameter. + +If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. + +### Plugin metadata + +This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. + +### Required parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `param_name` | string | required | Description of parameter | + +### [Category] parameters + +[Additional parameter tables organized by category] + +### TOML configuration + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `config_file_path` | string | none | TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration) | + +*To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3. + +#### Example TOML configurations + +- [config_file.toml](config_file.toml) - Description + +For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3_plugins/README.md](/README.md). + +## [Special Requirements Section - if applicable] + +[For example: Data requirements, Schema requirements, Software requirements] + +## Installation steps + +1. Start InfluxDB 3 with the Processing Engine enabled: + ```bash + influxdb3 serve \ + --node-id node0 \ + --object-store file \ + --data-dir ~/.influxdb3 \ + --plugin-dir ~/.plugins + ``` + +2. Install required Python packages (if any): + ```bash + influxdb3 install package package_name + ``` + +## Trigger setup + +### [Trigger Type 1] + +[Description and example] + +```bash +influxdb3 create trigger \ + --database mydb \ + --plugin-filename gh:influxdata/[plugin]/[plugin].py \ + --trigger-spec "[spec]" \ + --trigger-arguments '[arguments]' \ + trigger_name +``` + +### [Trigger Type 2] + +[Additional trigger examples as needed] + +## Example usage + +### Example 1: [Use case name] + +[Description of the example] + +```bash +# Create the trigger +influxdb3 create trigger \ + --database mydb \ + --plugin-filename gh:influxdata/[plugin]/[plugin].py \ + --trigger-spec "[spec]" \ + --trigger-arguments '[arguments]' \ + trigger_name + +# Write test data +influxdb3 write \ + --database mydb \ + "measurement,tag=value field=1" + +# Query results +influxdb3 query \ + --database mydb \ + "SELECT * FROM result_measurement" +``` + +### Expected output + +``` +[Expected output format] +``` + +**Details:** +- Before: [input state] +- After: [output state] + +### Example 2: [Use case name] + +[Additional examples following the same pattern] + +## Code overview + +### Files + +- `plugin_name.py`: Main plugin code +- `config.toml`: Example configuration file + +### Main functions + +#### `function_name(params)` +[Description of what the function does] + +Key operations: +1. [Operation 1] +2. [Operation 2] +3. [Operation 3] + +### Key logic + +[Description of the plugin's core logic and processing flow] + +## Troubleshooting + +### Common issues + +#### Issue: [Problem description] +**Solution**: [How to fix it] + +#### Issue: [Problem description] +**Solution**: [How to fix it] + +### Debugging tips + +1. **[Tip 1]**: [Description] +2. **[Tip 2]**: [Description] + +### Performance considerations + +- [Performance note 1] +- [Performance note 2] + +## Questions/Comments + +For questions or comments about this plugin, please open an issue in the [influxdb3_plugins repository](https://github.com/influxdata/influxdb3_plugins/issues). +``` + +### 1.2 Validation Requirements + +Each README must: +- Include all required sections in the correct order +- Use consistent parameter table formatting +- Provide at least 2 complete usage examples +- Include expected output for examples +- Document all configuration parameters +- Link to TOML configuration files (if applicable) + +## Phase 2: Transformation to docs-v2 + +### 2.1 Transformation Rules + +When porting content from `influxdb3_plugins` to `docs-v2`: + +#### Content Transformations + +1. **Remove emoji metadata line** (already in plugin metadata) + +2. **Convert relative links to GitHub URLs**: + - `[file.toml](file.toml)` ā†’ `[file.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/[plugin]/file.toml)` + - `/README.md` ā†’ `https://github.com/influxdata/influxdb3_plugins/blob/master/README.md` + +3. **Add product shortcodes**: + - Replace "InfluxDB 3" with `{{% product-name %}}` + - Add `{{% token-link %}}` for token references + - Use `{{% show-in "enterprise" %}}` for conditional content + +4. **Enhance opening paragraph** (optional): + - Can be expanded for better SEO and context + - Must maintain factual accuracy + +5. **Add docs-specific sections**: + +```markdown +## Logging + +Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: + +```bash +influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" +``` + +Log columns: +- **event_time**: Timestamp of the log event +- **trigger_name**: Name of the trigger that generated the log +- **log_level**: Severity level (INFO, WARN, ERROR) +- **log_text**: Message describing the action or error + +## Report an issue + +For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). + +## Find support for {{% product-name %}} + +The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. +``` + +### 2.2 File Structure in docs-v2 + +Plugin documentation location: +- **Official plugins**: `/content/shared/influxdb3-plugins/plugins-library/official/[plugin-name].md` +- **Example plugins**: `/content/shared/influxdb3-plugins/plugins-library/examples/[plugin-name].md` +- **Community plugins**: `/content/shared/influxdb3-plugins/plugins-library/community/[plugin-name].md` + +## Phase 3: Automation Implementation + +### 3.1 Validation Script (validate_readme.py) + +```python +#!/usr/bin/env python3 +""" +Validates plugin README files against the standard template. +""" + +import sys +import re +from pathlib import Path + +REQUIRED_SECTIONS = [ + "# ", # Title + "## Description", + "## Configuration", + "### Required parameters", + "## Installation steps", + "## Trigger setup", + "## Example usage", + "## Code overview", + "## Troubleshooting", + "## Questions/Comments" +] + +def validate_readme(readme_path): + """Validate a README file against the template.""" + with open(readme_path, 'r') as f: + content = f.read() + + errors = [] + + # Check for required sections + for section in REQUIRED_SECTIONS: + if section not in content: + errors.append(f"Missing required section: {section}") + + # Check for emoji metadata + if not re.search(r'^āš”.*šŸ·ļø.*šŸ”§', content, re.MULTILINE): + errors.append("Missing emoji metadata line") + + # Check for parameter tables + if '| Parameter |' not in content: + errors.append("No parameter tables found") + + # Check for code examples + if '```bash' not in content: + errors.append("No bash code examples found") + + # Check for expected output + if '### Expected output' not in content and '**Expected output' not in content: + errors.append("No expected output section found") + + return errors + +if __name__ == "__main__": + # Validate all plugin READMEs + plugin_dirs = Path('influxdata').glob('*/README.md') + + all_valid = True + for readme_path in plugin_dirs: + errors = validate_readme(readme_path) + if errors: + all_valid = False + print(f"\nāŒ {readme_path}:") + for error in errors: + print(f" - {error}") + else: + print(f"āœ… {readme_path}") + + sys.exit(0 if all_valid else 1) +``` + +### 3.2 Transformation Script (port_to_docs.py) + +```python +#!/usr/bin/env python3 +""" +Transforms plugin READMEs from influxdb3_plugins to docs-v2 format. +""" + +import re +import yaml +from pathlib import Path + +def transform_content(content, plugin_name): + """Transform README content for docs-v2.""" + + # Remove emoji metadata line + content = re.sub(r'^āš”.*šŸ”§.*$', '', content, flags=re.MULTILINE) + + # Convert relative links to GitHub URLs + base_url = f"https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/{plugin_name}/" + content = re.sub(r'\[([^\]]+)\]\(([^)]+\.toml)\)', + f'[\\1]({base_url}\\2)', content) + + # Replace InfluxDB 3 references + content = content.replace('InfluxDB 3 Core/Enterprise', '{{% product-name %}}') + content = content.replace('InfluxDB 3', '{{% product-name %}}') + + # Remove Questions/Comments section (will be replaced) + content = re.sub(r'## Questions/Comments.*', '', content, flags=re.DOTALL) + + # Add docs-specific sections + docs_sections = """ +## Logging + +Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: + +```bash +influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" +``` + +Log columns: +- **event_time**: Timestamp of the log event +- **trigger_name**: Name of the trigger that generated the log +- **log_level**: Severity level (INFO, WARN, ERROR) +- **log_text**: Message describing the action or error + +## Report an issue + +For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). + +## Find support for {{% product-name %}} + +The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. +""" + + return content + docs_sections + +def process_plugin(source_path, target_path, plugin_name): + """Process a single plugin README.""" + with open(source_path, 'r') as f: + content = f.read() + + transformed = transform_content(content, plugin_name) + + # Ensure target directory exists + target_path.parent.mkdir(parents=True, exist_ok=True) + + with open(target_path, 'w') as f: + f.write(transformed) + + print(f"āœ… Processed {plugin_name}") + +if __name__ == "__main__": + # Load mapping configuration + with open('docs_mapping.yaml', 'r') as f: + config = yaml.safe_load(f) + + for plugin_name, mapping in config['plugins'].items(): + source = Path(mapping['source']) + target = Path(mapping['target']) + + if source.exists(): + process_plugin(source, target, plugin_name) + else: + print(f"āŒ Source not found: {source}") +``` + +### 3.3 Mapping Configuration (docs_mapping.yaml) + +```yaml +plugins: + basic_transformation: + source: influxdata/basic_transformation/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/basic-transformation.md + category: official + + downsampler: + source: influxdata/downsampler/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/downsampler.md + category: official + + forecast_error_evaluator: + source: influxdata/forecast_error_evaluator/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/forecast-error-evaluator.md + category: official + + influxdb_to_iceberg: + source: influxdata/influxdb_to_iceberg/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/influxdb-to-iceberg.md + category: official + + mad_check: + source: influxdata/mad_check/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/mad-check.md + category: official + + notifier: + source: influxdata/notifier/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/notifier.md + category: official + + prophet_forecasting: + source: influxdata/prophet_forecasting/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/prophet-forecasting.md + category: official + + state_change: + source: influxdata/state_change/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/state-change.md + category: official + + stateless_adtk_detector: + source: influxdata/stateless_adtk_detector/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/stateless-adtk-detector.md + category: official + + system_metrics: + source: influxdata/system_metrics/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/system-metrics.md + category: official + + threshold_deadman_checks: + source: influxdata/threshold_deadman_checks/README.md + target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/official/threshold-deadman-checks.md + category: official +``` + +## Phase 4: GitHub Actions Automation + +### 4.1 Automated Workflow Architecture + +The plugin documentation sync process uses a two-repository workflow: + +1. **influxdb3_plugins**: Detects README changes and creates reminder comments +2. **docs-v2**: Handles the complete sync workflow via issue-triggered automation + +### 4.2 Reminder System (influxdb3_plugins) + +**File**: `.github/workflows/remind-sync-docs.yml` + +- **Triggers**: On pushes to `influxdata/**/README.md` files +- **Action**: Creates commit comment with pre-filled link to docs-v2 sync form +- **No secrets required**: Pure URL-based integration + +**Flow**: +``` +Plugin README change ā†’ Commit ā†’ Automatic comment with sync link +``` + +### 4.3 Sync Automation (docs-v2) + +**Files**: +- `.github/workflows/sync-plugins.yml` - Main sync workflow +- `.github/ISSUE_TEMPLATE/sync-plugin-docs.yml` - Issue template for requests + +**Triggers**: +- Issue creation with `sync-plugin-docs` label (from template) +- Manual workflow dispatch + +**Process**: +1. **Parse request** - Extract plugin names and source commit from issue +2. **Validate source** - Run `validate_readme.py` on specified plugins +3. **Transform content** - Apply docs-v2 formatting and enhancements +4. **Generate screenshots** - Capture Hugo-rendered plugin pages +5. **Create PR** - Submit changes with comprehensive summary +6. **Update issue** - Close with success/failure status + +### 4.4 Key Features + +- **No cross-repo secrets** - Uses URL parameters and public APIs +- **Visual validation** - Screenshots included in PRs +- **Comprehensive error handling** - Clear feedback on failures +- **Selective sync** - Individual plugins or "all" +- **Progress tracking** - Issue updates throughout process + +## Phase 5: Manual Workflow (Until Automation is Ready) + +### Step 1: Update Source README +1. Edit README in `influxdb3_plugins/influxdata/[plugin]/` +2. Run validation: `python validate_readme.py` +3. Fix any validation errors + +### Step 2: Transform Content +1. Change to docs-v2 directory: `cd ../docs-v2` +2. Run transformation: `npm run sync-plugins` +3. Review generated content: `git diff content/shared/influxdb3-plugins/` + +### Step 3: Manual Adjustments (if needed) +1. Add any docs-specific enhancements +2. Verify shortcodes and formatting +3. Test links and examples + +### Step 4: Commit and PR +1. Commit changes in influxdb3_plugins +2. Commit transformed content in docs-v2 +3. Reference source commit in docs-v2 commit message + +## Maintenance Guidelines + +### For New Plugins +1. Create README following template in `influxdb3_plugins` +2. Add entry to `docs_mapping.yaml` +3. Run validation and transformation scripts +4. Review and merge generated documentation + +### For Updates +1. Update source README in `influxdb3_plugins` +2. Automation (or manual script) creates PR to docs-v2 +3. Review for accuracy and merge + +### Exception Handling +- Use `[no-sync]` in commit message to skip automation +- Document special cases in `exceptions.yaml` +- Manual override allowed for urgent fixes + +## Quality Assurance Checklist + +Before merging any plugin documentation: + +- [ ] Source README passes validation (`validate_readme.py`) +- [ ] All required sections present and properly formatted +- [ ] At least 2 complete usage examples with expected output +- [ ] All configuration parameters documented +- [ ] Links to GitHub source files work +- [ ] TOML configuration documented (if applicable) +- [ ] Transformation script runs without errors +- [ ] Generated docs-v2 content is accurate +- [ ] Product shortcodes properly applied +- [ ] Support sections included + +## Benefits of This Workflow + +1. **Consistency**: Single source of truth eliminates divergence +2. **Automation**: Reduces manual work and human error +3. **Validation**: Ensures quality standards are met +4. **Scalability**: Easy to add new plugins +5. **Maintainability**: Clear ownership and update process +6. **Traceability**: Git history shows relationship between repos + +## Future Enhancements + +1. **Automated testing** of code examples +2. **Version tracking** for plugin compatibility +3. **Changelog generation** from commit history +4. **Multi-language** example generation +5. **API documentation** generation from metadata +6. **Search optimization** through automated tagging + +## Conclusion + +This workflow establishes a sustainable process for maintaining high-quality, consistent plugin documentation across both repositories while enabling progressive automation. The standardized template and transformation rules ensure that documentation remains synchronized while allowing for repository-specific requirements. \ No newline at end of file diff --git a/helper-scripts/influxdb3-plugins/docs_mapping.yaml b/helper-scripts/influxdb3-plugins/docs_mapping.yaml new file mode 100644 index 0000000000..73f6ee0fb8 --- /dev/null +++ b/helper-scripts/influxdb3-plugins/docs_mapping.yaml @@ -0,0 +1,122 @@ +# Mapping configuration for transforming plugin documentation from influxdb3_plugins to docs-v2 +# This file defines the source and target paths for each plugin's documentation + +plugins: + basic_transformation: + source: ../influxdb3_plugins/influxdata/basic_transformation/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/basic-transformation.md + category: official + additional_sections: + - schema_requirements + - logging + + downsampler: + source: ../influxdb3_plugins/influxdata/downsampler/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/downsampler.md + category: official + additional_sections: + - schema_management + - logging + + forecast_error_evaluator: + source: ../influxdb3_plugins/influxdata/forecast_error_evaluator/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/forecast-error-evaluator.md + category: official + additional_sections: + - logging + + influxdb_to_iceberg: + source: ../influxdb3_plugins/influxdata/influxdb_to_iceberg/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/influxdb-to-iceberg.md + category: official + additional_sections: + - logging + + mad_check: + source: ../influxdb3_plugins/influxdata/mad_check/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/mad-check.md + category: official + additional_sections: + - logging + + notifier: + source: ../influxdb3_plugins/influxdata/notifier/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/notifier.md + category: official + additional_sections: + - logging + + prophet_forecasting: + source: ../influxdb3_plugins/influxdata/prophet_forecasting/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/prophet-forecasting.md + category: official + additional_sections: + - logging + + state_change: + source: ../influxdb3_plugins/influxdata/state_change/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/state-change.md + category: official + additional_sections: + - logging + + stateless_adtk_detector: + source: ../influxdb3_plugins/influxdata/stateless_adtk_detector/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/stateless-adtk-detector.md + category: official + additional_sections: + - logging + + system_metrics: + source: ../influxdb3_plugins/influxdata/system_metrics/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/system-metrics.md + category: official + additional_sections: + - logging + + threshold_deadman_checks: + source: ../influxdb3_plugins/influxdata/threshold_deadman_checks/README.md + target: content/shared/influxdb3-plugins/plugins-library/official/threshold-deadman-checks.md + category: official + additional_sections: + - logging + +# Example plugins (if any exist in the examples directory) +example_plugins: + # Add example plugins here when they are created + # example_name: + # source: examples/example_name/README.md + # target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/examples/example-name.md + # category: examples + +# Community plugins (future expansion) +community_plugins: + # Add community plugins here as they are contributed + # author_plugin: + # source: community/author/plugin_name/README.md + # target: ../docs-v2/content/shared/influxdb3-plugins/plugins-library/community/plugin-name.md + # category: community + +# Configuration settings +settings: + # Whether to validate links in transformed content + validate_links: true + + # Whether to check for template remnants + check_template_remnants: true + + # Whether to create backup of existing docs before overwriting + create_backup: false + + # Default category if not specified + default_category: official + +# Exceptions - plugins that require special handling +exceptions: + # List plugins that should not be automatically synced + no_sync: [] + + # Plugins that require manual review after transformation + manual_review: + - influxdb_to_iceberg # Complex configuration may need manual verification + - forecast_error_evaluator # Multiple notification options need review \ No newline at end of file diff --git a/helper-scripts/influxdb3-plugins/port_to_docs.js b/helper-scripts/influxdb3-plugins/port_to_docs.js new file mode 100644 index 0000000000..af900e1857 --- /dev/null +++ b/helper-scripts/influxdb3-plugins/port_to_docs.js @@ -0,0 +1,486 @@ +#!/usr/bin/env node +/** + * Transforms plugin READMEs from influxdb3_plugins to docs-v2 format. + * Maintains consistency while applying documentation-specific enhancements. + */ + +import { promises as fs } from 'fs'; +import path from 'path'; +import yaml from 'js-yaml'; + +/** + * Load the mapping configuration file. + */ +async function loadMappingConfig(configPath = 'docs_mapping.yaml') { + try { + const content = await fs.readFile(configPath, 'utf8'); + return yaml.load(content); + } catch (error) { + if (error.code === 'ENOENT') { + console.error(`āŒ Error: Configuration file '${configPath}' not found`); + } else { + console.error(`āŒ Error parsing YAML configuration: ${error.message}`); + } + process.exit(1); + } +} + +/** + * Remove the emoji metadata line from content. + */ +function removeEmojiMetadata(content) { + // Remove the emoji line (it's already in the plugin's JSON metadata) + const pattern = /^āš”.*?šŸ”§.*?$\n*/gm; + return content.replace(pattern, ''); +} + +/** + * Convert relative links to GitHub URLs. + */ +function convertRelativeLinks(content, pluginName) { + const baseUrl = `https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/${pluginName}/`; + + // Convert TOML file links + content = content.replace( + /\[([^\]]+\.toml)\]\(([^)]+\.toml)\)/g, + (match, linkText, linkPath) => `[${linkText}](${baseUrl}${linkPath})` + ); + + // Convert Python file links + content = content.replace( + /\[([^\]]+\.py)\]\(([^)]+\.py)\)/g, + (match, linkText, linkPath) => `[${linkText}](${baseUrl}${linkPath})` + ); + + // Convert main README reference + content = content.replace( + '[influxdb3_plugins/README.md](/README.md)', + '[influxdb3_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md)' + ); + + return content; +} + +/** + * Replace product references with Hugo shortcodes. + */ +function addProductShortcodes(content) { + // Replace various forms of InfluxDB 3 references + const replacements = [ + [/InfluxDB 3 Core\/Enterprise/g, '{{% product-name %}}'], + [/InfluxDB 3 Core and InfluxDB 3 Enterprise/g, '{{% product-name %}}'], + [/InfluxDB 3 Core, InfluxDB 3 Enterprise/g, '{{% product-name %}}'], + // Be careful not to replace in URLs or code blocks + [/(? [!WARNING] +> #### Requires existing schema +> +> By design, the plugin returns an error if the schema doesn't exist or doesn't contain the expected columns. +`; + } else if (pluginName === 'downsampler') { + schemaSection = `## Schema management + +Each downsampled record includes three additional metadata columns: + +- \`record_count\` ā€” the number of original points compressed into this single downsampled row +- \`time_from\` ā€” the minimum timestamp among the original points in the interval +- \`time_to\` ā€” the maximum timestamp among the original points in the interval +`; + } else { + return content; + } + + // Insert after Configuration section + if (content.includes('## Installation steps')) { + content = content.replace( + '## Installation steps', + schemaSection + '\n## Installation steps' + ); + } + + return content; +} + +/** + * Apply all transformations to convert README for docs-v2. + */ +function transformContent(content, pluginName, config) { + // Apply transformations in order + content = removeEmojiMetadata(content); + content = convertRelativeLinks(content, pluginName); + content = addProductShortcodes(content); + content = enhanceOpeningParagraph(content); + content = fixCodeBlockFormatting(content); + + // Add schema requirements if applicable + if ( + config.additional_sections && + config.additional_sections.includes('schema_requirements') + ) { + content = addSchemaRequirements(content, pluginName); + } + + // Add logging section + content = addLoggingSection(content); + + // Replace support section + content = replaceSupportSection(content); + + return content; +} + +/** + * Process a single plugin README. + * Returns true if successful, false otherwise. + */ +async function processPlugin(pluginName, mapping, dryRun = false) { + const sourcePath = mapping.source; + const targetPath = mapping.target; + + try { + // Check if source exists + await fs.access(sourcePath); + } catch (error) { + console.error(`āŒ Source not found: ${sourcePath}`); + return false; + } + + try { + // Read source content + const content = await fs.readFile(sourcePath, 'utf8'); + + // Transform content + const transformed = transformContent(content, pluginName, mapping); + + if (dryRun) { + console.log(`āœ… Would process ${pluginName}`); + console.log(` Source: ${sourcePath}`); + console.log(` Target: ${targetPath}`); + return true; + } + + // Ensure target directory exists + await fs.mkdir(path.dirname(targetPath), { recursive: true }); + + // Write transformed content + await fs.writeFile(targetPath, transformed, 'utf8'); + + console.log(`āœ… Processed ${pluginName}`); + console.log(` Source: ${sourcePath}`); + console.log(` Target: ${targetPath}`); + return true; + } catch (error) { + console.error(`āŒ Error processing ${pluginName}: ${error.message}`); + return false; + } +} + +/** + * Check if docs-v2 repository is accessible. + */ +async function validateDocsV2Path() { + try { + await fs.access('../..'); + return true; + } catch (error) { + console.warn('āš ļø Warning: docs-v2 repository structure not detected'); + console.warn( + ' Make sure you are running this from docs-v2/helper-scripts/influxdb3-plugins' + ); + return false; + } +} + +/** + * Parse command line arguments. + */ +function parseArgs() { + const args = process.argv.slice(2); + const options = { + config: 'docs_mapping.yaml', + plugin: null, + dryRun: false, + validate: false, + help: false, + }; + + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + switch (arg) { + case '--config': + options.config = args[++i]; + break; + case '--plugin': + options.plugin = args[++i]; + break; + case '--dry-run': + options.dryRun = true; + break; + case '--validate': + options.validate = true; + break; + case '--help': + case '-h': + options.help = true; + break; + default: + console.error(`Unknown argument: ${arg}`); + options.help = true; + break; + } + } + + return options; +} + +/** + * Show help message. + */ +function showHelp() { + console.log(` +Transform plugin READMEs from influxdb3_plugins to docs-v2 format + +Usage: node port_to_docs.js [options] + +Options: + --config Path to mapping configuration file (default: docs_mapping.yaml) + --plugin Process only specified plugin + --dry-run Show what would be done without making changes + --validate Validate configuration only + --help, -h Show this help message + +Examples: + node port_to_docs.js # Process all plugins + node port_to_docs.js --plugin basic_transformation # Process specific plugin + node port_to_docs.js --dry-run # Preview changes + node port_to_docs.js --validate # Check configuration +`); +} + +/** + * Main transformation function. + */ +async function main() { + const options = parseArgs(); + + if (options.help) { + showHelp(); + process.exit(0); + } + + // Load configuration + const config = await loadMappingConfig(options.config); + + if (!config || !config.plugins) { + console.error('āŒ Invalid configuration file'); + process.exit(1); + } + + // Validate configuration + if (options.validate) { + console.log('Validating configuration...'); + let valid = true; + + for (const [pluginName, mapping] of Object.entries(config.plugins)) { + if (!mapping.source || !mapping.target) { + console.error( + `āŒ Invalid mapping for ${pluginName}: missing source or target` + ); + valid = false; + continue; + } + + try { + await fs.access(mapping.source); + } catch (error) { + console.warn( + `āš ļø Source not found for ${pluginName}: ${mapping.source}` + ); + } + } + + if (valid) { + console.log('āœ… Configuration is valid'); + } + process.exit(valid ? 0 : 1); + } + + // Check if we're in the right location + if (!options.dryRun && !(await validateDocsV2Path())) { + console.log( + '\nTo use this script, ensure you are in the correct directory:' + ); + console.log(' cd docs-v2/helper-scripts/influxdb3-plugins'); + process.exit(1); + } + + // Process plugins + let pluginsToProcess = Object.entries(config.plugins); + + if (options.plugin) { + if (!config.plugins[options.plugin]) { + console.error(`āŒ Plugin '${options.plugin}' not found in configuration`); + process.exit(1); + } + pluginsToProcess = [[options.plugin, config.plugins[options.plugin]]]; + } + + console.log( + `${options.dryRun ? 'DRY RUN: ' : ''}Processing ${pluginsToProcess.length} plugin(s)...\n` + ); + + let successCount = 0; + let errorCount = 0; + + for (const [pluginName, mapping] of pluginsToProcess) { + if (await processPlugin(pluginName, mapping, options.dryRun)) { + successCount++; + } else { + errorCount++; + } + } + + // Print summary + console.log('\n' + '='.repeat(60)); + console.log('TRANSFORMATION SUMMARY'); + console.log('='.repeat(60)); + console.log(`Successfully processed: ${successCount}`); + console.log(`Errors: ${errorCount}`); + + if (errorCount === 0) { + console.log('\nāœ… All plugins processed successfully!'); + if (!options.dryRun) { + console.log('\nNext steps:'); + console.log('1. Review the generated documentation in docs-v2'); + console.log('2. Test that all links work correctly'); + console.log('3. Verify product shortcodes render properly'); + console.log('4. Commit changes in both repositories'); + } + } else { + console.log(`\nāŒ ${errorCount} plugin(s) failed to process`); + process.exit(1); + } +} + +// Handle unhandled promise rejections +process.on('unhandledRejection', (reason, promise) => { + console.error('Unhandled Rejection at:', promise, 'reason:', reason); + process.exit(1); +}); + +// Run main function +if (import.meta.url === `file://${process.argv[1]}`) { + main().catch((error) => { + console.error('āŒ Fatal error:', error.message); + process.exit(1); + }); +} + +export { transformContent, processPlugin, loadMappingConfig }; diff --git a/package.json b/package.json index 10896b237a..bd6a7f2ce3 100644 --- a/package.json +++ b/package.json @@ -55,7 +55,10 @@ "test:codeblocks:v2": "docker compose run --rm --name v2-pytest v2-pytest", "test:codeblocks:stop-monitors": "./test/scripts/monitor-tests.sh stop cloud-dedicated-pytest && ./test/scripts/monitor-tests.sh stop clustered-pytest", "test:e2e": "node cypress/support/run-e2e-specs.js", - "test:shortcode-examples": "node cypress/support/run-e2e-specs.js --spec \"cypress/e2e/content/article-links.cy.js\" content/example.md" + "test:shortcode-examples": "node cypress/support/run-e2e-specs.js --spec \"cypress/e2e/content/article-links.cy.js\" content/example.md", + "sync-plugins": "cd helper-scripts/influxdb3-plugins && node port_to_docs.js", + "sync-plugins:dry-run": "cd helper-scripts/influxdb3-plugins && node port_to_docs.js --dry-run", + "validate-plugin-config": "cd helper-scripts/influxdb3-plugins && node port_to_docs.js --validate" }, "type": "module", "browserslist": [ From 76f0c3426868c9e11152bf1959450a3bb20b2cd9 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 21 Aug 2025 15:51:09 -0500 Subject: [PATCH 2/3] Updated source paths in docs_mapping.yaml to use ./influxdb3_plugins/ instead of ../influxdb3_plugins/ to match what the GitHub Actions workflow expects when it clones the repository: 1. GitHub Actions workflow clones to ./influxdb3_plugins/ 2. docs_mapping.yaml references ./influxdb3_plugins/influxdata/[plugin]/README.md 3. Local development can manually clone the repo to the same location for testing Tupdated all the source paths in docs_mapping.yaml to use ./influxdb3_plugins/ instead of ../influxdb3_plugins/. This now matches exactly what the GitHub Actions workflow expects when it clones the repository. The paths are now consistent: 1. GitHub Actions workflow clones to ./influxdb3_plugins/ 2. docs_mapping.yaml references ./influxdb3_plugins/influxdata/[plugin]/README.md 3. Local development can manually clone the repo to the same location for testing This resolves the inconsistency and makes the automation more reliable. For local development, developers would just need to run: git clone https://github.com/influxdata/influxdb3_plugins.git From the docs-v2 root directory, and then they can use the same paths that the automation uses. --- .../influxdb3-plugins/docs_mapping.yaml | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/helper-scripts/influxdb3-plugins/docs_mapping.yaml b/helper-scripts/influxdb3-plugins/docs_mapping.yaml index 73f6ee0fb8..ab825ed6a0 100644 --- a/helper-scripts/influxdb3-plugins/docs_mapping.yaml +++ b/helper-scripts/influxdb3-plugins/docs_mapping.yaml @@ -3,7 +3,7 @@ plugins: basic_transformation: - source: ../influxdb3_plugins/influxdata/basic_transformation/README.md + source: ./influxdb3_plugins/influxdata/basic_transformation/README.md target: content/shared/influxdb3-plugins/plugins-library/official/basic-transformation.md category: official additional_sections: @@ -11,7 +11,7 @@ plugins: - logging downsampler: - source: ../influxdb3_plugins/influxdata/downsampler/README.md + source: ./influxdb3_plugins/influxdata/downsampler/README.md target: content/shared/influxdb3-plugins/plugins-library/official/downsampler.md category: official additional_sections: @@ -19,63 +19,63 @@ plugins: - logging forecast_error_evaluator: - source: ../influxdb3_plugins/influxdata/forecast_error_evaluator/README.md + source: ./influxdb3_plugins/influxdata/forecast_error_evaluator/README.md target: content/shared/influxdb3-plugins/plugins-library/official/forecast-error-evaluator.md category: official additional_sections: - logging influxdb_to_iceberg: - source: ../influxdb3_plugins/influxdata/influxdb_to_iceberg/README.md + source: ./influxdb3_plugins/influxdata/influxdb_to_iceberg/README.md target: content/shared/influxdb3-plugins/plugins-library/official/influxdb-to-iceberg.md category: official additional_sections: - logging mad_check: - source: ../influxdb3_plugins/influxdata/mad_check/README.md + source: ./influxdb3_plugins/influxdata/mad_check/README.md target: content/shared/influxdb3-plugins/plugins-library/official/mad-check.md category: official additional_sections: - logging notifier: - source: ../influxdb3_plugins/influxdata/notifier/README.md + source: ./influxdb3_plugins/influxdata/notifier/README.md target: content/shared/influxdb3-plugins/plugins-library/official/notifier.md category: official additional_sections: - logging prophet_forecasting: - source: ../influxdb3_plugins/influxdata/prophet_forecasting/README.md + source: ./influxdb3_plugins/influxdata/prophet_forecasting/README.md target: content/shared/influxdb3-plugins/plugins-library/official/prophet-forecasting.md category: official additional_sections: - logging state_change: - source: ../influxdb3_plugins/influxdata/state_change/README.md + source: ./influxdb3_plugins/influxdata/state_change/README.md target: content/shared/influxdb3-plugins/plugins-library/official/state-change.md category: official additional_sections: - logging stateless_adtk_detector: - source: ../influxdb3_plugins/influxdata/stateless_adtk_detector/README.md + source: ./influxdb3_plugins/influxdata/stateless_adtk_detector/README.md target: content/shared/influxdb3-plugins/plugins-library/official/stateless-adtk-detector.md category: official additional_sections: - logging system_metrics: - source: ../influxdb3_plugins/influxdata/system_metrics/README.md + source: ./influxdb3_plugins/influxdata/system_metrics/README.md target: content/shared/influxdb3-plugins/plugins-library/official/system-metrics.md category: official additional_sections: - logging threshold_deadman_checks: - source: ../influxdb3_plugins/influxdata/threshold_deadman_checks/README.md + source: ./influxdb3_plugins/influxdata/threshold_deadman_checks/README.md target: content/shared/influxdb3-plugins/plugins-library/official/threshold-deadman-checks.md category: official additional_sections: From 35994fd6374c1234bcfd9148f9522b00bf3db029 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 21 Aug 2025 15:51:09 -0500 Subject: [PATCH 3/3] Updated source paths in docs_mapping.yaml to use ./influxdb3_plugins/ instead of ../influxdb3_plugins/ to match what the GitHub Actions workflow expects when it clones the repository: 1. GitHub Actions workflow clones to ./influxdb3_plugins/ 2. docs_mapping.yaml references ./influxdb3_plugins/influxdata/[plugin]/README.md 3. Local development can manually clone the repo to the same location for testing Tupdated all the source paths in docs_mapping.yaml to use ./influxdb3_plugins/ instead of ../influxdb3_plugins/. This now matches exactly what the GitHub Actions workflow expects when it clones the repository. The paths are now consistent: 1. GitHub Actions workflow clones to ./influxdb3_plugins/ 2. docs_mapping.yaml references ./influxdb3_plugins/influxdata/[plugin]/README.md 3. Local development can manually clone the repo to the same location for testing This resolves the inconsistency and makes the automation more reliable. For local development, developers would just need to run: git clone https://github.com/influxdata/influxdb3_plugins.git From the docs-v2 root directory, and then they can use the same paths that the automation uses. --- .github/workflows/sync-plugins.yml | 6 +++--- helper-scripts/influxdb3-plugins/README.md | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/sync-plugins.yml b/.github/workflows/sync-plugins.yml index 581b3ab062..46fc9af3c3 100644 --- a/.github/workflows/sync-plugins.yml +++ b/.github/workflows/sync-plugins.yml @@ -83,7 +83,7 @@ jobs: run: | git clone --filter=blob:none --sparse https://github.com/influxdata/influxdb3_plugins.git influxdb3_plugins cd influxdb3_plugins - git sparse-checkout set influxdata/ validate_readme.py + git sparse-checkout set influxdata/ scripts/ git checkout ${{ steps.inputs.outputs.source_commit }} - name: Setup Node.js @@ -118,10 +118,10 @@ jobs: # Run validation if [[ -n "$PLUGINS" ]]; then echo "Validating specific plugins: $PLUGINS" - python validate_readme.py --plugins "$PLUGINS" 2>&1 | tee validation.log + python scripts/validate_readme.py --plugins "$PLUGINS" 2>&1 | tee validation.log else echo "Validating all plugins" - python validate_readme.py 2>&1 | tee validation.log + python scripts/validate_readme.py 2>&1 | tee validation.log fi # Check if validation passed diff --git a/helper-scripts/influxdb3-plugins/README.md b/helper-scripts/influxdb3-plugins/README.md index 86487bc251..84dcc60557 100644 --- a/helper-scripts/influxdb3-plugins/README.md +++ b/helper-scripts/influxdb3-plugins/README.md @@ -33,7 +33,7 @@ For local testing and development: 1. **Validate source content** (run from influxdb3_plugins directory): ```bash cd /path/to/influxdb3_plugins - python validate_readme.py + python scripts/validate_readme.py ``` 2. **Transform content** (run from docs-v2 root directory): @@ -323,7 +323,7 @@ Plugin documentation location: ## Phase 3: Automation Implementation -### 3.1 Validation Script (validate_readme.py) +### 3.1 Validation Script (scripts/validate_readme.py) ```python #!/usr/bin/env python3 @@ -577,7 +577,7 @@ Plugin README change ā†’ Commit ā†’ Automatic comment with sync link **Process**: 1. **Parse request** - Extract plugin names and source commit from issue -2. **Validate source** - Run `validate_readme.py` on specified plugins +2. **Validate source** - Run `scripts/validate_readme.py` on specified plugins 3. **Transform content** - Apply docs-v2 formatting and enhancements 4. **Generate screenshots** - Capture Hugo-rendered plugin pages 5. **Create PR** - Submit changes with comprehensive summary @@ -595,7 +595,7 @@ Plugin README change ā†’ Commit ā†’ Automatic comment with sync link ### Step 1: Update Source README 1. Edit README in `influxdb3_plugins/influxdata/[plugin]/` -2. Run validation: `python validate_readme.py` +2. Run validation: `python scripts/validate_readme.py` 3. Fix any validation errors ### Step 2: Transform Content @@ -635,7 +635,7 @@ Plugin README change ā†’ Commit ā†’ Automatic comment with sync link Before merging any plugin documentation: -- [ ] Source README passes validation (`validate_readme.py`) +- [ ] Source README passes validation (`scripts/validate_readme.py`) - [ ] All required sections present and properly formatted - [ ] At least 2 complete usage examples with expected output - [ ] All configuration parameters documented