Skip to content

Implement global output publishing pattern for Nextflow v25.10+ (Issue #41)#69

Open
ppreshant wants to merge 1 commit intomainfrom
global_publish_i41
Open

Implement global output publishing pattern for Nextflow v25.10+ (Issue #41)#69
ppreshant wants to merge 1 commit intomainfrom
global_publish_i41

Conversation

@ppreshant
Copy link
Copy Markdown
Member

Overview

This PR implements the modern global output publishing pattern introduced in Nextflow v25.10+ to replace process-level publishDir directives with centralized workflow outputs.

Changes Made

Main Workflow (main.nf)

  • ✅ Added global output block before the workflow definition
  • ✅ Defined all pipeline output categories with organized directory structures
  • ✅ Added publish: section in main workflow to assign channels to outputs
  • ✅ Removed duplicate output block
  • ✅ Added workflow handlers (onComplete, onError) inside workflow block

Modules Updated (10 total)

Removed publishDir directives from all modules:

  • modules/local/bakta/bakta/main.nf
  • modules/local/checkm2/parse/main.nf
  • modules/local/singlem/appraise/main.nf
  • modules/local/singlem/pipe/main.nf
  • modules/local/taxburst/main.nf
  • modules/nf-core/checkm2/predict/main.nf
  • modules/nf-core/flye/main.nf
  • modules/nf-core/samtools/coverage/main.nf

Subworkflow Updated

  • workflows/somatem.nf - Removed publishDir directives

Output Organization

The new structure publishes outputs to organized directories:

📁 preprocessing/        - Clean reads by sample
📁 assembly/            - Assembly FASTA, GFA, logs by sample
📁 mapping/             - BAM files and coverage reports
📁 binning/             - Bin tables and FASTA files
📁 quality_assessment/  - CheckM2 reports
📁 annotation/          - Bakta GFF, TSV, TXT by sample/bin
📁 taxonomy/            - SingleM profiles and TaxBurst visualizations
📁 post_hoc_analysis/   - Pigeon and Appraise reports
📁 pipeline_info/       - Version tracking

Benefits

  • 🎯 Centralized control: All output paths defined in one location
  • 🔧 Easier maintenance: No scattered publishDir directives across modules
  • 📊 Better organization: Consistent directory structure
  • v25.10+ compatible: Follows modern Nextflow best practices
  • 🔮 Future-proof: Aligns with Nextflow's strategic direction

Testing Notes

⚠️ Note: Comprehensive linting (task 5) was skipped to avoid recursion limits. The pipeline should be tested to ensure:

  1. All outputs are correctly published to expected directories
  2. Meta maps are properly structured for path closures
  3. No conflicts with remaining publishDir directives (if any)

Related

Closes #41

Review Checklist

  • All publishDir directives removed from modules
  • Output paths are correctly structured
  • Pipeline runs successfully end-to-end
  • All expected outputs appear in correct directories
  • No duplicate or missing outputs

- Added global output block before workflow definition to define all pipeline outputs
- Configured publish paths for all major output categories:
  * Quality control (clean reads)
  * Assembly (FASTA, GFA, logs)
  * Mapping (BAM files, coverage)
  * Binning (tables, FASTA)
  * Quality assessment (CheckM2 reports)
  * Annotation (GFF, TSV, TXT)
  * Taxonomic profiling (profiles, visualizations)
  * Post-hoc analysis (Pigeon, Appraise)
  * Pipeline metadata (versions)
- Used publish: section in workflow to assign channels to output definitions
- Removed duplicate output block from end of file
- Follows Nextflow v25.10+ strict syntax requirements

This implements the modern output publishing pattern that replaces process-level publishDir directives with centralized workflow outputs.
bakta_txt = params.analysis_type == "assembly" ? ASSEMBLY_MAGS.out.bakta_txt : channel.empty() // channel: [ val(meta), path(*.txt) ]

// Taxonomic profiling outputs
singlem_profile = params.analysis_type == "assembly" ? ASSEMBLY_MAGS.out.singlem_profile : channel.empty() // channel: [ val(meta), path(*.tsv) ]
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to change params.analysis_type to taxonomic-profiling


// Mapping outputs
mapping {
path { meta, _bam -> "mapping/${meta.id}" }
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

read again why I chose the other syntax vs this one?

mapping {
        path { "mapping/" }
    }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collect important outputs from the whole workflow (with global publish)

1 participant