The amount of transcriptome data available

[tiantian-chen]

Thank you for the developers. I would like to ask a question. I have approximately 1,900 second-generation transcriptomes and 60 third-generation transcriptomes, as well as a 53G reference genome. If the running memory is supported, can I annotate all the transcriptome data together? Or should I split them for annotation? Or are there any other recommendations?

[murphyte]

The largest genome we've annotated with our internal EGAP pipeline is 40G (lungfish), and I think the largest we've tested with EGAPx is ~6G or thereabouts. So I would try it first with a small amount of RNAseq. You'll likely also need to relax many timeouts to allow for long running processes. Also, GenBank doesn't support chromosome lengths over ~2.1 Gbp, so if you have any chromosomes exceeding that length they need to be split. Here's an example:
https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_019279795.1/

We also saw problems with earlier large genomes and excessive problems with frameshifting indels. More recent ones using HiFi or ONT have done better, but if you've used any earlier technologies the sequence quality can be an issue. A small run may expose that if it shows a high proportion of "low quality" protein-coding genes.

For the amount of transcriptome evidence, the benefit plateaus quite quickly after ~5B reads, and for RefSeq EGAP runs we rarely go over 10B reads (5B paired-end spots). If by 1900 second-generation transcriptomes you mean 1900 Illumina (or similar) runs, you're probably exceeding that by 10x, so I would recommend picking and choosing. For the 3rd generation data, note we're sorting out some issue that occur with certain types of read ids, and you might run into that same problem. We hope to have a new release with that fixed later this month. For the volume, while with our internal EGAP pipeline we've used as many as 1B IsoSeq reads, I'd recommend staying under 100M reads for now with EGAPx. I don't think we've tested anything quite that high, but I think that should work. We have some additional improvements in the works that will help with using higher amounts, but they're not ready yet.

[tiantian-chen]

First, I used two second-generation transcriptomes and two third-generation transcriptomes to perform annotation tests on the genome of 5G3. I encountered the following error. Is this a problem with my memory or with egapx:gnomon_plane:chainer:run_chainer?

executor > local (715) [0e/efc7cb] ega…tup_genome:get_genome_info | 1 of 1 ✔ [24/9bc8a8] ega…_proteins:convert_proteins | 1 of 1 ✔ [11/3e666c] ega…ne:miniprot:split_proteins | 1 of 1 ✔ [f2/c41f34] ega…miniprot:run_miniprot (11) | 27 of 27, retries: 4 ✔ [14/5bfed0] ega…e:paf2asn:run_paf2asn (27) | 27 of 27 ✔ [e4/020075] ega…prot:run_best_aligned_prot | 1 of 1 ✔ [b4/8f899d] ega…ter_sa:run_align_filter_sa | 1 of 1 ✔ [fa/2a0272] ega…ign_sort_sa:run_align_sort | 1 of 1 ✔ [7d/3df7ee] ega…ane:star_index:build_index | 1 of 1 ✔ [70/bb60e2] ega…rt_plane:star:run_star (1) | 2 of 2 ✔ [06/fadb69] ega…seq_divide_by_strandedness | 1 of 1 ✔ [75/21df29] ega…d_sort:calc_assembly_sizes | 1 of 1 ✔ [1b/7889fe] ega…m_bin_and_sort:bam_bin (2) | 2 of 2 ✔ [0d/32051f] ega…bin_and_sort:merge_prepare | 1 of 1 ✔ [7a/586535] ega…am_bin_and_sort:merge (32) | 53 of 53 ✔ [ad/94771e] ega…plane:bam2asn:convert (52) | 53 of 53 ✔ [8e/2f6106] ega…seq_collapse:generate_jobs | 1 of 1 ✔ [c3/0c8f57] ega…se:run_rnaseq_collapse (2) | 10 of 10, retries: 10 ✔ [d2/d3c3ef] ega…lapse:run_gpx_make_outputs | 1 of 1 ✔ [d7/617700] ega…2_index:run_minimap2_index | 1 of 1 ✔ [3a/90946b] ega…e:minimap2:gpx_qsubmit (5) | 5 of 5 ✔ [69/308999] ega…imap2:minimap2_wnode (233) | 310 of 310 ✔ [27/a08b13] ega…g_plane:minimap2:gpx_qdump | 1 of 1 ✔ [2c/4fc978] ega…ter_est_align:align_filter | 1 of 1 ✔ [8c/762581] ega…ilter_est_align:align_sort | 1 of 1 ✔ [66/07bb3f] ega…sk_stats:run_winmask_stats | 0 of 1, retries: 3 [72/6f5a5a] ega…:winmask_wnode:gpx_qsubmit | 1 of 1 ✔ [44/db0513] ega…pare_masks:gp_annot_format | 1 of 1 ✔ [fa/5412ea] ega…re_masks:convert_rfam_mask | 1 of 1 ✔ [61/250990] ega…chainer:run_align_sort (1) | 1 of 1 ✔ [12/161604] ega…lane:chainer:generate_jobs | 1 of 1 ✔ [00/395c48] ega…ne:chainer:run_chainer (8) | 0 of 16, retries: 169 [11/1ce428] ega…plane:fetch_swiss_prot_asn | 1 of 1 ✔ [05/7a3464] ega…n_plane:get_swiss_prot_ids | 1 of 1 ✔ [0e/5293ca] ega…builder:annot_builder_main | 1 of 1 ✔ [16/efb39d] ega…_plane:print_fake_lxr_data | 1 of 1 ✔ [d2/ea2d66] ega…:fetch_ortholog_references | 1 of 1 ✔ [11/5bc701] ega…ext_genome:get_genome_info | 1 of 1 ✔ [9e/e885d4] ega…_proteins:convert_proteins | 1 of 1 ✔ [48/b3102d] ega…ogy_plane:get_prot_ref_ids | 1 of 1 ✔ Plus 36 more processes waiting for tasks… [8c/9b9fe4] NOTE: Process egapx:gnomon_plane:chainer:run_chainer (3)terminated with an error exit status (139) -- Execution is retried (13) [71/c39bbe] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (16)terminated with an error exit status (2) -- Execution is retried (12) [c3/13fe6e] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (10)failed -- Execution is retried (12) [27/b4b82c] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (9)failed -- Execution is retried (13) [eb/ec71fb] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (5)failed -- Execution is retried (11) [06/4c890b] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (8)failed -- Execution is retried (12) [57/33b9fc] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (1)failed -- Execution is retried (11) [f2/e7eaeb] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (3)failed -- Execution is retried (14) [bd/7545d5] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (16)terminated with an error exit status (2) -- Execution is retried (13) [6b/a9c8a4] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (10)failed -- Execution is retried (13) [f2/99c67a] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (9)failed -- Execution is retried (14) [19/ac32b4] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (5)failed -- Execution is retried (12) [00/395c48] NOTE: Processegapx:gnomon_plane:chainer:run_chainer (8)` failed -- Execution is retried (13)

There is insufficient memory for the Java Runtime Environment to continue.

Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.

An error report file with more information is saved as:

/share/org/YZWL/yzwl_chentt/soft/egapx_0.4/egapx_n/hs_err_pid1724059.log

Compiler replay data is saved as:

/share/org/YZWL/yzwl_chentt/soft/egapx_0.4/egapx_n/replay_pid1724059.log

!!WARNING!!
This is an alpha release with limited features and organism scope to collect initial feedback on execution. Outputs are not yet complete and not intended for production use.

eutil connection failure
None
To resume execution, run: sh /share/org/YZWL/yzwl_chentt/soft/egapx_0.4/egapx_n/egapx_out_merge_fan53te/nextflow/resume.sh`

[murphyte]

We're not sure what's causing that.

Please provide the following:

1. your input config.yaml
2. how many sequences are in your genome? How long is the biggest sequence?
3. the chainer logs under/share/org/YZWL/yzwl_chentt/soft/egapx_0.4/egapx_n/egapx_out_merge_fan53te/nextflow/execution_logs/gnomon_plane/chainer/
4. the java log at:
   /share/org/YZWL/yzwl_chentt/soft/egapx_0.4/egapx_n/hs_err_pid1724059.log

[tiantian-chen]

`cat fan53te.yaml
genome: /share/org/YZWL/yzwl_chentt/Public/analysis/wheat/analysis/fangenome_anno/IAAS_S1_S2_S3_All_NLR/final_anno/IAAS_S1_S2_S3_All_NLR_chrall.fa
taxid: 4565
locus_tag_prefix: PanRef
long_reads:

• /share/org/YZWL/yzwl_chentt/Public/data/new_tra3_20250622/wheat/F25A040000599_TRImdcnT_0619/annols/renamed_fasta/new/relength/XN3517AlimapolyA.fasta
• /share/org/YZWL/yzwl_chentt/Public/data/new_tra3_20250622/wheat/F25A040000599_TRImdcnT_0619/annols/renamed_fasta/new/relength/XN3517_ckAlimapolyA.fasta
• /share/org/YZWL/yzwl_chentt/Public/data/new_tra3_20250622/wheat/F25A040000599_TRImdcnT_0619/annols/renamed_fasta/new/relength/Z8425BAlimapolyA.fasta
• /share/org/YZWL/yzwl_chentt/Public/data/new_tra3_20250622/wheat/F25A040000599_TRImdcnT_0619/annols/renamed_fasta/new/relength/Z8425B_ckAlimapolyA.fasta
• /share/org/YZWL/yzwl_chentt/Public/data/new_tra3_20250622/wheat/cau_t2t_rna_seq/modifiedSRR28004290.fa
  short_reads: /share/org/YZWL/yzwl_chentt/Public/analysis/wheat/analysis/fangenome_anno/IAAS_S1_S2_S3_All_NLR/final_anno/tra2_te.txt`

Below are my chromosome information and some log information.

nextflow.log
replay_pid1724059.log

IAAS_S1_S2_S3_All_NLR_chrall.length.txt

[murphyte]

Do you still have disk space available?

[victzh]

I see in the logs you've uploaded that Nextflow reports error=12, Cannot allocate memory. How much RAM do your computers have?