BUG: Treats local read files as SRR, if it supplied as path to read list.

[Alteroldis]

v0.4.1

Traceback (most recent call last):                                                                            
  File "/home/Tools/egapx/ui/egapx.py", line 1983, in <module>                                                
    sys.exit(main(sys.argv))                                                                                  
             ^^^^^^^^^^^^^^                                                                                   
  File "/home/Tools/egapx/ui/egapx.py", line 1882, in main                                                    
    res, reads_msg = prepare_reads(run_inputs, args.force)                                                    
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                    
  File "/home/Tools/egapx/ui/egapx.py", line 331, in prepare_reads                                            
    save_sra_metadata_file(outfile_path, sras, nosra)                                                         
  File "/home/Tools/egapx/ui/egapx.py", line 1550, in save_sra_metadata_file                                  
    metadata = get_sra_metadata(sras).values()                                                                
               ^^^^^^^^^^^^^^^^^^^^^^                                                                         
  File "/home/Tools/egapx/ui/egapx.py", line 1525, in get_sra_metadata                                        
    metadata = sra_query(" OR ".join(runs), runs)                                                             
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                             
  File "/home/Tools/egapx/ui/egapx.py", line 1541, in sra_query
    sra_uids = sra_uids_query(query)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/Tools/egapx/ui/egapx.py", line 1398, in sra_uids_query
    esearch = urlopen(esearch_url+quote(sra_potential_query.encode('utf-8') ))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 414: Request-URI Too Long

[etvedte]

if it supplied as path to read list

Please provide the YAML formatting you are using to supply local reads.

[Alteroldis]

Then part of README should be changed:

If you have a large number of local RNA-seq data, you can list them in a file with a set name and a filepath in each line (see examples/input_D_farinae_small_reads.txt). Then you can read that file from the input yaml (see examples/input_D_farinae_small_readlist.yaml).

[etvedte]

No this was a question, not a request to change your format, sorry for the miscommunication. I wanted you to share how you formatted your input YAML.

I see now what you are saying though. I am testing this out...where are you seeing this error? Are you using SRA files downloaded locally, or your own data?

[Alteroldis]

Oh, sorry me too.

input.yaml

genome: /full/path/to/genome/hap1upcmt.fa
short_reads: readlist
taxid: 1774088
locus_tag_prefix: egapxtmp
proteins: /full/path/to/prots/3specorstrongmatch.echino.approved.egap7586.faa.gz
proteins_trusted: /full/path/to/egapx_local_cache/target_proteins/2/7586.trusted_proteins.gi
tasks:
  rnaseq_collapse:
    rnaseq_collapse: -high-identity 0.85
  convert_from_bam:
    sam2asn: -filter 'pct_identity_gap >= 90'
  star_wnode:
    star_wnode: -pct-identity 90
  star_index:
    STAR: --runThreadN 24
  miniprot:
    miniprot: -t 48

example of readlist

unpaired1.1 /full/path/to/unpair/unpaired1.R1.fq.gz
unpaired1.2 /full/path/to/unpair/unpaired1.R2.fq.gz
paired1.1 /full/path/to/unpair/paired1.rep1.R1.fq.gz
paired1.1 /full/path/to/unpair/paired1.rep1.R2.fq.gz
paired1.2 /full/path/to/unpair/paired1.rep2.R1.fq.gz
paired1.2 /full/path/to/unpair/paired1.rep2.R2.fq.gz

Issue is in the prepare_reads_by_type function: if I supply a file with a list of local, not SRR, files to the 'short_reads' tag, the function should return it inside of the nosra1 variable (beginning of the prepare_reads func) and maybe flush the sra1 var, but instead only sra1 is not an empty list. In v0.4.0 this error does not appear because save_sra_metadata_file function has the exception handling.

[jdamas13]

I'm finding the same issue.
Is there a work around or ETA for a fix?

[pstrope]

Hi,
Can you try editing the readlist 1st column as below and give it a try?

unpaired1 /full/path/to/unpair/unpaired1.R1.fq.gz
unpaired1 /full/path/to/unpair/unpaired1.R2.fq.gz
paired1 /full/path/to/unpair/paired1.rep1.R1.fq.gz
paired1 /full/path/to/unpair/paired1.rep1.R2.fq.gz
paired2 /full/path/to/unpair/paired1.rep2.R1.fq.gz
paired2 /full/path/to/unpair/paired1.rep2.R2.fq.gz

[jdamas13]

I'm using readlists as below, and I'm still getting the same error.
Do the read files need to have the R1/R2 notation? Or the ID to terminate in numbers?

Thank you!

eye /home/jdamas/data/gene-annotation/egapx/short-read-RNA/eye_1.fq.gz
eye /home/jdamas/data/gene-annotation/egapx/short-read-RNA/eye_2.fq.gz
liver /home/jdamas/data/gene-annotation/egapx/short-read-RNA/liver_1.fq.gz
liver /home/jdamas/data/gene-annotation/egapx/short-read-RNA/liver_2.fq.gz
ovary /home/jdamas/data/gene-annotation/egapx/short-read-RNA/ovary_1.fq.gz
ovary /home/jdamas/data/gene-annotation/egapx/short-read-RNA/ovary_2.fq.gz
testis /home/jdamas/data/gene-annotation/egapx/short-read-RNA/testis_1.fq.gz
testis /home/jdamas/data/gene-annotation/egapx/short-read-RNA/testis_2.fq.gz

[pstrope]

@jdamas13 Can you please post a few header lines of your fq files (eye_1 and eye_2)?

[jdamas13]

Here it goes.

@A01204:148:HT7M5DRX2:1:2101:1452:1000 1:N:0:TGCTACTG+CCTCCGGT
NGGGAGGTATAGCCAGGAGTGGGGTAGAAGGGAAGGCCGGCGTGTAGACAGGCTGAGGAGGGATTGCCCCTTCAAAGTCCTGAAACTA
+
#FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF,FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
@A01204:148:HT7M5DRX2:1:2101:1687:1000 1:N:0:TGCTACTG+CCTCCGGT
NGGATTTTATTAACTTGATATGCTTTTGCAGAAAAGCATTGACATGTACAAGTTGGACATCCGGGAAGAGATTTGTATACACTGTTTTGCACACAGCAATTTTTTTTTAAACTCTGGGTCAGACAAAATGTTCATTTCAATAGGACTTCCA
+
#FFF,FFFFFFFFFFFFFFFF:FFFFF:FFFFFFFFFFFFF:FFFFFFFFFFFF:FFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF,FFFFFFFFFFFFFFFFF:FFFFFFFFFFF:,FF:FFFFFFFF:FFFFFFF:F
@A01204:148:HT7M5DRX2:1:2101:1723:1000 1:N:0:TGCTACTG+CCTCCGGT
NATTTGGATTTGGCTTGTTATAGTTAGGTATATACACTTGCTTTAATTTGCTTTCTGCTTCAGCTGCCTTCAACCGCTTCTGTTGAACGTATCTGTCAGTTGTTTTCCACATGTAATAGTATTCAATGATGCTAGTTAAGGACTTCCATGG

@A01204:148:HT7M5DRX2:1:2101:1452:1000 2:N:0:TGCTACTG+CCTCCGGT
TAGTTTCAGGACTTTGAAGGGGCAATCCCTCCTCAGCCTGTCTACACGCCGGCCTTCCCTTCTACCCCACTCCTGGCTATACCTCCCA
+
FFFFFFFFFFFFFFFFFFFFFFF::FFFFFFFFFFFFFFFFFFFFFFFFFFFF,:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
@A01204:148:HT7M5DRX2:1:2101:1687:1000 2:N:0:TGCTACTG+CCTCCGGT
TGTTAGCTTGGACTAGAATTGGATTTACTAAGCCCCTAGTTCAGTATCTTTTCCTAATATAAATGAAGGGAGATTAAAAGCATACAGCAGAGTGCTCTGTCCAGTTCTGTCTGATTTCATTTTTGAATGTAGTGTACCAATTTTTTTCGTG
+
FF,FFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFF:FFF:FFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF,FFFFFFFFFFFFFFFF:F:FFFFFF,FF:FFFFFFFFFFFFFFFFFFFFFFFFFFF,FF::FF
@A01204:148:HT7M5DRX2:1:2101:1723:1000 2:N:0:TGCTACTG+CCTCCGGT
TGCAAGATTTTCTTCCATGGAAGTCCTTAACTAGCATCATTGAATACTATTACATGTGGAAAACAACTGACAGATACGTTCAACAGAAGCGGTTGAAGGCAGCTGAAGCAGAAAGCAAATTAAAGCAAGTGTATATACCTAACTATAACAA

[jdamas13]

Any updates on this?
Thank you!

[Alteroldis]

I only add below lines after line 550 in ui/egapx.py:

                else:
                    for vi in v:
                        sra_to_return.remove( vi )
                    paired = 'paired' if len(v) == 2 else 'unpaired'
                    out_rec = [run_name, 'NA', paired, '2', '2', 'NA', 'NA', 'SAMN_'+run_name, 'NA', 'NA', 'NA', 'NA', '0']
                    not_sra.append(out_rec)

But it is crutch. I work only with local read files listed inside readlist.

[pstrope]

Hi @jdamas13, we haven't had a chance to investigate on this bug. We'll make a ticket and look into it. Until we do so, can you try @Alteroldis 's fix as a temporary workaround?

[jdamas13]

I will do that! Thank you so much for your quick replies @Alteroldis & @pstrope !

[jdamas13]

@Alteroldis, would you mind providing your edited ui/egapx.py file to ensure I'm making the right edits?
Thanks!