[App request] seqenv: Assign environment ontology (EnvO) terms to short DNA sequences

[pjmramond]

👤 Your Marbits Username

pramond

🖥️ App Name

seqenv

🔗 App URL

https://github.com/xapple/seqenv?tab=readme-ov-file

📦 App Version

Development

📄 Main Publication

https://peerj.com/articles/3827/

📝 Additional Notes

[franlat]

I didn't forget, I am on it. But this is a pain in the ass... it's too old, and some dependencies are incompatible. I am trying to install it in a conda environment, but it is not working properly. Also, it uses an old NCBI DB from 2015-2016, the one which uses the old nomenclature, in case you want to consider using it. It won't work with the newer versions.

[franlat]

Ok!

I think I managed a patch for this. I installed seqenv 1.3.0 in a conda environment with Python3 and newer versions of the packages. I had to play around with some specifics, but at least seqenv -h returns the help page. I haven't testede more, though.

I did not download the database because it's an old one, and currently we don't have a dedicated space for them in Marbits/Smart, at least until the new storage units arrive and are installed in the near future.

How to activate:

module load seqenv/1.3.0
conda activate seqenv-1.3.0
seqnev -h

According to the GitHub page, an old image of the DB can be downloaded here:
https://www.dropbox.com/s/fxqwc4uwi046av3/ncbi_nt_01_07_2015.fasta.gz?dl=0

If you test it, let me know if it works or not, or if you have any issues.

Feel free to close the issue if you think it has been solved.

[pjmramond]

Hey Fran thanks for trying out this install, in an interactive session it looks like the conda environment cannot find git

seqenv wnsn.fasta --search_db ncbi_nt_01_07_2015                                                                                               
/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/bin/seqenv:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('seqenv==1.3.0')
seqenv version 1.3.0 (pid 18110)
Codebase at: <module 'seqenv' from '/home/apps/seqenv/src/seqenv-py3/seqenv/__init__.py'>
Traceback (most recent call last):
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/bin/seqenv", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqenv", line 64, in <module>
    seqenv.Analysis(input_path, **kwargs).run()
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/analysis.py", line 152, in run
    if git_repo: print("The exact version of the codebase is: " + git_repo.short_hash)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/common/git.py", line 36, in short_hash
    sha1 = sh.git(self.default + ["rev-parse", "--short", "HEAD"])
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/lib/python3.8/site-packages/sh.py", line 3669, in __getattr__
    return self.__env[name]
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/lib/python3.8/site-packages/sh.py", line 3444, in __getitem__
    raise CommandNotFound(k)
sh.CommandNotFound: git

I guess that
conda install -c conda-forge git could do the trick.

[pjmramond]

I have also installed and formatted the NCBI DB and formatted it using the following commands:

wget https://www.dropbox.com/s/fxqwc4uwi046av3/ncbi_nt_01_07_2015.fasta.gz
gunzip ncbi_nt_01_07_2015.fasta.gz
makeblastdb -in ncbi_nt_01_07_2015.fasta -dbtype nucl -parse_seqids -out ncbi_nt_01_07_2015 

the files are there: /mnt/smart/scratch/emm1/users/pramond/DB/seqenv

[franlat]

I installed git in the environment, can you try again?

[pjmramond]

Hey Fran, thanks for the install, after trying it out again I get several error messages and it stops at step 3 (out of 6).

Some of the errors are here (truncated from my tmux session):

    self.run()
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/blast.py", line 80, in run
    if "BLAST Database error" in result.stderr:
AttributeError: 'str' object has no attribute 'stderr'
Exception in thread Thread-9:
Traceback (most recent call last):
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/blast.py", line 80, in run
    if "BLAST Database error" in result.stderr:
AttributeError: 'str' object has no attribute 'stderr'
Elapsed time: 6:36:14.471722
--> STEP 3: Filter out bad hits from the search results
Traceback (most recent call last):
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/bin/seqenv", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqenv", line 64, in <module>
    seqenv.Analysis(input_path, **kwargs).run()
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/analysis.py", line 154, in run
    self.outputs.make_all()
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/outputs.py", line 41, in make_all
    self.tsv_seq_to_concepts()
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/outputs.py", line 108, in tsv_seq_to_concepts
    content = self.df_seqs_concepts.to_csv(None, sep=self.sep, float_format=self.float_format)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/common/cache.py", line 35, in retrieve_from_cache
    else: result = f(self)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/outputs.py", line 55, in df_seqs_concepts
    df = pandas.DataFrame(self.a.seq_to_counts)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/common/cache.py", line 35, in retrieve_from_cache
    else: result = f(self)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/analysis.py", line 346, in seq_to_counts
    print(msg % (len(self.unique_gis), len(self.gis_with_envo)))
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/common/cache.py", line 35, in retrieve_from_cache
    else: result = f(self)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/analysis.py", line 295, in unique_gis
    return set(gi for gis in list(self.seq_to_gis.values()) for gi in gis)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/common/cache.py", line 35, in retrieve_from_cache
    else: result = f(self)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/analysis.py", line 279, in seq_to_gis
    self.search_results
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/analysis.py", line 256, in search_results
    self.search.filter()
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/__init__.py", line 99, in filter
    return self.query.filter(self.filtering)
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/blast.py", line 110, in filter
    with open(temp_path, 'w') as handle: handle.writelines(filter_lines(self.out_path))
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/blast.py", line 106, in filter_lines
    if coverage < threshold: continue
TypeError: '<' not supported between instances of 'str' and 'float'

ChatGPT says this:

👉 This is a known bug in Python 3 compatibility of old seqenv. The original code was written for Python 2.7, where the sh library behaved differently.

Workarounds:
• Patch the code in /home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/blast.py around line 80:
Replace:

if "BLAST Database error" in result.stderr:

with

if isinstance(result, str):
    if "BLAST Database error" in result:
        raise RuntimeError("BLAST Database error")
else:
    if "BLAST Database error" in result.stderr.decode():
        raise RuntimeError("BLAST Database error")

As well as this:

if coverage < threshold: continue

with

coverage = float(coverage)
if coverage < threshold:
    continue

[franlat]

Okay, I created a copy of the original just in case, but modified these lines. Let's hope it works now.

[pjmramond]

New error:

seqenv wnsn.fasta --search_db ncbi_nt_01_07_2015
/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/bin/seqenv:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('seqenv==1.3.0')
Traceback (most recent call last):
  File "/home/apps/miniconda/4.5.x/envs/seqenv-1.3.0/bin/seqenv", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqenv", line 8, in <module>
    import seqenv
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/__init__.py", line 28, in <module>
    from seqenv.analysis import Analysis
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/analysis.py", line 15, in <module>
    from seqenv.seqsearch.parallel import ParallelSeqSearch
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/__init__.py", line 5, in <module>
    from seqenv.seqsearch.blast import BLASTquery
  File "/home/apps/seqenv/src/seqenv-py3/seqenv/seqsearch/blast.py", line 81
    if "BLAST Database error" in result:
                                       ^
TabError: inconsistent use of tabs and spaces in indentation

I'm sorry for this being such a pain the a$$...