-
Notifications
You must be signed in to change notification settings - Fork 1
[WIP] New Release v2 #74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mr-eyes
wants to merge
522
commits into
master
Choose a base branch
from
v2
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
522 commits
Select commit
Hold shift + click to select a range
87781f3
Merge branch 'v2' of github.com:dib-lab/kProcessor into mo_v2
mr-eyes 680e676
fix bug in the test frame work and in phmap
shokrof 0162d7e
decrease the probility of collisons
shokrof 205f146
Merge branch 'v2' into mo_v2
shokrof ff01016
Merge pull request #82 from dib-lab/mo_v2
shokrof 94d7898
merge bug fix
shokrof 6e38144
cleaning the codebase
shokrof 50c02d2
upgrade the kmer decoder
shokrof e8abcc3
add option to num vectors and vectors size
shokrof dc9db4b
use int_vector instead of vlc
shokrof f03ab97
tests for inner join and sets functions is fixed and optimized
shokrof 74ee35e
small enhancements
shokrof a2734af
try enc_vector to see the compression
shokrof 1b3a50e
change inedingPriority queue to create color column instead of defaul…
shokrof a2ce15d
first step to make a tree iterator
shokrof 1d5f0bd
an attempt to make n level recursive colors
shokrof aec7928
bug fixes
shokrof 12ea978
testing prefix column and impelementing cache in query and perfermanc…
shokrof 6366f12
didnt push in the last commit
shokrof 214cc7a
sucess compression
9bc91e3
draft compress
shokrof 159358e
fix
shokrof 5c3b6c2
build errors
shokrof 8a0c254
refactor(build
shokrof 5c8a6ae
bug
cf6c718
merge
88039aa
bug fixes
ce202c0
bug fix
shokrof 4efdf6c
kp
56a7905
bug fix
shokrof 5f42ef2
bugs
shokrof 4d9bbfd
sss
f051c80
bug fix
shokrof 9a01196
bugs
ca45882
bug fixes and commenting merge indexes
shokrof b48d550
code is building but I commented the use case
75bb5e7
bugs
shokrof 05c9d1a
more bug fixes
90f9f32
remove static
shokrof 2b56aa4
frame
3a8cfd3
fix filter
shokrof e1c85ad
fix aggregate
shokrof 1112a0d
mote fixes
shokrof b5da096
more line
shokrof e9c6fca
add count column
shokrof de458fb
bug fix
shokrof ec85287
bug fix
6748349
try to fx buffered slow
shokrof a0bbe7b
Merge branch 'v2' of github.com:dib-lab/kProcessor into v2
shokrof d2a1bd4
fix bug in test
shokrof 05d3ee6
fix in test
ca0e30f
fix bug in set functions
shokrof 9f75944
fixing bugs in test case
shokrof cfffa0c
refactor complete and bugs are fixed
shokrof bc0150d
t
2c3a034
Merge remote-tracking branch 'refs/remotes/origin/v2' into v2
01208ea
fix index
shokrof 2bf90ae
optimize index
shokrof 3770f84
optimize index
shokrof 905ea72
update
shokrof aa1aa3d
update
shokrof a350743
kjjjkjk
shokrof 43332a8
create static mqf
shokrof fe5e01d
fix double hashing
shokrof 8a841f9
serial version of non sorted join
shokrof 164b047
closing buffered test and add profile to cmake
shokrof 80b90e8
ignore
shokrof bfeb149
wwwww
shokrof 8b75303
Merge branch 'v2' of github.com:dib-lab/kProcessor into v2
shokrof 3ea4c79
change the signature for insert to make it return the order of the in…
shokrof 815162e
fix in test case to actually use multi columns
shokrof 940f272
new test case for prefix trie
shokrof 731d7a4
fix bugs in prefix trie
shokrof 603cc4a
add function getColumnNames
shokrof e8e284c
testing blight
shokrof 4a9ac50
small bug
shokrof adeba2e
remove the random working directory. blight is not stable
shokrof d7733f2
this file should be commited before
shokrof 0cfab5c
using parallel hashmap not the normal flat map
shokrof 06afb31
TEST DEFUALT COLUMNS DEFAULT VALUES
shokrof ff0a6e3
parallel version of inner Join completed
shokrof a738c72
test inserting a column value to a missing kmer
shokrof f0f01c2
bug fix in MAP
shokrof 78e29b4
more documentation
shokrof 6ac98f1
fixing memory and speeding up parallel join
shokrof 22a6ee1
documentation
shokrof 1aab7f4
dbg test
shokrof 5a18eed
Update README.md
shokrof 17020f5
remove kmer and multicolumn test case is added
shokrof 5f4a43d
bug fix that may causes memory leak
shokrof cae4811
Comments for innerjoin and parallel join
shokrof b576191
remove parseFromSequence as it is a replcate. use countsFromFile
shokrof 41628a0
simplifying teh interface for deduplicated columns
shokrof 4366273
bug fix from loadFromKMC
shokrof 36848a6
Test For KMC
shokrof c120a3d
add Optimize step to index
shokrof 18bd935
update MQF
shokrof a8bab99
fix MQF to handle novel kmers
shokrof 34a1183
bug fixes
shokrof 9bd9c9c
bug fixes
shokrof 79d0d34
bug fixes
shokrof bc1606f
implemeting kdataframe clone
shokrof 883d695
transform function is reimplemented to consider multi columns and enh…
shokrof 4624081
tests to transform inplace
shokrof 4b85667
bug fixes in prefixtrie
shokrof 1551256
bug fixes i and improvement
shokrof 652c292
bug fixes i and improvement
shokrof 849c44b
test btree
shokrof a29a5ae
different optimizations
shokrof 26d44d4
save some memory
shokrof 0eee9df
more tests
shokrof 2cc2b23
more tests
shokrof 22d9197
update kmerdecode
shokrof 62284bc
decrease locking
shokrof e6a09be
bug fix
shokrof 3f0f0c4
remove the check for now
shokrof 320555e
bug fix
shokrof 173c20c
remove bubles in the ids
shokrof 98b65d7
bug fix
shokrof 9d2f317
first implementatiosn for preifx forest
shokrof 460a74b
more messages for debuging
shokrof 981b83f
using cerr
shokrof 04541c5
bug fixes
shokrof 914fb2c
bug fixes
shokrof c5d8292
bug fixes
shokrof 1c74c9d
bug fixes in creating forest
shokrof 6319c19
bug fixes in creating forest
shokrof 7e98a0d
making new tool to covert kmc to kdataframe
shokrof afa4500
new use case corticall
shokrof ab3236c
update downlaod scripts
2542c09
run cortex
433a623
add outFolder
3a75df3
add clean kprocessor to snakemake
shokrof 7e9a28a
update corticall workflow
bc42065
corticall workflow
b8f694e
load from KMC to BQMF for indexing
shokrof 70ff575
adjust memmory for corticall
eac0eb8
load from KMC to BQMF for indexing and more coticall fixes
shokrof d19823d
vuf fix in bmqf
shokrof 091584d
add indexing to snakemake
shokrof e5e00f0
create kframe from references
d40097c
Merge branch 'v2' of github.com:dib-lab/kProcessor into v2
10760f1
split Snakefile between kprocessor and corticall
shokrof b96bddb
add dbg utils files
shokrof a70c4be
first version of aligning reads but not working
shokrof 1929dc7
copy snakmake corticall too
shokrof 2699ae8
updates
399a80d
merge
dbda149
add tool to convert from diffrerent kdataframes types
shokrof 3ccbb96
fix bug in index snakemake
shokrof a17ae59
join all kdraframes into one
shokrof d911f59
detect novelkmer
shokrof 5b0378a
bug fixes
shokrof 6fa6857
Update README.md
shokrof fc3fc00
buf fixes. all testcases running now
shokrof 899e3b0
Merge remote-tracking branch 'origin/v2' into v2
shokrof e7eabb1
debug messages for creating prefix forest
shokrof 04ff6db
new use case for sv
shokrof 88b956a
use cases files
shokrof 8b5ed8f
adding threshold
850a9bb
bug fix in cleaninng
shokrof 51ea1e3
bug fix in cleaninng
shokrof 2c05920
sv pop
shokrof 5d965be
more messages
shokrof 347012c
bug
shokrof d5dfb67
bug
shokrof b6b252a
more messages
shokrof d832b35
more messages
shokrof 3b1cef5
changes
7159b94
Merge branch 'v2' of github.com:dib-lab/kProcessor into v2
3221f2e
create sv kframe
shokrof e92bcd8
add the src
shokrof b26c8c1
updates
f9034f5
extract reads
shokrof b4a4fbf
code
shokrof 7eb4214
extract reads
shokrof 996ba68
bugs
shokrof 3c793a3
bugs
shokrof 061bb0c
code update for prefix forest(failed) and corticall usecase
shokrof c4a86e7
make kPorcessor compile. prefix forest to be removed
shokrof 26646f1
make it complie
shokrof 0c04d8f
kDataframe stl is implemented and kDataframePHMAP / MAP are removed a…
shokrof e98e3f9
make it complie on the server
shokrof 88ff4fc
adding btree to kdatafarames
shokrof 0b5f911
adding sshash
shokrof 7d703bb
big refactor to make interface easier
shokrof 01afb2b
making seperate header file for each kDataframe and remove kDataframe…
shokrof 0a66d66
removing kmer row from the kDataframe and only using it for internal …
shokrof 33d9563
adding sshash working but no save and load
shokrof 5b879a9
fix bug
shokrof 9e3fe2b
fix bug
shokrof b8d085c
bug fix
shokrof fa79406
bug fix
shokrof 908c0f5
fix bug
shokrof a0d3626
update load app
shokrof a00db03
fix bug
shokrof 70e908e
fix bug and test for sshahs
shokrof b4d522a
add test files
shokrof 73b07e9
fix
shokrof 7a20061
simplify columns template
shokrof 871d320
memory optimization to indexing genomes
shokrof 8ff94b9
kdifferential expression use case is updated
shokrof 04ead8f
load to btree
shokrof 75d02d6
re-activate build CI
mr-eyes 6b13fa6
fix for sshash
shokrof 37a597b
Merge remote-tracking branch 'origin/v2' into v2
shokrof fafe35e
free memory
shokrof 0cad890
update kmerDecoder
mr-eyes 1da0037
update kmerDecoder and parallel_hashmap
mr-eyes d8a3569
fixed
mr-eyes 6f57b6f
add includes
mr-eyes cc186e3
Merge pull request #99 from dib-lab/mo_dev_v2
shokrof 879cb38
update sshash
shokrof 2f1dd75
upgrade sshash
shokrof 2504693
bug fix
shokrof 50838dd
bug fix
shokrof 35491c3
Update ubuntu.yml
shokrof dd2209f
Update ubuntu.yml
shokrof 61ba558
Update ubuntu.yml
shokrof 1e452da
Update ubuntu.yml
shokrof 54039ff
Update ubuntu.yml
shokrof a45df64
delete TBB not needed anymore
shokrof 62a02ce
remove hidden characters
shokrof 858cf91
remove hidden characters
shokrof aac0d31
Update ubuntu.yml
shokrof 774c563
bug
shokrof e626e60
Merge branch 'v2' of github.com:dib-lab/kProcessor into v2
shokrof a63ca76
test github ci
shokrof 9df2043
fix the bug in kProcessor release mode
shokrof d4b5906
adjust sshahs paramters
shokrof 9ce783c
bug fix
shokrof a39641f
improvements in kDifferential expression usecases
shokrof 362f73e
improvements
shokrof 695e91e
refactor to normalize the colors structures for all index functions
shokrof afeaadf
add function getColor and getColorID
shokrof a9bd4bd
usecase upset plot
shokrof 2e28956
fix MQF load from KMC and make it faster with less memory
shokrof d26fb77
bug fix
shokrof d2992b8
bug fix
shokrof 9be8b48
upset first version
shokrof 1c3f16c
bug fixes
4534341
upset bug fixes
shokrof 2e4ab3e
use mosue data instead
e8d2f12
bugs for the use cases
shokrof 28090d8
bug fix
shokrof 3cdd76a
bug fix
shokrof ba62744
bug fix
shokrof 9457666
mqf in use case
shokrof d5c0c11
not needed anymore
shokrof 5f3598a
upgrading MQF
shokrof File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
name: cibuildwheel | ||
|
||
on: | ||
push: | ||
branches: | ||
- master | ||
pull_request: | ||
branches: | ||
- master | ||
|
||
jobs: | ||
build_wheels: | ||
name: Build wheels on ${{ matrix.os }} | ||
runs-on: ${{ matrix.os }} | ||
strategy: | ||
matrix: | ||
os: [ ubuntu-latest ] # , windows-2019, macos-10.15 ] | ||
|
||
steps: | ||
- uses: actions/checkout@v2 | ||
with: | ||
submodules: recursive | ||
|
||
- name: Build wheels | ||
uses: joerick/[email protected] | ||
|
||
env: | ||
CIBW_BUILD_VERBOSITY: 3 | ||
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.6" | ||
CIBW_SKIP: pp* cp27-manylinux* *-manylinux_{aarch64,ppc64le,s390x} | ||
CIBW_ARCHS_LINUX: x86_64 | ||
CIBW_MANYLINUX_X86_64_IMAGE: quay.io/pypa/manylinux2014_x86_64:latest | ||
CIBW_BEFORE_BUILD: | | ||
yum install bzip2 -y | ||
pip install --upgrade pip | ||
pip install cmake | ||
yum install zlib-devel -y | ||
yum install centos-release-scl-rh -y | ||
yum install doxygen -y | ||
yum install devtoolset-7-toolchain -y | ||
yum groupinstall -y "development tools" | ||
yum install boost-devel -y | ||
yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel | ||
cmake -BKP_BUILD | ||
cmake --build KP_BUILD -j4 | ||
|
||
|
||
- uses: actions/upload-artifact@v2 | ||
with: | ||
path: ./wheelhouse/*.whl | ||
|
||
upload_pypi: | ||
needs: build_wheels | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v2 | ||
|
||
- uses: actions/download-artifact@v2 | ||
id: fetch_artifacts | ||
with: | ||
path: 'wheels/' | ||
|
||
- name: Publish package to TestPyPI | ||
uses: pypa/gh-action-pypi-publish@release/v1 | ||
with: | ||
user: __token__ | ||
password: ${{ secrets.TEST_PYPI_KPROCESSOR_API_KEY }} #PYPI_KPROCESSOR_API_KEY | ||
repository_url: https://test.pypi.org/legacy/ | ||
packages_dir: ${{steps.fetch_artifacts.outputs.download-path}}/artifact/ | ||
skip_existing: true | ||
verbose: true | ||
|
||
upload_bintray: | ||
needs: build_wheels | ||
runs-on: ubuntu-latest | ||
env: | ||
kProcessor_version: 2.0 | ||
|
||
steps: | ||
- uses: actions/checkout@v1 | ||
- name: Fetch wheels from artifacts | ||
id: fetch_artifacts | ||
uses: actions/download-artifact@v2 | ||
with: | ||
path: 'wheels/' | ||
|
||
|
||
- name: Bintray | ||
run: | | ||
for whl_file in ${{steps.fetch_artifacts.outputs.download-path}}/artifact/*whl; | ||
do | ||
curl -T $whl_file -umr-eyes:${{ secrets.BINTRAY_API_KEY }} -H "X-Bintray-Override:1" -H "X-Bintray-Version:${{env.kProcessor_version}}" -H "X-Bintray-Package:kProcessor" https://api.bintray.com/content/mr-eyes/wheels/ | ||
done | ||
curl -X POST -umr-eyes:${{ secrets.BINTRAY_API_KEY }} https://api.bintray.com/content/mr-eyes/wheels/kProcessor/${{env.kProcessor_version}}/publish |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
name: Ubuntu | ||
|
||
on: [push, pull_request] | ||
|
||
env: | ||
CTEST_OUTPUT_ON_FAILURE: 1 | ||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} | ||
CPM_SOURCE_CACHE: ${{ github.workspace }}/cpm_modules | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-18.04 | ||
|
||
steps: | ||
- uses: actions/checkout@v2 | ||
with: | ||
submodules: recursive | ||
|
||
- name: Set up GCC | ||
uses: egor-tensin/setup-gcc@v1 | ||
with: | ||
version: 9 | ||
platform: x64 | ||
- name: Setup cmake | ||
uses: jwlawson/[email protected] | ||
with: | ||
cmake-version: 'latest' | ||
- name: Use cmake | ||
run: cmake --version | ||
- name: Use g++ | ||
run: g++ --version && git status | ||
- name: install dependecies | ||
run: sudo apt-get install swig cmake python3-dev zlib1g-dev libghc-bzlib-dev python3-distutils libboost-all-dev | ||
|
||
# - name: configure | ||
# run: cmake -Stest -Bbuild -DENABLE_TEST_COVERAGE=1 -DCMAKE_BUILD_TYPE=Debug | ||
|
||
- name: configure | ||
run: cmake -Bbuild -DBUILD_ALL=1 | ||
|
||
- name: build | ||
run: cmake --build build -j1 | ||
|
||
- name: test | ||
run: cd build/tests/kProcessorLibTests/ && ./testKprocessorLib | ||
|
||
# - name: test | ||
# run: | | ||
# cd build | ||
# ctest --build-config Debug | ||
- name: collect code coverage | ||
run: bash <(curl -s https://codecov.io/bash) || echo "Codecov did not collect coverage reports" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd recommend separating this into a different workflow that only works if the build succeeds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And it's currently failing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agree! can you do it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. I will do it later. BTW, what was the difference between the old and new GCC/CMAKE installation commands?