DNAnexus module within stor by anujkumar93 · Pull Request #2 · anujkumar93/stor

anujkumar93 · 2018-09-05T22:05:27Z

Implement apis for DNAnexus inside stor. Also implement unit and integration testing for DNAnexus paths inside stor. Setup DNAnexus credentials for testing.

Motivation

As part of the epic to support DNAnexus paths within stor.

Implementation

Setup DX canonical, virtual and super classes for stor
Setup vcrpy and cassettes for testing, and login token in tox
The following api's are implemented as part of this PR, along with their tests:

abspath
normcase
normpath
realpath
expanduser
expandvars
dirname
basename
expand
join
split
splitext
list
listdir
exists
isabs
isdir
isfile
islink
ismount
getsize
walkfiles
stat
temp_url

jtratner

bunch of comments

jtratner · 2018-09-07T17:52:02Z

+NotFoundError = stor_exceptions.NotFoundError
+
+
+def _parse_dx_error(exc, **kwargs):


this looks unused now - should you remove it? It's also missing a second else clause when none of the cases match that would lead to a confusing error message

Yep, outdated and removed.

jtratner · 2018-09-07T17:53:16Z

+        wrapper.__doc__ = 'No-op for %r' % attr_name
+        return wrapper
+
+    abspath = _noop('abspath')


is this going to lead to bugs around dx://Proj:/parent/subdir/../anotherfile.txt - do we care about that behavior?

abspath is not used for DXPaths in code base, and seeing that OBS paths in general have abspath as a noop, it is upto the user to use abspath cautiously with DXPaths.

jtratner · 2018-09-07T17:55:52Z

+                    project=self.canonical_project
+                )[0]
+            else:
+                raise DXError('DX Projects cannot have a temporary download url')


looks pretty close to the right way to do this; however a couple points:

since the if clause always raise an error, you can check it first, raise the exception and dedent here, i.e.:

if not self.resource: raise DXError('Can only generate a download URL for individual files') try: return file_handler.get_download_url(...) except: pass

I think you want to check resource and not canonical resource before raising an error, no reason to go hit an API to canonicalize a project before raising this exception.

Agreed on both points, and changed.

jtratner · 2018-09-07T17:56:30Z

+                return file_handler.get_download_url(
+                    duration=lifetime,
+                    preauthenticated=True,
+                    filename=filename,


if you're already describing this, should filename just be what's returned from describe() if not present? what happens if filename is None here?

If filename is None here, the get_download_url defaults to the name of the source file, which is the behavior we want.

jtratner · 2018-09-07T17:58:22Z

+            else:
+                raise DXError('DX Projects cannot have a temporary download url')
+        except ValueError:
+            raise DXError('DXPaths ending in folders cannot have a temporary download url')


how would you hit this exception? wouldn't you know this because you can't generate a canonical resource for this method?

I'm thinking it'd be clearer to write something like:

if not self.resource: error canonical = self.canonicalize() # causes error if it's a folder already??

in the second case I think it's fine to just have an error that's just about data objects, etc

Agreed after discussion. Changed the method to reflect this.

jtratner · 2018-09-07T18:09:18Z

+            if canonicalize:
+                yield DXCanonicalPath('dx://{}:/{}'.format(obj['project'], obj['id']))
+            else:
+                dx_p = DXVirtualPath(self.drive + proj_name + ':' + obj['describe']['folder'])


I think this'd be clearer written as:

'{drive}{proj_name}:{folder}/{name}'.format(drive=drive, proj_name=proj_name, folder=obj['describe']['folder'], name=obj['describe']['name']) ``` which I believe does the same thing but avoids a level of indirection

Yep, changed here and other places where it would be more intuitive to write like this.

jtratner · 2018-09-07T18:11:10Z

+            pass
+        # otherwise we could be a directory, so try to grab first
+        # file/subfolder
+        if self._is_folder():


can you help me understand what the _is_folder check adds here? why not just try to list it directly and return False on the NotFoundError?

can you ensure there's an integration test that shows the following is True?

# empty dirs should be seen as existing folder_path = dx_p / 'myfolder' folder_path.makedirs_p() assert stor.exists(folder_path)

should be a documented diff between swift vs. s3 vs. posix

Swift & S3 - "empty" folder is exists = False
Posix & DX - "empty" folder is exists = True

Removed _is_folder

There is a unit test TestExists.test_true_empty_dir to check that empty dirs are seen as existing. Should I add an integration test as well?

jtratner · 2018-09-07T18:14:09Z

+        """
+        if not self.resource:
+            return None
+        if self._is_folder():


when I look here, it feels like this should just raise a NotFoundError that specifies no data object could be found for the given path.

Agreed and changed.

jtratner · 2018-09-07T18:14:35Z

+            'folder': self.resource.parent,
+            'project': self.canonical_project
+        }]
+        results = dxpy.resolve_data_objects(objects=objects)[0]


aside, should you pass in limit=2 here, just cuz we only care 0 or 1

Unfortunately resolve_data_objects doesn't have a limit argument, hence cannot pass it.

jtratner · 2018-09-07T18:16:41Z

+                                               .format(self.project))
+                return proj_dict['id']
+            except DXError as e:
+                raise DuplicateProjectError('Duplicate projects were found with given name ({})'


this seems like a dangerous check, because it's possible you'd swallow up a different error (e.g., about authentication or something else). can you list projects with a limit of 2 and then handle 0/1/2 projects with appropriate exceptions?

Changed DXError to DXSearchError which is only raised when there are multiple projects.
Since find_one_project does not have a limit argument, the only other way is to pass zero_ok=False, in which case DXSearchError is raised for 0/2 projects and there's no way to differentiate the two scenarios.

jtratner · 2018-09-07T19:11:43Z

+
+        Args:
+            canonicalize (boolean): whether to return canonicalized paths
+            only (str): "objects" for only objects, "folders" for only folder,


makes sense to keep because using it below

jtratner · 2018-09-07T19:12:31Z

+        """List the path as a dir, returning top-level directories and files.
+
+        Args:
+            canonicalize (boolean): whether to return canonicalized paths


canonicalize (bool, default False): if True, return canonical paths

Changed here and other places. Will include in next commit.

anujkumar93 · 2018-09-11T18:08:20Z

want to add a max limit on number of iterations? Maybe wait for 10s at most or something?

Yes, added a maximum limit of 15s, after which the test case fails

anujkumar93 · 2018-09-11T18:08:51Z

with above, seems ripe for generate_closed_file(<path>) method

Agreed. Made a 'setup_file' method that waits for the files to close before returning the handler.

Anuj Kumar added 16 commits August 15, 2018 15:39

initial test commit

20877bc

partial unit tests for DX - committed for feedback

d2b24d0

adding few more unit tests for dxpath.list

f35311e

adding tests for dxpath.walkfiles

fbfe959

adding encoding env variables to tox for pycharm bug

91440f1

adding tests for dxIDs and TestGlob

9f81fce

adding DXVirtualPath, DXCanonicalPath and stat methods

39bcb9f

Cleaned up and added tests for initializing DXPaths

26e0c7b

incorporating review comments

d3fe87a

simplifying find_dx_class and other review comments

f782c2a

temporary non-working commit to sync up on vcr

85fa13f

implementations of list, listdir and walkfiles

b011328

adding functionalities and tests for DX

f436ecf

adding environment variables in toc

6420870

Cleaning up code and adding list_iter etc: code review comments

9167353

implementing last review comment on glob/walkfiles

2b3520f

anujkumar93 changed the title ~~DNAnexus testing module within stor~~ DNAnexus module within stor Sep 5, 2018

anujkumar93 requested a review from pkaleta September 6, 2018 21:49

anujkumar93 self-assigned this Sep 6, 2018

jtratner reviewed Sep 7, 2018

View reviewed changes

anujkumar93 mentioned this pull request Sep 11, 2018

(WIP) Implement mutation api's #1

Open

pkaleta removed their request for review December 16, 2022 17:47

		NotFoundError = stor_exceptions.NotFoundError


		def _parse_dx_error(exc, **kwargs):

Conversation

anujkumar93 commented Sep 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Implementation

Uh oh!

jtratner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anujkumar93 Sep 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anujkumar93 Sep 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anujkumar93 commented Sep 11, 2018

Uh oh!

anujkumar93 commented Sep 11, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anujkumar93 commented Sep 5, 2018 •

edited

Loading

anujkumar93 Sep 11, 2018 •

edited

Loading

anujkumar93 Sep 11, 2018 •

edited

Loading