Set rm_files to be a synchronous method #503
Conversation
adlfs/spec.py
Outdated
| @@ -1248,7 +1248,7 @@ async def _rm_files( | |||
| for file in file_paths: | |||
| self.invalidate_cache(self._parent(file)) | |||
|
|
|||
| sync_wrapper(_rm_files) | |||
| rm_files = sync_wrapper(_rm_files) | |||
There was a problem hiding this comment.
Ah this is something that I missed as well when we went over the original GitHub feature request... Based on the fsspec specification there is no rm_files() shared interface; there's only an rm_file(). The original GitHub issue: #497 is also only requesting for rm_file().
So, it is not appropriate to be setting a sync wrapper like this because it does not appear rm_files to be a shared interface across file systems. Instead, it would probably make sense to add an async _rm_file that is a simplified wrapper over the _rm implementation to implement the feature request.
Even more interesting, it seems there used to be a _rm_file() implementation prior to this PR: #383 and because the underlying AsyncFileSystem mirrors methods, I suspect that adlfs might have actually at one point supported rm_file() and could be a regression. It would be great to confirm if adlfs ever supported rm_file() in a version prior to that PR.
There was a problem hiding this comment.
This was the previous implementation of rm_file I found, but it looks like it was only ever used by rm and was not callable.
anjaliratnam-msft
force-pushed
the
rm_files_fix
branch
from
4af870c to
0c62e00
Compare
adlfs/spec.py
Outdated
| if p != "": | ||
| await self._rm_files(container_name, [p.rstrip(delimiter)]) | ||
| else: | ||
| await self._rmdir(container_name) |
There was a problem hiding this comment.
So looking more at rm_file it seems like it's sole purpose is to just delete a single file and does not support deleting a directory provided. This seems to be the intention based on other implementation's versions of rm_file as well:
s3fs- Only deletes a single object which would be a blob in adlfslocal- Usesos.remove()which only handles removing files and not directories.
I think it would make sense to stick with this contract to be consistent, especially if rm_file was not actually never exposed publicly.
adlfs/spec.py
Outdated
| @@ -1278,6 +1278,31 @@ async def _rm_files( | |||
|
|
|||
| sync_wrapper(_rm_files) | |||
|
|
|||
| async def _rm_file( | |||
| self, path: typing.Union[str, typing.List[str]], delimiter: str = "/", **kwargs | |||
There was a problem hiding this comment.
And on a similar note, I'm not sure if we should be exposing a delimiter for this method since there is not really any recursive nature to this public contract and instead always just use / if we need any splitting logic. Also not supporting a delimiter seems consistent with the other implementations I linked.
This addresses the github issue where rm_files is not implemented. This was a simple fix where sync_wrapper(_rm_files) just needed to be set to rm_files. Tests were also added to make sure it works as expected.