Folder uploads

[dobesv]

Currently if you drag/drop a folder onto the file upload area it is sent to the server as a file containing who-knows-what. This isn't really ideal - it would be better to report an error, or where supported (currently only Chrome) give the application the chance to walk the file tree and upload the files inside the folder.

For a bit of discussion on this, see

• http://stackoverflow.com/questions/8856628/detecting-folders-directories-in-javascript-filelist-objects
• http://hs2n.wordpress.com/2012/08/13/detecting-folders-in-html-drop-area/

[dobesv]

Here's an example of how to test for a folder / non-file upload; it's in coffeescript:

    uploader.bind 'afteraddingfile', (evt, item) ->
        continue_upload = -> ... code calls item.upload ...

        if uploader.isHTML5
            f = item.file
            # Assume it is really a normal file, if it is large
            if f.size > 1048576
                continue_upload()
            else
                # Smaller-looking file, might be a folder; the only reliable way to tell is to actually try to read it, which will error out for a folder
                skip_invalid_file_or_folder = ->
                    item.cancel()
                    alert(f.name+": Couldn't upload this file; if this is a folder try putting it into a zip file or upload its contents one at a time.")


                reader = new FileReader()
                reader.onloadstart = ->
                    # Avoid reading the whole file, we just want to see if we can open it
                    continue_upload()
                    reader.onerror = null
                    reader.abort()
                reader.onload = continue_upload
                reader.onerror = skip_invalid_file_or_folder
                reader.readAsArrayBuffer(f)
        else
            continue_upload()

[darynmitchell]

Note that drag/drop folder does not always result in an upload.
The 'folder' filter is enabled by default, and my testing of drag/drop a folder in IE11 correctly does not add an item to the upload queue.
The problem occurs in FF and Chrome (Windows 7), and only when I drop some folders, the folder filter accepts it and causes an unexpected upload, while for other folders the filter correctly blocks it.

To grab some info out of on @dobesv's first StackOverflow link, it's because folder file-like items can have a non-zero size, causing the current implementation of _folderFilter function to says it's a file.

The item passed to _folderFilter(item)_ looks like:

lastModifiedDate: Tue Jan 06 2015 ⋯
name: "UploadTest"
size: 12288
type: ""
__proto__: FileLikeObject

The SO discussion says that folder size is dependent on OS implementation. Many reports of Windows & Linux x64 having folder sizes be multiples of 4096, with size dependent on how many files were ever in the folder; but one commenter says that OS X will have different folder sizes not a multiple of 4096.

[darynmitchell]

Also a note on one part of @dobesv code above, the (f.size > 1048576) optimization in some cases will be problematic, since folder size can actually be larger than that. For a folder with a large number of files, I checked the size of my temp folder (Windows 7), and it's 2883584 bytes, so it would pass that condition but in reality needs to be checked by the FileReader approach.

[ronanww]

@darynmitchell So there is no way to track if the uploaded item is a folder or not? The folder filter doesn't work well

[darynmitchell]

There are browser-specific non-standard ways that I think would offer 100% confidence. Would required changing angular-file-upload to provide access at the time of drop. I didn't dig into the possibilities because I needed it cross browser, and found a good-enough workaround that didn't take a lot of effort.

The workaround isn't the prettiest, but it worked for me, so here you go. It's a variation on @dobesv's code above and entirely indebted to it for setting me in the right direction.

Notes:

1. I decided to put the check only when upload failed, since 99% of the time I figure the drop is a file not a folder, so why slow down the 99% to perform the check?
2. @dobesv's reader.onloadstart was desirable because you don't want to read the whole file, but it didn't work on Mac Safari — Safari calls it even when the item is a folder.
3. So instead we have to use onload, but then we're reading the whole file. And for very large files, that caused Firefox to crash(!)
4. Therefore, don't read the whole file, read just one byte. If it's a folder, that's enough to cause it to fail and not reach onload. That's the fileItem._file.slice(0, 1);
5. Need to wrap in try/catch because FF throws error, whereas webkit goes to reader.onerror

Okay, where are we at? When we can't read the item, it could be a folder, or it could be a file that is locked and therefore unreadable. How do we know which? This is where it gets a bit hacky. I made two different error messages to show the user.

One is the current not-totally-confident knowledge we have at this point: "Selected item is a folder or there was an error reading the file"
But almost all my users are on Windows not Mac. I confirmed that on Mac the "size" of a folder varies very widely. But and as mentioned above, on Windows, folder size is always multiples of 4096, presumably sector size. The hackyness is that this is not a promise, but in real life it worked very well for me. So if we get the "can't read dropped item" error AND the size is a multiple of 4096, then I feel confident showing the user an error message saying "You can't drop folders." Yes, if someone tries to drop a file that is both locked and 4096 bytes I'll show them the wrong error message, but I'm confident that will "never" happen, not enough for me to not do this.

Enough talking, here's the code:

function checkIfFolderThenHandleError(fileItem, response, status, headers) {
    // Seems to be no easy way to know it's a folder (no current web standard).
    // https://github.com/nervgh/angular-file-upload/issues/273 Offers this as a way to do it
    // Differences:
    // Use Blob.slice() to not read the whole file, which caused FF to crash (reported bug in FF).
    // "onload" is okay because we're only trying to read a "slice" (otherwise I'd use onprogress)
    // (Can't use "onloadstart" because it gets called on Safari/Mac OS X for folders - didn't in Windows).
    var reader = new $window.FileReader();
    reader.onload = function(e) {
        // It's a file, since directories cannot be opened to get any progress like this
        onDeterminedIsFile();
    };
    reader.onerror = function(e, anymore) {
        $log.warn('Unable to read file "' + fileItem.file.name + '", or requested file is a folder', reader.error);
        onDeterminedIsFolderOrUnreadableFile();
    };

    $log.debug('FileReader "' + fileItem.file.name + '": checking if is (readable) file');
    try {
        // Don't try to read the whole file, as FF bug crashes for large files. Use slice() to read just 1 byte
        var firstByteBlob;
        firstByteBlob = fileItem._file.slice(0, 1);
        reader.readAsArrayBuffer(firstByteBlob);
    }
    catch (e) {
        // For folder/unreadable file, FF throws error on readAsArrayBuffer(), vs. webkit goes to reader.onerror
        $log.error("FileReader error reading file: ", e);
        onDeterminedIsFolderOrUnreadableFile();
    }

    function onDeterminedIsFile() {
        var isReadableFile = true;
        onErrorItem(fileItem, response, status, headers, isReadableFile);
        // (abort needs to come at the end for FF; seems the onerror call stops in FF when abort called)
        reader.onerror = null;
        reader.abort(); // Stop reader (and ensure no more progress)
    }

    function onDeterminedIsFolderOrUnreadableFile() {
        // It's EITHER a folder, OR a file that couldn't be opened for reading
        var isReadableFile = false;
        onErrorItem(fileItem, response, status, headers, isReadableFile);
    }
}

function onErrorItem(fileItem, response, status, headers, isReadableFile) {
    var isFolderOrUnreadableFile = (isReadableFile === false);

    $timeout(function() {
        if (isFolderOrUnreadableFile) {
            // Jumping through some hoops here to make more beautiful user experience.
            // 1) When user drops folder in FF & Chrome there's no web standard for how to determine if it's a folder.
            //    Chrome has custom webkitGetAsEntry() but we're not willing to spend time on that right now.
            //
            // 2) "isFolderOrUnreableFile" tells us the item is either a folder or the file could not be read.
            //    But that's an ugly message to show a user? ("We don't know if it's a folder or a locked file")
            //
            // 3) Before now (e.g. filter) it didn't seem right to rely on the fact that that in CURRENT Windows
            //    and typical disk formatting, folders are multiples of 4096, because there is a tiny chance
            //    that a user could have a file, with no extension, with size a multiple of 4096.
            //
            // 4) But now that we're here, it would have to be that strange file name with size multiple of 4096
            //    AND ALSO unable to read the file for some reason (e.g. it's currently locked & being written to).
            //    This moves me from 99% confident to 99.99% confident. So, let's make this beautiful for the
            //    majority of users (Windows) by giving them a proper "No folders" error instead of "Um
            //    I think that's probably a folder, but it could also be a file that couldn't be read".
            //    Mac OS X users will still see the "I'm not sure which" error, but oh well, this was low effort.

            var windowsAndLinuxMostCommonDirectorySizeInBytes = 4096,
                unreadableFileIsConfidentlyAFolder = (!fileItem.file.type
                && (fileItem.file.size % windowsAndLinuxMostCommonDirectorySizeInBytes === 0)),
                errorType = unreadableFileIsConfidentlyAFolder ? $scope.enum.uploadError.folder : $scope.enum.uploadError.folderOrUnreadable;
            markUploadItemAsFailed(fileItem, errorType);
        }
        else {
            handleUploadError(fileItem, response, status);
        }
    });
}

After all of this, maybe I should have looked at the browser-specific features! But it seemed to work for me. YMMV!

[bettysteger]

Would it be possible to read the files inside the directory (in Chrome)?

[darynmitchell]

Chrome provides non-standard methods to let you do that. Check out webkitGetAsEntry(), described here: Drag and drop a folder onto Chrome now available.

If you do so, and find a way to add that smoothly within the context of angular-file-upload, I encourage you to share how you did it so others can do the same.

[bettysteger]

Thanks, I would do that of course - at the moment it is just a nice-to-have feature, so it is not sure if I will implement it at all.

[mcantrell]

A pretty decent article for not only detecting folders but also uploading their content:

http://code.flickr.net/2012/12/10/drag-n-drop/

function loadFiles(files) {
    updateQueueLength(count);

    for (var i = 0; i < files.length; i++) {
        var file = files[i];
        var entry, reader;

        if (file.isFile || file.isDirectory) {
            entry = file;
        }
        else if (file.getAsEntry) {
            entry = file.getAsEntry();
        }
        else if (file.webkitGetAsEntry) {
            entry = file.webkitGetAsEntry();
        }
        else if (typeof file.getAsFile === 'function') {
            enqueueFileAddition(file.getAsFile());
            continue;
        }
        else if (File && file instanceof File) {
            enqueueFileAddition(file);
            continue;
        }
        else {
            updateQueueLength(-1);
            continue;
        }

        if (!entry) {
            updateQueueLength(-1);
        }
        else if (entry.isFile) {
            entry.file(function(file) {
                enqueueFileAddition(file);
            }, function(err) {
                console.warn(err);
            });
        }
        else if (entry.isDirectory) {
            reader = entry.createReader();

            reader.readEntries(function(entries) {
                loadFiles(entries);
                updateQueueLength(-1);
            }, function(err) {
                console.warn(err);
            });
        }
    }
}

[topwood]

same question!

[dobesv]

Here's a version of onDrop that recursively processes folder contents, in browsers that support it (Chrome):

            FileDrop.prototype.onDrop = function(event) {
                var transfer = this._getTransfer(event);
                if (!transfer) return;
                var options = this.getOptions();
                var filters = this.getFilters();
                this._preventAndStop(event);
                dragdropCollection = $();
                angular.forEach(this.uploader._directives.over, this._removeOverClass, this);
                // Check for new-style items list
                var uploader = this.uploader;
                if(transfer.items) {
                    var files = [];
                    var pending = transfer.items.length;
                    var oneDone = function() {
                        pending--;
                        if(pending === 0) {
                            uploader.addToQueue(files, options, filters);
                        }
                    };
                    var oneFail = function(err) { console.warn(err); oneDone(); };

                    var processEntry = function(entry) {
                        if(entry.isFile) {
                            entry.file(function(file) {
                                files.push(file);
                                oneDone();
                            }, oneFail);
                        } else if(entry.isDirectory) {
                            reader = entry.createReader();

                            reader.readEntries(function(entries) {
                                pending += entries.length;
                                for(var i=0; i < entries.length; i++) {
                                    processEntry(entries[i]);
                                }
                                oneDone();
                            }, oneFail);
                        } else {
                            oneFail('Entry is not file or directory');
                        }
                    };
                    var processItem = function(item) {
                        // Check for new "Entry" API
                        var entry = (item.getAsEntry && item.getAsEntry()) ||
                            (item.webkitGetAsEntry && item.webkitGetAsEntry());
                        if(!entry) {
                            // No Entry support in this browser, convert to File
                            files.push(item.getAsFile());
                            oneDone();
                        } else {
                            processEntry(entry);
                        }
                    };
                    for(var i=0; i < transfer.items.length; i++) {
                        processItem(transfer.items[i]);
                    }
                } else {
                    this.uploader.addToQueue(transfer.files, options, filters);
                }

            };