Any there any issues webscraping/downloading PDFs directly to the watched folder?

[ScottCov]

I have a program that scrapes/downloads all court dockets for a particular court case. Typically the number of dockets is about 1000-3000 and about 4 dockets are downloaded per min. The process takes about 12-24 hours. Are there any potential issues if I chose to download the pdf files directly to the watched directory?

[dadoonet]

I have no idea. I'm wondering if this could fail if the file is not yet completely downloaded.

I think that the safer option is to download to a non watched dir and after each file is downloaded, move it to the watched dir.

My 2 cents 😉

[ScottCov]

I may already be doing that...I am using Selenium with python which doesn't seem to allow a user to specify anything outside of the default download directory

1)File gets downloaded to default download directory
2)Once completed, it gets moved to a specific directory with the corresponding case number (which could be within the watched directory)

So it could potentially work then?

[dadoonet]

I guess so.

[ScottCov]

One last question...I am guessing when fscrawler processes new files it records the date/time stamp of the newest file somewhere so that if newer files are subsequently added with a later date/time stamp it knows those are new and to process those. Where does it record this? or is this just a function of the last time the it processed the last batch of files regardless of the date/time stamp of those files?

[dadoonet]

Once a full run is done, a _status.json file is created/updated in the job file directory.
It contains the start date of the run.

Any file added after this date is then added on the next run.

[ScottCov]

see it thank you