(chromeredir) added some limits to number of open chrome pages

[ohanf]

chrome really does not like having > 1000 pages open, tried only doing 20 for now

I got a little excited yesterday and did not think to test at scale (as your tools are often run). I tried running what I wrote yesterday with 1000 hosts and it really did not like it, so I have made some changes. Sorry for the double PR, I will try to be a little more thoughtful going forward.

Here are some very rough metrics from my testing today (only ran each test once or twice and on my hardware so ymmv)

It is a little hard to parse, the tl;dr is that it appears that performance is best with 20-50 workers, so I have set the default to 20 as that is in line with your other tools (from what I can tell)

test command = time cat hosts | node checkredir.js

MAX_WORKERS = 1
	node checkredir.js  3.36s user 0.66s system 0% cpu 9:10.80 total
	<killed after ~400 urls>
MAX_WORKERS = 10
	cat hosts  0.00s user 0.00s system 2% cpu 0.102 total
	node checkredir.js  55.67s user 16.99s system 78% cpu 1:32.00 total
MAX_WORKERS = 20
	cat hosts  0.00s user 0.00s system 1% cpu 0.103 total
        node checkredir.js  56.34s user 17.06s system 101% cpu 1:12.39 total
MAX_WORKERS = 30
	cat hosts  0.00s user 0.00s system 2% cpu 0.121 total
	node checkredir.js  62.26s user 19.23s system 116% cpu 1:10.01 total
MAX_WORKERS = 40
	cat hosts  0.00s user 0.00s system 1% cpu 0.103 total
	node checkredir.js  64.98s user 20.60s system 121% cpu 1:10.37 total
MAX_WORKERS = 50
	cat hosts  0.00s user 0.00s system 2% cpu 0.106 total
	node checkredir.js  65.60s user 21.20s system 121% cpu 1:11.39 total
MAX_WORKERS = 60
	cat hosts  0.00s user 0.00s system 1% cpu 0.094 total
	node checkredir.js  66.28s user 21.67s system 119% cpu 1:13.89 total
MAX_WORKERS = 75
	cat hosts  0.00s user 0.00s system 1% cpu 0.101 total
	node checkredir.js  70.60s user 23.99s system 116% cpu 1:20.98 total
MAX_WORKERS = 100
	cat hosts  0.00s user 0.00s system 1% cpu 0.105 total
	node checkredir.js  72.22s user 24.93s system 110% cpu 1:28.18 total
MAX_WORKERS = 500
	<crashes>
MAX_WORKERS = 1000
	<crashes>

Regardless I do believe this is an improvement from both yesterday and the original, if there is interest I can keep working on / help work on this more as I don't think it is perfect yet (but what ever really is?).

Thanks for the quick response yesterday, it's great to see how involved you are with your projects