Split inference packages out into separate Manifest

This changes the `load!()` machinery to eval into `Main` instead of
`BaseBenchmarks`, which actually allows for a runtime-changable
dependency list.

Author: staticfloat

.gitignore

@@ -1,4 +1,7 @@
 *.jl.cov
 *.jl.*.cov
 *.jl.mem
+!src/inference/LocalPreferences.toml
+!src/inference/Manifest.toml
+!Manifest.toml
 !LocalPreferences.toml