fyi some other Stan benchmarking code

[SteveBronder]

@bbbales2 has been working on benchmarks using google bench. Thought you may find some of them useful

https://github.com/bbbales2/perf-math/pull/1/files

Also is there a reason you are not using googlebench? I find it pretty easy for setting up stuff and it supports manual timing so you can catch the forward and/or reverse pass times

[JamesYang007]

I remember having some trouble getting Adept, one of the other libraries, working with gbench, but it was for an earlier version so who knows if things changed. I just manually wrote this just to be safe, but I might clean this up and try porting it to gbench (though I don't think the times will be much different)!

[SteveBronder]

Do you remember what the issue was? I can try running some of this either this week or next.

Yeah if your stuff works then ya know that's fine. I just like gbench because it's pretty standard.

Also side note fastAD looks very neat! I like how you compound expressions together and a lot of the other design choices are very cool

[JamesYang007]

If I recall correctly, the gradient was actually not correct and there were memory errors (sometimes leading to segfault). But there's also a good chance I didn't write proper code for Adept back then. Actually, for my own curiosity, I'm going to try to use gbench instead, what the hell why not :D I agree gbench is super nice.

Also thanks for checking out FastAD! That means a lot coming from a stan dev!

[SteveBronder]

It may be useful to separate out each program into it's own executable. My guess is that would effect what the compiler looks at. There's also some oddities to watch out for. For instance in Stan we have a global memory arena that we need to keep track of. After each gradient evaluation the program needs to zero out the adjoints to run the same gradient again and for a new gradient calculation you want to run stan::math::recover_memory() so Stan reuses the already allocated memory in the arena. For our benchmarking we usually run some arbitrary big matrix first to allocate the memory once, run stan::math::recover_memory(), then run the actual benchmark. Since all the memory is allocated on the first pass this usually gives us a more realistic benchmark's for a particular function. Though for testing across different autodiff packages it may be a better idea to actually account for those allocations in the first pass

[JamesYang007]

I see, so in my benchmark run_test.hpp I'm currently doing this:

        sw.start(); // start stopwatch
        for (int i = 0; i < stan_pack.n_iter; ++i) { // iterate like 10000 times 
            stan::math::gradient(f, x, fx, grad_fx);
        }
        sw.stop(); // stop stopwatch

and you're recommending this instead:

        sw.start();
        for (int i = 0; i < stan_pack.n_iter; ++i) {
            stan::math::gradient(f, x, fx, grad_fx);
            stan::math::recover_memory();
        }
        sw.stop();

[SteveBronder]

@bbbales2 it's the second one right? Since gradient creates a vector of var

[bbbales2]

stan::math::gradient uses the nested stack: https://github.com/stan-dev/math/blob/develop/stan/math/rev/functor/gradient.hpp#L44 . The first should be good. That will time the forward and reverse passes and get rid of memory when it's all done.

[SteveBronder]

Oh yes good call! Forgot the nested stack recovers memory after leaving scope

[JamesYang007]

@SteveBronder update: I just ported everything to gbench - turns out I was just coding Adept wrong before! Stan-related code is in benchmark/stan if you're interested.

[SteveBronder]

Nice!! I'll try to take a look this week or next