(a) Higher-order hyperoptimization performance with SGD. (b) Higher-order hyperoptimization performance with Adam. (c) As the stacks of hyperoptimizers grow taller, each step of SGD takes longer by a small constant factor, corresponding to the extra step of stepping one node further in the backwards AD computation graph. (Performance)
{}