Yes, good catch! As this comment points out, OFU only makes sense when talking about stochastic environments (the OFU algorithm is described in terms of confidence intervals). The adversarial setting is quite different.