It's a tiny bit more than 10 years old now, but one of my favorite algorithms, period, is John Platt's Sequential Minimal Optimization. (http://portal.acm.org/citation.cfm?id=299094.299105)

It was a huge advancement to the state of the art in training SVMs.

This is not my field, but the hash array mapped trie from 2000-1 (http://en.wikipedia.org/wiki/Hash_array_mapped_trie) is the source of Clojure's "secret sauce" that is its answer to the functional programming trivial update problem. Updates are O(N) where N is a max of 32 and more likely 5-6, according to Rich Hickey.