Indeed; the simple thinking you apply to sort some numbers, or to understand a brutally simple logic such as a programming language spec, gives you incredible insight into all fields of human endeavour, past and present.
On a somewhat related note - I highly recommend visiting the Mayan ruins at the Coba archaelogical site. It has the tallest Mayan pyramid on the Yucatan peninsula, with half of it (the inaccessible rear) still inundated by dense jungle. A few tips: if you don't want to opt for a guided tour, prepare to turn down several pointed offers at the entrance. Also, there are small rickshaws for transportation - they will inflate the distances involved to get you to hire one - if you can take a few hours total of walking including breaks you don't need their services. Quite a place! Less squeaky clean than Chichen Itza.
The fact you think programming is about numbers shows you have no idea what programming is.
The fact is, programming at a high level means analyzing thousands of dimensions of information and flattening them out into something executable by a dumb machine.
We are literally expert at becoming experts.
For any given data set there are a number of possible interpretations. Is that notch natural? They say yes, but I'm sure others in the field disagree as well.
His claim was that programming is a simple kind of thinking. He gives two representative examples:
* The consideration of sorting algorithms (which in all fairness _is_ a staple in teaching algorithms), and
* The use of simple logic - such is representative of programming language specs.
I'd say that this is a fair summary - real life tends to be a lot more complicated that the problems faced by most programmers.
Even more brutally: he is aiming quite high, a lot of programming is the minor stitching together of APIs without making too big of a mess.
Few programmers ever seriously actually need to analyse thousands of dimensions of information - and I'd say that it is a safe bet that the ones who do are probably using numbers.
To "We are literally expert at becoming experts.", I'd reply with https://xkcd.com/1112/