When you mentioned infinite structures, I thought you'd bring up the idea that our goal to automate often pits us against problems defined as collections of infinitary instances. I don't think the potential infinity of tapes and running times poses as much of a concern by comparison.

Maybe computer science is about giving, to borrow a little bit from Hilbert, finitary representation to infinitary structures. (Finitary representation with other properties of interest, such as tractability and whatnot, of course.)