There's something like a "scrolling cost" -- people are only willing to skim so much of a comments thread, without seeing something interesting to them, before closing it.
This is why HN dislikes humorous fluff-posts: they both easily rise to the top, and encourage humorous fluff-replies, which means the first few screenfuls of comments will be guaranteed to induce the kind of "scroll-pain" that makes people close the tab.
Collapsing threads do the job pretty well. And you're always going to get comments that aren't relevant to you, because you're not the only person doing the endorsing. If a comment is useless fluff to you, but gets endorsed by someone else, then you still have a scrolling problem. Not so with collapsable commeents.
They use a very simple system, collapsing posts after a certain depth, and hiding more comments after a certain number (10, I believe) at the level below the top comment (sorted by score obviously).
It works extremely well, and thanks to the fast JS collapsing, it's not at all a hassle to read a subthread that happened to get collapsed if it piques your interest.
It's so simple it may easily be overlooked in its obviousness, but really you don't need a very complex system that is strictly a lot better than no collapsing at all.
What's wrong with (old) slashdot (I mean, technically) ?
While I'm usually a kind of hard core html-first, ajax/js/webapps later kind of guy -- I'd love for the comments to be served up in a json-blob, with a couple of user-settable preferences ("Show only comments rated higher than N, hide threads with lower (median/mean) rating than N, show all direct replies to my comments -- and similar).
You have to read a thread enough to know you want to collapse it, which incurs exactly the same "scroll-pain" as actually scrolling. It's not a physical tax from the action required to scroll, but rather a mental stress from making a decision to skip something. Enough of that stress building up at once, and you decide that the comments page itself is probably skippable.
If uninteresting comment threads could start collapsed, that'd be great. But they won't, because humor-fluff and other such things are superstimuli for upvotes, so you can't use anything about the vote tally to determine collapsed-ness. (If there was a secondary voting system--like, say, if enough people collapsing a thread would make it start collapsed for others--this might work. But then you'd have to take into account the people who collapse everything as they read it, to mark their place...)
The problem is, your definitions of what is uninteresting, useless and fluff are subjective, and no more relevant than anyone else's, which leads to a conflict of interests within the community about the bounds of what Hacker News content should and shouldn't be.
I think the models which would satisfy the most people are opt-in, in this case, choosing to collapse threads and ignore users rather than expecting the hivemind to do it for you.
Perhaps if users had a custom set of filters which automatically collapsed threads for them based on their own criteria, that would solve part of the problem. But I don't think it's too much to ask of people to actually take the minor effort to form an opinion on what they read, or curate their own account, if they expect content they don't like to be hidden from themselves and potentially from others.
Disagree on your first point, the top comment is often obviously spawning a huge argument that goes nowhere because it touched some hot-button political issue that was a minor part of the story.
Of course this is the most obvious solution, least intrusive that changes the least things and as far as I can see has only positive side-effects.
But for some reason or other pg won't do that.
I know there's some extensions/bookmarklets and I use them, but they can't do the auto-collapsing based on score+amount, the very simple yet elegant algorithm Reddit employs, pretty much the one thing that is absolutely necessary to have a well-usable threaded commenting system.
Instead, pg "just wrote the simplest thing first", or something.
