What are the good models for summarization? I have found all, particularly local models, to be poor. Is there a leaderboard for summarization somewhere?
How do you evaluate quality ? Also I suspect the performance between models would varry between datasets. Heck it would vary on same model/source if you included that your mother was being held hostage and will be killed unless you summarize the source correctly :).
I think you are still stuck with try if it works for you and hope it generalizes beyond your evaluation.
I think summarization quality can only be a subjective criterion measured using user studies and things like that.
The task itself is not very well-defined. You want a lossy representation that preserves the key points -- this may require context that the model does not have. For technical/legal text, seemingly innocuous words can be very load-bearing, and their removal can completely change the semantics of the text, but achieving this reliably requires complete context and reasoning.
