It would be nice if it was that easy but if retry behaviour is dependent on the specific operation attempted and the error information in the response then you need some degree of local handling even if its just to prepare information for a generic handler.
Exceptions work best when the error is fatal for the local scope but responses from external services aren't like that. The general problem is that the dividing line between errors and information becomes too blurry - your error might only be information to me. A simple example is where something like axios will (by default) throw on 404 responses but an external api might use 404 to indicate a resource does not exist. If your app logic makes a decision based on this information, you will find yourself using exception handlers for control flow despite not experiencing any actual errors.
Exceptions work best when the error is fatal for the local scope but responses from external services aren't like that. The general problem is that the dividing line between errors and information becomes too blurry - your error might only be information to me. A simple example is where something like axios will (by default) throw on 404 responses but an external api might use 404 to indicate a resource does not exist. If your app logic makes a decision based on this information, you will find yourself using exception handlers for control flow despite not experiencing any actual errors.