Certainly agree and we have seen a lot of cases of JWT tokens getting bloated with more and more authorization data.
Using JWTs to hold the autheNtication information is a standard now but consuming that and applying fine-grained access controls for authoriZation using that context with a system that can enforce policies is that next step.
Having a dedicated AuthZ system in place that sits after the AuthN layer should put your mind at rest as a good one will provide a central point for all authZ logic which can be managed, tested and audited for every action within a system.
