It matters if 1) you have long-lived sessions on the application (RP) or 2) the application (RP) does something on behalf of the users and you need to stop that.
For example, say I leave the company. My account in SSO (IdP) gets deactivated, so I can no longer use SSO to log in to the company GitLab. However, without some way to let GitLab know that I'm gone, I might still be able to access repos with my SSH keys or access tokens, and scheduled jobs in my account will still run. Deprovisioning not only lets GitLab know I won't be logging in through SSO again, but also that it should stop the scheduled jobs in my account, and block other access methods.
You're right that depending on what your application does, you might not need it at all.
With straight up fed you can autocreate users etc but as the relying party you don't know when their access has been revoked. Sometimes this doesn't matter, but... often it does. People come and go all the time in enterprise. You can more accurately show user state in your UI and (crucially) accurately adjust license seats. You can also de-activate API tokens or alternative auth methods the user might have configured to close the loop on security.
Big orgs will pay more for SCIM to avoid having to do user access reviews in the UIs of individual products they have bought. It directly translates to person-hours of busywork and admin burden of executing / tracking that work.
Deprovisioning is huge. Provisioning is also awfully nice. For example:
1. New employee joins the team.
2. SCIM creates their account in the ticket tracking system.
3. Employee's boss can create onboarding tickets and assign them to the new person before they've even logged into the system.
That's not such a big deal for small companies that don't use a gazillion services. It's huge for large companies with hundreds of vendors where you want everyone in an employee group ("engineering", "sales", "everyone") to have an account in some of those services.
