This is absolutely NOT true. (1) Signal doesn't store your contacts, and (2) Signal only stores a name and a profile photo if you want, and in a secure wayhttps://signal.org/blog/signal-profiles-beta/
I'm sorry to be the one to tell you, but Signal 100% stores your contacts. They keep your name, your photo, your phone number, and a list of every person you've contacted using Signal. That data is permanently stored on their servers.
> "This should be updated for the recent changes where contacts are uploaded to Signal’s servers and stored permanently along with Groups V2 and other data, protected by a 4-digit minimum PIN and Intel SGX – there have been concerns 5 raised 2 in these forums, particularly if one of your contacts chooses a brute-forceable PIN which in the context of an Intel SGX vulnerability 1 could leak a lot of contact data if hacked, even if you choose a strong password."
See the two links sited in that comment for more information on why it isn't actually stored in "secure" way.