I remember testing this theory when I lived in Los Gatos, CA around 1980 and had a PO Box there. Los Gatos has three zip codes: 95030 for the downtown area where I lived, 95031 for the PO boxes at the downtown post office, and 95032 for the east part of town and the hills.
I put a stamp on an envelope and wrote nothing but 95031-nnnn on the envelope, where nnnn was my PO Box number. Mailed it from some other local town, and sure enough it was delivered to my PO Box!
And since a small post office will have less than 9999 boxes you can identify one by zip+4.