Are you / have you considered leveraging postgres' row level security for this (row filtering)? Feels like a natural fit for ensuring the right subset of data gets to the right users https://www.postgresql.org/docs/current/ddl-rowsecurity.html
E2E encryption would also be nice to have, it could be worth having a look at https://www.etesync.com/ for inspiration
You definitely can do row-level filtering (and a lot more) using DDLX[1], an extension we made to Postgres language that allows expressing flexible permission rules in a data-centric way.
E2E encryption would also be nice to have, it could be worth having a look at https://www.etesync.com/ for inspiration