Weights are just numbers, essentially by using a neural network you are telling the system to "find the best way to represent the scene with a budget of X numbers/parameters". Modern NeRFs like instant-ngp also use some grid representations. I guess Gaussian Splatting is slightly more geometrically appealing because you get points around the surfaces that you are trying to model. These points are however not guaranteed to be exactly on the surface, which additional surface losses try to solve (e.g. NeuSG).