>3) LLMs do not have a mechanism for sampling from given probability distributions. E.g. if you ask LLM to sample a random number from 1 to 10, it will likely give you 3 or 7, as those are overrepresented in the training data.
I went and tested this, and asked chat gpt for a random number between 1 and 10, 4 times.
It gave me 7,3,9,2.
Both of the numbers you suggested as more likely came as the first 2 numbers. Seems you are correct!
I recall a video (I think it was Veritasium) which featured interviews of people specifically being asked to give a "random" number (really, the first one they think of as "random") between 1 and 50. The most common number given was 37. The video made an interesting case for why.
(It was Veritasium but it was actually a number from 1 to 100, the most common number was 7 and the most common 2-digit number was 37: https://www.youtube.com/watch?v=d6iQrh2TK98.)
I went and tested this, and asked chat gpt for a random number between 1 and 10, 4 times.
It gave me 7,3,9,2.
Both of the numbers you suggested as more likely came as the first 2 numbers. Seems you are correct!