> That the whole field seems to be moving in a direction where you need a lot of resources to do anything. You can have 10 different ideas on how to improve LLMs but unless you have the resources there is barely anything you can do.
I think you're confusing problems, or you're not realizing that improving the efficiency of a class of models is a research area on it's own. Look at any field that involves expensive computational work. Model reduction strategies dominate research.
I felt that way maybe an year or two ago. It seemed like the most research were only concerned about building bigger models to beat benchmarks. There was also this prevalent idea that models need to be big and have massive compute. Especially from companies like openai. I was glad that models like deepseek were made. Bought back some hope
I think you're confusing problems, or you're not realizing that improving the efficiency of a class of models is a research area on it's own. Look at any field that involves expensive computational work. Model reduction strategies dominate research.