It is interesting that the model I am proposing inverts many peoples expectation of how LLMs will benefit us. In one vision, we give a data-lake of information to LLMs, they tease out the relevant context and then make deductions.
In my view, we hand craft the context and then the LLM makes the deductions.
I guess it will come down to how important crafting the relevant context is for making useful deductions. In my experience with writing code using LLMs, the effectiveness increases when I very carefully select the context and the effectiveness goes down when I let the agent framework (e.g. Cursor) figure out the context. The ideal case is the entire project fits in the context window obviously, but that won't always be possible.
What I've found is that LLMs struggle to ask the right questions. I will often ask the LLM "what other information can I provide you to help solve this problem" and I rarely get a good answer. However, if I know the information that will help it solve the problem and I provide it to the agent then it often does a good job.
> In my view, we hand craft the context and then the LLM makes the deductions.
We (as in users) provide the source material and our questions, the LLM provides the answers. The entire concept of a context is incidental complexity resulting from technical constraints, it's not anything that users should need to care about, and certainly not something they should need to craft themselves.
But it makes a radical difference to the quality of the answer. How is the LLM (or collaboration of LLMs) going to get all the useful context when it’s not told what it is?
(Maybe it’s obvious in how MCP works? I’m only at the stage of occasionally using LLMs to write half a function for me)
In short, that's the job of the software/tooling/LLM to figure out, not the job of the user to specify. The user doesn't know what the context needs to be, if they did and could specify it then they probably don't need an LLM in the first place.
MCP servers are a step in the direction of allowing the LLM to essentially build up its own context, based on the user prompt, by querying third-party services/APIs/etc. for information that's not part of their e.g. training data.
In my view, we hand craft the context and then the LLM makes the deductions.
I guess it will come down to how important crafting the relevant context is for making useful deductions. In my experience with writing code using LLMs, the effectiveness increases when I very carefully select the context and the effectiveness goes down when I let the agent framework (e.g. Cursor) figure out the context. The ideal case is the entire project fits in the context window obviously, but that won't always be possible.
What I've found is that LLMs struggle to ask the right questions. I will often ask the LLM "what other information can I provide you to help solve this problem" and I rarely get a good answer. However, if I know the information that will help it solve the problem and I provide it to the agent then it often does a good job.