Hey! Congrats on the launch. I just signed up for a trial account and I’m pretty impressed with the search API (haven’t used websets yet but looks cool).
Our experimental use case is enabling quick and dirty integration of web-based docs into an employee service agentic chatbot - lots of the questions are around “how do I max out my 401k”, which connects to internal information, but some are more like “how do I link a calendar to calendly”.
The one thing I’d love to have in the search product is a cruft cleaner for the results of web queries. Where you have cached the data presumably this wouldn’t add much overhead. Reduces what you have to feed to the LLM downstream and might improve the embeddings performance.
By cruft cleaner, do you mean cleaning the HTML well? Right now, we do 2 things to help with that, a pretty robust parsing stack as well as a "summaries" feature that returns an LLM-generated query-biased text output for every webpage returned.
Our experimental use case is enabling quick and dirty integration of web-based docs into an employee service agentic chatbot - lots of the questions are around “how do I max out my 401k”, which connects to internal information, but some are more like “how do I link a calendar to calendly”.
The one thing I’d love to have in the search product is a cruft cleaner for the results of web queries. Where you have cached the data presumably this wouldn’t add much overhead. Reduces what you have to feed to the LLM downstream and might improve the embeddings performance.