Based on the fact that there are very few up-to-date English-language search indexes (Google, Bing, and Brave if you count it), it must be incredibly costly. I doubt they are maintaining their own.
I've been wondering can't this be done p2p? Didn't we solve most of the technical problems in the late 90s / early 2000s? And then just abandoned that entire way of thinking for some reason?
If many thousands of people care about having a free / private / distributed search engine, wouldn't it make sense for them to donate 1% of their CPU/storage/network to an indexer / db that they they then all benefit from?
Well, flesh it out more and it doesn't sound solved at all.
How do you make it trustless. How do you fetch/crawl the index when it's scattered across arbitrary devices. How do you index the decentralized index. What is actually stored on nodes. When you want to do something useful with the crawled info, what does that look like.
I think you could do it hierarchically, and with redundancy.
You'd figure out a replication strategy based on observed reliability (Lindy effect + uptime %).
It would be less "5 million flaky randoms" and more "5,000 very reliable volunteers".
Though for the crawling layer you can and should absolutely utilize 5 million flaky randoms. That's actually the holy grail of crawling. One request per random consumer device.
I think the actual issue wouldn't be the technical issue but the selection. How do you decide what's worth keeping.
You could just do it on a volunteer basis. One volunteer really likes Lizard Facts and volunteers to host that. Or you could dynamically generate the "desired semantic subspace" based on the search traffic...
Let me illustrate this with a more poetic example.
In 2015, I was working at a startup incubator hosted inside of an art academy.
I took a nap on the couch. I was the only person in the building, so my full attention was devoted to the strange sounds produced by the computers.
There were dozens of computers there. They were all on. They were all wasting hundreds of watts. They were all doing essentially nothing. Nothing useful.
I could feel the power there. I could feel, suddenly, all the computers in a thousand mile radius. All sitting there, all wasting time and energy.
perplexity added API today, got the following email:
> Dear API user,
We’re excited to launch the Perplexity Search API — giving developers direct access to the same real-time, high-quality web index that powers Perplexity’s answers.
Not particularly. Indexes are sort of like railroads. They're costly to build and maintain. They have significant external costs. (For railroads, in land use. For indexes, in crawler pressure on hosting costs.)
If you build an index, you should be entitled to a return on your investment. But you should also be required to share that investment with others (at a cost to them, of course).
This is now (more) dated. Copilots as an interface are dated. The current initiative is full agents with human in the loop at the very start, occasionally the middle and the end.
Huds are just good UI, something copilots can natively exist as part of in the form of contextual insights and alerts.
We’re moving on to agency where it’s everything else vs an entirely different entity taking the action of flying the plane from take off to landing.
I don't think you understood what a "copilot" and a "HUD" is. Confusingly, GitHub Copilot (the original one that suggests completions) is basically a HUD. On the other hand, agents that you give a task are clearly copilots that work via a natural language interface.
The article also mentions that agents are copilots:
> Here’s another personal example from AI coding. Let’s say you want to fix a bug. The obvious “copilot” way is to open an agent chat and ask it to do the fix.
I don’t understand this one at all. Say you need to update a somewhat unique implementation of a component across 5 files. In pseudocode, it might take you 30 seconds to type out whatever needs to be done. It would take maybe 3-4 minutes to do it.
I set that up to run then do something different. I come back in a couple minutes, scan the diffs which match expectations and move on to the next task.
That’s not everything but those menial tasks where you know what needs to be done and what the final shape should look like are great for AI. Pass it off while you work on more interesting problems.
more common than ever before, my 4 closest batchmates are each at 1.5 - 4m ARR and still only taken a seed. i think all but one are cfp as well so unclear what raising more gets them. And all teams under ~15
many YC companies have hit the Series A benchmark and just skipped it, a change from the past
1m, 2m ARR don’t have the same impact in fundraising so later Series A are bound to happen and are not a great indicator imo of how well a company is doing
throwaway but several from my batch (including us) are cash flow positive, higher ARR than these Series A companies and not on the fundraising treadmill
that cohort of YC companies is not represented in data anywhere because from what I’ve seen it is a bit of an anomaly vs previous YC batches
