When I was growing up 90-ies, a mix of using public metro and buses to roam the city (since I was in second grade, when I was allowed to take metro to do afterschool karate) and spending summers in various countryside locations where my grandparents resided was a good mix.
I disagree that the kids need or want to roam without grownups all the time. Grownups are not the problem. Kids are fun for the parents, the company of parents and their peers is kinda amazing.
Systems and institutions are the problem. When kids are stuck in the daycare or school, in a very limited space, grownups are stuck at the office and grandparents are in a different state for tax purposes - that is the problem.
I don't know if this is true, but Patagonia claimed at some point that they used to maintain daycares and allow kids to roam the campus...
It feels like to really censor the model it needs to be pre-trained on a distribution of data derived from a well defined and synthetic source, like TinyStories. Otherwise... world model would still be capable of modeling the original distribution.
Ablation in post isn't good enough - it usually does 10% of "expunge the data you want expunged", 70% of "make the data you want expunged less accessible", and 20% of "collateral damage". Training for refusals doesn't damage the capabilities much - it just make them harder to access. If someone has access to model weights, neither holds. GPT-OSS was SOTA at removing unwanted capabilities, and even that didn't hold for long.
Now, dataset curation/filtration does help against select capabilities. But a lot of capabilities are double edged, and can't be deleted without hurting performance at the task you want.
If an AI is good at coming up with novel ways to perform chemical synthesis, it can be reused to come up with pathways for synthesizing illegal drugs or poisons, no way around that. If an AI is good at writing software, it can be reused for writing malware. If an AI is good at autonomously finding vulnerabilities in your own network, it can be reused to do the same in some other dude's network.
AI may have an alignment, but raw capabilities sure don't.
I'm pretty sure that any world model that is inherently incapable of "bad outputs" would be too castrated in general to the point where it'd be actively detrimental to overall model quality. Even as it is, with RLHF "alignment", we already know that it has a noticeable downwards effect on raw scores.
Also note Karpathy notes the problems with agents are tractable but hard.
He's vague on the paths being explored to resolve them. His "higher level" view is probably awareness the solutions to software problems are hardware based fixes, but he cannot say that to software developers. Which has been the back and forth of tech since I was a kid in the 80s writing Basic; new state management unlocked by old software logic being embedded into new hardware.
Two main problems to solve for: too many people bought in to a status quo. And much simpler, the actual engineering of new hardware. One is only resolved by generational churn without resorting to all out police state action. So tech jobs as we know them will fade away slowly to not upset too many, and younger generations will not care as they will never experience anything else.
"Higher" than an EE with an MSc in elastic structures, ~30 years industry experience, now working with PhDs across the spectrum on energy models to embed in chips? Energy models in part, inferred from categorization of LLM contents and compression of those contents into geometric functions like I described?
"Higher level" implies acceptance of geometric structure. You place tokens like a Chomsky diagrams at each step up and down, where you should see parameters to transform geometry of the structure.
My team works "above" the contrived state management of software workers to more efficiently sync memory matrix to display matrix. LLMs are a form of compression [1]. My team is working on compressing them further into sets of points that make up each glyph and functions to recreate them.
Electromagnetic geometry transforms hardcoded[2] into hardware so reduce energy use of all the outdated string mangling of software dev as most know it.
What's higher level, relative to our machines, than design and implementation of the machine?
DnD dungeon master versus WOTC game designer.
Notice outside how there are no words and philosophy? Just color gradient and geometry?
Notice inside the human body no philosophy or words?
Language is not intelligence it's an emergent phenomena of geometry created by fundamental forces of physics organizing matter at various speeds relative to light.
You've read too much into an ultimately arbitrary statement meant to invoked a subtext, a subtle emotion context. You think of language as Legos, when it is music to feel.
Two decades ago, in the Bay Area we used to have a lot of books stores, specialized, chains, children's, grade school, college slugbooks, etc. Places like Fry's had a coffee and a book store inside. The population grew, number of book stores went down to near zero.
In that case, it's great that Microsoft is building their silicon. Keeps NVIDIA in check, otherwise these profits would evaporate into nonsense and NVIDIA would lose the AI industry to competition from China. Which, depending if AGI/ASI is possible or not, may or may not be a great move.
> There's a simple fact: humans prefer lies that they don't know are lies over lies that they do know are lies.
As an engineer and researcher, I prefer lies (models, simplifications), that are known to me, rather than unknown unknowns.
I don't need to know exact implementation details, knowledge of aggregate benchmarks, fault rates and tolerances is enough. A model is a nice to have.
This approach works, in science (physics, chemistry, biology, ...) and in engineering (including engineering agentic and social sustems- social engineering).
> As an engineer and researcher, I prefer lies (models, simplifications), that are known to me, rather than unknown unknowns.
I think you misunderstood.
I'll make a corollary to help:
~> There's a simple fact: humans prefer lies that they believe are truths over lies that they do know are lies.
I'm insure if you: misread "lies that they don't know are lies", conflated unknown unknowns with known unknowns, or (my guess) misunderstood that I am talking about the training process which involves a human evaluator evaluating an LLM output. That last one would require the human evaluator to preference a lie over a lie that they do not know is actually a lie. I think you can see how we can't expect such an evaluation to occur (except through accident). For the evaluator to preference the unknown unknown they would be required to preference what they believe to be a falsehood over what they believe is truth. You'd throw out such an evaluator for not doing their job!
As a researcher myself, yes, I do also prefer known falsehoods over unknown falsehoods but we can only do this from a metaphysical perspective. If I'm aware of an unknown then it is, by definition, not an unknown unknown.
How do you preference a falsehood which you cannot identify as a falsehood?
How do you preference an unknown which you do not know is unknown?
We have strategies like skepticism to deal with this help with this but this doesn't make the problem go away. It ends up with "everything looks right, but I'm suspicious". Digging in can be very fruitful but is more frequently a waste of time for the same reason: if a mistake exists we have not identified the mistake as a mistake!
> I don't need to know exact implementation details, knowledge of aggregate benchmarks, fault rates and tolerances is enough.
I think this is a place where there's a divergence in science and engineering (I've worked in both fields). The main difference in them is at what level of a problem you're working on. At the more fundamental level you cannot get away with empirical evidence alone.
Evidence can only bound your confidence in the truth of some claim but it cannot prove it. The dual to this is a much simpler problem, as disproving a claim can be done with a singular example. This distinction often isn't as consequential in engineering as there are usually other sources of error that are much larger.
As an example, we all (hopefully) know that you can't prove the correctness of a program through testing. It's a non-exhaustive process. BUT we test because it bounds our confidence about its correctness and we usually write cases to disprove certain unintended behaviors. You could go through the effort to prove correctness but this is a monumental task and usually not worth the effort.
But right now we're talking about a foundational problem and such a distinction matters here. We can't resolve the limits of methods like RLHF without considering this problem. It's quite possible that there's no way around this limitation since there are no objective truths the majority of tasks we give LLMs. If that's true then the consequence is that a known unknown is "there are unknown unknowns". And like you, I'm not a fan of unknown unknowns.
We don't actually know the fault rates nor tolerances. Benchmarks do not give that to us in the general setting (where we apply our tools). This is a very different case than, say, understanding the performance metrics and tolerances of an o-ring. That part is highly constrained and you're not going to have a good idea of how well it'll perform as a spring, despite those tests having a lot of related information.
There was a nice video, I've seen at some point where a "DJI Phantom 3 drone gets hit with an electrical impulse of 1.4MV - basically, a lightning strike."
And at the end, they were able to protect the drone, with a tiny bit of shielding...
A gutted microwave oven and a satellite tv dish have been demonstrated to disable DJI drones at ranges exceeding 500m - either having them fall out of he sky or trigger return to home. That's broadband jamming on the 2.4GHz radio frequencies though, not sending enough energy to screw with electronic (apart from the sensitive radio receiver frontends).
(This was original DJI Phantom era, so maybe 10 years or so back. I'm not aware of results of similar testing against newer DJI gear, but I doubt it'd be much different, at least for consumer DJI stuff.)
This is great to hear. Churches are often in the prime real estate areas, it'd be great, if the attendance / donations would fall to zero. And if these places would be redeveloped into open-for-all urban green islands. Rather than exclusive worship clubs.
I am unsure what your comment has to do with mine.
But I disagree with your value judgement. Open spaces are great - but men and women who believe in something greater than themselves is what makes the world go round.
Looking at fertility rates between religious and atheists, we'll need these churches and temples in the next generations.
That's an interesting argument, about believing in something greater than themselves. But what, if that belief happens to be outdated? This had happened before.
I'm not into demographics planning. But I do have two kids, girls. I certainly want them to have as much autonomy as they could have. A religious argument regarding "fertility rates" that need to be increased is not aligned with that.
I love green spaces. So I simply want to see churches gone, because they compete for the same (very limited) resource. Using the privilege of being grandfathered in.
If they'll move their activity online, it'll not be the case. So I'm happy to see this happening.
I really appreciate a chance to discuss this with a fellow dad. A lot of these things quickly touch on what we value and believe, rather than what we can prove.
So for example - even back when I was an atheist (I was born in the USSR) - I had a feeling that life had a purpose, there was a potential to live up to, there was meaning and reason beyond just me. In a way that's a religiously originated belief that happened to "survive" for a while unrooted from religion, but clearly originated from it.
Now that I've come back to religion, one lens on it is that it's simply the necessary context in which meaning and purpose can exist. For example, if as an atheist I encounter another atheist who is a nihilist or hedonist (as proxy for not having anything greater than themselves) I have no logical foundation from which to say there's more to life than that. But as a religious person talking to another religious person, establishing that baseline is trivial - that's not where the effort goes.
For a belief to be "outdated" requires something updated that works better. One obviously true attribute of atheism is that it doesn't enable its adherents to have enough kids - so from a purely Darwinian metric it obviously doesn't work. It's a hard sell to say "here's a better belief, sure it'll drive you into an evolutionary dead end but... uhh yeah it's better"
Congrats on your two girls and while I also want my kids (2 boys and 1 girl) to have maximal autonomy, I have a great role to play in shaping their thinking and values. For example - whether I live in a neighborhood with a lot of families of in a city filled primarily with spinsters, whether I convey my own values around children or not, whether I encourage a social group that is positive about parenthood vs one that sees kids as undesired burdens, these are all things within my control. And my personal belief is that my kids will be happiest and have the greatest meaning in life if they - like me and my wife - love life, and want to pass it on, and to use our children as a way to contribute positively to the future. So yes I want to pass on these values, yes religion helps that, and also - if we don't do that - the future will simply be inherited by the kids of parents who do. It's not about sustaining fertility rate in some mechanistic sense, it's about a dropping rate being a numeric symptom of a problem I don't want my kids to suffer from if I can help it.
I agree with the desire to not move your kids "online" but I don't think green spaces are gonna do that for you, as plenty of kids live near the park but never look up from their phone. I recently realized that a religious lifestyle "forces" more in person interaction, as I alluded to in my original comment here. For example, it's my religious friends that attend services together - physically, that go to each others houses for lifecycle events, that study as a community.
I disagree that the kids need or want to roam without grownups all the time. Grownups are not the problem. Kids are fun for the parents, the company of parents and their peers is kinda amazing.
Systems and institutions are the problem. When kids are stuck in the daycare or school, in a very limited space, grownups are stuck at the office and grandparents are in a different state for tax purposes - that is the problem.
I don't know if this is true, but Patagonia claimed at some point that they used to maintain daycares and allow kids to roam the campus...
reply