- I made this little thing with 100 levels, each easy enough to do in 30 seconds to 2 minutes: https://akkartik.name/post/2024-10-26-devlog. My kid has zero patience for lecturing, and this is me trying to sneak through that filter.
- When you get to the end, it turns out you're pretty close to a game of Pong or Snake, and it's been interesting to see people build toward one vs the other. I've been obsessed with Snake lately, alternating between playing it and hacking on it: https://merveilles.town/@akkartik/114547652849162554
"We did notice, and documented in our paper, instances when the DGM hacked its reward function.. To see if DGM could fix this issue.. We created a “tool use hallucination” reward function.. in some cases, it removed the markers we use in the reward function to detect hallucination (despite our explicit instruction not to do so), hacking our hallucination detection function to report false successes."
So, empirical evidence of theoretically postulated phenomena. Seems unsurprising.
Reward hacking is a well known and tracked problem at frontier labs - Claude 4’s system card reports on it for instance. It’s not surprising that a framework built on current llms would have reward hacking tendencies.
For this part of the stack the interesting question to me is how to identify and mitigate.
I don't care about dates on titles in HN, but strongly care about dates _somewhere_ on any post online. Some indication of the time (range) when it was published. Even if the author doesn't consider it important it helps future readers (including the author! kinda like code comments!) answer all sorts of unanticipated questions about the post down the road. And it's easy to do. I do it all the time by reflex in my private longhand notebooks. So if you also consider this precious, I don't understand what you mean by "precious" and I have trouble translating it to some concrete ill-consequence that would preclude doing this very easy thing that might help one's readers.
Top posting vs replying inline feels like an incomplete dichotomy. I often encounter threads on mailing lists where people are replying inline but not trimming the part they're not replying to. That makes it hard to follow. Or they reply inline and trim well but it's still hard to follow because you need more context than just the part they're replying to.
So the key is "the thread needs to be distilled" as OP puts it. And often that's more work than just finding the right sentence to quote.
My approach these days in my email is:
99% of the time I quote nothing and delete everything my client puts into the compose window, relying on the default thread-view in most email clients to supply the context for my readers.
1% of the time I need to quote, and I quote liberally, treating words as a wiki, and editing/sculpting the text in '> ' just as much as my own reply below it.
Yeah this is my strategy. The top/bottom post brigade are both happy and email clients seem to handle it well.
It's kind of alarming to me that the default in say gmail is to constantly re-send the original message chain back and forth. I guess it then gives the whole chain to newcomers to the thread?
Yeah and it prevents issues if the subject line changes in an unexpected way that confuses the email client, or issues where an email chain goes on for a long period of time but retention policies delete old emails. Etc…
I don't know if this will address your question, but I had a long-standing question about boids that might overlap, which I coincidentally only resolved to my satisfaction a month ago. Here's the Lua code I ended up with:
function update_positions(boids, dt)
local max_speed = 0.5 -- per frame
local max_accel = 20 -- per second
local max_turn_angle = math.pi/6 -- per second
for _,boid in ipairs(boids) do
boid.pos = vadd(boid.pos, boid.velocity)
end
for i,boid in ipairs(boids) do
local accel = {x=0, y=0}
accel = vadd(accel, vscale(avoid_others(boid, boids), 20*dt))
accel = vadd(accel, vscale(seek_others(boid, boids), 10*dt))
accel = vadd(accel, vscale(align_with_others(boid, boids), 10*dt))
accel = vadd(accel, vscale(remain_within_viewport(boid), 40*dt))
local curr_heading = vnorm(boid.velocity) -- could be nil
accel = vclamp2(accel, max_accel*dt, curr_heading, max_turn_angle*dt)
boid.velocity = vadd(boid.velocity, accel)
boid.velocity = vclamp(boid.velocity, max_speed)
end
end
Here, avoid_others, seek_others and align_with_others are the 3 rules you can find on Wikipedia (https://en.wikipedia.org/wiki/Boids): separation, cohesion, alignment. Each of the functions returns a unit vector, which I then weight using vscale.
The key is the last 4 lines. My intuition here is that the way muscle mechanics work, there are limits on both how fast you can accelerate and also how much you can turn per unit time. That's what vclamp2 is doing. It separately clamps both magnitude and angle of acceleration.
My rough sense after this experience was:
* Boids is not a simple program the way the Game of Life or Mandelbrot set is. The original paper had tons of nuance that we gloss over in the internet era.
* Every implementation I've found is either extremely sensitive to weights or does weird stuff in the steering. Stuff like subtracting velocity from acceleration when the units are different, and so on. There may be a numeric basis for them, but it's never been explained to my satisfaction. Whereas my vclamp2 idea roughly hangs together for me. And the evidence it's on the right track is that a wide variety of weights (the 10s, 20s and 40s above) result in behavior that looks right to me.
Wow! Thanks. The thought about "Boids is not a simple ..." is new to me and very good. The other vector in this is the evolution/genetic algorithm idea. It raises the question of what are the benefits of flocking? And could you plug those into a genetic algorithm to test survival.
It seems like perhaps the visual inputs are another interesting area. What do I (as a boid) do when I see one boid in front of me go right, one go left, for example.
But thanks!!
Yeah, the original paper gets into some of that. It was about 3D flocking! And Reynolds was very much thinking about what each bird sees, the minimum angle to turn to avoid a collision, etc. All on a then-powerful graphical workstation.
Creativity is not a tap that you turn on and off. Sometimes it is hard to be creative, even if you enjoy it. Even if you are good at it.
When a student feels pressure to perform, and they have an LLM to fall back on, they can poison the well of their creativity. LLMs are that tap that they can always turn on.
Then soon enough they no longer want to put in the effort. Why, when an LLM can do it perfectly right now?
> Then soon enough they no longer want to put in the effort. Why, when an LLM can do it perfectly right now?
Assuming we get to the point where the output of these models is adequate for the task, I don't think that's a problem.
Sure, we'll end up with many fewer poets, playwrights, songwriters, photographers, and the like - but those we do have will be doing it because they're passionate about it. They'll be doing it in spite of the economics, not because of them.
I'm in my 40s, and very much a software engineer at this point. I don't want to do anything else professionally if I can help it. Yet, I have a Fujifilm X-Pro3 sitting on my desk right now. I've got about $5k in lenses in my bag, and still love photography. I've done it professionally. I know with certainty that I can sustain my family on it as a career, and I know how much work that is. I'm not interested.
Instead, I'm mostly the "official photographer" for my wife's side businesses. I take on jobs here and there for friends and acquaintances when they seem fun and interesting. I do just enough to keep my name out there in the community, so I can fall back to it if my "real job" goes away unexpectedly -- which is always a possibility working for startups!
Basically, I do think creative fields will shrink significantly. I agree that we'll see AI-generated art used more and more frequently. The quality will improve - though, honestly, there is already a market of almost unlimited size that wasn't being served by humans because the expected value of those works didn't justify their creation.
In fact, that's also a good point: a rise in AI-generated art usage does not necessarily mean a fall in human-created art. I think it will mean less human art and fewer human artists in time, but that may not be the case. It very well could be the case that most brands use AI, but those that want to set themselves apart as particularly high quality or luxury will lean more heavily on human artists than ever before.
reply