I agree with that point, too.

My guess is that current models are not yet able to provide consistent, effective motivation to learners over the long term. Like a lot of new educational technology, they might be fun and engaging in the short term but will lose their effectiveness as motivators once the shine has worn off.

But the models might become better motivators as they become more multimodal, so that they are able to respond in real time to the student’s tone of voice and facial expressions, and as they acquire longer context windows, so that they are able to adapt their interactions to what the student has done weeks or months earlier.

The biggest issue, I think, is what the OP raises: Whether the bots will be able to—or allowed to—acquire sufficiently human-like personalities and indentities so that they can motivate learners as human parents, teachers, and mentors do—by making the learners want to please them, to be praised by them, to avoid angering them, to emulate them.

We already have highly efficient "stand-alone" motivators, they're called video games. And for Educational video games, there's been attempts, some would say it's too difficult to integrate the two fields together, but looking at what's been attempted vs what's actually done in industry I can't say they've really tried that hard.

The problem I would imagine is that the kind of people who go on to make an educational video game probably will lack the necessary creative or aesthetic chops to do it. And no game dev is going to be dreaming of building one either. But if one were to commission a veteran studio instead we might get better results.