I can see the argument for using a custom engine if you have specific design goals that can't be met by existing engines, but that seems seems like an edge case. I think 99% of game concepts can probably be done in Unity, Godot, or Unreal.
Meanwhile you could probably surpass Handmade Hero with any off the shelf engine with a tutorial and a few hours' work, or even a project template from an asset store. The biggest problem I have with Handmade Hero is that because Casey is putting so much effort into the coding and architecture up front, the game itself isn't interesting. It's supposed to be a "AAA game" but it's little more than a tech demo.
And that's why you use off the shelf engines - they allow you to put effort into designing the game rather than reinventing the wheel.
> 99% of game concepts can probably be done in Unity, Godot, or Unreal.
The vast majority of developers use these engines, so you would expect the vast majority of games to be stuff that's easy to make within those engines.
With how samey new games are, it's hard to argue that what we see comes close to the full design space of possible interesting games. That's partially developers copying games they've seen work and sell, but it's also developers making what is reasonably easy to make within Unity or Unreal with the resources they have.
I think it's hard to argue that there exists a vast space of untapped design potential for games that can't be realized only because of the limitations of off the shelf game engines. Most people who use custom engines use them for mainstream common game concepts, because they disagree with architectural decisions about the engine itself (most likely the language being used) and they would rather start from scratch than work with the engine.
Handmade Hero can be made in Unity or Godot. So could Braid. I'm actually struggling to think of a game built in a custom engine that's so radically out of pocket design-wise that it needs a custom engine. I'm not arguing that the use case doesn't exist, I'm arguing that most of the time using a custom engine is a matter of convenience and comfort rather than creative expression, and isn't strictly necessary.
A custom engine is never _needed_ technically speaking since COTS engines are often customizable to the point where you can do anything you want with them. That doesn't mean that they don't influence the design of games, though.
There's a talk that Casey gives where he explains how he implemented the movement system for The Witness, in which he shows examples of Unity-based "walking simulator"-type games dealing with limitations of the engine in ways that The Witness was able to totally avoid. This allowed the game's artists to be creative with set design without worrying as much about performance issues or collision bugs, thus potentially opening up more design space.
Here's some praise of The Witness by Fabien Giesen:
> That’s where I am right now. I have never seen another game as cohesive as this. Not even close. Without getting into specifics or spoiler territory, I have never seen a game with such manifest intention behind every single detail, nor one where all the details cohere this well into a whole. This goes for the design and gameplay itself, but also across traditional hard boundaries. Game design and art play off each other, and both of these are tightly coupled with some of the engine tech in the game. It’s amazing. There is no single detail in the game that would be hard to accomplish by itself, but the whole is much more than the sum of its parts, with no signs of the endless compromise that are normally a reality in every project.
> A custom engine is never _needed_ technically speaking since COTS engines are often customizable to the point where you can do anything you want with them
Yeah, no. You can make a lot of things with an engine, and if you don't already have the necessary talent to make a custom engine you probably should just not.
But there are certainly games which would not exist as "Just use COTS" games. The first which comes to mind is Outer Wilds.
Outer Wilds is running a tiny solar system model. Such a model isn't stable for very long even if you had a lot of compute power which a video game console does not, but in Outer Wilds there's an excuse for that [spoiler] the sun is about to explode, it's a time loop game. Still, this is a heavily specialised engine because normal games centre on the camera or player, Outer Wilds can't do that, the model would explode almost immediately if you do that, so the centre is the sun at all times.
I didn't know that. Although it seems as though they're sufficiently far outside what its makers thinks Unity is for that their attempt to go to Unity 5 had big obstacles. Thanks for correcting me!
[Also, another way I was wrong: to make Unity work the centre of the world is the player, and so the universe is implemented in reverse, your orbit around the sun is calculated with you at the middle, this works correctly in our actual Einsteinian universe - no position is privileged, there is no "center", but it would be crazy to do the maths, for the short time Outer WIlds needs it works well enough with their simple Newtonian physics model]
Meanwhile you could probably surpass Handmade Hero with any off the shelf engine with a tutorial and a few hours' work, or even a project template from an asset store. The biggest problem I have with Handmade Hero is that because Casey is putting so much effort into the coding and architecture up front, the game itself isn't interesting. It's supposed to be a "AAA game" but it's little more than a tech demo.
And that's why you use off the shelf engines - they allow you to put effort into designing the game rather than reinventing the wheel.