My experience was also that Plex is better at working around the limitations of Apple TV, compared to Jellyfin.
E.g., the native, built-in Apple TV video player expects everything in an MP4 container, and Plex appears to seamlessly repackage content into MP4, whereas you need to dig into the settings to force an MP4 container and force the native player on Jellyfin. Otherwise, Jellyfin will try to use VLCKit or some other player that has other issues (like with HDR content).
Apple TV also has very limited subtitle support in the native player, and Plex does better at accommodating that.
I'm assuming Jellyfin is closer to parity on hardware with more capable native players, though.
If you haven't already, check the CPU usage on your Jellyfin server during laggy playback. It could be transcoding on the fly, which can noticeably affect performance.
I recently used an old thin client with Kodi as Jellyfin client. Controlling it with a remote feels horrible, I’m now planning to reinstall some plain linux (instead of LibreELEC) and try out the jellyfin desktop client (it supposedly has a TV setting), surely it can’t be worse.
I've done both and the only major reason I used jellyfin was to track watching on that server rather than Infuse as for some reason Infuse tends to reset stats on me.
Of course transcoding on the fly and stuff is there but I just want to stream the direct file to the tv so none of that matters.
*edit: For anyone else reading down here the parent's solution was better for me vs Plex. Simply streaming through infuse via SMB removes all the issues I don't want to deal with with Plex as it has gotten worse over the years and has invaded my purely local setup more and more. As long as it is encoded in a way Infuse can read it works great. I didn't have a file Infuse couldn't play.
You're tied to the iOS family using the built-in library, and tags and indexing get to be a big burden if your collection grows into the 10+ TB. Jellyfin gives you location, platform, and codec independence.
My understanding is that Plex&Co's big selling point is transcoding, which might be valuable if your clients have less horsepower and don't natively support whatever format your content is in.
The plex app refuses to play 4K and says my connection is too low. Doesn’t matter if it’s wired or an ap 5 feet away, my iPhone gets almost 850mb down in the same room. If I play the 4K file directly from VLC I see absolutely 0 stuttering or glitches. I really don’t understand why plex has issues direct streaming (no transcoding). The computer has 16 cores so it’s not the problem, only plex has the issues.
Plex has a much better UI and UX than anything else.
Plex has a configurable bit rate limit on the server side for streaming. The default is too low for a lot of 4k media. Just set it to something stupid high.
That sucks, I was about to do the same (Jellyfin on the Synology, Apple TV for streaming). Seems there is no alternative to Plex that works with an Apple TV...
Actually, the Infuse app is almost perfect for Jellyfin on Apple TV. Highly recommended if you don't mind the small subscription. It takes load off of the server by playing a wide variety of codecs natively. There's also an OSS app in fast development that looks promising, but only Android/iPadOS for now.
The lack of a corporate entity behind Jellyfin makes a big impact on the quality of their streaming clients. On some platforms they work flawlessly, on others there are excellent third party clients, and some platforms don't attract a lot of open source devs and provide a lacking experience.
Jellyfin split off from Emby years ago, which does have a commercial side. Perhaps their Apple TV client is better?
But here is the thing: It is a wired Ethernet network between the server for both and the Apple TV, so there is absolutely no justification.
Jellyfin is going to be THE answer one day, but still has some growing to do.