To give a simplified but largely accurate summation: IP and Ethernet were each designed in different time periods and largely without knowledge of the other. Ethernet was historically used in such a fashion that multiple hosts (more than 2) occupied the same collision domain, that is, they were physically connected to the same cable, or through hubs that repeated frames to all interfaces without routing. This means that Ethernet required an addressing scheme so that hosts on the same media knew which frames were for them (higher-level protocols at the time did not necessarily handle this).
Ethernet's addressing scheme was not designed to accommodate large hierarchical networks and so is unsuitable for the IP use case, but more importantly, IP was designed completely separately from Ethernet, and was not used primarily with Ethernet until later, so IP could not "assume" that the layer below it handled addressing (typically there was either no layer below [point-to-point] or only a very simple one).
The result is that Ethernet and IP duplicate functionality to some extent. It is theoretically possible, although not common, to build a network which uses only layer 3 routing without any reliance on Ethernet addressing. A significant reason this is rare, arguably the most significant reason, is that IP is now carried over Ethernet a significant majority of the time and L2 Ethernet devices (like switches) require the use of Ethernet addressing for the network to function. You usually see "pure IP" in virtual networking environments where the IP is encapsulated in, well, more IP, but even then Ethernet frames are sometimes used because, well, just like network hardware, operating system network stacks generally expect them (examine, e.g., the linux bridge implementation). It is completely possible to build network stacks and network appliances which do not require the use of Ethernet but it is expensive and there's not much of a motivation to do so, and you'd run into issues with any kind of equipment not so designed.
Addressing is not the only duplicate functionality between Ethernet and IP, and it's one of the less significant ones since Ethernet addressing does provide utility even if not strictly required. Ethernet frames are checksummed, and IP headers are also checksummed, even though the Ethernet checksum is already over them. The IP header checksum exists because IP was historically carried over lower layers that did not provide integrity checking. This is basically pure wasted space in typical networks, so IPv6 drops the header checksum to remove the overhead.
In general, though, network protocols tend to make more sense when you have some awareness of the history of their development, as when you try to view the modern internet as an elegant, monolithic design as some authors attempt, a lot of things won't make sense because they simply are that way for historic reasons. Ethernet and IP were each designed in the '70s, but separately, and their use has accumulated significant cruft since then, including some radical changes in the ways that they were used (for example the transition of Ethernet from shared media to point-to-point, which occurred de facto earlier but became largely formalized with the introduction of GbE which prohibits more than two hosts in a collision domain, and of course ironically the introduction of multiple hosts in a collision domain as an even larger issue with wireless protocols, which requires additional handling below, or actually in lieu of, the ethernet layer, 802.11 being a replacement for ethernet that happens to behave similarly in many ways for compatibility).
Finally, the OSI model is something that tends to add complexity and confusion to these discussions, which is why I doggedly discourage its use in teaching. The OSI Model describes the OSI protocols, which were contemporaries competitors to the TCP/IP protocols. Arguably, one of the reasons that the OSI protocols fell out of use (in favor of IP) is exactly because they assumed seven layers, and each was fairly complex. Some OSI protocols are still in use, for example IS-IS (OSI layer 2) in the telecom industry and some backbone IP transit, but in niches and generally being replaced with IP. IP is intentionally simpler, and can be fully described using four layers, what's usually referred to as the TCP/IP model.
The OSI layers do not map 1:1 to the TCP/IP layers, even if you simply ignore the ones that map more poorly as instructors often do. Even worse, many instructors and textbook authors feel such a strong compulsion to map modern networks to the obsolete OSI model that they cram application-layer protocols into OSI layers 5 and 6 in order to have examples of them. I have seen cases as extreme as an instructor claiming that HTTP cookies represent the session layer. This kind of thing is nonsense and hinders understanding rather than contributing to it. If the OSI model is taught (not a bad idea at all as students should realize that TCP/IP is merely the popular way, and certainly not the only way), it should be taught specifically by contrasting it to the different TCP/IP model. Unfortunately few instructors and website authors today seem to even be aware that the OSI protocol stack existed separately from IP.
And, if you are wondering, yes, Ethernet can be used in a switched network completely independently from IP (although not really in a routed network unless you are generous about how you define routing). This was more common decades ago, the only equipment I have ever personally encountered that used bare Ethernet was a very outdated CNC setup.
Along with the above fantastic comment, I found https://apenwarr.ca/log/20170810 an interesting (if inflammatory/divisive) essay on the subject and its history.
Yes, that essay is outstanding! I largely left out mention of IPv6 because it's a whole different can of worms, but as that article presents, it aims to make the situation radically simpler but in practice, well, doesn't. Cue the XKCD about making a new standard.
A bit ago I touched on various competitors to IP on my blog-thing (https://computer.rip/) but I need to find time to give the topic a more thorough treatment. As with a lot of fields, you can probably learn more about what really matters in networking by studying the protocols that didn't make it than by studying the ones that did. It's hard for most people that entered the computing field in the last couple of decades to imagine IP and TCP/UDP not being the clearly correct design, but in the '80s to early '90s the expansion of microcomputers was accompanied by a flourishing of network protocols for use with them. There are multiple reasons that TCP/IP over Ethernet eventually became dominant but in the end it's mostly happenstance, it's pretty easy to imagine XNS becoming the norm if ARPANET had gone a little differently. Imagine the problems we'd be talking about today in that parallel universe, XNSv6 adoption is such a mess.
I'm honestly a bit sad to see the "all-IP" trend working its way through the telecom industry. It's reducing use of protocols like MPLS that I think are very cool. But now software-defined networking brings a whole new world of strange network technologies that we'll find ill-advised in fifty years.
Besides the choice between using IP or "bare" Ethernet, there are alternatives to IP as the layer on top of Ethernet that are used in routed networks. Two of the more-common examples historically are Novell Netware (IPX/SPX) and DECnet.
Another historic alternative was VINES IP which was used by Banyan Vines systems. Like IPX/SPX it was inspired by XNS.
What makes it particularly interesting is that Vines was based upon AT&T UNIX System V which means it is was a widely deployed commercial Unix implementation which did not use TCP/IP for it's network stack.
I don't know, it's hard to get that far with TLS because you get mired down pointing out all the problems and failed potential solutions in the CA infrastructure first. ;)
Ethernet's addressing scheme was not designed to accommodate large hierarchical networks and so is unsuitable for the IP use case, but more importantly, IP was designed completely separately from Ethernet, and was not used primarily with Ethernet until later, so IP could not "assume" that the layer below it handled addressing (typically there was either no layer below [point-to-point] or only a very simple one).
The result is that Ethernet and IP duplicate functionality to some extent. It is theoretically possible, although not common, to build a network which uses only layer 3 routing without any reliance on Ethernet addressing. A significant reason this is rare, arguably the most significant reason, is that IP is now carried over Ethernet a significant majority of the time and L2 Ethernet devices (like switches) require the use of Ethernet addressing for the network to function. You usually see "pure IP" in virtual networking environments where the IP is encapsulated in, well, more IP, but even then Ethernet frames are sometimes used because, well, just like network hardware, operating system network stacks generally expect them (examine, e.g., the linux bridge implementation). It is completely possible to build network stacks and network appliances which do not require the use of Ethernet but it is expensive and there's not much of a motivation to do so, and you'd run into issues with any kind of equipment not so designed.
Addressing is not the only duplicate functionality between Ethernet and IP, and it's one of the less significant ones since Ethernet addressing does provide utility even if not strictly required. Ethernet frames are checksummed, and IP headers are also checksummed, even though the Ethernet checksum is already over them. The IP header checksum exists because IP was historically carried over lower layers that did not provide integrity checking. This is basically pure wasted space in typical networks, so IPv6 drops the header checksum to remove the overhead.
In general, though, network protocols tend to make more sense when you have some awareness of the history of their development, as when you try to view the modern internet as an elegant, monolithic design as some authors attempt, a lot of things won't make sense because they simply are that way for historic reasons. Ethernet and IP were each designed in the '70s, but separately, and their use has accumulated significant cruft since then, including some radical changes in the ways that they were used (for example the transition of Ethernet from shared media to point-to-point, which occurred de facto earlier but became largely formalized with the introduction of GbE which prohibits more than two hosts in a collision domain, and of course ironically the introduction of multiple hosts in a collision domain as an even larger issue with wireless protocols, which requires additional handling below, or actually in lieu of, the ethernet layer, 802.11 being a replacement for ethernet that happens to behave similarly in many ways for compatibility).
Finally, the OSI model is something that tends to add complexity and confusion to these discussions, which is why I doggedly discourage its use in teaching. The OSI Model describes the OSI protocols, which were contemporaries competitors to the TCP/IP protocols. Arguably, one of the reasons that the OSI protocols fell out of use (in favor of IP) is exactly because they assumed seven layers, and each was fairly complex. Some OSI protocols are still in use, for example IS-IS (OSI layer 2) in the telecom industry and some backbone IP transit, but in niches and generally being replaced with IP. IP is intentionally simpler, and can be fully described using four layers, what's usually referred to as the TCP/IP model.
The OSI layers do not map 1:1 to the TCP/IP layers, even if you simply ignore the ones that map more poorly as instructors often do. Even worse, many instructors and textbook authors feel such a strong compulsion to map modern networks to the obsolete OSI model that they cram application-layer protocols into OSI layers 5 and 6 in order to have examples of them. I have seen cases as extreme as an instructor claiming that HTTP cookies represent the session layer. This kind of thing is nonsense and hinders understanding rather than contributing to it. If the OSI model is taught (not a bad idea at all as students should realize that TCP/IP is merely the popular way, and certainly not the only way), it should be taught specifically by contrasting it to the different TCP/IP model. Unfortunately few instructors and website authors today seem to even be aware that the OSI protocol stack existed separately from IP.
And, if you are wondering, yes, Ethernet can be used in a switched network completely independently from IP (although not really in a routed network unless you are generous about how you define routing). This was more common decades ago, the only equipment I have ever personally encountered that used bare Ethernet was a very outdated CNC setup.