a. absolute lowest latency I've seen to a major IX point in seattle is about 15.8ms. More often it's 22-23ms. On a longer term, averages between 20-30ms. That's for the combined total of four 550km paths through space plus terrestrial latency from their earth station to seattle.
Keep in mind that on something like a DOCSIS3 cable coax last mile segment can have a minimum latency of 12-16ms from just the cablemodem to the local CMTS in the neighborhood, so that's actually really good. It's not as good as fiber but it's significantly better than many other options. One has to account for the need for the antenna/modem to modulate and apply FEC on the Tx at both my CPE antenna and the spacex earth station on the other end (at least a few ms).
b. no ipv6 yet. They have people who know what they are doing for neteng so I am sure it is planned.
c. all IPs are in cgnat IP space right now. Nobody is getting a discrete dhcp-assigned public ipv4 /32 yet. You can hook up whatever router you want, anything you plug into the PoE injector that is an ordinary 1000BaseT 1500 mtu DHCP client will be given an address. The router they hand out is a very bare bones 802.11ac 2x2 MIMO thing that is based on openwrt.
restrictions, you can't move it from your designated service location. That's part of the beta TOS.
Using it, it is a real 150 Mbps down x 16.5 Mbps up. I have put non technical guest users on it doing various things and from their perspective it cannot be distinguished from the separate terrestrial connection at the same location.
> b. no ipv6 yet.
> c. all IPs are in cgnat IP space
Interesting, this is more like an internet proxy than connecting devices to the internet. Hope they fix it soon. Does the TOS forbid running eg ssh or remote desktop services? Or VPNs?
It is a consumer internet eyeball service. Having an assigned routable address is not a thing in many countries, and you really shouldn't expect it as a "feature" in the future.
Eventually the Comcasts of the world will sell off the majority of the IPv4 they hold to the AWSs of the world.
I's 50%, i would not call that "not a thing in many countries"
For mobile, it does not usually matter, you have a smartphone, where you would not host something.
For a huge percentage of the developing world the only internet connection people have is a smartphone, and dedicated residential internet at a fixed address is only for the upper middle class and the wealthy.
It matters for smartphones too (think video calls, IoT applications, and home/office networks that have mobile network as uplink) but fortunately IPv6 is common there.
People have torn it down and messed about with it, I believe I recall seeing some found the test pads on the pcb and were able to watch its bootloader through a serial connection.
> b. no ipv6 yet. They have people who know what they are doing for neteng so I am sure it is planned.
heh, my assumption when I hear that people who know what they're doing are involved is the opposite ;)
Thank you for the detailed report! I'm very excited for the tech, especially once the lasers are up and it doesn't rely on the base station (i.e. I can take my uplink on a road trip)
Lol, the fact that they just went with IPv4 supports the theory that they know what they're doing.
I've read multiple tutorials on IPv6 and it's so complicated that I still can't grasp how it all works in my head. For some context, I've personally written an Ethernet IPv4 network stack with DHCP from scratch. I also worked on Ethernet switch firmware, coincidentally for SpaceX...
I guess I don't have as much low level experience as you do, but in my eyes IPv6 is literally just IPv4 with more bits. Due to the increased number of possible addresses, IPv6 is used with some different (better) concepts than IPv4. For example having (very likely) unique local networks (ULAs) is amazing when you happen to merge previously independent networks together or do some VPN stuff. You pretty much never have to calculate subnets because you'll likely never run out of addresses. And even if you somehow manage to do the impossible and fill up a /48 ULA (at which point you would have died from NATing billions of entire IPv4 address spaces with multiple layers to somehow make it all one network on IPv4), it takes just two seconds to generate a new one.
Also getting an entire public prefix assigned instead of just a single address is amazing too. On my public servers I let my ssh service listen on a randomly generated static IPv6 address from my prefix, which is not used for anything else. On that same machine I can have dozens of other public services on their own IP each (they can even all run on the same port because they're using different addresses), and when someone then decides to scan the IP of my website for interesting ports, they won't find anything.
To be honest, if someone doesn't understand IPv6, I doubt they really understand IPv4 either. Many issues with IPv6 today come from the fact that everyone and everything is so used to V4 concepts, that most products focus on IPv4 first and then half-ass IPv6 support (if they implement it at all).
The thing about IP and therefore with IPv6 is it's not just an address format - it's a set of protocols and practices so a lot of it changes.
Address asignment in an IPv6 world is more complicated - you can do DHCPv6, or your can do stateless auto-config. Which should you choose and when? Many other aspects of the stack require more knowledge.
And THEN you ALSO need to understand how v4 and v6 interacts so you can provide suitable compatibility - the various different methods of tunnelling and negotiation.
Using SLAAC is the way to go in most cases. DHCPv6 should only really be used when you have a network that is for some reason smaller than a /64.
In my opinion you shouldn't really bother with compatibility. Just do dual stack and maybe NAT IPv4 if necessary.
For my personal use cases IPv6 just proved to be much more versatile than IPv4. It did take some getting used to since it's approached differently, but learning it is very worth it and also helps using IPv4 better
Look at all the additional optional bits DCHP offers in terms of being able to communicate additional information to clients to configure themselves. Two big ones that stand out are timeservers and network booting parameters.
SLAAC is only a drop in replacement in the (typical) home.
That's just the addressing/subnetting though. Ipv6 and 4 for that matter also have a lot more going on in the packet handling and specification. and Ipv6's packet handling stuff is way more complicated than Ipv4. I was reading through just the ICMP parts and the difference is significant. I imagine the rest of it is the same.
> it's so complicated that I still can't grasp how it all works
I have seen too many people that have the same excuse. The advise I gave them is to forgot all the things learnt from IPv4 and study IPv6 as a totally new IP stack.
The problem is people look at it from IPv4 view, which is a currently a mess with IPv4 exhaustion.
Going through the trouble of implementing CGNAT and skipping IPv6 seems like a partially degraded service for the end user though. It's becoming common practice for ISPs to slap all the users between two layers of NAT and calling it a day. You can't run essentially anything on your network (not even an SSH server) without having to have a server on standby and reverse tunnels. I know there are not enough IPv4s but at least let me having the option to pay for one. Or, you know, give me an IPv6. Otherwise it's practically a networked content consumption service at this point. It's 2021, the "IPv6 is complicated" excuse must stop at some point.
IPv6 really isn't that complicated compared to CGNAT!
Of course, today IPv6 only can't be considered full internet access, so providers usually need both – and given that CGNAT gets consumers 99% of what they want/are willing to pay for, providers sometimes can't be bothered to also implement IPv6.
I used to have GCNAT + IPv6 on my provider, but the v6 part of their CPE was worse than useless – no option to open ports since "port forwarding is not needed on IPv6" (nevermind the mandatory firewall that doesn't allow inbound traffic unless completely disabled).
It can’t? I at one point messed up my firewall rules and didn’t notice that IPv4 wasn’t accessible for a few days. Nobody complained. I eventually noticed something was amiss and fixed it but really most places seems to be running dual stack. I am starting to run some IPV6 only services.
It would be great if offering public IPv6 only services was viable, but according to Google statistics (https://www.google.com/intl/en/ipv6/statistics.html) you're still cutting off about two thirds of potential users.
This will vary greatly depending on how loaded your particular cablemodem segment is, how the channel bonding is set up, how enthusiastic your neighbors are about downloading things like 65GB torrents of 2160p60 movies. If there is an 8.8.8.8 anycast node somewhere in your city, and you are kinda close to downtown, less than 1.5ms of that could be the fiber path from your ISP to their peering with google and back, and 10.5ms of it or so would be accounted for by the cablemodem part.
It will also vary greatly once starlink is loaded. The variation you're seeing is purely due to slant range. Those numbers will increase once the satellites are oversubscribed.
Very true. I'm seeing an average of 100ms to the first hop (and about the same to 8.8.8.8) during the day on mine, with lows of 50-60ms, and the occasional 10 second peaks.
I am on DOCSIS (TPG NBN) and www.google.com is 8ms, 8.8.8.8 8.5ms (actually seems long as the family is watching netflix).
Docsis modem bridged to a linux box.
22mS sounds more like mine + wifi.
I had to try my fiber connection. I get 0.6 - 0.8 ms within our town which is not that big, < 10 km from me.
To Stockholm, which is only 80 or so km away if you measure straight line, is 3.5 ms away.
Iv'e always been curious as to what adds so much latency and what I did read was that its the network equipment itself mostly. The inherent latency in the fiber should be on the order of much less than 1 ms as far as I understand. 3.33 us per km, and 3,33 * 104 (more realistic distance) is only 346 microseconds.
Its weird though, using mtr its very clear that its the "long" hop that most of the latency, or at least 2.8 ms. So is the figures for speed of light in fiber wrong, or is there something else that adds up to a lot of latency in fiber connections? Is it the optical components?
Signals in fiber do not travel at c. They travel at lightspeed in glass. Light speed in a material is c/(the refractive index of the material). For commercial glass fiber, 1.5 is a reasonable estimate for it's refractive index. So the actual time for 1 km is ~5 us.
Traceroute work by routers decrementing the ttl as the packet passes. If a router doesn’t decrement, or if your packet gets bundled into an mpls, the hops may be hidden.
Then you’ve got the buffers - you could feed 5x100M streams on 1G carriers into a 1G uplink port and have plenty of bandwidth, but if all the packets of those 100M streams arrive in 100ms, then have no packet for the next 900ms, your outbound traffic will have to either delay your packets upto 900ms, or drop them.
And of course everything depends on the implementation. On my home fiber, the minimum latency to anything IPv4 is 20ms since the PPPoE concentrators are all on the other side of the country. IPv6 on the same line (which uses IPoE instead of PPPoE) gives me 5ms to a friend in the same city.
Keep in mind that on something like a DOCSIS3 cable coax last mile segment can have a minimum latency of 12-16ms from just the cablemodem to the local CMTS in the neighborhood, so that's actually really good. It's not as good as fiber but it's significantly better than many other options. One has to account for the need for the antenna/modem to modulate and apply FEC on the Tx at both my CPE antenna and the spacex earth station on the other end (at least a few ms).
b. no ipv6 yet. They have people who know what they are doing for neteng so I am sure it is planned.
c. all IPs are in cgnat IP space right now. Nobody is getting a discrete dhcp-assigned public ipv4 /32 yet. You can hook up whatever router you want, anything you plug into the PoE injector that is an ordinary 1000BaseT 1500 mtu DHCP client will be given an address. The router they hand out is a very bare bones 802.11ac 2x2 MIMO thing that is based on openwrt.
restrictions, you can't move it from your designated service location. That's part of the beta TOS.
Using it, it is a real 150 Mbps down x 16.5 Mbps up. I have put non technical guest users on it doing various things and from their perspective it cannot be distinguished from the separate terrestrial connection at the same location.