Producer doesn’t care how many partitions there are, it doesn’t even know about them, unless it wants to use its own partitioning algorithm. You can change the number of partitions on the topic after the fact.
There’s more to it than that. We don’t care about total order even within partitions. Every so often we get a message that must not be sent downstream until some subset of messages have been sent.
So most of the time we’re fine sending 100-200 parallel message batches, but sometimes we need to stop and wait for some batches to complete before sending any more.
We also want to control how hard we hammer specific resources downstream, which don’t correlate with the partitions we’d need. Additionally we want to scale up and scale down the parallelism per each of the previously mentioned resources depending on how fast they are coming in to maximize batch size (while keeping latency low).
There’s of course ways to do this with multiple partitions by having the consumers communicate with each other. But now we have added an additional consumer and topic to the pipeline, and an inter-consumer control system.
It was overall easier to have one consumer read from the existing topic and spawn goroutines, so that we can have more dynamic control, the ability to scale up and down immediately without worrying about rebalancing, and easy communication between threads.
Bruce Dawson says: I like to call this Dawson’s first law of computing: O(n^2) is the sweet spot of badly scaling algorithms: fast enough to make it into production, but slow enough to make things fall down once it gets there.
Fair, but `n log n` definitely is the historical "good enough to actually sleep at night" in my head, every time I see it I think of the prof who taught my first CSC course and our data structures course due to how often it came up.
Also, the wise statement that 'memory is fairly cheap compared to CPU for scaling'. It's insane to see how often folks would rather manually open and scan a 'static-on-deploy' 20-100MB Json file for each request vs just parsing it into structures in memory (where, for most cases, the in memory usage is a fraction of the json itself) and just caching the parsed structure for the length of the application.
But I also don't dabble in this area nearly enough to know whether there's years of tears and toil finding out repeatedly that O(n) is ~impossible to implement and verify :)
Depends on the constants and on the value of n. If the constant for the O(n log n) algorithm is five times that of the O(n) algorithm, the O(n) algorithm is faster for n < 100.
If you expect that n < 100 will always hold, it may be better to implement the O(n) algorithm and add a logging warning if n > 250 or so (and, maybe, a fatal error if n > 1000 or so), instead of spending time to write both versions of the algorithm and spend time finding the cut off value for choosing between the two.
Fatal errors tend to blow up in production rather than test.
One of the simplest solutions for detecting cyclic graphs is instead of collecting a lookup table or doing something non-concurrent like marking the nodes, is to count nodes and panic if the encountered set is more than an order of magnitude more than you expected.
I came onto a project that had done that before and it blew up during my tenure. The worst case graph size was several times the expected case, and long term customers were growing their data sets vertically rather than horizontally (eg, ever notice how much friction there is to making new web pages versus cramming more data into the existing set?) and now instead of 10x never happening it was happening every Tuesday.
I was watching the same thing play out on another project recently but it got cancelled before we hit that threshold for anything other than incorrect queries.
Just wanted to say you're one of my favorite posters. Can't put an exact reason on why, but at some point over the last 15 years I learned to recognize your name simply from consistent high quality contributions. Cheers.
In my mind, that's always been the point in dropping log factors. The algorithms are comparable enough that the actual implementation starts to matter, which is all we're really looking for in a Big-O analysis.
I made the “mistake” in an interview of equating two super-quadratic solutions in an interview. What I meant was what Dawson meant. It doesn’t matter because they’re both too ridiculous to even discuss.
If the cost of doing something goes above quadratic, you shouldn't do it at all. Because essentially every customer interaction costs you more than the one before. You will never be able to come up with ways to cover that cost faster than it ramps. You are digging a hole, filling it with cash and lighting it on fire.
If you can't do something well you should consider not doing it at all. If you can only do it badly with no hope of ever correcting it, you should outsource it.
All of modern Neural Network AI is based on GEMM which are O(n^2) algorithms. There are sub-cubic alternatives, but it's my understanding that the cache behavior of those variants mean they aren't practically faster when memory bound.
n is only rarely related to "customers". As long as n doesn't grow, the asymptotic complexity doesn't actually matter.
I’m on the fence about cubic time. I was mostly thinking of exponential and factorial problems. I think some very clever people can make cubic work despite my warnings. But most of us shouldn’t. General advice is to be ignored by masters when appropriate. That’s also the story arc of about half of kung fu movies.
Did chess solvers really progress much before there was a cubic approximation?
> I think some very clever people can make cubic work despite my warnings.
I think you're selling yourself short. You don't need to be that clever to make these algorithms work, you have all the tools necessary. Asymptotic analysis is helpful not just because it tells us a growth, but also because it limits that growth to being in _n_. If you're doing matmul and n is proportional to the size of the input matrix, then you know that if your matrix is constant then the matmul will always take the same time. It does not matter to you what the asymptotic complexity is, because you have a fixed n. In your program, it's O(1). As long as the runtime is sufficient, you know it will never change for the lifetime of the program.
There's absolutely no reason to be scared of that kind of work, it's not hard.
Right but back up at the top of the chain the assertion was that if n grows as your company does then IME you’re default dead. Because when the VC money runs out you can’t charge your customers enough to keep the lights on and also keep the customers.
That only matters when the constants are nontrivial and N has a potential to get big.
Not every app is a B2C product intending to grow to billions of users. If the costs start out as near-zero and are going to grow to still be negligible at 100% market share, who cares that it's _technically_ suboptimal? Sure, you could spend expensive developer-hours trying to find a better way of doing it, but YAGNI.
I just exited a B2B that discovered they invested in luxury features and the market tightened their belts by going with cheaper and simpler competitors. Their n wasn’t really that high but they sure tried their damnedest to make it cubic complexity. “Power” and “flexibility” outnumbered, “straightforward” and even “robust” but at least three to one in conversations. A lot of my favorite people saw there was no winning that conversation and noped out long before I did.
The devs voted with their feet and the customers with their wallets.
Almost every startup that has succeeded was utterly unscalable at first in tons of technical and business ways. Then they fixed it as they scaled. Over-optimizing early has probably killed far more projects and companies than the opposite.
That’s not a bold assumption it’s the predicate for this entire sidebar. The commenter at the top said some things can’t be done in quadratic time and have to be done anyway, and I took exception.
>> unless a more optimal solution does not exist
Dropping into the middle of a conversation and ignoring the context so you can treat the participants like they are confused or stupid is very bad manners. I’m not grumpy at you I’m grumpy that this is the eleventeenth time this has happened.
> Almost every startup
Almost every startup fails. Do you model your behavior on people who fail >90% of the time? Maybe you, and perhaps by extension we, need to reflect on that.
> Then we fixed it as we scaled
Yes, because you picked a problem that can be architected to run in reasonable time. You elected to do it later. You trusted that you could delay it and turned out to be right.
>> unless a more optimal solution does not exist
When the devs discover the entire premise is unsustainable or nobody knows how to make it sustainable after banging their heads against it, they quickly find someplace else to be and everyone wonders what went wrong. There was a table of ex employees who knew exactly what went wrong but it was impolitic to say. Don’t want the VCs to wake up.
Not all n's grow unbounded with the number of customers. If anything, having a reasonable upper bound for how high a n you have to support is the more common case - and you're going to need that with O(n) as well.
The first SAT solver case that comes to mind is circuit layout, and then you have a k vs n problem. Because you don’t SAT solve per chip, you SAT solve per model and then amortize that cost across the first couple years’ sales. And they’re also “cheating” by copy pasting cores, which means the SAT problem is growing much more slowly than the number of gates per chip. Probably more like n^1/2 these days.
If SAT solvers suddenly got inordinately more expensive you’d use a human because they used to do this but the solver was better/cheaper.
Edit: checking my math, looks like in a 15 year period from around 2005 to 2020, AMD increased the number of cores by about 30x and the transistors per core by about 10x.
What I’m saying is that the gate count problem that is profitable is in m³ not n³. And as long as m < n^2/3 then you are n² despite applying a cubic time solution to m.
I would argue that this is essentially part of why Intel is flagging now. They had a model of ever increasing design costs that was offset by a steady inflation of sales quarter after quarter offsetting those costs. They introduced the “tick tock” model of biting off a major design every second cycle and small refinements in between, to keep the slope of the cost line below the slope of the sales line. Then they stumbled on that and now it’s tick tick tock and clearly TSM, AMD and possibly Apple (with TSM’s help) can now produce a better product for a lower cost per gate.
Doesn’t TSM’s library of existing circuit layouts constitute a substantial decrease in the complexity of laying out an entire chip? As grows you introduce more precalculated components that are dropped in, bringing the slope of the line down.
Meanwhile NVIDIA has an even better model where they spam gpu units like mad. What’s the doubling interval for gpu units?
I’ll allow that perhaps I should have said “cubic” instead of “quadratic” - there are much worse orders in the menagerie than n^3. But it’s a constraint we bang into over and over again. We use these systems because they’re cheaper than humans, yes? People are still trying to shave off hundredths of the exponent in matrix multiplication for instance. It makes the front page of HN every time someone makes a “breakthrough”.
I was just helping out with the network at an event. Worked great in testing, but it failed in production due to unicast flooding the network core. Turns out that some of the PoE Ethernet switches had an insufficiently sized CAM for the deployment combined with STP topology changes reducing the effective size of the CAM by a factor of 10 on the larger switches. Gotta love when packet forwarding goes from O(1) to O(n) and O(n^2)! Debugging that in production is non-trivial as the needle is in such a large haystack of packets so as to be nearly impossible to find in the output of tcpdump and wireshark. The horror... The horror...
First big project I worked on a couple of us sped up the db initialization scripts so we could use a less trivial set of test data to stop this sort of shenanigans.
Things like inserting the test data first and turning on constraints and possibly indexes afterward.
Modern computers are pretty great at scanning small blocks of memory repeatedly, so n^2 can be faster than the alternative using a map in cases for small N.
I spent a lot of time fixing n^2 in blink, but there were some fun surprises:
For large N without a cache :nth-child matching would be very slow doing n^2 scans of the siblings to compute the index. On the other hand for small sibling counts it turned out the cache overhead was noticably worse than just doing an n^2. (See the end of the linked comment).
This turns out to be true in a lot of surprising places, both where linear search beats constant time maps, and where n^2 is better than fancy algorithms to compensate.
Memory latency and instruction timing is the gotcha of many fun algorithms in the real world.
This is true. But unless you are sure that current and future inputs will always be small I find it is better to start with the algorithm that scales better. Then you can add a special case for small sizes if it turns up in a hot path.
This is because performance is typically less important for the fast/small case and it is generally acceptable for processing twice as much to be twice (or slightly more than twice) as slow, but users are far more likely to hit and really burned by n^2 algorithms in things you thought would almost always be small and you never tested large enough sizes in testing to notice.
A lot of computations are really higher complexity order functions with stair steps at certain intervals based on hardware trying to pretend they are constant time. All operations cost the same amount until n doubles again and then it’s slower. If you zoom out toward infinity, the stair steps smooth out into a logarithmic curve. It becomes logarithmic in the former case and square root in the latter. Even dividing two numbers or doing a memory address lookup stops being C, which is part of why prime factoring worked for RSA for so long.
If anyone had made clockless logic work you would see that adding 1 + 1 is in fact faster than adding 2^63 + 1.
If you put enough data into a hash table the key length has to increase logarithmically to the table size in order to have distinct keys per record. Even Knuth points out that hash tables are really nlogn - something I’m pretty sure my CS professors left out. In multiple classes. Man, did I get tired of hash tables, but I see now why they harped on them. Case on point, this article.
Can you give an example? You said lots of good things in your post, but I struggling to believe this one. Also, it would help to see some wall clock times or real world impact.
Pick any compiled language and test it. Pick an algorithm making heavy use of a small (<10, maybe up to a hundred elements) hashset, and try using a linear structure instead. The difference will be most apparent with complicated keys, but even strings of more than a few characters should work.
Some example workloads include:
1. Tokenization (checking if a word is a keyword)
2. Interpretation (mapping an instruction name to its action)
3. ML (encoding low-cardinality string features in something like catboost)
4. JSON parsers (usually key count is low, so parse into a linear-scan hashmap rather than a general-purpose hashmap)
Details vary in the exact workload, the hardware you're using, what other instructions you're mixing in, etc. It's a well-known phenomenon though, and when you're doing a microoptimization pass it's definitely something to consider. 2x speedups are common. 10x or more happen from time to time.
It's similar to (but not _quite_ the same as) the reason real-world binary search uses linear scans for small element counts.
When you go to really optimize the system, you'll also find that the linear scan solution is often more amenable to performance improvements from batching.
As to how much it matters for your composite program? Even at a microoptimization level I think it's much more important to pay attention to memory access patterns. When we wrote our protobuf parser that's all we really had to care about to improve performance (33% less execution time for the entire workload, proto parsing being much better than that). You're much more likely to be able to write sane code that way (contrasted with focusing on instructions and whatnot first), and it's easier to layer CPU improvements on top of a good memory access pattern than to go the other way around.
You've got to keep in mind that computers aren't the 1-instruction-at-a-time purely sequential machines anymore.
Let's say you've got a linear array of bytes, and you want to see if it contains a specific value. What would a modern CPU need to execute? Well, we can actually compare 64 values at a time with _mm512_mask_cmp_epu8_mask! You still need a little bit of setup and a final "did any of them match" check, of course. Want to compare 512 values? You can probably do that in less than 10 clock cycles with modern machines
Doing the same with a hash set? Better make sure that hash algorithm is fast. Sure it's O(1), but if calculating the hash takes 20 cycles it doesn't matter.
A string search algorithm that uses SIMD to do quickly discard a majority of 16, 32 or 64 attempts in parallel, and then verify the surviving ones quadratically (again 16, 32 or 64 bytes at a time) can go a very long way against a sublinear algorithm that understands needle structure, but necessarily needs to process the haystack one byte at a time.
My rule of thumb for 80%-90% of the problems is, if you need complicated algorithm, it means your data model isn't right. Sure, you do need complicated algorithms for compilers, db internals, route planning et all, but all things considered, those are minority of the use cases.
This is not a complicated algorithm. A hash map (dictionary) or a hash set is how you would always do deduplication in Python, because it is easiest to write / least keystrokes anyway. That is not the case in C though, as it is much easier to use arrays and nested loops instead of hash maps.
I wrote a funny algorithm to group together words that end the same way to write them once in my binary wordlist file, since there is an array that points to the start character and a \0 to end the word. My initial solution was O(n²) but it was too slow on a real wordlist so I had to come up with something better. In the end I just sort the list with quicksort, but revert the order of the words and then the groupable ones end up nearby each other.
I'd say the exception is when `n` is under about 10, and is counting some sort of hardware constrained thing (e.g. some operation over all CAN interfaces pesent on an OBDII connector can be O(n^(2)) since n will always be between 1 and 4). If you wouldn't have to physically replace hardware for `n` to increase, you really need to avoid n^2 operations. And even then consider them carefully, perhaps explicitly failing if `n` gets too big to allow for noticing rework is needed before new hardware hits the field.
Real production use cases absolutely have small n. You don't hear about them, because it's very hard for them to cause issues. Unless the use case changes and now the n is not small anymore and nobody noticed the trap.
I have an app that's been running an O n^2 algorithm in "production" (free open source app used by various communities) for about half a year now.
It's been fine because "n" is "number of aircraft flying in this flight simulator" - and the simulator's engine starts to fail above around 2000 anyway. So even in the worst case it's still going to run within milliseconds.
For those of us without a formal comp sci background, is there a way for the IDE to detect and warn about these automatically? Or any other easy telltale signs to look for?
As a self taught dev, when I encounter nested loops, I have to mentally go through them and try to see if they iterate through each item more than once. But that's not a very foolproof method.
Too much domain knowledge for an IDE to catch. I'm self taught as well, and it comes down to spending more time thinking about the code than writing the code.
It's a fairly simple thought experiment to ask yourself what if there was 10x items in this array? 100x? That is essentially what the O(n) notation is trying to quantify. You just don't need to do it that formally.
I once looked into tree diffing algorithms (the literature is all about diffing XML even though it's really trees). The obvious dumb algorithm (which seems to be what everyone uses) is n^4.
Information is ~log(possible states) according to Shannon.
Log(25)/log(4) is 2.3. Among other things this definition has the nice property that two disks/pages/drives/bits together contain the sum of the capacities instead of their product.
What makes a computer blabbing all the time different from an ATC blabbing all the time? Just because a computer can speak some things doesn't mean it has to speak all the things.
The article says the reason is a bit different - that the route they were practicing is (in theory) sensitive information.
> But the Black Hawk did not operate with the technology because of the confidentiality of the mission for which the crew was practicing. That is because ADS-B Out positions can be obtained by anyone with an internet connection, making the system a potential risk to national security.
Seems like leaving it in listen-only mode would be wise, though.
Yes, this group transports VIPs and sometimes does so in secret. This training flight was a "simple" check-ride for the pilot (simple in scare quotes because part of the ride was using the NVGs, which strikes me as fairly ridiculous in the DCA air space).
When this specific helicopter/mission joins the route, how fast it goes, what callsign it uses, when it leaves the route, etc. may not be so public. Or at least be treated as "try not to make it unnecessarily public".
Overclassification is absolutely a thing, too. I recall when the Snowden NSA leaks came out, government employees were still forbidden from reading the documents, even if they were published in the newspapers. Pointless? Yes. But those were the rules.
> Overclassification is absolutely a thing, too. I recall when the Snowden NSA leaks came out, government employees were still forbidden from reading the documents, even if they were published in the newspapers. Pointless? Yes. But those were the rules.
Not just government employees. I was at a defense contractor at the time, and we were also instructed to not read any of the documents online, even for people who were technically cleared to read them through proper channels.
Edit: misremembering, wasn't the Snowden leaks, it was some earlier set of leaks on WikiLeaks
Training for a mission tends to mean pretending it's the real mission, as closely as possible. People fire off $100k missiles (https://www.youtube.com/watch?v=wrhybKEzb-0) so they know what it'll be like to do it in combat for real.
Competent people still make mistakes. I wouldn't want to be anywhere near DCA airspace, personally.
The ADSB is a simple switch. All it does is broadcast the position. It would have had zero impact on operational readiness. It’s not like they were actually flying “dark” - lights were on, they were in context with ATC, etc.
I don't think the Black Hawk can support ADS-B In and usually its the surveillance type aircraft that carry it. I updated my post above. There is limited cockpit space in Black Hawks anyways. There might be a specific modernization occurring for a variant of UH-60 that has ADS-B IN, but vast majority do not.
My dad's little four seat hobby plane has both In/Out. You can track him on FlightAware as a result, because it's continually broadcasting its location; it's certainly not rare or sophisticated equipment.
> The Army Black Hawk helicopter crew involved in the midair collision with an American Eagle CRJ700 last January at Reagan National Airport had turned off ADS-B because they were practicing a classified flight profile, according to a New York Times investigation.
We are both in agreement that ADS-B OUT is required. But, I am referring to ADS-B IN which most military aircraft do not have as a matter of practice. If ADS-B IN was running in addition to ADS-B OUT on both aircraft then it might have provided additional situational awareness assuming the Black Hawk pilot was operating the helicopter properly. The original comment was about putting the receiver in listening mode and that's simply not possible with the Black Hawk.
I have been running an ADS-B receiver at home for 6 years via PiAware along with an AIS receiver. So yes, low cost :)
> A majority of respondents had used ADS-B In, with 56% of respondents reported having experience with either an installed or portable system. Of the group who had experience with ADS-B In, 85% used portable systems and 30% used installed systems.
In retrospect, it was a bad plan to let a young Captain who mostly served as a liaison in DC and not a helicopter pilot to train on that route. A simpler one where she could progressively train up to would have been wiser. She also should have listened to her more well seasoned Warrant Officer copilot. ADS-B In wouldn’t have addressed any of those problems
The route they were training for was to evac government personnel during an emergency (terrorism, incoming attack, etc.). ADS-B is live location whereas transponder is delayed. In a real scenario, you wouldn’t want to be transmitting live location, since whatever the emergency is likely involves targeting of VIP government personnel. But in training, that would not effect your training, since the ADS-B is for others benefit, and doesn’t change your situational awareness or capability.
edit: To add and make clear, the route will be known for a training or real situation, but it will be delayed. So for training, turning off the ADS-B does not protect the route information and that is why there is no reason to fly with it off for training.
You are insisting that this was a training thing. But realistically, military just doesn’t like to be tracked and would rather put everyone else at risk.
It wasn't coming back from Langley. That's misinformation from people who don't know the subtleties of what's displayed by flight tracking sites. For more info see https://x.com/aeroscouting/status/1884983390392488306
It’s similar with the TSA facial recognition photos. “We delete your photo immediately” but what they don’t say is that they don’t delete the biometrics from that photo.
Literally not compelled in this case, the TSA signage says that the image capture is completely optional.
More generally, having your stuff screened for security to get on a commercial plane isn't a 4th amendment violation, the word "unreasonable" is right there in the amendment for a reason. You're in public in an enclosed flying object bringing your goods onto someone else's plane with 100+ strangers aboard, it is completely reasonable and necessary for the freedoms of everyone involved for the TSA to ensure that your stuff doesn't have dangerous objects aboard.
Don't forget that freedom also involves the freedom of other people to not be negatively impacted by you exercising your "freedom."
That is not the other option at all. The other option is essentially just the traditional screening process.
> Standard ID credential verification is in place – Travelers who decide not to participate in the use of facial recognition technology will receive an alternative ID credential check by the TSO at the podium. The traveler will not experience any negative consequences for choosing not to participate. There is no issue and no delay with a traveler exercising their rights to not participate in the automated biometrics matching technology.
My goodness this thread is just the most annoying tinfoil hat thread I've seen all day. Y'all are spending too much time online.
> The other option is essentially just the traditional screening process.
I know that, and you know that, but you have to convince the average traveler that nothing bad will happen if they say no. In the mind of the average traveler, it’s safer to just say “okay” to whatever the TSA wants. There needs to be some kind of neutral ombudsman to placate travelers’ fears of reprisal for opting to preserve their rights.
Did this change? Last time I tried to take them (ten+ years ago, because my license expired) they refused my ticket purchase because my id was expired.
For better or worse, we didn’t have to make such hard choices for the first 80 years of aviation. And Greyhound etc require photo ID these days as well
The TSA is - objectively, by their own audits - complete security theater. Why bother to defend them, exactly?
Also, the spirit of the 4th Amendment is most certainly not "here, this is the easy way!" (yes, we are conducting mass surveillance but you can sort of opt out of one piece of it by going through a manual process over here that we will make you feel like you are burdening us by requesting)
correcting disinformation isn't defending something. do you want to live in a world where we dislike someone and so we just make up random terrible things about them that aren't true, and it's fine and encouraged because they're someone we dislike, and people aren't allowed to say "hey that's not actually true, at all"
Yup,people are really good about it in my experience too. I just stand off to the side of the camera, and say "no biometrics please". They take a minute to check my documents and it's done. Try it.
I trust the TSA agents brain to not get hacked in the next 24 hours, a database run by them, not so much.
The purpose is to gather biometric data on people that will be used for future surveillance in our incipient fascist state with the implicit statement that opting out is suspicious and will lead to greater scrutiny.
Some of us want to be able to cross the country in an afternoon, and not have to spend days on a slow, uncomfortable train to make the same trip. I don't think that's unreasonable.
Certainly not unreasonable. But it does require you to commission your own transport subject to the rules that that private entity seeks to impose. Public entities which indiscriminately service residents and visitors of a given territory would obviate this requirement. But if you're in the US, good luck convincing taxpayers to agree to pay for that.
> subject to the rules that that private entity seeks to impose.
It's not the private entity taking a 3D face scan, nor are they necessarily wanting for that scan to be taken. It's federal laws and regulations being done by federal agents in spaces controlled by the federal government.
TSA is absolutely a government organization, it's a part of the Department of Homeland Security. It was created by an act of Congress, the Aviation and Transportation Security Act. You might as well argue the IRS or FBI or the US Marshalls aren't a government organization. What about absolutely absurd thing to suggest.
> The Transportation Security Administration (TSA) is an agency of the United States Department of Homeland Security (DHS) that has authority over the security of transportation systems within and connecting to the United States.
You can also walk. Lovers of freedom can walk from Manhattan to LA in 40-50 days. Of course if you look “wrong”, you’ll probably get rounded up in some flyover town.
Depends on where you walk the US is amazingly poorly situated for long walks outside of major cities. Sidewalks disappear first then lighting then one is liable to run into major stretches with no safe affordance for walking whatsoever where one is either inches from cars or in a ditch.
reply