It probes at "philosophy of computing" to ask what kind of tech is needed to solve the problem.
For many everyday tasks with personal records or small business, paper records continue to work fine provided the scale isn't too large and only a few hands are in the pot. Slightly larger, and you jump over to the spreadsheet.
Once you involve custom development you've moved from commodity solutions into the realm of architects making a bespoken design, and like with architecture, there's a strong desire to be a star and work on a monumental structure, not a shed.
But...there's a gap between spreadsheet and Bigtable. Where you can start adding requirements of more "9's" of reliability, deep access control policies, frontend dashboards and the like.
These things aren't the informational problem, they're the control of information problem. They don't follow the literal grain of the technology, but exist in an imagined universe where more and more power is consolidated into the hands of the system's owners.
That is the actual statement of purpose you have to make to justify a big tech kind of solution.
There are distributed systems that are not of that sort, the Internet itself among them. They exist, they have some value, they evolve and gain some complexity, but they don't naturally turn into platform monopolies.
And so the tension of "wanting to build distributed but being unable to justify it" is kind of specific to the economic thrust of SV style business and companies trying to ape that model. They're charged with leveraging tech to grow faster and control more, so they have to invent it. But if your business isn't that, you don't need it. But you can't conquer the world without doing that, so if you don't do that, you aren't playing for the real stakes. And that drives a certain kind of conflict in engineering orgs between pure problem solvers and the power hungry.
The only way out of thinking like that, really, is to let go and find balance. The people who are seriously happy with distributed systems work will do it with no paycheck. And for most other people, the spreadsheet, or at most a SQL database, is where it's at. For all the rest, it's the business card scene in American Psycho; the technical demonstration is simply keyfabe for one's personal advancement.
For many everyday tasks with personal records or small business, paper records continue to work fine provided the scale isn't too large and only a few hands are in the pot. Slightly larger, and you jump over to the spreadsheet.
Once you involve custom development you've moved from commodity solutions into the realm of architects making a bespoken design, and like with architecture, there's a strong desire to be a star and work on a monumental structure, not a shed.
But...there's a gap between spreadsheet and Bigtable. Where you can start adding requirements of more "9's" of reliability, deep access control policies, frontend dashboards and the like.
These things aren't the informational problem, they're the control of information problem. They don't follow the literal grain of the technology, but exist in an imagined universe where more and more power is consolidated into the hands of the system's owners.
That is the actual statement of purpose you have to make to justify a big tech kind of solution.
There are distributed systems that are not of that sort, the Internet itself among them. They exist, they have some value, they evolve and gain some complexity, but they don't naturally turn into platform monopolies.
And so the tension of "wanting to build distributed but being unable to justify it" is kind of specific to the economic thrust of SV style business and companies trying to ape that model. They're charged with leveraging tech to grow faster and control more, so they have to invent it. But if your business isn't that, you don't need it. But you can't conquer the world without doing that, so if you don't do that, you aren't playing for the real stakes. And that drives a certain kind of conflict in engineering orgs between pure problem solvers and the power hungry.
The only way out of thinking like that, really, is to let go and find balance. The people who are seriously happy with distributed systems work will do it with no paycheck. And for most other people, the spreadsheet, or at most a SQL database, is where it's at. For all the rest, it's the business card scene in American Psycho; the technical demonstration is simply keyfabe for one's personal advancement.