The sad reality is that programming as we knew years ago is going into extinction. The only thing that remains is to work as library integrator and debugger for X, where X is something that is popular this week. All interesting software development left is in open source, and SEs are now just required to integrate existing libraries into a GUI done by somebody else, or a server side component that talks to that GUI. With AI coming into the picture fast, the prospects are even worse.
I think there's a kind of pyramid. From the small number of people at the top who make the hardware, to a larger number making the OS, to a larger number making various libraries, to the most people (with least technical-skill requirements) making products by tacking API's together. The bottom of the pyramid may be increasing in size the fastest, meaning the most jobs (and if the labor supply can't keep up, we get this backwards system where the highly-trained engineers make less than far-less-skilled API users), but those jobs at the top of the pyramid haven't gone away, they've just been eclipsed.
You have to be extremely senior these days to be in a position where they actually let you solve problems of any kind. (The paradox is: At that level of seniority, you are no longer capable of actually seeing the problems).
I'm a software engineer with a Ph.D. from an elite university and 10 years experience. For me it has consistently gone like this for the past 10 years, and that was in a huge variety of different companies:
Manager: We need to add field X from data source Y to interface Z. Please do that for us.
Me: This will take 93 days.
Manager: That's perfectly fine.
Me: The reason it will take 93 days instead of 3 days is that your whole architecture is a pile of shit because of problem A, problem B, and problem C. Fortunately for you, I happen to be someone who is capable of seeing and solving problems. So here is what I'd like to do for you instead: I'll use 30 days to solve problem A, 30 days to solve problem B, 30 days to solve problem C, and, with those problems solved, 3 days to add field X from data source Y to interface Z. Your spending is the same, plus it's an investment in your software architecture that will pay dividends, because there's a good chance that next week you'll want to add field V from data source W to interface Z, and solving those problems, that too will then be a 3-day project instead of a 93-day project.
Manager: Wow, you are great at identifying and solving problems. I'm glad I hired someone who is such a great problem solver. But please don't do any of that. We just need you to add field X from data source Y to interface Z without touching any of the things you just said you'd like to fix.
Me: But it will take 93 days, and you are paying me a shit-ton of money, don't you care about that money and how it's put to use?
Manager: We have enough money to not give a shit about the money we give to you, and part of the reason we are giving it to you is so I don't have to expend my mental effort evaluating proposals like the one you just made. So, please just do as you're told.
If you want creative problem solving: Go to a small company (not necessary a startup), there you will get tons of this... but sadly at a much lower paying job.
I am oficially the "Head of IT" for a small industrial company, but in reality i plan networks, program my own tools or whole systems for the company... or simple hire someone who i need. Its really nice, though not really good paid.
I totally agree. The jobs with the most creative technical problem solving tend to be the tech jobs in non-tech environments. The best job I've ever had was doing Data Science consulting for an insurance company with no Data Science competence and very little IT competence. Unfortunately, the bad-pay part, too, is very much in line with my experience.
That project was very short-lived. Since my consulting rate was 2X what they pay an employee, I couldn't convince them to carry the consulting engagement for long. They would have gladly hired me fulltime but were unwilling to pay more than X. But I knew that, working for a tech company, I could easily make 2X as a long-term fulltime salary, so I was already taking a financial hit by doing the consulting for as little as 2X.
The irony is: They ended up covering my function by buying an out-of-the-box actuarial model and services from their reinsurance provider costing them something like 10X in licensing fees. But they don't seem to care. The piece they care about is keeping the peace within the workforce and ensuring that nobody ever has a reason to be jealous of anybody else's salary and by being a one-man-consulting-show I looked to the workforce too much like I was just one of them and shouldn't get paid more. -- This is where the cycle completes itself. The insurance company's willingness to pay 10X for licensing something that the labour market can create for a price of 2X is the reason why this 10X licensing company can easily afford to pay its employees 2X in salaries. Wasting money like that is probably also a big part of the reason why the insurance company itself can't afford to pay anyone who works there more than X.
According to neoclassical economic theory such at thing shouldn't be able to exist. But neoclassical economics assumes rationality on the part of decision makers where it should be assuming petty jealousy.
Insist on talking only about the "what" and make the "how" a non-negotiable? That doesn't tend to fly when you're a techie in a tech-environment with a manager who usually has a tech background -- typically 10 years in the past, but not stopping them from thinking that they know the day-to-day better than the people who actually do the day-to-day.
Now being a techie in a non-tech environment is different (see other subthread).
"What are you working on now? Why does that part take so long? Can we skip it altogether?" They're not stupid, they know a little bit about the path from point A to point B.
I have never given an estimate that was not later called into question and asked to be reduced. Usually much sooner in fact.
In many cases, asking for an estimate is fishing for a way to make the problem easier to solve. But they don't really want to make the problem easier to solve, they just want to get it done quicker (with no regard for whether or not it makes the next problem harder to get this problem done the quicker way.)
Really nobody is thinking this hard about it though, it's basically the "quarterly earnings report" problem. If we don't look good this quarter, then there's not much hope for next quarter, since we already told the boss this one would be done this quarter, and have started talking about what they'll want next quarter, though we're not ready to talk with you about that because they could change their mind at any time, so "93 days" sounds absolutely perfect, as you'll be ready for more soul crushing just in time for next quarter.
"But they don't really want to make the problem easier to solve, they just want to get it done quicker (with no regard for whether or not it makes the next problem harder to get this problem done the quicker way.)"
It is not just the quarter result, the managers also are evaluated based on what they did the last few months. If they miss their projects for the quarter there is no promotion, no pay raise.
Quarterly, monthly, whatever, ...it is the same problem. I'm talking about the dichotomy between optimizing for short-term results and maintaining a sustainable long-term vision (which does not necessarily devolve into a failed software project / team diaspora.)
If you have a good manager, you don't want them to be promoted because then it's a crap shoot whether you get another good one or not :) of course the pay raise is a compelling argument
I was only arguing against the idea that you can successfully keep your manager in the dark, you don't want that. You want a good manager who understands the problem domain well enough to anticipate these negative outcomes and who won't fixate on quarterly (or short-term) outcomes at the expense of long-term viability.
(A good manager also optimizes for your long-term success as well as their own!)
And it's not to say that optimizing for short-term goals is necessarily wrong every time. There's YAGNI – which is a perfectly reasonable argument to make at any time, but sometimes You Are Gonna Need It and there's an objective case in favor of not taking the shortcut because it will cost you in easily predictable ways.
I think the analogy is that software engineering used to be like carpentry and woodworking, but now it's more like assembling Ikea furniture. Sure, the final result is the same from a purely practical point of view, but if you learned and loved the former, then it definitely feels like something has been lost in the latter.
Honestly, in some areas it feels like adding libraries only adds more problems and a lot more cost to integration than one expects beforehand.
I'm not saying libraries are bad in general. But when it comes to dependencies I have seen no sign of change since the left-pad debacle and that concerns me, not only because I think fixing broken dependencies and builds is one of the most frustrating things to do in development.
Preventing problems is even better than solving them, and in contrast to at least some colleagues I know I tend to prefer to "waste" time writing some of my own code, because at least I know how to debug or adapt that.
At my company, we don't really solve problems. The business creates their business system/flow, then we implement it. They handle the strategy, be just obey. We might hit technical problems, but that's not really the same as solving the business problem. Usually it would make sense technically and from a business standpoint to tweak the business system to solve the problem, but that's never an option.
In most companies you still need to understand the business requirements, but only to better implement what was already decided. Management doesn't leave much room for techies to make decisions that impact business.
