Consider the Business Intelligence/Analyst roles. The industry is trying to replace people who write SQL with people who can use Tableau or similar. "Just connect to whatever datastore you have and non technical people can drag and drop."
Its got some problems:
1. They forget you need to hire many more (lower paid) people, because your output now linearly scales. Human hands have to turn the crank because its all UI-based work.
2. You still end up with very complex and disorganized business logic transform code, and now its buried in the UI. The PMs or business teams are the only ones who know what that logic is. The engineering org delivers high quality, tested datasets that are pretty raw for the purpose of answering business team questions.
The hard part was always solving the weird way business outputs are obtained from raw upstream data. Now that solution gets stored in a tableau workbook and cant be used as input for something else. It has to be copy pasted from the UI into a new tableau workbook. Well, now we bought the Tableau cloud service and our BI team can build and maintain SQL extracts in a more rigorous way. Tableau now looks like its trying to take a chunk of Databricks business here, but now its a non-engineering team doing it.
Respectfully disagree. Meaningful answers from raw data is hard, but the hardest part was always making business people to _ask the right question_.
Take as an example a typical business question: "which countries are our users from?"
But do they mean the country the user declared in the registration form?
Or country they're currently accessing your service from?
Or the one from their payment method?
Or the one where they where born?
Or the one they have citizenship from?
Or the one they're currently residing on?
Or their shipping address? Or they billing address? Or..?
If your dataset is sufficiently big, every one of those countries will output a different answer. I'm sure that in the "global fintech" space those weird cases become the norm.
Your typical lower-wage Tableau user will just look for whatever country codes show up and run a count, then declare it the truth.
A slightly smarter Tableau user will bribe an engineer to write SQL for them.
It'll take someone with knowledge of the dataset and probably the systems where the data is sourced from to push back and force the business to ask the proper question, and provide proper context.
Tableau and the like are good to replace "technical work" which is a guy copy/pasting the same SQL query into pgAdmin and emailing the resulting CSV on a daily basis, and then some, but it's not making "less skilled" UI-oriented workers to think better.
As a UI developer, I guess I had always assumed having close contact with end users and domain experts. Nothing I could articulate. It's just how things were done.
Then I served in a QA Manager role for a while. Naive me started out focusing on the QC & Test parts.
Eventually I figured out most of the value add comes from the Quality Assurance parts. Formalizing some of the stuff I used to do intuitively, like requirements gathering, sure.
But I'd say (without proof) most value (impact) came from a) asking the right questions and b) verifying the team solved the problem they had set out to solve. In other words, formalizing the team's internal feedback loops.
Alas. That was late 1990s. The Agile Manifesto cult swept aside all that silly formalism. "Too heavy!"
We now have "business analysts" backfilling QC/Test, without any training or guideance. And I haven't seen any QA style requirements gathering, analysis, and verification in probably 20 years.
> The Agile Manifesto cult swept aside all that silly formalism. "Too heavy!"
The Agile Manifesto puts close contact with the end users and domain experts as a fundamental principle (actually two principles, out of four). I do think you have the wrong culprit on your mind.
Agreed. When devs, QA, and other doers have a direct line to customers, there's no problem. In my personal experience.
That arrangement has been rare. More common is gatekeeping and incompetence. (Which may be the same thing.)
--
I've never figured out how to do "agile" QA/QC/Test. I'd don't even know what it'd look like. And, yes, my prior experiences and expectations may be keeping from seeing the new paradigm. Which is why I keep asking.
The best candidate I've read about is "Test Into Prod". But I have not yet done that strategy in real life. Soon (fingers crossed).
Oh, and "bug bashes", are pretty great. Where everyone examines logs together and either explains or eliminates exceptions. That needs to be the norm.
Well, with Scrum or whatever usually passes as agile, I have no idea either. And I imagine people can't really answer your question, because almost nobody practices the stuff on the manifesto. The motto would be certainly be to bring the customer around to specify your tests, but the actual procedure is a bit hard to imagine the details.
Anyawy, my comment was just to point that it's not exactly the manifesto stopping you.
For business ”leaders”, #1 is a feature, not a bug. Linear scaling of output to costs is something that can be modeled in spreadsheets, can be passed on and billed to a customer. Everyone in a business hierarchy feels comfortable with those types of dynamics. Whereas having a dedicated specialist who can produce 1000x output with little effort, but is not entirely sure if that will take a day or a week makes everyone uncertain and puts leaders on edge. This is a hard sell if everyone wants comfortable mediocrity.
#2 ironically means more specialists, different specialists though, but more of them to maintain all the logic.
At the end of the day it will work out as being a jobs program and will end up being more expensive, but also distribute proportionate value to the politically influential fiefdoms, and that will make it successful.
Sounds like a good business killer. First you fire all the back end folks cause they don't know nothing about the business and tableau is the new backend (nevermind that the backend folks have been absorbing and helping refine business requirements). Then when the business folks get tangled up in the web of shit they call in the tableau consultants who suck em dry and produce nothing of value.
Its got some problems:
1. They forget you need to hire many more (lower paid) people, because your output now linearly scales. Human hands have to turn the crank because its all UI-based work.
2. You still end up with very complex and disorganized business logic transform code, and now its buried in the UI. The PMs or business teams are the only ones who know what that logic is. The engineering org delivers high quality, tested datasets that are pretty raw for the purpose of answering business team questions.
The hard part was always solving the weird way business outputs are obtained from raw upstream data. Now that solution gets stored in a tableau workbook and cant be used as input for something else. It has to be copy pasted from the UI into a new tableau workbook. Well, now we bought the Tableau cloud service and our BI team can build and maintain SQL extracts in a more rigorous way. Tableau now looks like its trying to take a chunk of Databricks business here, but now its a non-engineering team doing it.
Not sure its going to work out.