The Method Maze
By Bianca Schulz
Maybe you know one of these scenarios. I have seen all of them during my career.
The backlog keeps growing, the sprint is already planned, and while you’re trying to work through the tickets, you get interrupted multiple times a day by urgent issues.
You’re extending a data model and trying to keep it stable, but there’s always a new request, a new change. You can barely keep up, and sometimes it feels like the whole thing is about to collapse like a house of cards.
You’re tasked with defining an MLOps model, but you’re not part of the machine learning engineering team. You already have a sense of what’s needed, but you’re disconnected from the engineers and their team lead heavily criticizes your work.
These are all examples of what happens when you do not know when to apply which method.
There are so many different activities in the data and AI field. I don’t need to tell you that. It’s hard to stay informed about all of them, and the job often demands specialization. The workload is frequently enormous, so it’s no surprise that some data professionals have limited background knowledge about the various methods for how work gets done. “Method” is a catch-all term here. It’s not entirely scientifically correct to call all of this simply “methods,” but this article is really about understanding what’s out there and why. We’ll look at software development models, project management frameworks, operations, and organizational design – with a few selected examples of each.
There’s actually a lot more out there. But I only want to make one point: at minimum, these are the things you should understand. When it comes to AI at scale with enterprise data, you need at least a solid grasp of these to decide what fits your own organization and how you might apply it.
My thesis: Not knowing, or half-knowing, often leads to choosing ways of working that aren’t quite right – or letting others make that choice for you, which breeds frustration. Often, important pieces get left out by accident. I’m convinced that more background knowledge about the history, purpose, and intent behind these different methods empowers data professionals to decide for themselves which approach fits their work best. The right material isn’t always easy to find online, and you have to know what you’re looking for in the first place.
So let’s understand what the difference between all of these actually is – and where they come from. Once you have a little background knowledge, the picture starts to clear up and choosing the right methods gets a lot easier.
To choose the right method – or methods – you need to understand the form of what you’re working on, the processes you need to apply, and the structure that makes sense. And for all three, you also need to understand the pace at which each one happens – or should happen.
Form:
Some systems are built like a house. The problem starts when one part needs to be rebuilt and everything else would fall apart if you changed it – think of a huge data model or monolithic legacy software that some organizations still run, especially in the public sector. Modern software is more like what you see on the right: features get added, functions get removed – it’s constantly changing shape.
Process:
When you’re building something larger that requires stability – like a data model or infrastructure – you often work in phases, because 3 requires 2 and 2 requires 1. When you make software, you work in iterations. But stop: infrastructure as a code is also software. So you need both ways of working at the same time!
Structure:
Left: a classical hierarchical org chart. Right: cross-functional teams. Sometimes you run both in parallel, and the big question is: who decides? If teams always have to escalate back to the hierarchy, the hierarchy can simply become the new bottleneck.
Time:
You could also add the dimension “time” to each: Some things change once in a lifetime or never (like the last name of someone), other things change three times a day (what the user liked with the heart symbol). Or strategy changes once a year, micro-decisions change multiple times a day. Or departments stay for decades and team setups change twice a year. For ops work, you often need to react immediately, while new features can usually be planned into a sprint, which means two different types of speed within one team.
One method for everything
Unfortunately, nearly nobody thinks about which forms, processes, structures, and time factors should apply to getting anything done. The vast majority operate in hierarchical organizations and have slapped Scrum on top somewhere without thinking about whether it makes any sense.
“Scrum is founded on […] Lean thinking”. Very few people have read the Scrum Guide and discovered that sentence, and even fewer have looked into where Lean actually comes from.
Lean is highly relevant for data and AI in many ways. I will explain that with examples. But first, what is Lean?
Toyota developed Lean Production because the company suffered from severe shortages of capital, materials, and space after World War II and simply could not afford the wasteful mass production à la Ford. To survive at all and build competitive, affordable cars, Toyota had to systematically eliminate waste, minimize inventory, and organize its production processes radically more efficiently.
Lean contains many principles, which you can find in the appendix. One component of Lean is to eliminate waste. In Japanese, that’s called Muda. Originally seven types were defined; the eighth was added later:
Overproduction – Producing more than needed, or too early. The worst form, because it causes all other types of waste.
Waiting – Idle time because something is missing. A part, a decision, a response, a machine.
Transport – Unnecessary movement of products or materials. Any movement that creates no value is waste.
Over-processing – Doing more than the customer needs. Features nobody wants. Quality nobody pays for.
Inventory – Storing more than necessary. Every stockpile costs money and hides problems.
Motion – Unnecessary movement by people. Searching, walking, reaching for something that isn’t where it should be.
Defects – Rework, incorrect results. Anything that isn’t right the first time.
Unused Potential – The eighth, later-added type. Failing to use people’s knowledge, creativity, and ideas. Not in the original TPS, but widely recognized today.
After reading the types of muda, what does it trigger for you – recognition, frustration, maybe even relief? Here are a few questions to help you spot it in your own work:
Where do we create data (pipelines, tables, dashboards, models) that nobody actually uses?
Where are we waiting – on access, approvals, upstream fixes, or someone else’s decision – before we can move forward?
Where do errors slip through and force rework downstream (quality issues, broken joins, drift, failed releases)?
Where are we carrying “inventory” that slows us down – half-finished work, stale backlogs, unused datasets, abandoned experiments?
Where are we adding handoffs, meetings, and documentation that don’t reduce risk or increase customer value?
It took until the 70s for Lean principles to become known in the Western world. By then, software already existed, but there was no Scrum yet. The Lean principles hadn’t arrived everywhere, but things were still quite different from what you all think.
Follow me and discover the forgotten world of software development models.
Software Development Models
Software was developed in Western countries for many decades through a Western lens. I can’t speak to other cultures here – I’m from Germany.
A software delivery model defines the sequence and structure of activities through which an idea gets turned into a working system. It defines phases like analysis, design, implementation, testing, and operations of software.
The Waterfall Model is from the 70s. I personally was never in a waterfall project. My first contact with the working world was 1993. After graduating, I worked as a software engineer at a large insurance company. There were mainframes and many colleagues programming in PL1. I was on the Java team, which was pretty new at the time. None of my “Hosties” – as the mainframe colleagues called themselves – were working strictly by waterfall. There was constant exchange with the business units, and things looped back all the time. The strict waterfall was practiced at most by external developers, and only when contracts were poorly structured – in which case you ended up with a pile of change requests and you were back in loops anyway. In my view, the idea that everything used to be pure waterfall before agile is a myth.
There have been several other models in software development like V-Model, Spiral Model or Rational Unified Process (RUP), just to name a few, there are many more. The Spiral Model and RUP were already incorporating iterations back in the ‘80s, so iterative development definitely isn’t that new. It’s crucial to keep this in mind; the narrative that the world only had Waterfall and then suddenly Scrum limits your room to maneuver when you’re faced with a project that requires both phases and iterations. The history of software development models proves that combining the two is not only possible but actually quite common.
There was even another method, I honestly don’t remember what it was officially called back then, but in 2005 I was working with a well-known large consulting firm at a major German freight transport company. Nobody in Germany was talking about Scrum yet, but we were working in four-week cycles. Software shipped every four weeks. Developers, business analysts, and the business itself were all sitting together in one open-plan office.
Modern data engineering and data science work looks a lot like software development these days – pipelines get versioned and tested, models get packaged and deployed, infrastructure gets written as code. So it’s worth knowing what kinds of process models exist and where they came from. Even though most software work can be iterative today, some work still has real phases – when you’re building foundational infrastructure (even as infrastructure-as-code), you often need A before B before C, and you iterate within each phase rather than pretending everything is purely iterative end to end. When no one is clear on which methods fit which purpose, you end up with projects that are “waterfall in sprints” .You all know the memes.
Now with the new possibilities through AI – vibe coding, and AI assistant coding – software development processes are experiencing a revival. People have realized that coding is only one part of the whole, and that architecture, non-functional requirements, operations, and more cannot be ignored either. Especially when it comes to agentic AI at scale, software architecture is becoming a highly relevant topic again.
I don’t know if you knew this, but in the 70s, software was also treated as daily business within the line organization. Projects came into the world of software development much later. If you work in data, you know perfectly well that a lot of data work is simply daily business and not a project at all. Project management frameworks on the other hand already existed in parallel, since the 50s, for all kinds of other things that were handled as projects like building houses or landing on the moon.
Project Management Frameworks
A huge number of people conflate software development models with project management frameworks. It happens for several reasons, and it’s worth understanding each one.
One lies in education. In the past, traditional software engineering programs barely covered project management frameworks, and business majors learned little about software development. Today, universities do teach Scrum and call it a project management framework.
Another factor is the Scrum Guide itself, the certifications, and the entire industry that sprouted up around them. The Scrum Guide is easy to grasp – at least on the surface – and there was virtually no barrier to entry for becoming a Scrum Master. A quick training, a simple certificate, and suddenly the market was flooded with career-changers who had no background in software, data engineering, or project management.
Then there are the organizations themselves. Many companies only entered this space when Scrum was already a global phenomenon, which meant they missed out on the basics entirely. On top of that, there simply wasn’t as much software or data back then – not every company had a long history with either. In those environments, you either hyper-focus on Scrum or do nothing at all – because you’re already struggling just to get things off the ground. That said, I’ve also seen companies with a long history in both software and project management where everything was handled exemplarily. You shouldn’t paint everyone with the same brush.
But Scrum alone doesn’t cover everything – and that’s where project management frameworks come in. Project management frameworks were created to manage projects of any kind – not just software.
The best-known representatives are PMBOK (PMI) and PRINCE2. I know IPMA well. It teaches competence elements like interested parties, cost & finance, procurement & contract, negotiation or legal (for the full list see the Appendix).
The reason why I mention this: a lot of it is still just as relevant today as it ever was. I went through IPMA training myself. There are very few points that could conflict with agile approaches. But there are many points you still need to know and master to deliver successfully in data and AI. So it’s not either/or – these are simply completely different things, and you need to know what matters in your own context. A lot of these competencies come into play the moment you need to convince stakeholders that certain fundamentals have to be built – so that pipelines stop failing silently, models stop drifting unnoticed, and data quality issues stop making it all the way into the boardroom dashboard.
Today, in the context of AI – but also in the data space – governance keeps coming up as a topic. The funny thing is, none of these are new disciplines: they are the old, proven project management categories that have been ignored for a while. They are now gathered under the term governance. Scrum, with its steady, human paced way of working, managed to hide a lot of this, or people somehow compensated for it along the way. With the speed of AI, that is no longer possible. Everything you do sloppily in project management will come back to haunt you with AI. Hence the call for governance. Everyone has understood this by now. The older ones among us know it is nothing new. It is requirements engineering, risk management, legal, control, cost & finance, ethics, and so on. So it is really worth pulling out the classics again.
Sometimes you hear people say, as an excuse for not doing proper stakeholder management or not thinking two steps ahead, well, we work agile. But agile is not a project management method. Let’s thoroughly clear up all the misconceptions.
Agile and Scrum
Agile is not a method It’s a mindset, built on values and principles and rooted in Lean. You’ve all heard of the Agile Manifesto. It doesn’t tell you to use Sprints or to have a Product Owner. Scrum is one example of how you can apply it in a framework – but there are many others. The Kanban method for development (distinct from its manufacturing roots) focuses on flow and limiting work in progress – no sprints, no fixed cadence. Lean Startup applies iterative thinking to business models. Teams can be highly agile without Scrum. Organizations can implement Scrum and remain entirely un-agile.
Three examples from the Agile Principles:
Business people and developers (or engineers) must work together daily throughout the project.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation (or a remote call with camera on).
Simplicity – the art of maximizing the amount of work not done – is essential.
How agile is your organization, really – just by looking at these three?
Scrum itself is explicit about this: it’s intentionally incomplete. It only defines what’s needed to apply its theory. Everything else – processes, techniques, methods – is meant to be added by the people using it.
When organizations tried to scale Scrum beyond a single team, the missing pieces – coordination, governance, portfolio planning – led many to SAFe, which reintroduced so much structure that it often prevents the original goal: being fast and nimble.
Over the years, far more software got built than before, and very different software, and for different people. There’s a huge difference between developing software for B2C – for thousands of people you’ve never met – and developing software for a handful of colleagues in another department.
The same applies to data work. Depending on what’s involved, I need different roles and different concepts. Scrum treats everything the same, and that’s a massive disadvantage of this method. Many people have recognized this and moved on to other approaches.
If you’re building a machine learning recommendation model for hundreds of sales reps across the same organization, it makes sense to aggregate needs through a product owner. But if you’re creating small dashboards for a handful of colleagues in another business department, putting a middle layer in between simply doesn’t make sense – your engineers can work directly with the people who will actually use it.
Some companies are already testing new ways of working: smaller teams, engineers and domain experts together, fewer coordinating roles. This is where it becomes very clear how an agile mindset can be applied without slavishly sticking to a framework.
It gets really chaotic when you start planning operational issues into sprints. I want to explain how operations used to work in the past, what changed through agile, what that means for data work, so you can decide whether you’re already on the right track.
Operations
We also have to look at operations – because once your pipeline, model, or AI system hits production, things break, models drift, and none of it runs itself. And it gets complicated fast when someone else is supposed to deal with it.
Classical IT operations were planning- and control-oriented. When I entered the workforce, deployments were still done manually and it was quite an effort. It took a long time and wasn’t done very often. It was risky, so it was very tightly controlled.
Agile work also changed operations. People noticed that the handoff from dev to ops was causing conflicts, both technically and interpersonally. Not everything that got developed was suited to run in production, and on the other hand the gate toward ops was too rigid and too slow.
That led to DevOps. It was a cultural shift. Developers take responsibility for operating their own software. “You build it, you run it.” A DevOps team has both developers and engineers who can cover the operations side. In large companies there’s always some central ops function left over, because otherwise it wouldn’t be efficient. That’s often called Platform Enablement.
DataOps and MLOps are the continuation of this idea – owning the full end-to-end path within one team.
The topic of operations for agentic AI systems at scale raises entirely new questions as well. What everyone has understood by now is that it makes no sense to completely separate the responsibility for operations from development.
And is operations still a separate department where you work? Org chart boxes are quite rigid. Sometimes so rigid that they actively get in the way of the work. That’s why my last stop on this journey with you is organizational design.
Organizational Design
The success of Scrum and agile work has meant that juniors often know nothing other than sprints. But it wasn’t rolled out the same way in every organization. Some introduced it only at the team level without adjusting the hierarchical organization around it, and many data teams are now paying the price for that. If you’ve ever been on an agile team where every decision had to go through someone outside the team, you know exactly what I’m talking about.
When the work spans business units, data teams, infrastructure, and AI – all on one goal – Scrum doesn’t tell you how to organize that, how to set boundaries, or how careers should move along that value chain. Because this is a question of organizational design.
In a traditional hierarchy, it’s common to sit in a “pure” data engineering team or a “pure” machine learning engineering team – even when the work actually requires tight, day-to-day collaboration with business units and other departments. The result is that engineers either never see (or fully understand) the business meaning behind what they’re building, or friction builds as misunderstandings pile up on both sides. Business teams start to feel unheard and create shadow IT to move forward on their own, while engineers retreat deeper into their engineering bubble because the criticism is constant and painful.
I’ve worked with companies that got their organizational restructuring right during the agile transformation, and I think those are the ones with a real advantage now that AI is in the picture. They oriented themselves around product or value stream oriented organizational forms.
This is where Lean comes back into play: you align your organization along how value is created for the customer from different resources, like Lean does it with the value stream (see Appendix). Essentially, an organization structured around value creation.
Think of data as a product at Netflix, and imagine the teams organizing themselves accordingly. Based on what I’ve seen of how successfully some companies have implemented this, I think the product oriented organizations are worth a second look.
Conclusion
If there’s one takeaway from the method maze, it’s this: methods only work when they match the kind of work you’re actually doing.
It will look different at every company, but the pattern is always the same: when you treat fundamentally different kinds of work with the same method, things start to break. Urgent operational work will collide with sprint planning, stable foundations like core data models will get managed like fast-changing feature work, and “central” enablement efforts like MLOps will fail when they’re designed outside the team that has to live with them.
That’s why you almost always need a mix: software development practices, project management skills, agile approaches, and some sequential work that happens in phases. You need operations close to the engineers, but also a few central capabilities and guardrails that change slowly. And more than anything, you have to think about organizational design – because org design determines the success of a data or AI team more than any method the team applies internally.
The open question for the coming years is how to build organizations that can scale AI with company data and how to do it in a way that is satisfying and motivating for everyone. None of the existing structures were really designed for that, and AI is raising entirely new questions on top.
I personally believe the best path forward is to understand proven principles and use them to develop new organizational forms, and I hope this article could help to start this conversation.
The goal is to know enough to be able to decide for yourself - and not be dependent on someone else making that call for you.
About the Author
Bianca Schulz has a long career, starting in database development and software engineering, then moving into roles as Project Lead, Scrum Master and Agile Coach. She worked inside or close to nearly 40 companies, from 5-person teams to 800,000-employee enterprises, across 4 continents and in 3 languages.