The Software Industry’s Professionalism Problem

When compared to other engineering disciplines, software engineering is still in its infancy. It's facing a professionalism problem - in this post, Senior Technical Consultant Nick Bialaszewski discusses some common issues and suggests solutions.

The Software Industry’s Professionalism Problem
Professional

Our industry has a problem with professionalism

I know that’s quite a statement to make, but I believe that it’s true. When compared to other engineering disciplines, the world of software engineering is still in its infancy and will be for some time to come. Considering that engineering as a discipline began around 4,000 years ago when the Egyptians built the pyramids, software has only been in existence for the past 70 years or so. Seventy years of history in the software industry compared to 4,000 in structural engineering: If you do the math, that’s just 1.75% of the history of engineering that has included software.

Compare that to a typical human lifetime–if you live to be 80 years old, you’ve live 1.75% of your life by the time you’re 18 months old. So, calling this a young industry is a gross understatement.

If we want to understand this problem, we have to examine how the practice of engineering software has evolved over the past 15-20 years.

In the beginning, software products were planned, designed, and built much like how houses are built, using the venerable waterfall approach. Unfortunately, it’s not the case that teams have stopped using waterfall to build software–it’s still the norm in most software development organizations I’ve worked with and for. The idea behind the waterfall approach isn’t a bad one–it’s a logical and reasonable approach to building a product that was borrowed from the conventional engineering disciplines.

However, the difference is that someone who commissions the building of a structure or machine is far less likely to “change their mind” about an attribute of the product mid-way through development. It would be crazy to ask a civil engineer to design a bridge to handle 1,000 cars per day and change the requirement to 10,000 cars per day after the concrete has already been poured. But this is the norm in the software industry, and if you can’t deal with it, many organizations will replace you with someone who can get the job done to their level of satisfaction.

Enter Agile

Agile was supposed to be the answer to all of the software industry’s delivery problems.

Agile

Over the past 15 years, managers of software teams have been promising their executives faster delivery times and an improved ability to deal with change. Always under pressure to “deliver more, faster,” these managers set goals for their teams to “go agile” and use it as a license to throw out the traditional rulebook. So, they toss some agile training into next year’s budget, order the team some pizzas, and BAM, everyone looks good.

However, with nearly every client I see, teams complain about the word “process” (they call it a dirty word), they don’t want to track their time, and they really don’t want to bother with those QA folks. Collecting and analyzing data about a team’s effort or a product’s size and quality aren’t just seen as academic and useless, but also as uninteresting and flawed approaches to obtaining insight. As a result, these teams produce poor-quality products, the organizations have no idea what they’re actually spending to develop a product, and everyone is constantly confused.

What’s worse is that management is terrified to demand time tracking or quality management practices from their teams, because they know that the team won’t buy into it. Because most of the industry functions this way, anyone who doesn’t like these rigid rules and processes can simply give their notice and walk out the door to a new company that will let them continue getting away with this behaviour. It’s simply not hip to work like the software engineers of yore, because, after all, it’s not like we’re sending anyone to the moon with our web app.

Software quality is important for all types of products, not just the ones whose failure could result in death, destruction of property, or a breach in security. Poor quality software is simply bad for business — product defects that escape our quality control processes at best negatively impact a customer’s experience, and at worst can ruin companies.

Furthermore, finding and fixing defects in a shipped product is expensive and disruptive. So, if management’s goal is to deliver more, faster, why not approach that goal with a strategy of improving product quality? High-performing teams with a professional attitude toward quality management will set team-specific goals to improve their process by examining their quality data. For instance, simply understanding the concrete phases of your team’s development workflow gives you the ability to correlate defect injection and removal actions with process phases. If this data is tracked, teams can review it regularly during sprint retrospectives. Being able to spot trends such as the percentage of defects injected in “Design” but removed in “Test” tells us that there might be something worth improving in our design and code review steps.

 

Why aren’t more teams doing this?

Firstly, it’s easy for everyone to make the argument that all of this extra work to collect the data and analyze it contributes nothing to increasing team velocity. Secondly, teams are usually so deep in their work that the idea of spending time on process improvement is the furthest thing from their minds. It’s understandable. However, the best teams out there aredoing this — maybe not all in the same way, but they’re executing on such a high level that continuous improvement is part of their DNA.

I believe that this is a shining example of agile in practice. One way to “be agile” is to work iteratively. We develop products iteratively, so why not improve our professionalism iteratively? Looking at the whole picture feels too overwhelming, but focusing on gradual improvement over time seems more approachable. Teams with top-notch processes typically have predictable releases that permit the business to plan reliably.

Agile is the mindset that the industry needs. However, I haven’t had a single client who understands that agile isn’t a prescriptive methodology. The purpose of the agile mindset — yes, mindset — is to reduce the amount of time between making a decision and seeing the consequences of that decision. Agile doesn’t say anything about how to write requirements, design APIs, or test the products we build. That’s still up to us to decide. Moreover, it’s also up to us to be professional about how we go about our business.

All of that being said, software teams still need support from management—and they’re not always getting it. Here are some common themes that I’ve seen:

Theme

Examples

Why it’s a problem

Team structure

Teams lack cross-functional perspective from QAs, ScrumMasters/Project Managers, technical writers, etc.

Teams don’t operate (even somewhat) consistently across the organization

Teams can’t be autonomous–they require too much support from other teams

The typical toss-it-over-the-wall style of software development leads to a schism between engineers and QAs. When there’s no partnership between these two perspectives, the engineers tend to ignore the quality aspects of their work and the testing effort gets squeezed. The engineers say “it’s done” and management starts bugging the QA people to “wrap it up, already!” And, that’s only the engineer-QA relationship. Engineers need to work closely with product managers, solutions architects, and UX architects to develop a quality product.

Context switching

Poor product management and overall product strategy is causing backlog volatility

In the course of my work I’ve actually heard staff such as Chief Product Officers state that “anyone can own a product initiative.” This sure sounds wonderful, but the question is: how does the senior management develop a strategy for a product when they have no control over it? They do this in the name of fostering creativity, but they’re actually spending untold amounts of money to develop product features that no one may want. The ones caught in the middle are the product managers–they end up with no ability to set the direction of their product according to what they understand from the market. Everyone is pulled in different directions and no one is able to spend the amount of time that’s required to fix the problems that everyone sees.

Unplanned work

Individuals are spending most of their time fighting fires instead of executing the business’ strategy and plans

When product quality is poor – either due to high defect injection rate, low defect removal rate, or simple “could ya just do this for me” requests – engineers and QAs lose ground when it comes to their schedules. Unplanned work is expensive and frustrating.The entire industry suffers from this. For instance, I’ve worked with clients who are terrified to rein in their releases (releasing, for example, over 50 times per day) because they don’t want to lose ground to their biggest competitors or miss out on a big political win with the company.

Identity

Organizations have no identity for how initiatives are run, products are designed, or how software is built and tested

When you look at really well-known, successful software companies, they always have an identity. Too many organizations lack this unique characteristic that comes as a result of fostering the right culture and the right governance. Organizations aren’t at the point where they can paint a picture that shows “this is how OUR company builds profitable, quality products” — and they need to get there.

Engineering governance

Organizations are lacking strong leadership when it comes to cross-cutting initiatives such as improving security practices, continuous integration, quality management processes, and tool adoption

Consistency with regard to process and technology is a huge problem in the industry. Some software managers take it too far when they try to address the problem, by dictating how teams will become more consistent through decrees and mandates. Even with these powerful positions taken up by management, I’ve still worked with few companies who have successfully implemented continuous integration and build automation in a maintainable, appropriate way.Secure coding practices are also a concern, but recent breaches have successfully scared most organizations into taking this more seriously. Nonetheless, security is still a major concern in the industry and many of today’s coders aren’t as up-to-speed on the subject as they should be.

Solving the Problem

If you’re not too depressed by now and you’re still with me, here are some ideas I have for how to begin solving these problems:

 1. Fix the Management Problem

Many software managers haven’t written a line of code in 20 years and have no idea how software is built in modern times. Hell, if you haven’t written a line of code or worked as a member of a team in the past 2 years, you can’t say that you still understand modern software development. It changes that fast. So, how do you manage these people?

You need to trust your engineers to be professionals who are capable of not only meeting your delivery goals, but also building quality products.

They also need to be kept in line. Engineers can be lazy people with a lot of power. Establishing clear, company-wide goals for cross-cutting initiatives such as security, continuous integration, quality management, static analysis, and architecture needs to be a priority.

Furthermore, these goals mustn’t be black-and-white. If management can lay out a set of principles and trust their engineers to make sound, justifiable decisions that lead to eventual implementation, they will be more successful. Lastly, it also needs to be seen as an important, strategic program within these organizations. These initiatives need to be managed by someone to ensure that things get done and get done properly.

2. Start with Skill

The skill of an engineer is the single most important factor in reducing the cost of building a product. Steve Jobs once said that the difference between an average cab driver and the world’s best cab driver might mean you get to your destination 5 minutes quicker. However, the difference between an average software engineer and the world’s best software engineer is far more dramatic. Hiring better engineers and mentoring the weaker ones should be a top priority for any company that wants to get serious about reducing its costs.

3. Stop Saving Pennies and Wasting Dollars

Companies will have a hard time recruiting and retaining top talent if they don’t foster a good working culture for those people. Adopting modern, integrated ALM and development tools is a good place to start.

Think about it — how can a top mechanic be expected to work with crappy tools? The good engineers will leave if they’re unhappy and the bad ones will stay. That’s how it goes. Turnover is ridiculously expensive. Software managers need to do a better job of being on the lookout for telltale signs of unhappiness before it goes too far and they’re being handed a resignation letter.

4. Whole-Org Agile

It’s wonderful that software teams are embracing agile, but it’s time for the rest of the organization to follow suit. Being an agile team inside a larger organization that operates the old-school way is not fun. If organizations can begin to operate more consistently throughout, they’ll realize the benefits of improved teamwork and throughput.

5. Optimize the Organization for Software Development

Agile Pods are the latest thing:

Agile pods are small custom agile teams, ranging from four to eight members, responsible for a single task, requirement, or part of the backlog. This organizational system is a step toward realizing the maximum potential of agile teams by involving members of different expertise and specialization, giving complete ownership and freedom, and expecting the best quality output.

6. Know What it Costs

I believe that it’s possible to judge the health of an organization by its ability to tell me how an initiative is going by looking at a balance sheet. If companies don’t know what they’re really spending to develop a product, how can they reasonably expect to be profitable?

In short, money hides problems, and if companies are making money in one arm of the business, it’ll simply cover up the mistakes being made in the product development area. This can be a terribly risky and dangerous position to be in. By ignoring these problems simply because money from other initiatives is keeping things afloat, senior management runs the risk of being unable to keep the organization running when markets change or the business climate suddenly tanks. Plus, if you think it’s hard to make changes when things are going well, how can you expect to make changes when you’re in crisis mode?

Companies need to do a better job of planning their product roadmaps, tracking estimated vs. actual effort, and seeking market feedback. So many organizations are reluctant to collect data, but they all wish they have it when it comes time to make tough decisions.

Time For Action

Organizations who commit to building and retaining highly-skilled teams with diverse perspectives and skill sets will be successful. By focusing on the people factor and committing to product quality, we can collectively improve the standard of professionalism in the software industry while helping our organizations produce better products for planned costs.

For more information on how Clearvision can help organizations in becoming more Agile and improving their processes, visit our services page or get in touch today

Atlasssian expert resources

Visit our blog for expert news and articles from the Atlassian world. On our resources page you will find recorded webinars, white papers, podcasts, videos and more.

The Software Blog

Read our blog for articles offering best practice advice written by Atlassian experts, as well as the latest news concerning your software.

Software White Papers and Guides

Dive deep into Atlassian software with our white papers and guides on individual tools, partner products, services, and best practices, written by the experts.

Expert Webinars

All of our webinars are pre-recorded and available to watch on-demand. Enjoy everything from partner features to application demos and updates from Atlassian experts.

Subscribe to our Newsletter

Subscribe to our Newsletter