I’ve been focusing a lot on testing and web testing in my last few posts. I have a post that talks about the different tests and the value of integration tests. Today I want to do a little story time to help illustrate some of the costs of a software bug that aren’t focused on quite as often.
Once upon a time…
Once upon a time I worked on a great product. I was on the frontend side and had a really great team. We weren’t perfect in our code but we made some pretty awesome stuff. We launched the product and we started getting some sales. We had a dedicated QA team that was solid. We had a three week cadence. This meant that we developed for three weeks, then QA tested it for three weeks, then we released it to production should it be ready.
At first, this didn’t feel like such a long time. Our company was used to older software ways with ~two releases a year. Compared to that, we were moving at lightining speed.
One of the first times I noticed a real problem came when a big sale depended on a feature we really wanted in the product anyway. We had just started our sprint and so to get it into the normal process, we were looking at 7.5 weeks (one and a half weeks until the next sprint, three weeks development, three weeks QA) before we could deliver this feature. By that time, this customer would be long gone. If we wanted to bend the rules and rush the feature out, we would have to put on pause a lot of the work that we had begun in our current sprint and rush to get this feature completed in time. I’ll be honest, I don’t remember if we got the sale or not but I will tell you that I really started to feel the pain of our previously lightning now sluggish pace as we stressed over this decision.
Why couldn’t we go faster?
I’m a big fan of the book Accelerate: The Science of Lean Software and Devops. One of the main metrics the book uses to determine high performing teams is lead time. Or…”time it takes to go from a customer making a request to the request being satisfied”.
I remember thinking that development of the feature that the customer wanted could really be done if we pushed in maybe 7 business days. I remember thinking that we could really have this feature out to production in 7 business days. That was a LOT faster than the 7.5 weeks if we did the normal process or even the 4.5 weeks if we bumped it into our current sprint.
The fact was, though, that I (and my team) didn’t have confidence that we could deliver that without breaking existing product functionality. Our frontend especially just didn’t have enough high quality tests that I believe we could trust it to catch regressions. We didn’t have an enough automated tests in place and that meant that our iteration time getting a feature request to a customer went up by almost four times. This made me sad.
What could have been
Our QA team was dang good. I really trusted them and they caught a lot of our bugs. At this time I would bet they were spending two thirds of their time on mandatory regression testing to confirm that we hadn’t broken existing functionality with each feature.
Let’s say we had automated testing that we trusted to do that. Our QA team could have been doing exploratory testing during that time and finding bugs that were hidden deep within the product. AND we could run it all in about 30 minutes. 7 business days to develop the feature and then another 30 minutes to run the automated testing. Then we deploy it out. We could have been SPEED.
Earlier tests are definitely better. Unit tests can catch the bugs during development. But web testing is something that even a third party, like myself, can do. In a few previous posts I have done web testing on both Citadel Packaging and Amateur Dota 2 League. I didn’t need access to the code to do so. I didn’t need any special credentials.
These can be automated to constantly test your site or to just test them when changes are pushed to an environment (prod, dev, staging, etc). It won’t catch bugs as early as something like unit tests but it can catch them before customers do.
If you need web testing, hopefully some of these posts have helped explain how to do so. Also, reach out and I’ll be happy to see what I can do to help you get your testing framework in place or set up web tests for you.