Humans are creatures of habit and routine, which makes trying new things and breaking the shell of comfort zones a challenge. But being open-minded to new changes and flexible in seeking solutions or improvements is a necessity in an industry that evolves as rapidly as the IT world does today. Before providing visionary solutions to others, sometimes it’s necessary to initiate a change within. That’s why our QA Lead, Andrei, decided to act and speak up, proposing a switch from TestRail to Azure DevOps when the limitations of TestRail and its influence on our efficiency became more evident over time.
What is Azure DevOps and TestRail?
Azure DevOps and TestRail are tools with modules that help organize the QA testing process. Azure DevOps is a service for teams to share code, track work, and ship software. It provides a flexible, extensible and feature-rich platform for continuous integration, agile planning, and continuous delivery to cover the full development life-cycle. In relation to QA testing, it integrates a fancy module named Test Plan that provides all the tools needed to successfully create and run tests.
On the other hand, TestRail is an efficient tool designed for QA and development teams to manage, track and organize their software testing effort. It’s an intuitive web-based user interface that is incredibly user-friendly and easy to use, making the creation of QA documentation a breeze. With that said, we have a clear winner regarding QA processes… or not.
The limitations of TestRail
Although our experience and history with TestRail has been very pleasant, we encountered a few limitations that needed to be addressed sooner rather than later. One major issue was finding a balanced and efficient manner to write both manual test cases and test scripts for automation. And more importantly, to easily and quickly link manual test cases and automated tests. This fed into the bigger problem of time allocation – an issue tackled frequently in our QA meetings – as allocating the right amount of time to maintain this balance was a complication we wanted to improve on.
Thus, the journey to research and find a new tool began. The goal was to find a tool that was both suitable for our needs and could overcome the limitations of TestRail.
An unexpected challenge: overcoming the resistance to something new
It was an admittedly short research journey spanning only a couple of days, partially because of the trust placed in Microsoft services. Azure DevOps offered what we were looking for: a new test management tool. It also provided a wide range of functionalities from storing meeting resolutions and delivery models in Wiki, to managing tasks in Board. As a new tool, Azure DevOps initially felt unfamiliar and a bit difficult to use. Things got easier after trying it out a few times, and our QA Lead was convinced that it would bring about efficiency and improvements to our processes moving forward. The problem: Andrei was the only one.
One of the biggest challenges was convincing fellow colleagues that Azure DevOps was more suitable for us than TestRail. Despite attempts to approach and persuade colleagues, many remained hesitant and doubtful. We decided to adopt a different approach and mindset: Let’s give it a try for a few projects and introduce it to our QA Internship program. We’ve now experienced Azure DevOps in several internal workshops and use it as a main test management tool in our Internship program – with very good results. It’s now the main tool we use for deployment, backlog management, and test management in the Carflow project.
Making the transition to Azure DevOps
A change of this magnitude for any QA department cannot overlook the importance of having a back-up plan. In this particular case, the risks lay in the Azure Test Plan being too rigid, lacking in user-friendliness, and decreasing rather than boosting our productivity in the long term. This worst-case scenario was almost realized in the first weeks as many continued to struggle with importing their old test cases from TestRail to Azure DevOps. On the verge of returning to TestRail, we decided to kick in our back-up plan and use Azure DevOps for one functionality: linking manual test cases and automated tests.
Things progressed smoothly from there, as Visual Studio’s connection with the server made linking manual test cases and automated tests exceedingly easy and efficient. As a result, we can now run an entire automated test suite from Visual Studio and report bugs directly from Visual Studio. With these improvements, we’ve reached the next step in automating our processes.
The positive outcomes don’t end there, as we’ve also experienced increased transparency for our manual test cases. With TestRail, a special account only accessible by the QA department was required to gain QA documentation. This lack of transparency made acquiring documented QA practices difficult and time-consuming. With Azure DevOps, every member of the team can view, run, or raise an issue (if that’s the case) with our QA documentation.
Azure DevOps
+ flexible and extensible, provides integration with most leading CI/CD tools and third-party services across the entire DevOps workflow
+ reliable, scalable, and globally available
+ mature, feature-rich platform with a wide range of functionalities to cover the full development life-cycle
+ easy to link manual test cases and automated tests with Visual Studio
+ increased transparency as the whole team can view QA documentation
– doesn’t surpass TestRail in terms of intuitiveness, user-friendliness, or ease of use
TestRail
+ intuitive web-based user interface is seamless, user-friendly, and easy to use
+ tool designed for QA and development teams
+ efficient tool to manage, track and organize software testing efforts
– difficult to link manual test cases and automated tests
– limited transparency as only the QA department has access to documentation
Final thoughts
We’re still riding a wave of discovery with Azure DevOps. At times, we can’t help feeling nostalgia for TestRail’s intuitiveness when things seem less straightforward with the new tool. TestRail remains a favorite amongst colleagues today due to its user-friendliness, ease of use, and its specific design for QA and development testers. Although we continue to use both tools, the advantages of Azure DevOps are now widely acknowledged and implemented by our team.
All in all, it was a valuable learning experience with an excellent outcome. Our team benefits from Azure DevOps functionalities mostly in terms of delivery, task management, code source, and automated tests. As QA members of the Carflow ecosystem, it was particularly important to fill in the need to balance manual and automated testing. And it did.
For Andrei personally, it was an experience met with enthusiasm and responsibility. It made him realize that the concept of cross-pollination is real and should always take place in our department. Individual ideas do not need to be accepted all the time, but going “against the current” to suggest new approaches and remaining flexible as a team is how we’ll continue to improve, making our work (and subsequently our clients’) more efficient.