​How to Adopt Open-Source Software as Part of Your Test Automation Stack

Digital transformation demands reduced development cycle times and high quality, driving early adopters to seek gains via open-source tools. Application leaders must take a strategic and systematic approach to adopt and support open-source testing tools or continue to fall behind digital leaders.

Key Aspects

To operationalize an open-source software testing tool strategy, application leaders responsible for application development and platforms should:

1. Build relevant technical and team skills by investing in training and reconfiguring from testing silos to continuous quality practices.

2. Determine the need for commercial versus self-support, by evaluating their organization’s risk profile and need for productivity enhancement.

3. Select effective tools by creating a weighted index of criteria, such as integration with the existing toolchain and active community support.

4. Manage tools sprawl by creating an internal tools group to assemble an open-source testing ecosystem

Adopting Open-Source Test Automation Tools

The move toward agile and DevOps requires organizations to adopt shift-left practices, and drives demand for a high degree of test automation. The majority of early adopters of these practices have relied on a variety of popular open-source tools (such as SonarQube, JUnit, Selenium and Apache JMeter). This is true across many components of the DevOps toolchain, creating a desire to follow these early adopters and their success.

However, while OSS testing tools may help organizations lower license costs and benefit from community contribution, ad hoc adoption without adequate planning may lead to a lack of success, higher maintenance costs and a fragmented toolset. The pace of innovation in open source, and a growing number of commercial offerings, will continue to drive organizations toward open-source solutions. There are a broad number of components, frameworks and layers that must be woven together, and the individual pieces change frequently.

Understanding the benefits and risks of open-source testing tools, and creating a well-defined OSS testing tools strategy, are key to achieving sustainable success. Because agile and DevOps practices are built upon collaborative work and an interconnected toolchain, Leading analysts recommend not looking at OSS testing tools abstractly from the rest of the toolchain. Instead, organizations should understand their overall needs and create a DevOps toolchain.

This strategy will help decrease technical complexity, and either streamline the process to implement open-source test automation as part of the complete DevOps toolchain (see Figure 1) or help determine if it is a good fit for your organization.

​

Four Steps to Adopt Open-Source Test Automation Tools

​Figure 1. Four Steps to Adopting Open-Source Test Automation Tools

#1: Build Relevant Technical and Team Skills

Technical Skills

Open-source testing tools are built by developers, for developers. It is true that many have record/playback front ends, but this is a poor way to create maintainable tests. Therefore, organizations must plan to:

1.    Rely on developers to create automation (relegating testers to manual exploratory tests)

2.    Hire/train testers with a basic level of coding skill

3.    Utilize commercial solutions that leverage open-source tool foundations, but provide a layer of insulation (such as object repository, keyword framework or model-driven), which in trade will create vendor dependency

Open-source software is guided by a mindset that is of the developer, by the developer and for the developer. If your testers do not have a developer mindset or skills, raw open-source tools will not be a good match. This is about more than the ability to script a test. In many organizations, this has led to the use of software development engineers in test (SDET), who can participate in the design and development of the application, as well as in the design and building of test automation. These individuals need strong collaboration skills, should understand design patterns and be able to support BDD and TDD efforts. In short, these are developers focused on the testing efforts (see Table 1).

Leading organizations have blended the roles of the developer and tester. They are all part of the same team, and both will be involved in ensuring software quality and testing. As a result, the organization shifts from testing as an event to a continuous quality approach. It should also be noted that the team will need to understand a number of components that make up the overall “code pipeline.” This is the collection of source repository, a continuous integration engine and the various services this engine will call to build and test the software.

Table 1: Example Skills to Develop

DevOps Skills Required

Beyond the actual test automation tool, the team will need to understand a number of other OSS components, including interactions with the source repository (typically Git) and scripting and configuring the continuous integration engine (such as Jenkins). Testers must understand the mechanics of the workflow and the practices that should be utilized to support CI/CD. Thus, a key element of building a talent pool isn’t just the technical understanding of the OSS tools, but the culture and practices.

Application leaders should assess the ability of the team members to manage all artifacts associated with application delivery (including tests and configurations) in the code repository. This will come naturally for developers, but testers with little programming background will need to be comfortable with the tools and branching strategy defined by the team. In addition, rather than developing and testing being two silos that pass things back and forth, quality should be a team goal and the responsibility of a single team.

Moving to OSS requires a continuous learning mindset to keep up with the pace of change in OSS testing tools. Note that frequent updates to open-source testing tools can break working tests, and so users should version the tools and frameworks used with the software under test. The key is to ensure that deploying and supporting OSS testing tools becomes a repeatable process without creating a dependency on any single individual. We often see organizations that lack institutional knowledge of the tools they use. This is usually the case when a few sysadmins and developers build a CI/CD toolchain, but the knowledge of the toolchain is not disseminated. DevOps team members must share expertise through trainings, documentation and participation in communities of practice dedicated specifically to OSS adoption and even more broadly to develop communities of practice (CoP) or “guilds.” This will encourage continuous learning.

Culture, Training and Collaboration

IT product teams will need to collaborate with the OSS community not only to seek support for deployment, but also to give feedback and influence the direction of the tools. The active collaboration with the open-source community aligns well with the goals of DevOps — an accelerated innovation cycle and reduced time to value. However, this involves substantial investment of time and effort to maintain the tools in the long term. Many organizations devote a small but fixed amount of time to contribute to the open source they use.

Training is available in many formats, be it a wide variety of books, free online courses or commercial online tools (such as Pluralsight, Udemy and others). We also suggest that organizations build internal competency centers that capture best practices and create an internal sense of community. Note, additionally, that it is not enough to train developers on a single tool. Organizations must also include practices that enable shift-left and continuous quality focus by the team.

Overall, shifting to OSS means a shift in the tools, techniques and structure, and this needs the

development of change leadership in the organization. This can begin with the formation of a team that will pilot solutions and practices, and that can act as a support and leadership team. Leading analysts have developed the ESCAPE model to aid CIOs in achieving sustainable. Culture shifts can also be brought about by introducing small changes or hacks.

#2: Determine the Need for Commercial Support Versus Self-Support

Determine the need for commercial support for open-source testing tools, as it is easy to conflate the idea of “open” as being “free of all costs.”

It is true that OSS does not incur license fees. However, application leaders incur training and operational costs to take the tool from prototype use to full production (see Figure 2).

​

Support for Open-Source Testing Tools​

​Figure 2. Balance Value of Self-Support Versus Commercial Support

A focus on self-support to lower the total cost of ownership (TCO) can lead to long-term costs. All too often, organizations underestimate OSS testing complexity (especially in regard to scripting, maintenance and upgrades). The market does have solid community support, but finding the right information for the right version and the matching frameworks will create a cost to productivity. In a practice built around continuous integration, it is critical that the tools are stable.

The decision between self-support and external support must be based on a combination of factors that go beyond the availability of technical skills. These factors include level of stability (time in market/maturity), risk, the ability to scale tests to cover different scenarios (browser, devices and distributed users), skills match to users, budget, and bandwidth to dedicate resources.

While the most common test automation tools are Selenium, Appium and Apache JMeter, there are many other components that either build on top of these or support common UI frameworks. There are several commercial tools that also support Selenium in particular (see Table 2). These can include simply providing organization and coding support, and extend up to adding features such as object repositories to aid with reuse and maintenance. Other tools are built on top of the core Selenium WebDriver test execution engine layering a proprietary framework and tools on top. These support greater ease of use and reduced maintenance via model-driven approaches and native support for various widget frameworks. This market is very dynamic and has been driven by several investments and acquisitions. We expect that to continue, and organizations must determine the balance of training versus ease of use and value provided by the ease versus the cost of the commercial product.

Table 2: Sample Commercial Open-Source Support

We expect growing support for open-source testing from Amazon, Google and Microsoft. Microsoft recently announced that it would drop support for its own Coded UI framework to focus on Selenium and other open-source technologies. In these cases, expect to see service-based pricing along with scale and support.

Additionally, the large system integrators have used OSS testing tools for several years and have leveraged these to create extended offerings. Generally, they have not entered the tools space itself. However, large system integrators are also contributing to the open-source codebase, as well as contributing a great deal of content on how to utilize the tools (via formal online training and YouTube videos). For test execution itself, OSS testing tools are well-integrated into the DevOps toolchain (see Figure 3), where the primary interface is via CI. However, how these ties into the management tools or features like change-based test execution tend to be part of what commercial offerings add.

​

​Figure 3. The DevOps Toolchain

Open-source software assets share the same core risks as any other software technology, yet undermanagement stands out as a risk that is particularly associated with OSS solutions more than others.

1.    Tackle open source (either commercially supported or self-supported) as inevitable investments that by being properly managed, will yield considerable TCO and “business value” benefits. When unmanaged (or undermanaged), these same OSS technologies will instead introduce considerable technical, security and legal risks to the enterprise.

2.    Scrutinize successful open-source software investments by implementing a robust enterprise governance program that is driven by the CIO from the top down and supported by a wide range of key stakeholders.

3.    Initiate open-source management, but do not conclude, by instilling a foundational enterprise software asset management strategy. The OSS model introduces new rules that requires an evolution of these efforts but should not be considered an entirely separate task altogether.

#3: Select Tools Based on a Weighted Index of Criteria

DevOps Toolchain methodology directs users to not be tool directed but to start with desired business outcomes, the practices that are required to create this outcome, the skills and training needed to support these practices and then selecting tools. Another element of selecting tools is how they will fit together with other tools that are already in use. Application leaders must follow a four-step process to select open-source tools:

1.    Define the functional requirements of the toolchain to support business objectives. With testing tools this may be supporting new technologies such as multimodel or adopting practices to shift-left with quality.

2.    Explore practices with small teams that will drive continuous quality (e.g., BDD, TDD, pairing, static analysis, code review).

3.    Once practices are understood and the value is understood, look where additional tooling will aid in automation, consistency, and a faster build/evaluate loop.

4.    Analyze features and functionalities of associated tools to assess them on their ability to support the identified activities.

5.    Assess each of the tools against a set of agreed-upon criteria to select the tools for your toolchain.

We find that testing tools are generally implemented independently to support a specific task (e.g., function automation, unit testing, code linting) and often these specific actions are independent simplifying integration to the toolchain. However, this can lead to a fragmented view of quality that doesn’t provide both a realistic view and limits the ability to understand flow and determine bottlenecks, root cause and defect containment.

This means that you must focus on the data format of test results. It should be noted that the integration needs will be different based on the role of the specific OSS tool. Much of the focus of OSS testing currently is on test execution which requires a limited amount of integration beyond interacting with the CI system. However, to meet the capabilities of commercial offerings users may see trade-offs. Commercial tools are increasing their use of AI and ML to automatically create, maintain, and select which tests need to run based on risks and changes. Some of these tools are layered on top of open source and this should be part of the selection matrix.

#4: Manage Tools Sprawl

Open-source testing tools tend to be much more componentized than commercial solutions. This means that to create a complete solution, organizations must create their own combination of tools and frameworks. This may be purely open source or a combination of OSS and commercial components. This drives the need to establish an internal team to manage the formation and evolution of selected open-source technologies.

It is possible you already have a team or process that is vetting open source. This may exist in a procurement, operations or sometimes security setting, as there are multiple reasons for this type of practice. The team building an OSS testing platform needs to coordinate with the greater OSS internal community. This should be done both through formal and informal methods. Formal methods (managed by a COE) include a corporate set of guidance on acceptable licenses and an internal distribution point. Informal support will come in the form of communities of practice and the use of information sharing portals. This is especially true when not utilizing commercially supported components but, in all cases, an internal distribution team must track the latest security patches, zero-day vulnerabilities, feature improvements and bug fixes. This should lead to the creation of internal distribution points for vetted tool collections and a process to evaluate new options.

There are a number of tools that can aid with ensuring that the open-source components of a specific version are secure and viable.

A key element of the internal tools group is ensuring that all of the selected version of components fit together and support the system under tests technology. For this reason, the specific configurations should be stored as part of the development project to simplify later maintenance projects. We are seeing an increasing number of options designed to be used as part of a containerized development environment and expect this to grow in use. The use of an internal distribution portal will simplify this and encourage users to only utilize the vetted components.

​

​

​