CI-CD is helping software engineers give speed to the release cycle while preserving the integrity of the software. Every small change to the code is extensively tested to eliminate problems at the root. CI-CD involves automating tasks such as code building, unit testing, functional testing, and delivering to the respective environment. In the end, the efforts deliver smoother product operations and reliability.
Integration testing of an API has always been an inefficient process. Increasing automation is creating additional challenges to those found in manual testing, such as maintaining multiple tools, which are often difficult to maintain and integrate. These underlying complexities often cause the DevOps engineers to focus on other parts of the CI while shifting the focus away from API testing.
Challenges in CI-CD
The challenges involved in integrating the API testing into the CI-CD pipeline are,
- Additional tools and configuration for CI
- Different procedure for Dev and CI environment
Often the DevOps engineers have to expend unnecessary effort to make the automation test work on the CI environment. This includes additional tooling to support the headless CI environments and configuration for test execution. This configuration takes an additional toll on the engineers: as requirements change, the various configurations have to be continuously updated, otherwise resulting in CI job failures and leading to complicated troubleshooting experiences.
A Real World Problem
When a DevOps engineer decides to automate API testing with the help of CI, there are a lot of blockers ahead. First, the way the automation tests were run on the Dev environment is different from the way it will be run on the CI. This causes the engineer to introduce additional configurations and scripts as make-shift solutions. The issues are often related to how (poorly) the tools integrate into the tools involved in the CI pipelines.
Further, integration tests are not part of the code repository: they are maintained in a separate tool. Whenever the tests are modified they have to be updated, otherwise failures result and the project stalls. At this point, the developer often has to add even more tools and scripts to push the automation test through through the CI.
And even then, when the CI has been configured, the developer often encounters job failures due to mismatch in the configuration and out-ofdate tests.
Understanding the Issues
The primary problem in this story is the complexity of learning and managing multiple tools. As projects grow and more features are added, this complexity only worsens.
The result is that benefits of CI are lost due to the pain of managing the tests in multiple locations, keeping them in synchronization, etc. The DevOps engineer’s valuable time–that should be used to automate the tasks to improve the development velocity–is now being spent keeping these “custom” configurations and solutions current, just to keep the CI pipeline going.
The issues can be summed us as follows
- Multiple tools should not be used to solve a single problem
As developers add more and more tools to solve deficiencies in the ones they already use, it only creates more problems - In-house solutions should not be a response to patch faulty tooling
Engineers are always tempted to fix the missing pieces either with existing ones or with their version of it. This approach, though, is destined to reach the point where maintaining the custom solution becomes unmanageable. This can also lead to duplicate efforts and raise scalability concerns in the future.
But the biggest problem is that, currently, automated testing solutions are often not native and require extensive maintenance in the form of configuration, additional tools. They can work as a functional testing tool, but they lacking the seamless experience across the different environments, particularly CI-CD.
Still, from a product development perspective, these problems all lead to the problem.
Time
Simply put, solving CI-CD problems the way we have noted above consumes the valuable time of the engineers and lowers productivity. This can not only lead to unsatisfying work experiences and disengaged team members, but these simple problems can also penetrate to the bottom causing extensive release delays and even failure.
An Ideal Solution
The solution is choose a tool which is easy to learn, requires only the basic skills sets and integrates well with your existing build tools. A multi objective build tool can greatly improve usability, leading to increased productivity and much more satisfying work experiences (and, perhaps more importantly, satisfied team members).
Zerocode Samurai offers integration into the existing build tools like Maven and Gradle. These integrations help in maintaining integration tests along with the code in the central repository. The executions offered within Samurai are as simple as executing unit tests. Further, all the BDD test scenarios and necessary configurations are packed into your project’s build file and maintained in the same repository.
BDD integration testing with Zerocode Samurai is declarative, easily configurable for automation and can be executed by simply configuring it in your build tools.
Here are some primary examples:
Maven
<?xml version="1.0" encoding="UTF-8"?>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<includes>
<include>io.zerocode.samurai.demo.tests.get.ApiCrudTest</include>
<include>io.zerocode.samurai.demo.tests.kafka.KafkaE2eTest</include>
</includes>
</configuration>
</plugin>
mvn test
Switching Environments
mvn test -Denv=dev
mvn test -Denv=sit
or
<?xml version="1.0" encoding="UTF-8"?>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.22.1</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<includes>
<include>io.zerocode.samurai.demo.openapi.**</include>
<include>zerocode.v2.petstore.**</include>
</includes>
</configuration>
</plugin>
Gradle
test {
include 'io/zerocode/samurai/demo/tests/get/ApiCrudTest.class'
include 'io/zerocode/samurai/demo/tests/kafka/KafkaE2eTest.class'
}
gradle test
This approach can execute integration tests in the Dev environments as well as the CI environment.
Samurai supports all the CI platforms (Gitlab, GitHub actions, Bamboo CI, etc.) without any restrictions. Further, since it is already integrated into your build tool, there is no additional overhead in configuring the CI.
Docker
Projects using other languages (Python, Ruby) can benefit from the Docker image pre-baked in Samurai. This image can also be used in the CI jobs, which avoids the need for additional installations for CI pipelines.
docker run -v /my/laptop/tests-folder:/zerocode \
registry.gitlab.com/prosamurai/zerocode-cli:latest \
-a path/to/root-test-folder \
-h path/to/api_hosts.properties
Also, docker-compose can be used to fire the tests by configuring it in a compose YAML file, as follows:
docker-compose up
Build Feedback
Zerocode Samurai outputs valuable and traceable test results in the following formats:
- Build console logs
- Granular CSV test result data
- HTML report dashboard
- A persistent log file with traceable step IDs
These reports are generated in the target folder of the built project every time CI-CD job runs, which can be displayed as a build output or streamed into another location or machine for analysis.
Test failures are easily tracked using the search and filter mechanism provided by the HTML dashboard or by using the correlationId of the test step of the BDD scenario.
With the integrations at the right level, Zerocode Samurai offers a seamless experience across Dev, Test, and CI environments eliminating the need for additional tools and configurations. It is as flexible as an in-house made solution, while still preserving ease of use and scalability.
The Bottom Line
Samurai can ensure CI-CD efficiency and productivity while providing DevOps engineers a satisfying work experience. This allows them to focus on more urgent tasks instead of worrying about the maintainability of the existing tools, while also ensuring faster, more seamless build pipelines, improved release cycles and–ultimately–increasingly defect free products and services.
* The example we worked through in this article was achieved using the Zerocode Samurai tooling. Download a Free Trial of Samurai today.
Share this blog
