Get insightful engineering articles delivered directly to your inbox.
Give it a try — it only takes a click to unsubscribe.
By

— 5 minute read

Automated Mobile Testing with Docker and Codeship

Automated mobile functional testing is hard. When you support a plethora of devices, OS versions, screen sizes and complicated math for positioning it can be very difficult to automate the validation of your application in a reliable way. So for a while we didn’t… Gasp! Hiss! I know…

As an Engineering Manager, I had some specific requirements if we were going to go down the path of investing the development time and effort into setting this up properly. We knew there was a better way, but hadn’t the expertise to execute on it. When Senior Quality Engineer, Manju Kasireddy joined the team she believed that she could accomplish our objectives and get us to a place where our developing engineers could add to a suite of automated functional mobile tests running on emulated devices in a way that didn’t interrupt their development workflow. Hope was alive to get us back to a better place.

So we began with my requirements:

Manju accepted these requirements and began planning out a better way forward…

The Solution

A Docker container built off of a GitHub repo written in JS that could be run locally with all dependencies encapsulated. A developer can add and trigger tests locally, which execute in a mobile emulator running on SauceLabs and report back to the container. When a developer completes their work and commits to the repo, CodeShip CI triggers a build and executes all tests on SauceLabs and reports the status to our #mobile-team-build Slack channel.

Test Framework Stack

The test framework is built to work inline with the rest of our application development stack, and hence built with Mocha, test framework and Chai, assertion library with Appium and WebDriver as the underlying tools to drive the user actions on Mobile and Desktop Web platforms respectively. The test repo is built as a Docker Image just like the rest of our services, so that test developers can pull in the container with all dependencies encapsulated.

Handling the Requirements

Tests must be effective

One of the difficult things about mobile functional tests is that all of the different device types, OS versions, and browser versions adds a lot of test variables. For our tests to be effective, we needed to be able to verify on the most frequently used devices that what our tests showed us was what the user would see. After investigation into the different possible tools, Manju recommended SauceLabs due to the cost, service and tools which fit with our needs. SauceLabs allows for you to dynamically generate emulated device tests with your defined OS version and execute your functional test code. It gives you a lot of information about the test run, including: a video screencast, screenshots of points of failure and logs at each point of test code execution making debugging easy and clear.

To complete end-to-end testing on this product, we needed to be able to automate both mobile and desktop web interactions. The mobile actions are automated using appium.io, which runs iOS, Android and Windows apps using the Webdriver Protocol. The desktop actions we automated using webdriver.io, which is a Node.js binding library of Selenium 2.0. The end result is that by using these tools, we were able to automate the user actions end-to-end and see the functional tests executing to validate that they are effective. Requirement pass!

Tests must be able to be added by developing engineers without any gotchas

In a startup environment developers need to be shipping working functional code as frequently as possible. In my experience, when adding functional tests is cumbersome, it typically gets abandoned as a process or test become stale and do not accurately represent what the user of the app is seeing. Thus, it was imperative that adding to the functional tests could be done without a brittle environment. Also, developers need to be able to get the tests going on their machine and keep them running effectively without assistance from Manju or any other Quality Engineer.

That meant that we needed all of the special test framework setup considerations encapsulated in the Docker container so that when a developer created a new piece of functionality or updated our application, they could pull down the container and have simple steps to begin adding to the test suite. The majority of this repo is written in JS since it is a familiar language for every developer and SauceLabs tests can be executed from within the Docker container, removing as many gotchas as possible. Gotchas contained, let’s automate!

Test runs must be automated

In most cases test automation would be a simple task with Jenkins. Since nearly all test frameworks integrate with Jenkins for most setups this would not be a major consideration. However, when we switched from our Vagrant/Chef based system to Docker/Kubernetes we also shifted away from Jenkins CI to CodeShip CI. Since CodeShip is so new there was no available documentation on triggering SauceLabs tests from Docker using CodeShip, but Manju was up to the task.

Manju added in the integration within CodeShip so that when a new commit is made to the GitHub repo, our CodeShip CI takes over and begins executing the tests found in the repo. We added a slack webhook to our team-builds channel which reports back on the success or failure of that CodeShip build closing the loop. If a build fails, everyone on the team is aware and we know that we need to look at the tests or review the SauceLabs screencast to determine the point of failure.

Conclusion

With Manju’s help we were able to overcome the struggles of automated mobile functional testing. We are now in a much better state and all of our engineers add to our test suite without pain! It’s an amazing place to be! If you’ve got any questions about mobile test automation with SauceLabs, Docker, and CodeShip leave a comment below!

By
Jeremy is an Engineering Manager at InVision.

Like what you've been reading? Join us and help create the next generation of prototyping and collaboration tools for product design teams around the world. Check out our open positions.