T’was the night before Christmas, when all through the office, not a dev was stirring, not even through our VPN service. The UI tests were set up on our CI server with care, In hopes that a release would soon be there.

The devs were at home all snug in their beds, While visions of successful builds danced in their heads. And a recent version of xcode with new tools for our app, Had just settled our brains for a long winters nap.

When out on my desk there arose such a clatter, I sprang from the bed to see what was the matter. I picked up my phone, vibrating with terror, Looked at the screen and saw there was an error.

With a lil’ ol’ icon and a familiar message on display, I knew in a moment it must be our friend Jenkins at play. He whistled “build failed” and shouted in dismay, “Assertion Failure: UI Testing Failure - Failed to perform AX action for monitoring the animations of Target Application, error: Failed to perform AXAction 2043 after 30 retries: kAXErrorServerNotFound (see <rdar://problem/15530121>)”

Traditionally at OVO, at the end of every sprint, all of the mobile devs would gather around with their test devices and a cup ‘o tea at hand and regression test the app, one feature file at a time. To newcomers, this process was useful in that it would give them a decent overview of what the app had to offer. To those of us that were already familiar with it, it was a chore, one that we didn’t look forward to. Not just because it was boring, but because it wasn’t exaustive. We knew we couldn’t possibly test our app on all combinations of device + ios version in a reasonable amount of time, not with the various accounts we get back from our servers. We compromised.

It didn’t help that Apple had neglected developing proper tools for UI Automation. But now that’s all changed. With the release of Xcode 7 and iOS 9 we finally got a functional testing solution integrated directly into Apple’s IDE and more importantly a clear statement from Apple that this is something they’re invested in for the long haul. We’re now able to quickly write up tests in a language that’s familiar to iOS developers (Swift or Obj-C) using a framework that we know inside out from typing up our unit tests, mainly, XCTest. On top of all that Xcode provides nifty little tools to help you work with your tests. It doesn’t get easier than this.

Immediately, we recorded some tests for the simplest journeys, the sort that don’t require any data from the backend. Satisified with the results, we proceeded to step 2 - UI tests for the most critical journeys. We wrote up a mock server that would run on the device and redirected all requests to localhost. In a few days our mock server had a data model in place and was able to serve it to the app. We transcribed our feature files to Swift and in less than a quarter of a sprint had our tests running on different devices with different iOS versions with data we were unable to serve up previously. As a bonus, we cleared up some bugs that were lingering in the app that we found in the process.

But for all the upsides and hype, we found things weren’t perfect in the Apple automation world. We’ve encountered a lot of bizarre bugs once we’ve started running tests on our Jenkins build server.

  1. Our tests worked fine on iPhone devices, but on iPad simulators they weren’t able to input text into secure text fields. Turning off the simulators “Connect hardware keyboard” option fixed the issue.

  2. 95% of our Jenkins builds fail because the app times out when launching after having gone through a lot of tests. This is one we still haven’t been able to solve, Apple gives no clear indication as to what may be wrong.

  3. Random issues like the one above in the christmas poem that aren’t documented.

Regardless, we’ll be sticking with this solution and count on Apple to get it’s tools in order. We’ve not automated everything just yet, but we’re hard at work at getting there. We still gather together at the end of every sprint, with our test devices and tea’s and go through regression testing. But we don’t spend as much time on it. The time we put into writing UI tests trimmed half a day off manual testing. And with manual testing we were never able to test on all devices with rare server data. Now we don’t have to, Xcode does it for us.