The ability to run a suite of tests is a great way to cut down on manual testing time, whilst still having the knowledge that any new changes have not broken the codebase.
There are a number of different options when it comes to writing automation tests. Cross platforms programs such as Appium, let you write tests in one language, which can then be run across both iOS and Android. From previous experience, these tests have not been reliable. Appium’s web driver that runs the tests is very flakey on iOS and is always subject to failures when a new SDK version is released. With that in mind, this post will focus on how to run XCTests from the command line so they can then be run by a CI, on a device farm or simulator.
XCTest and XCUITest
Xcode has a built-in framework called XCTest which facilitates developers in writing unit tests, performance tests and UI tests. One main difference between unit tests and UI tests is that UI tests need a target application to be run on. Trying to access the XCUIApplication class in a target that is not set up for UI tests will return the following error.
This is why separate targets are needed to run both sets of tests.
Multiple Schemes
A common practice is to create separate schemes for UI testing. By default when an Xcode project is set up it will add the unit test and UI test targets, and add them to the main scheme, so they are tested together. By creating a separate scheme, it allows testing to be done individually on different test targets. For it to be run on a CI server, the scheme will have to be shared so it is committed into source control.
Build for Testing
Before the tests can be run, they first need to be built. Commonly during development, this is done within Xcode with the ⌘⇧U command. Tests are also built when they are run using the ⌘U command. To manually build tests, Xcode supplies a set of command-line scripts, specifically a script called xcodebuild. This script allows for many of the commands which can be run through Xcode to be run on the command line. To build for testing the following parameters are needed.
-workspace – not compulsory, but needed if the project uses a workspace
-scheme – the scheme which is going to be run. In this case this the main scheme which runs both UI and unit tests.
-sdk – by default the os will be set to iphoneos, so for targeting a different sdk, this value will need to be changed.
–destination – is the platform the build is going to be run on. This is needed so it can create the build for the right architecture.
–derviedDataPath – by default this will be the derivedData directory setup in Xcode, but for the CI to know where the artefacts are built, it is easier to provide a relative directory.
build-for-testing – finally the build action is supplied, and in this case, it is build-for-testing which will create the .xctestrun file to be able to run the tests.
Once the command is successful, it will generate the relevant files within the derivedDataPath supplied.
Inside Build/Products, it will create everything needed to run the app, including .xctest files for all the schemes, and a UITests-Runner.app, which is used to run the UI tests. It will also create a .xctestrun file which explains exactly how all the tests are run, and where to find the assets to run each test.
Run the Tests
Now the tests have been built, they can be run, and to do that a different xcodebuild command is used.
The same workspace and scheme used to build the target are still needed, along with a couple of new parameters.
-xctestrun – is the path to the generated .xctestrun file created from the previous xcodebuild command.
-destination – now the test needs to be run on an actual device and the id for a device needs to be passed through.
-resultBundlePath – not needed, but makes it easier for the CI to find any generate artefacts such as screenshots from the UI tests.
test-without-building – this time the build action is test-without-building, which will take the .xctestrun file, and run it on the device.
If all the data is correct, the tests will be run on the device, and the results will be in the resultBundlePath supplied. If the .xctestrun file supplied has been built with multiple target e.g unit test and UI tests, it will run both targets on the device back to back.
Conclusion
With these two commands, it’s easy for any CI to build and run unit test targets. When connected to real devices it can then run UI tests across a stream of different devices.