The ability to run a suite of tests is a great way to cut down on manual testing time, whilst still having the knowledge that any new changes have not broken the codebase.
There are a number of different options when it comes to writing automation tests. Cross platforms programs such as Appium, let you write tests in one language, which can then be run across both iOS and Android. From previous experience, these tests have not been reliable. Appium’s web driver that runs the tests is very flakey on iOS and is always subject to failures when a new SDK version is released. With that in mind, this post will focus on how to run XCTests from the command line so they can then be run by a CI, on a device farm or simulator.
XCTest and XCUITest
Xcode has a built-in framework called XCTest which facilitates developers in writing unit tests, performance tests and UI tests. One main difference between unit tests and UI tests is that UI tests need a target application to be run on. Trying to access the XCUIApplication class in a target that is not set up for UI tests will return the following error.
This is why separate targets are needed to run both sets of tests.
A common practice is to create separate schemes for UI testing. By default when an Xcode project is set up it will add the unit test and UI test targets, and add them to the main scheme, so they are tested together. By creating a separate scheme, it allows testing to be done individually on different test targets. For it to be run on a CI server, the scheme will have to be shared so it is committed into source control.
Build for Testing
Before the tests can be run, they first need to be built. Commonly during development, this is done within Xcode with the ⌘⇧U command. Tests are also built when they are run using the ⌘U command. To manually build tests, Xcode supplies a set of command-line scripts, specifically a script called xcodebuild. This script allows for many of the commands which can be run through Xcode to be run on the command line. To build for testing the following parameters are needed.
-workspace – not compulsory, but needed if the project uses a workspace
-scheme – the scheme which is going to be run. In this case this the main scheme which runs both UI and unit tests.
-sdk – by default the os will be set to iphoneos, so for targeting a different sdk, this value will need to be changed.
–destination – is the platform the build is going to be run on. This is needed so it can create the build for the right architecture.
–derviedDataPath – by default this will be the derivedData directory setup in Xcode, but for the CI to know where the artefacts are built, it is easier to provide a relative directory.
build-for-testing – finally the build action is supplied, and in this case, it is build-for-testing which will create the .xctestrun file to be able to run the tests.
Once the command is successful, it will generate the relevant files within the derivedDataPath supplied.
Inside Build/Products, it will create everything needed to run the app, including .xctest files for all the schemes, and a UITests-Runner.app, which is used to run the UI tests. It will also create a .xctestrun file which explains exactly how all the tests are run, and where to find the assets to run each test.
Run the Tests
Now the tests have been built, they can be run, and to do that a different xcodebuild command is used.
The same workspace and scheme used to build the target are still needed, along with a couple of new parameters.
-xctestrun – is the path to the generated .xctestrun file created from the previous xcodebuild command.
-destination – now the test needs to be run on an actual device and the id for a device needs to be passed through.
-resultBundlePath – not needed, but makes it easier for the CI to find any generate artefacts such as screenshots from the UI tests.
test-without-building – this time the build action is test-without-building, which will take the .xctestrun file, and run it on the device.
If all the data is correct, the tests will be run on the device, and the results will be in the resultBundlePath supplied. If the .xctestrun file supplied has been built with multiple target e.g unit test and UI tests, it will run both targets on the device back to back.
With these two commands, it’s easy for any CI to build and run unit test targets. When connected to real devices it can then run UI tests across a stream of different devices.
- Cheshire Half Marathon 2020 September 27, 2020With most of 2020 being a complete right off, it meant I could knuckle down and get plenty of good training in. Races should have been on the cards with Manchester marathon, Holkham triathlon and Valencia marathon being the main goals. However, like most events...
- Helvellyn September 13, 2020Hannah and I were due to get married on the 29th August and were supposed to be spending two weeks hiking around Canada. Then the coronavirus happened and all our plans went out the window. Instead of sitting home doing nothing, we decided to take...
- Uploading content, React, GraphQL, Rails and Active Storage April 26, 2020There are some great blog posts out there on all these topics but I found a lot to be out of date. The versions I’m using are below. Rails 6.0 React 16.3.1 Apollo 3.0.0 Rails To start off with we will set up our backend...
- Full-stack introduction using Rails, React, Docker and Heroku February 9, 2020Coming from mainly a mobile background, I’ve always been interested in learning new technologies that I wouldn’t use in my day to day field. I find it fascinating that especially with today’s technology it is easier for an individual to write an entire end to...
- Malaga Marathon 2019 December 21, 2019All the training these last few months had been leading up to one goal – to run a sub-3-hour marathon in the Malaga marathon. The training had been going great, and with a 1:22 in the Tatton Half, I felt like I had a great...
- Run Tatton Half Marathon 2019 November 9, 2019It’s been a while since my last run event, with the Chester half being back in May. With my marathon training now in full swing for Malaga, I was going to treat this as a fast training run. I had no real goal time in...
- Property Wrappers October 31, 2019First introduced in Swift 5.1, property wrappers allow us devs to add our own implementation details to a property before they are initialized. Currently, by delaying initialization we can declare a property as lazy, and then return our property when it’s first requested. Property wrappers...
- Shropshire Triathlon 2019 September 15, 2019A step up was on the cards for this race, as this would be my first shot at an Olympic distance triathlon after my sprint distance race that I did back in April. I did not have the best prep going into the race, as...
- Running Xcode Tests from CI August 3, 2019The ability to run a suite of tests is a great way to cut down on manual testing time, whilst still having the knowledge that any new changes have not broken the codebase. There are a number of different options when it comes to writing...
- Method Swizzling June 29, 2019When anyone mentions swizzling I automatically get worried. Method swizzling is an Objective C runtime feature which allows for the switching of method implementations. This means any function can theoretically be switch out for another one at run time. The Objective C runtime as a...