mokacoding

Some weeks ago I decided to focus my writing on mokacoding more on unit and acceptance testing, automation, and productivity, mainly in the iOS world.

This led to posts like Setting up Calabash for iOS projects with CocoaPods and Build Configurations and Running Xcode tests from the terminal.

This week we'll take a few steps back, or better said a few steps higher and take a look at the unit and acceptance tests scene and at the resources to run continuous integration in the cloud.

Like one would do when setting up a walking skeleton, we'll take a look at the tools that Cocoa and Xcode offer to developer, then to the open source libraries we can use to achieve better results, and finally to the solution to run the written test in a continuous integration environment in the cloud.

Where it all begins: Xcode

With iOS 7 and Xcode 5 Apple introduced XCTest a simple yet powerful framework for writing tests in a xUnit way.

Writing XCTest tests is easy, and Xcode keeps the feedback loop quick by allowing developers to run tests inside the IDE by hitting ⌘U.

Xcode also has a "Test Navigator" section, in which we can see all the test, and their success or failure state in the last run of the suite.

Also note how failing tests are red, and passing ones are green. Colors are important in the red, green, refactor loop.

XCTest is super integrated with Xcode, and very simple to use. These are it's main strengths but also it's weakness. The XCTAssert... APIs are not very expressive or versatile, and running tests from outside Xcode is not as straightforward as it could be.

While in the past two year unit testing for iOS and OS X has become better and better, the acceptance testing side of things hasn't made any improvements.

To write UI automation test, Apple provides the UIAutomation framework. UIAutomation tests are written in Javascript, and allow us to drive the application UI and make assertion on its state. Despite how promising this all sounds, working with UIAutomation turns out to be quite tedious, and the Javascript APIs not as powerful as the unit testing native counter parts.

Here's a snippet of UIAutomation test:

UIATarget.localTarget().frontMostApp().navigationBar().buttons()["Add"].tap();

UIATarget.localTarget().frontMostApp().mainWindow().tableViews()[0].cells()[0].elements()["Chocolate Cake"];

UIATarget.localTarget().frontMostApp().mainWindow().tableViews()[0].scrollToElementWithPredicate("name beginswith 'Turtle Pie'");

As you can see they managed to make the Javascript APIs even more verbose than the ones in Foundation. Add to that the fact that these tests need to be run from Instruments and you'll get the idea of how unpleasant working on this framework could be.

The last piece of the puzzle is Apple's answer to the CI question: Xcode Bots. We can configure an Xcode Bot to get triggered and do work, for example running tests, for us, and host it on an Xcode Server.

I'll admit that I haven't done any work with Xcode Bots, but all the feedbacks I got on theme where on the lines of "I just doesn't work". 😞

To sum it all up, today developers lacking curiosity and big enterprises can have suite of unit and acceptance testing running on CI, using only Apple's technology. Everything is gonna work, more or less.

If you're reading this post you're probably full of curiosity, so let's move on and see what the open source community has to offer.

Open source libraries for Unit Testing

The iOS and OS X open source community is full of nice people and interesting projects. At the time of writing pod list reports that 8625 pods were found.

The open source libraries for unit testing show a trend in the style of the tests, they're in fact all in an xSpec style, which is taken from the RSpec Ruby testing library, and gives more relevance to the testing the behaviour of a class, rather than enumerating its methods.

Kiwi

Kiwi is a drop-in full stack replacement for XCTest, that provides an xSpec syntax. A Kiwi test, or better spec, looks like this:

describe(@"Team", ^{
  context(@"when newly created", ^{
    it(@"has a name", ^{
      id team = [Team team];
      [[team.name should] equal:@"Black Hawks"];
    });

    it(@"has 11 players", ^{
      id team = [Team team];
      [[[team should] have:11] players];
    });
  });
});

Kiwi specs are usually easier to read and communicate effectively what the system under test is meant to do, working as good documentation for the code.

Kiwi comes with it's own set of expectations, mocks, and stubs, and even supports asynchronous testing.

Specta

Specta is very similar to Kiwi, but takes a different architectural approach. Where Kiwi is a monolithic drop-in replacement, Specta values modularity and composition. The only thing the library takes care of is the framework for writing and running xSpec style tests, it's then up to the user to plug in libraries to take care of expectations, matching, mocking and stubbing.

I personally prefer this approach to library design, little self contained pieces that can be combined together.

Here's a Specta spec:

SpecBegin(Thing)

describe(@"Thing", ^{
  it(@"should do stuff", ^{
    // This is an example block. Place your assertions here.
  });

  it(@"should do some stuff asynchronously", ^{
    waitUntil(^(DoneCallback done) {
      // Async example blocks need to invoke done() callback.
      done();
    });
  });
});

Notice how the implementation on the it blocks is left empty. It's left to the library's user to fill them using the tools they prefer.

Speaking of tools, here's a list of libraries that can be used with Specta, and Kiwi, for testing:

• Expecta a matcher framework, expect(foo).to.equal(bar).
• OCHamcrest another matcher framework, assertThat(foo, equalTo(bar)).
• OCMock a mocking framework.
• OCMockito another mocking framework.
• OHTTPStubs a library to stub network requests, with block based syntax to match URLs.
• Nocilla another library to stub network requests, with a nice chain-able API, stubRequest(@"POST", <url>).withHeaders(...).withBody(...).

Quick

Quick is the new kid in the block, and it's a very cool kid indeed. Quick is mainly written in Swift and it's best suited to write test components written in the new language.

import Quick

class ThingSpec: QuickSpec {
  override func spec() {
    describe("a 'Thing'") {
      it("should do stuff) {
        //
      }
    }
  }
}

Thanks to Swift's syntax and closures Quick specs look far more readable that Kiwi's or Specta's.

Quick comes together with Nimble it's matchers library, which allows us to write neat things like expect(10) > 2.

Whether is Objective-C or Swift, monolith framework or composition of your favourite libraries, the open source scene offers plenty of valuable testing libraries, specially focused on writing clearer tests thanks to expressive syntax.

Open source libraries for Acceptance Testing

The same discrepancy in the quality of unit versus acceptance testing tools that we see in the Apple frameworks is reflected in the open source tools. This is probably due to the fact that while XCTest provide a solid foundation for developers to build frameworks on top of, UIAutomation doesn't, and all we're left with are hacks.

KIF

KIF, Keep It Functional, is a framework written in Objective-C that lets us write acceptance tests using XCTest and running them through Xcode in the same way we would do with unit tests.

The KIF tester uses private APIs to gain knowledge of the view hierarchy, and lets us query and interact with it using the accessibility label value of the views.

- (void)testSuccessfulLogin {
  [tester enterText:@"[email protected]" intoViewWithAccessibilityLabel:@"Login User Name"];
  [tester enterText:@"thisismypassword" intoViewWithAccessibilityLabel:@"Login Password"];
  [tester tapViewWithAccessibilityLabel:@"Log In"];

  // Verify that the login succeeded
  [tester waitForTappableViewWithAccessibilityLabel:@"Welcome"];
}

A big downside of KIF is the slow response time of the maintainers. This doesn't want to be a critique though, in the open source world everything is done for free, and since we all need to pay the bills, the amount of time that can be spent on such projects is limited. But when mixed with an foundation hard to work with, this results in unstable tools.

Update 2015/06/04 Since the months right after the publication of this article KIF has seen a remarkable burst in the response time of the maintainers, such that the observation above is not valid any more. Right now KIF is not only the best candidate for UI automation and acceptance testing, but also has an active community, and new version that will bring architectural and performance improvements is on its way. KIF's future is definitely bright.

Subliminal

Subliminal is an Objective-C framework that, like KIF, integrates with XCTest. Unlike KIF though, Subliminal is written on top of UIAutomation itself, and aims to hide away it's complexity from the developers.

- (void)testLogInSucceedsWithUsernameAndPassword {
  SLTextField *usernameField = [SLTextField elementWithAccessibilityLabel:@"username field"];
  SLTextField *passwordField = [SLTextField elementWithAccessibilityLabel:@"password field" isSecure:YES];
  SLElement *submitButton = [SLElement elementWithAccessibilityLabel:@"Submit"];
  SLElement *loginSpinner = [SLElement elementWithAccessibilityLabel:@"Logging in..."];

  NSString *username = @"Jeff", *password = @"foo";
  [usernameField setText:username];
  [passwordField setText:password];

  [submitButton tap];

  // wait for the login spinner to disappear
  SLAssertTrueWithTimeout([loginSpinner isInvalidOrInvisible], 3.0, @"Log-in was not successful.");

  NSString *successMessage = [NSString stringWithFormat:@"Hello, %@!", username];
  SLAssertTrue([[SLElement elementWithAccessibilityLabel:successMessage] isValid],
  @"Log-in did not succeed.");

  // Check the internal state of the app.
  SLAssertTrue(SLAskAppYesNo(isUserLoggedIn), @"User is not logged in.")
}

Subliminal states that it can enable testing In-App Purchase alerts, and even put the app to sleep. This all sounds great, but the fact that, at the time of writing, the last commit is from September 2014, and there are 13 active PR throws a bad signal on the status of the library 😕.

Calabash

Of all the tools seen so far Calabash is certainly the most original one. It's a Ruby gem to write acceptance tests in full BDD style using Cucumber, and it's maintained by Xamarin, which is a framework for writing iOS and Android apps in C#. What a mix of languages!

Unlike KIF and Subliminal, Calabash is not integrated with Xcode, at all. My setup for example uses Vim and Rake.

We write Cucumber features, implements the step, and run the tests using a command line tool. For this to work we need to embed an HTTP server in the app, that is used by the test runner to query and drive the UI.

Needless to say, this is all a big hack.

A Cucumber/Calabash test looks like this

# rating_a_stand.feature

Feature: Rating a stand
  Scenario: Find and rate a stand from the list
    Given I am on the foodstand list
    Then I should see a "rating" button
    And I should not see "Dixie Burger & Gumbo Soup"

# steps.rb

Given(/^I am on the foodstand list$/) do
  wait_for_element_exists "view marked:'Foodstand'"
end

Given(/^I should see a "([^\"]*)" button$/) do |button_title|
  wait_for_element_exists "button marked:'#{button_title}'"
end

Given(/^I should not see "([^\"]*)"$/) do |view_label|
  wait_for_element_does_not_exists "view marked:'#{view_label}'
end

The upside of Calabash is very declarative tests, that the management will like if they'll ever read them, and the ability for a tester to port most of their know-how across the two platforms.

On the other hand, the toolchain is not very robust. Tests run slower [quote needed], and one needs to constantly swap between Cucumber features, Ruby steps, and Objective-C view code, taking up a considerable amount of time.

Like the unit testing scene, the open source world provides different choices that can be used to improve your workflow. The only difference here is that the tools are no way near as mature, and the interest of the community is not as active.

Continuous Integrations Platforms

The final step of putting in place a good test harness for our project is having Continuous Integration. Running tests only on the developers machine does not guarantee that the code won't break when merged with the changes made by other team members, and it's safer to have someone who will always run the tests.

It goes without saying, the best CI setups are in the cloud. The time it takes to setup an maintain good old Jenkins is... too damn high!

The CI scene is probably the one where there's more variety. Here's a list of the major CI services that support iOS projects.

• Travis CI
• Ship.io
• Sauce Labs
• Bitrise
• Testdroid

The difference between these are in the pricing, how easy it is to get started, and how to configure them. For example Travis CI uses a .travis.yml file do define all the steps, while Bitrise has an interface where every step is represented by a block that can be added to the process.

---

The lists above are certainly not comprehensive, and I might have missed something. I hope this will serve as a good starting point for people interested in testing and CI.

The intent for the next weeks is to take a deeper look at some of the tools listed above. If you don't want to miss out subscribe to get updates to your inbox.

If you have suggestions or if you have an addition to make to the list tweet me at @mokacoding.

Happy coding, and keep the codebase better than you found it.