2020-12-24

Xamarin Pipeline Demo

Table Of Contents

Introduction

I'm making this demo repo and writeup because it was surprisingly and frustratingly difficult to get Xamarin.UITest tests for Android to run on a Microsoft-hosted agent in an Azure DevOps pipeline. NO App Center. NO self-hosted agents. I just wanted to do everything in Azure DevOps.

So, this demo shows how to accomplish that, and some other common goals for an Azure Devops continuous integration pipeline for the Android portion of a Xamarin app...

  • Each build gets its own versionCode and versionName.
  • Build the APK.
  • Sign the APK.
  • Publish the APK as a pipeline artifact.
  • Do unit tests (NUnit).
  • Do UI tests (Xamarin.UITest), which involves several Android emulator steps.
  • Publish test results.

This demo is not about getting started on unit testing or UI testing; the demo is about getting these things to work in an Azure DevOps pipeline.

You can see a successful run, a successful job overview, published artifacts, and unit+UI test results (also alernate view for unit test run and UI test run).

This repo is available as a visualstudio.com repo and a github repo. As of 2020-Dec-24, Azure DevOps offers a free tier with 30 build hours per month and 2 GiB of artifact storage. The free tier was more than enough for all the pipeline needs of this demo.

This writeup is available as a github readme, visualstudio.com readme, and blog post. The repo readmes will be kept up to date, but the blog post may not receive many updates after 2020-12-24. Readme section links are oriented for GitHub.

Notable Files

The XamarinPipelineDemo.Android/AzureDevOps/ folder has most of the notable files...

  • pipeline-android.yml: the pipeline definition and heart of this demo.
  • AndroidSetVersion.ps1: the script that manipulates the Android manifest file to update the versionName and versionCode attributes.
  • example.keystore: for signing the APK. Normally keystore files are sensitive and you wouldn't put them (and their passwords) in your repo, but this is a demo.

XamarinPipelineDemo.UITest/AppInitializer.cs: the autogenerated AppInitializer.cs has been modified so that you can specify which APK file to install for testing, or which keystore to match an already installed (signed) APK. I suggest the APK file methodology.

uitest_run.ps1: script to run UITest tests in way most similar to how the pipeline will do it.

Screenshots folder has some screenshots of the results of a working pipeline run, and some of the web interface you need to tangle with to get the pipeline working.

Getting Started On Local Machine

First, check that it works on your machine. Open the solution in Visual Studio 2019, and deploy the Release build to an Android emulator or connected Android device (just select Release build configuration and launch the debugger). The app should show you a page with a label that says "Some text.".

In Visual Studio's test explorer, run both the Nunit and UITest tests. Everything should pass.

Also, to run the UITest tests in the way most similar to how the pipeline will do it, install a recent stable nunit3-console release, go into the LocalScripts folder and run uitest_run.ps1. You'll get a test results file TestResult.xml and a detailed log uitest.log that is useful for troubleshooting. The script tries to use adb and msbuild from your PATH environment variable and a few other locations. You might have to add your adb or msbuild directories to your PATH. Also, you might have to set the ANDROID_HOME environment variable to something like C:\Program Files (x86)\Android\android-sdk and the JAVA_HOME environment variable to something like C:\Program Files\Android\jdk\microsoft_dist_openjdk_1.8.0.25

Getting Started On Azure DevOps

In the Pipelines => Library section of your Azure DevOps project, you need to do a few things.

You need to set up the keystore file and variables...

  • Upload example.keystore as a secure file.
  • Create a variable group named android_demo_var_group. In it, create the following variables...
    • androidKeystoreSecureFileName: example.keystore
    • androidKeyAlias: androiddebugkey
    • androidKeystorePassword: android
    • androidKeyPassword: android
  • Make the androidKeystorePassword and androidKeyPassword secret by clicking the padlock icon.

You need to create a pipeline from the yaml pipeline definition file...

  • Upload the repo to Azure DevOps.
  • Create a new pipeline.
  • When asked "where is your code?", choose "Azure Repos Git".
  • Select the XamarinPipelineDemo repo.
  • Select "Existing Azure Pipelines YAML file".
  • Select the XamarinPipelineDemo/XamarinPipelineDemo.Android/AzureDevOps/pipeline-android.yml as the existing yaml file.

Run the pipeline and you'll have to click a few things to give the pipeline access to these secure files and secret variables. To grant permission to the pipeline, you might have to go to the run summary page (see Screenshots folder).

Explanation Of The Journey: Deadends, Pitfalls, Solutions

Note that things may change after this demo was made (2020-12-19). Some limitations may go away, and some workaround no longer needed. I'd love to hear about them if you ever encounter a way this demo should be updated.

Must Use MacOS Agent

First of all, doing Xamarin.UITest tests on Microsoft-hosted agent in an Azure DevOps pipeline has some important constraints. Microsoft-hosted agents for Window and Linux run in a virtual machine that can not run the Android emulator, so only the MacOS agent can run the Android emulator (MS docs page).

With a self-hosted agent that is not a virtual machine, you can use any of the three OSes. With App Center, the tests are run on a real device, and there is no need to run the Android emulator, so again, you can use any of the three OSes.

MacOS Agent Pitfalls

The MacOS agent has a few pitfalls to watch out for.

  • MacOS must use Mono when dealing with .NET Framework stuff (originally made just for Windows). So, .NET Framework stuff that works on your Windows machine may not do so well in the pipeline.
    • Try to make your project target .NET Core or .NET 5 where possible, especially your unit test project.
    • You can't use DotNetCoreCLI task on a MacOS agent to run test projects that target .NET Framework. Mono's open issue 6984 says that you can do "dotnet build" on a .NET Framework project, but you can't "dotnet test".
  • Xamarin.UITest MUST be .NET Framework, so you can not use DotNetCoreCLI task to run Xamarin.UITest tests.
  • MacOS agent also doesn't support VSTest or VsBuild tasks.
  • The only thing left to do for Xamarin.UITest is a MSBuild task to build it, then directly run nunit3-console to run the Xamarin.UITest tests.
  • MacOS agents are case sensitive for path stuff while Windows is not, so make sure your pipeline stuff is case-appropriate.
  • On Windows, you might be used to using Unix-inspired PowerShell aliases like "ls" and "mv". Do not use those aliases. In MacOS, even inside a PowerShell script, commands like "ls" will invoke the Unix command instead of the PowerShell cmdlet.

MacOS Agent Directories

During pipeline execution, there are three major directories to think about:

  • Build.SourcesDirectory, which is often /Users/runner/work/1/s/.
  • Build.BinariesDirectory, which is often /Users/runner/work/1/b/.
  • Build.ArtifactStagingDirectory, which is often /Users/runner/work/1/a/.

The repo for your pipeline is automatically put in the Build.SourcesDirectory. The other two directories are just natural places for you to put stuff. For instance, build outputs to Build.BinariesDirectory and special files you want to download later (artifacts) to Build.ArtifactStagingDirectory. The PublishBuildArtifacts task even defaults to publishing everything in the Build.ArtifactStagingDirectory.

Fresh Autogenerated Pipeline

If you make a new pipeline for a Xamarin Android app, you get an autogenerated yaml file that...

  • Triggers on your main or master branch.
  • Selects desired agent type (the pool section vmImage value).
  • Sets buildConfiguration and outputDirectory variables.
  • Does usual nuget stuff so you download the nuget packages used by your solutions.
  • The XamarinAndroid task builds all "*droid*.csproj" projects (probably just one for you), generating an unsigned APK file.

That's it. You can't even access the unsigned APK file after the pipeline runs; you just get to know whether the agent was able to make the unsigned APK.

I'll explain how and why we add to the pipeline to accomplish the goals I mentioned in the introduction.

General Pipeline YAML Advice

You're going to have to learn nuances of yaml. If you don't already know yaml and the unique quirks of pipeline yaml, it's going to trip you up somewhere.

Pipeline Variables

One of the first learning hurdles for dealing with pipeline is learning enough to use variables effectively.

The variable section in a fresh autogenerated pipeline looks like this...

variables:
  name1: value1
  name2: value2

...which is nice and compact. But if you need to use a variable group, you have to go with the more verbose way...

variables:
  - group: nameOfVariableGroup
  - name: name1
    value: value1
  - name: name2
    value: value2

I still haven't read the entire MS Docs page on pipeline variables because it is so long. Unfortunately there are three different syntaxes for referencing variables. You can mostly use macro syntax, which looks like $(someVariable) and leads to the variable being processed just before a task executes. Macro syntax can not be used in trigger or resource sections, and can not be used as yaml keys.

If the pipeline encounters $(someVariable) and doesn't recognize someVariable as a variable, then the expression stays as is (because maybe it'll be usable by PowerShell or whatever you're executing).

So, if you get errors that directly talk about $(someVariable) rather than the value of someVariable, then someVariable isn't defined. You need to check your spelling, and if it's a variable from a variable group (defined in Library section of web interface), you need to explicitly reference the variable group in your variables section.

My pipeline yaml mostly uses macro syntax. One notable exception is runtime expression syntax ($[variables.someVariable]) in conditions and expressions, as is recommended. You can see the runtime expression syntax in my pipeline's step conditions, just search for "condition:" or "variables.". Another exception is Azure DevOps's surprising (but reasonable) way of setting/creating pipeline variables from scripts: outputting a line to standard output that conforms to logging command syntax; here's an example:

- pwsh: Write-Output "##vso[task.setvariable variable=someVariable]some string with spaces allowed"

Non-secret variables are mapped to environment variables for each task.

Pipeline Triggers

A freshly autogenerated pipeline might have a trigger section...

trigger:
- main

...which will make the pipeline trigger for every change to the main branch. But if you have multiple target platforms (android, iOS, uwp), each having their own pipeline, then you get a lot of unnecessary builds when you update something only relevant to one platform.

So, you probably want a path-based trigger. Note that wildcards are unsupported and all paths are relative to the root of the repo. Here's a trigger section for a hypothetical android pipeline...

trigger:
  branches:
    include:
    - main
  paths:
    include:
    # common
    - 'MyApp'
    - 'MyApp.NUnit'
    - 'MyApp.UITest'
    - 'Util'
    - 'XamarinUtil'
    - 'MyApp.sln'
    # platform
    - 'MyApp.Android'

Also, this path-based trigger stuff is why this demo's android pipeline yml file and android version script are under XamarinPipelineDemo.Android/AzureDevOps rather than under a root-level AzureDevOps folder. A change to these android-pipeline-specific file should only trigger an android pipeline build, and putting them under an android folder makes that easy trigger-wise.

Similarly, uitest_run.ps1 is in a LocalScripts folder instead of the XamarinPipelineDemo.UITest folder because changes to a local-use-only script should not trigger a pipeline build. There is also the option of having a XamarinPipelineDemo.UITest/LocalScripts folder and listing that folder in the yaml's trigger-paths-exclude list.

Pipeline Tasks

Some tasks support path wildcards in their inputs, some don't. Always check the task reference before using path wildcards. If you get an error message like "not found PathToPublish: /User/runner/work/1/a/*.apk", the fact that the path it couldn't find has a wildcard should make you double check whether wildcards are supported for that task input.

Sometimes the task is a wrapper around some tool, and the task's documentation doesn't go into much detail into the behavior of the tool. For instance, AndroidSigning is a wrapper around apksigner, and you have to get all the way down to the --out option section of the apksigner doc to learn that the absence of the option leads to the APK file being signed in place, overwriting the input APK.

Sometimes looking at the Azure pipeline tasks source code is useful.

Pipeline Scripts And Strings

In your pipeline, you might want to do something simple, like copy some files. Sometimes there is a task for what you want to do, like CopyFiles, but often there isn't. A good way to accomplish these small things is to use one of the script tasks...

  • Bash: runs on MacOS, Linux, and Windows.
  • BatchScript: runs on Windows.
  • CmdLine: uses bash on Linux and MacOS; uses cmd.exe on Windows.
  • PowerShell: runs on MacOS, Linux, and Windows.

I prefer PowerShell because...

  • It runs on all the agents in the same way.
  • It will run on people's local machines. It comes preinstalled in Windows and I think it's easy enough to install on Linux and MacOS.
  • I think it's the most capable of the languages. I think PowerShell helps keep simple tasks easy and can use anything in the .net ecosystem, like System.Collections.Generic.Dictionary<K,V>, which is especially nice for Xamarin developers.

In fact, I learned PowerShell because of dealing with Xamarin pipelines, and PowerShell is now my go-to language for quick Windows scripts.

There are a few ways to do scripts in pipelines, but first you should understand yaml multi-line strings. The > character causes the following indented block to be treated "folded style": as a single string with no line breaks (except one at the end). The | character causes the following indented block to be treated "literal style": as a single string with line breaks preserved. Good explanation of mult-line strings at this stackoverflow answer and yaml-multiline.info.

Here are some script examples...

- pwsh: SomeCommand | CommandReceivingPipedStuffFromPreviousCommand; SomeSeparateCommand
  displayName: 'some inline one-liner script'

- pwsh: |
    SomeCommand | CommandReceivingPipedStuffFromPreviousCommand
    SomeSeparateCommand
  displayName: 'some inline multi-liner script'

- task: PowerShell@2
  displayName: 'calling a PowerShell script file in the repo'
  inputs:
    filePath: '$(theScriptDir)/SomeScript.ps1'
    # '>' used so we can have multiple lines treated as one line
    arguments: >
      -SomeScriptArg "SomeValueInQuotes"
      -AnotherScriptArg AnotherValueShowingQuotesNotAlwaysNeeded

Note how | characters can appear in the scripts; that's totally fine.

Give Each Build An Increasing Android App Version

If the Azure DevOps pipeline is going to be making the APKs we'll be releasing, we need unique versionCode and versionName values for each build.

Reminder: versionCode is a positive integer that is used by Android to compare versions and is not shown to the user. versionName is text displayed to the user and that is its only use.

Short version: The 'Set build-based Android app version' task uses the YAML counter function on the pipeline name (Build.DefinitionName) to set the versionCode and the Build.BuildNumber to set the versionName. This task is executed right before the XamarinAndroid build task and calls a PowerShell script to modify the Android manifest file.

How To Set The Android App Version

James Montemagno's and Andrew Hoefling's "Mobile App Tasks for VSTS" (Azure DevOps plugin, Github repo) has an AndroidBumpVersion task that does half of the job: setting the versionCode and versionName.

Some people are not allowed to use Azure DevOps plugins (perhaps for security by their employer), so we will not use this as a plugin. Azure DevOps plugins are run via a Node server, so the plugin would use tasks/AndroidBumpVersion/task.ts, but thankfully James has also provided PowerShell and bash equivalents of his plugin tasks, so you can look at those files.

I went with his PowerShell script, fixed a bug, and cleaned it up (pull request 39, current code). The result is this demo's AndroidSetVersion.ps1.

(Note: recent versions of PowerShell are cross platform, so you can run PowerShell on MacOS and Linux. But again, be mindful of Unix commands overriding PowerShell aliases and you can't be case-insensitive.)

The essence of the script is that the Android manifest file is XML and inside the manifest element, set the android:versionCode and android:versionName attributes appropriately. Thankfully PowerShell has the XmlDocument class and the Select-XML cmdlet that gives you easy-to-manipulate SelectXmlInfo objects.

How To Choose The Version

The second half of the problem is how to have an increasing and meaningful versionCode and versionName. Azure DevOps pipelines will have pre-defined variables, including...

  • Build.BuildId: a positive integer that is build id that is unique across your organization and will appear in the build's URL (ex: dev.azure.com/SomeOrganization/SomeProject/_build/results?buildId=123456).
  • Build.BuildNumber: a string (not a number, especially if you set the name variable. The default format is "$(Date:yyyyMMdd)$(Rev:.r)", which looks like "20201231.7" and is unique only within the pipeline.
  • Build.DefinitionName: the name of the pipeline.

I think that the default Build.BuildNumber makes sense for versionName; it's unique, increasing, and easy for you to lookup the build/commit for the version name a user sees. I don't like Build.BuildId for versionCode because consecutive builds will probably not have consecutive versionCode values because of all the other builds in your Azure DevOps organization. Build.BuildId is probably just going to be a large, meaningless number for you.

Thankfully, Andrew Hoefling wrote “Azure Pipelines Custom Build Numbers in YAML Templates”, which shows how you can get a simple {1,2,3,...} progression for a build using the yaml counter function. MS docs on defining pipeline variables has a counter example too.

Here's a snippet that shows a simple pipelineBuildNumber that goes up {0,1,2,...} and a versionRevision that counts up but gets reset everytime you change the versionMajorMinor value.

variables:
  # for doing Major.Minor.Revision;
  # any time you change versionMajorMinor,
  # versionRevision uses a new counter
  - name: 'versionMajorMinor'
    value: '0.0'
  - name: 'versionRevision'
    value: $[counter(variables['versionMajorMinor'], 0)]
  # for doing simple pipeline build counter
  - name: 'pipelineBuildNumber'
    value: $[counter(variables['Build.DefinitionName'], 1)]

Build The APK File

Thankfully autogenerated android pipelines and internet examples give you a XamarinAndroid step that can build the apk for you. Here's the demo's step for that...

- task: XamarinAndroid@1
  inputs:
    projectFile: '**/*droid*.csproj'
    outputDir: '$(outputDir)'
    configuration: '$(buildConfiguration)'

One confusing thing though is some places will say to use the Gradle task instead of the deprecated Android build task. I am 90% sure Gradle is for native Android apps, not Xamarin. I do know that I've never had to use anything Gradle-related for my Xamarin stuff and XamarinAndroid seems fine.

Sign The APK File

Keystore Background

You'll want to sign the APK file so it can be installed on users' devices and distributed on Google Play. This repo already comes with a keystore file (remember: don't put your keystore in your repo; it should be more tightly controlled and uploaded as a secure file to Azure DevOps), but you can create your own keystore by following these MS Docs instructions (don't do the "Sign the APK" section).

You might get confused that if you make a keystore in Visual Studio, you have to choose a "keystore password", but not a "key password", and lots of other places talk about the "key password". The "key and certificate options" section of the apksigner doc might help you understand. A keystore can contain multiple keys, each identified by a key alias. The keystore itself password-protected, and each key might have its own password. This keytool example makes me think a common behavior is for a key password to default to the same as the keystore password.

One approach that has worked for me so far: when you are asked for a key password, and you don't recall there being a key password, you can probably put the keystore password.

Another confusion you may have is that the AndroidSigning@3 task has an input named "keystoreAlias" (also called "apksignerKeystoreAlias"), but keystores do not have aliases; keys within keystores have aliases. You specify the keystore by the file name, then you specify the key by the key's alias. I have reported this misnaming as a problem on Developer Community.

AndroidSigning Task

This is the demo's AndroidSigning task (and required reference to appropriate variable group)...

variables:
  - group: android_demo_var_group

...

- task: AndroidSigning@3
  displayName: 'sign APK with example keystore'
  inputs:
    apkFiles: '$(outputDir)/*.apk'
    apksignerKeystoreFile: '$(androidKeystoreSecureFileName)'
    apksignerKeystoreAlias: '$(androidKeyAlias)'
    apksignerKeystorePassword: '$(androidKeystorePassword)'
    apksignerArguments: '--verbose --out $(finalApkPathSigned)'

Remember to follow the steps from Getting Started On Azure DevOps for uploading the keystore as secure file and creating the needed variable group with needed variables.

The task doc says it accepts wildcards for apkFiles. (Remember, don't assume tasks accept wildcards, check the task doc). Also, the doc states that the referenced keystore file must be a secure file, which should be fine for you. However, if you want to get around this restriction, you could use a PowerShell task to call apksigner directly.

Here is the error message if you try to use something other than a secure file for your keystore:

There was a resource authorization issue: "The pipeline is not valid. Job Job: Step AndroidSigning2 input keystoreFile references secure file /path/to/nonsecure/file which could not be found. The secure file does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

If you get errors like "can't find $(someVariable) secure file", that means the someVariable is not defined. Check that you are referencing the appropriate variable group in your yaml's variables section, and check that someVariable exactly matches what you have in your variable group.

By default, apksigner overwrites the APK file, and therefore the AndroidSigning task overwrites the APK file, which could be fine for you. But I wanted the signed APK to go into the artifact staging directory (path held in predefined variable Build.ArtifactStagingDirectory) with a particular file name (not the default com.demo.XamarinPipelineDemo.apk), so I used apksigner's --out argument.

Note that finalApkPathSigned puts the Build.BuildNumber and pipelineBuildNumber in the file name.

If you ever want to double check whether an APK has been signed, and by which keystore, use apksigner (possibly at C:\Program Files (x86)\Android\android-sdk\build-tools\SOME_VERSION\apksigner.bat). Do apksigner verify --print-certs THE_APK_PATH and the first line tells you about the key that signed the APK or DOES NOT VERIFY if not signed.

Likewise, for looking at keystore files, you can use keytool (possibly at C:\Program Files\Android\jdk\microsoft_dist_openjdk_1.8.0.25\bin\keytool.exe). keytool -v -list -keystore KEYSTORE_PATH will tell you about keys in the keystore, even if you provide no keystore password.

Publish The APK Files As Build Artifacts

I wanted to have both unsigned and signed APKs as build artifact.

Here is the MS Doc description of build artifacts and pipeline artifacts:

Build artifacts are the files that you want your build to produce. Build artifacts can be nearly anything that your team needs to test or deploy your app. For example, you've got .dll and .exe executable files and a .PDB symbols file of a .NET or C++ Windows app.

You can use pipeline artifacts to help store build outputs and move intermediate files between jobs in your pipeline. Pipeline artifacts are tied to the pipeline that they're created in. You can use them within the pipeline and download them from the build, as long as the build is retained. Pipeline artifacts are the new generation of build artifacts. They take advantage of existing services to dramatically reduce the time it takes to store outputs in your pipelines. Only available in Azure DevOps Services.

The "Pipeline artifacts are the new generation of build artifacts" makes me think maybe I should be producing pipeline artifacts instead of build artifacts, but build artifacts have been satisfactory so far. Publishing the APKs as a build artifact makes it easy for me to download the APKs generated by a build, and that's what I wanted. See [this screenshot](Screenshots/published artifacts.png) for how the web interface looks for displaying build artifacts, which can be downloaded by clicking on them.

The demo's PublishBuildArtifacts step for APKs...

- task: PublishBuildArtifacts@1
  displayName: 'publish APK artifacts'
  inputs:
    artifactName: 'apks'

Previously, an inline powershell script copied an unsigned APK file to Build.ArtifactStagingDirectory, and then AndroidSigning task created its APK and idsig outputs in Build.ArtifactStagingDirectory. PublishBuildArtifacts's pathToPublish input defaults to publishing the directory Build.ArtifactStagingDirectory, so the default works out. PublishBuildArtifact's source code suggests to me that published files are not removed, so keep that in mind when doing multiple publishes.

When you download the apks artifact, the download will be a zip file named apks.zip, which will contain an apks folder that will contain all the published files.

Note that pathToPublish does not support wildcards.

The demo does not specify the publishLocation input value, so the default of container is being used. I'm not sure what a container is, and I can't find anything that offers an explanation. There is this MS Doc about container jobs, but it talks about containers in the Docker sense. The publishLocation input reference says the container option will "store the artifact in Azure Pipelines" and that sounds good, and does make the artifact available for looking at and downloading when I view the build run. The alternate option for publishLocation is filePath, which copies the artifacts to "a file share", which I guess you'd have to set up

Build And Run Unit Tests

To build and run unit tests, DotNetCoreCLI will take care of...

  • Building the test project and its dependencies.
  • Discovering and running the tests in the test project.
  • Publishing the test results so you can see and explore them in Azure DevOps's web interface. This includes the build being marked with something like "90% of tests passing".

One requirement is that your test project is .NET Core or .NET 5. (Currently, "dotnet test" does not support Mono, but that may change.) Even if you get "dotnet test" to work on your Windows machine by making the project SDK-style ( format article, overview article), it won't work on the MacOS agent; you'll get errors about not having the references assemblies...

##[error]/Users/runner/.dotnet/sdk/3.1.404/Microsoft.Common.CurrentVersion.targets(1177,5): Error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks
/Users/runner/.dotnet/sdk/3.1.404/Microsoft.Common.CurrentVersion.targets(1177,5): error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks [/Users/runner/work/1/s/XamarinPipelineDemo.NUnit/XamarinPipelineDemo.NUnit.csproj]

The step for unit tests is...

- task: DotNetCoreCLI@2
  displayName: 'unit tests'
  inputs:
    command: 'test'
    projects: '**/*NUnit*.csproj'
    configuration: '$(buildConfiguration)'

The projects input supports path wildcards. The code we are testing is already built with the Release build configuration, so if we build our test project with the same build configuration, we won't have to rebuild our dependencies.

Just so you know, if you dig in to the dotnet test doc, the configuration option defaults to Debug. The default build configuration is Debug for msbuild and other dotnet commands as well.

Remember that VSTest task is not available on MacOS agents. If you need an alternative to DotNetCoreCLI for testing, you'd have to do...

Run App Center UI Tests

This demo isn't really about App Center tests, but I'd love to go over some workarounds/solutions to some of the problems you might have, possibly saving you a lot of time.

Here are the relevant pipeline steps...

- task: NuGetCommand@2
  displayName: 'nuget-restore solution'
  inputs:
    restoreSolution: '$(solution)'
    # Xamarin.UITest nuget packages sometimes go to global nuget cache but we need
    # a known folder for Xamarin.UTTest's test-cloud.exe for AppCenterTest
    # https://github.com/microsoft/azure-pipelines-tasks/issues/6868#issuecomment-547565284
    restoreDirectory: '$(nugetPackageDir)'

# ... lots of steps omitted here ...

# default nodejs version (v12) is not compatible with stuff used in AppCenterTest task
# https://github.com/microsoft/appcenter-cli/issues/696#issuecomment-553218361
- task: UseNode@1
  displayName: 'Use Node 10.15.1'
  condition: and(succeeded(), eq(variables.wantAppCenter, true))
  inputs:
    version: 10.15.1

- pwsh: |
    $uiTestToolsDir = (Get-ChildItem "$(nugetPackageDir)/Xamarin.UITest/*/tools")[0].FullName
    Write-Output "##vso[task.setvariable variable=uiTestToolsDir]$uiTestToolsDir"
  displayName: 'find Xamarin.UITest tools directory'
  condition: and(succeeded(), eq(variables.wantAppCenter, true))

- task: AppCenterTest@1
  condition: and(succeeded(), eq(variables.wantAppCenter, true))
  inputs:
    appFile: '$(finalApkPathSigned)'
    frameworkOption: uitest
    uiTestBuildDirectory: '$(uiTestDir)'
    uiTestToolsDirectory: '$(uiTestToolsDir)'
    # alternatively, you can skip the previous search for uiTestToolsDir, but you'll have to update
    # the "3.0.12" below everytime you update the Xamarin.UITest nuget package
    #uiTestToolsDirectory: '$(nugetPackageDir)/Xamarin.UITest/3.0.12/tools'

If you get errors like "Error: Command test prepare uitest ... is invalid", then you have a nodejs version problem. Unfortunately, (as of time of writing), Azure DevOps pipelines default to nodejs version 12, but AppCenterTest task requires nodejs version 10. Thus, the demo uses the UseNode task to set nodejs to version 10.15.1, just like this appcenter-cli issue 696 thread suggests.

Strangely enough, I can't find UseNode task doc via searching or looking through all the other task docs. There's NodeTool task doc, but looking at the source code of these tasks (usenode.ts, nodetool.ts) and comments, they don't seem to do the same thing.

If you get a test-cloud.exe related error like this...

Preparing tests... failed.
Error: Cannot find test-cloud.exe, which is required to prepare UI tests.
We have searched for directory "packages\Xamarin.UITest.*\tools" inside "D:\" and all of its parent directories.
Please use option "--uitest-tools-dir" to manually specify location of this tool.
Minimum required version is "2.2.0".
##[error]Error: D:\a\_tasks\AppCenterTest_ad5cd22a-be4e-48bb-adce-181a32432da5\1.152.3\node_modules\.bin\appcenter.cmd failed with return code: 3

...then you need to be sure that your AppCenterTest task's uiTestToolsDirectory input is set to the tools folder underneath the Xamarin.UITest package folder. The tools folder contains test-cloud.exe.

Unfortunately, there's a few places that Xamarin.UITest's nuget package folder could go, so I altered my NuGetCommand step to use nugetPackageDir as the restoreDirectory. There's also the unfortunate complication that the tools folder will be underneath a different version folder each time you update yur Xamarin.UITest package, so I used an inline PowerShell script to find the exact folder and set the uiTestToolsDir variable. That uiTestToolsDir variable is used for the uiTestToolsDirectory task input. The PowerShell script would not be necessary if the uiTestToolsDirectory supported path wildcards; one of the reasons that getting comfortable with PowerShell is very helpful for working with pipelines.

Note: this azure-pipelines-tasks issue discussion is where I learned about the nature of the test-cloud.exe problem and some workarounds. The PowerShell script to solve version folder problem is my own invention.

Set Up And Start Android Emulator

For setting up and starting the Android emulator, there are some good examples out there.

Eric Labelle's "Android UI Testing in Azure DevOps" article is for native Android apps, not Xamarin Android. The article covers more than just setting up the Android emulator. It talks about caching the AVD. I found that caching the AVD took the same or longer than just downloading the AVD fresh, but maybe I was doing something wrong.

Jan Piotrowski's azure-pipelines-android_emulator repo is good in that it gives you a pipeline yaml file with steps definitions for setting up and starting the Android emulator.

The MS Docs article "Build, Test, And Deploy Android Apps" has a section on starting the Android emulator.

You can see that the bash code in these articles are all pretty much the same. I think they're all derived from Andrey Mitsyk's comment on the azure-devops-docs issue thread about missing Android emulator documentation.

I made a few changes to Jan Piotrowski's pipeline steps for this demo...

variables:
  - name: adb
    value: '$ANDROID_HOME/platform-tools/adb'
  - name: emulator
    value: '$ANDROID_HOME/emulator/emulator'

# .. lots of stuff omitted here ...

- bash: |
    set -o xtrace
    $ANDROID_HOME/tools/bin/sdkmanager --list
  displayName: 'list already installed Android packages'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- bash: |
    set -o xtrace
    echo "y" | $ANDROID_HOME/tools/bin/sdkmanager --install 'system-images;android-30;google_apis;x86'
  displayName: 'install Android image'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- bash: |
    set -o xtrace
    $(emulator) -list-avds
    echo "no" | $ANDROID_HOME/tools/bin/avdmanager create avd -n uitest_android_emulator -k 'system-images;android-30;google_apis;x86' --force
    $(emulator) -list-avds
  displayName: 'create AVD'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- bash: |
    set -o xtrace
    $(adb) devices
    nohup $(emulator) -avd uitest_android_emulator -no-snapshot -no-boot-anim -gpu auto -qemu > /dev/null 2>&1 &
  displayName: 'start Android emulator'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))

- task: MSBuild@1
  displayName: 'build ui tests'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  inputs:
    solution: '**/*UITest*.csproj'
    configuration: '$(buildConfiguration)'

- bash: |
    set -o xtrace
    $(adb) wait-for-device shell 'while [[ -z $(getprop sys.boot_completed | tr -d '\r') ]]; do sleep 1; done; input keyevent 82'
    $(adb) devices
  displayName: 'wait for Android emulator'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  timeoutInMinutes: 5

The set -o xtrace lines are so that script lines are printed before they are executed.

I wanted it to be easy to choose emulator/AppCenter/none for UI testing, so you can see the condition of steps depending on wantEmulatorUITests.

I wish I could find the place that I first saw the idea of building the UI tests while waiting for the Android emulator to start.

If there is an Android device that is especially beneficial to do emulator UI tests on, you can create an AVD for that device. The avdmanager command line reference seems incomplete, but googling will get you some avdmanager examples to learn from.

I've read that unsigned APKs can be installed on emulators, but I got the following error when trying to do a adb install unsigned.apk...

adb: failed to install /Users/runner/work/1/a/unsigned.apk: Failure [INSTALL_PARSE_FAILED_NO_CERTIFICATES: Failed collecting certificates for /data/app/vmdl1550487669.tmp/base.apk: Failed to collect certificates from /data/app/vmdl1550487669.tmp/base.apk: Attempt to get length of null array]

So, install a signed APK on your emulator, even if the APK is signed by the debug keystore.

Build UI Tests

Here's the step definition again...

- task: MSBuild@1
  displayName: 'build ui tests'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  inputs:
    solution: '**/*UITest*.csproj'
    configuration: '$(buildConfiguration)'

The situation is pretty simple once you learn that you can't use DotNetCoreCLI task for Xamarin.UITest project. MSBuild task is the only build task available on the MacOS agent (other than doing some script task that calls msbuild). You probably want to match the same build configuration that was used to compile the other projects, unless your UI test project has no dependencies that were built before.

Run UI Tests

Currently, dotnet test does not work with Mono, so the DotNetCoreCLI task does not work on the MacOS agent for running Xamarint.UITest tests. So, you have to run the tests via nunit3-console and publish the test results via the PublishTestResults task (which is nicely integrated into the Azure DevOps web interface for that build and for analysis across builds). For troubleshooting, you may want to publish the detailed UI test log (not the same thing as test results).

Here are the steps and relevant variable definitions...

variables:
  - name: uiTestDir
    value: '$(Build.SourcesDirectory)/XamarinPipelineDemo.UITest'
  - name: uiTestResultPath
    value: '$(Build.ArtifactStagingDirectory)/uitest_result.xml'
  - name: uiTestLogPath
    value: '$(Build.ArtifactStagingDirectory)/uitest.log'

# ... lots of stuff omitted here ...

- pwsh: |
    Set-PSDebug -Trace 1
    $env:UITEST_APK_PATH = "$(finalApkPathSigned)"
    $testAssemblies = Get-Item "$(uiTestDir)/bin/$(buildConfiguration)/XamarinPipelineDemo.UITest*.dll"
    nunit3-console $testAssemblies --output="$(uiTestLogPath)" --result="$(uiTestResultPath)"
  displayName: 'run ui tests'
  condition: and(succeeded(), eq(variables.wantEmulatorUiTests, true))
  continueOnError: true
  timeoutInMinutes: 120

Note that nunit3-console defaults to putting the test results into ./TestResult.xml file, but the demo specifies a path for --result.

The demo app is installed as package com.demo.XamarinPipelineDemo, and the UI tests need to install package com.demo.XamarinPipelineDemo.test and the signatures of those two packages must match. Look at AppInitializer.cs for how it's done, but the basics is you either use ApkFile by itself or InstalledApp and KeyStore together. The ApkFile method is simpler. It even takes care of installing the APK onto the emulator.

If the app package and the test package don't have the same signature, you'll get an error like this:

System.Exception : Failed to execute: /Users/runner/Library/Android/sdk/platform-tools/adb -s emulator-5554 shell am instrument com.demo.XamarinPipelineFiddle.test/sh.calaba.instrumentationbackend.ClearAppData2 - exit code: 1
java.lang.SecurityException: Permission Denial: starting instrumentation ComponentInfo{com.demo.XamarinPipelineFiddle.test/sh.calaba.instrumentationbackend.ClearAppData2} from pid=5635, uid=5635 not allowed because package com.demo.XamarinPipelineFiddle.test does not have a signature matching the target com.demo.XamarinPipelineFiddle

If you get the error System.Exception : Timed out waiting for result of ClearAppData2 in your job log, and the detailed UI test log file contains...

AdbArguments: '-s emulator-5554 shell run-as com.demo.XamarinPipelineDemo.test ls "/data/data/com.demo.XamarinPipelineDemo.test/files/calabash_failure.out"'.
Finished with exit code 1 in 184 ms.
ls: /data/data/com.demo.XamarinPipelineDemo.test/files/calabash_failure.out: No such file or directory

...then the most likely explanation is that your app crashed. When you encounter this error, try out the exact APK file that the pipeline was using. The crash might happen only in Release build configuration, or something else.

The job log having the error System.Exception : Post to endpoint '/ping' failed after 100 retries. No http result received is most likely due to an app crash, but can also be due to problems with the agent pool (as in it's not your fault, you might have to re-run the build a few times and hopefully the problem passes).

Also, these "ClearAppData2" and "post to endpoint" errors might be followed by a TearDown: System.NullReferenceException error, and that is because the test runner still calls the test TearDown method, which might try to use the IApp object that was supposed to be set to the result of AppInitializer.StartApp. The real problem is the earlier errors, not the exception in TearDown.

The error Tcp transport error is something I've only seen due to agent pool problems. I just had to wait and retry a few times.

Publish UI Tests

Running nunit3-console will run the tests and generate a test result xml file and a test log file, but we still need to publish at least the test results...

- task: PublishBuildArtifacts@1
  displayName: 'publish ui test log artifact'
  inputs:
   artifactName: 'UI test log'
   pathToPublish: '$(uiTestLogPath)'
  continueOnError: true

- task: PublishTestResults@2
  condition: eq(variables.wantEmulatorUiTests, true)
  inputs:
    testRunTitle: 'Android UI Test Run'
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(uiTestResultPath)'
    # Android tests may randomly fail because of the System UI not responding (if you're using Prism);
    # see https://github.com/PrismLibrary/Prism/issues/2099 ;
    # tests may also fail due to pool agent problems;
    # using the following line still makes builds have warning status when UI tests fail
    # failTaskOnFailedTests: false

Publishing the UI test log as a general build artifact is for troubleshooting; the normal job log is pretty helpful, but you need to look at the UI test log in order to see any console printing your UI tests did.

Publishing the test results makes them nicely integrated with the Azure DevOps web interface and associated with the build.

Sometimes UI tests fail due to things like agent problems. Up to you whether you want to treat failing UI tests as warning or failure via the failTaskOnFailedTests input.

Problems Accessing Stuff

You might have someone who can't access something, like build artifacts, regardless of permissions (and rememeber there are permissions under project settings, then permissions for pipelines, then permissions for EACH pipeline). The problem might be their “access level”. If their access level is “Stakeholder”, then it probably needs to be changed to "Basic" or better. “Basic”. You can check anyone’s organization-specific access level at URLs like this: dev.azure.com/TheAppropriateOrganization/_settings/users

Thanks To Those Who Helped Me

Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)