SFO15-506 - Capabilities of Workload Automation Test Harness - setup, execution flow, limitations

Sep 11, 2026

SFO15-506 introduces a Workload Automation (WA) test harness designed for repeatable benchmark execution across Android, Linux, and ChromeOS. The content covers setup requirements on a Ubuntu host, target-board networking assumptions, and Python-based installation steps using pip and required packages. It details the WA execution flow, including YAML-driven agendas, instrumentation and perf event selection, workload sequencing, and demo guidance for remote board runs. Key limitations include one-to-one WA instances, Python 2.7 dependency, static target IP assumptions, constrained reporting, and reliance on SSH, followed by next-step improvements toward automated PASS/FAIL reporting.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages15

Reading Time5 min

AccessFree to download

Introduction

Setup example

Demo

WA Execution Flow

Linaro’s use of WA

Limitations

Next steps

Presented by Lisa Nguyen Date Friday 25 September 2015 Event SFO15 SFO15-506: Capabilities of Workload Automation Test Harness Lisa Nguyen, Linaro Introduction ● Created by ARM ● Run benchmarks repeatably ● Supports Android, Linux, and ChromeOS ● Written in python Setup example ● Host (laptop) running Ubuntu 14.04 LTS ● Pandaboard as target (running either Android or Ubuntu) ● Networking required! ● Host has python 2.7, build-essential, git, and OpenSSH packages installed Setup commands On Ubuntu: ● sudo apt-get install python-pip ● sudo apt-get install python-serial Install packages via pip: ● sudo pip install--upgrade pip ● sudo pip install--upgrade setuptools ● sudo pip install pexpect ● sudo pip install pyserial ● sudo pip install docutils ● sudo pip install python-dateutil ● sudo pip install wlauto Demo Run WA remotely on a board in Kevin Hilman’s lab and show some nifty commands! WA Execution Flow Sample agenda global: iterations: 2 \# instrumentation: [perf] \# perf: \# events: ['task-clock', 'faults', 'migrations', 'cs', 'sched: sched_stat_runtime'] workloads: -name: hackbench -name: ebizzy workload_params: chunks: 4 -name: rt-app workload_params: config: browser-short.json -name: rt-app workload_params: config: mp3-short.json -name: cyclictest -name: stream workload_params: threads: 4 ● written in YAML ● global config settings for workloads ● lists workloads to execute in order ● Download a sample agenda at [http://people.linaro.org/](http://people.linaro.org/)~lisa. nguyen/agenda.yaml Linaro’s use of WA ● Collect power measurements ○ Pre-instrumented boards ● Run set of benchmarks on various boards Limitations ● One-to-one. One WA instance per board. ● Only supported in python 2.7 ● Assumes that target IP is static ● Limited reporting capabilities ● SSH is still required for running WA on Linux Next steps ● Investigate feasibility of using reports to declare PASS/FAIL ○ Every board has different thresholds for performance/power ○ Intel’s LKP handles this well ● Work with [kernelci.org](kernelci.org) to add another dimension to automated testing

FAQs

What platforms and programming language does the workload automation test harness support?

The harness supports Android, Linux, and ChromeOS, and it is written in Python.

What are the primary setup requirements for running WA?

A Ubuntu 14.04 LTS host with Python 2.7 and required packages installed is used, along with a Pandaboard target board. Networking is required, and SSH is needed when running WA on Linux targets.

What limitations affect WA execution and reporting?

WA is limited to a one-to-one model (one WA instance per board), requires Python 2.7, assumes the target IP is static, offers limited reporting capabilities, and depends on SSH for Linux execution.

Sep 11, 2026

Uploaded by Genevieve