How to measure and report on accessibility

How to measure and report on accessibility

This resource was created by our Technology Taskforce, a group of senior IT accessibility individuals from leading UK and global organisations. For more information, including how to join, see our website. 

Introduction

Accessibility is a core requirement of any business – whether they realise it or not. Inaccessible organisations lose vast sums of money, both from disabled people whose custom they have lost, and from lost productivity from disabled employees.

This resource addresses the challenge of turning accessibility from a cause championed by a few passionate enthusiasts into a mainstream concern within the organisation. The experiences of Technology Taskforce members tells us that accessibility cannot meaningfully progress within an organisation until it shifts from being seen as an ‘extra’ or ‘bolt on’ feature, to a mainstream business issue.

All mainstream business issues must be managed if they are to be delivered. Accessibility is no different. New initiatives need deadlines and targets to work towards. Operational processes need service level agreements and measures to understand how effective they are. Reports are needed both to keep those responsible for delivery accountable and to keep accessibility within management’s focus.

This resource shares our experience of the type of targets and measures that our members use, and the approaches to measurement and reporting we find useful.

What should be measured?

Areas to measure

There are two areas where measurements are useful.

Systems development

This relates to the design of IT systems to ensure that they meet the organisation’s accessibility standards. This is necessary so that systems can be used by disabled staff and customers using relevant assistive technology (AT) tools.

Workplace adjustments / accommodations

This relates to providing the necessary tools and support to disabled staff. Organisations also have a duty to provide adjustments where needed for disabled customers – though these should be measured separately.

The tools might include:

• AT products such as screen magnifiers or text-to-speech
• Specialist hardware such as ergonomic keyboards and mice
• The ability to use the accessibility features built into work computers, mobile phones, and other work devices.

Support might include IT helpdesk provision, training and mentoring.

Types of measurement

Each of these areas has two types of measurement:

‘Business as usual’ processes

For systems development, this might include measuring:

• The number or percentage of projects which have accessibility implications – for example, those with a user interface.
• The number of new or amended systems delivered that meet the organisation standards – or fail to do so – and the number of projects seeking advice.

For workplace adjustments / accommodations, it might be:

• The number of staff in the organisation who have such adjustments
• The number of helpdesk calls they make and their fix times
• The time taken to deliver recommended workplace adjustments
• The number of incidents and complaints raised
• The total spend.

New initiatives

For both systems development and workplace adjustments, organisations might

1. Review existing processes
2. Prioritise areas for improvement
3. Develop plans
4. Set targets.

Measures would then include:

• Progress
• Milestones achieved
• Increases in user satisfaction
• Reduction in costs or delivery times
• Anything else deemed useful.

As already mentioned, these are no different for accessibility than for any other type of initiative.

Approaches to measurement and reporting

Accessibility Maturity Model

Members of the Technology Taskforce have created the Accessibility Maturity Model (AMM) to help organisations measure their performance in ten different areas of accessibility.

This tool provides a means of setting priorities, targets and benchmarks for the full range of accessibility features and hence for measuring progress during the course of a reporting year.

Business objectives and key performance indicators (KPIs)

Many organisations set key business objectives for the year, with KPIs in place to measure progress and targets.

We recommend including accessibility as a business objective. This means also creating a set of associated KPIs. It should be reported on using the same process used for all other business objectives.

Accessibility KPIs might include a mix of initiatives and operational process performance measures.

Process metrics

It is useful to use existing processes to collect data (for example, IT helpdesk cases, provision of equipment, software development lifecycle gateways, time spent on accessibility). A key feature here is incorporating flags to highlight accessibility data. For example:

• Time recording codes for accessibility work
• A specific accessibility or workplace adjustments / accommodations budget
• A register of staff IDs for those members of staff who have been given specialist hardware or software. For example, their data subset might be extracted from a database of all calls to an IT helpdesk, so that it would be simple to know from a list of staff if a new system has any disabled users.

Surveys

We recommend using surveys to gather data rather than solely relying on the raw numbers gathered by process metrics. Surveys can gather: 

• Raw numbers (too) – such as a sample of users to ask about the effectiveness of solutions provided. Example questions could be: Are you actually using the products recommended? What is your productivity compared to the expected average for your role? 
• Information – By allowing recipients to enter text as well as numbers. For example, do you have unresolved accessibility issues? How could the process be improved? 
• Qualitative data such as perceptions and opinions – for example, how would you rate your experience of this process? There is an important element to this type of data. It is easy for accessibility practitioners to think they know what disabled users want, but to get it wrong. It is good practice to consult and involve disabled users directly – for example, we can’t do everything at once, so what would you set as our priorities? Our resource, ‘Staff consultation methods,’ has more information about how to do this. 

When conducting surveys, the experience of our members is: 

• Where applicable, it is valuable to survey the views of managers of disabled staff, as well as the staff themselves. 
• Care needs to be taken when asking for textual information as opposed to just numbers, as the time required analysis can be great. Using a limited sample can help manage this. 
• Some surveys need to be conducted anonymously – for example, questions about personal productivity. 
• It is not inappropriate to ask personal questions if the overall purpose is to seek improvements and there is a genuine need to gather the data, providing of course that it is done sensitively. 
• All data needs to be protected.  
• If issues are reported, they must be investigated. Anonymous responses cannot be investigated of course, but a survey indicating major hidden issues should prompt follow-up work. 
• Involve disabled staff volunteers in the design, data gathering and analysis stages. This not only opens up the possibility of extra resources, but it also actively involves disabled users in finding solutions. 
• Designing surveys with questions that can be asked again in future years can be useful in measuring progress over time. For example, “Overall, how would you rate this service?” 
• Analysis of such responses needs to consider other factors. For example, disabled user satisfaction may fall overall if an organisation is going through a period of job reductions, even if in fact the core services have improved (such as delivery times for reasonable adjustments).