1 Pushing the Boundaries of Automatic Testing Through Machine LearningAlistair Garrison Director of Accessibility Research

2 Formerly Known As SSB BART GroupWho is Level Access? Formerly Known As SSB BART Group New Corporate Identity – Name and Branding New Website – Coming Spring 2017!! No Change in our Innovative Technology or Excellent Service CSUN 2017 Assistive Technology Conference

3 W.R.T. Accessibility Testing - Why do we need more automation?Today, Continuous Delivery – in the form of fast, frequent and stable software releases - is the goal of many companies; Implemented through DevOps, via collaborative “inter-team working” with a heavy focus on automation. CSUN 2017 Assistive Technology Conference

4 A11y better in, than thrown out!Modern build development pipe-line In this heavy automation environment we’re wanting to ensure accessibility testing is embraced, and included into the build pipeline; Rather than being seen as a bottleneck, and thrown out. CSUN 2017 Assistive Technology Conference

5 So, we’re talking about pushing the boundaries - but in what?Technical Accessibility testing; Relating to Accessibility Requirements: WCAG X.0, Section 508, etc… CSUN 2017 Assistive Technology Conference

6 And, what boundaries within a11y testing are we really talking about?The boundaries between the three test type silos that have solidified in the accessibility testing field: Fully automated tests; Guided automatic tests; Manual tests; CSUN 2017 Assistive Technology Conference

7 CSUN 2017 Assistive Technology ConferenceBoundaries created by limitations in what we believe only humans can test? If we’ve looking for 100% Accessibility Test Coverage Manual tests Guided automatic tests Fully automated tests CSUN 2017 Assistive Technology Conference

8 For example, people say that…Only around 25% of things we need to check for accessibility can be checked using fully automated tests. But that means we still need human involvement in 75% of accessibility tests - right??? CSUN 2017 Assistive Technology Conference

9 But for what? What skills / abilities do we believe only humans have…Recognizing content; Recognizing similar objects; Recognizing the meaning of content - image of a dog; or “Toto”? Recognizing when things have been misused – a Recognizing the Cause-Effect relationship between content (for example, understanding that what is being shown is an error message when an error is entered); CSUN 2017 Assistive Technology Conference

10 So, I’m guessing, it’s our ability to recognize stuff…To answer questions like - “Choose the things in the picture that are red”? Picture shows: Apple Strawberry Cat. CSUN 2017 Assistive Technology Conference

11 But, there’s also expert judgment???Accessibility testing Yes?… But, what if that’s only because we’ve been asking a small number of big over- arching questions; Rather than breaking things down as far as they can be broken down; So big questions become large sets of atomic ones… CSUN 2017 Assistive Technology Conference

12 Here’s an example of what I mean…From the bbc website - Highly-trained Clinicians identifying cancerous tissue samples… Sounds a highly complex task, that needs an expert? CSUN 2017 Assistive Technology Conference

13 CSUN 2017 Assistive Technology ConferenceWell… From the bbc website - Boiling the testing down… The key skill seemed to be the ability to distinguish cancerous tissue samples from healthy tissue samples… CSUN 2017 Assistive Technology Conference

14 CSUN 2017 Assistive Technology ConferenceWell… From the bbc website - CSUN 2017 Assistive Technology Conference

15 CSUN 2017 Assistive Technology ConferenceBut… From the bbc website - Boiling it down some more… The key skill was actually the ability to recognize characteristics within images that would allow them to be categorize as one of two things – benign or malignant… There was in fact no need to have medical knowledge at all; or in fact any level of expert human judgement. CSUN 2017 Assistive Technology Conference

16 CSUN 2017 Assistive Technology ConferenceBut… How can I say this? From the bbc website - CSUN 2017 Assistive Technology Conference

17 Because, they got pigeons to do it…From the bbc website - After two weeks of training, the pigeons reached a level of 85% accuracy. Pooling the decisions from a group of four birds led to an impressive 99% accuracy. The exact same score as the highly-trained Clinicians. CSUN 2017 Assistive Technology Conference

18 Oddly, pigeons are apparently just very good at (image) recognition…"Pigeons can distinguish identities and emotional expressions on human faces, letters of the alphabet, misshapen pharmaceutical capsules, and even paintings by Monet vs Picasso" said Prof Edward Wasserman from the University of Iowa CSUN 2017 Assistive Technology Conference

19 CSUN 2017 Assistive Technology ConferenceSo, in this example… It clearly shows that the actual need for Expert judgement can be reduced, or in fact removed, when looking at the most atomic part(s) of what was previously thought of as a technically complex task. CSUN 2017 Assistive Technology Conference

20 CSUN 2017 Assistive Technology ConferenceAnd, it’s not just pigeons that are adept at learning to recognize things… CSUN 2017 Assistive Technology Conference

21 CSUN 2017 Assistive Technology ConferenceAnd, it’s not just pigeons that are adept at learning to recognize things… But, probably more like this… In terms of evolution / sophistication. But, moving very fast… CSUN 2017 Assistive Technology Conference

22 CSUN 2017 Assistive Technology ConferenceMachine Learning… Massive field within Computer Science inc. Neural networks; Deep learning; Enables computers to do things that they have not been explicitly programmed to do. CSUN 2017 Assistive Technology Conference

23 CSUN 2017 Assistive Technology ConferenceHere, we’ll focus on possible areas of interest in Accessibility Testing… Classification (Supervised Learning); Natural Language Processing (NLP); Image recognition; Image similarity comparison; CSUN 2017 Assistive Technology Conference

24 Classification (Supervised Learning)…Supervised Learning is the ML task of inferring a function from pre-labeled training data; CSUN 2017 Assistive Technology Conference

25 Classification (Supervised Learning)…In ML, classification is the problem of identifying into which set of categories a new observation belongs; Achieved on the basis of a training set of data whose category membership is known e.g. pre-labeled. Classification is an example of pattern recognition. CSUN 2017 Assistive Technology Conference

26 Simple example of Classification…Classification (Supervised Learning) Assigning a given as "spam" or "non-spam". CSUN 2017 Assistive Technology Conference

27 A more complex example…Classification (Supervised Learning) Assigning a diagnosis to a patient based on observed characteristics: gender; blood pressure; presence or absence of certain symptoms; etc… CSUN 2017 Assistive Technology Conference

28 How can a machine infer a function?Collect training data – e.g. spam s marked as spam, and non-spam s marked as non-spam; Then use functionality that allows each to be plotted on a chart; Then use statistical analysis to determine the function that separates plots for spam and non-spam s in the best way. CSUN 2017 Assistive Technology Conference

29 Next - What is Natural Language Processing (NLP)?It is the ability of a computer program to understand text that has been written by a human; or words that have been spoken by a human. Amazon’s Alexa provides a good example of a Machine that uses NLP techniques. Aside: In case you search for NLP - It is not - Neuro- linguistic programming (NLP). CSUN 2017 Assistive Technology Conference

30 CSUN 2017 Assistive Technology ConferenceWhat can NLP be used for? Information extraction - taking plain text, and extracting the pertinent information from it; Summarization - taking plain text, and summarizing it; Determining the main topics within text; And, loads, loads more… CSUN 2017 Assistive Technology Conference

31 Next - What is Image Recognition…The process of identifying and detecting an object or a feature in a digital image or video. Powering apps like Google Goggles / CamFind Apps – take a photo and the app tells you what is in the photo; CSUN 2017 Assistive Technology Conference

32 What can image recognition be used for?Optical Character Recognition - the identification of text in images using computer software. CSUN 2017 Assistive Technology Conference

33 Next - What is Image Similarity Comparison…The process of determining how similar different images are. There are loads of methods, with some of the lighter- weight ones boiling an image down to a small square of pixels (16 x 16). CSUN 2017 Assistive Technology Conference

34 Machine Learning tools available for Accessibility TestingSo, we have… Machine Learning tools available for Accessibility Testing Classification (Supervised Learning); Natural Language Processing (NLP); Image recognition; Image similarity comparison; And there will be many more. CSUN 2017 Assistive Technology Conference

35 Now, where can they be used in Automatic Testing?…CSUN 2017 Assistive Technology Conference

36 As input into our atomic tests…For example, determining: a data-table from a layout table; a decorative image from a non- decorative image; if an image contains text; if an image shows charted information; if text is in a foreign language. Etc, etc, etc, etc… CSUN 2017 Assistive Technology Conference

37 Or, as atomic tests themselves…Determining: if alternative text is suspicious; if the alternative text for a complex image is a good summary of its long text description; if an alternative text value is similar to accepted alternative text values for broadly similar images used in a broadly similar contexts Etc, etc, etc, etc… CSUN 2017 Assistive Technology Conference

38 In the field of Accessibility TestingGoing forward… In the field of Accessibility Testing Ensure that we don’t limit ourselves to what “we believe can only be tested by humans”. Look to create more and more atomic tests; To enable things to be determined with less and less reliance on human skills / expertise. Look to collect vast numbers of examples of things – in ML such collections are called Corpora. Look to take / apply more and more thinking from Machine Learning. CSUN 2017 Assistive Technology Conference

39 Questions? Pushing the Boundaries of Automatic Testing Through Machine Learning