google / automation-inspector Goto Github PK

License: Apache License 2.0

JavaScript 87.75% CSS 6.51% HTML 5.73%

automation-inspector's Introduction

Automation Inspector

An inspection tool for Chrome Automation API

The Chrome Automation API is an experimental accessibility API that allows developers to access the automation tree and events for the browser. The tree resembles the DOM tree, but only exposes the semantic structure of a page. It can be used to programmatically interact with a page by examining names, roles, and states, listening for events, and performing actions on nodes.

One of Chrome Automation's main purposes is to enable the development of assistive technologies written in JavaScript, for Chrome OS. This is similar to how MSAA, IA2 and UIA are used on Windows, and AX Accessibility is used for OS X. It can also be used by extensions, but because it is experimental, it is currently available only on the dev channel and to internal assistive technologies such as ChromeVox.

Automation Inspector is used to exercise the Automation API, to inspect the entire Chrome OS desktop, or to inspect a specific browser tab.

This tool is mainly useful for the following types of people:

Developers/QA working on Chrome accessibility support
Developers/QA working on assistive technologies for Chrome OS
Developers/QA working on accessibility of web applications looking (less so)

Installation

Chrome OS desktop: to inspect the Chrome OS desktop, you must switch your Chrome OS to the Dev Channel, and then install Automation Inspector as an app (insert link here).
Chrome tabs: you can inspect individual browser tabs on any OS. To do this, install Google Chrome Canary and then install Automation Inspector as an extension (insert link here).

Development

Building

The first time you build, do an npm install This will automatically run bower install for you. If you want to update the dependencies, use npm update && bower update

Next, type the following from the command line in the automation-inspector folder: grunt to build once, or: grunt default watch to keep the build updated as source files change.

Running from the local file system

Using developer mode from the Chrome extensions page at chrome://extensions, use the load unpacked extension feature, pointing it at the build/extension folder.

Releasing

Update the version number via: npm version patch, npm version minor or npm version major and grunt default compress to create app.zip and extension.zip for uploading to the Chrome web store in the build/package folder.

Usage tips

Finding a node

The find field is very powerful, and can be used in a number of ways:

Plain text search: this will search rows for the visible text shown. This can be slow on complex pages, it it will cause the entire page to be loaded a bit at a time.
/RegEx/ search: include slashes to search for rows with matching visible text. Slow on complex pages (similiar to plain text search).
${selector}: use jQuery-like format, which is a CSS selector wrapped inside $(), to find a node matching a given selector. This will currently return only the first item. This method uses the Automation API's domQuerySelector method. For example, use $('#my-special-element') to find the nearest automation node to that element.
{JSON FindParams}: this executes the Automation API's findAll method with syntax similar to FindParams. However, this method accepts valid JSON syntax only. You must provide strings rather than constants. For example, use { "state" : {"disabled": true }} rather than { state: { StateType.disabled: true }}.

automation-inspector's People

Contributors

Stargazers

Watchers

Forkers

nicetester buhijs arjunreddyd paradoxbusinessgroup parveen-rx ecutdavid robo-cloud casebell beingsane neotim isabella232 ghas-results thecodeofmontecristo

automation-inspector's Issues

Ability to hide more events

There can be a lot of noisy mutations in the page.

It might be nice to limit event groups to those only from keyboard usage.

Highlight the currently selected node in the web content

Draw a rectangle around the current item.

We will need to create a chrome highlight api that will set an attribute such as
automation-api-highlight=true

Then we can inject css with a rule like *[automation-api-highlight=true] { outline: 3px solid #f00 !important; }

Property changes not always highlighted

Currently we highlight what properties were changed in a nodeChanged notification.

However, we are only able to do this if the node was previously displayed in the DOM tree -- otherwise we don't know what the original properties were.

It might be nice if the nodeChanged notification passed this info to us.

Show a slug for each node that has one

A slug is something like p#para1

This is useful for understanding where the current node is in the tree. It's probably a good idea to expose this somewhere in the UI. Maybe it could be a property.

Allow JSON filters to be constructed via UI

The JSON syntax is easy to forget. We should allow users to say what roles, states, etc. they are interested in filtering for.

Use and display automation node ids

AutomationNode instances already have an internal id. These would be useful to have instead of generating our own id. It could also be useful for debugging Chrome.

Let's just expose and consume these.

Hit testing

Ability to pick an item with the mouse.

There's a hitTest function in automationNode that can be used.

Ability to limit views to a specific window

In app mode for Chrome OS, viewing the entire desktop tree along with all events provides an overwhelming ream of data.

We should prompt the user for a window that they wish to inspect.

Support execution of functions that require parameters

Support boundsForRange, setSelection and other functions that require parameters.

Local dev mode /third_party/*.js NOT FOUND

Hi @aleventhal
Very new to this extension, please forgive me if I missed something obvious.

Followed the "Running from the local file system" section from
https://github.com/google/automation-inspector/blob/master/README.md

Using developer mode from the Chrome extensions page at chrome://extensions, use the load unpacked extension feature, pointing it at the build/extension folder.

Unfortuently getting 404 for /third_party/*.js
(e.g., chrome-extension://enlmgkkaebjkhjbeajabgddgmggdeadf/third_party/jquery.fancytree.table.js)

A quick guess is something wrong with the Gruntfile.js, not sure right now though.
Happy to dig into it sometime late this week(maybe it's just a mistake I made 😅).

And ideas? Thanks!

Resizable panels

We should allow for panel resizing. Should work with the keyboard ala https://codepen.io/rstrahl/pen/eJZQej

[need investigate] extension break on canary with ReferenceError

Getting ReferenceError: ids is not defined after downloading the extension and click the extension icon when viewing a webpage.

The error trace is not very helpful...
Guessing an investigation need to be made.

Chrome Version: 77.0.3847.0 (Official Build) canary (64-bit)
System Version: macOS 10.14.5 (18F132)

Screen reader accessibility

Make it work with a screen reader.

Example accessible treegrid:
https://rawgit.com/w3c/aria-practices/treegrid/examples/treegrid/treegrid-1.html

Depends on Fancytree work here:
mar10/fancytree#709

Does not walk into iframes

Steps:

Open a document with an iframe
Inspect

Notice that the contents of the iframe are not shown in the inspector tree view.

Using $(selector) syntax only finds one node

This is because automationNode.querySelectorAll() doesn't exist. Only querySelector() is available.

I started a patch to work on this for Chrome, but it was hard because WebNode also didn't have querySelectorAll(), nor was there a concept of a static node list in the public web APIs.

Remote access

Allow one Automation Inspector on one computer to inspect Chrome on a different computer.

This would have a big benefit in that focus would remain in the inspected app, instead of moving back and forth between the app and inspector, which can have big side effects.