How to describe elements in Natural Language to AI

When using an AI-powered action, such as Click Element, Fill Input or others, you can describe the element to interact with in natural language.

It may be confusing at first, but you will quickly realise it's much simpler than using CSS selectors.

How our AI works

To best describe elements, you need to understand how our AI works.

Our AI is based on two main sources of information:

1. The DOM of the page (the structure of the page and its code)
2. The visible content of the page (what you see if you take a screenshot)

When you describe the element, the AI will first try to find it in the DOM using some custom-built heuristics. If it fails to find the element, it will try to find it in the visible content of the page using some other homemade algorithms.

If none of these methods work you will see an error message indicating that the element was not found.

Troubleshooting

All web pages are unique and there may be cases when Magic Inspector will fail to find the element.

These cases also depend on the action you are trying to perform.

Common failing reasons:

1. The element is not visible or not interactive (this is often the case for clickable elements with poor implementation)
2. The element is very similar to another element on the page and the AI is confused (this is often the case in applications with many similar components)

Solution 1

If an element is not found, try to refine your element description and be more specific.

For example, instead of "The credentials", try "The email input". Instead of "The add user button" try "The button in to bottom left corner with a plus (+) icon".

Solution 2

If after refining the element description the AI still fails to find the element, disable the AI and use a traditional CSS selector.

We made a guide on how to use CSS selectors to describe elements instead of using AI.