Skip to content

Computer Vision

Using the methods below you will be able to use computer vision to interact with the elements that appear on the screen.

Add Image

By default, element images are fetched from the project's resources folder and added to the image map. However, it is possible to add an image that is in another directory in the image map and thus access it normally through the defined label.

Note

This operation is not necessary for images that are already in the project's default resources folder.

# Adding an image to the image map that is in a specific directory.
bot.add_image("element", "/my_directory/images/element.png")

// Adding an image to the image map that is in a specific directory.
addImage("element", "/my_directory/images/element.png");

// Adding an image to the image map that is in a specific directory.
desktopBot.addImage('element', '/my_directory/images/element.png')

// Adding an image to the image map that is in a specific directory.
desktopBot.addImage('element', '/my_directory/images/element.png')

Get Image from Map

You can get a specific image from the image map using a defined label.

# Getting the image that has the "element" label.
img = bot.get_image_from_map("element")

// Not yet implemented.

// Getting the image that has the "element" label.
const img = desktopBot.getImageFromMap('element')

// Getting the image that has the "element" label.
const img: Jimp = desktopBot.getImageFromMap('element')

Finding Elements

Using the find methods it is possible to search for an element on the screen using computer vision until a timeout happens. You can use the return of the find method to check if an element was found or not and based on that perform the necessary treatment.

# Print if the first_element was NOT found on the screen.
if not bot.find(label="first_element", matching=0.97, waiting_time=10000):
    print("This element was not found.")

# Print if the "second element" was found on the screen.
if bot.find(label="second_element", matching=0.97, waiting_time=10000):
    print("This element was found!")

# Using find_until will have the same effect as using find.
if bot.find_until(label="another_element", matching=0.97, waiting_time=10000):
    print("Found!")

// Print if the first_element was NOT found on the screen.
if(!find("first_element", 0.97, 10000)) {
    System.out.println("This element was not found");
}

// Print if the second_element was found on the screen.
if(find("second_element", 0.97, 10000)) {
    System.out.println("This element was found!");
}

// Print if the first_element was NOT found on the screen.
if (await desktopBot.find('first_element') === null) 
    console.log('This element not found')

// Print if the second_element was found on the screen.
if (await desktopBot.find('second_element') !== null) 
    print("This element was found!")

// Print if the first_element was NOT found on the screen.
if (await desktopBot.find('first_element') === null) 
    console.log('This element not found!')

// Print if the second_element was found on the screen.
if (await desktopBot.find('second_element') !== null) 
    print("This element was found!")

The find method above is the most common to use and will basically work for all cases where we need to search for an element on the screen. However, it is possible to use find methods for specific situations.

Find Text

# Searching for an element that contains only text.
if bot.find_text(label="txt_element", matching=0.97, waiting_time=10000):
    print("This text was found on screen!")

// Searching for an element that contains only text.
if(findText("txt_element", 10000)){
    System.out.println("This text was found on screen!");
}

// Searching for an element that contains only text
if (await desktopBot.findText('second_element') !== null)
    console.log('This text was found on screen!')

// Searching for an element that contains only text
if (await desktopBot.findText('second_element') !== null)
    console.log('This text was found on screen!')

Find All

In addition to performing operations with single elements, it is also possible to search for several different elements or all occurrences of the same element.

# Search for all occurrences of the element.
elements = bot.find_all(label="test_element", matching=0.97, waiting_time=10000)

# For each element found, print the coordinates.
for ele in elements:
    print(ele)

// Search for all occurrences of the element.
List<UIElement> elements = findAllUntil("test_element", 0.97, 20000);

// For each element found, print the coordinates.
for(UIElement ele : elements) {
    System.out.println(ele.getX());
    System.out.println(ele.getY());
    System.out.println(ele.getWidth());
    System.out.println(ele.getHeight());
}

// Not yet implemented.

// Not yet implemented.

Find Multiple

# List with the element's label.
elements_to_find = ["ele1", "ele2", "ele3", "ele4"]

# Searching for all elements will return a dictionary with the label and coordinates of each one.
elements = bot.find_multiple(labels=elements_to_find, matching=0.97, waiting_time=20000)

print(elements)

// Not yet implemented.

// List with the element's label.
const elements_to_find = ["ele1", "ele2", "ele3", "ele4"]

// Searching for all elements will return a dictionary with the label and coordinates of each one.
const elements = await desktopBot.find_multiple(elements_to_find)

console.log(elements)

// List with the elements label.
const elements_to_find: string[] = ["ele1", "ele2", "ele3", "ele4"]

// Searching for all elements will return a dictionary with the label and coordinates of each one.
const elements: Element[] | null[] = await desktopBot.find_multiple(elements_to_find)

console.log(elements)

Getting Elements

You can get information about the last element found and also the coordinates of a specific element.

Get Last Element

# Get the last element found on the screen.
last_element = bot.get_last_element()

# Printing the element coords.
print(last_element)

// Get the last element found on the screen.
UIElement lastElement = getLastElement();

// Printing the element coords
System.out.println(lastElement.getX());
System.out.println(lastElement.getY());
System.out.println(lastElement.getWidth());
System.out.println(lastElement.getHeight());

// Get the last element found on the screen.
const lastElement = desktopBot.getLastElement()

// Get the last element found on the screen.
const lastElement: Element = desktopBot.getLastElement()

Get Element Coordinates

# This method will search for the element and return its X and Y.
ele = bot.get_element_coords(label="test_element", matching=0.97)
print(ele)

# This method will search for the element and return its X and Y centered.
element_centered = bot.get_element_coords_centered(label="test_element", matching=0.97)
print(element_centered)

// This method will search for the element and return its X and Y.
Point ele = getCoordinates("<path_to_element.png>", 10000, false);
System.out.println(ele);

// This method will search for the element and return its X and Y.
const elementCoords = await desktopBot.getElementCoords('test_element')

// This method will search for the element and return its X and Y.
const elementCoords: object = await desktopBot.getElementCoords('test_element')