Skip to content

Desktop Automation

Using the power of computer vision to find and interact with elements on your computer, BotCity's Desktop Automation framework allows you to automate any task on your computer.

The following sections will help you to understand how to use the Desktop Automation framework and all the features available divided in different categories.

  • Display: Learn how to interact with the display and get screenshots, display size and much more.
  • Computer Vision: Learn how to use computer vision to find elements on your screen.
  • Keyboard: Learn how to type, use shortcuts and more.
  • Mouse: Learn how to interact with the mouse and perform clicks, drag and drop and more.
  • Clipboard: Learn how to exchange information with the clipboard.
  • System: Learn how to interact with process on the system and start applications.
  • Browser: Learn how to interact with the browser to open webpages and files.
  • Waits: Learn about wait methods such as sleep, wait for file and more.
  • Windows Applications: Learn how to interact with the elements of an application.

System Dependencies for Python

If you are using Python on a Linux system, you will need to install the following dependencies:

apt install python3-tk
apt install scrot
apt install xclip
  • The code above is for Ubuntu. If you are using a different distribution, please refer to the documentation for your distribution.

System issues when using computer vision

If you are using a Linux system, you may have problems using computer vision features.

In some cases, the elements may not be found and also when saving a screenshot the result is an image with no content.

This could be caused by the Wayland graphic interface protocol. This resource is common in newer versions of Linux systems, and can cause some blocks when trying to use the graphics resources.

To resolve this issue, you can try the following steps:

  • Using an editor, access the file at: /etc/gdm3/custom.conf.
  • Find and uncomment the line: #WaylandEnable=false.
  • After saving the changes, use the sudo systemctl restart gdm3 command to restart the system.
  • After that, your Linux system will be using X instead of Wayland.

Note

The steps above have been tested on Ubuntu. If you are using a different distribution, please refer to the documentation for your distribution.