Skip to content

Automating a Windows Application

In some cases, we can automate Windows applications through element properties, as an alternative to computer vision.

This can be a useful alternative when we don't want to worry about resolution issues. We can simply interact with the application structure through the attributes of elements and windows.

This tutorial will guide you through the process of creating a simple automation that interacts with the elements of a Windows application.

Creating a New Project

The first step is to create a project for a desktop automation, you can follow the same steps defined in the previous section.

See how to create a Desktop Bot project.

Target Application and Spy Tools

For this simple example, we are going to use the standard Windows notepad.

To help us inspect this application, we are going to use the Accessibility Insights for Windows spy tool.

See more details about the spy tools usage.

Interacting with the Application

Connecting

With the project open, let's insert the first step in the bot.py file, which is to establish the connection with the target application.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "notepad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)
...

With the application open, we can use the spy tool to see some information about the elements.

Spy tool inspect

Finding Windows and Elements

With the app connected, we can now use the reference from the main window and collect the element referring to the text editor, for example.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "notepad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the notepad main window context using 'title_re' selector
    main_window = bot.find_app_window(title_re="Untitled - Notepad")

    # Getting the edit text area using the element 'title'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, title="Text Editor")
...

Note that we used the title property that we got from the spy tool. You can basically use any information that appears to filter a certain element.

Element properties

Tip

Another way to inspect the application's control elements is to use the print_control_identifiers method. After opening the dialog box and assigning it to a variable, the print_control_identifiers method should print all available controls. See more details about this here.

Performing Actions

With the element found, we can perform the operations. As this element is of type Edit, we can insert a text content.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "notepad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the notepad main window context using 'title_re' selector
    main_window = bot.find_app_window(title_re="Untitled - Notepad")

    # Getting the edit text area using the element 'title'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, title="Text Editor")

    # Typing some text
    edit.type_keys("Hello! Welcome to BotCity!", with_spaces=True)
...

Info

See more details about the methods available for each different type of element at this link.

Running the Bot

Just the code above would be enough for us to start notepad and insert a text content.

We have established a connection with the application and are performing operations in the context of a specific element.

Running the code, the final result will look like this:

Result

Accessing other elements in Notepad

With the same approach as above, using the control_type and title property, let's access other notepad elements.

Accesing menu_select

We'll start by accessing the Page Setup option from the File menu, through the menu_select method.

# Acessing option Page Setup of File Menu
main_window.menu_select("File -> Page Setup")
...

Using select to choose an option on the box of selection

We will interact with the Size element, as we see below:

Paper-Setup

To select the paper size, let's map the control_type and title of the size element, thats we can select them via the select method.

This time we will use another approach to get the property of the element, the print_control_identifiers, as we see below:

# Priting the control identifiers from the "Page Setup" window.
main_window.print_control_identifiers()
...
Running the code above, the controls of the Page Setup window will be generated in your terminal.

Control-Identifiers

Where we can see the title and control_type of element, which will be used below, as parameters of find_app_element method.

# Finding element to select page size
size = bot.find_app_element(from_parent_window=main_window, control_type="ComboBox",         
                             title="Size:")
# Selecting page type
size.select("A5")
...

Using invoke to choose an the radio button

Here, we will use invoke to select the Landscape radio button.

Radio-button

Still using controls identifiers.

Control-Identifiers

Let's get the title and control_type of the Landscape element to find it below:

# Finding element to select radio button "Landscape"
orientation = bot.find_app_element(from_parent_window=main_window, 
                                    control_type="RadioButton", title="Landscape")
# Method `click` is an alias of `invoke`
orientation.invoke()

# Finding element to click on button "OK"
btn_ok = bot.find_app_element(from_parent_window=main_window, title="OK")
# Clicking on button
btn_ok.click()
...

Info

In this case, it was not possible to use the methods click, and click_input to select the radio button, because they only have control over elements with property control_type="Button".

Info

invoke uses the so called InvokePattern that represents a default action for the control. When using Application(backend="uia"), the click method is an alias of invoke, where the InvokePattern action brings the window to the foreground, allowing interaction with the element.

After the above actions, the page configuration looked like this:

Page-setup

Using type_keys and click to save the notepad in the file explorer.

To finish, we will access the save as option in the file menu. 

# Acessing Menu Save As
main_window.menu_select("File -> Save As...")
...

This action will open the file explorer window, as we see below:

Save-As

Thus, we will define a name for the file, in this case it will be tutorial.txt, and we will click on the save button, through the click method.

# Finding element to type the name of file
file_name = bot.find_app_element(from_parent_window=main_window, class_name="Edit", 
                                  title="File name:")
# Typing the file name
file_name.type_keys("tutorial.txt")

# Finding element to clicks on button save
btn_save = bot.find_app_element(from_parent_window=main_window, class_name="Button", 
                                 title="Save")
# Clicking on button save
btn_save.click()
...

Complete Code

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "notepad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the notepad main window context using 'title_re' selector
    main_window = bot.find_app_window(title_re="Untitled - Notepad")

    # Getting the edit text area using the element 'title'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, title="Text Editor")

    # Typing some text
    edit.type_keys("Hello! Welcome to BotCity!", with_spaces=True)

    # Acessing option Page Setup of File Menu
    main_window.menu_select("File -> Page Setup")

    # Finding element to select page size
    size = bot.find_app_element(from_parent_window=main_window, class_name="ComboBox", 
                                    title="Size:")
    # Selecting page type
    size.select("A5")

    # Finding element to select radio button "Landscape"
    orientation = bot.find_app_element(from_parent_window=main_window, 
                                        control_type="RadioButton", title="Landscape")
    # Method click() is an alias of invoke()
    orientation.invoke()

    # Finding element to click on button "OK"
    btn_ok = bot.find_app_element(from_parent_window=main_window, title="OK")
    # Clicking on button
    btn_ok.click()

    # Acessing option Save As of file Menu
    main_window.menu_select("File -> Save As...")

    # Finding element to type the name of file
    name_file = bot.find_app_element(from_parent_window=main_window, class_name="Edit", 
                                        title="File name:")
    # Typing the file name
    name_file.type_keys("tutorial.txt")

    # Finding element to clicks on button save
    btn_save = bot.find_app_element(from_parent_window=main_window, class_name="Button", 
                                        title="Save")
    # Clicking on button save
    btn_save.click()
...

Important

Some applications may be more difficult to inspect and find information about elements.

Use the spy tools as a reference to determine if this is the best alternative for your automation.

Conclusion

Under this tutorial you learned:

  • How to use a spy tool to inspect a Windows application.

  • How to connect to an application and view its attributes and properties.

  • How to interact with an element that has been found.