Automating a Windows Application¶

In some cases, we can automate Windows applications through element properties, as an alternative to computer vision.

This can be a useful alternative when we don't want to worry about resolution issues. We can simply interact with the application structure through the attributes of elements and windows.

This tutorial will guide you through the process of creating a simple automation that interacts with the elements of a Windows application.

Creating a New Project¶

The first step is to create a project for a desktop automation, you can follow the same steps defined in the previous section.

See how to create a Desktop Bot project.

Target Application and Spy Tools¶

For this simple example, we are going to use the standard Windows wordpad.

To help us inspect this application, we are going to use the Accessibility Insights for Windows spy tool.

See more details about the spy tools usage.

Interacting with the Application¶

Connecting¶

With the project open, let's insert the first step in the bot.py file, which is to establish the connection with the target application.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)
...

With the application open, we can use the spy tool to see some information about the elements.

Finding Windows and Elements¶

With the app connected, we can now use the reference from the main window and collect the element referring to the text editor, for example.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the Wordpad main window context using 'title_re',  'class_name' and 'control_type' selector
    main_window = bot.find_app_window(title_re='Document - WordPad', class_name="WordPadClass",
                                        control_type="Window")

    # Getting the edit text area using the element 'class_name' and 'control_type'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, control_type="Document",
                                    class_name='RICHEDIT50W')
...

Note that we used the class_name and control_type property that we got from the spy tool. You can basically use any information that appears to filter a certain element.

Tip

Another way to inspect the application's control elements is to use the print_control_identifiers method. After opening the dialog box and assigning it to a variable, the print_control_identifiers method should print all available controls. See more details about this here.

Performing Actions¶

With the element found, we can perform the operations. As this element is of type Document, we can insert a text content.

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the Wordpad main window context using 'title_re',  'class_name' and 'control_type' selector
    main_window = bot.find_app_window(title_re='Document - WordPad', class_name="WordPadClass",
                                         control_type="Window")

    # Getting the edit text area using the element 'class_name' and 'control_type'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, control_type="Document",
                                     class_name='RICHEDIT50W')

    # Typing some text
    edit.type_keys("Hello! Welcome to BotCity!", with_spaces=True)
...

Info

See more details about the methods available for each different type of element at this link.

Running the Bot¶

Just the code above would be enough for us to start Wordpad and insert a text content.

We have established a connection with the application and are performing operations in the context of a specific element.

Running the code, the final result will look like this:

Accessing other elements¶

With the same approach as above, using the control_type and title_re property, let's access other Wordpad elements.

Accesing `Print Preview`¶

# Accessing WordPad's 'Print Preview' option through the `control_type` and` title_re` selectors
print_preview = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                        title_re='Print preview')
print_preview.click()
...

Then we will select the Page Setup option

# Accessing the 'Page Setup' option through selectors `control_type` and` title_re`
page_setup = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                        title_re='Page setup')
page_setup.click()
...

Using `select` to choose an option on the box of selection¶

We will interact with the Size element, as we see below:

Before any manipulation, we will enter the context of the Page Setup window. It is a window within the scope of main_window but its attributes are not directly exposed to it. This way, to find the elements and attributes of the window Page Setup, it will be necessary to return it first.

# Obtaining context of the 'Page Setup' window through the 'Title_re' selector
page_setup_window = bot.find_app_element(from_parent_window=main_window, title_re='Page Setup')
...

Now yes, to select the paper size, let's map control_type and title from the size element so we can select them through theselect method.

This time we will use another approach to get the property of the element, the print_control_identifiers, as we see below:

# Priting the control identifiers from the "Page Setup" window.
page_setup_window.print_control_identifiers()
...

Running the code above, the controls of the Page Setup window will be generated in your terminal.

Where we can see the title and control_type of element, which will be used below, as parameters of find_app_element method.

# Finding element to select page size
size = bot.find_app_element(from_parent_window=main_window, control_type="ComboBox",         
                                title="Size:")
# Selecting page type
size.select("A5")
...

Using `invoke` to choose an the radio button¶

Here, we will use invoke to select the Landscape radio button.

Still using controls identifiers.

Let's get the title and control_type of the Landscape element to find it below:

# Finding element to select radio button "Landscape"
orientation = bot.find_app_element(from_parent_window=main_window, control_type="RadioButton",
                                        title="Landscape")
# Method `click` is an alias of `invoke`
orientation.invoke()

# Finding element to click on button "OK"
btn_ok = bot.find_app_element(from_parent_window=main_window, title="OK")
# Clicking on button
btn_ok.click()
...

Info

In this case, it was not possible to use the methods click, and click_input to select the radio button, because they only have control over elements with property control_type="Button".

Info

invoke uses the so called InvokePattern that represents a default action for the control. When using Application(backend="uia"), the click method is an alias of invoke, where the InvokePattern action brings the window to the foreground, allowing interaction with the element.

After the above actions, the page configuration looked like this:

Using `type_keys` and `click` to save the Wordpad in the file explorer.¶

To finish, we will access the Save option in the menu.

    # Accessing 'Save' icon of the top menu through the 'Control_Type' and 'Title' selectors
    save = bot.find_app_element(from_parent_window=main_window, control_type="Button",         
                                    title="Save")
    save.click()
...

This action will open the file explorer window, as we see below:

Thus, we will define a name for the file, in this case it will be tutorial.txt, and we will click on the save button, through the click method.

# Finding element to type the name of file
file_name = bot.find_app_element(from_parent_window=main_window, class_name="Edit", 
                                    title="File name:")
# Typing the file name
file_name.type_keys("tutorial.txt")

# Finding element to clicks on button save
btn_save = bot.find_app_element(from_parent_window=main_window, class_name="Button", 
                                    title="Save")
# Clicking on button save
btn_save.click()
...

Complete Code¶

from botcity.core import DesktopBot, Backend
...
def main():
    bot = DesktopBot()

    # Application path
    app_path = "wordpad.exe"

    # Launching the app
    bot.execute(app_path)

    # Connecting to the application using 'path' selector
    bot.connect_to_app(backend=Backend.UIA, path=app_path)

    # Getting the Wordpad main window context using 'title_re',  'class_name' and 'control_type' selector
    main_window = bot.find_app_window(title_re='Document - WordPad', class_name="WordPadClass",
                                        control_type="Window")

    # Getting the edit text area using the element 'class_name' and 'control_type'
    # We were able to view this property through the spy tool
    edit = bot.find_app_element(from_parent_window=main_window, control_type="Document",
                                    class_name='RICHEDIT50W')

    # Typing some text
    edit.type_keys("Hello! Welcome to BotCity!", with_spaces=True)

    # Accessing WordPad's 'print preview' option through the `control_type` and` title_re` selectors
    print_preview = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                            title_re='Print preview')
    print_preview.click()

    # Accessing the 'Page Setup' option through selectors `control_type` and` title_re`
    page_setup = bot.find_app_element(from_parent_window=main_window, control_type='Button',
                                         title_re='Page setup')
    page_setup.click()

    # Obtaining context of the 'Page Setup' window through the 'Title_re' selector
    page_setup_window = bot.find_app_element(from_parent_window=main_window, title_re='Page Setup')

    # Priting the control identifiers from the "Page Setup" window.
    page_setup_window.print_control_identifiers()

    # Finding element to select page size
    size = bot.find_app_element(from_parent_window=main_window, class_name="ComboBox", 
                                    title="Size:")
    # Selecting page type
    size.select("A5")

    # Finding element to select radio button "Landscape"
    orientation = bot.find_app_element(from_parent_window=main_window, 
                                           control_type="RadioButton", title="Landscape")
    # Method click() is an alias of invoke()
    orientation.invoke()

    # Finding element to click on button "OK"
    btn_ok = bot.find_app_element(from_parent_window=main_window, title="OK")
    # Clicking on button
    btn_ok.click()

    # Accessing 'Save' icon of the top menu through the 'Control_Type' and 'Title' selectors
    save = bot.find_app_element(from_parent_window=main_window, control_type="Button",         
                                    title="Save")
    save.click()

    # Finding element to type the name of file
    file_name = bot.find_app_element(from_parent_window=main_window, class_name="Edit", 
                                        title="File name:")
    # Typing the file name
    file_name.type_keys("tutorial.txt")

    # Finding element to clicks on button save
    btn_save = bot.find_app_element(from_parent_window=main_window, class_name="Button", 
                                        title="Save")
    # Clicking on button save
    btn_save.click()
...

Important

Some applications may be more difficult to inspect and find information about elements.

Use the spy tools as a reference to determine if this is the best alternative for your automation.

Conclusion¶

Under this tutorial you learned:

How to use a spy tool to inspect a Windows application.
How to connect to an application and view its attributes and properties.
How to interact with an element that has been found.

Automating a Windows Application¶

Creating a New Project¶

Target Application and Spy Tools¶

Interacting with the Application¶

Connecting¶

Finding Windows and Elements¶

Performing Actions¶

Running the Bot¶

Accessing other elements¶

Accesing Print Preview¶

Using select to choose an option on the box of selection¶

Using invoke to choose an the radio button¶

Using type_keys and click to save the Wordpad in the file explorer.¶

Complete Code¶

Conclusion¶

Accesing `Print Preview`¶

Using `select` to choose an option on the box of selection¶

Using `invoke` to choose an the radio button¶

Using `type_keys` and `click` to save the Wordpad in the file explorer.¶