Datapool¶

Datapool functionality can be used as a way to manage batch processing.

In this section, you will see more details on how to interact with this BotCity Maestro feature through your automation code.

Tip

See more details on how to create a Datapool and add new items through the Maestro interface by accessing this link.

Consuming items from a Datapool¶

The first step is to obtain the Datapool reference through its unique identifier (label).

After obtaining the Datapool reference, we can use a repeat loop to check as long as there are items to be processed.

Python

# Consuming the next available item and reporting the finishing state at the end
datapool = maestro.get_datapool(label="Items-To-Process")

while datapool.has_next():
    # Fetch the next Datapool item
    item = datapool.next(task_id=execution.task_id)
    if item is None:
        # Item could be None if another process consumed it before
        break

    # Processing item...

    # Finishing as 'DONE' after processing
    item.report_done()

Warning

Remember to always include in the code the report regarding the finishing state of the item that was processed.

This is extremely important so that item states are updated within the Datapool in Maestro.

Manipulating a Datapool item¶

In addition to reporting the finishing state of an item, we can perform other operations in the code.

It is possible to obtain some information about the item, as well as specific data based on the Schema that was created.

Python

# Fetch the next Datapool item
item = datapool.next(task_id=execution.task_id)

# Getting the value of some specific field of the item
item_data = item["data-label"]

# Finishing as 'DONE' after processing
item.report_done()

# Finishing item processing as 'ERROR'
item.report_error()

Datapool operations¶

Through the Maestro SDK, we can perform other operations with the Datapool in addition to checking and consuming the items to be processed.

Python

# Getting the Datapool reference
datapool = maestro.get_datapool("ProductsData")

# Checking if the Datapool is active
print(datapool.is_active())

# Checking if the Datapool is empty
print(datapool.is_empty())

# Marking the Datapool as active
datapool.activate()

# Marking the Datapool as inactive
datapool.deactivate()

Adding new items¶

We can add new items to the Datapool using the Maestro SDK.

Tip

This method can be useful if you want to use a Python script to populate the Datapool with the items that will be processed.

Python

# Instantiating a new Datapool item based on the Schema that has been defined
new_item = DataPoolEntry(
    values={
        "product_name": "Smartphone",
        "product_category": "Electronic",
        "product_value": "2000"
    }
)

# Getting the Datapool reference
datapool = maestro.get_datapool("ProductsData")

# Adding a new item
datapool.create_entry(new_item)