Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ai-assistance.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ After you upload the [Documentation Context](https://docs.tilebox.com/llms-full.

Here's an example prompt to get you started.

```txt Example prompt
```plaintext Example prompt
Generate a script that

- Creates a cluster
Expand Down
6 changes: 3 additions & 3 deletions api-reference/tilebox.workflows/Client.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@ def Client.clusters() -> ClusterClient
A client for managing clusters.

```python
def Client.recurrent_tasks() -> RecurrentTaskClient
def Client.automations() -> AutomationClient
```

A client for scheduling recurrent tasks.
A client for scheduling automations.

## Task runners

Expand All @@ -65,7 +65,7 @@ client = Client(
# access sub clients
job_client = client.jobs()
cluster_client = client.clusters()
recurrent_task_client = client.recurrent_tasks()
automation_client = client.automations()

# or instantiate a task runner
runner = client.runner("dev-cluster", tasks=[...])
Expand Down
Binary file modified assets/console/api-keys-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/console/api-keys-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/automation-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/automation-edit-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/automation-edit-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/automation-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/automations-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/automations-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/console/cluster-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/console/cluster-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/datapoints-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/datapoints-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/console/datasets-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/console/datasets-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed assets/console/recurrent-task-dark.png
Binary file not shown.
Binary file removed assets/console/recurrent-task-light.png
Binary file not shown.
Binary file added assets/console/storage-event-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/console/storage-event-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/console/usage-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/console/usage-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed assets/workflows/recurrent-tasks/cron-dark.png
Binary file not shown.
Binary file removed assets/workflows/recurrent-tasks/cron-light.png
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Diff not rendered.
Diff not rendered.
Diff not rendered.
6 changes: 3 additions & 3 deletions console.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,11 @@ data = sentinel2_msi.collection("S2A_S2MSI1C").load(

## Workflows

The workflows section of the console allows you to view jobs, create clusters and create recurring and near real-time tasks.
The workflows section of the console allows you to view jobs, create clusters and create near real-time automations.

<Frame>
<img src="/assets/console/recurrent-task-light.png" alt="Tilebox Console" className="dark:hidden" />
<img src="/assets/console/recurrent-task-dark.png" alt="Tilebox Console" className="hidden dark:block" />
<img src="/assets/console/automation-light.png" alt="Tilebox Console" className="dark:hidden" />
<img src="/assets/console/automation-dark.png" alt="Tilebox Console" className="hidden dark:block" />
</Frame>

## Account
Expand Down
2 changes: 1 addition & 1 deletion mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@
"group": "Near-Real Time",
"icon": "bolt",
"pages": [
"workflows/near-real-time/recurrent-tasks",
"workflows/near-real-time/automations",
"workflows/near-real-time/cron",
"workflows/near-real-time/storage-events"
]
Expand Down
1 change: 1 addition & 0 deletions vale/styles/config/vocabularies/docs/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,4 @@ coroutines
(?i)Imager
(?i)pushbroom
rollout
(?i)automations
2 changes: 1 addition & 1 deletion workflows/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ This section provides guides showcasing how to use the Tilebox workflow orchestr
<Card title="Configure shared data access" icon="box-archive" href="/workflows/caches" horizontal>
Learn to configure shared data access for all tasks of a job using caches.
</Card>
<Card title="Trigger Jobs in near-real-time" icon="bolt-lightning" href="/workflows/near-real-time/recurrent-tasks" horizontal>
<Card title="Trigger Jobs in near-real-time" icon="bolt-lightning" href="/workflows/near-real-time/automations" horizontal>
Trigger jobs based on events or schedules, such as new data availability or CRON schedules.
</Card>
</CardGroup>
Expand Down
92 changes: 92 additions & 0 deletions workflows/near-real-time/automations.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
---
title: Automations
description: Process data in near-real-time by triggering jobs based on external events
icon: repeat
---

## Introduction

Tilebox Workflows can execute jobs in two ways: a one-time execution triggered by a user, typically a batch processing, and near-real-time execution based on specific external events.
By defining trigger conditions, you can automatically submit jobs based on external events.

Tilebox Workflows currently supports the following trigger conditions:

<CardGroup cols={1}>
<Card title="Cron Triggers" icon="clock" href="/workflows/near-real-time/cron" horizontal>
Trigger jobs based on a Cron schedule.
</Card>
<Card title="Storage Event Triggers" icon="right-to-line" href="/workflows/near-real-time/storage-events" horizontal>
Trigger jobs after objects are created or modified in a storage location such as a cloud bucket.
</Card>
</CardGroup>

<Info>
Dataset Event Triggers, which will trigger jobs when new data points are ingested into a Tilebox dataset, are on the roadmap. Stay tuned for updates.
</Info>

## Automations

To create a trigger, define a special task that serves as a prototype. In response to a trigger condition met, this task will be submitted as a new job. Such tasks are referred to as automations.

Each automation has a [task identifier](/workflows/concepts/tasks#task-identifiers), a [version](/workflows/concepts/tasks#semantic-versioning), and [input parameters](/workflows/concepts/tasks#input-parameters), just like regular tasks.
Automations also automatically provide a special `trigger` attribute that contains information about the event that initiated the task's execution.

## Automation Client

The Tilebox Workflows client includes a sub-client for managing automations. You can create this sub-client by calling the `automations` method on the main client instance.

### Listing Registered Automations

To list all registered automations, use the `all` method on the automation client.

```python Python
from tilebox.workflows import Client

client = Client()
automations = client.automations()

automations = automations.all()
print(automations)
```

```plaintext Output
[
AutomationPrototype(
name='Run MyCronTask every hour at 15 minutes past the hour',
prototype=TaskSubmission(
cluster_slug='dev-cluster',
identifier=TaskIdentifier(name='MyCronTask', version='v0.0'),
input=b'{"message": "Hello"},
dependencies=[],
display='MyCronTask',
max_retries=0),
storage_event_triggers=[],
cron_triggers=[CronTrigger(schedule='15 * * * *')],
)
]
```

### Registering Automations

To register an automation, use the `create_*_automation` methods specific to each trigger type provided by the automation client. Refer to the documentation for each trigger type for more details.

<CardGroup>
<Card title="Cron Triggers" icon="clock" href="/workflows/near-real-time/cron" horizontal />
<Card title="Storage Event Triggers" icon="right-to-line" href="/workflows/near-real-time/storage-events" horizontal />
</CardGroup>

## Overview in the Tilebox Console

You can also use the Tilebox Console to manage automations. Visit [the automations section](https://console.tilebox.com/workflows/automations) to check it out.

<Frame>
<img src="/assets/console/automations-light.png" alt="Tilebox Workflows automations in the Tilebox Console" className="dark:hidden" />
<img src="/assets/console/automations-dark.png" alt="Tilebox Workflows automations in the Tilebox Console" className="hidden dark:block" />
</Frame>

You can also register new automations or edit and delete existing ones directly from the console.

<Frame>
<img src="/assets/console/automation-edit-light.png" alt="Tilebox Workflows automations in the Tilebox Console" className="dark:hidden" />
<img src="/assets/console/automation-edit-dark.png" alt="Tilebox Workflows automations in the Tilebox Console" className="hidden dark:block" />
</Frame>
36 changes: 19 additions & 17 deletions workflows/near-real-time/cron.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,12 @@ icon: clock

## Creating Cron tasks

Cron tasks run repeatedly on a specified [cron](https://en.wikipedia.org/wiki/Cron) schedule. To create a Cron task, use `tilebox.workflows.recurrent_tasks.CronTask` as your tasks base class instead of the regular `tilebox.workflows.Task`.
Cron tasks run repeatedly on a specified [cron](https://en.wikipedia.org/wiki/Cron) schedule.
To create a Cron task, use `tilebox.workflows.automations.CronTask` as your tasks base class instead of the regular `tilebox.workflows.Task`.

```python Python
from tilebox.workflows import ExecutionContext
from tilebox.workflows.recurrent_tasks import CronTask
from tilebox.workflows.automations import CronTask

class MyCronTask(CronTask):
message: str
Expand All @@ -25,15 +26,16 @@ class MyCronTask(CronTask):

## Registering a Cron trigger

After implementing a Cron task, register it to be triggered according to a Cron schedule. When the Cron expression matches, a new job is submitted consisting of a single task instance derived from the Cron task prototype.
After implementing a Cron task, register it to be triggered according to a Cron schedule.
When the Cron expression matches, a new job is submitted consisting of a single task instance derived from the Cron task prototype.

```python Python
from tilebox.workflows import Client

client = Client()
recurrent_tasks = client.recurrent_tasks()
cron_task = recurrent_tasks.create_recurring_cron_task(
"my-cron-task", # name of the recurring cron task
automations = client.automations()
cron_automation = automations.create_cron_automation(
"my-cron-automation", # name of the cron automation
"dev-cluster", # cluster slug to submit jobs to
MyCronTask(message="World"), # the task (and its input parameters) to run repeatedly
cron_triggers=[
Expand All @@ -51,7 +53,7 @@ cron_task = recurrent_tasks.create_recurring_cron_task(

## Starting a Cron Task Runner

With the Cron task registered, a job is submitted whenever the Cron expression matches. But unless a [task runner](/workflows/concepts/task-runners) is available to execute the Cron task the submitted jobs remain in a task queue.
With the Cron automation registered, a job is submitted whenever the Cron expression matches. But unless a [task runner](/workflows/concepts/task-runners) is available to execute the Cron task the submitted jobs remain in a task queue.
Once an [eligible task runner](/workflows/concepts/task-runners#task-selection) becomes available, all jobs in the queue are executed.

```python Python
Expand Down Expand Up @@ -79,30 +81,30 @@ This task was triggered at 2023-09-25 19:12:00

## Inspecting in the Console

The [Tilebox Console](https://console.tilebox.com/workflows/recurrent-tasks) provides a straightforward way to inspect all registered Cron tasks.
The [Tilebox Console](https://console.tilebox.com/workflows/automations) provides a straightforward way to inspect all registered Cron automations.

<Frame>
<img src="/assets/workflows/recurrent-tasks/cron-light.png" alt="Tilebox Workflows recurrent tasks in the Tilebox Console" className="dark:hidden" />
<img src="/assets/workflows/recurrent-tasks/cron-dark.png" alt="Tilebox Workflows recurrent tasks in the Tilebox Console" className="hidden dark:block" />
<img src="/assets/console/automation-edit-light.png" alt="Tilebox Workflows automations in the Tilebox Console" className="dark:hidden" />
<img src="/assets/console/automation-edit-dark.png" alt="Tilebox Workflows automations in the Tilebox Console" className="hidden dark:block" />
</Frame>

Use the console to view, edit, and delete the registered Cron tasks.
Use the console to view, edit, and delete the registered Cron automations.

## Deleting Cron triggers
## Deleting Cron automations

To delete a registered Cron task, use `recurrent_tasks.delete`. After deletion, no new jobs will be submitted by that Cron trigger. Past jobs already triggered will still remain queued.
To delete a registered Cron automation, use `automations.delete`. After deletion, no new jobs will be submitted by that Cron trigger. Past jobs already triggered will still remain queued.

```python Python
from tilebox.workflows import Client

client = Client()
recurrent_tasks = client.recurrent_tasks()
automations = client.automations()

# delete the task as returned by create_recurring_cron_task
recurrent_tasks.delete(cron_task)
# delete the automation as returned by create_cron_automation
automations.delete(cron_automation)

# or manually by id:
recurrent_tasks.delete("0190bafc-b3b8-88c4-008b-a5db044380d0")
automations.delete("0190bafc-b3b8-88c4-008b-a5db044380d0")
```

## Submitting Cron jobs manually
Expand Down
91 changes: 0 additions & 91 deletions workflows/near-real-time/recurrent-tasks.mdx

This file was deleted.

Loading
Loading