dt.batch

Scheduling batch jobs at scale

Up to 10K jobs in 10 seconds in parallel and/or with dependencies

Create and schedule batch job DAGs programmatically via the GUI, Python API, or JSON

Deploy batch jobs on Datatailr with 3 lines of code. Just put what you want to run into __batch_main__ function.

Deploy services as you would deploy an application.

Jobs can be parameterized and created on the fly, as part of another, scheduled job

Customizable display; user settings saved between sessions

Flexible graph-based batch processing and orchestration
Automatically pre-warm machines
Automatically spin VMs up and down for enormous scale
Optimal memory and CPU allocationFailed jobs retried automatically until success
Optimal memory and CPU allocationFailed jobs retried automatically until success
Rerun only failed parts of batch runs
Full reproducibility to test your code: all batch jobs run the latest version of the image by default
When triggering a re-run of a previously executed batch, we can choose to run it with newer versions of the images

Batch jobs planning

Next run time is displayed for each scheduled job

When the expiration date is reached, batch job transitions to a 'stopped' state

Batch jobs visibility

Access to stdout and stderr of each job directly from the batch run view
Display each job's custom parameters

Gantt view of execution times

Every process is entitled in order to track and allocate costs by user, and set group limits

Deploying Applications

Add 3 lines of code to deploy your application - Datatailr integrates Dash, Streamlit and Bokeh.
Or, take your code out of Datatailr and run it outside by commenting out those lines.

Excel Add-In

Use Excel as an interface, with the unlimited power of the cloud
Built-in security: data encryption, version control, user level permissioning
Write your own custom functions
Array operations
Streaming data

Add 3 lines of code to create and deploy a powerful Python-based Excel add-in.

Run back-tests of your code against your data pipeline