Introduction to Workflow Operations
A Cargo workflow is a living entity that processes data continuously based on triggers and conditions. Understanding how to operate, monitor, and maintain workflows is crucial for ensuring reliable data processing in your organization.
Inspecting runs
Once a workflow is live it will continuously process data based on defined triggers and conditions.
Given the complex nature of modern data flows it is virtually impossible to predict all possible edge cases and data combinations.
To address that Cargo gives full, detailed visibility into historical workflow runs. With runs view you can view specific workflow invocation with all data-points that were processed by the workflow.
This allows to identification of possible improvements to the workflow logic and data mapping to cater to all possible scenarios.
Learn more about the runs view in the Editor reference.
Publishing updates
Whenever an improvement is needed, whether is it a result of organizational change, dataflow refinement, or simplified fixing bugs, Cargo editor offers a simple, yet robust versioning system.
It separates a draft version, that you can always work on, testing new things, without worrying about impacting the existing production setup.
When your updated draft is ready you can publish it and that will replace the current live version.
On the other hand, you can also start drafting again from any historical release if you need to rollback you workflow.
Learn more about the versions in the Editor reference.
Re-running failed runs
As indicated above multiple errors can cause runs to fail to execute correctly and need to be re-run post-corrections to the execution logic. Cargo provides a feature to re-run these failing nodes directly from the runs view.
After the failed run and the bug are identified just update the workflow logic. When the logic is ready to handle the erroneous case publish it. Once the update is live navigate to the runs view again, find the failed runs and run them again.
Failed runs can be re-run again from scratch or only from the moment it failed ensuring that successful steps are not duplicated.
Learn more about the runs view in the Editor reference.
Health Status
Not every situation when a workflow fails must mean an error. Some of the workflows may work in the best-effort model and by the design they can't succeed in all cases. On the other hand, workflow health needs to be closely monitored to avoid disruptions in key processes.
Cargo's editor exposed another feature to cater to the need to continuously monitor workflow health status and if a defined threshold is hit send a slack notification.
Learn more about the health status in the Editor reference.
What's next?
Check our best practices to get real life insights into building reliable and scalable workflows.