Amazon Simple Workflow Service (Amazon SWF) is an AWS service that coordinates work across distributed application components. Amazon SWF enables developers to build, run, and scale background jobs that have parallel or sequential steps. The coordination of tasks involves managing execution dependencies, scheduling, and concurrency in accordance with the logical flow of the application. With Amazon SWF, developers get full control over implementing processing steps and coordinating the tasks that drive them.
- Amazon SWF enables applications for a range of use cases, such as edia processing, web application back-ends, business process workflows, and analytics pipelines, to be designed as a coordination of tasks.
- Amazon SWF provides the AWS Flow Framework to allow developers use asynchronous programming in the development of their applications.
- By using Amazon SWF, developers benefit from ease of programming and have the ability to improve their applications’ resource usage, latencies, and throughputs.
- Amazon SWF stores tasks and assigns them to workers when they are ready, tracks their progress, and maintains their state, including details on their completion.
- Amazon SWF maintains an application’s execution state durable so that the application is resilient to failures in individual components. With Amazon SWF, AWS customers can implement, deploy, scale, and modify these application components independently.
Amazon SWF Features
Amazon SWF promotes a separation between the control flow of customers’ background job’s stepwise logic and the actual units of work that contain their unique business logic.
- It enables customers to separately manage, maintain, and scale “state machinery” of their application from the core business logic that differentiates it.
Amazon SWF replaces the complexity of custom-coded workflow solutions and process automation software with a fully managed cloud workflow web service.
- This eliminates the need for developers to manage the infrastructure plumbing of process automation so they can focus their energy on the unique functionality of their application.
Amazon SWF runs within Amazon’s high-availability data centers, so the state tracking and task processing engine is available whenever applications need them.
- Amazon SWF redundantly stores the tasks, reliably dispatches them to application components, tracks their progress, and keeps their latest state.
Amazon SWF lets you write your application components and coordination logic in any programming language and run them in the cloud or on-premises.
- It centralizes the coordination of steps in customers’ applications. The customer coordination logic does not have to be scattered across different components, but can be encapsulated in a single program.
Simple Workflow Concepts
The mechanism by which both the activity workers and the decider receive their tasks respectively is by polling the Amazon SWF service. Amazon SWF informs the decider of the state of the workflow by including, with each decision task, a copy of the current workflow execution history.
- The workflow execution history is composed of events, where an event represents a significant change in the state of the workflow execution. The history is a complete, consistent, and authoritative record of the workflow’s progress.
Amazon SWF access control uses AWS Identity and Access Management (IAM), allow customers to provide access to AWS resources in a controlled and limited way that doesn’t
I server endpoint. A node group is one or more Amazon EC2 instances that are deployed in an Amazon EC2 Auto Scaling group.
A cluster can contain several node groups, and each node group can contain several worker nodes. The managed node groups are able to have a maximum number of nodes. All instances in a node group must:
- Be the same instance type
- Be running the same Amazon Machine Image (AMI)
- Use the same Amazon EKS Worker Node IAM Role.
Amazon EKS provides a specialized Amazon Machine Image (AMI) called the Amazon EKS-optimized AMI. This AMI is built on top of Amazon Linux 2, and is configured to serve as the base image for Amazon EKS worker nodes.
- The AMI is configured to work with Amazon EKS out of the box, and it includes Docker, kubelet, and the AWS IAM Authenticator. The AMI also contains a specialized bootstrap script that allows it to discover and connect to the customers cluster’s control plane automatically.
Customers can deploy one or more worker nodes into a node group. Nodes are Worker machines in Kubernetes. Amazon EKS worker nodes run in customers’ AWS accounts, and it connects their cluster’s control plane via the cluster AP
The fundamental concept in Amazon SWF is the workflow. A workflow is a set of activities that carry out some objective, together with logic that coordinates the activities. Each workflow runs in an AWS resource called a domain, which controls the workflow’s scope. An AWS account can have multiple domains, each of which can contain multiple workflows, but workflows in different domains can’t interact.
When designing an Amazon SWF workflow, AWS clients need to precisely define each of the required activities. Once that is done, they register each activity with Amazon SWF as an activity type. While registering the activity, clients also need to provide information such as a name and version. Such expectations would inform the timeout values that you specify when registering your activities.
In the process of carrying out the workflow, some activities may need to be performed more than once, perhaps with varying inputs. Amazon SWF has the concept of an activity task that represents one invocation of an activity.
- An activity worker is a program that receives activity tasks, performs them, and provides results back. Note that the task itself might actually be performed by a person, in which case the person would use the activity worker software for the receipt and disposition of the task.
Both Activity tasks and the Activity workers that perform them can run synchronously or asynchronously. They can be distributed across multiple computers, potentially in different geographic regions, or they can all run on the same computer. Different activity workers can be written in different programming languages and run on different operating systems.
- The coordination logic in a workflow is contained in a software program called a decider. The decider schedules activity tasks, provides input data to the activity workers, processes events that arrive while the workflow is in progress, and ultimately ends (or closes) the workflow when the objective has been completed.
The role of the Amazon SWF service is to function as a reliable central hub through which data is exchanged between the decider, the activity workers, and other relevant entities such as the person administering the workflow. Amazon SWF also maintains the state of each workflow execution, which saves customers applications from having to store the state in a durable way.
The decider directs the workflow by receiving decision tasks from Amazon SWF and responding back to Amazon SWF with decisions. A decision represents an action or set of actions which are the next steps in the workflow.
- A typical decision would be to schedule an activity task. Decisions can also be used to set timers to delay the execution of an activity task, to request cancellation of activity tasks already in progress, and to complete or close the workflow.
SWF Use cases
Data Center Migration
Business critical operations are hosted in a private datacenter but need to be moved entirely to the cloud without causing disruptions. Amazon SWF-based applications can combine workers that wrap components running in the datacenter with workers that run in the cloud.
- To transition a datacenter worker seamlessly, new workers of the same type are first deployed in the cloud.
- The workers in the datacenter continue to run as usual, along with the new cloud-based workers.
- The cloud-based workers are tested and validated by routing a portion of the load through them. During this testing, the application is not disrupted because the workers in the datacenter continue to run.
- After successful testing, the workers in the datacenter are gradually stopped and those in the cloud are scaled up, so that they move entirely to a cloud workflow management application.
- This cloud workflow process can be repeated for all other workers in the datacenter so that the application moves entirely to the cloud.
AWS customers can encode Video using Amazon S3 and Amazon EC2. To do this customers large videos are uploaded to Amazon S3 in chunks. The upload of chunks has to be monitored. After a chunk is uploaded, it is encoded by downloading it to an Amazon EC2 instance. The encoded chunk is stored to another Amazon S3 location. After all of the chunks have been encoded in this manner, they are combined into a complete encoded file which is stored back in its entirety to Amazon S3.
- The entire application is built as a workflow where each video file is handled as one workflow execution.
- The tasks that are processed by different workers are: upload a chunk to Amazon S3, download a chunk from Amazon S3 to an Amazon EC2 instance and encode it, store a chunk back to Amazon S3, combine multiple chunks into a single file, and upload a complete file to Amazon S3.
- The decider initiates concurrent tasks to exploit the parallelism in the use case.
- The application state kept by Amazon SWF helps the decider control the workflow.
- The execution progress is continuously tracked in the Amazon SWF Management Console.
- If there are failures, the specific tasks that failed are identified and used to pinpoint the failed chunks.
Product Catalogs With Human Workers
While validating data in large catalogs, the products in the catalog are processed in batches. Different batches can be processed concurrently. For each batch, the product data is extracted from servers in the datacenter and transformed into CSV (Comma Separated Values) files required by Amazon Mechanical Turk’s Requester User Interface (RUI).
- The CSV is uploaded to populate and run the HITs (Human Intelligence Tasks).
- When HITs complete, the resulting CSV file is reverse transformed to get the data back into the original format.
- The results are then assessed and Amazon Mechanical Turk workers are paid for acceptable results.
- Failures are weeded out and reprocessed, while the acceptable HIT results are used to update the catalog.
- As batches are processed, the system needs to track the quality of the Amazon Mechanical Turk workers and adjust the payments accordingly.
- Failed HITs are re-batched and sent through the pipeline again
When using Amazon SWF, AWS customers implement workers to perform tasks. These workers can run either on cloud infrastructure, such as Amazon Elastic Compute Cloud (Amazon EC2), or on customers own premises. They can create tasks that are long-running, or that may fail, time out, or require restarts, or that may complete with varying throughput and latency.
Amazon SWF is suitable for a range of use cases that require coordination of tasks, including media processing, web application back-ends, business process workflows, and analytics pipelines.
Amazon SWF is supported by the AWS SDKs for Java, .NET, Node.js, PHP, PHP version 2, Python and Ruby, providing a convenient way to use the Amazon SWF HTTP API in the programming language of your choice.
- AWS customers can develop deciders, activities, or workflow starters using the API exposed by these libraries.
- Customers can access visibility operations through these libraries, and develop their own Amazon SWF monitoring and reporting tools.
AWS Flow Framework
The AWS Flow Framework is an enhanced SDK for writing distributed, asynchronous programs that can run as workflows on Amazon SWF. It is available for the Java and Ruby programming languages, and it provides classes that simplify writing complex distributed programs.
- Using AWS Flow Framework, customers can use preconfigured types to map the definition of their workflow directly to methods they program.
- The AWS Flow Framework supports standard object-oriented concepts, such as exception-based error handling.
- Programs written with the AWS Flow Framework can be created, executed, and debugged entirely within customers’ preferred editor or IDE.
HTTP Service API
Amazon SWF provides service operations that are accessible through HTTP requests. Customers can use these operations to communicate directly with Amazon SWF. They can also use them to develop their own libraries in any language that can communicate with Amazon SWF through HTTP.
- By using the service API customers can develop deciders, activity workers, or workflow starters.
- Customers can access visibility operations through the API to develop your own monitoring and reporting tools.