Myujth

I am trying to build a process that invokes AWS lambda, which then utilizes AWS SNS to send messages that trigger more lambdas. Each such triggered lambdas write an output file to S3. The process is as depicted below -

My question is this - How can I know that all lambdas are done with writing files? I want to execute another process that collects all these files and does merging. I could think of two obvious ways -

1. Constantly monitor s3 for as many output files as SNS messages. Once, total count reaches, invoke the final merging lambda.


2. Use a db as sync source, write counts for that particular job/session and keep monitoring it till the count reaches SNS messages count.

Both solutions require constant polling, which i would like to avoid. I want to do this in an event driven manner. I was hoping for Amazon SQS would come to my rescue with some sort of "empty queue lambda trigger", but SQS only supports lambdas triggering on new messages. Is there any known way to achieve this in an event driven manner in AWS? Your suggestions/comments/answers are much appreciated.

AWS Step Functions (a managed state machine service) would be the obvious choice. AWS has some examples as starting points. I remember one being a looping state that you could probably apply to this use case.

Another idea off top of my head...

Create an "Orchestration Lambda" that has the list of your files...

1. Orchestration Lambda invokes a "File Writer Lambda" in a loop, passing the file info. The invokeAsync(InvokeRequest request) returns a Future object. Orchestration Lambda can check the future object state for completion.




2. Orchestration Lambda can make a similar call to the "File Writer Lambda" but instead use the more flexible method: invokeAsync(InvokeRequest request, AsyncHandler asyncHandler). You can make an inner class that implements this AsyncHandler and monitor the completion there in the Orchestration Lambda. It is a little cleaner than all the loops.

There are probably many ways to solve this problem, but there are two ideas.

Personally, I prefer the idea with "Step Functions".

But if you want to simplify your architecture, you could create trigered lambda function. Chose 'S3 trigger' in left side of lambda function designer and configure it bottom.

Check out more - Using AWS Lambda with Amazon S3

But in this case you have to create more sophisticated lambda function wich will check that all apropriate files are uploaded on S3 and after this start your merge.

I would propose a couple of options here:

Step Functions:

This is a managed service for state machines. It's great for co-ordinating workflows.

Atomic Counting:

If you know the number of things in advance, you could initialize an Atomic Counter in DynamoDB and then atomically decrement it as work completes. Use DynamoDB Streams to trigger Lambda invocation when the counter is mutated, and trigger your next phase (or end of work) when the counter hits zero. Note that whenever an application creates, updates, or deletes items in the table, DynamoDB Streams writes a stream record, so every mutation of the counter would trigger your Lambda.

Note that DynamoDB Streams guarantees the following:

• Each stream record appears exactly once in the stream.




• For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

The stated problem seems a suitable candidate for the Saga Pattern.
Basically Saga is described like any long running , distributed process.

As mentioned earlier , the AWS platform allows using Step functions to implement a Saga, as described here enter