Triggering a Lambda function by an EventBridge Events rule can be used as a _serverless _replacement of cron job. The highest frequency of it is one invocation per minute so that it cannot be used directly if you need to schedule a Lambda function more frequently. For example, it may be refreshing an application with real time metrics from an Amazon Connect instance where some metrics are updated every 15 seconds. There is a post in the AWS Architecture Blog, and it suggests using AWS Step Functions. Or a usual recommendation is using Amazon EC2. Albeit being serverless, the former gets a bit complicated especially in order to handle the hard quota of 25,000 entries in the execution history. And the latter is not an option if you look for a serverless solution. In this post, I’ll demonstrate another serverless solution of scheduling a Lambda function at a sub-minute frequency using Amazon SQS.

Architecture

The solution contains 2 Lambda functions and each of them has its own event source: EventBridge Events rule and SQS.

  1. A Lambda function (sender) is invoked every minute by an EventBridge Events rule.
  2. The function sends messages to a queue with different delay seconds values. For example, if we want to invoke the consumer Lambda function every 10 seconds, we can send 6 messages with delay seconds values of 0, 10, 20, 30, 40 and 50.
  3. The consumer is invoked after the delay seconds as the messages are visible.

I find this architecture is simpler than other options.

Lambda Functions

The sender Lambda function sends messages with different delay second values to a queue. An array of those values are generated by generateDelaySeconds(), given an interval value. Note that this function works well if the interval value is less than or equal to 30. If we want to set up a higher interval value, we should update the function together with the EventBridge Event rule. The source can be found in the GitHub repository.

 1// src/sender.js
 2const AWS = require("aws-sdk");
 3
 4const sqs = new AWS.SQS({
 5  apiVersion: "2012-11-05",
 6  region: process.env.AWS_REGION || "us-east-1",
 7});
 8
 9/**
10 * Generate delay seconds by an interval value.
11 *
12 * @example
13 * // returns [ 0, 30 ]
14 * generateDelaySeconds(30)
15 * // returns [ 0, 20, 40 ]
16 * generateDelaySeconds(20)
17 * // returns [ 0, 15, 30, 45 ]
18 * generateDelaySeconds(15)
19 * // returns [ 0, 10, 20, 30, 40, 50 ]
20 * generateDelaySeconds(10)
21 */
22const generateDelaySeconds = (interval) => {
23  const numElem = Math.round(60 / interval);
24  const array = Array.apply(0, Array(numElem + 1)).map((_, index) => {
25    return index;
26  });
27  const min = Math.min(...array);
28  const max = Math.max(...array);
29  return array
30    .map((a) => Math.round(((a - min) / (max - min)) * 60))
31    .filter((a) => a < 60);
32};
33
34const handler = async () => {
35  const interval = process.env.SCHEDULE_INTERVAL || 30;
36  const delaySeconds = generateDelaySeconds(interval);
37  for (const ds of delaySeconds) {
38    const params = {
39      MessageBody: JSON.stringify({ delaySecond: ds }),
40      QueueUrl: process.env.QUEUE_URL,
41      DelaySeconds: ds,
42    };
43    await sqs.sendMessage(params).promise();
44  }
45  console.log(`send messages, delay seconds - ${delaySeconds.join(", ")}`);
46};
47
48module.exports = { handler };

The consumer Lambda function simply polls the messages. It is set to finish after 1 second followed by logging the delay second value.

 1// src/consumer.js
 2const sleep = (ms) => {
 3  return new Promise((resolve) => {
 4    setTimeout(resolve, ms);
 5  });
 6};
 7
 8const handler = async (event) => {
 9  for (const rec of event.Records) {
10    const body = JSON.parse(rec.body);
11    console.log(`delay second - ${body.delaySecond}`);
12    await sleep(1000);
13  }
14};
15
16module.exports = { handler };

Serverless Service

Two Lambda functions (sender and consumer) and a queue are created by Serverless Framework. As discussed earlier the sender function has an EventBridge Event rule trigger, and it invokes the function at the rate of 1 minute. The schedule interval is set to 10, which is used to create delay seconds values. The consumer is set to be triggered by the queue.

 1# serverless.yml
 2service: ${self:custom.serviceName}
 3
 4plugins:
 5  - serverless-iam-roles-per-function
 6
 7custom:
 8  serviceName: lambda-schedule
 9  scheduleInterval: 10
10  queue:
11    name: ${self:custom.serviceName}-queue-${self:provider.stage}
12    url: !Ref Queue
13    arn: !GetAtt Queue.Arn
14
15…
16
17provider:
18  name: aws
19  runtime: nodejs12.x
20  stage: ${opt:stage, 'dev'}
21  region: ${opt:region, 'us-east-1'}
22  lambdaHashingVersion: 20201221
23  memorySize: 128
24  logRetentionInDays: 7
25  deploymentBucket:
26    tags:
27      OWNER: ${env:owner}
28  stackTags:
29    OWNER: ${env:owner}
30
31functions:
32  sender:
33    handler: src/sender.handler
34    name: ${self:custom.serviceName}-sender-${self:provider.stage}
35    events:
36      - eventBridge:
37          schedule: rate(1 minute)
38          enabled: true
39    environment:
40      SCHEDULE_INTERVAL: ${self:custom.scheduleInterval}
41      QUEUE_URL: ${self:custom.queue.url}
42    iamRoleStatements:
43      - Effect: Allow
44        Action:
45          - sqs:SendMessage
46        Resource:
47          - ${self:custom.queue.arn}
48  consumer:
49    handler: src/consumer.handler
50    name: ${self:custom.serviceName}-consumer-${self:provider.stage}
51    events:
52      - sqs:
53          arn: ${self:custom.queue.arn}
54
55resources:
56  Resources:
57    Queue:
58      Type: AWS::SQS::Queue
59      Properties:
60        QueueName: ${self:custom.queue.name}

Performance

We can filter the log of the consumer function in the CloudWatch page. The function is invoked as expected, but I see the interval gets shortened periodically especially when the delay second value is 0. We’ll have a closer look at that below.

I created a chart that shows delay (milliseconds) by invocation. It shows periodic downward spikes, and they correspond to the invocations where the delay seconds value is 0. For some early invocations, the delay values are more than 1000 milliseconds, which means that the consumer function’s intervals are less than 9 seconds. The delays get stable at or after the 200th invocation. The table in the right-hand side shows the summary statistics of delays after that invocation. It shows the consumer invocation delays spread in a range of 300 milliseconds in general.

Caveats

An EventBridge Events rule can be triggered more than once and a message in an Amazon SQS queue can be delivered more than once as well. Therefore, it is important to design the consumer Lambda function to be idempotent.

Conclusion

In this post, I demonstrated a serverless solution for scheduling a Lambda function at a sub-minute frequency with Amazon SQS. The architecture of the serverless solution is simpler than other options and its performance is acceptable in spite of some negative delays. Due to the at-least-once delivery feature of EventBridge Events and Amazon SQS, it is important to design the application to be idempotent. I hope this post is useful to build a Lambda scheduling solution.