updated docs with stage info and setup scripts

Author: Michael Bahr

README.md

@@ -10,28 +10,35 @@ You can deploy this yourself or use our SAAS offer.
 
 ## SAAS Offer
 
-While we're ironing things out you can use our service free of charge. Just publish an event to `arn:aws:sns:us-east-1:256608350746:scheduler-input-dev`.  This topic is public so anyone can publish to it. 
+While we're ironing things out you can use our service free of charge. Just publish an event to `arn:aws:sns:us-east-1:256608350746:scheduler-input-prod`.  This topic is public so anyone can publish to it. 
 
 If you become a heavy user with more than 100.000 events per month we might want to get in touch with you, so make sure to fill out the `user` field with some way to contact you (email, twitter handle, ...).
 
 ## Usage
 
 ### Setup
-First of all you need an output topic that we can publish events to once the scheduled datetime arrives. To do this go to the AWS Console, then open the service SNS and create a topic that includes the following access policy.
+First of all you need an output topic that we can publish events to once the scheduled datetime arrives. To do this run `python setup/init_output_topic.py <stage>`. This will create a topic called `scheduler-output-<stage>` and grant our account the right to publish messages. You can see the added policy below.
 
 ```json
-    {
-      "Sid": "HereComesYourSidName",
-      "Effect": "Allow",
-      "Principal": {
-        "AWS": "arn:aws:sts::256608350746:assumed-role/aws-scheduler-dev-us-east-1-lambdaRole/aws-scheduler-dev-emitter"
-      },
-      "Action": "SNS:Publish",
-      "Resource": "arn:aws:sns:us-east-1:256608350746:scheduler-output-dev"
-    }
+{
+  "Sid": "scheduler-output-<stage>-publish-access",
+  "Effect": "Allow",
+  "Principal": {
+    "AWS": "arn:aws:iam::256608350746:root"
+  },
+  "Action": "SNS:Publish",
+  "Resource": "arn:aws:sns:us-east-1:256608350746:scheduler-output-<stage>"
+}
 ``` 
 
-This policy allows our emitter function to publish events to your topic which you can then process with e.g. AWS Lambda. 
+If you don't want to grant our account the right to publish, you can pass your own accountId as a third argument.
+
+```
+Creating topic scheduler-output-<stage>
+Created topic scheduler-output-<stage> with arn arn:aws:sns:us-east-1:<your-account-id>:scheduler-output-<stage>
+Granting publish rights to scheduler-output-<stage> for accountId 256608350746
+Done
+```
 
 Write down the ARN of your output topic as you will need it for the input events.
 
@@ -47,7 +54,27 @@ To schedule a trigger you have to publish an event which follows the structure b
 }
 ```
 
-All fields are mandatory. If you submit an event that does not follow the spec, it will be dropped. Future versions will improve on this.
+First string encode the json structure and then publish it to the input topic.
+
+```python
+# Python example
+import json
+import boto3
+
+client = boto3.client('sns')
+
+event = {
+          "date": "2019-07-27T12:20:24.919071",
+          "target": "arn:aws:sns:us-east-1:256608350746:scheduler-output-prod",
+          "user": "Twitter @michabahr",
+          "payload": "46607451-3e67-49bc-972b-425c150c5456"
+        }
+
+input_topic = 'arn:aws:sns:us-east-1:256608350746:scheduler-input-prod'
+client.publish(TopicArn=input_topic, Message=json.dumps(event))
+```
+
+All fields are mandatory. Please make sure that the `payload` can be utf-8 encoded. If you submit an event that does not follow the spec, it will be dropped. Future versions will improve on this.
 
 So far there is no batch publishing available for SNS. Make sure the event stays within the 256KB limit of SNS. We recommend that you only submit IDs and don't transfer any real data to the service.
 
@@ -69,9 +96,11 @@ You must have the following tools installed:
 - python3
 - pip
 
-Run `setup/init_table.py` to setup the database.
+Run `setup/init_table.py <stage>` to setup the database. Replace `<stage>` with the stage of your application.
+
+Run `setup/init_input_topic.py <stage> [public]`. Replace `<stage>` with the stage of your application. You may append the parameter `public` to grant public publish rights.
 
-Create an SNS topic named `scheduler-input-{stage}` and a SQS queue named `scheduler-queue-{stage}`. Replace `{stage}` with the stage that you use for the serverless deployment, e.g. `dev`. Adjust the access policies as necessary.
+Run `setup/init_queue.py <stage>`. Replace `<stage>` with the stage of your application.
 
 ### Deploy
 1. Navigate into the project folder
@@ -91,8 +120,8 @@ Our results showed that most of the events arrive within one second of the speci
 
 The charts show the amount of events received on the y axis and the distribution by delay on the x axis.
 
-![Regular Scaled 100000 events wihtin 10 minutes](https://github.com/bahrmichael/aws-scheduler/raw/master/pictures/regular-scaled-100k-10m.png)
-![Log Scaled 100000 events wihtin 10 minutes](https://github.com/bahrmichael/aws-scheduler/raw/master/pictures/log-scaled-100k-10m.png)
+![Regular Scaled 100000 events within 10 minutes](https://github.com/bahrmichael/aws-scheduler/raw/master/pictures/regular-scaled-100k-10m.png)
+![Log Scaled 100000 events within 10 minutes](https://github.com/bahrmichael/aws-scheduler/raw/master/pictures/log-scaled-100k-10m.png)
 
 ## Limitations
 Events may arrive more than once at the output topic.
@@ -101,11 +130,12 @@ Events may arrive more than once at the output topic.
 Contributions are welcome, both issues and code. Get in touch at twitter [@michabahr](https://twitter.com/michabahr) or create an issue.
 
 ## TODOs
-- test user check
 - test with huge messages (first make as big as possible until fails, then execute them with more than 10 minutes target so they get bulked)
-- prod deployment and update arns/policies
+- attach to own project
+- helper script to create an output topic with proper access policy (should take name and region as parameters)
 - use a proper logger
 - secure the PoC with test
 - include a failure queue and adjust the docs
 - add a (video) guide on how to create a proper output queue
 - add a safe guard that pulls messages from dead letter queues back into the circuit
+- handling for messages that can't be utf-8 encoded

serverless.yml

@@ -3,8 +3,8 @@ service: aws-scheduler
 provider:
   name: aws
   runtime: python3.7
-  stage: dev
-  region: us-east-1
+  stage: ${opt:stage, 'dev'}
+  region: stage: ${opt:region, 'us-east-1'}
   iamRoleStatements:
     - Effect: Allow
       Action:
@@ -21,10 +21,6 @@ provider:
       Action:
         - SQS:SendMessage
       Resource: "${self:custom.queue.arn}"
-    - Effect: Allow
-      Action:
-        - dynamodb:*
-      Resource: { "Fn::Join": [":", ["arn:aws:dynamodb:${self:provider.region}", { "Ref": "AWS::AccountId" }, "table/aws-scheduler-measurement" ] ]  }
     - Effect: Allow
       Action:
         - lambda:InvokeFunction
@@ -39,11 +35,11 @@ custom:
     arn: { "Fn::Join": [":", ["arn:aws:sqs:${self:provider.region}", { "Ref": "AWS::AccountId" }, "${self:custom.queue.name}" ] ]  }
     url: { "Fn::Join": ["/", ["https://sqs.${self:provider.region}.amazonaws.com", { "Ref": "AWS::AccountId" }, "${self:custom.queue.name}" ] ]  }
   table:
-    arn: { "Fn::Join": [":", ["arn:aws:dynamodb:${self:provider.region}", { "Ref": "AWS::AccountId" }, "table/aws-scheduler-events" ] ]  }
+    arn: { "Fn::Join": [":", ["arn:aws:dynamodb:${self:provider.region}", { "Ref": "AWS::AccountId" }, "table/aws-scheduler-events-${self:provider.stage}" ] ]  }
     index:
       name: "status-date-index"
-      arn: { "Fn::Join": [":", ["arn:aws:dynamodb:${self:provider.region}", { "Ref": "AWS::AccountId" }, "table/aws-scheduler-events/index/${self:custom.table.index.name}" ] ]  }
-  scheduleFunction: "aws-scheduler-${self:provider.stage}-scheduleBatch"
+      arn: { "Fn::Join": [":", ["arn:aws:dynamodb:${self:provider.region}", { "Ref": "AWS::AccountId" }, "table/aws-scheduler-events-${self:provider.stage}/index/${self:custom.table.index.name}" ] ]  }
+  scheduleFunction: "aws-scheduler-${self:provider.stage}-scheduler"
   pythonRequirements:
     pythonBin: python3
 
@@ -62,7 +58,7 @@ functions:
           topicName: "${self:custom.inbound.name}"
     environment:
       SCHEDULE_FUNCTION: "${self:custom.scheduleFunction}"
-      ENFORCE_USER: true
+      ENFORCE_USER: false
   eventLoader:
     handler: handler.event_loader
     events: