Amazon Web Services Simple Queue Service Using the Java 2 Software Development Kit

Introduction

Message Queues are a way to exchange messages between applications. Senders send data objects to a queue and receivers, receive objects from a queue. Amazon’s Simple Queue Service (SQS) is a service offered by AWS that offers scalability and reliability by being distributed across Amazon.

A message queue decouples applications. A message producer only knows about the queue and knows nothing about the queue’s consumers. Likewise, a message consumer only knows about the queue and knows nothing about the queue’s other consumers or producers. Moreover, producers and consumers know nothing about timing, and are asynchronous.

For more on queues and message-passing in general, there are many resources online. Here is a good reference from MIT: Reading 22: Queues and Message-Passing.

Use Case

Suspend disbelief, or more accurately, simply build the system regardless of what you think about the soundness behind the business plan. Famous entrepreneur John Bunyan from Washington State has a plan to get rich and famous by finally proving conclusively that Bigfoot – or Sasquatch for the cultured – exists and uses the extensive system of hiking trails to move around.

Against his accountant’s advice, he liquidated half his fortune to install a series of hidden cameras along Washington State’s hiking trails to take photos every fifteen minutes. As he is a busy man, he does not have time to analyze all the photos personally, and so he wants image analysis software to analyze the images. If the software registers a Sasquatch, he wants the images to personally go to his email account so he can register the image as a Squatch or not.

Now, if 10,000 cameras take a picture every 15 minutes, that is 600,000 images per hour. Assume each image takes up to five minutes to process. Hopefully you can see, we have a scalability issue.

There are various ways to deal with this scalability issue, but as this is a tutorial on SQS, we use AWS SQS. And, as I am fond of admonishing in all my tutorials, if the “business case” seems suspect, then suspend disbelief and focus on the AWS code.

Design

Enough apologizing for the business case, let’s focus on the application’s design. The following diagram illustrates the dilemma.

• Every n minutes, a Station sends an observation to an AWS queue.
• There are 1 or more SquatchFinder components whose job is to pick up an observation from the queue and process the observation.
• Station is the producer while SasquatchFinder is the consumer.

We can formalize our requirements with a simple class diagram. A Station creates an Observation. A SasquatchFinder processes an Observation.

All communication with AWS from external processes is via its REST API. SQS is no different. Moreover, SQS queues only accept textual data. But a common need is for the queue to accept binary data, such as an image. Also, JSON is a textual data transport format.

We can translate the Observation into a JSON document. The image is converted to base64 encoding so it can be represented as text. Note the encodedImage in this tutorial is always truncated with <snip>, as the base64 string is quite long.

{ 
  timestamp: “1558493503”,
  latitude:”46.879967”,
  longitude:”-121.726906”,
  encodedImage:"/9j/4AA <snip> 3QEUBGX/9k="
}

Base64 Encoding

Images are binary. However, all binary can be represented by a String provided it is encoded and decoded correctly. Base64 is an encoding scheme that is converts binary to a string. It’s useful because it allows embedding binary data, such as an image, in a textual file, such as a webpage or JSON document. AWS queues only allow textual data, and so if you wish to store an image on an SQS queue, you must convert it to a string. And the easiest way to accomplish this is by using Base64 format to encode binary data to strings when transporting data and decode strings to binary data when storing the data. For an example of Base64 and DynamoDB, refer to this site’s tutorial: Using the AWS DynamoDB Low-Level Java API – Sprint Boot Rest Application.

Station – Producer

Before coding the application, let’s create a queue. You can create a queue via the Java 2 API SDK; however, here we create the queue manually and then use this queue to send and receive messages.

Create SQSQueue

• Navigate to the SQS console and select standard Queue.
• Click the Configure Queue button.

  

• Name the queue SasquatchImageQueue.

  

• Accept the defaults for the Queue Attributes.

  

• After creating the queue, you should see a screen similar to the following:

  

• Click on the Permissions tab and notice that we have not created a permission. We return to the Permissions tab after creating the two necessary users.

  

There are two types of queues offered by AWS, Standard Queues and First In First Out (FIFO) Queues. Standard queues provide what is called best-effort ordering. Although messages are usually delivered in the order they are received, there are no guarantees. Moreover, messages can also be processed more than once. FIFO queues, in contrast, guarantee first in first out delivery and processing only once.

In this tutorial, we primarily use standard queues. However, towards the end of this tutorial, we illustrate using a FIFO queue.

Create SQSQueue Users

We need to create two users, one to interact with the queue for sending messages and another for receiving messages. If you have created IAM users before, note we do not assign the user to any group or assign any policies. Instead, we allow the queue to determine its permissions. Of course, we assign the user programmatic access and download the credentials file.

• Navigate to the IAM console and create a new user called SasquatchProducerUser that has programmatic access.
• Save the user’s credentials locally.

  

• Create a second user called SasquatchConsumerUser that also has programmatic access.
• Save the user’s credentials locally.
• You should have two users created with programmatic access.

  

Queue Permissions

Initially only a queue’s creator, or owner, can read or write to a queue. The creator must grant permissions. We do this using a queue policy. We write the policy using the ASW SQS Console, although you write it manually if you wished.

Consumer Permissions

• Navigate to the SasquatchConsumerUser summary screen and copy the Amazon Resource Name (ARN).

  

  The ARN should appear similar to the following:

  arn:aws:iam::743327341874:user/SasquatchConsumer

  The Amazon Resource Number, or ARN, uniquely identifies an Amazon resource, in this case, the SasquatchConsumer user.

• Return to the SQS console and select the SasquatchImageQueue and click on the Permissions tab.

  

• Click Add a Permission.
• In the resultant popup, paste the ARN in the Principal text box.
• Check the DeleteMessage, GetQueueUrl, and ReceiveMessage Action