18CSC449 Internet of Things

workshop

Lecture-3
Data Protocols in IoT

Nalinadevi Kadiresan
Department of CSE
Amrita Vishwa Vidyapeetham
Traditional TCP/IP suite vs IoT Protocol
Stack
 Protocols in IoT
Rather than trying to fit all of the IoT Protocols on top of
existing architecture models like OSI Model, protocols are
broken the following layers to provide some level of
organization:
– Data Protocols (ex: MQTT, CoAP, AMQP, Websocket, Node)
– Discovery (ex: Physical Web, mDNS, DNS-SD)
– Identification (ex: EPC, uCode, IPv6, URIs)
– Comms / Transport (ex: Wifi, Bluetooth, LPWAN)
– Infrastructure (ex: 6LowPAN, IPv4/IPv6, RPL)
– Device Management (ex: TR-069, OMA-DM)
– Semantic (ex: JSON-LD, Web Thing Model)
– Multi-layer Frameworks (ex: Alljoyn, IoTivity, Weave, Homekit)
IoT Protocols and standards
Data Protocols
 Feature enhancement of HTTP
• HTTP/1.1
– persistent , pipelined connection
– chunked transfers
– new header fields in request/response body
– Keep-Alive header to set policies for long-lived
communications between hosts (timeout period and
maximum request count to handle per connection)
• HTTP 2.0
– Upgrade header to switch the connection to an enhanced
protocol mode such as HTTP/2.0 (h2,h2c) or Websockets
(websocket)
 HTTP request message (Recap)

• two types of HTTP messages: request, response

• HTTP request message:
– ASCII (human-readable format)
carriage return character
line-feed character
request line
(GET, POST, GET /index.html HTTP/1.1\r\n
HEAD commands) Host: www-net.cs.umass.edu\r\n
User-Agent: Firefox/3.6.10\r\n
Accept: text/html,application/xhtml+xml\r\n
headerAccept-Language: en-us,en;q=0.5\r\n
linesAccept-Encoding: gzip,deflate\r\n
Accept-Charset: ISO-8859-1,utf-8;q=0.7\r\n
carriage return, Keep-Alive: 115\r\n
line feed at start Connection: keep-alive\r\n
\r\n
of line indicates
end of header lines
Application Layer 2-7
 HTTP request message: general format
(Recap)
method sp URL sp version cr lf request
line
header field name value cr lf
header
~
~ ~
~ lines

header field name value cr lf

cr lf

~
~ entity body ~
~ body

Application Layer 2-8

 Uploading form input (Recap)

POST method:
• web page often includes form input
• input is uploaded to server in entity body

URL method:
• uses GET method
• input is uploaded in URL field of request line:

www.somesite.com/animalsearch?monkeys&banana

Application Layer 2-9

 Method types (Recap)
HTTP/1.0: HTTP/1.1:
• GET • GET, POST, HEAD
• POST • PUT
• HEAD – uploads file in
– asks server to leave entity body to path
requested object specified in URL
out of response field
• DELETE
– deletes file
specified in the URL
Application Layer
field 2-10
 HTTP response message (Recap)
status line
(protocol
status code HTTP/1.1 200 OK\r\n
status phrase) Date: Sun, 26 Sep 2010 20:09:20 GMT\r\n
Server: Apache/2.0.52 (CentOS)\r\n
Last-Modified: Tue, 30 Oct 2007 17:00:02
GMT\r\n
header ETag: "17dc6-a5c-bf716880"\r\n
Accept-Ranges: bytes\r\n
lines Content-Length: 2652\r\n
Keep-Alive: timeout=10, max=100\r\n
Connection: Keep-Alive\r\n
Content-Type: text/html; charset=ISO-8859-1\
r\n
\r\n
data, e.g., data data data data data ...
requested
HTML file
Application Layer 2-11
HTTP response status codes(Recap)
 status code appears in 1st line in server-to-
client response message.
 some sample codes:

200 OK
– request succeeded, requested object later in this msg
301 Moved Permanently
– requested object moved, new location specified later in this msg
(Location:)
400 Bad Request
– request msg not understood by server
404 Not Found
– requested document not found on this server
505 HTTP Version Not Supported
Application Layer 2-12
 Feature enhancement of HTTP
• REST Web Services: Rest stands for Representational State Transfer.
Let's understand the meaning of each word in the REST acronym.
• State means data
• Representational means formats (such as XML, JSON, YAML, HTML, etc)
• Transfer means carry data between consumer and provider using the HTTP
protocol
A REST API is an intermediary Application Programming Interface that enables two
applications to communicate with each other over HTTP, much like how servers
communicate to browsers.
 Feature enhancement of HTTP
• HTTP Short Polling - A lot of requests are processed as they come to
server, creating a lot of traffic (uses resources, but frees them as soon as
response is sent back).

• HTTP Long Polling: One request goes to server and client is waiting for the
response to come. The server holds the request open until new data is
available (it’s unresolved and resources are blocked). You are notified with
no delay when the server event happens.
 Feature enhancement of HTTP
• HTTP Periodic Polling:
– There’s a predefined time gap between two requests.
– This is an improved/managed version of polling.
– You can reduce server consumption by increasing time gap between
two requests.
– But if you need to be notified with no delay when the server event
happens, this is not a good option.
 Feature enhancement of HTTP
• HTTP Streaming : Client makes an HTTP request, and the
server trickles out a response of indefinite length . This is an
alternate to websocket.
– Issue: Intermediaries can interrupt the connection (e.g.
timeout, intermediaries serving other requests in round-
robin manner). In such cases, it cannot guarantee the
complete realtimeness.
 Feature enhancement of HTTP
• SSE (Server Sent Events / EventSource)
– server pushes data to browser (browsers can only subscribe to data updates
originated by server, but cannot send any data to the server)
– Sample applications: Twitter updates, stock quotes, cricket scores, notifications
to browser
• HTTP/2 Server Push
– A mechanism for a server to proactively push assets (stylesheets, scripts, media)
to the client cache in advance
– Sample applications: Social media feeds, single page apps
– Issues: no support of browser for infinitely open connection, intermediate
devices like proxies, routers and hub
Feature enhancement of HTTP: WebSocket
• WebSockets allow both the server and the client to push messages
at any time without any relation to a previous request.
• One notable advantage in using websockets is, almost every
browser support websockets.
• WebSocket solves a few issues with HTTP:
– Bi-directional protocol — either client/server can send a message to the
other party (In HTTP, the request is always initiated by client and the
response is processed by server — making HTTP a uni-directional
protocol)
– Full-duplex communication — client and server can talk to each other
independently at the same time.
– Single TCP connection — After upgrading the HTTP connection in the
beginning, client and server communicate over that same TCP
connection throughout the lifecycle of Websocket connection.
Feature enhancement of HTTP: WebSocket
Feature enhancement of HTTP: WebSocket

• C

• f
Feature enhancement of HTTP: WebSocket

HTTP vs WebSocket
Feature enhancement of HTTP: WebSocket
Feature enhancement of HTTP: WebSocket
 Feature enhancement of HTTP:
WebSocket
The WebSocket protocol was designed to work
well with the legacy web infrastructure.
–It uses an HTTP-compatible handshake to establish a
connection between the client and server.
–The process to start a Web Socket connection begins
with the client sending an HTTP GET with the Web
Socket "Upgrade" field.
– After a connection is initialized communication
switches to a bidirectional binary protocol unlike the
HTTP protocol.
 Feature enhancement of HTTP: WebSocket

The handshake from the client looks as follows:

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

The handshake from the server looks as follows:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: chat
Feature enhancement of HTTP: WebSocket

Sending frames in WebSocket

• Data is transmitted using a sequence of frames
• Data frame may be transmitted by either the client or server
between handshake and endpoint close request.
• A WebSocket header is only 2 to 6 bytes long.

• A client MUST mask all frames that it sends to the server.

– To avoid confusing network intermediaries (such as intercepting
proxies)
• For security reasons
Feature enhancement of HTTP: WebSocket

•HTTP does not define any limit. However most web servers do
limit size of headers they accept. For example in Apache default
limit is 8KB, in IIS it's 16K. Server will return 413 Entity Too
Large error if headers size exceeds that limit.

•As a comparison, Google’s SPDY research reports HTTP

request headers can vary in size from ~200 bytes to over 2KB.
Think of all the network traffic and communication latency you
can reduce by switching from an HTTP-based implementation to
a WebSocket implementation.
Feature enhancement of HTTP: WebSocket
• If an encrypted WebSocket connection is used, then the use of Transport
Layer Security (TLS) in the WebSocket Secure connection ensures that an
HTTP CONNECT command is issued when the browser is configured to use
an explicit proxy server.
• This sets up a tunnel, which provides low-level end-to-end TCP
communication through the HTTP proxy, between the WebSocket Secure
client and the WebSocket server.
Feature enhancement of HTTP: WebSocket
Feature enhancement of HTTP: WebSocket
Feature enhancement of HTTP: WebSocket

Need for WebSockets

 Feature enhancement of HTTP: MQTT
MQTT vs HTTP – Which one is Best ???
1. MQTT is data centric whereas HTTP is document-centric.
– HTTP is request-response protocol for client-server
computing
• not always optimized for mobile devices.
– MQTT is
• lightweight (MQTT transfers data as a byte array) and
• publish/subscribe model
– perfect for resource constrained devices and help to save battery.
– provides clients with independent existence from one another and
enhance the reliability of the whole system.
– When one client is out of order the whole system can keep on
working properly.
 Feature enhancement of HTTP: MQTT
MQTT vs HTTP – Which one is Best ???
2. Speed and Delivery
– According to measurements in 3G networks, throughput of
MQTT is 93 times faster than HTTP’s.
– MQTT Protocol ensures high delivery guarantees.
– There are 3 levels of Quality of Services:
at most once: guarantees a best effort delivery.
at least once: guaranteed that a message will be delivered
at least once. But the message can also be delivered more
than once.
exactly once: guarantees that each message is received
only once by the counterpart
 Feature enhancement of HTTP: MQTT
MQTT vs HTTP – Which one is Best ???
– MQTT also provides users with options of Last will &
Testament and Retained messages.
The last will and Testament means that in case of unexpected
disconnection of a client all subscribed clients will get a message from a
broker.
Retained message means that a newly subscribed client will get an
immediate status update.
– HTTP Protocol has none of these abilities.
3. Complexity and Message Size:
– MQTT has pretty short specification. There are only CONNECT,
PUBLISH, SUBSCRIBE, UNSUBSCRIBE and DISCONNECT types
that are significant for developers. Whereas HTTP
specifications are much longer.
 Feature enhancement of HTTP: MQTT
MQTT vs HTTP – Which one is Best ???
– MQTT has a very short message header and the smallest
packet message size of 2 bytes.
– Using text message format by HTTP protocol, allows it to
compose lengthy headers and messages
• MQTT is for IoT
– MQTT Protocol is easy of use. It is essential when response
time, throughput, lower battery and bandwidth usage are on
the first place for future solutions. MQTT is also perfect in
case of intermittent connectivity.
– HTTP is worthy and extendable. But MQTT is more suitable
when it is referred to IoT development.
 Feature enhancement of HTTP: MQTT
MQTT – A Client Server Publish-Subscribe Messaging
Transport Protocol
•Three Components of MQTT
–Publisher is a client sending a particular message.
–Subscriber is a another client (or more clients), who is
receiving the message .
•Publisher and subscriber don’t know about the existence of
one another.
–Broker, which is known by both the publisher and
subscriber, which filters all incoming messages and
distributes them accordingly.
Feature enhancement of HTTP: MQTT
 Feature enhancement of HTTP: MQTT
Decoupling between Publisher and Subscriber

• The main aspect in pub/sub is the decoupling of publisher and

receiver, which can be differentiated in more dimensions:
– Space decoupling: Publisher and subscriber do not need to know each
other (by IP address and Port for example)
– Time decoupling: Publisher and subscriber do not need to run at the
same time.
– Synchronization decoupling: Operations on both components are not
halted during publish or receiving
• In summary publish/subscribe decouples publisher and receiver
of a message, through filtering of the messages it is possible that
only certain clients receive certain messages.
 Feature enhancement of HTTP: MQTT

Scalability
– Pub/Sub also provides a greater scalability than the
traditional client-server approach.
• Broker can
– highly parallelize and process event-driven.
– perform message caching and intelligent routing of messages.
– But it is definitely a challenge to scale publish/subscribe to
millions of connections.
• This can be achieved using clustered broker nodes in order to
distribute the load over more individual servers with load balancers.
 Feature enhancement of HTTP:MQTT

MQTT Message Filtering

• Option 1: Subject-based filtering
– The filtering is based on a subject or topic, which is part of each message.
– Topics are in general strings with an hierarchical structure, that allow filtering
based on a limited number of expression.
• Option 2: Content-based filtering
– Content-based filtering is as the name already implies, when the broker
filters the message based on a specific content filter-language.
– A big downside to this is, that the content of the message must be known
beforehand and can not be encrypted or changed easily.
• Option 3: Type-based filtering
– When using object-oriented languages it is a common practice to filter based
on the type/class of the message (event).
– In this case a subscriber could listen to all messages, which are from type
Exception or any subtype of it.
 Feature enhancement of HTTP: MQTT
Who is a MQTT Client?
• MQTT client can be both a publisher and/or subscribers at the same time.
•MQTT client is any device from a micro controller up to a full fledged
server
– That has a MQTT library running and is connecting to an MQTT broker
over any kind of network.
– Example
• A small and resource constrained device, that is connected over a wireless
network and has a library strapped to the minimum
– MQTT is ideally suitable for small devices.
• A typical computer running a graphical MQTT client for testing purposes,
basically any device that has a TCP/IP stack and speaks MQTT over it.
• MQTT client libraries are available for a huge variety of programming
languages, for example Android, Arduino, C, C++, C#, Go, iOS, Java,
JavaScript, .NET.
 Feature enhancement of HTTP: MQTT

Broker
•The counterpart to a MQTT client is the MQTT broker, which is the
heart of any publish/subscribe protocol.
– A broker can handle up to thousands of concurrently connected MQTT
clients.
• The broker is primarily responsible for receiving all messages, filtering them, decide
who is interested in it and then sending the message to all subscribed clients.
• It also holds the session of all persisted clients including subscriptions and missed
messages .
– A broker provides authentication and authorization of clients.
– A broker is also extensible, which allows to easily integrate custom
authentication, authorization and integration into backend systems.
– All in all the broker is the central hub, which every message needs to pass.
• Highly scalable, integrate into backend systems, easy to monitor and failure-
resistant.
 Feature enhancement of HTTP: MQTT

MQTT Connection
The MQTT protocol is based on top of TCP/IP and
both client and broker need to have a TCP/IP
stack.
 Feature enhancement of HTTP: MQTT
MQTT Connection
• Connection itself is always between one client and the broker
–no client is connected to another client directly.
•The connection is initiated through a client sending a CONNECT
message to the broker.
•The broker response with a CONNACK and a status code.
•Once the connection is established, the broker will keep it open
as long as the client doesn’t send a disconnect command or it
looses the connection.
 Feature enhancement of HTTP: MQTT

Topics in MQTT
• Topics are the ways you register the interest for incoming
messages or how you specify where you want to publish your
message.
– A topic is a hierarchical structured string, which is used for message
filtering and routing.
• Topics are represented with strings separated by slashes. The
slashes indicate the topic level.
• Example on how you would create a topic for a lamp in your home
office:
 Feature enhancement of HTTP: MQTT

Topics in MQTT
• For example if you want to turn on a lamp in your home office
using an ESP and a RPi, you publish a message to a topic using
a client (Node-RED) saying “ON”.
• Your device needs to be subscribed to that same topic, so it
would receive the “ON” message and finally turn on the lamp.
 Feature enhancement of HTTP: MQTT

Topics in MQTT
Note: topics are case-sensitive, which makes
these two topics different:
 Feature enhancement of HTTP: MQTT
Command in MQTT
You can send a command with a client (like Node-RED)
to control an output:

Or you can read data from a sensor and publish it to a

client (like Node-RED):
 Feature enhancement of HTTP: MQTT

MQTT connection through a NAT

•Generally MQTT clients are behind routers
–It requires network address translation (NAT) in order to
translate from a private network address (like 192.168.x.x,
10.0.x.x) to a public facing one.
•We know that MQTT client is doing the first step by
sending a CONNECT message.
–So there is no problem at all with clients behind a NAT,
because the broker has a public address and the connection
will be kept open to allow sending and receiving message
bidirectional after the initial CONNECT.
Feature enhancement of HTTP: MQTT
 Feature enhancement of HTTP: MQTT
ClientId
• The client identifier (short ClientId) is an identifier of
each MQTT client connecting to a MQTT broker.
• It should be unique per broker. The broker uses it for
identifying the client and the current state of the client.
• If you don’t need a state to be hold by the broker, in
MQTT 3.1.1 (current standard) it is also possible to
send an empty ClientId, which results in a connection
without any state. A condition is that clean session is
true, otherwise the connection will be rejected.
 Feature enhancement of HTTP: MQTT

QoS levels
• The Quality of Service (QoS) level is an agreement
between sender and receiver of a message regarding the
guarantees of delivering a message. There are 3 QoS levels
in MQTT:
– At most once (0) (Fire and Forget) (Best Effort Delivery)
– At least once (1) it is guaranteed that a message will be
delivered at least once to the receiver. But the message can
also be delivered more than once.
– Exactly once (2). The highest QoS is 2, it guarantees that each
message is received only once by the counterpart. It is the
safest and also the slowest quality of service level.
 Feature enhancement of HTTP: MQTT
Clean Session
• The clean session flag indicates the broker, whether
the client wants to establish a persistent session or not.
• A persistent session (CleanSession is false) means, that
the broker will store all subscriptions for the client and
also all missed messages, when subscribing with
Quality of Service (QoS) 1 or 2.
• If clean session is set to true, the broker won’t store
anything for the client and will also purge all
information from a previous persistent session.
 Feature enhancement of HTTP: MQTT
Username/Password
• MQTT allows to send a username and password for
authenticating the client and also authorization.
However, the password is sent in plaintext, if it isn’t
encrypted or hashed by implementation or TLS is used
underneath.
• We highly recommend to use username and password
together with a secure transport of it.
• In brokers like HiveMQ it is also possible to authenticate
clients with an SSL certificate, so no username and
password is needed.
 Feature enhancement of HTTP: MQTT
Will Message
• The will message is part of the last will and
testament feature of MQTT. It allows to notify other
clients, when a client disconnects ungracefully.
• A connecting client will provide his will in form of an
MQTT message and topic in the CONNECT message.
• If this clients gets disconnected ungracefully, the
broker sends this message on behalf of the client.
 Feature enhancement of HTTP: MQTT

KeepAlive
• The keep alive is a time interval, the clients
commits to by sending regular PING Request
messages to the broker.
• The broker response with PING Response and
this mechanism will allow both sides to
determine if the other one is still alive and
reachable.
Feature enhancement of HTTP: MQTT
 Feature enhancement of HTTP: MQTT
Broker responds with the CONNACK message
When a broker obtains a CONNECT message, it is obligated to respond with a
CONNACK message. The CONNACK contains only two data entries: session present
flag, connect return code.
• Session Present flag
The session present flag indicate, whether the broker already has a persistent session
of the client from previous interactions. If a client connects and has set CleanSession
to true, this flag is always false, because there is no session available.
If the client has set CleanSession to false, the flag is depending on, if there are session
information available for the ClientId. If stored session information exist, then the flag
is true and otherwise it is false.
• Connect acknowledge flag
The second flag in the CONNACK is the connect acknowledge flag. It signals the client,
if the connection attempt was successful and otherwise what the issue is.
Feature enhancement of HTTP: MQTT
Feature enhancement of HTTP: MQTT
Feature enhancement of HTTP: MQTT
Feature enhancement of HTTP: MQTT
Feature enhancement of HTTP: MQTT
Feature enhancement of HTTP: MQTT
Feature enhancement of HTTP: MQTT
Feature enhancement of HTTP: MQTT
 What is a Constrained Device?
• “A node where some of the characteristics that are
otherwise pretty much taken for granted for
Internet nodes at the time of writing are not
attainable, […] due to cost, size, and energy
constraints”
• Significant constraints on:
– maximum code complexity (ROM/Flash)
– size of state and buffers (RAM)
– available computational power
– connectivity
 What is a Constrained Network?
• “A network where some of the characteristics
pretty much taken for granted with link layers in
common use in the Internet at the time of writing
are not attainable”
• Significant constraints on:
– low achievable throughput
– high packet loss
– highly asymmetric links
– severe penalties for using larger packets
– limits on reachability over time
Classes of Constrained devices

Very constrained sensor-like

Constrained Class-0 Devices
Constrained Class-1 devices
Constrained Class-2 Devices
Energy Constrained Devices
Evolution of Constrained Devices in IoT
 Feature enhancement of HTTP: CoAP

Constrained Application protocol

Feature enhancement of HTTP: CoAP
 Feature enhancement of HTTP: CoAP

URL Resolution
 Feature enhancement of HTTP: CoAP
Request/ response Transaction
 Feature enhancement of HTTP: CoAP
• COAP Architecture
Feature enhancement of HTTP: CoAP
COAP Design Requirements

See draft-shelby-core-coap-req
Feature enhancement of HTTP: CoAP
 Feature enhancement of HTTP: CoAP
• COAP Messaging Basics
 Feature enhancement of HTTP: CoAP
• COAP Messaging
 Feature enhancement of HTTP: CoAP
COAP Request/response
message COAP Response only message
 Feature enhancement of HTTP: CoAP
• COAP Non confirmable request message
 Feature enhancement of HTTP: CoAP
• Message Header (4 bytes)
 Feature enhancement of HTTP: CoAP
• COAP dealing with Packet Loss
 Feature enhancement of HTTP: CoAP
• Proxying/Caching
 Feature enhancement of HTTP: CoAP
• COAP Observation
Feature enhancement of HTTP: CoAP
 Feature enhancement of HTTP: CoAP
• COAP Block Transfer
 Feature enhancement of HTTP: CoAP
• COAP Discovery and semantics
 Feature enhancement of HTTP: CoAP
• COAP Discovery and Semantics