MQTT (MQ Telemetry Transport) is a lightweight open messaging protocol that provides resource-constrained network clients with a simple way to distribute telemetry information in low-bandwidth environments. The protocol, which employs a publish/subscribe communication pattern, is used for machine-to-machine (M2M) communication.
Created as a low-overhead protocol to accommodate bandwidth and CPU limitations, MQTT was designed to run in an embedded environment where it could provide a reliable, effective path for communication. Suitable for connecting devices with a small code footprint, MQTT is a good choice for wireless networks that experience varying levels of latency due to occasional bandwidth constraints or unreliable connections. The protocol has applications in industries ranging from automotive to energy to telecommunications.
Although MQTT started as a proprietary protocol used to communicate with supervisory control and data acquisition (SCADA) systems in the oil and gas industry, it has become popular in the smart device arena and today is the leading open source protocol for connecting internet of things (IoT) and industrial IoT (IIoT) devices.
While the TT in MQTT stands for Telemetry Transport, the MQ is in reference to a product called IBM MQ. Although the spell-out for MQTT is sometimes given as Message Queuing Telemetry Transport, there is no message queuing in MQTT communication.
How does MQTT work?
Aimed at maximizing the available bandwidth, MQTT's publish/subscribe (pub/sub) communication model is an alternative to traditional client-server architecture that communicates directly with an endpoint. By contrast, in the pub/sub model, the client that sends a message (the publisher) is decoupled from the client or clients that receive the messages (or the subscribers). Because neither the publishers nor the subscribers contact each other directly, third parties -- the brokers -- take care of the connections between them.
MQTT clients include publishers and subscribers, terms that refer to whether the client is publishing messages or subscribed to receive messages. These two functions can be implemented in the same MQTT client. When a device (or client) wants to send data to a server (or broker) it is called a publish. When the operation is reversed, it is called a subscribe. Under the pub/sub model, multiple clients can connect to a broker and subscribe to topics in which they are interested.
If the connection from a subscribing client to a broker is broken, then the broker will buffer messages and push them out to the subscriber when it is back online. If the connection from the publishing client to the broker is disconnected without notice, then the broker can close the connection and send subscribers a cached message with instructions from the publisher.
An IBM writeup describes the pub/sub model: "Publishers send the messages, subscribers receive the messages they are interested in, and brokers pass the messages from the publishers to the subscribers. Publishers and subscribers are MQTT clients, which only communicate with an MQTT broker. MQTT clients can be any device or application (from microcontrollers like the Arduino to a full application server hosted in the Cloud) that runs an MQTT library."
What is an MQTT broker?
An MQTT broker acts as a go-between for the clients who are sending messages and the subscribers who are receiving those messages. In a post office analogy, the broker is the post office itself. All messages have to go through the broker before they can be delivered to the subscriber.
Brokers may have to handle millions of concurrently connected MQTT clients, so when choosing an MQTT broker, enterprises should rate them based on their scalability, integration, monitoring and failure-resistance capabilities.
Types of MQTT messages
An MQTT session is divided into four stages: connection, authentication, communication and termination. A client starts by creating a Transmission Control Protocol/Internet Protocol (TCP/IP) connection to the broker by using either a standard port or a custom port defined by the broker's operators. When creating the connection, it is important to recognize that the server might continue an old session if it is provided with a reused client identity.
The standard ports are 1883 for nonencrypted communication and 8883 for encrypted communication -- using Secure Sockets Layer (SSL)/Transport Layer Security (TLS). During the SSL/TLS handshake, the client validates the server certificate and authenticates the server. The client may also provide a client certificate to the broker during the handshake. The broker can use this to authenticate the client. While not specifically part of the MQTT specification, it has become customary for brokers to support client authentication with SSL/TLS client-side certificates.
Because the MQTT protocol aims to be a protocol for resource-constrained and IoT devices, SSL/TLS might not always be an option and, in some cases, might not be desired. On such occasions, authentication is presented as a cleartext username and password, which are sent by the client to the server -- this, as part of the CONNECT/CONNACK packet sequence. In addition, some brokers, especially open brokers published on the internet, will accept anonymous clients. In such cases, the username and password are simply left blank.
MQTT is considered a lightweight protocol because all its messages have a small code footprint. Each message consists of a fixed header -- 2 bytes -- an optional variable header, a message payload that is limited to 256 megabytes (MB) of information and a quality of service (QoS) level.
During the communication phase, a client can perform publish, subscribe, unsubscribe and ping operations. The publish operation sends a binary block of data -- the content -- to a topic that is defined by the publisher.
MQTT supports message binary large objects (BLOBs) up to 256 MB in size. The format of the content will be application-specific. Topic subscriptions are made using a SUBSCRIBE/SUBACK packet pair, and unsubscribing is similarly performed using an UNSUBSCRIBE/UNSUBACK packet pair.
Topic strings form a natural topic tree with the use of a special delimiter character, the forward slash (/). A client can subscribe to -- and unsubscribe from -- entire branches in the topic tree with the use of special wildcard characters. There are two wildcard characters: a single-level wildcard character, the plus character (+); and a multilevel wildcard character, the hash character (#). A special topic character, the dollar character ($), excludes a topic from any root wildcard subscriptions. Typically, $ is used to transport server-specific or system messages.
Another operation a client can perform during the communication phase is to ping the broker server using a PINGREQ/PINGRESP packet sequence. This packet sequence roughly translates to ARE YOU ALIVE/YES I AM ALIVE. This operation has no other function than to maintain a live connection and ensure the TCP connection has not been shut down by a gateway or router.
When a publisher or subscriber wants to terminate an MQTT session, it sends a DISCONNECT message to the broker and then closes the connection. This is called a graceful shutdown because it gives the client the ability to easily reconnect by providing its client identity and resuming where it left off.
Should the disconnect happen suddenly without time for a publisher to send a DISCONNECT message, the broker may send subscribers a message from the publisher that the broker has previously cached. The message, which is called a last will and testament, provides subscribers with instructions for what to do if the publisher dies unexpectedly.
What are the benefits of using MQTT?
The lightweight properties and minimum overhead of the MQTT protocol architecture help ensure smooth data transfer with low bandwidth and reduce the load on the CPU and RAM. Among MQTT's advantages over competing protocols are the following:
- efficient data transmission and quick to implement, due to its being a lightweight protocol;
- low network usage, due to minimized data packets;
- efficient distribution of data;
- successful implementation of remote sensing and control;
- fast, efficient message delivery;
- uses small amounts of power, which is good for the connected devices; and
- optimizes network bandwidth.
What are the drawbacks of MQTT?
Potential downsides to MQTT include the following:
- MQTT has slower transmit cycles compared to Constrained Application Protocol (CoAP).
- MQTT's resource discovery works on flexible topic subscription, whereas CoAP uses a stable resource discovery system.
- MQTT is unencrypted. Instead, it uses TLS/SSL (Transport Layer Security/Secure Sockets Layer) for security encryption.
- It is difficult to create a globally scalable MQTT network.
- Other MQTT challenges relate to security, interoperability and authentication.
Because the MQTT protocol was not designed with security in mind, the protocol has traditionally been used in secure back-end networks for application-specific purposes. MQTT's topic structure can easily form a huge tree, and there is no clear way to divide a tree into smaller logical domains that can be federated. This makes it difficult to create a globally scalable MQTT network because, as the size of the topic tree grows, the complexity increases.
Another negative aspect of MQTT is its lack of interoperability. Because message payloads are binary, with no information as to how they are encoded, problems can arise -- especially in open architectures where different applications from different manufacturers are supposed to work seamlessly with each other.
As touched upon previously, MQTT has minimal authentication features built into the protocol. Usernames and passwords are sent in cleartext, and any form of secure use of MQTT must employ SSL/TLS, which, unfortunately, is not a lightweight protocol.
Authenticating clients with client-side certificates is not a simple process, and there is no way in MQTT to control who owns a topic and who can publish information on it, except using proprietary, out-of-band means. This makes it easy to inject harmful messages into the network, either willfully or by mistake.
Furthermore, there is no way for the message receiver to know who sent the original message unless that information is contained in the actual message. Security features that must be implemented on top of MQTT in a proprietary fashion increase the code footprint and make implementations more difficult.
MQTT protocol applications and use cases
Due to its lightweight properties MQTT works well for applications involving remote monitoring, including the following:
- synchronization of sensors, such as fire detectors or motion sensors for theft detection, to determine if a hazard is valid;
- monitoring health parameters using sensors for patients leaving a hospital; and
- sensors alerting people of danger.
Another application is a text-based messaging application for real-time communication that capitalizes on MQTT's low data and energy usage. For example, Facebook uses MQTT for its Messenger app, not only because the protocol conserves battery power during mobile phone-to-phone messaging, but also because the protocol enables messages to be delivered efficiently in milliseconds, despite inconsistent internet connections across the globe.
Most major cloud services providers, including Amazon Web Services (AWS), Google Cloud, IBM Cloud and Microsoft Azure, support MQTT.
MQTT is well suited to applications using M2M and IoT devices for purposes such as real-time analytics, preventative maintenance and monitoring in environments, including smart homes, healthcare, logistics, industry and manufacturing.
How is MQTT used in IoT?
Because MQTT clients are small, they require minimal resources and thus can be used on small microcontrollers, according to MQTT.org. To optimize network bandwidth, MQTT headers are small. Plus, MQTT "can scale to connect with millions of IoT devices," according to the organization.
As a result, MQTT is one of the most commonly used protocols in IoT and IIoT infrastructure -- for example, for utilities industries to efficiently transmit data between their services and their customers and devices.
Examples of MQTT use in IoT or IIoT instructure include the following:
- Smart metering. The MQTT protocol can be used to transmit data with guaranteed message delivery to provide accurate meter readings in real time. This helps make billing more accurate.
- Gathering ambient sensor data. Sensors used in remote environments are often low-power devices, so MQTT is a good fit for IoT sensor buildouts with lower-priority data transmission needs.
- Machine health data. Ably, which provides a pub/sub messaging platform, gives the example of a wind turbine requiring "guaranteed delivery of machine health data to local teams even before that information hits a data center."
- Billing systems. MQTT helps eliminate duplicate or lost message packets in billing or invoicing.
Whether used for smart metering, fast outage response or other IoT and IIoT applications, MQTT enables resource-constrained IoT devices to send, or publish, information about a specific topic to a server that acts as an MQTT message broker. The broker then pushes the information out to those clients that have subscribed previously to the topic.
To a human, a topic looks like a hierarchical file path. Clients can subscribe to a specific level of a topic's hierarchy or use a wildcard character to subscribe to multiple levels. The client could be an IoT sensor in the field or an application in a data center that processes IoT data.
As examples, the Carriots, Evrythng and ThingWorx IoT platforms support the MQTT protocol.
Competing protocols
Other transfer protocols that compete with MQTT include the following:
- Constrained Application Protocol (CoAP). Well suited for IoT, it uses a request/response communication pattern.
- Advanced Message Queuing Protocol (AMQP). Like MQTT, it uses a publish/subscribe communication pattern.
- Simple/Streaming Text Oriented Messaging Protocol (STOMP). It is a text-based protocol. However, STOMP does not deal with queues and topics; it uses a send semantic with a destination string.
- Mosquitto. It is an open source MQTT broker.
- Simple Media Control Protocol (SMCP). A CoAP stack that is used in embedded environments, is C-based.
- SSI (Simple Sensor Interface). This is a communications protocol for data transfer between a combination of computers and sensors.
- Data Distribution Service (DDS). For real-time systems, it is a middleware standard that can directly publish or subscribe communications in real time in embedded systems.
Quality of service levels
QoS refers to an agreement between the sender of a message and the message's recipient. It acts as a key feature in MQTT, giving the client the ability to choose between three levels of service.
The three different QoS levels determine how the content is managed by the MQTT protocol. Although higher levels of QoS are more reliable, they have more latency and bandwidth requirements, so subscribing clients can specify the highest QoS level they would like to receive.
The simplest QoS level is unacknowledged service. This QoS level uses a PUBLISH packet sequence; the publisher sends a message to the broker one time, and the broker passes the message to subscribers one time. There is no mechanism in place to make sure the message has been received correctly, and the broker does not save the message. This QoS level may also be referred to as at most once, QoS0 or fire and forget.
The second QoS level is acknowledged service. This QoS level uses a PUBLISH/PUBACK packet sequence between the publisher and its broker, as well as between the broker and subscribers. An acknowledgment packet verifies that content has been received, and a retry mechanism will send the original content again if an acknowledgment is not received in a timely manner. This may result in the subscriber receiving multiple copies of the same message. This QoS level may also be referred to as at least once or QoS1.
The third QoS level is assured service. This QoS level delivers the message with two pairs of packets. The first pair is called PUBLISH/PUBREC, and the second pair is called PUBREL/PUBCOMP. The two pairs ensure that, regardless of the number of retries, the message will only be delivered once. This QoS level may also be referred to as exactly once or QoS2.
MQTT protocol versions and history
MQTT was created by Dr. Andy Stanford-Clark of IBM and Arlen Nipper of Arcom -- now Eurotech -- in 1999. MQTT was created as a cost-effective and reliable way to connect monitoring devices used in the oil and gas industries to remote enterprise servers. When challenged with finding a way to push data from pipeline sensors in the desert to off-site SCADA systems, they decided upon a TCP/IP-based pub/sub topology that would be event-driven to keep satellite link transmission costs down.
Although MQTT is still closely associated with IBM, it is now an open protocol that is overseen by the Organization for the Advancement of Structured Information Standards (OASIS).
Though the name suggests it, MQTT is not part of the original IBM MQSeries; however, as of version 7.1, it is available in WebSphere MQ. MQTT was previously known as the SCADA protocol, MQ Integrator SCADA Device Protocol (MQIsdp) and WebSphere MQTT (WMQTT), although all these variations have fallen out of use.
MQTT has different specifications depending on the specific version. Version 5.0 superseded the last version of MQTT, version 3.1.1. Some newer specifications, as defined by OASIS, include the following:
- the use of pub/sub message patterns;
- a mechanism that can notify users when abnormal disconnections occur;
- the three levels of message delivery: at most once, at least once and exactly once;
- the minimization of transport overhead and protocol exchanges to reduce network traffic; and
- an agnostic messaging transport referring to the content of the payload.
Further specifications can be found at OASIS' website here.
Updates of MQTT
MQTT was officially approved as an OASIS standard on Oct. 28, 2015. At the end of January 2016, it was accepted as an International Organization for Standardization (ISO) standard. The protocol is continuously improving and now supports WebSocket, another protocol that enables two-way communication between clients and brokers in real time. Later, notable versions included the v3.1.1 standard and the v5.0 standard, both having been approved as OASIS standards. As an example of some of its updates, version 5.0 included better error reporting, including metadata in message headers, shared subscriptions, message and session expiries, and topic aliasing.
