MQTT Introduction

This section will further deepen the study of MQTT.

What is MQTT

MQTT (Message Queuing Telemetry Transport) is a lightweight, low-bandwidth, publish/subscribe-based message transport protocol. It is designed for network environments with limited bandwidth, high latency, and instability, making it particularly suitable for IoT (Internet of Things) scenarios.

Main Features of MQTT

1. Lightweight and Efficient: MQTT clients are small in size and consume minimal resources, making them suitable for running on microcontrollers. Its message header is also very small, optimizing the use of network bandwidth.
2. Bidirectional Communication: MQTT supports bidirectional message delivery from device to cloud and from cloud to device, facilitating broadcasting messages to device groups.
3. Scalability: MQTT can scale to connect millions of IoT devices.
4. Reliable Message Delivery: MQTT defines three levels of Quality of Service (QoS) to meet different reliability requirements:
   • QoS 0: At most once (messages may be lost)
   • QoS 1: At least once (ensures messages are delivered at least once)
   • QoS 2: Exactly once (ensures messages are delivered only once)
5. Support for Unstable Networks: For many IoT devices connected via unstable cellular networks, MQTT supports persistent sessions, reducing the time for clients to reconnect to the broker.
6. Security: MQTT supports using TLS to encrypt messages and supports modern authentication protocols (such as OAuth) for client authentication.

MQTT Publish/Subscribe Architecture:

MQTT uses a publish/subscribe model, where MQTT clients can be publishers or subscribers:

• Publisher: Publishes messages to specific topics.
• Subscriber: Subscribes to topics of interest and receives messages published to those topics.
• MQTT Broker: Acts as a message relay station, responsible for receiving messages from publishers and forwarding them to subscribers.

History of MQTT

The history of MQTT (Message Queuing Telemetry Transport) can be traced back to the 1990s when it was initially invented by IBM. The original design of MQTT was to meet the data transmission needs between oil pipeline sensors and satellites, requiring the protocol to be lightweight, low-bandwidth, and capable of reliably transmitting data in unstable network environments.

Development of MQTT:

1. MQTT v3.1.1: This version of MQTT was officially released in October 2014. It was widely adopted and became an OASIS (Organization for the Advancement of Structured Information Standards) protocol standard. This means that MQTT v3.1.1 has been upgraded to an international IoT standard, paving the way for billions of low-cost IoT devices to connect to the network.
2. MQTT v5.0: This major update was released in May 2018 and became the new OASIS standard in March 2019. Compared to previous versions, v5.0 introduced significant enhancements and improvements but is not backward compatible with older versions. This decision reflects the need for the MQTT protocol to introduce many new features to adapt to the rapid development of the IoT industry.

The detailed development history of MQTT is shown in the figure

OASIS

OASIS (Organization for the Advancement of Structured Information Standards) is an international non-profit organization dedicated to promoting the development of structured information standards. It provides an open forum for various industries and communities, allowing members from around the world to collaborate in developing and promoting impactful technical standards.

Main Goals and Activities of OASIS

1. Standard Development: OASIS develops standards through its Technical Committees (TCs), which consist of experts from different organizations working together to create a series of industry standards.
2. Global Collaboration: As an international organization, OASIS encourages global enterprises and organizations to participate in the standard development process, ensuring the broad applicability and international nature of the standards.
3. Promotion and Application: OASIS not only focuses on standard development but also promotes the application of these standards, helping enterprises and organizations improve their products and services using these standards.
4. Cross-Industry Collaboration: OASIS covers a wide range of fields, including but not limited to security, cloud computing, IoT, emergency management, content technology, service-oriented architecture (SOA), and XML namespaces.
5. Open and Transparent: The work process of OASIS is open and transparent, allowing anyone to participate in discussions, provide opinions, and suggestions, ensuring the fairness and diversity of the standards.

Impact of OASIS Standards

The standards developed by OASIS (official website: https://www.oasis-open.org/) have been widely adopted globally, for example:

• Security: OASIS has successfully promoted standards related to secure communication and data protection, such as Security Token Service (STS).
• Cloud Computing: OASIS has contributed to the cloud computing field, such as Cloud Audit Data Service (CADS).
• Internet of Things (IoT): OASIS has promoted interoperability between IoT devices and platforms through standardization work, such as the MQTT protocol.
• Content Technology: OASIS has developed standards related to content management and distribution, such as DITA (Darwin Information Typing Architecture).
• Service-Oriented Architecture (SOA): OASIS has promoted the development of SOA-related standards, helping enterprises build more flexible and integrated IT systems.

Core Concepts

The MQTT (Message Queuing Telemetry Transport) protocol includes several core concepts that form the basis of MQTT message transmission. Here are some key terms and concepts in MQTT:

1. Topic:
   • A topic is a core concept in the MQTT protocol used to identify message channels.
   • It is a string that can contain multiple levels, separated by slashes (/).
   • Topics can be broadcast, allowing a message to be received by multiple subscribers.
2. Broker:
   • Also known as the MQTT server, it is the central node of the MQTT system.
   • Responsible for maintaining connections between clients and receiving, storing, and forwarding messages.
   • The broker allows publishers to publish messages to specific topics and distributes these messages to subscribers who have subscribed to those topics.
3. Publisher:
   • A publisher is a client that sends messages.
   • It publishes messages to specific topics on the broker for other clients to subscribe to.
4. Subscriber:
   • A subscriber is a client that receives messages.
   • It subscribes to one or more topics on the broker, and when a publisher publishes messages to those topics, the broker forwards the messages to the subscriber.
5. Will:
   • The MQTT protocol allows clients to set a will message.
   • If a client unexpectedly disconnects for some reason, the broker will send the will message to all topics previously subscribed to by the client.
   • The will message can notify other clients that the client has disconnected and may contain the client’s last status.
6. Quality of Service (QoS):
   • MQTT defines three levels of message quality of service:
     • QoS 0: At most once (messages may be lost, no acknowledgment)
     • QoS 1: At least once (messages will be delivered at least once but may be duplicated)
     • QoS 2: Exactly once (ensures messages are delivered only once)
7. Persistence:
   • MQTT supports message persistence, allowing clients to set messages as persistent.
   • Persistent messages are retained by the broker even after the client disconnects until the message is successfully delivered.
8. Session:
   • A session in MQTT refers to the connection state between a client and the broker.
   • Sessions can be clean or persistent, with persistent sessions allowing the restoration of connection and message state after network interruptions.
9. Connection:
   • The network connection between a client and the broker.
   • When establishing a connection, the client can specify various parameters such as client ID, clean session flag, keep-alive time, etc.

Publish/Subscribe

The essence of the MQTT (Message Queuing Telemetry Transport) protocol lies in its publish/subscribe model, which provides an efficient and flexible mechanism for information flow. Here is a detailed explanation of the MQTT publish/subscribe model:

Decoupled Communication Architecture

MQTT achieves decoupling between message senders (publishers) and receivers (subscribers) through the publish/subscribe model. This design allows both to operate independently without knowing each other’s existence or state. Publishers only need to send messages to the broker, while subscribers subscribe to specific topics of interest, with the broker responsible for distributing messages to the appropriate subscribers.

Role of the Broker

In this model, the broker acts as an intermediary agent, serving as the central node of the MQTT system, responsible for handling message routing and distribution tasks. The broker receives messages from publishers and, based on the subscription rules of subscribers, delivers the messages to the correct recipients.

Dynamic System Scalability

The publish/subscribe model of MQTT provides high communication flexibility, allowing the system to dynamically add or remove publishers and subscribers at runtime. This dynamic nature makes the MQTT protocol very suitable for the ever-changing IoT environment, where devices may frequently join or leave the network.

Topic-Based Communication

Message publishing and subscribing are based on topics. Topics are labels for message classification, allowing subscribers to subscribe to one or more topics according to their needs. Publishers publish messages based on topics, and the broker ensures that these messages can be received by all subscribers who have subscribed to the topic.

Diverse Message Distribution

The MQTT protocol, utilizing the publish/subscribe mechanism, can achieve various message distribution methods:

• Broadcast: Messages are sent to all subscribers who have subscribed to a specific topic.
• Multicast: Messages are sent to all subscribers within a specific group.
• Unicast: Messages are sent to a specific single subscriber.

This flexibility allows MQTT to adapt to communication needs ranging from one-to-one to one-to-many, and even many-to-many.

Broker

In the MQTT (Message Queuing Telemetry Transport) protocol, the server is usually referred to as the broker, which plays a crucial role in the publish/subscribe communication architecture. The broker acts as the core intermediary for message delivery, responsible for receiving messages from clients and forwarding them to the appropriate subscribing clients.

Role of the Central Intermediary

The main function of the broker is to act as a relay station for messages. In the MQTT architecture, publishers send messages to the broker, and subscribers indicate their interest in specific topics to the broker. The broker maintains these subscription details and ensures that when a message is published to a topic, it is correctly forwarded to all clients subscribed to that topic.

Message Routing and Distribution

The broker is responsible for routing and distributing messages. It decides which messages should be sent to which clients based on the interests of the subscribers (i.e., the topics they have subscribed to). This mechanism makes the message delivery process very efficient, as the broker only sends messages to clients that actually need to receive them.

Maintaining Connection State

The broker is also responsible for maintaining the connection state with clients. It tracks the connection and disconnection process of each client and handles reconnections and message recovery when necessary.

Supporting Scalability

The design of the broker allows it to support connections from a few to millions of clients. This scalability makes the MQTT protocol very suitable for large-scale distributed systems, including IoT applications.

Ensuring Message Quality of Service

The broker is also responsible for implementing the three levels of Quality of Service (QoS) defined by the MQTT protocol. It ensures that messages are delivered at the appropriate QoS level as requested by the clients.

Security and Authentication

The broker typically provides security and authentication features, ensuring that only authorized clients can connect to the broker and publish or subscribe to messages. This includes support for TLS/SSL encrypted connections and various client authentication mechanisms.

Message Persistence

Some broker implementations also support message persistence, meaning that even if the broker restarts, messages will not be lost and can be re-delivered to subscribers.

Client

In the MQTT (Message Queuing Telemetry Transport) protocol ecosystem, clients play a crucial role. Clients can be any device or application that uses the MQTT protocol to connect to the MQTT server (broker). They are responsible for communicating with the broker, both publishing messages to topics and subscribing to and receiving messages from topics.

Dual Role of Publisher and Subscriber

MQTT clients have flexible identities; they can act as publishers, sending messages to the broker, or as subscribers, indicating their interest in specific topics and receiving messages. In some cases, a client may have both roles, both publishing and subscribing to messages.

Connection Management

Clients are responsible for establishing and maintaining network connections with the broker. This includes handling the initialization of connections, disconnections, and reconnection mechanisms in case of network instability.

Message Publishing

As publishers, clients can publish messages to one or more topics. These messages are then distributed by the broker according to the subscription rules of the subscribers. Clients can control the Quality of Service (QoS) of messages to ensure that messages are delivered with the desired level of reliability.

Message Subscription

As subscribers, clients subscribe to specific topics on the broker. Once subscribed, clients will receive all messages published to those topics. Clients can subscribe to multiple topics according to their needs to receive different types of messages.

Message Handling

Clients need to be capable of handling received messages. This includes parsing message content, executing corresponding business logic, and possibly responding.

Will Message

Clients can set a will message, a special mechanism used to publish a final message by the broker if the client disconnects unexpectedly, informing other clients of the client’s status.

Persistence

Clients can set their subscriptions as persistent subscriptions. This way, even if the client disconnects, it can still receive messages published during its offline period when it reconnects.

Security and Authentication

When connecting to the broker, clients need to ensure secure communication and authentication. This may include using TLS/SSL encrypted connections, client certificates, or other authentication mechanisms.

Topic

In the MQTT (Message Queuing Telemetry Transport) protocol, topics are the core of the message routing system, used to identify and distinguish different messages. Topics not only define the content or type of messages but also determine how messages are delivered to interested recipients.

Message Identification and Classification

Topics serve as the identification of messages, allowing clients to clearly specify the type of messages they want to publish or receive. By assigning appropriate topics to messages, clients can ensure that information is correctly classified and delivered to the target audience.

Publish and Subscribe Mechanism

Publishers specify topics when sending messages, and subscribers subscribe to messages based on topics. This mechanism allows the broker to route messages from publishers to all subscribers who have subscribed to the same topic.

Hierarchical Structure of Topics

MQTT topics usually have a hierarchical structure, similar to the path of a file system. Topic levels are separated by slashes (”/”), allowing the creation of topics with different levels of granularity. For example, “home/living_room/temperature” can be a topic representing the temperature of the living room at home.

Use of Wildcards

MQTT supports the use of wildcards to subscribe to multiple topics. There are two types of wildcards:

• Single-level wildcard (+): Matches one level in the topic, such as “home/living_room/+” can match “home/living_room/temperature” or “home/living_room/humidity”.
• Multi-level wildcard (#): Matches all subsequent levels, such as “home/living_room/#” can match “home/living_room/temperature”, “home/living_room/humidity”, and deeper levels.

Message Filtering

Topics allow subscribers to filter messages based on specific topics, receiving only the messages they are interested in. This reduces unnecessary message delivery and improves communication efficiency.

Wildcards

Subscribers can use wildcards in the topics they subscribe to achieve the purpose of subscribing to multiple topics at once. MQTT provides two types of topic wildcards to meet different subscription needs.

QoS

In the MQTT (Message Queuing Telemetry Transport) protocol, Quality of Service (QoS) is a core feature that determines the reliability of message transmission between publishers and subscribers. MQTT provides different levels of QoS to meet the needs of various application scenarios.

QoS Levels

MQTT defines three different levels of QoS, each meeting different message delivery requirements:

1. QoS 0 - At most once delivery
   • This is the most basic level of QoS, providing the lowest guarantee of message delivery.
   • The publisher sends the message without any acknowledgment or retransmission, so the message may be lost during transmission.
   • This level of QoS is suitable for scenarios where message delivery reliability is not critical, such as status updates or environmental monitoring information.
   • QoS 0 messages are transmitted the fastest because they reduce the overhead of network interactions.
2. QoS 1 - At least once delivery
   • This level of QoS provides a higher guarantee of message delivery.
   • The publisher sends the message, and the broker ensures that the message is received at least once by the subscriber.
   • If the message is lost during transmission, a retransmission mechanism is triggered until the message is successfully delivered.
   • This level of QoS is suitable for scenarios where it is important to ensure that the message is delivered at least once, such as email notifications or message alerts.
   • QoS 1 improves message reliability through acknowledgment and retransmission mechanisms but may introduce message duplication.
3. QoS 2 - Exactly once delivery
   • This is the highest level of QoS, providing the highest guarantee of message delivery.
   • QoS 2 ensures that the message is received only once by the subscriber and maintains the order of messages.
   • This level of QoS is suitable for scenarios where message delivery accuracy and order are critical, such as financial transaction confirmations or critical mission control commands.
   • QoS 2 ensures the exact once delivery of messages through complex acknowledgment and tracking mechanisms but increases protocol complexity and transmission latency.

Choosing QoS

Choosing the appropriate QoS level depends on the specific needs of the application scenario. Developers need to balance message delivery reliability, efficiency, and cost:

• For scenarios where message delivery does not need to be guaranteed, QoS 0 can be chosen for the fastest transmission speed.
• For scenarios where it is important to ensure that the message is delivered at least once, QoS 1 can be chosen to balance reliability and transmission efficiency.

1. QoS 0 - At most once delivery
   • This is the most basic level of QoS, providing the lowest guarantee of message delivery.
   • The publisher sends the message without any acknowledgment or retransmission, so the message may be lost during transmission.
   • This level of QoS is suitable for scenarios where message delivery reliability is not critical, such as status updates or environmental monitoring information.
   • QoS 0 messages are transmitted the fastest because they reduce the overhead of network interactions.
2. QoS 1 - At least once delivery
   • This level of QoS provides a higher guarantee of message delivery.
   • The publisher sends the message, and the broker ensures that the message is received at least once by the subscriber.
   • If the message is lost during transmission, a retransmission mechanism is triggered until the message is successfully delivered.
   • This level of QoS is suitable for scenarios where it is important to ensure that the message is delivered at least once, such as email notifications or message alerts.
   • QoS 1 improves message reliability through acknowledgment and retransmission mechanisms but may introduce message duplication.
3. QoS 2 - Exactly once delivery
   • This is the highest level of QoS, providing the highest guarantee of message delivery.
   • QoS 2 ensures that the message is received only once by the subscriber and maintains the order of messages.
   • This level of QoS is suitable for scenarios where message delivery accuracy and order are critical, such as financial transaction confirmations or critical mission control commands.
   • QoS 2 ensures the exact once delivery of messages through complex acknowledgment and tracking mechanisms but increases protocol complexity and transmission latency.

Choosing QoS

Choosing the appropriate QoS level depends on the specific needs of the application scenario. Developers need to balance message delivery reliability, efficiency, and cost:

• For scenarios where message delivery does not need to be guaranteed, QoS 0 can be chosen for the fastest transmission speed.
• For scenarios where it is important to ensure that the message is delivered at least once, QoS 1 can be chosen to balance reliability and transmission efficiency.
• For scenarios where message delivery accuracy and order are critical, QoS 2 can be chosen to ensure the exact once delivery of messages.

Session

In the MQTT (Message Queuing Telemetry Transport) protocol, session is a key concept that defines the stateful interaction between the client and the server (Broker). Session management is crucial for ensuring message delivery and maintaining subscription status.

Two types of sessions

1. Temporary session
   • A temporary session exists only for the duration of the network connection.
   • When the client disconnects from the Broker, the temporary session ends, and all state information will not be retained.
   • Suitable for scenarios where message or subscription information does not need to be persisted across connections.
2. Persistent session
   • A persistent session can exist across multiple network connections.
   • When the client disconnects from the Broker, if the client is configured for a persistent session, the Broker will retain its subscription information and undelivered messages (except for will messages).
   • When the client reconnects, it can resume from the existing persistent session and continue to receive messages from previously subscribed topics.

Session recovery

• Recover from an existing session
  • MQTT clients can choose to recover from an existing session, which is usually used for persistent sessions.
  • Recovering a session allows the client to continue receiving messages published during its offline period, ensuring that messages are not lost.
• Start a new session
  • Clients can also choose to ignore the existing session state and start a new session.
  • This will cause the Broker to discard previous state information, and the client will not be able to receive messages published during its offline period.

Session configuration

• Clean Session
  • When the client sets the Clean Session flag, the Broker will not retain any previous state information.
  • Each connection is a new session, suitable for scenarios where subscription information does not need to be persisted.
• Persistent Session
  • When the client does not set the Clean Session flag, it expects the Broker to retain its subscription information and undelivered messages.
  • This allows the client to resume the persistent session when reconnecting.

Retained Messages

In the MQTT (Message Queuing Telemetry Transport) protocol, retained messages are an important feature that allows messages to be stored in the Broker and delivered to new subscribers when appropriate.

Concept of retained messages

Retained messages are different from regular messages; they can be stored in the MQTT server (Broker). When a client publishes a message with the retained flag, the Broker will retain this message and associate it with a specific topic.

Delivery mechanism of retained messages

• Immediate delivery to new subscribers
  • If a new subscriber subscribes to a topic that matches the retained message topic, even if the message was published before the subscription, the Broker will immediately deliver the retained message to these new subscribers.
  • This ensures that new subscribers can receive the latest or most important information related to the topic.
• Only deliver the latest retained message
  • If a topic has multiple retained messages, the Broker will only deliver the latest retained message to subscribers.
  • This avoids subscribers receiving duplicate or outdated information.

Application scenarios of retained messages

Retained messages are useful in various scenarios, including but not limited to:

• Status indicators: Retained messages can serve as indicators of the status of a device or system, for example, a device can publish a retained message indicating its online or offline status.
• Providing the latest data: In data collection and monitoring systems, retained messages can provide the latest measurements of environmental parameters (such as temperature, humidity).
• Message notifications: In scenarios where it is necessary to notify all subscribers of specific events or messages, retained messages can ensure that all new subscribers receive the notification.