· Zen HuiFer · Tutorial  · 2 min read

Data Flow Link

Understand the core data structure and installation method of the data flow link in the Go IoT development platform, and optimize the data processing flow of IoT projects.

Understand the core data structure and installation method of the data flow link in the Go IoT development platform, and optimize the data processing flow of IoT projects.

Data Flow Link

Core Data Structure

On-site installation methods for IoT projects:

  1. Physical devices can directly communicate over the network and report data
  2. Physical devices need to report data through gateway devices

For these two installation modes, the core data structure within this system is proposed.

type DataRowList struct {
	Time      int64     `json:"time"`       // 秒级时间戳
	DeviceUid string    `json:"device_uid"` // 能够产生网络通讯的唯一编码
	IdentificationCode string `json:"identification_code"` // 设备标识码
	DataRows  []DataRow `json:"data"`
	Nc        string    `json:"nc"`
}
type DataRow struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}


In the first installation mode, DeviceUid and IdentificationCode are the same, while in the second installation mode, they are different.

Note: This data structure is not the data structure reported by the device, but the internal circulation data structure. After the device reports, it needs to be parsed into this data structure through a parsing script.

Below is an example of data reported by an MQTT client. The DeviceUid field is the unique identifier of the MQTT client in this system. This identifier is determined when the MQTT client is created, so the user only needs to copy the relevant unique identifier. In addition, if the MQTT wildcard subscription mode is not used, IdentificationCode needs to be set the same as DeviceUid. The processing modes of other protocols are the same.

The following diagram shows the complete process from device data reporting to data processing.

  1. Devices send data to the specified server through COAP, MQTT, TCP/IP, WebSocket, and HTTP.
  2. After receiving the data, the server forwards it to different message queues according to different reporting protocols.
Reporting ProtocolMessage Queue
COAPpre_coap_handler
MQTTpre_handler
TCP/IPpre_tcp_handler
WebSocketpre_ws_handler
HTTPpre_http_handler
  1. The message queue will find the corresponding parsing script to convert the original message into DataRowList data. Data storage will be completed in pre_xxx_handler.
  2. After passing through the pre_xxx_handler queue, the data will be placed into the waring_handler, waring_delay_handler, and transmit_handler message queues for subsequent data processing. Among them, waring_handler will put the data judged to be an alarm into the notification queue waring_notice.

设备数据流转链路.drawio

Data Storage

This project uses Influxdb2 as the data storage tool. The overall design scheme is as follows.

  1. bucket is the content of the configuration file
  2. measurement calculation rule: protocol_${DeviceUid}_${IdentificationCode}
  3. field details are as follows
Data FieldData Value
storage_timeSystem storage time, this data is generated by the system, the time before writing to influxdb.
push_timeData reporting time, this data is obtained from ${DataRowList.Time}
DataRowList.DataRows.NameDataRowList.DataRows.Value
Back to Blog

Related Posts

View All Posts »
MQTT Client Management Solution

MQTT Client Management Solution

This article details the solution for designing and managing a large number of MQTT clients in IoT projects, including constraints, solution design, load balancing, and failover, helping developers optimize system performance and ensure stable operation.

Data Alarm Design

Data Alarm Design

This article details the design scheme of data alarms in IoT projects, including temperature monitoring alarms, equipment performance degradation alarms, and multi-device linkage alarms, helping developers better understand and apply data alarm technology.

Basic Knowledge of IoT

Basic Knowledge of IoT

This chapter summarizes the basic knowledge of IoT, covering the definition of IoT, the three-layer architecture model of IoT, and its applications in smart homes, industry, agriculture, healthcare, and other fields.