Data Flow Link

Core Data Structure

On-site installation methods for IoT projects:

1. Physical devices can directly communicate over the network and report data
2. Physical devices need to report data through gateway devices

For these two installation modes, the core data structure within this system is proposed.

type DataRowList struct {
	Time      int64     `json:"time"`       // 秒级时间戳
	DeviceUid string    `json:"device_uid"` // 能够产生网络通讯的唯一编码
	IdentificationCode string `json:"identification_code"` // 设备标识码
	DataRows  []DataRow `json:"data"`
	Nc        string    `json:"nc"`
}
type DataRow struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

In the first installation mode, DeviceUid and IdentificationCode are the same, while in the second installation mode, they are different.

Note: This data structure is not the data structure reported by the device, but the internal circulation data structure. After the device reports, it needs to be parsed into this data structure through a parsing script.

Below is an example of data reported by an MQTT client. The DeviceUid field is the unique identifier of the MQTT client in this system. This identifier is determined when the MQTT client is created, so the user only needs to copy the relevant unique identifier. In addition, if the MQTT wildcard subscription mode is not used, IdentificationCode needs to be set the same as DeviceUid. The processing modes of other protocols are the same.

Core Data Flow Link

The following diagram shows the complete process from device data reporting to data processing.

1. Devices send data to the specified server through COAP, MQTT, TCP/IP, WebSocket, and HTTP.
2. After receiving the data, the server forwards it to different message queues according to different reporting protocols.

3. The message queue will find the corresponding parsing script to convert the original message into DataRowList data. Data storage will be completed in pre_xxx_handler.
4. After passing through the pre_xxx_handler queue, the data will be placed into the waring_handler, waring_delay_handler, and transmit_handler message queues for subsequent data processing. Among them, waring_handler will put the data judged to be an alarm into the notification queue waring_notice.

Data Storage

This project uses Influxdb2 as the data storage tool. The overall design scheme is as follows.

1. bucket is the content of the configuration file
2. measurement calculation rule: protocol_${DeviceUid}_${IdentificationCode}
3. field details are as follows