WO2023109806A1

WO2023109806A1 - Method and apparatus for processing active data for internet of things device, and storage medium

Info

Publication number: WO2023109806A1
Application number: PCT/CN2022/138668
Authority: WO
Inventors: 贾水钦; 朱明�; 任勇强; 丁霞; 王世杰
Original assignee: 天翼物联科技有限公司
Priority date: 2021-12-14
Filing date: 2022-12-13
Publication date: 2023-06-22
Also published as: CN114385378A

Abstract

A method and apparatus for processing active data for an Internet of things device, and a storage medium. The method comprises: obtaining a device reporting message, and pushing the device reporting message to a distributed message queue, the device reporting message comprising a message type (S100); constructing, on the basis of a stream computing service, a data stream of the device reporting message in the message queue (S110); cleaning the data stream according to the message type to obtain a data stream only comprising device active data (S120); processing device active data in a cache according to the data stream (S130); processing device active data in Hbase according to the device active data in the cache (S140); and performing batch processing on the device active data in the HBase on the basis of a batch processing service to obtain active basic statistical data and storing same into a database (S150). In the method, high concurrency requirements generated by mass device data are satisfied by means of a stream processing service; the cache is introduced as a buffer, thereby reducing data interaction with HBase, and improving data processing performance.

Description

Active data processing method, device and storage medium of Internet of things equipment

technical field

The present application relates to the field of data processing, and in particular to an active data processing method, device and storage medium of an Internet of Things device.

Background technique

With the development of the Internet of Things technology, the number of devices in the Internet of Things is increasing, and the active data of the devices is an important indicator and operational data for the operation of the Internet of Things platform. Due to the large number of devices in the Internet of Things, and the differences in device forms, the behavior of devices varies greatly. In related technologies, traditional database collection and recording methods are used to collect active data such as daily active data and monthly active data of devices. It is difficult to meet the high data concurrency requirements generated by massive devices.

Contents of the invention

This application aims to solve one of the technical problems in the related art at least to a certain extent. To this end, the present application proposes a method, device and storage medium for processing active data of an Internet of Things device.

In the first aspect, the embodiment of the present application provides a method for processing active data of an Internet of Things device, including: obtaining a device report message, and pushing the device report message to a distributed message queue; wherein, the device report message Including the message type; based on the flow computing service, constructing the data flow of the device reporting message in the message queue; according to the message type, cleaning the data flow to obtain the data flow containing only the active data of the device ; According to the data stream after cleaning, the active data of the device in the cache is added, modified or not processed; according to the active data of the device in the cache, the active data of the device in Hbase is Adding, modifying or not processing; based on the batch processing service, batch processing is performed on the device active data in the Hbase to obtain active basic statistical data, and store the active basic statistical data in a database.

Optionally, the active device data includes the device ID, the date of going online and offline of the device, and the time of going online and offline of the device. According to the data stream after cleaning, the active data of the device in the cache is added, Modifying or not processing, including: querying the cache according to the device ID and the online and offline dates in the data stream; According to the device active data in the date, according to the device ID in the data stream, the online and offline date and the online and offline time, add the active data of the device in the cache; The device active data of the same device in the same date exists in the data stream and the cache, and according to the online and offline time, the modification processing or no processing is performed on the device active data in the cache.

Optionally, when querying the data flow and the device active data of the same device in the same date in the cache, according to the online and offline time, the active data of the device in the cache Performing the modification processing or not processing includes: when the device active data of the same device in the same date exists in the data stream and the cache, the first online and offline time in the data stream and the The second online and offline time in the cache is compared; when the first online and offline time is later than the second online and offline time, replace the second online and offline time in the cache with the first online and offline time Online time; when the first online and offline time is earlier than or equal to the second online and offline time, no processing is performed.

Optionally, adding, modifying or not processing the device active data in Hbase according to the device active data in the cache includes: setting a timed task according to business requirements; the timed task Including the operation interval; every time the operation interval passes, according to the device active data in the cache, the device active data in the Hbase is added, modified or not processed.

Optionally, adding, modifying or not processing the device active data in Hbase according to the device active data in the cache includes: according to the device ID in the cache, The Hbase is inquired; when the device active data of the same device exists in the cache and the Hbase, the first log-off date in the cache is compared with the second log-off date in the Hbase; When the first online and offline dates are different from the second online and offline dates, according to the device ID in the cache, the online and offline dates and the online and offline time, add the new online and offline time in the Hbase The active data of the device; when the first online and offline date is the same as the second online and offline date, and the first online and offline time in the cache is earlier than the second online and offline time in the Hbase, the The device active data in the Hbase is replaced with the device active data in the cache; when the first online and offline date is the same as the second online and offline date, and the first online and offline time in the cache is later If it is equal to or equal to the second online and offline time in the Hbase, it will not be processed.

Optionally, the batch-based service performs batch processing on the device active data in the Hbase, obtains active basic statistical data, and stores the active basic statistical data in a database, including: based on the batch Processing services, obtaining the device active data in the Hbase in batches; analyzing the device active data in batches to obtain user fields, product fields and device fields in the device active data; analyzing the device active data The active data is assembled into an on-line and off-line data sequence, and the on-line and off-line data sequence is composed of a plurality of unit data; starting from the last unit data in the on-line and off-line data sequence, recursively forward, judging from the unit data that the device is in The active status within the specified time period; according to the user field, the product field, the device field and the specified time period, perform aggregation calculations on the active status to obtain the active basic statistical data, and storing said activity base statistics data in said database.

Optionally, the unit data includes device operation and operation time; the recursion starts from the last unit data in the online and offline data sequence, and it is judged according to the unit data that the device is within the specified time period The active status of the device includes: when the operation time is within the specified period, judging that the device is active within the specified period; when the operation time is later than the end time of the specified period, take the previous The unit data is re-judged; when the operation time is earlier than or equal to the start time of the specified period, and the operation of the device is online, it is determined that the device is active within the specified period; when the operation time is earlier If it is at or equal to the start time of the specified time period, and the operation of the device is offline, it is determined that the device is inactive within the specified time period.

In the second aspect, the embodiment of the present application provides an active data processing system of an Internet of Things device, including: a first module, configured to obtain a device report message, and push the device report message to a distributed message queue; wherein , the message reported by the device includes a message type; the second module is configured to construct a data flow of the message reported by the device in the message queue based on the flow computing service; the third module is configured to generate the message according to the message type The data stream is cleaned to obtain the data stream containing only the active data of the device; the fourth module is used to add, modify or modify the active data of the device in the cache according to the data stream after cleaning. No processing; the fifth module is used to add, modify or not process the device active data in Hbase according to the device active data in the cache; the sixth module is used for batch-based services, Perform batch processing on the device active data in the Hbase to obtain active basic statistical data, and store the active basic statistical data in a database.

In a third aspect, the embodiment of the present application provides an apparatus, including: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the At least one processor implements the active data processing method of the IoT device as described in the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a processor-executable program, and the processor-executable program is used to implement the program described in the first aspect when executed by the processor. The active data processing method of the IoT device described above.

The beneficial effects of the embodiments of the present application are as follows: firstly obtain the device report message, and push the device report message to the distributed message queue; wherein, the device report message includes the message type; based on the flow computing service, construct the device report message in the message queue According to the message type, the data flow is cleaned to obtain a data flow containing only the active data of the device; according to the cleaned data flow, the active data of the device in the cache is added, modified or not processed; according to the cache Add, modify or not process the device active data in Hbase; based on the batch processing service, perform batch processing on the device active data in Hbase to obtain active basic statistical data and convert the active basic statistical data into the database. This application processes device active data through stream processing services, which can further meet the high concurrency requirements generated by massive device data. Moreover, this application introduces cache as a buffer for device active data, which can reduce data interaction with Hbase and reduce a large amount of disk I/O processing, thereby further improving data processing performance.

Description of drawings

The accompanying drawings are used to provide a further understanding of the technical solution of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solution of the present application, and do not constitute a limitation to the technical solution of the present application.

FIG. 1 is a flow chart of the steps of the active data processing method of the Internet of Things device provided by the embodiment of the present application;

Fig. 2 is the implementation flowchart of the data collection optimization algorithm that the embodiment of the present application proposes;

Fig. 3 is the flow chart of the steps of the synchronization process of the device active data provided by the embodiment of the present application from cache to Hbase;

FIG. 4 is a schematic diagram of an active data processing system of an Internet of Things device provided by an embodiment of the present application;

Fig. 5 is a schematic diagram of the device provided by the embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

It should be noted that although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, it can be executed in a different order than the module division in the system or the flow chart steps shown or described. The terms "first", "second" and the like in the specification and claims and the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence.

The embodiments of the present application will be further described below in conjunction with the accompanying drawings.

Referring to FIG. 1, FIG. 1 is a flow chart of the steps of the active data processing method of the Internet of Things device provided by the embodiment of the present application. The method includes but is not limited to steps S100-S150:

S100. Obtain a device report message, and push the device report message to a distributed message queue; wherein, the device report message includes a message type;

Specifically, the Internet of Things platform accesses a large number of different types of Internet of Things devices through various protocols such as MQTT (Message Queuing Telemetry Transport) and LWM2M (Lightweight Machine-To-Machine, Lightweight M2M). , in the daily operation of the Internet of Things, the device needs to report its own information to the Internet of Things platform, and this information is called device reporting information. Because the information reported by devices is often huge, after the IoT platform obtains the information reported by these devices, it will simply analyze the information reported by the devices, and push it to the distributed message queue, which is temporarily stored in the message queue.

It can be understood that the information reported by the device can include various conditions of the IoT device during operation, so the information reported by the device may include device failure data, device energy consumption data, or device activity data. The message type of the device reported information can be used for Multiple types of device data are distinguished.

S110. Based on the flow computing service, construct a data flow of the message reported by the device in the message queue;

Specifically, based on the flow computing service, build a data source flow object, and specify the data source that needs to be consumed in the distributed message queue, so as to build the data flow of the message reported by the device in the message queue, and use the flow computing engine to consume in the message queue The data (that is, the message reported by the device) is monitored in real time.

S120. Clean the data stream according to the message type to obtain a data stream containing only device active data;

Specifically, it is mentioned in the above content that the message reported by the device reflecting different situations of the device can be distinguished through the message type of the information reported by the device. Since the embodiment of the present application proposes a method for processing active data of IoT devices, in this step, the data flow is cleaned according to the message type of the message reported by the device, specifically, the data in the data flow such as device failure data, device consumption data, etc. Non-device active data such as performance data are cleaned and filtered to obtain a data stream containing only device active data.

It can be understood that since the device is in the active state, it generally means that the device is in the online state; similarly, the device is in the inactive state, which generally means that the device is in the offline state, so the active data of the device is the data that can reflect the state of the device in the offline state. .

More specifically, device active data includes but not limited to device ID, device log-off date, and device log-off time. It can be understood that the log-off date is divided into the log-on date and the log-off date, and the log-off time is divided into the log-on time And offline time, the time corresponding to the operation of device online or device offline is the online time or offline time of the device.

S130. Add, modify, or not process the device active data in the cache according to the cleaned data stream;

Specifically, the embodiment of the present application introduces a cache in the process of processing the active data of the Internet of Things device. The cache is a window aggregation database. The active data of the device enters the cache from the message queue first, and the embodiment of the present application is run in the cache. The proposed data acquisition optimization algorithm further screens and integrates the active data of the equipment, and then integrates the data in the cache into Hbase, thereby greatly reducing the amount of data entering Hbase, and reducing the disk I/O by reducing the amount of data. O processing, to achieve the purpose of improving data processing capabilities.

As mentioned above, device active data includes but is not limited to device ID, device log-off date and device log-off time. The data acquisition optimization algorithm proposed in the embodiment of the present application can integrate the device activity data of the data stream into the device activity data in the cache according to the device ID and other data. The specific flow of the data acquisition optimization algorithm is described below.

With reference to Fig. 2, Fig. 2 is the implementation flowchart of the data acquisition optimization algorithm proposed by the embodiment of the present application, the method includes but not limited to steps S200-S250:

S200. Query the cache according to the device ID and the online and offline dates in the data stream;

Specifically, the query is performed in the cache using the device ID and the online and offline dates in the data stream as keys.

It is understandable that when the data is moved from the data stream to the cache, or from the cache to Hbase, some storage format adjustments may be required, resulting in differences in the data stream, cache, and device active data in Hbase , but the device active data in the data stream, cache, and Hbase all contain information that can reflect the activity of a certain device, such as the device ID, the date of the device's log-in and log-out, and the device's log-in and log-out time. Therefore, in the embodiment of this application, the Data streams, caches, and data stored in Hbase that reflect device activity are collectively referred to as device activity data, and device activity data located in different places in data streams, caches, and Hbase can be compared to determine subsequent data processing steps.

S210, judging whether there is device active data of the same device in the same date in the data stream and cache;

Specifically, step S200 queries the active data of the device in the cache through the two conditions of the device ID and the online and offline date. If the device ID in the data stream is found in the cache, it means that there is data of the same device in the data stream and the cache. ; If the online and offline date of the device queried in the cache is the same as the online and offline information in the data stream, it means that there is device active data of the same device on the same date in the data stream and cache.

Therefore, this step is to judge the result of the query realized in step S200, that is, to judge whether there is device active data of the same device in the same date in the data stream and cache, if not, jump to step S220; if yes, jump to Go to step S230.

S220, according to the device ID in the data stream, the date of going online and going online, and the time of going online and going online, add active data of the device in the cache;

Specifically, according to the judgment of step S210, it is determined that there is no device active data of the same device in the same date in the data stream and cache, then add a device active data in the cache for the device ID, log-off date and log-off time.

S230. Determine whether the first log-off time in the data stream is later than the second log-off time in the cache;

Specifically, according to the judgment of step S210, it is determined that there is device active data of the same device in the same date in the data stream and the cache, then a further judgment is made in this step, and the first online and offline time in the data stream and the first online and offline time in the cache are combined. 2. Comparing and judging the online and offline time.

As mentioned above, the online and offline time is divided into the online time and the offline time. It can be understood that when the first online and offline time is compared with the second online and offline time, it should be the online time in the first online and offline time Compared with the online time in the second online and offline time, the offline time in the first online and offline time is the offline time in the second online and offline time.

According to the comparison method above, it is judged whether the first log-off time in the data stream is later than the second log-off time in the cache, if not, go to step S240; if yes, go to step S250.

S240, do not process the device active data in the cache;

Specifically, according to step S230, it is determined that the first log-off time in the data stream is earlier than or equal to the second log-off time in the cache, and then the device active data in the cache is not processed.

It can be understood that the first online and offline time is earlier than or equal to the second online and offline time specifically means: the online and offline time in the first online and offline time is earlier than or equal to the online and offline time in the second online and offline time, and the first online and offline time The offline time in the online time is earlier than or equal to the offline time in the second online and offline time.

S250, replacing the second online and offline time in the cache with the first online and offline time;

Specifically, according to step S230, it is determined that the first log-off time in the data stream is later than the second log-off time in the cache, and replace the second log-off time corresponding to the device ID in the cache with the first log-off time.

It can be understood that when the first online and offline time is later than the second online and offline time, it specifically means that the online time in the first online and offline time is later than the online time in the second online and offline time and/or in the first online and offline time If the offline time of is earlier than or equal to the offline time in the second online and offline time, then replace the second online and offline time with the first online and offline time. The online time in the second online online time and/or replace the offline time in the second online online time with a later offline time in the first online online time.

Through steps S200-S250, the embodiment of the present application provides a data collection optimization algorithm, and running the data collection optimization algorithm on the data stream can greatly simplify the active data of the device. Taking the reporting frequency of elevator equipment in a smart community as an example, traditional equipment can store as many as 86,400 records per day. Using the equipment activity optimization collection algorithm to calculate daily activity only needs to store 1 original record for daily and monthly activity calculations. Taking the reporting frequency of electric meter equipment in smart power as an example, the traditional daily activity of the equipment stores as many as 96 records per day, and the calculation of the daily activity using the equipment activity optimization collection algorithm only needs to store one original record for daily and monthly activity calculations. Therefore, the data acquisition optimization algorithm in the embodiment of the present application helps to reduce the amount of data stored in the cache, and effectively avoids the storage and calculation pressure brought by high-frequency offline devices to the system.

Step S130 has been described through the above content, and step S140 will be described below.

S140. Add, modify or not process the device active data in Hbase according to the device active data in the cache;

Specifically, various businesses in the Internet of Things have different requirements for time limits, and a timing task can be set according to business requirements; the timing task includes the duration of the operation interval. After each operation interval, according to the device active data in the cache, add, modify or not process the device active data in Hbase, so as to achieve the purpose of synchronizing the device active data in the cache to Hbase at regular intervals.

In the process of data synchronization, the data in the cache can be further simplified, and the number of data interactions with Hbase can be further reduced, thereby improving the performance of data processing. The following describes the synchronization process of device active data from cache to Hbase.

Referring to Fig. 3, Fig. 3 is a flow chart of the steps of the synchronization process of the device active data from the cache to the Hbase provided by the embodiment of the present application. The method includes but is not limited to steps S300-S370:

S300, query Hbase according to the device ID in the cache;

Specifically, use the device ID in the cache as the key to query in Hbase.

S310, judging whether there is device active data of the same device in the cache and Hbase;

Specifically, if there is a device ID identical to the device ID in the cache in the Hbase, it means that the device active data of the same device exists in the cache and Hbase. Therefore, in this step, it is judged whether there is device active data of the same device in the cache and Hbase, if not, jump to step S320; if yes, jump to step S330.

S320, according to equipment ID, on-line date and on-line time, add equipment active data in Hbase;

Specifically, according to step S310, it is judged that the device active data of the same device does not exist in the cache and Hbase, then according to the device ID, the date of going online and the time of going online and going online, the device active data corresponding to the device ID is added in Hbase. It can be understood that, in the future, the device activity data corresponding to the device ID will be recorded under the newly created device ID.

S330, judging whether the first log-off date in the cache is the same as the second log-off date in Hbase;

Specifically, according to step S310, it is judged that there is device active data of the same device in the cache and Hbase, and then the date corresponding to the device ID is further judged.

It can be understood that the online date in the first online and offline date is compared with the online date in the second online and offline date, and the offline date in the first online and offline date is compared with the offline date in the second online and offline date . In this step, it is judged whether the first log-off date in the cache is the same as the second log-off date in Hbase, if not, go to step S340; if yes, go to step S350.

S340, adding device active data in Hbase;

Specifically, according to the judgment in step S330, the first online and offline date is different from the second online and offline date, that is, the online date in the first online and offline date is different from the online date in the second online and offline date, and the first online and offline date If the offline date in the offline date is different from the offline date in the second online and offline date, then under the device ID in Hbase, insert the active number of the device in the cache.

S350, judging whether the first online and offline time in the cache is earlier than the second online and offline time in Hbase;

Specifically, according to the judgment in step S330, the first online and offline date is the same as the second online and offline date, that is, the online date in the first online and offline date is different from the online date in the second online and offline date, and/or the first online and offline date is different. The offline date in the online date is different from the offline date in the second online and offline date, whether the first online and offline time in the further cache is earlier than the second online and offline time in Hbase, if not, skip to the step S360; if so, go to step S370.

It can be understood that the online time in the first online and offline time is compared with the online time in the second online and offline time, and the offline time in the first online and offline time is compared with the offline time in the second online and offline time.

S360, abandoning the current data flow;

Specifically, according to the judgment of step S350, the first online and offline time is later than or equal to the second online and offline time in Hbase, that is, the online time in the first online and offline time is later than or equal to the online time in the second online and offline time , and the offline time in the first online and offline time is later than or equal to the offline time in the second online and offline time, then Hbase will not be processed, and the current data stream in the cache will be discarded.

S370. Replace the device active data in the Hbase with the device active data in the cache;

Specifically, according to the judgment of step S350, the first online and offline time is earlier than the second online and offline time in Hbase, that is, the online time in the first online and offline time is earlier than the online time in the second online and offline time and/or the first online and offline time. If the offline time in the first online and offline time is earlier than the offline time in the second online and offline time, the device active data in Hbase is replaced with the device active data in the cache, specifically, the device active data in Hbase is deleted, and Add a piece of device active data based on the device active data in the cache.

Through steps S300-S370, the embodiment of the present application provides a synchronization process of device active data from cache to Hbase. Step S140 has been described, and step S150 will be described below.

S150, based on the batch processing service, perform batch processing on the device active data in the Hbase, obtain the active basic statistical data, and store the active basic statistical data in the database;

Specifically, the embodiment of the present application proposes to establish a batch processing task based on a batch processing engine, thereby performing batch processing on device active data in Hbase, and finally obtaining active basic statistical data, and storing the active basic statistical data in the database. In the future, when the IoT platform needs to perform user-level, product-level, or platform-level equipment daily and monthly activity statistics, it can directly call the active basic statistical data in the database for statistics.

In the embodiment of the present application, the method for processing the device active data in Hbase according to the batch processing engine includes: firstly, based on the batch processing service, obtaining the device active data in Hbase in batches; Batch parsing to obtain user fields, product fields, and device fields in device active data, and assemble the parsed device active data into an on-line and on-line data sequence composed of multiple unit data. Then, according to the active determination optimization algorithm proposed in the embodiment of the present application, logically process the online and offline data sequence, that is, start recursively from the last unit data in the online and offline data sequence, and judge the status of the device within a specified period according to the unit data. Active situation. After completing the processing of the online and offline data sequence, the user field, product field, and device field are obtained according to the analysis, and the activity status is aggregated and calculated within a specified period of time to obtain basic statistics of activity at different levels. For example, it is possible to calculate the active basic statistics of multiple IoT devices corresponding to the same user within a specified period of time, and it is also possible to calculate all the same products (such as all cameras in the smart community, face access control, etc. IoT devices) The active basic statistical data within a specified period can also be calculated to obtain the active basic statistical data of all devices in the IoT platform. Finally, the calculated active basic statistics are stored in the database.

The implementation process of the liveness determination optimization algorithm proposed in the embodiment of the present application is described below. Firstly, determine the last unit data in the online and offline data sequence. The unit data includes equipment operation and operation time. Equipment operation refers to the time when the device goes online or goes offline, and the operation time refers to the time when the device goes online or when the device goes offline. Taking the current unit data as the standard, when the operation time is within the specified period, it is judged that the device is active within the specified period; when the operation time is later than the end time of the specified period, the previous unit data in the offline data sequence is taken and re Perform active judgment; when the operation time is earlier than or equal to the start time of the specified period, and the device operation is online, it is judged that the device is active within the specified period; when the operation time is earlier than or equal to the start time of the specified period, and the device operation is offline , to determine that the device has been inactive for a specified period of time.

Based on the above content, run the activity determination optimization algorithm on the online and offline data series, and finally determine the active basic statistical data.

Through steps S100-S150, the embodiment of the present application provides a method for processing active data of an Internet of Things device. The method includes: first obtaining a device report message, and pushing the device report message to a distributed message queue; wherein, the device report The message includes the message type; based on the flow computing service, the data flow of the device reporting the message is constructed in the message queue; according to the message type, the data flow is cleaned to obtain a data flow containing only the active data of the device; according to the cleaned data flow, Add, modify or not process the device active data in the cache according to the data collection optimization algorithm; add, modify or not process the device active data in Hbase according to the device active data in the cache; based on batch processing services, According to the active determination optimization algorithm, the device active data in Hbase is batch-processed to obtain active basic statistical data, and the active basic statistical data is stored in the database. This application processes device active data through stream processing services, which can further meet the high concurrency requirements generated by massive device data. Moreover, this application introduces cache as a buffer for device active data, which can reduce data interaction with Hbase and reduce a large amount of disk I/O processing, thereby further improving data processing performance.

Next, in combination with the above-mentioned multiple embodiments and the actual situation, the active data processing method of the Internet of Things device provided by the embodiment of the present application is described, and the specific steps are as follows:

1) Prepare 21 virtual machines configured with 16G memory and 8-core CPU, deploy distributed real-time stream computing engine Flink on 9 servers, and deploy a Flink cluster (Flink cluster 1-3) for every 3 servers; Deploy the Redis cache service on the server; deploy the distributed message queue Pulsar on the 3 servers; deploy the HBase database on the 3 servers, and deploy the Mysql database on the 3 servers.

2) Create a push queue for device reporting messages on Pulsar, which is used for the IoT platform to receive messages reported by a large number of devices and push them to the designated queue.

3) Deploy stream computing services for data collection and storage on the 1st-2nd Flink cluster, and deploy batch processing services for active computing of massive devices on the 3rd Flink cluster.

4) Set the number of job tasks for the flow computing service of Flink 1-2 clusters to 30, and support a maximum of 30 data streams for data processing services at the same time; set the flow computing service for data collection and storage to monitor the device reporting message Topic queue of the Pulsar cluster .

5) After the stream computing service obtains the message reported by the device, it first cleans the message, cleans and filters the non-device offline message, and then analyzes and processes the cleaned data to identify the user, product, device, operation time, operation identifier, etc. information to form structured data.

6) Perform data acquisition optimization algorithm processing on the structured data, and refer to the above steps S200-S250 for specific processing methods:

7) Set the number of job tasks for the batch computing service of Flink's third cluster to 15, and support a maximum of 15 data streams for batch data services at the same time.

8) The scheduling service of the IoT platform starts to trigger the batch processing service at 00:05 every night to calculate the daily and monthly active status of massive devices on the platform on that day.

9) Batch processing service loads massive equipment online and offline data in Hbase at one time, and parses the data to form a structured list.

10) Perform active determination optimization algorithm processing on the device log-off data List, and refer to the above-mentioned step S150 for the specific processing method. If it is necessary to calculate the daily activity of the device, the specified period is a specified day; if it is necessary to calculate the monthly activity of the device, the specified period is a specified month. And the active device is marked as 1, and the inactive device is marked as 0.

11) After the judgment, the device data is aggregated and calculated from the dimensions of user, product and time range to generate active basic statistical data.

12) After the 15 job tasks are calculated, the massive active basic statistical data will be put into the database, inserted into the mysql database, and then the subsequent business layer data statistics will be processed.

13) After obtaining the active basic data of a large number of devices, the IoT platform can use the basic data to make daily and monthly activity statistics of user equipment, product-level equipment daily and monthly activity statistics, or platform-level equipment daily and monthly activity statistics.

In addition, referring to FIG. 4, FIG. 4 is a schematic diagram of an active data processing system of an Internet of Things device provided by an embodiment of the present application. The system 400 includes a first module 410, a second module 420, a third module 430, a fourth module 440, The fifth module 450 and the sixth module 460 . The first module is used to obtain the device report message, and push the device report message to the distributed message queue; wherein, the device report message includes the message type; the second module is used to build the device report message in the message queue based on the stream computing service The data flow; the third module is used to clean the data flow according to the message type, and obtain the data flow containing only the device active data; the fourth module is used to clean the device active data in the cache according to the cleaned data flow Add, modify or not process; the fifth module is used to add, modify or not process the device active data in Hbase according to the device active data in the cache; Batch processing of device activity data to obtain active basic statistical data and store active basic statistical data in the database.

Referring to FIG. 5, FIG. 5 is a schematic diagram of a device provided by an embodiment of the present application. The device 500 includes at least one processor 510 and at least one memory 520 for storing at least one program; in FIG. 5, a processor and a memory as an example.

The processor and the memory may be connected through a bus or in other ways, and connection through a bus is taken as an example in FIG. 5 .

As a non-transitory computer-readable storage medium, memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, flash memory device or other non-transitory solid-state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, which remote memory may be connected to the device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Another embodiment of the present application further provides an apparatus, which can be used to execute the control method in any of the above embodiments, for example, execute the method steps in FIG. 1 described above.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

The embodiment of the present application also discloses a computer storage medium, which stores a processor-executable program, which is characterized in that the processor-executable program is used to implement the Internet of Things device proposed in this application when executed by the processor. Active data processing method.

Those skilled in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

The above is a specific description of the preferred implementation of the application, but the application is not limited to the above-mentioned implementation, and those skilled in the art can also make various equivalent deformations or replacements without violating the spirit of the application. Equivalent modifications or replacements are all within the scope defined by the claims of the present application.

Claims

A method for processing active data of an Internet of Things device, characterized in that it comprises:

Obtaining a device report message, and pushing the device report message to a distributed message queue; wherein, the device report message includes a message type;

Based on the flow computing service, constructing the data flow of the message reported by the device in the message queue;

According to the message type, the data stream is cleaned to obtain the data stream containing only device active data;

Adding, modifying or not processing the active data of the device in the cache according to the data stream after cleaning;

According to the active data of the device in the cache, the active data of the device in Hbase is added, modified or not processed;

Based on the batch processing service, perform batch processing on the device active data in the Hbase to obtain active basic statistical data, and store the active basic statistical data in a database.
The active data processing method of an Internet of Things device according to claim 1, wherein the active data of the device includes the device ID, the date of going online and offline of the device, and the time of going online and offline of the device. Data flow, adding, modifying or not processing the active data of the device in the cache, including:

Querying the cache according to the device ID and the online and offline dates in the data stream;

When it is found that the data stream and the cache do not contain the device active data of the same device on the same date, according to the device ID in the data stream, the online and offline date, and the online and offline time , adding active data of the device in the cache;

When it is found that the data stream and the cache have the device active data of the same device on the same date, according to the online and offline time, perform the modification process on the device active data in the cache or No processing.
The method for processing active data of an Internet of Things device according to claim 2, wherein when the device active data of the same device in the same date is found in the data stream and the cache, according to the The above-mentioned online and offline time is used to modify or not process the active data of the device in the cache, including:

When the device active data of the same device on the same date exists in the data stream and the cache, compare the first log-off time in the data stream with the second log-off time in the cache;

When the first online and offline time is later than the second online and offline time, replace the second online and offline time in the cache with the first online and offline time;

When the first log-off time is earlier than or equal to the second log-off time, no processing is performed.
The active data processing method of the Internet of Things device according to claim 2, wherein, according to the active data of the device in the cache, the active data of the device in Hbase is added, modified or deleted. processing, including:

Set a timing task according to business requirements; the timing task includes an operation interval;

Every time the operation interval passes, according to the device active data in the cache, the device active data in the Hbase is added, modified or not processed.
The active data processing method of an Internet of Things device according to any one of claims 2-4, wherein the active data of the device in Hbase is updated according to the active data of the device in the cache. addition, modification or non-processing, including:

According to the device ID in the cache, query the Hbase;

When the device active data of the same device exists in the cache and the Hbase, compare the first log-off date in the cache with the second log-off date in the Hbase;

When the first online and offline dates are different from the second online and offline dates, according to the device ID in the cache, the online and offline dates and the online and offline time, add the new online and offline time in the Hbase active data of the above device;

When the first online and offline date is the same as the second online and offline date, and the first online and offline time in the cache is earlier than the second online and offline time in the Hbase, the device in the Hbase The active data is replaced with the device active data in the cache;

When the first log-off date is the same as the second log-off date, and the first log-off time in the cache is later than or equal to the second log-off time in the Hbase, no processing is performed.
The active data processing method of an Internet of Things device according to claim 1, wherein the batch-based service performs batch processing on the active data of the device in the Hbase to obtain active basic statistical data, and The active basic statistical data is stored in the database, including:

Based on the batch processing service, obtain the device active data in the Hbase in batches;

Batch parsing the device active data to obtain user fields, product fields and device fields in the device active data;

Assembling the parsed active data of the device into an on-line and off-line data sequence, the on-line and off-line data sequence is composed of a plurality of unit data;

starting from the last unit data in the online and offline data sequence, and recursing forward, judging the activity of the device within a specified period of time according to the unit data;

According to the user field, the product field, the device field and the specified time period, the active status is aggregated and calculated to obtain the active basic statistical data, and the active basic statistical data is stored in the database.
The active data processing method of an Internet of Things device according to claim 6, wherein said unit data includes device operation and operation time; said starting from the last said unit data in said online and offline data sequence to Pre-recursion, judging the activity of the device within the specified time period according to the unit data, including:

When the operation time is within the specified period, it is determined that the device is active within the specified period;

When the operation time is later than the end time of the specified time period, the previous unit data is used for re-judgment;

When the operation time is earlier than or equal to the start time of the specified period, and the operation of the device is online, it is determined that the device is active within the specified period;

When the operation time is earlier than or equal to the start time of the specified period, and the operation of the device is offline, it is determined that the device is inactive within the specified period.
An active data processing system for an Internet of Things device, characterized in that it includes:

The first module is configured to obtain a device report message, and push the device report message to a distributed message queue; wherein the device report message includes a message type;

The second module is configured to construct a data flow of the message reported by the device in the message queue based on the flow computing service;

A third module, configured to clean the data stream according to the message type, and obtain the data stream containing only device active data;

The fourth module is used to add, modify or not process the active data of the device in the cache according to the data stream after cleaning;

The fifth module is used to add, modify or not process the device active data in Hbase according to the device active data in the cache;

The sixth module is configured to perform batch processing on the device active data in the Hbase based on a batch processing service, obtain active basic statistical data, and store the active basic statistical data in a database.
A device, characterized in that it comprises:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the active data processing method of the Internet of Things device according to any one of claims 1-7.
A computer storage medium, in which a program executable by a processor is stored, wherein the program executable by the processor is used to implement any one of claims 1-7 when executed by the processor Active data processing methods for IoT devices.