WO2021151279A1

WO2021151279A1 - Method and apparatus for cloud monitoring based on edge computing, electronic device, and storage medium

Info

Publication number: WO2021151279A1
Application number: PCT/CN2020/099092
Authority: WO
Inventors: 王鑫; 楼泽君; 宋永亮
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-06-17
Filing date: 2020-06-30
Publication date: 2021-08-05
Also published as: CN111723727A

Abstract

A cloud monitoring method based on edge computing, comprising: receiving media stream data packets sent by a video recording device (S11), then sequentially parsing the media stream data packets to obtain a media frame queue of a pre-set type and adding the media frame queue to a pre-set buffer space (S12), then, reading media frames to be processed from the media frame queue (S13) and using a pre-established analysis model to perform analysis processing on the media frames to be processed to obtain feature information corresponding to the media frames to be processed, then finding feature information satisfying a pre-set condition in the feature information, and causing the media frames to be processed that correspond to the found feature information to serve as target media frames (S14), and finally, on the basis of the target media frames, generating a monitoring analysis result and sending same to a cloud device (S15). The present method improves monitoring analysis efficiency.

Description

Cloud monitoring method, device, electronic equipment and storage medium based on edge computing

This application requires the priority of a Chinese patent application filed with the Chinese Patent Office on June 17, 2020, the application number is CN202010557709.7, and the invention title is "Cloud monitoring methods, devices, electronic equipment and storage media based on edge computing". The entire content is incorporated into this application by reference.

Technical field

This application relates to cloud monitoring technology, and in particular to a cloud monitoring method, device, electronic equipment, and storage medium based on edge computing.

Background technique

The monitoring system is the physical basis for real-time monitoring of specific targets (for example, specific places, people, objects, etc.) in various industries. The management department can obtain effective data, image, video, or sound information through it, and carry out the process of sudden abnormal events. Timely monitoring and memory. The inventor realizes that the traditional monitoring system still requires a lot of manpower to identify and analyze the monitoring data, which results in low efficiency in obtaining monitoring and analysis results.

Therefore, how to improve the efficiency of monitoring and analysis has become an urgent technical problem to be solved.

Summary of the invention

The main purpose of this application is to provide a cloud monitoring method, device, electronic equipment, and computer-readable storage medium based on edge computing, aiming to improve the efficiency of monitoring and analysis.

To achieve the above objective, this application provides a cloud monitoring method based on edge computing, which is applied to edge devices, including:

Receive media stream data packets sent by video recording equipment;

Parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to the preset buffer space;

Read the media frame to be processed from the media frame queue;

Use the pre-established analysis model to analyze and process each media frame to be processed to obtain feature information corresponding to each media frame to be processed, search for feature information that meets preset conditions in the feature information, and retrieve the feature information The corresponding to-be-processed media frame is used as the target media frame;

A monitoring analysis result is generated based on the target media frame, and the monitoring analysis result is sent to a cloud device.

In order to solve the above problems, this application also provides a cloud monitoring method based on edge computing, which is applied to cloud devices, including:

Receiving and storing the monitoring analysis result sent by the edge device, where the monitoring analysis result is obtained by the edge device analyzing the media stream data packet sent by the camera recording device;

In response to a query request carrying query conditions sent by the user terminal, query the stored monitoring analysis results for monitoring analysis results that meet the query conditions to obtain the query results;

Sending the query result to the user terminal.

In order to solve the above-mentioned problems, this application also provides a cloud monitoring device based on edge computing, the device including:

The receiving module is used to receive the media stream data packet sent by the camera equipment;

A buffer module, configured to parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to the preset buffer space;

An extraction module, which is used to read media frames to be processed from the media frame queue;

The analysis module is used to analyze and process each media frame to be processed using a pre-established analysis model to obtain feature information corresponding to each media frame to be processed, search for feature information that meets preset conditions in the feature information, and combine The media frame to be processed corresponding to the found feature information is used as the target media frame;

The sending module is configured to generate a monitoring analysis result based on the target media frame, and send the monitoring analysis result to a cloud device.

In order to solve the above-mentioned problems, the present application also provides an electronic device, which includes:

Memory, storing at least one instruction; and

The processor implements the following steps when executing instructions stored in the memory:

Receive media stream data packets sent by video recording equipment;

Read the media frame to be processed from the media frame queue;

Memory, storing at least one instruction; and

Sending the query result to the user terminal.

In order to solve the above-mentioned problem, the present application also provides a computer-readable storage medium having at least one instruction stored in the computer-readable storage medium, and when the at least one instruction is executed by a processor in an electronic device, the following steps are implemented:

Receive media stream data packets sent by video recording equipment;

Read the media frame to be processed from the media frame queue;

Sending the query result to the user terminal.

The embodiment of the application receives the media stream data packet sent by the camera, and then parses the media stream data packet according to the receiving order to obtain a preset type of media frame queue, and adds the media frame queue to the preset buffer space Then, read the media frame to be processed from the media frame queue, and use the pre-established analysis model to analyze and process each media frame to be processed to obtain the characteristic information corresponding to each media frame to be processed, and then obtain the In the feature information, the feature information that meets the preset conditions is searched, and the to-be-processed media frame corresponding to the found feature information is used as the target media frame. Finally, the monitoring analysis result is generated based on the target media frame, and the monitoring The analysis result is sent to the cloud device. Compared with the prior art, in this embodiment, the edge device analyzes the media stream data packet to obtain the monitoring analysis result, thereby improving the monitoring analysis efficiency. In addition, since the edge device generates the monitoring analysis result based on the target media frame only, The monitoring and analysis results are then sent to the cloud device for storage, so that the amount of data uploaded to the cloud device is smaller, the upload speed of the monitoring analysis results is improved, and the high bandwidth and storage space occupation is reduced.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, without creative work, other drawings can be obtained based on the structure shown in these drawings.

FIG. 1 is a schematic flowchart of a cloud monitoring method based on edge computing provided by an embodiment of this application;

2 is a schematic flowchart of a cloud monitoring method based on edge computing provided by another embodiment of this application;

3 is a schematic diagram of modules of a cloud monitoring device based on edge computing provided by an embodiment of the application;

4 is a schematic diagram of modules of a cloud monitoring device based on edge computing provided by another embodiment of this application;

FIG. 5 is a schematic diagram of the internal structure of an electronic device that implements a cloud monitoring method based on edge computing according to an embodiment of the application.

The realization, functional characteristics, and advantages of the purpose of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Detailed ways

The principles and features of the application are described below in conjunction with the accompanying drawings. The examples cited are only used to explain the application, and are not used to limit the scope of the application.

This application provides a cloud monitoring method based on edge computing. Referring to FIG. 1, it is a schematic flowchart of a cloud monitoring method based on edge computing provided by an embodiment of this application. This method can be applied to edge devices, and the edge devices can respectively communicate and connect with a video camera and a cloud device. It should be emphasized that the edge device can be any device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions, such as edge computing servers, desktop computers, notebooks, palms. Computers, mobile phones, smart watches, etc., this application is not limited. In addition, the method can also be executed by one electronic device or multiple electronic devices, and the electronic device can be implemented by software and/or hardware.

In this embodiment, the cloud monitoring method based on edge computing includes:

Step S11, receiving the media stream data packet sent by the camera recording device.

In detail, the video recording device collects media data in real time or at regular intervals to obtain multiple media frames, and the media frames are rendered, encoded, and encapsulated to generate corresponding media stream data packets. The so-called media stream, also known as streaming media, refers to media formats played on the Internet by means of streaming transmission, including audio streams, video streams, text streams, image streams, animation streams, etc.

The video recording device sends the media stream data packet to the edge device, and the edge device receives the media stream data packet sent by the video recording device.

Step S12: Parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to a preset buffer space.

The preset types of media frame queues include video frame queues and/or audio frame queues. For example, the media stream data packet is parsed according to the encapsulation format of the media stream data packet. When the media stream is a video stream that can play audio, the media stream data packet can be parsed to obtain an audio frame queue and a video frame queue. Only the video frame queue needs to be processed, and the video frame queue is added to the preset buffer space as a preset type of media frame queue.

Step S13: Read the media frame to be processed from the media frame queue.

In this embodiment, the media frame can be read from the media frame queue as the to-be-processed media frame according to the requirements of the specific application scenario and according to the preset sampling rule. For example, a media frame may be read every first preset number of media frames (for example, twenty media frames every interval) or every first preset duration (for example, 10 seconds) in the media frame queue. As a pending media frame.

Step S14: Use the pre-established analysis model to analyze and process each to-be-processed media frame to obtain feature information corresponding to each to-be-processed media frame, search for feature information that meets preset conditions in the feature information, and find The to-be-processed media frame corresponding to the feature information of is used as the target media frame.

In detail, the edge device uses the analysis model to analyze and process each media frame to be processed to obtain feature information corresponding to each media frame to be processed. Wherein, the analysis model includes one or more of a face recognition model, a voiceprint recognition model, a text recognition model, and an object recognition model. The corresponding analysis model type can be selected according to specific application scenarios. For example, if you want to recognize the number of people in the monitoring area, you can choose a face recognition model. If you want to recognize the number of vehicles in the monitoring area, you can choose an object recognition model. If you want to recognize both the number of people in the monitoring area and the vehicles Quantity, you can choose face recognition model and object recognition model.

In this embodiment, the analysis model may be sent from the cloud device to the edge device. For example, the cloud device receives a setting request sent by the user terminal, the setting request includes application service type information, and the cloud device determines the corresponding application service type information according to the pre-established application service type information and the mapping relationship between the analysis model. Analyze the model and send the analysis model to the edge device, and the edge device receives and saves the analysis model. It should be emphasized that in other application examples, the analysis model can also be imported into the edge device by any other applicable device, which is not limited in this application.

The following takes the object recognition model as an example to illustrate how the analysis model analyzes and processes the media frames to be processed to obtain characteristic information. To identify the number of a certain target object, the type of the acquired media frame to be processed is a video frame (ie, a frame of image). First, the media frame to be processed is segmented to obtain several characteristic images. Wherein, the segmentation processing methods include segmentation methods based on edge monitoring, segmentation methods based on region growth, segmentation methods based on neural networks, segmentation methods based on thresholds, etc. The segmentation method can be selected according to needs. Not limited. Then, each feature image is analyzed for similarity with a plurality of preset sample images. Among them, the similarity between the feature image and the preset sample image can be calculated based on parameters such as contour, color, and texture. The preset sample image is a sample image of the target object. Then, whenever the similarity between a feature image and any preset sample image is greater than or equal to a preset similarity threshold, the feature image is recognized as a target object. Finally, count the number of feature images recognized as the target object, and output the counted number as feature information.

For the above-mentioned feature information, when the specific application scenarios are different, the above-mentioned feature information may also be different. For example, to identify the number of people and the number of vehicles in the monitoring area, the output feature information includes the number of people and the number of vehicles. To identify whether there is a target person in the monitoring area, the output characteristic information includes the identified person information (for example, the name of the person).

After obtaining the feature information, the edge device searches the obtained feature information for feature information that meets preset conditions. Among them, the preset conditions can be set according to specific application scenarios. For example, if you want to identify the number of people and the number of vehicles in the monitoring area, and you need to warn you when the number of people is greater than 4 or the number of vehicles is equal to 5, you can set the preset conditions Set the number of people greater than 4 or the number of vehicles equal to 5.

When the edge device finds feature information that meets the preset condition, it uses the to-be-processed media frame corresponding to the found feature information as the target media frame.

Step S15: Generate a monitoring analysis result based on the target media frame, and send the monitoring analysis result to a cloud device.

In this embodiment, there may be multiple ways to generate the monitoring analysis result based on the target media frame. For example, when the target media frame is a video frame, the following way may be used to generate the monitoring analysis result.

Manner 1: Save the target media frame as a screenshot, and generate the monitoring analysis result based on the screenshot. Regarding the first method, in some application scenarios, the target media frame also carries time information (for example, a timestamp), and the monitoring analysis result can be generated based on the screenshot and time information.

Manner 2: Intercept the media frame sub-queue containing the target media frame from the media frame queue, and generate the monitoring analysis result based on the media frame sub-queue. For example, obtaining a second preset number of video frames arranged in front of the target media frame in a media frame queue, and obtaining a second preset number of video frames arranged after the target media frame, that is, obtaining The media frame sub-queue formed by the video frame and the target media frame. Or, obtain a second preset number of video frames arranged in front of the target media frame in the media frame queue, and obtain a second preset number of video frames arranged after the target media frame, that is, obtain The media frame sub-queue formed by the video frame and the target media frame. Based on the media frame sub-queue, the monitoring analysis result is generated. In some application scenarios, a video file with a second preset duration (for example, 20 seconds) may be generated based on the media frame sub-queue, and then the video file may be used as the monitoring analysis result.

In this embodiment, the media stream data packet sent by the camera is received, and the media stream data packet is parsed according to the receiving order to obtain a preset type of media frame queue, and the media frame queue is added to the preset buffer space. Then, read the media frames to be processed from the media frame queue, and use the pre-established analysis model to analyze and process each media frame to be processed to obtain the feature information corresponding to each media frame to be processed, and then use the obtained feature Find feature information that meets preset conditions in the information. When found, use the to-be-processed media frame corresponding to the found feature information as the target media frame. Finally, generate monitoring and analysis results based on the target media frame, and The monitoring analysis result is sent to the cloud device. Compared with the prior art, in this embodiment, the edge device analyzes the media stream data packet to obtain the monitoring analysis result, thereby improving the monitoring analysis efficiency. In addition, since the edge device generates the monitoring analysis result based only on the target media frame, The monitoring and analysis results are then sent to the cloud device for storage, so that the amount of data uploaded to the cloud device is smaller, the upload speed of the monitoring analysis results is improved, and the high bandwidth and storage space occupation is reduced.

Further, before the step S11, the method further includes:

Send a registration request carrying device identification information to the cloud device for the cloud device to perform a registration operation based on the device identification information; receive device operating parameters and model inference parameters sent by the cloud device after completing the registration operation And an analysis model; configure operating parameters based on the equipment operating parameters; save the analysis model, and generate and save the preset conditions for finding the characteristic information based on the model inference parameters.

Among them, the above-mentioned device identification information includes product model, device name, and device password.

The device operating parameters include video streaming address, video streaming address, video frame rate, video scanning interval time (corresponding to the above-mentioned first preset duration), monitoring analysis result report type (for example, screenshot, short video), short video Interception duration (corresponding to the second preset duration mentioned above), etc. According to specific application scenarios, corresponding device operating parameters can be set, which is not limited in this application.

In this embodiment, the user can send a setting request to the cloud device through the user terminal to realize the setting of device operating parameters and model inference parameters, so as to meet the diverse needs of users.

Referring to FIG. 2, it is a schematic flowchart of a cloud monitoring method based on edge computing provided by another embodiment of this application. This method can be applied to cloud devices, and the cloud devices can communicate with edge devices and user terminals respectively. It should be emphasized that the method can be executed by one electronic device or multiple electronic devices, and the electronic device can be implemented by software and/or hardware.

Step S21, receiving and storing the monitoring analysis result sent by the edge device.

The monitoring analysis result is obtained by the edge device analyzing the media stream data packet sent by the camera recording device. For example, the edge device receives the media stream data packet sent by the camera recording device, parses the media stream data packet according to the receiving order, obtains a preset type of media frame queue, adds the media frame queue to the preset buffer space, and then Read the to-be-processed media frames from the media frame queue, and use the pre-established analysis model to analyze and process each to-be-processed media frame to obtain the feature information corresponding to each to-be-processed media frame, and in the obtained feature information The feature information that meets the preset condition is searched, and when the feature information is found, the to-be-processed media frame corresponding to the found feature information is taken as the target media frame, and finally, a monitoring analysis result is generated based on the target media frame. For the specific description of the analysis method, please refer to the content of the previous embodiment, which will not be repeated here.

Step S22, in response to the query request carrying the query condition sent by the user terminal, query the stored monitoring analysis results for the monitoring analysis result that meets the query condition to obtain the query result.

In detail, the user can select the query condition as needed. For example, if you choose to view the monitoring analysis result in a certain time interval, the query condition is set to a specific time interval. When the cloud device receives a query request carrying query conditions sent by the user terminal, it queries the stored monitoring analysis results for the monitoring analysis results that meet the query conditions to obtain the query results.

Step S23: Send the query result to the user terminal.

This embodiment receives and stores the monitoring analysis results sent by the edge device, and then responds to the query request carrying the query conditions sent by the user terminal, and queries the stored monitoring analysis results for the monitoring analysis results that meet the query conditions to obtain the query results And finally, sending the query result to the user terminal. Compared with the prior art, this embodiment analyzes the media stream data packet through the edge device to obtain the monitoring analysis result, thereby improving the monitoring analysis efficiency. In addition, the user can query the monitoring analysis result stored in the cloud device through the user terminal , Improve the efficiency of obtaining monitoring and analysis results, and make user operations more convenient.

Further, before step S21, the method further includes:

Receive a registration request sent by the edge device, where the registration request includes device identification information; perform a registration operation based on the device identification information; after completing the registration operation, combine predetermined device operating parameters, model inference parameters, and analysis models Send to the edge device.

In detail, the cloud device receives and responds to a registration request sent by an edge device. The registration request includes device identification information, and the device identification information is used to uniquely identify the edge device, including product model, device name, and device password. The cloud device searches for the received device identification information in the pre-established device information database (the device information database includes the device identification information of all legal devices), if found, the edge device is considered to be a legal device, and a message indicating successful registration is returned Go to the edge device, and mark the state of the edge device as the active state. If it is not found, the edge device is considered to be an illegal device, and a registration failure message is returned to the edge device.

After completing the registration operation, the user can set equipment operating parameters, model reasoning parameters, and select an analysis model through the user terminal. For example, the cloud device receives a setting request sent by the user terminal, the setting request includes application service type information, and the cloud device determines the corresponding application service type information according to the pre-established application service type information and the mapping relationship between the analysis model. Analyze the model and send the analysis model to the edge device, and the edge device receives and saves the analysis model. In another example, the cloud device receives a setting request sent by the user terminal, where the setting request includes device operating parameters and/or model inference parameters, and stores the device operating parameters and/or model inference parameters in a preset storage space. After completing the settings, the cloud device sends the device operating parameters, model inference parameters, and analysis model to the edge device. The edge device configures operating parameters based on the device operating parameters, saves the analysis model, and generates and saves the preset conditions for searching the characteristic information based on the model inference parameters.

Wherein, the device operating parameters include video streaming address, video streaming address, video frame rate, video scanning interval time (corresponding to the above-mentioned first preset duration), monitoring analysis result reporting type (for example, screenshot, short video), Short video interception duration (corresponding to the second preset duration mentioned above), etc. According to specific application scenarios, corresponding device operating parameters can be set, which is not limited in this application.

The analysis model includes one or more of a face recognition model, a voiceprint recognition model, a text recognition model, and an object recognition model. The corresponding analysis model type can be selected according to specific application scenarios. For example, if you want to recognize the number of people in the monitoring area, you can choose a face recognition model. If you want to recognize the number of vehicles in the monitoring area, you can choose an object recognition model. If you want to recognize both the number of people in the monitoring area and the vehicles Quantity, you can choose face recognition model and object recognition model.

In this embodiment, the user can send a setting request to the cloud device through the user terminal to realize the setting of device operating parameters, model reasoning parameters, and selection of analysis models, so as to meet the diverse needs of users.

Further, in this embodiment, the method further includes:

Receiving a model training request sent by a user terminal, where the model training request includes a first preset training data set;

An analysis model trained in advance based on a second preset training data set is acquired, and the analysis model is trained using the first preset training set to obtain a new analysis model.

For example, if a user wants to train an object recognition model, he can set a sample set through the user terminal. The sample set includes multiple sample pictures, and then annotate each sample picture to obtain the label data. Then, the sample set and The annotation data is sent to the cloud device as the first preset training set and the cloud device is requested to perform model training. The cloud device obtains an analysis model (for example, a basic item recognition model) trained in advance based on a second preset training data set from a preset storage space, and uses the first preset training set to perform a secondary operation on the analysis model Training to generate a new analysis model (ie a customized analysis model).

This embodiment can implement custom analysis model training according to user needs, can meet the diverse needs of users, save model development costs for users, and improve model training efficiency.

Further, in this embodiment, after the new analysis model is obtained, the method further includes:

In response to a model update request sent by the user terminal, based on the new analysis model, generate model update data, and send the model update data to the edge device for the edge device to update the data according to the model and perform analysis Model update operation.

In this embodiment, batch analysis model updates can also be performed on multiple edge devices. For example, the model update request includes device identification information of multiple edge devices, and the cloud device generates model updates based on the new analysis model. Data, and send the model update data to the edge device corresponding to each device identification information.

In this embodiment, the analysis model of the edge device can be remotely updated.

As shown in FIG. 3, it is a schematic diagram of modules of a cloud monitoring device based on edge computing provided by an embodiment of the present application.

The cloud monitoring device 100 based on edge computing described in this application can be installed in an electronic device. According to the implemented functions, the cloud monitoring device based on edge computing may include a receiving module 101, a cache module 102, an extraction module 103, an analysis module 104, and a sending module 105. The module described in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The receiving module 101 is configured to receive a media stream data packet sent by a camera recording device.

The camera recording device sends the media stream data packet to the receiving module 101, and the receiving module 101 receives the media stream data packet sent by the camera recording device.

The buffer module 102 is configured to parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to the preset buffer space.

The preset type of media frame queue includes a video frame queue and/or an audio frame queue. For example, the media stream data packet is parsed according to the encapsulation format of the media stream data packet. When the media stream is a video stream that can play audio, the media stream data packet can be parsed to obtain an audio frame queue and a video frame queue. Only the video frame queue needs to be processed, and the video frame queue is added to the preset buffer space as a preset type of media frame queue.

The extraction module 103 is configured to read the media frame to be processed from the media frame queue.

The analysis module 104 is configured to analyze and process each media frame to be processed using a pre-established analysis model to obtain feature information corresponding to each media frame to be processed, find feature information that meets preset conditions in the feature information, and Use the to-be-processed media frame corresponding to the found feature information as the target media frame.

In detail, the analysis module 104 uses the analysis model to analyze and process each media frame to be processed to obtain feature information corresponding to each media frame to be processed. Wherein, the analysis model includes one or more of a face recognition model, a voiceprint recognition model, a text recognition model, and an object recognition model. The corresponding analysis model type can be selected according to specific application scenarios. For example, if you want to recognize the number of people in the monitoring area, you can choose a face recognition model. If you want to recognize the number of vehicles in the monitoring area, you can choose an object recognition model. If you want to recognize both the number of people in the monitoring area and the vehicles Quantity, you can choose face recognition model and object recognition model. In this embodiment, the analysis model may be sent to the analysis module 104 by the cloud device. For example, the cloud device receives a setting request sent by the user terminal, the setting request includes application service type information, and the cloud device determines which application service type information corresponds to the application service type information and the mapping relationship between the analysis model established in advance. The analysis model is analyzed, and the analysis model is sent to the analysis module 104, and the analysis module 104 receives and saves the analysis model. It should be emphasized that in other application examples, the analysis model can also be imported into the analysis module 104 by any other applicable equipment, which is not limited in this application.

After obtaining the characteristic information, the analysis module 104 searches the obtained characteristic information for characteristic information that meets a preset condition. Among them, the preset conditions can be set according to specific application scenarios. For example, if you want to identify the number of people and the number of vehicles in the monitoring area, and you need to warn you when the number of people is greater than 4 or the number of vehicles is equal to 5, you can set the preset conditions Set the number of people greater than 4 or the number of vehicles equal to 5.

When the analysis module 104 finds feature information that meets the preset condition, it uses the to-be-processed media frame corresponding to the found feature information as the target media frame.

The sending module 105 is configured to generate a monitoring analysis result based on the target media frame, and send the monitoring analysis result to a cloud device.

In this embodiment, the media stream data packet sent by the camera and recording device is received, and the media stream data packet is parsed according to the receiving order to obtain a media frame queue of a preset type, and the media frame queue is added to the preset buffer space. Next, read the media frames to be processed from the media frame queue, and use the pre-established analysis model to analyze and process each media frame to be processed to obtain the feature information corresponding to each media frame to be processed, and then use the obtained feature Find feature information that meets preset conditions in the information, and when found, use the to-be-processed media frame corresponding to the found feature information as the target media frame, and finally, generate a monitoring analysis result based on the target media frame, and The monitoring analysis result is sent to the cloud device. Compared with the prior art, this embodiment obtains the monitoring analysis result by analyzing the media stream data packet, thereby improving the monitoring analysis efficiency. In addition, since the monitoring analysis result is generated only based on the target media frame, the monitoring The analysis result is sent to the cloud device for storage, so that the amount of data uploaded to the cloud device is smaller, the upload speed of the monitoring analysis result is improved, and the high bandwidth and storage space occupation is reduced.

Further, the device further includes: a registration module and a setting module (not shown in the figure), wherein:

The registration module is configured to send a registration request carrying device identification information to the cloud device for the cloud device to perform a registration operation based on the device identification information;

The setting module is used to receive the device operating parameters, model inference parameters, and analysis models sent by the cloud device after the registration operation is completed; perform operating parameter configuration based on the device operating parameters; save the analysis model, and based on all The model reasoning parameter generates and saves the preset conditions for searching the characteristic information.

As shown in FIG. 4, it is a schematic diagram of modules of a cloud monitoring device based on edge computing provided by another embodiment of the present application.

The cloud monitoring apparatus 200 based on edge computing described in this application may be installed in electronic equipment. According to the implemented functions, the cloud monitoring device based on edge computing may include a storage module 201, a query module 202, and a feedback module 203. The module described in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The storage module 201 is used to receive and store the monitoring and analysis results sent by the edge device.

The monitoring analysis result is obtained by the edge device analyzing the media stream data packet sent by the camera recording device. For example, the edge device receives the media stream data packet sent by the camera recording device, parses the media stream data packet according to the receiving order, obtains a preset type of media frame queue, adds the media frame queue to the preset buffer space, and then Read the to-be-processed media frames from the media frame queue, and use the pre-established analysis model to analyze and process each to-be-processed media frame to obtain the feature information corresponding to each to-be-processed media frame, and in the obtained feature information The feature information that meets the preset condition is searched, and when the feature information is found, the to-be-processed media frame corresponding to the found feature information is taken as the target media frame, and finally, a monitoring analysis result is generated based on the target media frame. For the specific description of the analysis method, reference may be made to the content of the foregoing embodiment, which will not be repeated here. In addition, it should be emphasized that, in order to further ensure the privacy and security of the above-mentioned monitoring and analysis results, the above-mentioned monitoring and analysis results can also be stored in a node of a blockchain.

The query module 202 is configured to respond to a query request carrying query conditions sent by the user terminal, query the stored monitoring analysis results from the monitoring analysis results that satisfy the query conditions, and obtain the query results.

In detail, the user can select the query condition as needed. For example, if you choose to view the monitoring analysis result in a certain time interval, the query condition is set to a specific time interval. When the query module 202 receives a query request carrying query conditions sent by the user terminal, it queries the stored monitoring analysis results for the monitoring analysis results that meet the query conditions to obtain the query results.

The feedback module 203 is configured to send the query result to the user terminal.

Further, the device also includes an activation module (not shown in the figure) for:

In detail, the activation module receives and responds to a registration request sent by an edge device, the registration request includes device identification information, and the device identification information is used to uniquely identify the edge device, including product model, device name, and device password. The activation module searches for the received device identification information in the pre-established device information database (the device information database includes the device identification information of all legal devices), if found, the edge device is considered to be a legal device, and a message indicating successful registration is returned Go to the edge device, and mark the state of the edge device as the active state. If it is not found, the edge device is considered to be an illegal device, and a registration failure message is returned to the edge device.

After completing the registration operation, the user can set equipment operating parameters, model reasoning parameters, and select an analysis model through the user terminal. For example, the activation module receives a setting request sent by the user terminal, the setting request includes application service type information, and the cloud device determines the corresponding application service type information according to the pre-established application service type information and the mapping relationship between the analysis model. Analyze the model and send the analysis model to the edge device, and the edge device receives and saves the analysis model. For another example, the activation module receives a setting request sent by the user terminal, where the setting request includes equipment operating parameters and/or model inference parameters, and stores the equipment operating parameters and/or model inference parameters in a preset storage space. After completing the settings, the activation module sends the device operating parameters, model inference parameters, and analysis model to the edge device. The edge device configures operating parameters based on the device operating parameters, saves the analysis model, and generates and saves the preset conditions for searching the characteristic information based on the model inference parameters.

Further, in this embodiment, the device further includes a model training module (not shown in the figure) for:

For example, if a user wants to train an object recognition model, he can set a sample set through the user terminal. The sample set includes multiple sample pictures, and then annotate each sample picture to obtain the label data. Then, the sample set and The labeled data is sent to the model training module as the first preset training set and the model training module is requested to perform model training. The model training module obtains an analysis model (for example, a basic item recognition model) trained in advance based on a second preset training data set from a preset storage space, and uses the first preset training set to perform two operations on the analysis model. After training, a new analysis model (ie a customized analysis model) is generated.

Further, in this embodiment, the device further includes an update module (not shown in the figure) for:

In this embodiment, batch analysis model updates can also be performed on multiple edge devices. For example, the model update request includes device identification information of multiple edge devices, and the update module generates model updates based on the new analysis model. Data, and send the model update data to the edge device corresponding to each device identification information.

As shown in FIG. 4, it is a schematic diagram of the internal structure of an electronic device that implements a cloud monitoring method based on edge computing provided by an embodiment of the present application.

The electronic device 1 may include a processor 10, a memory 11, and a bus, and may also include a computer program stored in the memory 11 and running on the processor 10, such as a cloud monitoring program 12 based on edge computing.

Wherein, the memory 11 includes at least one type of computer-readable storage medium, and the computer-readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, Disks, CDs, etc.

The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, for example, a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart media card (SMC), and a secure digital (Secure Digital) equipped on the electronic device 1. , SD) card, flash card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can be used not only to store application software and various data installed in the electronic device 1, such as the code of a cloud monitoring program based on edge computing, etc., but also to temporarily store data that has been output or will be output.

The processor 10 may be composed of integrated circuits in some embodiments, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions, including one or more Combinations of central processing unit (CPU), microprocessor, digital processing chip, graphics processor, and various control chips, etc. The processor 10 is the control unit of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and runs or executes programs or modules stored in the memory 11 (for example, based on Cloud monitoring programs for edge computing, etc.), and call data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

The bus may be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to implement connection and communication between the memory 11 and at least one processor 10 and the like.

FIG. 4 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure. Components, or combinations of certain components, or different component arrangements.

For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, thereby controlling power The device implements functions such as charge management, discharge management, and power consumption management. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used in the electronic device 1 Establish a communication connection with other electronic devices.

Optionally, the electronic device 1 may also include a user interface. The user interface may be a display (Display) and an input unit (such as a keyboard (Keyboard)). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display the information processed in the electronic device 1 and to display a visualized user interface.

It should be understood that the embodiments are only for illustrative purposes, and are not limited by this structure in the scope of the patent application.

The edge computing-based cloud monitoring program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can realize:

Receive media stream data packets sent by video recording equipment;

Read the media frame to be processed from the media frame queue;

Alternatively, the edge computing-based cloud monitoring program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, and when running in the processor 10, it can realize:

Sending the query result to the user terminal.

Specifically, for the specific implementation method of the above-mentioned instructions by the processor 10, reference may be made to the description of the relevant steps in the embodiment corresponding to FIG. 1, which will not be repeated here. It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned monitoring and analysis results, the above-mentioned monitoring and analysis results can also be stored in a node of a blockchain.

Further, if the integrated module/unit of the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium may be volatile or non-volatile. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile Hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory).

The specific implementation of the computer-readable storage medium of the present application is basically the same as the foregoing embodiments of the cloud monitoring method based on edge computing, and will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed equipment, device, and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of hardware plus software functional modules.

For those skilled in the art, it is obvious that the present application is not limited to the details of the foregoing exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application.

Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of this application is defined by the appended claims rather than the above description, and therefore it is intended to fall into the claims. All changes in the meaning and scope of the equivalent elements of are included in this application. Any reference signs in the claims should not be regarded as limiting the claims involved.

The blockchain referred to in this application is a new application mode of computer technology such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information for verification. The validity of the information (anti-counterfeiting) and the generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer. In addition, it is obvious that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. The second class words are used to indicate names, and do not indicate any specific order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application and not to limit them. Although the application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the application can be Make modifications or equivalent replacements without departing from the spirit and scope of the technical solution of the present application.

Claims

A cloud monitoring method based on edge computing is applied to edge devices, wherein the method includes:

Receive media stream data packets sent by video recording equipment;

Parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to the preset buffer space;

Read the media frame to be processed from the media frame queue;

Use the pre-established analysis model to analyze and process each media frame to be processed to obtain feature information corresponding to each media frame to be processed, search for feature information that meets preset conditions in the feature information, and retrieve the feature information The corresponding to-be-processed media frame is used as the target media frame;

A monitoring analysis result is generated based on the target media frame, and the monitoring analysis result is sent to a cloud device.
The cloud monitoring method based on edge computing according to claim 1, wherein before said receiving the media stream data packet sent by the camera recording device, the method further comprises:

Sending a registration request carrying device identification information to the cloud device for the cloud device to perform a registration operation based on the device identification information;

Receiving device operating parameters, model inference parameters, and analysis models sent by the cloud device after completing the registration operation;

Perform operating parameter configuration based on the equipment operating parameters;

The analysis model is saved, and the preset conditions for searching the characteristic information are generated and saved based on the model inference parameters.
The cloud monitoring method based on edge computing according to claim 1, wherein, when the target media frame is a video frame, generating a monitoring analysis result based on the target media frame comprises:

Saving the target media frame as a screenshot, and generating the monitoring analysis result based on the screenshot;

Alternatively, the media frame sub-queue containing the target media frame is intercepted from the media frame queue, and the monitoring analysis result is generated based on the media frame sub-queue.
The edge computing-based cloud monitoring method according to any one of claims 1 to 3, wherein the pre-established analysis model includes a face recognition model, a voiceprint recognition model, a text recognition model, and an object recognition model. One or more.
A cloud monitoring method based on edge computing is applied to cloud devices, wherein the method includes:

Receiving and storing the monitoring analysis result sent by the edge device, where the monitoring analysis result is obtained by the edge device analyzing the media stream data packet sent by the camera recording device;

In response to a query request carrying query conditions sent by the user terminal, query the stored monitoring analysis results for monitoring analysis results that meet the query conditions to obtain the query results;

Sending the query result to the user terminal.
The cloud monitoring method based on edge computing according to claim 5, wherein, before the receiving and storing the monitoring analysis result sent by the edge device, the method comprises:

Receiving a registration request sent by the edge device, where the registration request includes device identification information;

Perform a registration operation based on the device identification information;

After the registration operation is completed, the predetermined device operating parameters, model inference parameters, and analysis model are sent to the edge device.
The cloud monitoring method based on edge computing according to claim 5, wherein the method further comprises:

Receiving a model training request sent by a user terminal, where the model training request includes a first preset training data set;

An analysis model trained in advance based on a second preset training data set is acquired, and the analysis model is trained using the first preset training set to obtain a new analysis model.
A cloud monitoring device based on edge computing, wherein the device includes:

The receiving module is used to receive the media stream data packet sent by the camera equipment;

A buffer module, configured to parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to the preset buffer space;

An extraction module, which is used to read media frames to be processed from the media frame queue;

The analysis module is used to analyze and process each media frame to be processed using a pre-established analysis model to obtain feature information corresponding to each media frame to be processed, search for feature information that meets preset conditions in the feature information, and combine The media frame to be processed corresponding to the found feature information is used as the target media frame;

The sending module is configured to generate a monitoring analysis result based on the target media frame, and send the monitoring analysis result to a cloud device.
An electronic device, wherein the electronic device includes:

At least one processor; and,

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the following steps are implemented:

Receive media stream data packets sent by video recording equipment;

Parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to the preset buffer space;

Read the media frame to be processed from the media frame queue;

Use the pre-established analysis model to analyze and process each media frame to be processed to obtain feature information corresponding to each media frame to be processed, search for feature information that meets preset conditions in the feature information, and retrieve the feature information The corresponding to-be-processed media frame is used as the target media frame;

A monitoring analysis result is generated based on the target media frame, and the monitoring analysis result is sent to a cloud device.
9. The electronic device according to claim 9, wherein, before the receiving the media stream data packet sent by the camera recording device, when the instruction is executed by the at least one processor, the following steps are further implemented:

Sending a registration request carrying device identification information to the cloud device for the cloud device to perform a registration operation based on the device identification information;

Receiving device operating parameters, model inference parameters, and analysis models sent by the cloud device after completing the registration operation;

Perform operating parameter configuration based on the equipment operating parameters;

The analysis model is saved, and the preset conditions for searching the characteristic information are generated and saved based on the model inference parameters.
9. The electronic device according to claim 9, wherein when the target media frame is a video frame, generating a monitoring analysis result based on the target media frame comprises:

Saving the target media frame as a screenshot, and generating the monitoring analysis result based on the screenshot;

Alternatively, the media frame sub-queue containing the target media frame is intercepted from the media frame queue, and the monitoring analysis result is generated based on the media frame sub-queue.
The electronic device according to any one of claims 9 to 11, wherein the pre-established analysis model includes one or more of a face recognition model, a voiceprint recognition model, a text recognition model, and an object recognition model .
An electronic device, wherein the electronic device includes:

At least one processor; and,

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the following steps are implemented:

Receiving and storing the monitoring analysis result sent by the edge device, where the monitoring analysis result is obtained by the edge device analyzing the media stream data packet sent by the camera recording device;

In response to a query request carrying query conditions sent by the user terminal, query the stored monitoring analysis results for monitoring analysis results that meet the query conditions to obtain the query results;

Sending the query result to the user terminal.
The electronic device according to claim 13, wherein, before the receiving and storing the monitoring analysis result sent by the edge device, when the instruction is executed by the at least one processor, the following steps are further implemented:

Receiving a registration request sent by the edge device, where the registration request includes device identification information;

Perform a registration operation based on the device identification information;

After the registration operation is completed, the predetermined device operating parameters, model inference parameters, and analysis model are sent to the edge device.
The electronic device according to claim 13, wherein, when the instruction is executed by the at least one processor, the following steps are further implemented:

Receiving a model training request sent by a user terminal, where the model training request includes a first preset training data set;

An analysis model trained in advance based on a second preset training data set is acquired, and the analysis model is trained using the first preset training set to obtain a new analysis model.
A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the following steps:

Receive media stream data packets sent by video recording equipment;

Parse the media stream data packets according to the receiving order to obtain a preset type of media frame queue, and add the media frame queue to the preset buffer space;

Read the media frame to be processed from the media frame queue;

Use the pre-established analysis model to analyze and process each media frame to be processed to obtain feature information corresponding to each media frame to be processed, search for feature information that meets preset conditions in the feature information, and retrieve the feature information The corresponding to-be-processed media frame is used as the target media frame;

A monitoring analysis result is generated based on the target media frame, and the monitoring analysis result is sent to a cloud device.
16. The computer-readable storage medium according to claim 16, wherein, before the receiving the media stream data packet sent by the camera recording device, the following steps are further implemented when the computer program is executed by the processor:

Sending a registration request carrying device identification information to the cloud device for the cloud device to perform a registration operation based on the device identification information;

Receiving device operating parameters, model inference parameters, and analysis models sent by the cloud device after completing the registration operation;

Perform operating parameter configuration based on the equipment operating parameters;

The analysis model is saved, and the preset conditions for searching the characteristic information are generated and saved based on the model inference parameters.
15. The computer-readable storage medium of claim 16, wherein when the target media frame is a video frame, the generating a monitoring analysis result based on the target media frame comprises:

Saving the target media frame as a screenshot, and generating the monitoring analysis result based on the screenshot;

Alternatively, the media frame sub-queue containing the target media frame is intercepted from the media frame queue, and the monitoring analysis result is generated based on the media frame sub-queue.
The computer-readable storage medium according to any one of claims 16 to 18, wherein the pre-established analysis model includes one of a face recognition model, a voiceprint recognition model, a text recognition model, and an object recognition model Or multiple.
A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the following steps:

Receiving and storing the monitoring analysis result sent by the edge device, where the monitoring analysis result is obtained by the edge device analyzing the media stream data packet sent by the camera recording device;

In response to a query request carrying query conditions sent by the user terminal, query the stored monitoring analysis results for monitoring analysis results that meet the query conditions to obtain the query results;

Sending the query result to the user terminal.