CN116597337A

CN116597337A - Method, system, equipment and storage medium for monitoring operation behaviors

Info

Publication number: CN116597337A
Application number: CN202310348511.1A
Authority: CN
Inventors: 王明杰; 高晓国; 陈建华; 易力; 秦钢; 施杰; 陈昌杰; 续杰; 王键; 张猛; 赵宇; 顾永兴
Original assignee: Ouye Yunshang Co ltd
Current assignee: Ouye Yunshang Co ltd
Priority date: 2023-03-31
Filing date: 2023-03-31
Publication date: 2023-08-15

Abstract

The application provides a monitoring method, a monitoring system, monitoring equipment and a storage medium for operation behaviors, and relates to the technical field of intelligent warehouse management. A method of monitoring job behavior, comprising: acquiring video image data of an operation area, wherein the video image data comprises operation behaviors of operators; generating a job analysis result of the operator according to the video image data; and monitoring the operation behaviors of the operators through the operation analysis results. According to the embodiment of the application, the local video stream of the operation area can be collected and analyzed, and the operation area can be linked with the operation area in real time on site, so that the behaviors of operators in the operation area are standardized.

Description

Method, system, equipment and storage medium for monitoring operation behaviors

Technical Field

The application relates to the technical field of warehouse intelligent management, in particular to a monitoring method, a monitoring system, a monitoring device and a storage medium for operation behaviors.

Background

At present, an automatic monitoring and intelligent control system based on machine vision is gradually popularized and applied to various fields of industrial production, and the key points extend to front-end manufacturing and rear-end logistics links.

Along with the continuous development of business and the expansion of the warehouse area under jurisdiction, a plurality of manufacturing enterprises in the industry adopt a method for improving the production safety in the whole logistics warehouse process and the intelligent measurement and correction in the material hoisting process by video AI identification analysis so as to realize the demands of the manual check mode on the work such as the safety of the whole logistics process.

The inventor of the application discovers that the common AI visual analysis needs to concentrate video stream data on a visual analysis platform, but because a plurality of warehouses in a service scene are distributed all over the country, a large amount of public network bandwidth is occupied by the video gathering to the cloud for visual analysis, and higher flow cost is consumed. In addition, stability is poor when video streaming is carried out between the warehouse and the visual analysis platform through the public network, and the condition of video frame loss often occurs, so that the detection result of visual analysis is affected.

Disclosure of Invention

According to an aspect of the present application, there is provided a method for monitoring operation behavior, including: acquiring video image data of an operation area, wherein the video image data comprises operation behaviors of operators; generating a job analysis result of the operator according to the video image data; and monitoring the operation behaviors of the operators through the operation analysis results.

According to some embodiments, acquiring video image data of a work area includes: acquiring video stream data from video equipment in the operation area; decoding the video stream data to obtain a plurality of video frames; the video image data is generated by format conversion of the plurality of video frames.

According to some embodiments, generating a job analysis result of the worker from the video image data includes: inputting the video image data into an algorithm model; and obtaining the operation analysis result through the algorithm model.

According to some embodiments, obtaining the job analysis result by the algorithm model includes: deducing the video image data through a preset function type in the algorithm model; and carrying out logic analysis according to the inferred result to output the operation analysis result.

According to some embodiments, performing a logic analysis based on the inferred results to output the job analysis results includes: determining whether the operator is present in the video image data; if the video image data exists, determining the operation behavior of the operator in the video image data; outputting the operation analysis result including the operation behavior of the operator.

According to some embodiments, monitoring the job behavior of the job personnel by the job analysis results includes: determining whether the operation behavior of the operator is unsafe or not according to the operation analysis result; if yes, sending out warning information to the operator.

According to an aspect of the present application, there is provided a monitoring system for operation behavior, including: the data acquisition module is used for acquiring video stream data in the operation area; decoding the video stream data; obtaining video image data by performing format conversion on a plurality of video frames obtained after decoding; the data processing module analyzes the video image data through an algorithm model to obtain an operation analysis result; the communication module is used for sending the operation analysis result to the vision processing module; the visual processing module receives and displays the operation analysis result; and sending an alarm instruction according to the operation analysis result.

According to some embodiments, the system further comprises: the warning module receives the warning instruction; and sending out warning information to operators in the operation area according to the warning instruction.

According to an aspect of the present application, there is provided an electronic apparatus including: one or more processors; a storage means for storing one or more programs; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the methods as described above.

According to an aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method as described above.

According to the embodiment of the application, the local video stream of the operation area can be collected and analyzed, the transmission bandwidth of the public network of the video stream is reduced, the stability of the video stream image is ensured, the video stream can be linked with the operation area in real time on site, the behaviors of operators in the operation area are standardized, and the safety supervision and guarantee capability in the service operation process is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application.

Fig. 1 shows a flow chart of a method of monitoring job behavior according to an example embodiment of the present application.

Fig. 2 shows a schematic diagram of a monitoring system of job behavior according to an example embodiment of the present application.

Fig. 3 shows a schematic diagram of acquiring video image data according to an exemplary embodiment of the present application.

Fig. 4 shows a flowchart of generating a job analysis result according to an example embodiment of the present application.

Fig. 5 shows a block diagram of an electronic device according to an example embodiment of the application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted.

The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, devices, operations, etc. In these instances, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

The terms first, second and the like in the description and in the claims and in the above-described figures are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.

The application provides a monitoring method, a system, equipment and a storage medium for operation behaviors, which reduce the requirement on video stream transmission bandwidth, ensure the stability of video stream images and monitor the behaviors of operators in real time through the acquisition and analysis of video streams in an operation area.

A method, system, apparatus and storage medium for monitoring operation behavior according to an embodiment of the present application will be described in detail with reference to the accompanying drawings.

As shown in fig. 1, in step S110, the monitoring system acquires video image data of a work area.

For example, in step S110, the monitoring system performs network connection with the video device in the operation area through preset video device information, and acquires video stream data in the operation area from the video device through the network.

According to some embodiments, the video stream data within the work area includes a work behavior of a worker within the work area.

The monitoring system decodes the acquired video stream data through the decoder to obtain a plurality of video frames. The monitoring system performs format conversion on the decoded video frames to generate video image data which can be processed by a preset algorithm model.

According to some embodiments, the monitoring system may set the resolution of the video frames to 720P or 1080P according to the requirements of the preset algorithm model, and set the number of video frames input to the algorithm model to between 1-25 frames per second.

In step S120, the monitoring system generates a job analysis result of the worker from the video image data.

For example, in step S120, the monitoring system inputs the video image data that has been format-converted into an algorithm model.

According to some embodiments, an algorithm model may be preset in the monitoring system, and the algorithm model may be adjusted or updated through the cloud according to actual requirements.

The monitoring system deduces the content corresponding to the preset function type in the video image data through the preset function type in the algorithm model.

For example, the preset function type in the algorithm model is detection of the wearing condition of the safety helmet, the monitoring system uses the function type as a label to infer the content related to the wearing of the safety helmet in the video image data, and an inference result is obtained, such as whether the safety helmet exists in a working area, the storage position of the safety helmet and the like.

Further, the monitoring system performs logic analysis on the inferred result of the content corresponding to the preset function type. The monitoring system first determines whether an operator is present in the video image data. In the presence of an operator, the monitoring system determines whether the operator performed the corresponding work action.

For another example, according to the inferred result of the detection of the wearing condition of the helmet, the monitoring system determines whether an operator exists in the working area in the video image data through the detection of the human body (such as the detection of the head and the shoulders of the human body). Under the condition that an operator exists in the operation area, the monitoring system judges whether the head of the operator is provided with the safety helmet or not, and determines the operation behavior of the operator according to a judging result, wherein the operation behavior comprises that the operator is provided with the safety helmet or the operator is not provided with the safety helmet.

According to some embodiments, the algorithm model may include a plurality of sub-models used in series. And the monitoring system performs comprehensive logic analysis according to the output results of the sub-models and outputs the analysis results to serve as final operation analysis results. The operation analysis result comprises description of operation behaviors, time and place of the operation behaviors, and field screenshot or video corresponding to the operation behaviors.

For another example, in the event that it is inferred that the helmet is hanging on a wall of the work area and no worker is detected within the work area, the monitoring system cannot determine whether the worker has worn the work action of the helmet.

In step S130, the monitoring system monitors the operation behavior of the operator through the operation analysis result.

For example, in step S130, the monitoring system determines the operation analysis result according to the preset condition, so as to determine whether the operation analysis result corresponds to the operation behavior of the operator, and displays the determined operation analysis result.

For example, if the worker does not wear a helmet in the work area, the monitoring system determines that the action is unsafe according to preset conditions. If the worker wears the safety helmet in the working area, the monitoring system determines that the behavior is safe according to preset conditions. And the monitoring system displays the action of whether the worker wears the safety helmet or not as a determined operation analysis result.

Under the condition that the operation behaviors of the operators are unsafe, the monitoring system sends out warning information to the operators in the operation area, so that the operators can timely normalize or correct the unsafe operation behaviors.

According to some embodiments, the warning information sent by the monitoring system includes warning information sent by using an acousto-optic device, such as sending a warning by flashing a warning lamp or playing a warning prompt tone by a loudspeaker.

According to the embodiment of the application, the monitoring system can realize the acquisition and analysis of the local video stream of the operation area, the transmission of the video stream is not required to be carried out through a public network, the stability of the video stream image is ensured, and the monitoring system can be linked with the operation area in real time on site, so that the monitoring of the behaviors of operators in the operation area is realized.

As shown in fig. 2, the monitoring system 100 includes a data acquisition module 110, a data processing module 120, a communication module 130, a visual processing module 140, and a warning module 150, wherein the data acquisition module 110, the data processing module 120, the communication module 130, and the warning module 150 are disposed in the operation area 200 together with a video device 210 for acquiring video stream data.

The data acquisition module 110 presets video equipment information within the work area 200. The data acquisition module 110 is connected to the video device 210 in the work area 200 through a network according to the video device information, and acquires video stream data of the work area 200 from the video device 210.

The data acquisition module 110 decodes and format-converts the acquired video stream data to video image data that can be processed by the algorithm module. The video image data may be stored in a memory (not shown in fig. 2) of the monitoring system 100.

The data processing module 120 obtains the video image data that has been format-converted by the data acquisition module 110, and inputs the video image data into a preset algorithm model.

The data processing module 120 performs inference on the video image data through the algorithm model, performs logic analysis on the inferred result, and determines the operation behavior of the operator as the operation analysis result.

The data processing module 120 determines whether the operation behavior in the operation analysis result is safe according to the preset condition, and temporarily stores the determined operation analysis result in a local database (not shown in fig. 2) of the monitoring system 100.

The communication module 130 transmits the determined job analysis result obtained by the data processing module 120 to the vision processing module 140.

According to some embodiments, job analysis results that are cached in the local database of the monitoring system 100 may be deleted after being sent to the vision processing module 140.

The vision processing module 140 receives and displays the determined operation analysis result. In the case that the operation behavior of the operator is unsafe in the operation analysis result, the vision processing module 140 sends an alert command to the alert module 150.

According to some embodiments, the vision processing module 140 may be configured at the cloud.

According to some embodiments, the vision processing module 140 may be in data communication with the data acquisition module 110 via the communication module 130 for configuring parameters of the data acquisition module 110, such as opening or closing channels for acquiring video stream data, and the like.

According to some embodiments, the vision processing module 140 may also be used to configure parameters of the algorithm model in the data processing module 120. The vision processing module 140 can communicate data with the data processing module 120 through the communication module 130 according to actual requirements to adjust and upgrade the algorithm model.

The warning module 150 receives the warning command sent by the vision processing module 140, and starts the acousto-optic equipment (not shown in fig. 2) in the working area 200 to send warning information according to the warning command, so as to prompt the operator in the working area 200.

It should be understood that the number of modules and devices in fig. 2 is merely illustrative. Any number of modules and devices may be provided as desired.

As shown in fig. 3, the data acquisition module 110 includes a channel manager 111, a streamer 112, a decoder 113, and a converter 114.

The channel manager 111 is connected to the video device through a network, and acquires multiple video stream data from the video device. The channel manager 111 sets a corresponding plurality of channels according to the multi-channel video stream to manage the video stream data.

According to some embodiments, the channel manager 111 may open or close a channel according to information of a visual processing module (not shown in fig. 3) to acquire or stop acquiring video stream data.

The streamer 112 obtains the video stream data from the channel manager 111 and converts the video stream data into byte stream queues.

According to some embodiments, the streamer 112 may employ RTSP (Real Time Streaming Protocol, real-time streaming protocol) based clients.

The decoder 113 decodes the byte stream data obtained by the streamer 112 to obtain a corresponding video frame queue.

The converter 114 format-converts the video frame queue according to a data format processable by the algorithm model and outputs the video image data after the format conversion.

It should be understood that the number of modules and devices in fig. 3 is merely illustrative. Any number of modules and devices may be provided as desired.

As shown in fig. 4, in step S210, the monitoring system inputs video image data into an algorithm model.

For example, in step S210, the monitoring system inputs the video image data subjected to format conversion into a preset algorithm model, where the preset algorithm model may include a plurality of sub-models according to actual requirements, and may be adjusted or updated through the cloud.

In step S220, the monitoring system makes an inference on the video image data.

For example, in step S220, the monitoring system deduces content corresponding to the preset function type in the video image data through the preset function type in the algorithm model, and obtains the deduced result.

The monitoring system carries out logic analysis on the inferred result through an algorithm model. The monitoring system firstly judges whether an operator exists in the operation area. In the case of the existence of the operator, the monitoring system determines whether the operator executes the operation behavior corresponding to the inferred result, and takes the operation behavior as the operation analysis result.

Further, the monitoring system judges whether the operation behavior in the operation analysis result is unsafe or not according to preset conditions.

In step S230, the monitoring system displays the job analysis result.

For example, in step S230, when no operator exists in the working area, the operator does not perform the working action corresponding to the inferred result, or the working action of the operator belongs to the safety action, the monitoring system displays the determined working analysis result through the vision processing module.

Under the condition that the operation behaviors of operators belong to unsafe behaviors, the monitoring system displays the determined operation analysis results through the visual processing module and sends warning instructions to the warning modules on the site of the operation area through the visual processing module.

In step S240, the monitoring system sends out alarm information.

For example, in step S240, the monitoring system sends out warning information through the warning module according to the warning command, so as to prompt the operator in the operation area.

According to the embodiment of the application, the monitoring system can realize the acquisition and analysis of the local video stream of the operation area and monitor whether the behaviors of operators in the operation area are safe or not in real time.

As shown in fig. 5, the electronic device 600 is merely an example, and should not be construed as limiting the functionality and scope of use of the embodiments of the present application.

As shown in fig. 5, the electronic device 600 is embodied in the form of a general purpose computing device. Components of electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different system components (including the memory unit 620 and the processing unit 610), a display unit 640, etc. In which a storage unit stores program codes that can be executed by the processing unit 610, so that the processing unit 610 performs the methods according to various exemplary embodiments of the present application described in the present specification. For example, the processing unit 610 may perform the method as shown in fig. 1.

The storage unit 620 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 6201 and/or cache memory unit 6202, and may further include Read Only Memory (ROM) 6203.

The storage unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Bus 630 may be a local bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or using any of a variety of bus architectures.

The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and/or any device (e.g., router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 650. Also, electronic device 600 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 600, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

From the description of the embodiments above, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. The technical solution according to the embodiments of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the method according to the embodiments of the present application.

The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The computer-readable medium carries one or more programs which, when executed by one of the devices, cause the computer-readable medium to perform the aforementioned functions.

Those skilled in the art will appreciate that the modules may be distributed throughout several devices as described in the embodiments, and that corresponding variations may be implemented in one or more devices that are unique to the embodiments. The modules of the above embodiments may be combined into one module, or may be further split into a plurality of sub-modules.

According to some embodiments of the application, the technical scheme of the application analyzes the safety operation behaviors in the warehouse logistics scene based on machine vision, links the analysis result to the site, reminds operators to observe the safety standard, corrects the operation behaviors which are not standard for the operators, and improves the safety supervision and guarantee capability in the business operation process.

The foregoing detailed description of the embodiments of the application has been presented only to assist in understanding the method and its core ideas of the application. Meanwhile, based on the idea of the present application, those skilled in the art can make changes or modifications on the specific embodiments and application scope of the present application, which belong to the protection scope of the present application. In view of the foregoing, this description should not be construed as limiting the application.

Claims

1. A method for monitoring operation behavior, comprising:

acquiring video image data of an operation area, wherein the video image data comprises operation behaviors of operators;

generating a job analysis result of the operator according to the video image data;

and monitoring the operation behaviors of the operators through the operation analysis results.

2. The method of claim 1, wherein acquiring video image data of a work area comprises:

acquiring video stream data from video equipment in the operation area;

decoding the video stream data to obtain a plurality of video frames;

the video image data is generated by format conversion of the plurality of video frames.

3. The method of claim 1, wherein generating job analysis results for the job personnel from the video image data comprises:

inputting the video image data into an algorithm model;

and obtaining the operation analysis result through the algorithm model.

4. A method according to claim 3, wherein obtaining the job analysis result by the algorithm model comprises:

deducing the video image data through a preset function type in the algorithm model;

and carrying out logic analysis according to the inferred result to output the operation analysis result.

5. The method of claim 4, wherein performing a logic analysis based on the inferred results to output the job analysis results comprises:

determining whether the operator is present in the video image data;

if the video image data exists, determining the operation behavior of the operator in the video image data;

outputting the operation analysis result including the operation behavior of the operator.

6. The method of claim 1, wherein monitoring the job behavior of the job personnel via the job analysis results comprises:

determining whether the operation behavior of the operator is unsafe or not according to the operation analysis result;

if yes, sending out warning information to the operator.

7. A system for monitoring operation behavior, comprising:

the data acquisition module is used for acquiring video stream data in the operation area; decoding the video stream data; obtaining video image data by performing format conversion on a plurality of video frames obtained after decoding;

the data processing module analyzes the video image data through an algorithm model to obtain an operation analysis result;

the communication module is used for sending the operation analysis result to the vision processing module;

the visual processing module receives and displays the operation analysis result; and sending an alarm instruction according to the operation analysis result.

8. The system of claim 7, further comprising:

the warning module receives the warning instruction; and sending out warning information to operators in the operation area according to the warning instruction.

9. An electronic device, comprising:

one or more processors;

a storage means for storing one or more programs;

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.