EA041812B1 - METHOD AND DEVICE FOR FIXING ALARM EVENTS ON A SERVICE VEHICLE - Google Patents

Description

Technical field

The invention relates to the field of video analytics systems used on service vehicles (STS) for fixing certain events.

State of the art

Solutions used by various approaches to video analysis of information on vehicles (TS) are widely used today. As a rule, such solutions are used in freight transport when transporting goods, such as containers. Video recording of data is carried out both in the area of the cargo compartment to obtain data on events occurring directly with the cargo, and outside the vehicle to process events in the surrounding space. Such monitoring systems are based on the processing of data received from surveillance cameras installed outside and inside the vehicle. An example of such a solution is known, for example, from US patent 9033116 (Intelligent Technologies International Inc, May 19, 2015). Often, when implementing such solutions, approaches are used based on machine learning algorithms that provide event detection and classification based on images of the received video stream from surveillance cameras. One example of this kind of solution is the application of the machine learning model architecture based on the ResNet neural network (Watkins et al. Vehicle classification using ResNets, localization and spatially-weighted pooling//2018).

Despite this, for a certain type of situation in production activities, such as, for example, cash collection service, it is necessary to take into account a certain range of events, which are characterized by specified triggers that must be recorded in the received video stream, which imposes the need to expand the functionality of the video analytics complex for its efficient operation in the required conditions. Thus, based on the existing solutions, there is a technical problem due to the shortcomings of the known solutions in terms of their limited functionality when used to monitor events on the CTS that can occur in its various states.

The essence of the invention

The claimed invention is aimed at solving a technical problem inherent in the known solutions from the prior art.

The technical result is to increase the efficiency of monitoring situations on official vehicles by identifying events in the video stream corresponding to various states of the production process on the official vehicle.

The technical result is achieved due to a computer-implemented method of fixing alarm events on a service vehicle (STS), performed using a processor and containing the steps at which data is received from CCTV cameras installed in the cabin and outside on the STS;

processing the received data using at least one machine learning model configured to fix the first state of the STS based on the processing of data showing the position of at least one employee in the STS cabin when he moves along the route;

fixing the second state of the STS upon receipt of data indicating the opening of the STS door and a change in the position of at least one employee in the STS cabin; moreover, when fixing the second state, the position of at least one employee at the open door outside the STS is determined and the order in which the employees leave the STS salon with a given time interval; and if the employees are closer than the specified distance value at the open door outside the STS and/or the employees do not leave the STS cabin one by one within the specified time interval, then an alarm event is recorded.

In one of the particular embodiments of the method, the fixation of the first state of the STS is carried out with the employees in the sitting position of the STS. In another particular embodiment of the method, the fixation of the second state of the STS is performed upon receipt of an image that at least one employee gets up in the STS cabin.

In another particular embodiment of the method, the analysis of the position of the employees of the STS when the second state of the STS is activated additionally includes processing data showing the location of employees outside the STS.

In another particular embodiment of the method, the first state of the STS is analyzed dynamically during the movement of the STS.

In another particular embodiment of the method, when the STS is moving, data from the STS cameras is additionally analyzed for the analysis of state registration plates (GRZ) of vehicles (TC).

In another particular embodiment of the method, the analysis takes place in order to recognize repeated GRZ along the path of the STS.

In another particular embodiment of the method, the presence of data is additionally analyzed

- 1 041812 from cameras displaying information about the interaction of the STS employee with the radio after leaving the STS cabin.

In another particular embodiment of the method, the transportation of cargo is additionally fixed by an employee of the STS.

In another particular embodiment of the method, the cargo images obtained from the STS cameras are analyzed in the second state of the STS.

In another particular embodiment of the method, when analyzing the second state of the STS, the movement of cargo in the process of loading and/or unloading is additionally recorded.

In another particular embodiment of the method, the movement of the STS employees is analyzed when leaving the cabin with the load.

The claimed technical result is also achieved by a device for fixing alarm events on the STS, which contains at least one processor and at least one memory that stores machine-readable instructions that, when executed by the processor, implement the above method.

In one of the particular implementation examples, the device is a computing device hosted on the STS.

Brief description of the drawings

Fig. 1 illustrates an example of the placement of cameras on the STS, FIG. 2 is a block diagram of a video stream analysis method, FIG. 3 is a block diagram of a method for detecting alarm events in the first state of the STS, FIG. 4 is a block diagram of a method for detecting alarm events in the second state of the CTC, FIG. 5 is a general view of the computing device.

Implementation of the invention

In various production processes (for example, cash collection), various violations occur in the sequence, type of actions of the participants in the process. Checking for violations in such processes consists of a combination of performing checks on individual events and post-processing the results. Various machine learning models are responsible for testing different events, which can be combined into a set of models to identify combinations of events.

The main task for the effective implementation of a video analytics system is the selection of the optimal combination of models that allows you to achieve the required level of quality while minimizing the volume and speed, and also imposes minimal requirements on the order of creation and maintenance of such models. For example, in cash collection processes there are regular violations in the sequence, type of actions of collectors. Example: collectors, being nearby and opening the STS door, put the container together in the car, etc.

As shown in FIG. 1, the claimed solution is implemented by placing a group of video cameras (101-109) on the STS (100), some of which capture events inside the STS (100) cabin, the other is placed on the body of the STS (100) to capture the external situation. The processing of incoming video streams from cameras (101-109) is performed by a computing device (110), for example, a computer, which can be installed directly on the STS (100) or be an external server, data exchange with which is carried out by means of data transmission over a computer network, for example, a network Internet.

Such a solution architecture makes it possible to implement the accounting of many different events on the STS (100) recorded during various processes of operational activities, for example, when the STS moves, stops at the STS on the way, the process of parking and unloading the STS, exit of personnel from the salon, entry of personnel into the STS salon And so on.

The processing of the video stream from the cameras (101-109) is performed by one or more machine learning models, for example, an artificial neural network (ANN) with the Resnet_18 architecture or another ANN suitable for detecting events on the frames of the video stream. ANN training was performed for all types of defined events, and at the last training step, data pruning models (pruning neural network) in TLT were used, which made it possible to reduce the volume of the final model.

The training of the applied one or more machine learning models was carried out as follows.

Frames from individual cameras were used to collect data portions (batch). The portion size depends on the number of cameras, classes, threshold for removing outliers in the model responses. Batch characteristics are determined during testing - measuring the speed of the pipeline for various combinations of batch parameters.

Next, the batch is fed to the input of the model: the model response contains the class values for all frames (from different cameras) included in the batch. From the cameras, data is obtained not only about recognized events, but also about the characteristics of the environment. For example, whether the event takes place on the street or in the STS salon (100). It is important that separate classes of detected events cannot be related to different types of environment (for example, an open door is only detected outdoors, but not in a passenger compartment). Thus, it is possible to obtain information not only about the class to which the image belongs, but also about the location of the camera from which the image was received. There are outliers in the model's responses - one-time false responses. To smooth the responses, responses were selected for 10 successive frames, if 8 frames out of 10 (the order is not important) received one response, then such a response was considered reliable.

Implementation example: data comes from 2 cameras. 1 camera looks at the street, 2 - into the interior of the STS. The camera facing the street detects door opening/closing events; the number and position of people in the cabin is determined by the camera looking into the STS salon, thus, both the position of the doors and the position of people are determined by one model. A batch of 8 frames is formed: 4 frames from camera 1 and 4 frames from camera 2. Based on the response of the model, the required characteristics are determined (the described approach is used to remove outliers: 8 out of 10 identical responses), and also, if necessary, which camera the event belongs to: to internal or external.

This approach allows optimizing the running time of the solution for analyzing images from frames due to the fact that with the help of one model, the possibility of generating a response to events of various types is realized, without the need to use different models to detect a specific type of event. At the same time, the time for additional training of models is optimized and reduced, because when a new type of CTS appears, only one model changes (retrains).

Since the camera can rotate through a small angle during the operation of STS (100), it is necessary to take into account the augmentation of images from frames with rotation (the angle was taken in the range from -45 to 45°).

Example machine learning model classes:

no_person - no people in the salon;

only_seat_1 - there is one person in the cabin, he is sitting;

only_seat_2 - in the cabin from two people, and everyone is sitting;

stand_up_l - there is one person in the cabin, he is standing;

stand_up_2 - in the cabin from two people, at least one is standing;

opened - STS side door is open;

closed - STS side door is closed.

Class size example (before augmentation):

{'no_person':5505, 'only_seat_1':1985, 'only_seat_2':9540, 'stand_up_1':945, 'stand_up_2':1390, 'opened':1250, closed':2370}

For the model stability to rotations and camera pollution, changes in the illumination of the object, data augmentation methods were used - expanding the data set (dataset) by adding transformations to the original images.

Applied augmentation methods: probabilistic change of brightness, contrast, addition of noise, conversion to gray, geometric transformations (https://github.com/albumentations-team/albumentations).

As one example, when training and testing a machine learning model for the purposes of classifying events, the final data was divided in the proportion: 80% / 20%. For validation, 10% of the training sample was taken (for validation, data were taken before the use of augmentation).

The claimed approach is applicable to a large number of tasks analyzed in various situations and states of the CTS. The state of the CTS should be understood as the presence of the CTS in a specific situation during the implementation of one or another production action, for example, the movement of the CTS along the route, the parking of the CTS for loading / unloading actions, waiting at the loading / unloading point, etc. Each event is analyzed in its own state of the CTS, which allows more efficient switching of model operation modes and fixing the required type of events, increasing the accuracy and quality of video data analysis.

In FIG. 2 shows the general flow of the method (200) for detecting alarm events on the CTC. The incoming data from the cameras (101-109) at step (201) is input to the computing device (110), which stores a machine learning model in machine-readable memory that performs subsequent processing of video streams at step (202) for event analysis (203). Events at step (203) are analyzed at the first (300) and second (400) state of the CTC. For each state, the analysis of the corresponding parameters is performed, signaling the occurrence of the corresponding alarm event, information about which is recorded at step (204).

In FIG. 3 is a flowchart of a method for analyzing events in the first CTC state (300).

At the first stage (301), from one or more cameras (for example, 103, 105, 106, 108, 109), which record the situation inside the STS cabin when it is moving (first state), a video stream is obtained for analysis. The video stream is transmitted to the computing device (110). Based on the received frames of the video stream, at least one STS employee is analyzed, in particular, at step (302), it is checked whether he has changed his sitting position during the movement. If the analysis of frames depicting the employee's posture at step (303) is acceptable, then the operation of the video monitoring system continues in the normal mode with the receipt and analysis of incoming frames from surveillance cameras. In the event that the fact of a change in the position of an employee is recognized, for example, the employee takes a standing position, and

- 3 041812

The STS (100) stops, then the start of the second state is latched in step (304).

Additionally, when the STS (100) moves along the route, the environment can also be monitored using cameras installed on the outer perimeter of the STS (100) body. One of the examples of events analyzed in this situation can be the license plate numbers of vehicles around the STS. In particular, an analysis is made of the presence of repeated GRZ numbers along the route of the STS in different time periods, which may indicate a possible alarming event in terms of unauthorized tracking and/or prosecution of the STS (100).

In one of the particular examples of the implementation of this approach, if a similar (with an accuracy of two digits) GRZ number was recorded earlier than a specified time (for example, 30 minutes) before the detection event within a week before the detection event, then the event is recorded and sent to the server a message about the appearance of a repeated number of the GRZ with an image containing a vehicle with this number, while one image is sent to one repeated number. If a similar (up to two digits) GRZ number was recorded, for example, earlier than 30 minutes (the time range can be changed) before the detection event during the current day, on the tablet of the STS employee and / or on a remote server (for example, the coordination center ) a message is sent with the repeated number, in particular, one message per repeated number.

Detection of the vehicle number is performed within a specified time range, for example, 2 minutes from the moment the number appears in the video stream. Number symbols must be visually distinguishable on the video. If these conditions are violated, notifications are generated for collectors about potential violations and information with photos of events is stored in the database, which can be on a computing device (110) or on a remote server.

In FIG. 4 is a flow diagram of a method for analyzing events in a second CTC state (400). The analysis of the second state is carried out when the STS (100) stops and the employee gets up to open the side door of the STS (100). At step (401), the data from the cameras inside the cabin is analyzed for the presence of at least one employee at the side door of the STS and its subsequent opening.

Next, at step (402), the data is analyzed from external cameras installed on the perimeter of the CTC body (100). The information received from the cameras (for example, 101, 102) is checked for frames showing employees leaving the STS salon, and the images of each employee and the time after which they leave the STS salon (100) are recognized. If the employees do not leave the salon one by one or the time range for employees leaving the salon is less than the set value (for example, 10 s between the exit of each employee), and there is no employee outside the STS at the open door, then an alarm event is recorded at step (403). In this case, the date, time and image are recorded with the corresponding violation. Also, fixing can be carried out for the location of employees outside the STS, in particular, that they are nearby at a given distance from each other. The analysis can be performed by calculating the distance between the pixels of the image, taking into account the resolution of the cameras (101, 102). Additionally, it can also be recorded that the employee who first left the STS salon interacts with the radio, which, as a rule, is located on his chest, which indicates that the procedure for reporting the arrival of the STS is being followed.

When carrying out the collection service procedure, an analysis of the cargo is performed, for example, a case or a special bag for cash, which can be in the STS cabin (100) when the second state is activated or directly moved to the CTS cabin by employees.

In this case, the images from the cameras inside the STS cabin, when the second mode is activated, are checked for the fact that the employee initially leaves the STS and waits for the second employee with the cargo. If this analysis fixes a violation of the procedure, then an alarm event is recorded. At the same time, frames that characterize the movement of cargo outside the STS, as well as the fact that employees move in a regulated manner, can also be analyzed. If there is a violation of the procedure for moving cargo, then data on an alarm event is generated.

In FIG. 5 is a general view of a computing device (500) suitable for performing any of the methods described above, and is also partially or completely equivalent to the device (110). Device (500) may be, for example, a server or other type of computing device that can be used to implement the claimed technical solution. Including being part of a cloud computing platform.

In the general case, the computing device (500) contains one or more processors (501) connected by a common information exchange bus, memory facilities such as RAM (502) and ROM (503), input/output interfaces (504), input/output devices ( 505) and a device for networking (506).

The processor (501) (or multiple processors, multi-core processor) may be selected from a variety of devices currently widely used, such as those from Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, and the like. The processor (501) can also be a graphics processor such as Nvidia, AMD, Graphcore, etc.

-

Claims (8)

RAM (502) is a random access memory and is designed to store machine-readable instructions executable by the processor (501) to perform the necessary data logical processing operations. The RAM (502) typically contains the executable instructions of the operating system and associated software components (applications, program modules, and the like). The ROM (503) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state data drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R/ RW, DVD-R/RW, BlueRay Disc, MD), etc. Various types of B/B interfaces (504) are used to organize the operation of device components (500) and organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific implementation of the computing device, which can be, but not limited to, PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc. To ensure user interaction with the computing device (500), various means (505) of B/B information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse, a light pen, a stylus, a touchpad, a trackball , speakers, microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc. The networking means (506) enables the communication of data by the device (500) via an internal or external computer network, such as an Intranet, Internet, LAN, and the like. As one or more means (506), an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and/or BLE module, a Wi-Fi module, and others Additionally, satellite navigation tools in the device (500) can also be used, such as GPS, GLONASS, BeiDou, Galileo. The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology. CLAIM 1. A computer-implemented method for fixing alarm events on a service vehicle (STS), performed using a processor and containing the steps at which data is received from video surveillance cameras installed in the passenger compartment and outside on the STS; processing the received data using at least one machine learning model capable of fixing the first state of the STS based on the processing of data showing the position of at least one employee in the STS cabin when he moves along the route; fixing the second state of the STS upon receipt of data indicating the opening of the STS door and a change in the position of at least one employee in the STS cabin; moreover, when fixing the second state, the position of at least one employee at the open door outside the STS is determined and the order in which the employees leave the STS salon with a given time interval; and if the employees are closer than the specified distance value at the open door outside the STS and/or the employees do not leave the STS cabin one by one within the specified time interval, then an alarm event is recorded. 2. The method according to claim 1, characterized in that the fixation of the first state of the STS is carried out with the employees sitting in the STS cabin. 3. The method according to claim 2, characterized in that the fixation of the second state of the STS is performed when an image is received that at least one employee gets up in the STS cabin. 4. The method according to claim 1, characterized in that the analysis of the position of the employees of the STS when the second state of the STS is activated additionally includes processing data showing the location of employees outside the STS. 5. The method according to claim 1, characterized in that the first state of the STS is analyzed dynamically during the movement of the STS. 6. The method according to claim 5, characterized in that when the STS moves, data from the STS cameras is additionally analyzed for the analysis of state registration plates (GRZ) of vehicles (TC). 7. The method according to claim 6, characterized in that the analysis takes place in order to recognize repeated GRZ along the path of the STS. 8. The method according to claim 4, characterized in that it additionally analyzes the presence of data from cameras that display information about the interaction of the STS employee with the radio after leaving the cabin -