RU2738664C1 - Vehicle control and monitoring system - Google Patents

Abstract

FIELD: automated control systems.

SUBSTANCE: invention relates to automated control systems, dispatcher control and safety of vehicle location on the route. Vehicle control and monitoring system comprises a response center, third-party systems and a video analytics software and hardware system (VDA SHS) configured to connect third-party systems and including video cameras, distributed video data analysis network and installed on motor vehicle master calculator, made with possibility to obtain data from third-party systems and video cameras, storing and transmitting to the response center data and analytics obtained during operation. VDA SHS is designed to determine situations that may be undesirable. Said situations are determined based on automated analysis of images obtained from video cameras mounted on a vehicle. Analysis is carried out by an electronic computing system using computer vision algorithms.

EFFECT: higher traffic safety of vehicles.

7 cl, 1 dwg, 1 tbl

Description

The invention relates to systems for automated control, dispatching control and ensuring the safety of being on the route of vehicles (TS).

An automated system for monitoring the technical condition of monitoring vehicles and a recorder of vehicle parameters is known from the prior art (RU 53049 U1, G08G 1/01, 27.04.2006). It consists of a base station, mobile units, a central control room and route sensors. The base station has transceivers installed. The base station microcontroller is configured to generate commands for setting and polling the mobile unit. Each mobile unit is equipped with a sensor unit for its technical and operational state. The central control center contains a transceiver device, the output of which is connected through the corresponding bus of the input-output port to the second PC, made with the ability to summarize and analyze the information of the base station microcontroller according to the boundary values of the monitored parameters of the vehicle parameter recorder. Route sensors incorporate transmitting devices capable of generating signals that record the fact that moving units pass by them. The integration into a single automated system is carried out by radio channels between the mobile unit and the base station by transceiver devices, between the base station and the central control center and between the mobile unit and the route sensors, the transceiver device of the recorder and the transmitting device of the route sensor.

The disadvantage of this system is the lack of a high degree of its automation and control efficiency, lack of video data analysis.

From the prior art, a control and monitoring system for routed vehicles is known (RU 2712404 C2, G08G 1/123, patent publication date 01/28/2020), selected as a prototype of the claimed invention. The prototype system includes an on-board terminal of the system that collects data from passenger flow sensors or load sensors, driver's driving style and manner control devices, a safe driving video analytics module, video recorders, a device for continuous voice communication between a dispatcher and a driver, emergency buttons, temperature sensors, etc. transfer of all received information to the telematics server with software and to the dispatcher's computer. The telematics server receives data from telemedicine terminals, from a radio frequency identification device for the right to access / exit a vehicle to / from the vehicle fleet (s) and a reporting module, into which trip reports are entered. While the vehicle is in motion, the terminal receives data from the safe driving video analytics module, made in the form of a software and hardware module and video surveillance cameras installed in the passenger compartment, and which allows you to monitor the driver's behavior while the vehicle is moving, identify the bus driver's conversation on a mobile phone, and etc.

The main disadvantage of the prototype system is the inability to analyze all the data received by the system directly on board the vehicle, which makes it vulnerable, for example, in the event of an emergency disconnection of communication. In this case, the safety of the vehicle may be compromised. Other disadvantages of the prototype system include the lack of biometric face recognition technology, the lack of using computer vision algorithms and artificial intelligence systems (neural networks) for event detection.

The objective of the claimed invention is to create a control and monitoring system for vehicles, which improves the safety of vehicle traffic, due to the ability to analyze all information received by the system and detect events directly on board the vehicle, configure each analytics module to solve its problem, and configure the generation of messages according to the type of situation , control of the condition of the driver, the roadway, compliance with traffic rules by the driver, automation of control over the technical and operational condition of the vehicle, automated transmission of data arrays on the technical and operational condition of the vehicle to the response center, improving the safety of road transport by constantly monitoring the condition of the vehicle and preventing outputs of monitored parameters about the technical and operational state.

The task is achieved due to the fact that the vehicle control and monitoring system includes a response center, third-party systems and a video analytics software and hardware complex (VDA PAK), made with the ability to connect third-party systems and including video cameras and a main computer installed on the vehicle, made with the ability to receive data from third-party systems and video cameras, save and transfer data and analytics obtained during operation to the response center, in the declared system, the VDA PAK is designed to determine situations that may turn out to be undesirable, while these situations are determined on the basis of automated image analysis received from video cameras installed on a vehicle, the analysis is carried out by an electronic computing complex using computer vision algorithms, and machine recognition and artificial intelligence systems are used to identify incidents related to loss of attention and violation of driving a vehicle, monitoring the state of facial expressions, eye movements and closing the driver's eyelids with the identification of incidents that indirectly reveal threats, namely, yawning, closing the eyelids, slowing down psychological reactions, is performed continuously, while the PAK ACA additionally includes a distributed video data analysis network installed on a vehicle, which is an analytics module and is capable of obtaining information about the state of a vehicle and using it to detect undesirable situations (events), generate messages about detected undesirable situations for their subsequent sending to the response center , with the ability to configure the generation of messages for the response center for each type of situation: when detecting a situation, when detecting the end of a situation, the frequency of messages when the situation continues, with the ability to configure each m Analytics module to solve its problem, indicating the address of the video camera from which it is necessary to receive a video stream or disk space, setting events and specifying the address of an external system, while the vehicle control and monitoring system additionally includes an access control and management system that performs biometric recognition persons, an additional module made with the ability to perform non-invasive temperature measurement using a thermal imager, namely, to detect increased body temperature, the main computer is configured to receive data from a distributed video data analysis network, an access control and management system, a thermal imager and configure analytics modules included into a distributed video data analysis network, and specifying the location of data acquisition for analytics. In the preferred embodiment of the claimed system, third-party systems are a video recorder installed on a vehicle, a tracker, and an on-board network of a vehicle. The video recorder can be configured to connect video cameras. The tracker can be configured to connect at least one type of sensors selected from the sequence: discrete, analog, frequency, pulse sensors, tachometer, digital fuel level sensors, tachograph. The declared system may additionally include optional modules: display panel, digital sensors, analog sensors. PAK VDA can be configured with the ability to connect optional modules, and the main computer - with the ability to receive data from digital and analog sensors. The main computer can be configured to inform the driver about the operation of the vehicle control and monitoring system using the display panel.

The presence of a distributed video data analysis network allows you to configure analytics modules to detect certain events, as well as to detect events directly at the place of their detection, which increases the speed of information transfer and decision-making and, consequently, the safety of the entire vehicle. Due to the fact that the analytics takes place on board the vehicle, and not on the cloud server, the claimed system can work offline in areas where there is no connection with the server, make decisions right on board, and not later (when it enters the area with cellular communication ). The control over the condition of the driver is improved through continuous monitoring of his behavior and the identification of indirect threats to disrupt the driving process. In addition, the feedback with the driver can be improved when the event message is transmitted to him directly, and not through the dispatcher, that an illegal fuel discharge is taking place, or about the hatch not closed, etc.

The control and monitoring system of vehicles includes the following nodes (see figure 1): a response center (11), third-party monitoring systems, a hardware-software complex for video analytics (PAC VDA), as well as an access control system and an additional module for non-invasive measurement body temperature using a thermal imager. Connection to the system of optional modules is possible. Let's consider each node separately.

The Response Center (11) is a remote server connected to the vehicle control and monitoring system via a radio modem. Allows you to perform the following functions:

- Database, saves all data coming from vehicles equipped with PAK VDA;

- analytics module for automatic decision-making to inform the operator;

- module for integration with external systems by the state fire service;

- software update module for PAK VDA;

- a module for creating configuration files for the PAK VDA software;

- operator interface.

Third-party monitoring systems are standard monitoring systems already installed on the vehicle system or installed to expand the capabilities of the VDA PAK. Third-party monitoring systems consist of: tracker (4), video recorder (5) and vehicle on-board network (6)

The tracker (4) is connected to the VDA PAK via the RS-485 interface. It is designed to control the movement of vehicles, fuel level, vehicle operation. The tracker can be connected to:

- Discrete sensors: sensors for motion, ignition, door opening, passenger presence, tilt, etc.: P0, P1, P2, P3 for sensors with a positive polarity of the output signal, P4, P5 for sensors with a negative polarity of the output signal.

- Analog sensors (fuel level, voltage, temperature, etc.): P0, P1, P2, P3.

- Frequency sensors (fuel level, frequency, engine speed, temperature, etc.): P2, P3, P4, P5.

- Pulse sensors (fuel flow meters, passenger counting sensors and others): P2, P3, P4, P5.

- Tachometer: P4, P5.

- Digital fuel level sensors LLS: RS-485.

- Tachograph and sensors using the interface: RS-485.

The video recorder (5) supports connection of up to 4 (8) analog or 4 (8) HD (720P) AHD CCTV cameras, it can also act in hybrid mode for a different combination of analog and HD cameras (channel switching). Communication with PAK VDA is carried out depending on the model of the DVR, it can be USB, Ethernet, WiFi (wired or wireless).

The vehicle on-board network (6) is connected to the VDA PAK via the CAN bus. Information received from the vehicle's on-board network and provided by the information transmission unit (external system) about the state of the vehicle: speed, direction of travel, supply voltage, the fact of engine operation, parking brake activation, data from the headlight control system, steering column and control pedals, data from systems for monitoring the use of seat belts, temperature inside the cabin, etc.

PAK VDA is designed to identify situations that may be undesirable. Description of potentially undesirable situations (events) is given in Table 1.

Situations are determined based on an automated analysis of images obtained from video cameras installed on the vehicle. The analysis is carried out by an electronic computer complex using computer vision algorithms. Using a machine recognition system and Artificial Intelligence (neural network) to identify incidents associated with a drop in attention and a violation of vehicle control. Continuous monitoring of the state of facial expressions and eye movements (including closing of the eyelids) of the driver is carried out with the identification of incidents indirectly revealing threats (yawning, closing of the eyelids, slowing down of psychological reactions). With the help of an additional module, a non-invasive temperature measurement is performed using a thermal imager (detection of increased body temperature).

The complexes are located on the vehicle and analyze the images in real time. The analysis results are archived and transferred to the monitoring center.

To optimize the operation of the electronic computing complex, it is supposed to use additional data, for example, data from the on-board data transmission network of the vehicle.

At the same time, the VDA PAK must ensure the recording, storage and authorized removal of all received video streams in high quality.

Optionally, it is provided for the implementation of additional functionality in the VDA PAK, such as the collection and transmission of data of various nature directly to the response center, the transmission of voice messages to the driver, etc.

The VDA PAK consists of a Distributed video data analysis network (1) (analytics modules), a main computer (2), digital video cameras (3), a PoE switch (s), a loudspeaker and a speaker. Data from digital video cameras (3) are transmitted to the general network, from analog cameras the data is received as to a local disk. The distributed network of video data analysis (1) allows, through the Web interface or by downloading a configuration file, to configure each analytics module to solve its problem (definitions of the event given in Table 1), to specify the address of the camera from which it is necessary to receive a video stream or disk space for saving video or event metadata, configure events and specify the address of an external system (for example, a dispatch center). Using the loudspeaker and microphone, contact the driver. The distributed network of video data analysis makes it possible to instantly inform the vehicle driver about the detected events (see table 1). The main computer receives events from the distributed video data analysis network, data from the tracker, video recorder, vehicle on-board network, access control system (ACS), thermal imager, digital and analog sensors, saves them and transmits them to the response center via radio modem. To inform the driver about the operation of the system, it is possible to install an indication panel, as well as light and sound alarms.

Let's consider the complex element by element:

1. Distributed network of video data analysis. (1) It consists of modules, single-board transport analytics computers and acts as sensors for third-party systems, reporting discrete events over the Ethernet network. Each event sent to external systems is a reported type of detected potentially unwanted situation.

A distributed network of video data analysis allows:

- receive video streams containing images generated by video cameras installed on the vehicle;

- receive information from the data transmission unit (third-party system) about the state of the vehicle and use it to detect situations, while the list of necessary data and the data transfer protocol can be adjusted;

- receive information about the state of the vehicle from the communication module with the CAN bus of the vehicle, use it to detect the Situations (in the required amount);

- to process the received images taking into account (if necessary) information about the state of the vehicle;

- process all images at the same time, except for those cases when it is redundant, the processing includes analysis to determine all the situations indicated in Table 1, while the quality of detection of situations in video is measured by precision / recall metrics on a deferred tagged sample of images and should not be less than the values listed in Table 1 for each event, respectively, taking into account the recognizable sequence, for example, of 5 frames, it is allowed to provide detection of situations with built-in video cameras with obtaining information about the fact of detection in the recognition module with its subsequent transfer to the response center, compliance with metrics does not depend on the intensity and direction of external lighting;

- in case of detecting situations, generate appropriate messages with events for their subsequent sending to the response center via a data transmission unit (third-party system) or via a mobile communication module, messages should be accompanied by a confirmation photo, in which the recognized situation should be highlighted with a colored frame;

- make sure that messages are received by the response center using the corresponding receipts returned by it; if the receipt is not received, send messages again;

- due to the relatively long duration of the situations under consideration, in order to increase the quality of recognition, it allows one to allow several consecutive checks for the fact of a situation occurrence before a message is generated for the response center;

- configure the generation of messages for the response center for each type of situation: when detecting a situation, when detecting the end of a situation, the frequency of messages when the situation continues;

- transmit a signal to the video recording module about blocking from erasure of files corresponding to the detected situation, while it is allowed to write the corresponding files to its own storage while ensuring the storage of such data;

- collect diagnostic information (temperature, processor load, etc.) from various subsystems of the VDA PAK for their subsequent transfer to the response center;

2. The main computer (2). Installed directly in the vehicle (bus, truck, specialized motor transport for various purposes). Connects to the existing intelligent vehicle control system or can operate independently. Receiving power from the vehicle battery (AKB), it filters it from interference and converts it into operating voltage, charges the built-in battery, monitors the system temperature and, if necessary, turns on the heating or cooling of the system. The system is turned on either from turning on the ignition or from preset settings. Receiving data from sensors connected via various interfaces (CAN, RS-232, RS-485, USB, Ethernet, WiFi, BlueTooth and GPIO), connected video cameras via MIPI, USB, Ethernet or WiFi interfaces), uses built-in machine vision analytics , examples are shown in Table 1, performs various actions, such as: logging and storing data bound to UTC, transmitting data via radio modems (duplication), controlling external devices.

3. Digital video cameras (3). Images transmitted by digital video cameras to a distributed video data analysis network and the main computer satisfy the requirements for video data analysis regardless of the intensity and direction of external lighting (including backlighting, including at night).

The location of video cameras should be selected so that the degree of contamination during a 12-hour shift does not lead to a deterioration in the image quality below that required by the video data analysis module to operate with the specified characteristics (except for occasional one-time contamination). It is possible to use cameras with a washing system. The washer is controlled using GPIO from the Main Computer or from the tracker.

The location of the cameras should be selected in such a way that during operation the zones of interest could not be outside the field of view of the cameras.

The system time and the name of the video camera should be superimposed in the metadata of the image from the video camera.

4. Switch PoE. Responsible for communication between the main computer and a distributed network of video data analysis and digital cameras, as well as their power supply. Can be equipped with an uninterruptible power supply.

5. Control and management system for access (ACS) (7). When entering the salon, biometric face recognition is performed. Faces are processed by a preanalyzer that prepares data for the operation of the ACS system using Face ID. A request is made to the Face ID database. Identification "friend / foe" is performed. The status is issued to the access control system (opening of the turnstile to enter the salon).

6. Thermal imager (8). It is connected to the PAK VDA complex via Ethernet. The driver and passengers in the vehicle must have a body temperature not higher than 37.2 degrees. When processing face biometrics, a thermal imaging face picture is added. Processing is carried out by the "machine vision" system with the analysis of body temperature. If the “threshold temperature” is exceeded above 37.2 degrees, a warning signal is issued + access to the salon is denied (the turnstile does not open). The incident is processed and transferred to the response center.

7. Display panel (9). To inform the driver about the operation of the complex, the VDA PAK can be equipped with a tablet with a computer that allows you to diagnose the system, inform the driver about various situations.

8. Digital and analog sensors (10). Designed to control the movement of vehicles, fuel level, vehicle operation. They can be connected to VDA PAK via RS-485 interface or via GPIO ports.

The following sensors can be connected:

- Discrete sensors: sensors for motion, ignition, door opening, presence of a passenger, tilt, etc.: I0, I1, I2 for sensors with positive polarity of the output signal.

- Analog sensors (fuel level, voltage, temperature and others): 10, 11, 12.

- External outputs: O0, O1, O3.

- Digital fuel level sensors LLS: RS-485.

- Tachograph and sensors using the RS-485 interface.

The technical result from the use of the invention is to ensure an increase in the safety of the vehicle movement, due to the possibility of analyzing all the information received by the system and detecting events directly on board the vehicle, setting each analytics module to solve its problem, setting up the generation of messages by the type of situation and measuring the quality of situation detection , control of the condition of the driver, the roadway, compliance with traffic rules by the driver, automation of control over the technical and operational condition of the vehicle, automated transfer of data arrays on the technical and operational condition of the vehicle to the response center, improving the safety of road transport by constantly monitoring the condition of the vehicle and preventing outputs of monitored parameters about the technical and operational state.

Claims (7)

1. A control and monitoring system for vehicles, including a response center, third-party systems and a video analytics software and hardware complex (PAC VDA), made with the ability to connect third-party systems and including video cameras and a main computer installed on a vehicle, made with the ability to receive data from third-party systems and video cameras, saving and transferring to the response center data and analytics obtained during operation, characterized in that the VDA PAK is designed to determine situations that may turn out to be undesirable, while the mentioned situations are determined based on the automated analysis of images received from video cameras installed on a vehicle, the analysis is carried out by an electronic computer complex using computer vision algorithms, and machine recognition and artificial intelligence systems are used to identify incidents associated with falling attention iya and violation of driving a motor vehicle, monitoring the state of facial expressions, eye movements and closing the driver's eyelids with the identification of incidents that indirectly reveal threats, namely, yawning, closing the eyelids, slowing down psychological reactions, is carried out continuously, while the PAK VDA additionally includes the installed on the motor vehicle means a distributed network of video data analysis, representing analytics modules and made with the ability to obtain information about the state of a vehicle and use it to detect unwanted situations (events), generate messages about detected undesirable situations for their subsequent sending to the response center, with the ability to customize the generation of messages for the response center for each type of situation: when detecting a situation, when detecting the end of a situation, the frequency of messages when the situation continues, with the ability to configure each analytics module to solve its problem, with indicating the address of the video camera from which it is necessary to receive a video stream or disk space, setting events and specifying the address of an external system, while the vehicle control and monitoring system additionally includes an access control and management system that performs biometric face recognition, an additional module capable of performing non-invasive temperature measurement using a thermal imager, namely, detecting increased body temperature, the main computer is configured to receive data from a distributed video data analysis network, an access control system, a thermal imager and configure analytics modules included in a distributed video data analysis network, and instructions places of obtaining data for analytics. 2. The system according to claim 1, characterized in that the third-party systems are a video recorder installed on a vehicle, a tracker, an on-board network of a vehicle. 3. The system according to claim 2, characterized in that the video recorder is configured to connect video cameras. 4. The system according to claim. 2, characterized in that the tracker is configured to connect at least one type of sensors selected from the sequence: discrete, analog, frequency, pulse sensors, tachometer, digital fuel level sensors, tachograph. 5. The system according to claim 1, characterized in that it additionally includes optional modules: display panel, digital sensors, analog sensors. 6. The system according to claim 5, characterized in that the VDA PAK is configured to connect optional modules, and the main computer is configured to receive data from digital and analog sensors. 7. The system according to claim 5, characterized in that the main computer is configured to inform the driver about the operation of the vehicle control and monitoring system using the display panel.

Images