KR102627908B1 - A monitoring system for safety of autonomous vehicle remote control - Google Patents

Abstract

The present invention relates to a monitoring system for ensuring stability when remotely controlling an autonomous vehicle. More specifically, the safety of controlling an autonomous vehicle is ensured through an independent monitoring system even when a dangerous command value is issued from the remote controller due to a delay in transmitting and receiving video information. When there is a delay in the video data transmission and reception process, the temporary manipulation space of the remote control target vehicle is calculated in real time based on the remote control parameter values, and the autonomous driving remote control can perform evasion and emergency stop in the vehicle. It is about a monitoring system for safety control.

Description

{A monitoring system for safety of autonomous vehicle remote control}

The present invention relates to a monitoring system to ensure stability when remotely controlling an autonomous vehicle. More specifically, it calculates the temporary manipulation space of the vehicle subject to remote control in real time based on remote control parameter values to reduce the risk to an acceptable level. It is about a monitoring system for safety control during remote control of autonomous driving.

Vehicle remote control technology not only makes it easier to respond to emergency situations when autonomous vehicles are driving, but also enables logistics transportation without spatial limitations, so it is used in various fields such as the cargo transportation industry.

In addition, with the development of autonomous driving technology, there is a need to ensure safety through test runs for commercialization of autonomous driving. In particular, during unmanned autonomous driving, autonomous driving may often be canceled, so the control center located at a distance The service is also expanding to provide remote control in case of emergency.

Meanwhile, during unmanned autonomous driving, autonomous driving may often be canceled, and services that provide remote control in case of emergency from a remote control center are expanding.

However, when general wireless communication (cellular) communication is used in the process of transmitting signals to or from autonomous vehicles, the safety of remote control is not sufficiently secured when communication delays occur intermittently. there is a problem.

In particular, if such a communication delay exists in the process of remote control based on video data, there is a possibility that the remote controller may issue an incorrect command value, and there is a problem that the safety of incorrect remote control cannot be sufficiently secured.

That is, in the remote control system of a conventional autonomous vehicle as shown in FIG. 1, especially with reference to the reception state of image data, the average delay time is about 14 ms at a bandwidth of 12 Mbps for camera images, and such communication delay As a result, reception of video data may be delayed during the remote control process.

In addition, when there is a delay in receiving video data as shown in FIG. 2, there is a delay in receiving video data even though the vehicle is moving normally, and there is a problem in that many dangerous situations in vehicle control occur. This communication delay occurs. It often occurs during the processing delay (Input) of the video transmitting device, communication bottleneck (Communication), and processing delay (Output) of the video receiving device.

That is, as shown in FIG. 3, looking at the problems in transmitting network video data and remotely controlling a vehicle, remote control mainly uses cellular wireless mobile data transmission methods such as 4G/5G, mobile communication methods. Due to its characteristics, it is difficult to expect a communication environment with consistent and stable delay times regardless of region and time, no matter how small the data size.

In addition, the IP video transmission method transmits video and audio through network IP rather than the existing baseband transmission method SDI/HDMI. In general, when packet loss or delayed pixels exist, real-time transmission of high-definition video data is difficult. To ensure performance, received data is displayed first, so there is a high possibility of delays occurring in each layer while other data is processed as an afterimage.

Korean Patent Publication No. 10-2021-0103377

The purpose of the present invention to solve the above problems is to improve the safety of autonomous vehicle control through an independent monitoring system even when a dangerous command value is issued from a remote controller due to a delay in transmitting and receiving image information.

In addition, if there is a delay in the video data transmission and reception process, there is a possibility of remote control being performed based on incorrect environmental data, so if there is a risk source in the temporary operation space, a system that can perform avoidance and emergency stop inside the vehicle The purpose is to provide.

The technical problem to be achieved by the present invention is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

The configuration of the monitoring system for safety control during remote control of autonomous driving according to the present invention to achieve the above object is an autonomous driving control unit including an autonomous driving control unit provided in the autonomous vehicle and a remote control unit provided outside the autonomous vehicle. In the monitoring system for safety control during remote control, the autonomous driving control unit includes: an image data acquisition unit that measures various image data; an image transmitting unit that synthesizes the acquired image data and provides it to the remote control unit; An environmental data acquisition unit that measures various environmental data; A real-time location measurement module that measures the movement and location of the autonomous vehicle; an autonomous driving PC that generates comprehensive location data by combining the acquired environmental data and measured real-time location data; and a communication module providing the generated comprehensive location data to the remote control unit. It is characterized by including.

In addition, it is preferable that data is provided by the video transmitter and the communication module to the remote control unit through a communication terminal.

Meanwhile, the remote control unit includes a data receiving unit that receives a data signal provided from the communication terminal; an image display unit that displays the image data provided from the image transmitter and the comprehensive location data provided from the communication module; and a central processing unit that determines whether an obstacle exists and whether remote control continues based on the image data and the comprehensive location data. It is characterized by including.

In addition, the central processing unit temporarily suspends remote control of the autonomous vehicle according to the determination result of the presence of an obstacle and whether remote control continues, and sends a signal for an emergency stop or evasive maneuver to the autonomous vehicle. It is characterized in that it is transmitted to the autonomous driving control unit.

In addition, the central processing unit is configured to calculate a temporary manipulation space when an obstacle exists near the autonomous vehicle according to the image data and the comprehensive location data.

In addition, if there are no obstacles near the autonomous vehicle according to the image data and the comprehensive location data, an emergency action system is implemented.

Meanwhile, a remote safety control method for an autonomous vehicle, which is another embodiment of the present invention, includes the steps of disabling the autonomous driving mode of the autonomous vehicle; Implementing a remote control mode of the autonomous vehicle; generating the image data and comprehensive location data by the autonomous driving control unit; Transmitting the image data and the comprehensive location data from the autonomous driving control unit to the remote control unit; measuring a delay time of data transmitted from the autonomous driving control unit to the remote control unit; The remote control unit determining whether the delay time exceeds a preset standard value; It is characterized by including.

In addition, when the delay time exceeds a preset standard value, the central processing unit is configured to determine whether an obstacle exists near the autonomous vehicle according to the image data and the comprehensive location data.

In addition, when the central processing unit determines that an obstacle exists near the autonomous vehicle according to the image data and the comprehensive location data, the temporary manipulation space is calculated.

In addition, the central processing unit determines whether the calculated temporary manipulation space is invaded, temporarily suspends remote control of the autonomous vehicle, and sends a signal for an emergency stop or evasive maneuver to the autonomous vehicle. It is characterized in that it is transmitted to the control unit.

In addition, when the central processing unit determines that there is no obstacle in the vicinity of the autonomous vehicle and that it does not invade the temporary manipulation space, according to the image data and the comprehensive location data, an emergency action system is implemented. It is characterized by being done so as to do so.

In addition, the emergency response system is preferably a risk-minimizing operation or minimum risk operation (MRM) in an autonomous driving system.

The effect of the present invention according to the above configuration has the advantage of enabling stable communication and safe remote control through delay time stabilization considering the safety integrity level of the general automobile industry, and through this, autonomous driving in an unmanned autonomous vehicle is possible. Even when released, safety of operation can be ensured.

In addition, since wireless communication can be carried out in two directions rather than one way, even feedback on the state of the unmanned vehicle can be included in securing communication stability, and the remote driver can understand and evaluate the situation without problems, enabling the autonomous vehicle to operate autonomously. It has the advantage of being able to easily respond to emergency situations while driving.

The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

Figures 1 to 3 are diagrams illustrating a network video transmission method and problems in remote control when remotely controlling an autonomous vehicle according to the prior art.
Figure 4 is a configuration diagram showing an embodiment of the monitoring system for safety control during remote control of autonomous driving according to the present invention.
Figure 5 is a diagram showing the structure of remote control CAN communication according to an embodiment of the present invention.
Figure 6 is a diagram showing a remote control SW architecture according to an embodiment of the present invention.
Figure 7 is a flowchart showing a remote safety control method for an autonomous vehicle according to an embodiment of the present invention.
Figures 8 and 9 are graphs and schematic diagrams showing the avoidance and emergency stop process using a temporary manipulation space according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the attached drawings. However, the present invention may be implemented in various different forms and, therefore, is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected (connected, contacted, combined)" with another part, this means not only "directly connected" but also "indirectly connected" with another member in between. "Includes cases where it is. Additionally, when a part is said to “include” a certain component, this does not mean that other components are excluded, but that other components can be added, unless specifically stated to the contrary.

The terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

The present invention relates to a monitoring system for ensuring stability when remotely controlling an autonomous vehicle. More specifically, the safety of controlling an autonomous vehicle is ensured through an independent monitoring system even when a dangerous command value is issued from the remote controller due to a delay in transmitting and receiving video information. When there is a delay in the video data transmission and reception process, the temporary manipulation space of the remote control target vehicle is calculated in real time based on the remote control parameter values, and the autonomous driving remote control can perform evasion and emergency stop in the vehicle. It is about a monitoring system for safety control.

Referring to FIG. 4, which is a configuration diagram showing an embodiment of a monitoring system for safety control during remote control of autonomous driving, the present invention relates to an autonomous driving control unit provided in an autonomous vehicle and an autonomous driving control unit provided outside the autonomous vehicle. It includes a remote control unit, wherein the autonomous driving control unit includes an image data acquisition unit that measures various image data; an image transmitting unit that synthesizes the acquired image data and provides it to the remote control unit; An environmental data acquisition unit that measures various environmental data; A real-time location measurement module that measures the movement and location of the autonomous vehicle; an autonomous driving PC that generates comprehensive location data by combining the acquired environmental data and measured real-time location data; and a communication module providing the generated comprehensive location data to the remote control unit. It is characterized by including.

To describe the autonomous driving control unit provided in an autonomous vehicle in more detail, it is desirable for the image data acquisition unit (V-WFH-L) to be installed in many locations of the vehicle to acquire as much image data as possible from around the vehicle.

In this embodiment, the image data acquisition unit is shown in four locations: the front, rear, left front, and rear front of the vehicle, but it is not limited to this, and image data is acquired in at least two or as many locations as possible, if necessary. Additional parts may be installed.

In addition, the image data acquisition unit may include analog and digital cameras that acquire general images and videos, as well as infrared cameras to increase nighttime visibility, and various image data acquisition units such as 2D or 3D cameras may be used.

In this way, the video data acquired from various locations of the autonomous vehicle are synthesized by the video transmitter and provided to the remote control unit, and the video display unit included in the remote control unit displays each video data provided from the video transmitter. You can.

In this embodiment, it is shown that the image data acquired from the four image data acquisition units are displayed on four different monitors, but depending on the configuration of the remote control unit, they can also be comprehensively displayed on one monitor, and can be displayed inside the vehicle and at a remote location. After configuring the same, each video information can be displayed.

Meanwhile, in addition to such video data, the autonomous driving control unit further includes an environmental data acquisition unit that measures various environmental data and a real-time location measurement module that measures the movement and location of the autonomous vehicle. Representative examples of this environmental data acquisition unit include: Lidar (Velodyne VLP-16) can be used, and a global satellite navigation system such as GNSS/IMU can be used as a real-time position measurement module.

A plurality of environmental data acquisition units such as LIDAR can also be installed in various locations of the autonomous vehicle as needed, and in this embodiment, two units are installed each at the front and rear of the vehicle.

These environmental data and real-time location data are each collected by the self-driving PC. The self-driving PC synthesizes the collected data to generate comprehensive location data, and the generated comprehensive location data is ultimately provided to the remote control unit through the communication module. do.

The video data and environmental data described above are provided to the remote control unit by the video transmitter and communication module. The data can be transmitted to the remote control unit through a communication terminal, and the communication at this time uses various communication environments such as 5G/4G. It can be achieved through.

According to the remote control CAN communication structure of the present invention as shown in Figure 5, a VCU for basic control of the vehicle, a BMS module for battery management, a transmission/reception VCU module for remote control, an inverter for transforming, and control for autonomous driving. It may include a PC, etc., and can receive data from each node and transmit it.

Meanwhile, a remote control unit located at a certain distance outside the vehicle receives various image data and environmental data obtained from the autonomous driving control unit and enables remote control.

This remote control unit includes a data receiving unit that receives the data signal provided from the communication terminal, an image display unit that displays the image data provided from the video transmitter and comprehensive location data provided by the communication module, and an obstacle detection unit according to the image data and comprehensive location data. It is characterized by including a central processing unit that determines the presence or absence of remote control.

The data receiver and video display unit are as described above, and in particular, the remote control unit can sequentially judge them and issue commands according to each standard when conducting remote control based on various image data and environmental data obtained from the autonomous driving control unit. there is.

That is, referring to FIG. 7 showing a remote safety control method of an autonomous vehicle according to an embodiment of the present invention, the remote safety control method of an autonomous vehicle performed by a central processing unit included in the remote control unit includes autonomous driving. Deactivating the autonomous driving mode of the vehicle; Implementing a remote control mode of the autonomous vehicle; generating the image data and comprehensive location data by the autonomous driving control unit; Transmitting the image data and the comprehensive location data from the autonomous driving control unit to the remote control unit; measuring a delay time of data transmitted from the autonomous driving control unit to the remote control unit; The remote control unit determining whether the delay time exceeds a preset standard value; may include.

At the stage before starting remote control in the remote control unit, it is first determined whether the autonomous driving state of the autonomous vehicle is in progress or has been canceled. If the autonomous driving state is in progress, autonomous driving continues without being converted to remote control mode. The state continues, and if the autonomous driving state is canceled, it switches to remote control mode outside the vehicle.

After being converted to this remote control mode, the remote control unit transmits/receives various video information, environmental information, and control information necessary for remote control. Typically, the autonomous driving control unit as described above transmits and receives video data and comprehensive information. After generating the location data, it is desirable for the image data and comprehensive location data to be transmitted from the autonomous driving control unit to the remote control unit.

At this time, in the process of transmitting data from the autonomous driving control unit to the remote control unit, the transmitted data may be delayed unexpectedly, and due to the nature of the mobile communication method, no matter how small the data size, the delay is consistent and stable regardless of region/time. Since a communication environment with time cannot be expected, if data transmission is delayed, there is a possibility that the remote controller may issue an incorrect command value, which may lead to problems in which the safety of incorrect remote control is not sufficiently secured.

Therefore, it is essential to measure the delay time in the video data transmission process, and it is determined whether this delay time is above the Intelligent Transport Systems (ITS) standard in the automobile/road traffic field.

If the delay time is below the ITS standard, video data for remote control is received smoothly, so various data for remote control are continuously transmitted and received. Conversely, if the delay time is above the ITS standard, video data for remote control is continuously transmitted. Since reception is not smooth, it is necessary to prevent or control problems that may occur during remote control.

That is, if the delay time exceeds a preset ITS standard value, the central processing unit may be configured to determine whether an obstacle exists near the autonomous vehicle based on image data and comprehensive location data.

If there are no obstacles, it is likely that the situation does not require an immediate emergency stop or evasive maneuver, so a general emergency response system can be implemented. It is desirable for this emergency response system to be a risk-minimizing operation or minimal risk operation (MRM: Minimum Risk Maneuver) in an autonomous driving system.

However, if the presence of an obstacle is recognized in the image data and environmental data obtained from the autonomous driving control unit and this data is received and confirmed by the remote control unit, it is highly likely that an immediate emergency stop or evasive maneuver is required.

Accordingly, in the present invention, if an obstacle exists near an autonomous vehicle according to image data and comprehensive location data, the temporary manipulation space can be calculated, and avoidance and emergency stop using this vehicle temporary manipulation space are also provided. It is shown in Figures 8 and 9.

That is, when the central processing unit determines that an obstacle exists near the autonomous vehicle according to the image data and the comprehensive location data, the temporary manipulation space is calculated.

The central processing unit determines whether the calculated temporary manipulation space is invaded, temporarily suspends remote control of the autonomous vehicle, and transmits a signal for emergency stop or evasive maneuver for the autonomous vehicle to the autonomous driving control unit. It is done so that

Each variable in the graph shown in FIG. 8 represents the following values.

That is, in the present invention, based on the delay time of the vehicle's remote control and the time it takes to recognize and stop it, the tentative manipulation space of the autonomous vehicle using the above remote control parameter values is calculated in real time, and the calculated result is Accordingly, there is an advantage in that the risk on the dangerous side can be reduced to an acceptable level by performing an emergency stop on the autonomous vehicle or controlling it to make an evasive maneuver in a dangerous area through remote control.

The avoidance and emergency stop method through the temporary operation space during remote control as in the present invention is calculated on the assumption that there is a self-diagnosis function to satisfy the safety level that an autonomous vehicle must have, and that the autonomous vehicle has a self-diagnosis function. It is a collision avoidance algorithm that can be implemented when existing safety auxiliary sensors operate normally.

Therefore, according to the present invention, not only can the safety of autonomous vehicles be improved through an independent monitoring system even if a dangerous command value is issued from the remote controller due to video information delay, but also the safety of the autonomous vehicle can be improved through avoidance actions taking into account the safety integrity level of the automobile industry. It has the advantage of stable communication and safe remote control.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the patent claims described below, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention.

Claims (12)

In the monitoring system for safety control during remote control of autonomous driving, which includes an autonomous driving control unit provided in the autonomous vehicle and a remote control unit provided outside the autonomous vehicle,
The autonomous driving control unit,
An image data acquisition unit that measures various image data;
an image transmitting unit that synthesizes the acquired image data and provides it to the remote control unit;
An environmental data acquisition unit that measures various environmental data;
A real-time location measurement module that measures the movement and location of the autonomous vehicle;
an autonomous driving PC that generates comprehensive location data by combining the acquired environmental data and measured real-time location data; and
It includes a communication module that provides the generated comprehensive location data to the remote control unit,
The remote control unit includes a central processing unit, and the central processing unit,
Determine whether to continue remote control according to the image data and the comprehensive location data,
When remote control continues, the delay time of data transmitted from the autonomous driving control unit to the remote control unit is measured,
If the delay time exceeds a preset standard value, determine whether an obstacle exists in the vicinity of the autonomous vehicle,
When an obstacle exists in the vicinity of the autonomous vehicle, the movement of the obstacle for each of the vehicle's self-diagnosis delay time, vehicle communication delay time, time from start to end of vehicle stopping operation, and time interval generated by the operation margin. Based on the speed and moving speed of the vehicle, the temporary manipulation space of the autonomous vehicle is calculated in real time,
When the obstacle invades the calculated temporary manipulation space, temporarily stopping remote control of the autonomous vehicle and transmitting a signal for an emergency stop or evasive maneuver for the autonomous vehicle to the autonomous driving control unit. A monitoring system for safety control during remote control of autonomous driving.
According to paragraph 1,
A monitoring system for safety control during remote control of autonomous driving, characterized in that data provided by the video transmitter and the communication module is provided to the remote control unit through a communication terminal.
According to paragraph 2,
The remote control unit,
a data receiving unit that receives a data signal provided from the communication terminal; and
A monitoring system for safety control during remote control of autonomous driving, further comprising a video display unit that displays the video data provided from the video transmitter and the comprehensive location data provided from the communication module.
delete delete According to paragraph 3,
A monitoring system for safety control during remote control of autonomous driving, characterized in that an emergency action system is implemented when there are no obstacles near the autonomous vehicle according to the image data and the comprehensive location data.
In the remote safety control method of an autonomous vehicle using a monitoring system for safety control during remote control of autonomous vehicle of claim 1,
Deactivating the autonomous driving mode of the autonomous vehicle;
Implementing a remote control mode of the autonomous vehicle;
generating the image data and comprehensive location data by the autonomous driving control unit;
Transmitting the image data and the comprehensive location data from the autonomous driving control unit to the remote control unit;
measuring a delay time of data transmitted from the autonomous driving control unit to the remote control unit;
The remote control unit determining whether the delay time exceeds a preset standard value;
When the delay time exceeds a preset standard value, the central processing unit determines whether an obstacle exists near the autonomous vehicle according to the image data and the comprehensive location data;
When the central processing unit determines that an obstacle exists near the autonomous vehicle, the vehicle's self-diagnosis delay time, vehicle communication delay time, time from start to end of vehicle stopping operation, and time interval generated by the operation margin, respectively. Calculating in real time the temporary manipulation space of the autonomous vehicle based on the moving speed of the obstacle and the moving speed of the vehicle; and
When the obstacle invades the calculated temporary manipulation space, temporarily stopping remote control of the autonomous vehicle and transmitting a signal for an emergency stop or evasive maneuver for the autonomous vehicle to the autonomous driving control unit. A remote safety control method for an autonomous vehicle, comprising:
delete delete delete In clause 7,
According to the image data and the comprehensive location data, when the central processing unit determines that there is no obstacle in the vicinity of the autonomous vehicle and that it does not invade the temporary manipulation space, an emergency action system is implemented. A remote safety control method for an autonomous vehicle, characterized in that.
According to clause 11,
The emergency action system is a remote safety control method for an autonomous vehicle, characterized in that risk minimization operation or minimum risk operation (MRM: Minimum Risk Maneuver) in an autonomous vehicle system.

Images