KR20240046095A - Safe remote control system for unmanned vehicles using edge computing and artificial intelligence - Google Patents

Abstract

The present invention relates to an unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology. More specifically, it is possible to safely control the driving and operation of an unmanned vehicle on the ground remotely by installing artificial intelligence technology on an edge computing module. This is about a remote driving control system for unmanned vehicles based on computing and artificial intelligence convergence technology.
The remote driving control system for an unmanned vehicle based on edge computing and artificial intelligence convergence technology according to the present invention includes an unmanned vehicle capable of driving with a motor as a power source; A sensor unit equipped with a plurality of sensors capable of obtaining driving information of the unmanned vehicle and environmental information around the unmanned vehicle; an interface capable of receiving various information from the sensor unit and controlling the unmanned vehicle according to control signals; And transmitting and receiving information to the interface unit remotely using a cloud network or wireless communication, analyzing and monitoring the information received from the interface, and transmitting a control signal capable of controlling the driving of the unmanned vehicle to the interface unit. Includes a remote control unit, wherein the remote control unit is composed of a ground server or an edge computer, and recognizes the driving environment including obstacles, objects, lanes, driving areas, and road surface conditions from the information received from the interface, and recognizes the driving environment. Depending on the driving environment, artificial intelligence techniques are used to determine situational response, including collision avoidance, response to emergency situations, and implementation of designated special purpose tasks, and generate control signals corresponding to the determined situational response.

Description

Safe remote control system for unmanned vehicles using edge computing and artificial intelligence}

The present invention relates to an unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology. More specifically, it is possible to safely control the driving and operation of an unmanned vehicle on the ground remotely by installing artificial intelligence technology on an edge computing module. This is about a remote driving control system for unmanned vehicles based on computing and artificial intelligence convergence technology.

Recently, research has been actively conducted on autonomous vehicles or remote control vehicles that run without a driver or without directly operating the vehicle even if there is a driver.

In particular, the demand for unmanned delivery vehicles, special-purpose unmanned vehicles (construction machinery, cleaning trucks, etc.), and robots is rapidly increasing due to the recent COVID-19 period, population decline, lack of skilled manpower, and safety issues.

As disclosed in Korean Patent Publication Nos. 10-2016-0050957 and 10-2019-0135390, an autonomous vehicle is a vehicle in which driving, stopping, turning, acceleration or deceleration are automatically performed by a computer. , A remote control driving vehicle is a vehicle whose driving is controlled from a distant location, as disclosed in Republic of Korea Patent Publication No. 10-2012-0076830.

Self-driving vehicles are equipped with sensors that recognize the surrounding environment, including the surrounding terrain and location, to maintain driving lanes, secure a safe distance from adjacent vehicles, detect nearby obstacles and avoid collisions, and determine vehicle speed according to traffic conditions or road conditions. Control is performed automatically, and the speed and steering of the remote control vehicle are controlled by control commands transmitted through a communication network from a remote location.

However, existing special-purpose work vehicles rely on the driver's experience and vision, or in the case of unmanned vehicles, rely on onboard computing techniques, resulting in lack of experience of the experienced person, lack of computing power of the unmanned vehicle, and sensor errors. In many cases, malfunctions and poor response could lead to accidents.

These remote driving control technologies are integrated to fulfill multiple missions such as multiple sensors, remote control speed, remote control driving precision, communication speed for control, safety functions for emergency stops, distance recognition precision, and semantic map generation. To be considered, the purpose of the present invention is to provide a system that can safely control the driving and operation of an unmanned vehicle on the ground remotely by mounting artificial intelligence technology on an edge computing module.

In order to achieve the above object, an unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to the present invention includes an unmanned vehicle capable of traveling with a motor as a power source; A sensor unit equipped with a plurality of sensors capable of obtaining driving information of the unmanned vehicle and environmental information around the unmanned vehicle; an interface capable of receiving various information from the sensor unit and controlling the unmanned vehicle according to control signals; And transmitting and receiving information to the interface unit remotely using a cloud network or wireless communication, analyzing and monitoring the information received from the interface, and transmitting a control signal capable of controlling the driving of the unmanned vehicle to the interface unit. Includes a remote control unit, wherein the remote control unit is composed of a ground server or an edge computer, and recognizes the driving environment including obstacles, objects, lanes, driving areas, and road surface conditions from the information received from the interface, and recognizes the driving environment. Depending on the driving environment, artificial intelligence techniques are used to determine situational response, including collision avoidance, response to emergency situations, and implementation of designated special purpose tasks, and generate control signals corresponding to the determined situational response.

Here, the remote control unit applies a deep learning-based mapless local motion planning technique as an artificial intelligence technique used to determine situational response for collision avoidance, and the deep learning-based mapless local motion planning technique is a semantic and Estimating the drivable area of the unmanned vehicle using segmentation, training an encoder-decoder model using transfer learning technology, and using the encoder-decoder model to estimate the drivable area of the unmanned vehicle using the semantic segmentation. It is characterized by being configured to divide the space and make situational response decisions to avoid dynamic and static collisions using a nonlinear model predictive control (NMPC) method.

Here, the sensor unit may include a GPS capable of acquiring driving information, an Around View Monitor (AVM) capable of capturing images around the unmanned vehicle, a 3D LiDAR sensor capable of acquiring information on surrounding obstacles around the unmanned vehicle, or It is characterized by including a RADAR sensor.

Here, the remote control unit displays the current location, overlaid driving path, driving speed, altitude, and direction information on the map based on driving information obtained from the GPS, and images captured by the around view monitor (AVM). And, a monitoring device that displays information on surrounding obstacles obtained through the 3D LiDAR sensor or RADAR sensor is provided.

Here, the remote control unit includes a goal setting function for setting a target route using waypoints based on driving information including the current location and driving route acquired through the sensor unit, and a driving function obtained through the sensor unit. A target speed setting function that sets the target speed based on driving information including speed, and a function that determines whether driving is safe through surrounding obstacle information acquired through the sensor unit and stops the driving of the unmanned vehicle if it is determined to be unsafe. It is characterized by including an emergency stop function.

Here, the remote control unit is characterized in that it is capable of manual driving control of the unmanned vehicle by a user or autonomous driving control that allows the unmanned vehicle to autonomously move along a determined travel path.

With the above configuration, the unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to the present invention has multiple sensors, remote control speed, remote control driving precision, communication speed for control, and safety functions for emergency stop, etc. , it has the advantage of being able to satisfy a number of missions required for remote driving control of unmanned vehicles, such as distance recognition precision and semantic map generation.

Figure 1 is a configuration diagram of an unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to an embodiment of the present invention.
Figure 2 is an example diagram showing a state in which a sensor unit is mounted on an unmanned mobile device according to an embodiment of the present invention.
Figure 3 is an exemplary communication and cloud configuration for remote control according to an embodiment of the present invention.
Figure 4 is a configuration diagram of an unmanned vehicle operating system and interface for remote control according to an embodiment of the present invention.
Figure 5 is an exemplary monitor device of the remote control unit according to an embodiment of the present invention.
Figure 6 is a power and data flow diagram of an unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to an embodiment of the present invention.
Figure 7 is a software environment configuration diagram of the remote control unit according to an embodiment of the present invention.
Figure 8 is an example of an autonomous driving algorithm for an unmanned vehicle according to an embodiment of the present invention.
Figure 9 is an example of determining the driving state of an unmanned vehicle through driving information and environmental information obtained from the sensor unit according to an embodiment of the present invention.
Figure 10 is an example of speed and steering control of an unmanned vehicle according to an embodiment of the present invention.
Figure 11 is an example of estimating the drivable area of an unmanned vehicle using semantic segmentation in a remote control unit according to an embodiment of the present invention.
Figure 12 is a conceptual diagram of transfer learning technology used in the remote control unit according to an embodiment of the present invention.
Figure 13 is an overall block diagram of the artificial intelligence algorithm of the remote control unit according to an embodiment of the present invention.

Hereinafter, the unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to the present invention will be described in more detail with reference to the embodiments shown in the drawings.

Figure 1 is a configuration diagram of a remote driving control system for an unmanned vehicle based on edge computing and artificial intelligence fusion technology according to an embodiment of the present invention, and Figure 2 shows a state in which a sensor unit is mounted on an unmanned vehicle according to an embodiment of the present invention. It is an example diagram, and Figure 3 is an example diagram of communication and cloud configuration for remote control according to an embodiment of the present invention, and Figure 4 is an unmanned vehicle operating system and interface for remote control according to an embodiment of the present invention. It is a configuration diagram, and Figure 5 is an exemplary diagram of a monitoring device of a remote control unit according to an embodiment of the present invention, and Figure 6 is a diagram of an unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to an embodiment of the present invention. It is a power and data flow diagram, Figure 7 is a software environment configuration diagram of the remote control unit according to an embodiment of the present invention, Figure 8 is an example diagram of an autonomous driving algorithm of an unmanned vehicle according to an embodiment of the present invention, and Figure 9 is It is an example diagram of determining the driving state of an unmanned vehicle through driving information and environmental information obtained from a sensor unit according to an embodiment of the present invention, and Figure 10 shows speed and steering control of an unmanned vehicle according to an embodiment of the present invention. It is an exemplary diagram, and FIG. 11 is an exemplary diagram of estimating the drivable area of an unmanned vehicle using semantic segmentation in a remote control unit according to an embodiment of the present invention, and FIG. 12 is a remote control unit according to an embodiment of the present invention. This is a conceptual diagram of the transfer learning technology used in, and Figure 13 is an overall block diagram of the artificial intelligence algorithm of the remote control unit according to an embodiment of the present invention.

Looking at Figure 1, the unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to an embodiment of the present invention includes an unmanned vehicle 10, a sensor unit 20, an interface unit 30, and a cloud. It includes a network or wireless communication (40) and a remote control unit (50).

This proposed technology is an edge computing (GPU, NPU) platform that allows remote ground work on special-purpose work environment information data in addition to real-time object recognition, lane recognition, driving area recognition, and road surface condition recognition, which are a class of artificial intelligence machine learning techniques. Uncertain and dangerous work environments are recognized in advance through transmitted and installed artificial intelligence techniques, and automatic motion control of work vehicles or unmanned vehicles is performed using remotely recognized environmental information, as shown in Figure 3. It is a technology that can be implemented based on cloud and telecommunication, where both manual and remote control are possible.

Remote safe driving technology is linked to the cloud environment and can be controlled from a remote location. At the same time, there is no need to board a work vehicle in an environment near the work space, and the entire mission is performed automatically or manually based on the content displayed on the screen. It is designed to do this and has the advantage of increasing work efficiency.

The unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to the present invention has multiple sensors, remote control speed, remote control driving precision, communication speed for control, safety functions for emergency stops, distance recognition precision, and semantics. This is to fulfill a number of required missions such as map creation.

The unmanned vehicle 10 is configured to be capable of traveling using a motor as a power source.

The sensor unit 20 is equipped with a plurality of sensors capable of obtaining driving information of the unmanned vehicle 10 and environmental information around the unmanned vehicle 10.

In one embodiment of the present invention, the sensor unit 20 is equipped with a GPS capable of obtaining driving information as shown in FIGS. 1 and 2, and an unmanned mobile device 10 to capture images around the unmanned mobile device 10. A plurality of Around View Monitors (AVM) mounted around the outer edge of the unmanned vehicle 10, and a 3D LiDAR installed at the upper center of the unmanned vehicle 10 to enable acquisition of information on surrounding obstacles around the unmanned vehicle 10. It may include a sensor or a RADAR sensor installed at the front and rear of the unmanned vehicle 10.

Obstacle information obtained from the 3D LiDAR sensor or RADAR sensor includes the distance of surrounding objects from the unmanned vehicle 10 and the speed of objects approaching the unmanned vehicle 10.

Additionally, the sensor unit 20 may further include a Camera (Stereo, RGB), IMU sensor, Wheel Encoder, and other sensors.

The interface unit 30 is configured to receive various information from the sensor unit 20 and control the unmanned vehicle 10 according to control signals, and is a remote control unit to be described later through a cloud network or wireless communication 40. It is configured to transmit and receive information with (50).

In one embodiment of the present invention, the interface unit 30 can execute commands based on the Linux OS (or other OS) in the vehicle when receiving a remote command for remote safe driving as shown in FIG. 4. The API for logic and interlocking control was implemented based on ROS, and a separate driver that could control the motor through ROS was designed for platform motion control.

In particular, it has the advantage of increasing both work efficiency and stability through automatic recognition of worker tasks and work spaces based on artificial intelligence techniques installed in the edge computing platform.

It is designed to automatically send a signal from the operator's edge computer screen when danger is detected or separate work is required, and to be equipped with a function to remotely control the vehicle platform manually based on the detected signal.

The cloud network or wireless communication 40 is a configuration that allows the interface unit 30 and the remote control unit 50 to transmit and receive information to each other through wireless communication.

The cloud network or wireless communication 40 may use various communication means such as LTE, 5G, WiFi, and Radio Signal.

The remote control unit 50 transmits and receives information to and from the interface unit 30 remotely using a cloud network or wireless communication 40, analyzes and monitors the information received from the interface 30, and controls the unmanned mobile device. This is a configuration that transmits a control signal that can control the driving of (10) to the interface unit (30).

In one embodiment of the present invention, the remote control unit 50 is composed of a ground server or an edge computer, and determines the driving environment including obstacles, objects, lanes, driving areas, and road surface conditions from the information received from the interface 30. It is configured to recognize and use artificial intelligence techniques according to the recognized driving environment to determine situational response, including collision avoidance, response to emergency situations, and implementation of designated special purpose tasks, and to generate control signals corresponding to the determined situational response.

In relation to this, FIG. 8 illustrates the autonomous driving algorithm in the remote control unit 50, FIG. 9 illustrates a state for determining the driving state of the unmanned vehicle, and FIG. 10 illustrates a state for controlling the speed and steering of the unmanned vehicle. is exemplified.

In addition, the unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology according to the present invention uses deep learning-based mapless as an artificial intelligence technique used to determine situational response to collision avoidance of the remote control unit 50. Local motion planning techniques can be applied.

The deep learning-based mapless local motion planning technique first estimates the drivable area of the unmanned vehicle 10 using semantic segmentation.

Figure 11 illustrates virtual laser scanning after estimating the drivable area of the unmanned vehicle 10 using semantic segmentation, and Figure 13B shows the configuration of the perception module based on semantic segmentation. is exemplified.

Additionally, an encoder-decoder model is trained using transfer learning technology, and the encoder-decoder model is used to segment the free space from the drivable area estimated through the semantic segmentation.

Figure 12 shows a conceptual diagram of training an encoder-decoder model using transfer learning technology, which increases the generalization ability of the encoder-decoder model by using a transfer learning technology that uses several frames in each scenario. It is for.

Afterwards, the nonlinear model predictive control (NMPC) method is used to determine situational response to avoid dynamic and static collisions.

Figure 13C illustrates the configuration of a local motion planning module based on nonlinear model predictive control (NMPC). Since the nonlinear model predictive control (NMPC) method itself is a known technology, detailed description below will be omitted. Do this.

As described above, training time can be reduced by the artificial intelligence technique according to the present invention, transferability from simulation to reality can be increased through transfer learning technology, and the estimated drivable free space does not increase the computational complexity of the problem, By being used with the relaxed barrier function in the NMPC formulation, the unmanned vehicle 10 can automatically discover complex movements and avoid cluttered scenes with moving obstacles, thereby improving stability and autonomy.

In addition, in one embodiment of the present invention, the remote control unit 50 displays the current location and the overlaid driving path, driving speed, altitude, and direction on the map based on the driving information obtained from the GPS, as shown in FIG. 5. A monitoring device is provided that displays information, images captured by the around view monitor (AVM), and surrounding obstacle information acquired through the 3D LiDAR sensor or RADAR sensor.

In addition, in one embodiment of the present invention, the remote control unit 50 sets a target route using waypoints based on driving information including the current location and driving route acquired through GPS of the sensor unit 20. A goal setting function that sets a target speed, a target speed setting function that sets a target speed based on driving information including the driving speed acquired through GPS of the sensor unit 20, and a 3D LIDAR of the sensor unit 20 It determines whether driving is safe through information on surrounding obstacles obtained through a LiDAR sensor or RADAR sensor, and includes an emergency stop function that stops the driving of the unmanned vehicle if it is determined to be unsafe.

In addition, in one embodiment of the present invention, the remote control unit 50 enables manual driving control of the unmanned vehicle 10 by a user, and enables the unmanned vehicle 10 to autonomously move a determined travel path. Of course, driving control is also possible.

Most existing remote control products receive camera or sensor information and manually operate basic remote control based on work environment monitoring.

In addition, existing autonomous driving control techniques are aimed at motion control and collision avoidance for autonomous driving and take the form of performing onboard control of the platform by installing algorithms inside the vehicle.

However, this technology is a new technology that processes edge computing-based artificial intelligence algorithms on the ground based on work environment data remotely and improves both stability and efficiency of work performance so that both autonomous work of special purpose vehicles and remote manual work can be performed. This is about intelligent remote safety control techniques.

The unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology described above and shown in the drawings is only one embodiment for carrying out the present invention, and should not be construed as limiting the technical idea of the present invention. The scope of protection of the present invention is determined only by the matters stated in the claims below, and improved and changed embodiments without departing from the gist of the present invention are obvious to those skilled in the art to which the present invention pertains. It will be said to fall within the scope of protection of the present invention.

10 Unmanned vehicle
20 sensor unit
30 Interface section
40 Cloud network or wireless communication
50 Remote control unit

Claims (6)

An unmanned vehicle capable of driving using a motor as a power source;
A sensor unit equipped with a plurality of sensors capable of obtaining driving information of the unmanned vehicle and environmental information around the unmanned vehicle;
an interface capable of receiving various information from the sensor unit and controlling the unmanned vehicle according to control signals; and
A remote device transmits and receives information to the interface unit remotely using a cloud network or wireless communication, analyzes and monitors the information received from the interface, and transmits a control signal to control the driving of the unmanned vehicle to the interface unit. Including a control unit;
The remote control unit,
Consists of a ground server or edge computer,
Recognizes the driving environment including obstacles, objects, lanes, driving areas, and road surface conditions from the information received from the interface, and uses artificial intelligence techniques according to the recognized driving environment to avoid collisions, respond to emergency situations, and perform designated special-purpose tasks. An unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology, which determines situational response, including implementation, and generates control signals corresponding to the determined situational response. According to paragraph 1,
The remote control unit,
A deep learning-based mapless local motion planning technique is applied as an artificial intelligence technique used to determine situational response to collision avoidance.
The deep learning-based mapless local motion planning technique estimates the drivable area of the unmanned vehicle using semantic segmentation, trains an encoder-decoder model using transfer learning technology, and trains the encoder-decoder model. An edge characterized by dividing the free space from the drivable area estimated through the semantic division and making situational response decisions to avoid dynamic and static collisions using a non-linear model predictive control (NMPC) method. Unmanned vehicle remote driving control system based on computing and artificial intelligence convergence technology. According to paragraph 1,
The sensor unit,
GPS that can acquire driving information, an Around View Monitor (AVM) that can capture images around the unmanned vehicle, and a 3D LiDAR sensor or RADAR sensor that can acquire information on surrounding obstacles around the unmanned vehicle. An unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology, characterized in that it includes. According to paragraph 3,
The remote control unit,
Based on the driving information obtained from the GPS, the current location, overlaid driving path, driving speed, altitude, and direction information are displayed on the map, and the image captured by the around view monitor (AVM) and the 3D LIDAR ( An unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology, which is equipped with a monitoring device that displays information on surrounding obstacles obtained through a LiDAR sensor or RADAR sensor. According to paragraph 3,
The remote control unit,
A goal setting function that sets a target route using waypoints based on driving information including the current location and driving path acquired through the sensor unit, and driving information including driving speed acquired through the sensor unit It includes a target speed setting function that sets the target speed, and an emergency stop function that determines whether driving is safe through surrounding obstacle information obtained through the sensor unit, but stops the driving of the unmanned vehicle if it is determined to be unsafe. An unmanned vehicle remote driving control system based on edge computing and artificial intelligence convergence technology. According to clause 5,
The remote control unit,
A remote driving control system for an unmanned vehicle based on edge computing and artificial intelligence convergence technology, characterized in that it is capable of manual driving control of the unmanned vehicle by a user or autonomous driving control that allows the unmanned vehicle to autonomously move a determined driving path.