KR102609489B1 - Obstacle detection system for autonomous vehicle or robot - Google Patents

Abstract

The present invention includes a sensor unit including a plurality of sensors for acquiring visual information and sound information; an encoder unit extracting compressed information of each data from the data acquired by the plurality of sensors; And machine learning is performed to restore the compressed spatial information to the spatial information of the obstacle with a spatial encoder using pre-given obstacle information, and spatial prediction information about the obstacle is generated from the compressed information of the visual information and sound information extracted through the encoder unit. It relates to an obstacle detection system for an autonomous vehicle or robot, including a spatial decoder that outputs.

Description

Obstacle detection system for autonomous vehicle or robot}

The present invention relates to an obstacle detection system that can detect obstacles that cannot be visually observed during autonomous driving of an autonomous vehicle or robot using visual and non-visual sensors.

As technology development for autonomous driving technology for vehicles or robots accelerates, technology to detect blind spots, i.e. obstacles that appear from spaces that cannot be observed through visual sensors such as lidar or cameras, and cause obstacles to autonomous vehicles or driving. It is also being developed in various ways.

For example, there is a technology that acquires spatial information from another vehicle or robot by receiving surrounding information from another vehicle or robot located in another space through communication means. According to this, it is possible to obtain not only spatial information about blind spots but also information about the surroundings of the space where a cooperative vehicle or robot capable of communicating with the vehicle or robot exists. However, this technology also has a limitation in that it requires a cooperative vehicle or robot to detect obstacles.

Another example is a technology that analyzes the position and speed of an object by emitting strong electromagnetic waves through a radar sensor and analyzing the reflected waves reflected from the object. In the case of this method, it is possible to observe objects located in blind spots by analyzing electromagnetic waves doubly reflected through objects such as surrounding walls, rather than analyzing electromagnetic waves directly reflected by objects in an alley. However, this technology has the disadvantage that not only is a radar sensor with a complex configuration used, but the detection method using sensing information is very complicated.

Registered Patent Publication No. 10-1149800 (2012.05.18)

The present invention was conceived in consideration of the above points, and aims to provide an obstacle detection system capable of detecting obstacles located in blind spots using acoustic information acquired using acoustic sensors such as microphones, which are simple sensors. Make it an assignment.

The technical problems to be achieved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

According to one embodiment of the present invention, a sensor unit including a plurality of sensors for acquiring visual information and sound information; an encoder unit extracting compressed information of each data from the data acquired by the plurality of sensors; And machine learning is performed to restore the compressed spatial information to the spatial information of the obstacle with a spatial encoder using pre-given obstacle information, and spatial prediction information about the obstacle is generated from the compressed information of the visual information and sound information extracted through the encoder unit. An obstacle detection system for an autonomous vehicle or robot, including a spatial decoder that outputs, is disclosed.

Additionally, the plurality of sensors may include lidar, cameras, and microphones.

In addition, the encoder unit includes a LiDAR encoder that extracts compressed LiDAR information using map information and LiDAR information acquired by the LiDAR; a vision encoder that extracts compressed vision information using map information and vision information acquired by the camera; and a sound encoder that extracts compressed sound information using map information and sound information acquired by the microphone.

Additionally, the spatial decoder may output spatial prediction information about the obstacle using an average value of the compressed LiDAR information, the compressed vision information, and the compressed sound information.

Additionally, the spatial prediction information about the obstacle may include one or more of the position, speed, acceleration, and future path of the predicted obstacle.

Additionally, the LIDAR encoder, the vision encoder, the sound encoder, and the spatial decoder may be learned by a machine learning unit through an autoencoder.

In addition, the machine learning unit unsupervisedly trains the LiDAR encoder and LiDAR decoder using pre-given LiDAR information and map information, and unsupervisedly trains the vision encoder and vision decoder using pre-given vision information and map information. The sound encoder and sound decoder can be trained unsupervised using pre-given acoustic information and map information, and the spatial encoder and spatial decoder can be unsupervised trained using pre-given obstacle information and map information. .

In addition, the machine learning unit may conduct learning in the direction of narrowing the mutual distance within the feature space of compressed information compressed through each of the LiDAR encoder, the vision encoder, the sound encoder, and the spatial encoder.

In addition, when acquisition of LiDAR information through the LiDAR is impossible, the LiDAR decoder restores the LiDAR information using compressed information of a vision encoder or sound encoder, and the vision decoder acquires vision information through the camera. When it is impossible to obtain sound information through the microphone, the vision information is restored using compressed information from the LiDAR encoder or the sound encoder, and when acquisition of sound information through the microphone is impossible, the sound decoder uses compressed information from the LiDAR encoder or the vision encoder. Thus, the sound information can be restored.

In addition, the spatial decoder may restore the obstacle information using compressed information of the sound encoder when it is impossible to obtain lidar information through the lidar and vision information through the camera.

According to an embodiment of the present invention, the compressed information of the visual sensor and the non-visual sensor is machine-learned to have similar values in the feature space, and by using this, the visual information and acoustic information acquired through the visual sensor and the non-visual sensor are used. It is effective in detecting obstacles in the surrounding space, including blind spots.

In addition, according to an embodiment of the present invention, since the compressed information of each sensor is learned to have similar values, even in a situation where measurement data cannot be acquired because a specific sensor is broken or cannot be used, information from other sensors can be restored. There are benefits to using it. In addition, even when there is no information from a specific sensor, there is an advantage that obstacle information can be predicted even if the accuracy is slightly lower.

1 is a conceptual diagram illustrating the operation of an obstacle detection system for an autonomous vehicle or robot according to an embodiment of the present invention.
2 is a block diagram of an obstacle detection system for an autonomous vehicle or robot according to an embodiment of the present invention.
Figures 3 to 6 are diagrams showing a machine learning method for the sensor-specific encoder and spatial decoder of Figure 2.
Figure 7 is a diagram showing the learning flow of the machine learning method shown in Figures 3 to 6.
Figures 8 and 9 are diagrams showing methods for restoring different types of sensor information using different types of encoders and decoders.

Since the present invention can be modified in various ways and can have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all transformations, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, embodiments of an obstacle detection system for an autonomous vehicle or robot according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components are denoted by the same drawing numbers. will be given and any redundant explanation will be omitted.

1 is a conceptual diagram illustrating the operation of an obstacle detection system for an autonomous vehicle or robot according to an embodiment of the present invention, and FIG. 2 is a conceptual diagram illustrating the operation of an obstacle detection system for an autonomous vehicle or robot according to an embodiment of the present invention. This is a block diagram.

The obstacle detection system according to this embodiment is mounted on an autonomous vehicle or robot (1), and is an obstacle located in an area that cannot be observed by the visual sensor of the autonomous vehicle or robot (1), that is, a blind spot (3). By detecting (2), spatial information of the obstacle (2), for example, location information and speed information of the obstacle (2) are predicted.

Referring to FIG. 2, the obstacle detection system according to this embodiment includes a sensor unit 100, an encoder unit 200, and a spatial decoder 300.

The sensor unit 100 includes a plurality of sensors for acquiring visual information and acoustic information, and these sensors include a visual sensor for acquiring visual information and a non-visual sensor for acquiring acoustic information. Visual sensors can be divided into active field-of-view sensors and passive field-of-view sensors. In the case of this embodiment, Lidar 110 is exemplified as an active in-field sensor among visual sensors, the camera 120 is exemplified as a passive in-field sensor, and the microphone 130 is exemplified as a non-visual sensor.

However, the sensors of the sensor unit 100 applicable to the present invention are not limited to the sensors illustrated in this embodiment, and may include various types of sensors, such as radar, ultrasonic sensors, and infrared cameras. You can.

The LIDAR 110 irradiates a laser to the surrounding space and acquires a point cloud from the reflected laser signal, and the camera 120 acquires a vision image by photographing the surrounding space. In addition, the microphone 130 senses sound in the surrounding audible band (KHz band) and outputs sound information.

The encoder unit 200 extracts compressed information for each data from data acquired by the plurality of sensors 110, 120, and 130. The encoder unit 200 is a LiDAR unit that extracts compressed LiDAR information 14 using map information 10 around the autonomous vehicle or robot 1 and LiDAR information 11 acquired by LiDAR 120. An encoder 210, a vision encoder 220 that extracts compressed vision information 15 using the map information 10 and the vision information 12 acquired by the camera 120, map information 10, and a microphone It includes a sound encoder 230 that extracts compressed sound information 16 using the acquired sound information 13.

The spatial decoder 300 uses pre-given obstacle information (e.g., obstacle position, speed, acceleration, future path, etc.) to compress spatial information 19 (see FIG. 6) together with the spatial encoder 310 (see FIG. 6). ) is machine learned to restore the spatial information 20 of the obstacle 2.

The spatial decoder 300 outputs spatial prediction information 18 about the obstacle 2 from the compressed information 14, 15, and 16 of the visual information and sound information extracted through the encoder unit 200. Here, the spatial prediction information 18 of the obstacle 2 includes the predicted position, speed, acceleration, future path, etc. of the obstacle 2.

The spatial decoder 300 predicts the space of the obstacle 2 using common compressed information 17 derived from the average value of compressed lidar information 14, compressed vision information 15, and the compressed acoustic information 16. Outputs information (18). At this time, map data around the autonomous vehicle or robot (1) can be used together.

Visual sensors such as LIDAR 110 and camera 120 have the advantage of being able to measure accurate distance and direction in obtaining spatial information because they use the straight-line nature of light, but have the disadvantage that blind spots always exist. there is. On the other hand, the acoustic information emitted by the obstacle 2 is easily diffracted, so there is an advantage in that data can be acquired without blind spots. However, it is difficult to measure the exact distance or direction of the acoustic information of the obstacle 2, but in the case of the obstacle detection system of the present invention, the encoder unit 200 and the spatial decoder 300 are trained through the machine learning unit to obtain visual information and audio. Using the information, accurate spatial prediction information 18 about the obstacle 2 can be output.

The machine learning unit can learn the LIDAR encoder, vision encoder 210, sound encoder 230, and spatial decoder 300 through an autoencoder, and FIGS. 3 to 6 show the sensor-specific encoder 210 of FIG. 2. , 220, 230) and the machine learning method of the spatial decoder 300.

Autoencoder is a deep learning network consisting of an encoder/decoder that can compress and restore data. An encoder compresses data and extracts the compressed information (or implicit information) contained in the data. The space where compressed information exists is called feature space, and the shorter the distance between information in the feature space, the more information it contains. The decoder restores the compressed information in the feature space to its original state. Since the compressed information in the feature space is not in a form that humans can understand, the information is restored to its original form that can be used by humans. The learned encoder and decoder can be used separately.

According to this embodiment, by adding map information 10 that is known in advance during machine learning using an autoencoder, more accurate information compression and restoration are possible. This is to provide additional conditions so that the autoencoder can learn better. In this embodiment, map information 10, which best represents the implied spatial information, is provided as an additional condition to facilitate learning more smoothly. You can make it progress.

The data required for machine learning through an autoencoder is pre-given lidar information (11, point cloud data), vision information (12, vision image), acoustic information (13), and map information around the autonomous vehicle or robot (1). (10), and includes obstacle information (20) within a certain range. At this time, the obstacle (2) includes the obstacle in the blind spot (3).

The above lidar information 11, vision information 12, acoustic information 13, map information 10, and obstacle information 20 used in machine learning may be provided in temporal synchronization with each other.

As shown in FIG. 3, the LiDAR encoder 210 and the LiDAR decoder 211 can be trained unsupervised through an autoencoder using pre-given LiDAR information 11 and map information 10, and as shown in FIG. 4 Likewise, the vision encoder 220 and the vision decoder 221 can be subjected to unsupervised learning using the previously given vision information 12 and map information 10. In addition, as shown in FIG. 5, the sound encoder 230 and the sound decoder 231 can be trained unsupervised using pre-given acoustic information 13 and map information 10, and as shown in FIG. 6, pre-given obstacles The spatial encoder 310 and the spatial decoder 300 can be subjected to unsupervised learning using the information 20 and map information 10.

Figure 7 shows the learning flow of the machine learning method shown in Figures 3 to 6.

Compressed information for each sensor, that is, compressed lidar information (14), compressed vision information (15), compressed acoustic information (16), and compressed spatial information (19), are each temporally synchronized and must have the same form. . However, because the LiDAR encoder 210, vision encoder 220, sound encoder 230, and spatial encoder 310 were individually learned from each other, as shown in (a) of FIG. 7, compressed LiDAR information 14 , Compressed vision information 15, compressed sound information 16, and compressed space information 19 are located at a distance from each other in the feature space. A large distance within the feature space means that they are not similar to each other.

However, since the four types of compressed information must actually be the same, the machine learning unit uses compressed lidar information (14), compressed vision information (15), compressed acoustic information (16), and compressed space information, as shown in (b) of Figure 7. (19) The learning proceeds in the direction of narrowing the mutual distance within the feature space. Specifically, by adding the cosine similarity between feature spaces to the loss function during learning, the encoder and decoder of each sensor can be learned correctly.

After learning is completed, the compressed LIDAR information (14), compressed vision information (15), compressed acoustic information (16), and compressed spatial information (19) compressed through each encoder (210, 220, 230, 310) are, As shown in (c) of Figure 7, they have almost common values, and this common compressed information can be viewed as information that properly compresses the spatial information at that point in time.

Referring again to FIG. 2, the LiDAR encoder 210, vision encoder 220, and sound encoder 230 use measurement information measured in real time through the sensor unit 100 to compress LiDAR information 14, Compressed vision information (15) and compressed sound information (16) are each extracted, averaged to obtain common compressed information (17), and passed through the spatial decoder (300) together with the map information (10) to determine the obstacle (2). Spatial prediction information 18 can be output.

Figures 8 and 9 are diagrams showing methods for restoring different types of sensor information using different types of encoders and decoders.

When the vision information 12 cannot be acquired due to a breakdown or unavailability of the camera 120, the vision decoder 14 uses the compressed information of the lidar encoder 210 or the sound encoder 230 to provide the vision information of the camera 120. Information 12 can be restored. FIG. 8 shows a case where the vision decoder 221 restores the vision information 12 using compressed LIDAR information 14 through the LIDAR encoder 210.

In addition to this case, when it is impossible to acquire the LiDAR information 11 through the LiDAR 110, the LiDAR decoder 211 uses the compressed information of the vision encoder 220 or the sound encoder 230 to provide LiDAR information. (11) can be restored, and when it is impossible to acquire acoustic information through the microphone 130, the sound decoder 231 uses the compressed information of the LiDAR encoder 210 or the vision encoder 220 to obtain acoustic information 13. can be restored.

According to this, even in a situation where measurement data cannot be acquired because a specific sensor is broken or cannot be used, information from other sensors can be restored and used to derive obstacle prediction information 18.

On the other hand, as shown in FIG. 9, the spatial decoder 300 cannot acquire the lidar information 11 through the lidar 110 or the vision information 12 through the camera 120, and compressed information of the sound encoder 230 The prediction information 18 about the obstacle can be restored using only . According to this, even when lidar information 11 or vision information 12 cannot be acquired, obstacle information can be predicted even if the accuracy is slightly lower.

Although the present invention has been described above with reference to specific embodiments, those skilled in the art can vary the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be modified and changed.

100: sensor unit 110: lidar
120: Camera 130: Microphone
200: Encoder unit 210: LiDAR encoder
211: Lidar decoder 220: Vision encoder
221: Vision decoder 230: Sound encoder
231: sound decoder 300: spatial decoder
310: spatial encoder

Claims (10)

A sensor unit including a lidar, a camera, and a microphone as a plurality of sensors for acquiring visual information and acoustic information;
an encoder unit extracting compressed information of each data from the data acquired by the plurality of sensors; and
Using pre-given obstacle information, machine learning is performed with a spatial encoder to restore the compressed spatial information to the spatial information of the obstacle, and spatial prediction information about the obstacle is output from the compressed information of the visual and acoustic information extracted through the encoder unit. Includes a spatial decoder;
The encoder unit,
A LiDAR encoder that extracts compressed LiDAR information using map information and LiDAR information acquired by the LiDAR;
a vision encoder that extracts compressed vision information using map information and vision information acquired by the camera; and
An obstacle detection system for an autonomous vehicle or robot, comprising a sound encoder that extracts compressed sound information using map information and sound information acquired by the microphone.
According to paragraph 1,
An obstacle detection system for an autonomous vehicle or robot, wherein the spatial prediction information about the obstacle includes one or more of the position, speed, acceleration, and future path of the predicted obstacle.
delete delete The method of claim 1, wherein the spatial decoder,
An obstacle detection system for an autonomous vehicle or robot, characterized in that it outputs spatial prediction information about the obstacle using an average value of the compressed LiDAR information, the compressed vision information, and the compressed acoustic information.
According to paragraph 1,
An obstacle detection system for an autonomous vehicle or robot, characterized in that the lidar encoder, the vision encoder, the sound encoder, and the spatial decoder are learned by a machine learning unit through an autoencoder.
The method of claim 6, wherein the machine learning unit,
Unsupervised learning of the LiDAR encoder and LiDAR decoder using pre-given LiDAR information and map information,
The vision encoder and vision decoder are trained unsupervised using pre-given vision information and map information,
Unsupervised learning of the sound encoder and sound decoder using pre-given sound information and map information,
An obstacle detection system for an autonomous vehicle or robot, characterized by unsupervised learning of the spatial encoder and the spatial decoder using pre-given obstacle information and map information.
The method of claim 7, wherein the machine learning unit,
An autonomous vehicle or robot, characterized in that learning is conducted in the direction of narrowing the mutual distance within the feature space of compressed information compressed through each of the LiDAR encoder, the vision encoder, the sound encoder, and the spatial encoder. obstacle detection system for
In clause 7,
The LiDAR decoder restores the LiDAR information using compressed information of a vision encoder or sound encoder when acquisition of LiDAR information through the LiDAR is impossible,
When it is impossible to obtain vision information through the camera, the vision decoder restores the vision information using compressed information from the LIDAR encoder or the sound encoder,
The sound decoder is an obstacle detection system for an autonomous vehicle or robot, characterized in that when acquisition of sound information through the microphone is impossible, the sound information is restored using compressed information of the LIDAR encoder or the vision encoder.
In clause 7,
The spatial decoder is for an autonomous vehicle or robot, characterized in that it restores the obstacle information using compressed information of the sound encoder when it is impossible to obtain lidar information through the lidar and vision information through the camera. Obstacle detection system.

Images