KR20210115493A - Method and system for analysing obstacle using analysis priority for obstacle candidate based on attention of driver - Google Patents

Abstract

The present invention provides a method and a system for analyzing an obstacle capable of analyzing obstacle candidates according to applied analysis priorities by providing the analysis priority of the obstacle candidates according to vision of a driver by previously obtaining and analyzing image data of the obstacle candidates in a direction where the vision of the driver does not reach.

Description

Method and system for analyzing obstacles using analysis priority for obstacle candidates based on driver's gaze

The following description relates to a method and system for analyzing an obstacle using an analysis priority for obstacle candidates based on a driver's gaze.

In order to prevent collisions with pedestrians and obstacles due to inexperienced driving or negligence, a technology that recognizes the external environment using sensors and warns the driver of the risk of collision or actively decelerates is being applied

In addition, the driver can react quickly to prevent a collision with only a warning about a collision when driving a vehicle, and an active reaction is also applied, which is a great help in safe driving. In the case of a normal vehicle, it is necessary to prevent collision with any object, so passive/active intervention is helpful for safe driving regardless of the driver's intention.

[Prior literature number]

Korean Patent No. 10-1628176

By first acquiring and analyzing image data for obstacle candidates in a direction that does not reach the driver's gaze, it is possible to give the analysis priority of the obstacle candidates according to the driver's gaze, and to analyze the obstacle candidates according to the given analysis priority. An obstacle analysis method and system are provided.

A method for analyzing an obstacle performed by a computer device included in a vehicle, the method comprising: determining a gaze direction of a driver by at least one processor included in the computer device; selecting, by the at least one processor, an obstacle candidate using the point cloud data collected from the radar of the vehicle; determining, by the at least one processor, an analysis priority for the selected obstacle candidate based on the driver's gaze direction and the position of the selected obstacle candidate; and sequentially analyzing, by the at least one processor, the obstacle candidate image data corresponding to the position of the selected obstacle candidate from the image data collected from the camera of the vehicle according to the determined analysis priority. A method for analyzing obstacles is provided.

According to one side, in the determining of the analysis priority, the larger the angle between the driver's gaze direction and the direction determined according to the position of the selected obstacle candidate, the higher the analysis priority for the selected obstacle candidate. It may be characterized in that it is given.

According to another aspect, the determining of the analysis priority comprises determining the analysis priority for the selected obstacle candidate further based on a distance determined according to the location of the vehicle and the selected obstacle candidate. can do.

According to another aspect, the determining of the analysis priority may include determining the analysis priority of the selected obstacle candidate further based on the traveling direction of the vehicle.

According to another aspect, the analyzing may include first obtaining and analyzing obstacle candidate image data corresponding to the position of an obstacle candidate having a relatively higher analysis priority in the collected image data from the collected image data. can be characterized as

According to another aspect, in the selecting of the obstacle candidate, the collected point cloud data is applied to a machine learning module trained to select an obstacle candidate for the input point cloud data by using the learning data composed of the point cloud data. It may be characterized in that an obstacle candidate is selected by input.

According to another aspect, in the determining of the driver's gaze direction, the driver's gaze is detected according to the detected gaze point by detecting the driver's gaze point from an image acquired through a camera included in the vehicle. It may be characterized in that the direction is determined.

Provided is a computer program stored in a computer-readable recording medium in combination with a computer device to execute the method on the computer device.

There is provided a computer-readable recording medium in which a program for executing the method in a computer device is recorded.

A computer device included in a vehicle, comprising at least one processor embodied to execute computer-readable instructions, wherein the at least one processor determines a gaze direction of a driver and collects data collected from the vehicle's radar. Selecting an obstacle candidate using point cloud data, determining an analysis priority for the selected obstacle candidate based on the driver's gaze direction and the position of the selected obstacle candidate, and by the at least one processor, It provides a computer device, characterized in that the obtained obstacle candidate image data corresponding to the position of the selected obstacle candidate from the image data collected from the camera of the vehicle according to the determined analysis priority and sequentially analyzed.

By first acquiring and analyzing image data for obstacle candidates in a direction that does not reach the driver's gaze, the analysis priority of the obstacle candidates according to the driver's gaze is given, and the obstacle candidates can be analyzed according to the given analysis priority. .

1 is a diagram illustrating an example of a driving environment according to an embodiment of the present invention.
2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention.
3 is a flowchart illustrating an example of an obstacle analysis method according to an embodiment of the present invention.
4 is a diagram illustrating an example of using a distance determined according to a location of a vehicle and a selected obstacle candidate according to an embodiment of the present invention.
5 is a diagram illustrating an example of utilizing a driving direction of a vehicle according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The obstacle analysis system according to embodiments of the present invention may include at least one computer device included in a vehicle, and the obstacle analysis method according to embodiments of the present invention includes at least one computer device included in the obstacle analysis system. can be done through In this case, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the obstacle analysis method according to the embodiments of the present invention under the control of the driven computer program. . The above-described computer program may be stored in a computer-readable recording medium in order to be combined with a computer device to execute the obstacle analysis method in the computer.

1 is a diagram illustrating an example of a driving environment according to an embodiment of the present invention. 1 shows a vehicle 110 and obstacles 120 , 130 and 140 . At this time, since the field of view 112 according to the driver's gaze direction 111 is limited, it is difficult to simultaneously look at these obstacles 120 , 130 , and 140 . For example, in the embodiment of FIG. 1 , the obstacle 1 120 and the obstacle 2 130 are highly likely to be included in the driver's field of view 112 , and accordingly, the obstacle 1 120 and the obstacle 2 130 are directly included by the driver. ), while the obstacle 3 140 is out of the driver's field of view 112, it may be difficult for the driver to directly respond.

In this embodiment, the computer device included in the vehicle may basically detect and respond to all obstacles, but may utilize the driver's gaze to recognize and respond to more dangerous obstacles more quickly. For example, the computer device included in the vehicle may determine the directions 121 of the obstacles 120 , 130 and 140 based on the driver's gaze direction 111 and the positions of the obstacles 120 , 130 and 140 . Based on 131 and 141 , it is possible to determine an analysis priority for each of the obstacles 120 , 130 and 140 , and sequentially analyze the obstacles 120 , 130 and 140 according to the analysis priority. For example, when the analysis priority of the obstacle 3 140 is determined to be the highest among the obstacles 120 , 130 , and 140 , the computer device included in the vehicle may analyze the obstacle 3 140 first. As such, the computer device according to the present embodiment may recognize and respond to a relatively more dangerous obstacle relatively quickly by sequentially analyzing the obstacle candidates according to the analysis priority.

2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. As an example, the computer device included in the vehicle described above may correspond to the computer device 200 illustrated in FIG. 2 . As shown in FIG. 2 , the computer device 200 may include a memory 210 , a processor 220 , a communication interface 230 , and an input/output interface 240 .

The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-volatile mass storage device such as a ROM and a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210 . Also, the memory 210 may store an operating system and at least one program code. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory 210 through the communication interface 230 instead of a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computer device 200 based on a computer program installed by files received through the network 260 .

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230 . For example, the processor 220 may be configured to execute a received instruction according to a program code stored in a recording device such as the memory 210 .

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices via the network 260 . For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as the memory 210 is transmitted to the network ( 260) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 260 . A signal, command, or data received through the communication interface 230 may be transferred to the processor 220 or the memory 210 , and the file may be a storage medium (described above) that the computer device 200 may further include. persistent storage).

The input/output interface 240 may be a means for an interface with the input/output device 250 . For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 240 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. At least one of the input/output devices 250 may include the computer device 200 and one device. For example, a touch screen, a microphone, a speaker, etc. may be implemented in a form included in the computer device 200 such as a smart phone or a navigation system.

Also, in other embodiments, the computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the computer device 200 may be implemented to include at least a portion of the above-described input/output device 250 or may further include other components such as a transceiver and a database.

3 is a flowchart illustrating an example of an obstacle analysis method according to an embodiment of the present invention. The obstacle analysis method according to the present embodiment may be performed by the computer device 200 . In this case, the processor 220 of the computer device 200 may be implemented to execute a control instruction according to a code of an operating system included in the memory 210 or a code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform the steps 310 to 340 included in the method of FIG. 3 according to a control command provided by a code stored in the computer device 200 . can control

In operation 310 , the computer device 200 may determine the driver's gaze direction. For example, the computer device 200 may detect the driver's gaze point from an image acquired through a camera included in the vehicle and determine the driver's gaze direction according to the detected gaze point. Such a camera may be a camera included in the vehicle itself, or may be a camera included in a separate device connected to the computer device 200 such as a smart phone, a navigation device, and/or a black box device. A technology for recognizing the driver's face and determining the driver's gaze direction according to the recognized face may utilize well-known technologies.

In operation 320 , the computer device 200 may select an obstacle candidate using point cloud data collected from the vehicle's radar. Radar is a wireless monitoring device that emits electromagnetic waves or lasers of about microwave (microwave, 10cm to 100cm wavelength) to an object, receives electromagnetic waves or lasers reflected from the object, and finds the distance, direction, and altitude of the object. . When an obstacle is located in the traveling direction of the electromagnetic wave or laser, the radar obtains data (hereinafter, 'point cloud data') in which a plurality of points form a cloud by the electromagnetic wave or laser reflected by the area where the obstacle is located. . In other words, the computer device 200 may recognize that an obstacle candidate exists at a position where the point cloud data is formed. To this end, the computer device 200 may utilize the machine learning module learned to select an obstacle candidate for the input point cloud data by using the learning data composed of the point cloud data. In this case, the computer device 200 may select an obstacle candidate by inputting the point cloud data collected through the radar into the machine learning module. Such a machine learning module may be based on a neural network such as a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), or a Recurrent Neural Network (RNN), or may be based on a classification model such as a Support Vector Machine (SVM).

In operation 330 , the computer device 200 may determine an analysis priority for the selected obstacle candidate based on the driver's gaze direction and the position of the selected obstacle candidate. For example, the computer device 200 may determine an analysis priority for the selected obstacle candidate based on the angle between the driver's gaze direction and the direction determined according to the position of the selected obstacle candidate. As a more specific example, the computer device 200 may give a relatively higher analysis priority to the selected obstacle candidate as the angle between the driver's gaze direction and the direction determined according to the position of the selected obstacle candidate increases. As the above-mentioned angle is larger, the probability that the corresponding obstacle candidate is out of the driver's field of vision is high, and thus there is a possibility that it is a relatively dangerous obstacle. Accordingly, the computer device 200 may give the obstacle candidate a higher analysis priority so that the obstacle candidate is analyzed more quickly.

In another embodiment, the computer device 200 may determine an analysis priority for the selected obstacle candidate further based on a distance determined according to the location of the vehicle and the selected obstacle candidate. For example, with respect to two obstacle candidates having the same/similar angle between directions determined according to the driver's gaze direction and the position of the selected obstacle candidate, a relatively higher analysis priority may be given to an obstacle candidate having a closer distance. . As a more specific example, an analysis streamline ranking may be given according to a weight obtained by adding a score according to an angle and a score according to a distance.

In another embodiment, the computer device 200 may determine an analysis priority for the selected obstacle candidate further based on the traveling direction of the vehicle. In the case of an obstacle candidate that is not related to the direction of the vehicle, a relatively low priority for analysis may be given. As a more specific example, the analysis wired priority may be given according to a weight obtained by adding a score according to an angle, a score according to a distance, and a score according to a driving direction of a vehicle and a position of an obstacle candidate.

In operation 340, the computer device 200 may obtain and sequentially analyze the obstacle candidate image data corresponding to the position of the selected obstacle candidate from the image data collected from the camera of the vehicle according to the determined analysis priority. In this case, the computer device 200 may first obtain and analyze the obstacle candidate image data corresponding to the position of the obstacle candidate having a high analysis priority in the collected image data from the collected image data. For example, since it is difficult to accurately analyze the type of obstacle or the location and size of an obstacle only with point cloud data collected from radar, image data collected from a camera may be further utilized. In this case, since a certain time is required to accurately recognize an obstacle in the image data, the computer device 200 sequentially analyzes the obstacle candidates according to the analysis priority based on the driver's gaze, thereby recognizing a relatively more dangerous obstacle. You will be able to recognize and respond more quickly.

4 is a diagram illustrating an example of using a distance determined according to a location of a vehicle and a selected obstacle candidate according to an embodiment of the present invention. 4 shows a vehicle 110 and an obstacle A 410 and an obstacle B 420 . At this time, the first angle between the driver's gaze direction 111 and the direction 411 determined according to the position of the obstacle A 410, and the driver's gaze direction 111 and the position of the obstacle B 420 When only the second angle between the directions 421 is considered, a higher analysis priority may be determined for the obstacle B 420 . However, in the present embodiment, since the obstacle A 410 is closer than the obstacle B 420 , there is a need to first analyze the obstacle A 410 . The computer device 200 described above calculates the score according to the angle and the score according to the distance, respectively, and adds them up to determine the analysis priority of each of the obstacles A 410 and B 420 according to the weight according to the summed score. can decide In other words, an obstacle A 410 relatively closer to the vehicle 110 may have a higher analysis priority than an obstacle B 420 positioned at a relatively larger angle with respect to the driver's gaze direction 111 . do.

5 is a diagram illustrating an example of utilizing a driving direction of a vehicle according to an embodiment of the present invention. 5 shows the vehicle 110 and an obstacle a 510 and an obstacle b 520 . It is assumed that the distance between the vehicle 110 and the obstacle a 510 and the distance between the vehicle 110 and the obstacle b 520 are the same. In this case, the first angle between the driver's gaze direction 111 and the direction 511 determined according to the position of the obstacle a 510 and the driver's gaze direction 111 and the position of the obstacle b 520 are determined according to the first angle. When only the second angle between the directions 521 is considered, a higher analysis priority may be determined for the obstacle b 520 . However, in the present embodiment, since the obstacle a 510 exists in the traveling direction 530 of the vehicle 110 , there is a need to first analyze the obstacle a 510 . The computer device 200 described above calculates the score according to the angle and the score according to the position of each of the obstacles 510 and 520 in the traveling direction 530 of the vehicle, and then adds them up and adds a weight according to the summed score. Accordingly, it is possible to determine an analysis priority of each of the obstacle a 510 and the obstacle b 520 . In other words, the analysis that the obstacle a (510) having a higher relevance to the traveling direction 530 of the vehicle 110 is higher than the obstacle b (520) located at a relatively larger angle with respect to the driver's gaze direction 111 . You can even have priorities.

As described above, according to embodiments of the present invention, by first acquiring and analyzing image data for obstacle candidates in a direction that does not reach the driver's gaze, the analysis priority of the obstacle candidates according to the driver's gaze is given, and the given Obstacle candidates can be analyzed according to the analysis priority.

The system or apparatus described above may be implemented as a hardware component or a combination of a hardware component and a software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The medium may be to continuously store a computer executable program, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute other various software, and servers. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In the obstacle analysis method performed by a computer device included in a vehicle,
determining a driver's gaze direction by at least one processor included in the computer device;
selecting, by the at least one processor, an obstacle candidate using the point cloud data collected from the radar of the vehicle;
determining, by the at least one processor, an analysis priority for the selected obstacle candidate based on the driver's gaze direction and the position of the selected obstacle candidate; and
acquiring and sequentially analyzing, by the at least one processor, obstacle candidate image data corresponding to the position of the selected obstacle candidate from the image data collected from the camera of the vehicle according to the determined analysis priority;
An obstacle analysis method comprising a. According to claim 1,
The step of determining the analysis priority,
The obstacle analysis method, characterized in that the analysis priority for the selected obstacle candidate is determined based on an angle between the driver's gaze direction and a direction determined according to the position of the selected obstacle candidate. According to claim 1,
The step of determining the analysis priority,
The obstacle analysis method, characterized in that, as the angle between the driver's gaze direction and the direction determined according to the position of the selected obstacle candidate increases, a relatively higher analysis priority is given to the selected obstacle candidate. According to claim 1,
The step of determining the analysis priority,
The obstacle analysis method, characterized in that the analysis priority for the selected obstacle candidate is further based on a distance determined according to the location of the vehicle and the selected obstacle candidate. According to claim 1,
The step of determining the analysis priority,
Obstacle analysis method, characterized in that the analysis priority for the selected obstacle candidate is determined further based on the traveling direction of the vehicle. According to claim 1,
The analyzing step is
The obstacle analysis method, characterized in that in the collected image data, the obstacle candidate image data corresponding to the position of the obstacle candidate having a relatively higher analysis priority is first obtained and analyzed from the collected image data. According to claim 1,
The step of selecting the obstacle candidate comprises:
Obstacle analysis method, characterized in that selecting an obstacle candidate by inputting the collected point cloud data into a machine learning module trained to select an obstacle candidate for the input point cloud data by using the learning data composed of the point cloud data. According to claim 1,
The step of determining the driver's gaze direction includes:
An obstacle analysis method, characterized in that by detecting the driver's gaze point from an image acquired through a camera included in the vehicle, and determining the driver's gaze direction according to the detected gaze point. A computer-readable recording medium in which a computer program for executing the method of any one of claims 1 to 8 in a computer device is recorded. A computer device included in a vehicle, comprising:
at least one processor implemented to execute computer-readable instructions
including,
by the at least one processor,
determine the driver's gaze direction;
Selecting an obstacle candidate using the point cloud data collected from the radar of the vehicle,
determining an analysis priority for the selected obstacle candidate based on the driver's gaze direction and the position of the selected obstacle candidate;
Obtaining and sequentially analyzing, by the at least one processor, obstacle candidate image data corresponding to the position of the selected obstacle candidate in the image data collected from the camera of the vehicle according to the determined analysis priority
A computer device characterized by a. 11. The method of claim 10,
by the at least one processor,
Determining an analysis priority for the selected obstacle candidate based on an angle between the driver's gaze direction and a direction determined according to the position of the selected obstacle candidate
A computer device characterized by a. 11. The method of claim 10,
by the at least one processor,
Giving a relatively higher analysis priority to the selected obstacle candidate as the angle between the driver's gaze direction and the direction determined according to the position of the selected obstacle candidate increases
A computer device characterized by a. 11. The method of claim 10,
by the at least one processor,
Determining an analysis priority for the selected obstacle candidate further based on a distance determined according to the location of the vehicle and the selected obstacle candidate
A computer device characterized by a. 11. The method of claim 10,
by the at least one processor,
Determining an analysis priority for the selected obstacle candidate further based on the traveling direction of the vehicle
A computer device characterized by a. 11. The method of claim 10,
by the at least one processor,
In the collected image data, the obstacle candidate image data corresponding to the position of the obstacle candidate having a relatively higher analysis priority is first obtained and analyzed from the collected image data.
A computer device characterized by a.

Images