KR102569898B1 - Driving assistance system using mixed reality and method for controlling the same - Google Patents

Abstract

The present invention relates to a driving assistance system capable of providing an environment similar to a driver's usual driving environment through mixed reality technology in a situation where the driving direction and driver's seat position are opposite to the driver's usual driving environment, and a control method thereof. A control method of a driving assistance system using mixed reality according to an embodiment of the present invention includes acquiring peripheral cognitive information related to a driver's senses through a cognitive information acquiring unit; converting the acquired information to correspond to a preset desired driving environment when a predetermined condition is satisfied; and outputting the converted information through an output unit, wherein the desired driving environment may be set to either left-hand drive or right-hand drive.

Description

Driving assistance system using mixed reality and its control method

The present invention relates to a driving assistance system capable of providing an environment similar to a driver's usual driving environment through mixed reality technology in a situation where the driving direction and driver's seat position are opposite to the driver's usual driving environment, and a control method thereof.

In most countries including Korea, forward driving is performed in the right lane based on the center line of the road, and vehicles operated/sold in these countries are often called "left-hand drive" vehicles because the driver's seat is located on the left side.

Unlike this, in some countries, such as the UK and Japan, forward driving is performed in the left lane based on the center line of the road, and the vehicle's driver's seat is on the right side, so it is called "right-hand drive".

Therefore, when a citizen of a right-hand drive country drives a left-hand drive vehicle in a left-hand drive country or a citizen of a left-hand drive country drives a right-hand drive vehicle in a right-hand drive country, left/right turn rules and signal systems are different, There is a problem that driving fatigue increases and the accident rate increases.

An object of the present invention is to provide a driving assistance system using mixed reality and a control method thereof capable of providing a local driving environment regardless of driving direction and steering wheel position.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

In order to solve the above technical problem, a control method of a driving assistance system using mixed reality according to an embodiment of the present invention includes acquiring peripheral cognitive information related to a driver's senses through a cognitive information acquisition unit; converting the acquired information to correspond to a preset desired driving environment when a predetermined condition is satisfied; and outputting the converted information through an output unit, wherein the desired driving environment may be set to either left-hand drive or right-hand drive.

In addition, a driving assistance system using mixed reality according to an embodiment of the present invention includes a cognitive information acquiring unit that acquires peripheral cognitive information related to a driver's senses; a driving environment controller that converts the acquired information to correspond to a preset desired driving environment when a predetermined condition is satisfied; and an output unit outputting the converted information, wherein the desired driving environment may be set to either left-hand drive or right-hand drive.

Through the driving assistance system using mixed reality and its control method according to at least one embodiment of the present invention configured as described above, it is possible to drive with the same sense even in situations where the driving direction and steering wheel direction are opposite to those familiar to the driver. do.

In addition, since the present invention reverses not only sight, but also hearing and body leaning, discomfort caused by mixed reality can be minimized.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a block diagram showing an example of a structure of a driving assistance system according to an embodiment of the present invention.
2 illustrates a driver's driving environment according to an embodiment of the present invention.
3 is a flowchart illustrating an example of a method for controlling a driving assistance system according to an embodiment of the present invention.
4 shows an example of a form in which time information conversion is performed according to an embodiment of the present invention.
5 illustrates an example of a form in which visual information conversion is performed when an omnidirectional camera is provided in a driving assistance system according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated. Also, parts denoted with the same reference numerals throughout the specification mean the same components.

In one embodiment of the present invention, the surrounding environment information perceived through the driver's senses while driving in an environment opposite to the direction of the steering wheel familiar to the driver is reversed in the vehicle and provided again to the driver in the form of mixed reality, thereby providing the existing driving sensation. It is proposed to provide the same driving environment as In addition, for this purpose, it is proposed that inputs related to directions, such as steering wheel operation and direction indicator operation, be reversed before being transmitted to the subject performing the function of the vehicle. Here, mixed reality generally refers to a technology in which virtual reality and augmented reality are mixed, and in the present invention, it may mean a technology that separates the driver's senses from the real environment and provides them.

1 is a block diagram showing an example of a structure of a driving assistance system according to an embodiment of the present invention.

Referring to FIG. 1 , a driving assistance system according to an embodiment includes a recognition information acquisition unit 110, a manipulation unit 120, a driving environment controller 130, an output unit 140, a body controller 150, and a steering controller 160. ) may be included. The components shown in FIG. 1 represent only the components related to the implementation of the present invention, and it goes without saying that more or fewer components may be included in the actual implementation of the driving assistance system. Hereinafter, each component will be described in detail.

First, the cognitive information acquisition unit 110 may obtain information related to a sense of a driver's perception of a driving environment while driving. To this end, the cognitive information acquisition unit 110 may include a camera 111 that obtains visual information, a microphone 112 that acquires sound information, and a sensor 113 that detects the acceleration felt by the driver due to the behavior of the vehicle. can

For example, the camera 111 includes a front camera that acquires an image of at least the front of the vehicle, an in-cabin camera that acquires an image of a side mirror from the driver's point of view, a side camera disposed at or adjacent to the side mirror, It may include at least one of omnidirectional cameras that acquire 360-degree images.

The microphone 112 is preferably a stereo microphone so that at least the left and right sides of the sound can be distinguished, and more preferably, it may be an omnidirectional microphone.

The sensor 113 may include an acceleration sensor or a gyro sensor to detect leaning felt by the driver according to the behavior of the vehicle.

Next, the control unit 120 is a vehicle component that is operated by a driver while driving in relation to a direction, and may correspond to a steering wheel and a direction indicator control lever.

The driving environment controller 130 receives the image, sound, and acceleration information obtained from the recognition information acquisition unit 110 and determines whether the driver's desired driving environment matches the driving environment elements obtained from the corresponding information, and determines whether or not they match. If not, the corresponding information may be converted to correspond to the driver's desired driving environment. At this time, the driving environment controller 130 may also convert a manipulation input input through the manipulation unit 120 . Here, transformation may mean left-right inversion. The converted manipulation input may be transmitted to individual function execution controllers such as the body controller 150 and the steering controller 160.

Of course, the driving environment controller 130 may transmit the corresponding information and manipulation input to the output unit 140 and the other controllers 150 and 160 as they are without conversion when the driver's desired driving environment matches the driving environment element. .

The output unit 140 may include a head mounted display 141 covering the driver's field of view, headphones 142 outputting sound, and a tilting seat 143. A specific form in which the converted image is displayed on the head mounted display 141 (HMD: Head Mounted Display) will be described later with reference to FIG. 4 .

The headphone 142 is preferably at least a stereo headphone, and when an omnidirectional microphone is supported, it is more preferable that the headphone 142 is a multi-channel headphone that separately transmits front and rear sounds as well as left and right.

Of course, the head-mounted display 141 and the headphones 142 are illustrative, and if they can block the actual driving environment and transmit the visual and auditory information processed by the driving environment controller to the driver, other components such as a omnidirectional monitor or speaker It is obvious to those skilled in the art that it can be replaced with.

The tilting seat 143 includes an actuator capable of tilting the driver's seat left and right at least, and can be tilted left or right according to a driving signal generated by the driving environment controller 130 based on a signal from the sensor 113. Due to the provision of such a tilting seat, when the visual information and the auditory information are reversed, the sense of incongruity can be minimized by harmonizing the inverted information with the body leaning. For example, when the driver attempts to turn right while the visual, auditory and steering wheel operations are reversed left and right, the vehicle actually turns left. At this time, if the tilting seat 143 is not provided, the driver's body leans to the right even though he thinks he is making a right turn, resulting in incongruity between his senses. Therefore, if the tilting seat 143 cancels the acceleration leaning to the right and tilts to the left with the information that the body will lean to the left, the driver's sense of incongruity for the inverted sense is consistent with the rest of the inverted senses even the body tilt corresponds to the right turn. is minimized

For example, the driver drives in the situation shown in FIG. 2 . 2 illustrates a driver's driving environment according to an embodiment of the present invention. Referring to FIG. 2 , a driver sits in a driver's seat implemented as a tilting seat 143 and drives while wearing a head mounted display 141 .

Returning to FIG. 1 again, the body controller 150 receives the lighting signal generated as the direction indicator lever is operated in the control unit 120 through the driving environment controller 130 as it is or receives the lighting signal whose direction is reversed. , the direction indicator lamp corresponding to the corresponding signal may be turned on.

The steering controller 160 is preferably a controller that controls an electronic steering system using an electric motor that is not a rack and pinion type. This is because the rotation direction and the steering direction of the steering wheel can be reversed only when the electronic steering system is provided. Similar to the body controller 150, the steering controller 160 may receive an inverted or non-inverted steering signal from the driving environment controller 130 and control the steering system in response thereto.

3 is a flowchart illustrating an example of a method for controlling a driving assistance system according to an embodiment of the present invention.

Referring to FIG. 3 , first, a driver's desired driving environment may be determined (S210). Determination of the desired driving environment may be a process in which the driver inputs a command through manipulation of a predetermined control system, or a process in which a smart device carried by the driver but storing information about the driver's usual driving direction transmits the information to the vehicle. . For example, the desired driving environment may be performed in the form of selecting either left-hand drive or right-hand drive.

The driving environment controller 130 may obtain peripheral cognitive information such as visual, auditory, and acceleration through the cognitive information acquisition unit 110 (S220).

The driving environment controller 130 may acquire driving environment elements based on the acquired information (S230). Here, the driving environment factor is information that is the basis for determining the driving direction (or steering wheel direction) of the road, and the position of the center line relative to the driving direction, the position of vehicles driving in the opposite direction, and signs that can determine the country And the like may correspond to this, but is not necessarily limited thereto.

The driving environment controller 130 determines the current driving direction based on the acquired driving environment elements, and determines whether the determined driving direction coincides with the driver's desired driving environment (S240).

As a result of the determination, if they do not match, the driving environment controller 130 may convert the surrounding cognitive information to correspond to the desired driving environment, and transmit the converted information to the output unit 140 (S250). At this time, the driving environment controller 130 may invert a signal corresponding to a manipulation related to a direction among signals input through the manipulation unit 120 and transmit the inverted signal to the body controller 150 and the steering controller 160 . In addition, converting information may be a concept including additional image processing in addition to simple horizontal reversal. A specific form in which additional image processing is performed will be described later with reference to FIG. 4 .

In contrast, when the determined driving direction matches the desired driving environment of the driver, the driving environment controller 130 transfers the information obtained from the recognition information acquisition unit 110 to the output unit 140 as it is without reversal, and the operation unit ( The signal input from 120) can also be transmitted to the body controller 150 and the steering controller 160 without inversion (S260).

4 shows an example of a form in which time information conversion is performed according to an embodiment of the present invention. In FIG. 4 , a description is given based on an image of the front of the vehicle for easy understanding, and it is assumed that the driver selects the left steering wheel as the desired driving environment.

First, referring to (a) of FIG. 4 , an image captured through a front camera is shown. In this image, the driving environment controller 130 may detect the center line 410 and determine the current driving direction based on the relative direction within the image. For example, in the situation shown in (a) of FIG. 4 , since the center line 410 is located on the left side, the driving environment controller 130 may determine that the driving direction corresponds to the left steering wheel. Therefore, since the desired driving environment and the current driving direction do not match, the driving environment controller 130 determines the conversion of the image.

Accordingly, the driving environment controller 130 may invert the front image left and right as shown in (b) of FIG. 4 . At this time, the left-right reversed image may include surrounding objects that need to be referred to for driving, such as traffic signs. can judge Here, it is obvious to those skilled in the art that widely known technologies such as feature point extraction-based shape recognition technology or text recognition technology can be used to recognize a surrounding object in an image, so description of a specific recognition method will be omitted.

For example, an object to be output in an inverted state may be an object 430 indicating a direction. In addition, the objects 420 and 440 including standardized signs or texts that may cause a driver to feel uncomfortable when left and right reversed may correspond to objects having directions that should not be reversed.

As shown in (c) of FIG. 4 , the driving environment controller 130 may re-invert the directional object ( 420 ′) to finally generate an image to be provided to the head mounted display 141 . If a desired language is set in setting the desired driving environment, the text of the object 440 including text may be converted into the desired language (440'). For example, when the desired language is Korean, a “STOP” sign may be converted to “stop” and output.

5 illustrates an example of a form in which visual information conversion is performed when an omnidirectional camera is provided in a driving assistance system according to an embodiment of the present invention.

Referring to (a) of FIG. 5, one frame of an image photographed through an omnidirectional camera is shown. When the desired driving environment and the current driving direction do not match, the driving environment controller 130 horizontally reverses the acquired image as shown in FIG. An image of a portion corresponding to the field of view of the user may be provided to the head mounted display.

The above-described present invention can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is

Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (19)

Acquiring peripheral cognitive information while the vehicle is driving as information related to a driver's senses through a cognitive information acquisition unit;
determining whether forward driving in the right lane or forward driving in the left lane is based on the obtained information;
converting the acquired information to correspond to a preset desired driving environment when a preset condition is satisfied; and
Outputting the converted information through an output unit,
The desired driving environment is set to either a left-hand drive or a right-hand drive,
The recognition information obtaining unit includes a sensor for detecting acceleration according to the behavior of the vehicle,
Further comprising tilting the sheet in the reverse direction of the acceleration sensed by the sensor,
Control method of driving assistance system using mixed reality. According to claim 1,
The recognition information acquisition unit further includes a camera and a microphone for taking at least a front image,
The method of controlling a driving assistance system using mixed reality, wherein the surrounding perception information includes a front image and ambient sound. According to claim 2,
The conversion step is
A method of controlling a driving assistance system using mixed reality, comprising the step of horizontally reversing an image acquired through the camera and an ambient sound obtained through the microphone. According to claim 3,
The output unit includes a head mounted display and headphones,
The outputting step is
outputting the inverted image to a head mounted display; and
A method of controlling a driving assistance system using mixed reality, comprising outputting the inverted ambient sound through headphones. According to claim 3,
The conversion step is
extracting at least one object from an image obtained through the camera;
determining whether the at least one object has a directivity; and
The method of controlling a driving assistance system using mixed reality further comprising the step of re-inverting an object determined to have directionality in the inverted image. delete According to claim 1,
When the preset condition is satisfied,
A method of controlling a driving assistance system using mixed reality, further comprising: inverting a signal corresponding to a manipulation related to a direction input through a manipulation unit of a vehicle, and transmitting the signal to a function execution controller of the vehicle. According to claim 7,
The control unit includes a direction indicator lever and a steering wheel,
The function execution controller,
A method of controlling a driving assistance system using mixed reality, comprising: a first controller controlling a direction indicator corresponding to an operation of the direction indicator lever and a second controller controlling a steering system corresponding to an operation of the steering wheel. According to claim 1,
The preset condition is,
The method of controlling a driving assistance system using mixed reality, which is satisfied when the desired driving environment and the driving direction determined based on the information acquired by the recognition information obtaining unit are different from each other. A computer readable recording medium storing a program for executing the control method of a driving assistance system using mixed reality according to any one of claims 1 to 5 and 7 to 9. A cognitive information acquisition unit for obtaining peripheral recognition information while driving the vehicle as information related to the driver's senses;
a driving environment controller that determines whether forward driving in the right lane or forward driving in the left lane is based on the acquired information, and converts the obtained information according to a preset desired driving environment when a preset condition is satisfied; and
Including an output unit for outputting the converted information,
The desired driving environment is set to either left-hand drive or right-hand drive,
Further comprising a tilting seat that is tilted at least to the left or right,
The recognition information obtaining unit includes a sensor for detecting acceleration according to the behavior of the vehicle,
The driving environment controller controls the tilting seat to be tilted in the reverse direction of the acceleration sensed by the sensor.
Driving assistance systems using mixed reality. According to claim 11,
The recognition information acquisition unit further includes a camera and a microphone for taking at least a front image,
The driving assistance system using mixed reality, wherein the surrounding perception information includes a front image and ambient sound. According to claim 12,
The driving environment controller,
A driving assistance system using mixed reality that reverses the image acquired through the camera and the ambient sound acquired through the microphone in the left and right directions. According to claim 13,
the output unit,
a head mounted display outputting the inverted image; and
A driving assistance system using mixed reality, including headphones outputting the inverted ambient sound. According to claim 13,
The driving environment controller,
Driving using mixed reality that extracts at least one object from the image acquired through the camera, determines whether the at least one object has directionality, and re-inverts the object determined to have directionality in the inverted image. auxiliary system. delete According to claim 11,
When the preset condition is satisfied,
The driving environment controller,
A driving assistance system using mixed reality, wherein a signal corresponding to an operation related to a direction input through a control unit of a vehicle is horizontally reversed and transmitted to a controller performing a function of the vehicle. According to claim 17,
The control unit includes a direction indicator lever and a steering wheel,
The function execution controller,
A driving assistance system using mixed reality, comprising: a first controller controlling direction indicators corresponding to manipulation of the direction indicator lever and a second controller controlling a steering system corresponding to manipulation of the steering wheel. According to claim 11,
The preset condition is,
A driving assistance system using mixed reality, which is satisfied when the desired driving environment and the driving direction determined based on the information acquired by the recognition information obtaining unit are different from each other.

Images