KR102270577B1 - Apparatus and method for controlling outputting external image of vehicle - Google Patents

Abstract

The present invention proposes a vehicle external image output control apparatus and method for output control of an image around a vehicle by calculating a driver's viewing angle based on a marker attached to a wearable device. An apparatus for controlling an external image of a vehicle according to the present invention includes an image acquisition unit configured to acquire an image of the exterior of the vehicle; a viewing angle calculator configured to calculate the driver's viewing angle using a marker attached to the wearable device worn by the driver; a target image generator for extracting an image in at least one direction from an image for the outside of the vehicle based on the viewing angle and generating an output target image; and an image display unit configured to display an output target image on the wearable device.

Description

Apparatus and method for controlling outputting external image of vehicle}

The present invention relates to an apparatus and method for controlling the output of an image. More particularly, it relates to an apparatus and method for controlling the output of an image to the outside of a vehicle.

Today's vehicles are equipped with a variety of electronic equipment. As such equipment, a navigation device that tracks and displays the location of the vehicle and outputs a map of the location where the vehicle is driving, a camera device that acquires images by photographing the inside/outside of the vehicle, and stores the images captured by the camera. LCDs that output or output playback images such as CDs or DVDs on the screen are examples.

However, a conventional camera device mounted on a vehicle has the following problems.

First, there is a limitation in collecting only image information in a limited area because the range of collecting the screen is limited. Accordingly, there is a problem in that it is not possible to show an image around the driver's seat whose view is obscured by the vehicle body.

Second, if there is a 3D structure around the vehicle, there is a disadvantage of showing an image of distorted incorrect information on the video screen.

Korean Patent Application Laid-Open No. 2013-0013410 proposes a device for outputting an image around a vehicle. However, since this device merely outputs an image around the vehicle according to the driving state of the vehicle, the above problem cannot be solved.

The present invention has been devised to solve the above problems, and it is an object of the present invention to propose an apparatus and method for controlling an external image output of a vehicle for output control of an image around a vehicle by calculating a driver's viewing angle based on a marker attached to a wearable device do it with

However, the object of the present invention is not limited to the above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

The present invention has been devised to achieve the above object, the image acquisition unit for acquiring an image of the outside of the vehicle; a viewing angle calculator configured to calculate a viewing angle of the driver using a marker attached to a wearable device worn by the driver; a target image generator for extracting an image of at least one direction from the image for the outside of the vehicle based on the viewing angle and generating an output target image; and an image display unit configured to display the output target image on the wearable device.

Preferably, the image acquisition unit acquires images of directions other than the front of the vehicle as an image of the outside of the vehicle or acquires images of all directions of the vehicle.

Preferably, the viewing angle calculation unit and the image display unit use a glasses-type device as the wearable device.

Preferably, the viewing angle calculator uses markers attached to upper ends of both eyeglass frames and markers attached to lower ends of both eyeglass frames as the markers.

Preferably, the viewing angle calculating unit calculates the driver's viewing angle based on the position and movement direction of the driver's eyes acquired by the marker and the position of the marker acquired by the camera.

Preferably, the viewing angle calculation unit tracks the movement of the driver's eyes in real time within a range of 90 degrees to the left and right with respect to the front of the vehicle by using the marker.

Preferably, the viewing angle calculation unit further uses markers attached to left ends of the both eyeglass frames and markers attached to right ends of the both eyeglass frames as the markers.

Preferably, the target image generator generates the output target image as a stereo image, and when images for at least two directions are extracted from the image for the outside of the vehicle, the output target image is generated as a panoramic image.

Preferably, the target image generating unit generates the output target image when it is determined that the viewing angle deviates from 90 degrees to the left or 90 degrees to the right with respect to the front of the vehicle.

In addition, the present invention comprises the steps of obtaining an image of the outside of the vehicle; calculating a viewing angle of the driver using a marker attached to a wearable device worn by the driver; extracting an image of at least one direction from the image of the exterior of the vehicle based on the viewing angle and generating an image to be output; and displaying the output target image on the wearable device.

Preferably, the acquiring includes acquiring images for the directions other than the front of the vehicle as the image for the outside of the vehicle or acquiring images for all directions of the vehicle.

Preferably, in the calculating and displaying, a device in the form of glasses is used as the wearable device.

Preferably, in the calculating, markers attached to upper ends of both eyeglass frames and markers attached to lower ends of both eyeglass frames are used as the markers.

Preferably, the calculating step calculates the driver's viewing angle based on the position and movement direction of the driver's eyes acquired by the marker, and the position of the marker acquired by the camera.

Preferably, in the calculating, the movement of the driver's eyes is tracked in real time within a range of 90 degrees to the left and right with respect to the front of the vehicle using the marker.

Preferably, the calculating step further uses the markers attached to the left ends of the both eyeglass frames and the markers attached to the right ends of the both eyeglass frames as the markers.

Preferably, in the generating, the output target image is generated as a stereo image, and when images for at least two directions are extracted from the image for the outside of the vehicle, the output target image is generated as a panoramic image.

Preferably, in the generating, if it is determined that the viewing angle deviates from 90 degrees in the left direction or 90 degrees to the right with respect to the front of the vehicle, the output target image is generated.

According to the present invention, the following effects can be obtained by calculating the driver's viewing angle based on the marker attached to the wearable device and controlling the output of the image around the vehicle.

First, it is possible to check the image around 360 degrees from the driver's view, not the two-dimensional flat image viewed from the top of the image around the vehicle.

Second, the driver can intuitively view an image around the driver's seat whose field of vision is obscured by the vehicle body using a wearable display device.

1 is a conceptual diagram illustrating an internal configuration of a 3D AVM system according to an embodiment of the present invention.
FIG. 2 is a perspective view of a wearable display device constituting the 3D AVM system of FIG. 1 .
FIG. 3 is a reference diagram for explaining an image collection method of a camera constituting the 3D AVM system of FIG. 1 .
4 is a flowchart illustrating a method of operating a 3D AVM system according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, it should be noted that in adding reference numerals to the components of each drawing, the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, preferred embodiments of the present invention will be described below, but the technical spirit of the present invention is not limited thereto and may be variously implemented by those skilled in the art without being limited thereto.

The present invention relates to a 3D AVM (Around View Monitor) system using glasses-type wearable display equipment, and it is characterized in that it collects 360-degree images around a car and outputs the peripheral images through the wearable display equipment.

1 is a conceptual diagram illustrating an internal configuration of a 3D AVM system according to an embodiment of the present invention.

The 3D AVM system 100 uses a wearable display device in the form of glasses to intuitively view an image around the driver's seat whose field of view is obscured by the vehicle body. The wearable display device in the form of glasses will be described later with reference to FIG. 2 .

The 3D AVM system 100 may obtain the following effects according to the above-described characteristics. First, the driver can check the image outside the vehicle just by looking around the wearable display device. Second, it becomes possible for the driver to check the surroundings of the vehicle while feeling the effect of riding in a vehicle made of glass.

The viewing angle measurement logic unit 110 performs a function of calculating the driver's viewing angle. The viewing angle measurement logic unit 110 may calculate the driver's viewing angle using a marker attached to the wearable device worn by the driver.

When the marking data is input by the marker (111), the viewing angle measurement logic unit 110 measures the viewing angle based on the marking data and transmits the measurement data to the additional information attachment logic unit 130 (112). The viewing angle measurement logic unit 110 may receive marking data from four predetermined points (4 Point Marking).

When measuring the direction of the driver's gaze, the viewing angle measurement logic unit 110 uses a viewing angle measurement technique using a marker, rather than tracking the gaze using a camera.

Existing gaze tracking techniques have a problem in that a delay occurs due to a system load, and accurate gaze tracking is difficult. Since the viewing angle measurement logic unit 110 tracks the gaze with an optical marker using 4 point marking, there is an advantage that a relatively small amount of load and fast tracking are possible than in the prior art.

In addition, the viewing angle measurement logic unit 110 can measure a distance angle in a three-dimensional space within 2 ms. The viewing angle measurement logic unit 110 can calculate an expected position value to perform position estimation faster than before, and by predicting and reflecting the next viewing angle, it can move directly to the image mapping step according to conditions.

The image collection unit 120 performs a function of acquiring an image of the outside of the vehicle. The image collection unit 120 collects a screen for the outside of the vehicle according to the following logic.

The image collection unit 120 uses a camera 121 located at the front of the vehicle, a camera 122 located on the left side of the vehicle, a camera 123 located on the right side of the vehicle, and a camera 124 located at the rear of the vehicle. Acquire front/rear/lateral images around the car.

FIG. 3 is a reference diagram for explaining an image collection method of a camera constituting the 3D AVM system of FIG. 1 . The image collection unit 120 uses the camera 310 as shown in FIG. 3 , for images about the vehicle surroundings 320a, 320b, and 320c, for example, an image for the front of the vehicle 320a, and the vehicle side 320b. It is possible to obtain an image for the vehicle, an image for the rear of the vehicle 320c, and the like.

The image collection unit 120 generates a 360-degree image by merging the acquired camera images ( 125 ). The image collection unit 120 stores the generated image in an image or video format. When the driver wearing the wearable display device looks to the side, the rear, etc., the image collection unit 120 shows an image of a part that is hidden inside the vehicle on a monitor.

The additional information attachment logic unit 130 performs a function of converting the image generated by the image collecting unit 120 based on the viewing angle measured by the viewing angle measurement logic unit 110 . The additional information attachment logic unit 130 synthesizes the monitor image according to the position of the driver's gaze ( 131 ).

The wearable display image transmission unit 140 outputs the image converted by the additional information attachment logic unit 130 to the monitor ( 141 ). Also, the wearable display image transmitting unit 140 collects and stores the image output to the monitor as a vehicle image ( 142 ).

The 3D AVM system 100 described above is capable of providing an augmented reality image service of the side and the rear rather than a simple front augmented reality data service by synthesizing the augmented reality function. This is because blind spots around the driver disappear.

In addition, the 3D AVM system 100 can intuitively check the surrounding image in a rear parking situation.

In addition, the 3D AVM system 100 can be utilized as a 360-degree image acquisition system function by replacing the black box image system by using a camera mounted on the vehicle.

In addition, since the 3D AVM system 100 uses a 4-point marker to measure the driver's viewing angle, it is effective for fast location tracking and reducing the cost of sensor IC of wearable equipment.

FIG. 2 is a perspective view of a wearable display device constituting the 3D AVM system of FIG. 1 .

A method of recognizing a driver's gaze using the wearable display device 200 in the form of glasses is as follows.

4 POINT MARKING (201, 202, 203, 204) is attached to the front of the glasses 200. First, a camera that takes a picture of the driver's face is installed inside the vehicle, and the 4 point marking video image attached to the glasses is acquired.

Then, the angle and size of the 4 POINT MAKING (201, 202, 203, 204) are measured to calculate the driver's gaze angle and the position coordinates of the face. In this case, POINT MAKING designates a specific color to improve image processing recognition rate. Here, a specific color means a color that is distinguished from the surrounding colors.

Thereafter, the distance, rotation angle, etc. of the driver are calculated using the position of the 4 points and the shape of the glasses frame in the rectangular area 205 .

4 is a flowchart illustrating a method of operating a 3D AVM system according to an embodiment of the present invention. The following description refers to FIGS. 1 and 4 .

First, the image collection unit 120 acquires an image around the vehicle and stores 360-degree image information (S405).

Thereafter, the marker position information calculation unit constituting the viewing angle measurement logic unit 110 collects the marker position information of the wearable device worn by the driver, and the marker storage unit stores the marker position information ( S410 ).

Thereafter, the marker position information calculating unit calculates the marker position information including the initial position of the marker mounted on the wearable device (S415).

Thereafter, the marker position information calculating unit calculates relative information including the current position and current angle of the wearable device based on the marker position information ( S420 ).

Thereafter, the wearable device position calculating unit constituting the viewing angle measurement logic unit 110 calculates the driver's viewing angle ( S425 ).

Thereafter, the wearable device position calculating unit determines whether the driver's gaze direction is lateral or rearward based on the driver's viewing angle (S430).

When it is determined that the driver's gaze direction is lateral or rearward, the additional information attachment logic unit 130 matches the surrounding image according to the viewing angle ( S435 ). Thereafter, the additional information attachment logic unit 130 outputs the surrounding image to the wearable display (S440).

On the other hand, if it is determined that the driver's gaze direction is not to the side or rear, the marker next step position estimating unit constituting the additional information attachment logic unit 130 estimates the expected movement position of the marker based on the position information of the marker. do (S445).

The marker next step position estimating unit determines whether the driver's gaze direction is lateral or rearward in the next step (S450).

If it is determined that the driver's gaze direction is lateral or rearward in the next step, the marker next step position estimation unit moves to the image matching step (S435) immediately after waiting for the next step image image (S460), so that unnecessary wearable position calculation is not performed.

On the other hand, if it is determined that the driver's gaze direction is lateral or rearward in the next step, the marker next step position estimator stores the surrounding image in the storage (S455).

An embodiment of the present invention has been described above with reference to FIGS. 1 to 4 . Hereinafter, a preferred embodiment of the present invention that can be inferred from such an embodiment will be described.

A vehicle external image output control apparatus according to a preferred embodiment of the present invention includes an image acquisition unit, a viewing angle calculation unit, a target image generation unit, an image display unit, a power supply unit, and a main control unit.

The power supply unit performs a function of supplying power to each component constituting the external image output control device of the vehicle. The main controller performs a function of controlling the overall operation of each component constituting the external image output control device of the vehicle. Considering that the vehicle external image output control device is mounted on the vehicle, the power supply unit and the main control unit may not be provided in the present embodiment.

The image acquisition unit performs a function of acquiring an image of the outside of the vehicle. The image acquisition unit is a concept corresponding to the image collection unit 120 of FIGS. 1 and 4 .

The image acquisition unit may acquire images of directions other than the front of the vehicle as an image of the outside of the vehicle or may acquire images of all directions of the vehicle.

The viewing angle calculator performs a function of calculating the driver's viewing angle using a marker attached to the wearable device worn by the driver. The viewing angle calculator is a concept corresponding to the viewing angle measurement logic unit 110 of FIGS. 1 and 4 .

The viewing angle calculator may use a device in the form of glasses as a wearable device. In the present embodiment, the wearable device may be implemented as a glasses-type wearable device such as smart glasses.

The viewing angle calculator may use markers attached to upper ends of both eyeglass frames and markers attached to lower ends of both eyeglass frames as markers.

The viewing angle calculator may calculate the driver's viewing angle based on the position and movement direction of the driver's eyes acquired by the marker and the position of the marker acquired by the camera.

The viewing angle calculator may track the movement of the driver's eyes in real time within a range of 90 degrees to the left and right with respect to the front of the vehicle by using the marker. When it is determined that the driver's gaze deviates from 90 degrees to the left or out of 90 degrees to the right with respect to the front of the vehicle (0 degrees), the viewing angle calculation unit determines that the driver is looking to the side or rear of the vehicle and no longer drives the driver. It does not track eye movements.

The viewing angle calculator may further use markers attached to left ends of both eyeglass frames and markers attached to right ends of both eyeglass frames as markers.

The target image generator extracts an image in at least one direction from an image of the outside of the vehicle based on the driver's viewing angle and generates an output target image. The target image generating unit is a concept corresponding to the additional information attachment logic unit 130 of FIGS. 1 and 4 .

The target image generator may generate an output target image as a stereo image when an image for at least one direction is extracted from an image for the outside of the vehicle. The target image generator may generate the output target image as a panoramic image when images for at least two directions are extracted from the image for the exterior of the vehicle.

The target image generator may generate an output target image when it is determined that the viewing angle deviates from 90 degrees to the left or 90 degrees to the right with respect to the front of the vehicle.

The image display unit performs a function of displaying an output target image on the wearable device. The image display unit is a concept corresponding to the wearable display image transmission unit 140 of FIGS. 1 and 4 .

The image display unit may use a device in the form of glasses as a wearable device.

Next, a method of operating an external image output control device of a vehicle will be described.

First, an image of the outside of the vehicle is acquired.

Thereafter, the driver's viewing angle is calculated using a marker attached to the wearable device worn by the driver.

Thereafter, an image of at least one direction is extracted from an image of the outside of the vehicle based on the driver's viewing angle and generated as an output target image.

Thereafter, the output target image is displayed on the wearable device.

Even if all the components constituting the embodiment of the present invention described above are described as being combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, although all the components may be implemented as one independent hardware, some or all of the components are selectively combined to perform some or all of the functions of one or more hardware components It may be implemented as a computer program having In addition, such a computer program is stored in a computer readable media such as a USB memory, a CD disk, a flash memory, etc., read and executed by a computer, thereby implementing an embodiment of the present invention. The computer program recording medium may include a magnetic recording medium, an optical recording medium, a carrier wave medium, and the like.

In addition, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless otherwise defined in the detailed description. Commonly used terms such as terms defined in the dictionary should be interpreted as being consistent with the contextual meaning of the related art, and are not interpreted in an ideal or excessively formal meaning unless explicitly defined in the present invention.

The above description is merely illustrative of the technical idea of the present invention, and various modifications, changes, and substitutions are possible within the range that does not depart from the essential characteristics of the present invention by those of ordinary skill in the art to which the present invention pertains. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are for explaining, not limiting, the technical spirit of the present invention, and the scope of the technical spirit of the present invention is not limited by these embodiments and the accompanying drawings. . The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (15)

an image acquisition unit for acquiring an image of the outside of the vehicle;
a viewing angle calculator configured to calculate a viewing angle of the driver using a marker attached to a wearable device worn by the driver;
a target image generator for extracting an image of at least one direction from the image of the outside of the vehicle based on the viewing angle and generating an output target image; and
An image display unit for displaying the output target image on the wearable device
including,
The viewing angle calculator is configured to photograph the driver through a camera provided in the vehicle, calculate marker position information including a position of a marker attached to a wearable device worn by the driver from the captured image, and calculate the calculated marker. The apparatus for controlling an external image output of a vehicle, characterized in that the driver's viewing angle is calculated based on location information. The method of claim 1,
The image acquisition unit acquires images for the directions other than the front of the vehicle as the image for the outside of the vehicle or acquires images for all directions of the vehicle. The method of claim 1,
The apparatus for controlling an image output outside the vehicle, characterized in that the viewing angle calculating unit and the image display unit use a glasses-type device as the wearable device. 4. The method of claim 3,
and the viewing angle calculator uses markers attached to upper ends of both eyeglass frames and markers attached to lower ends of both eyeglass frames as the markers. delete The method of claim 1,
and the viewing angle calculating unit tracks, in real time, the movement of the driver's eyes within a range of 90 degrees left and right with respect to the front of the vehicle by using the marker. 5. The method of claim 4,
The viewing angle calculator further uses the markers attached to the left ends of the both eyeglass frames and the markers attached to the right ends of the both eyeglass frames as the markers. The method of claim 1,
The target image generating unit generates the output target image as a stereo image, and when images for at least two directions are extracted from the image for the outside of the vehicle, the output target image is generated as a panoramic image. controller. The method of claim 1,
The target image generating unit generates the output target image when it is determined that the viewing angle deviates from 90 degrees to the left or 90 degrees to the right with respect to the front of the vehicle. acquiring an image of the outside of the vehicle;
calculating a viewing angle of the driver using a marker attached to a wearable device worn by the driver;
extracting an image in at least one direction from the image of the exterior of the vehicle based on the viewing angle and generating an image to be output; and
displaying the output target image on the wearable device
including,
The calculating may include photographing the driver through a camera provided in the vehicle, calculating marker position information including a position of a marker attached to a wearable device worn by the driver from the captured image, and calculating the calculated position. The method for controlling the output of an image outside the vehicle, characterized in that the driver's viewing angle is calculated based on the marker position information. 11. The method of claim 10,
The calculating and displaying are performed by using a device in the form of glasses as the wearable device. 12. The method of claim 11,
In the calculating, the markers attached to the upper ends of both spectacle frames and the markers attached to the lower ends of the both spectacle frames are used as the markers. 11. The method of claim 10,
The calculating may include using the marker to track the movement of the driver's eyes within a range of 90 degrees to the left and right with respect to the front of the vehicle in real time. 11. The method of claim 10,
In the generating, the output target image is generated as a stereo image, and when images for at least two directions are extracted from the image for the outside of the vehicle, the output target image is generated as a panoramic image. control method. 11. The method of claim 10,
The generating step may include generating the output target image when it is determined that the viewing angle deviates from 90 degrees in the left direction or 90 degrees to the right with respect to the front of the vehicle.

Images