KR20240020051A - A method of calibrating a head up display using an in-car camera and an apparatus of calibrating a head up display using an in-car camera - Google Patents

Abstract

A method of measuring optical properties according to embodiments includes generating a multi-viewpoint image, the multi-viewpoint image including a left image, a center image, and a right image; Displaying the position of a virtual image for a multi-view image; calculating the depth of the position of the virtual image; estimating the depth of the virtual image; Matching the depth of the location and the depth of the virtual image; and calibrating the error of the multi-view image based on the matched depth; may include. In addition, the method of measuring optical properties includes generating a multi-view image, the multi-view image including a left image, a center image, and a right image; Displaying positions of virtual images for multi-view images; estimating the depth of locations in the virtual image; Grouping locations based on depth; selecting some locations from the grouped locations; and projecting the corrected virtual object based on some selected positions and depths. may include.

Description

HUD calibration method and device using a car interior camera {A METHOD OF CALIBRATING A HEAD UP DISPLAY USING AN IN-CAR CAMERA AND AN APPARATUS OF CALIBRATING A HEAD UP DISPLAY USING AN IN-CAR CAMERA}

The present invention relates to a HUD calibration method and device using an automobile interior camera, and more specifically, to a method and device for measuring optical characteristics of a three-dimensional virtual image generated by an augmented reality device.

Augmented reality (AR) is a field of virtual reality (AR) and is a computer graphics technique that synthesizes virtual objects or information in the real environment to make them look like objects that exist in the original environment. It is frequently used in digital media. It is used.

Augmented reality is also called mixed reality (MR) because it combines the real world with a virtual world with additional information in real time and displays it as a single image. Research and development has been underway in the United States since the late 1990s on a hybrid VR system that fuses real and virtual environments.

For example, augmented reality can be used in remote medical diagnosis, broadcasting, architectural design, and manufacturing process management. In addition, with the recent widespread adoption of smartphones, it has entered a full-fledged commercialization phase, and various products are being developed in the game and mobile solution industries as well as in the education field.

Meanwhile, a wearable computer may be used to realize augmented reality outdoors. In particular, a head mounted display (HMD) enables augmented reality by displaying computer graphics and text in real time over the actual environment seen by the user. Additionally, a head up display (HUD) enables augmented reality by displaying various information necessary for vehicle operation on the outside of the vehicle's windshield.

For example, a head-up display displays the light source output from the inside of the vehicle to the outside of the windshield on a virtual plane located outside the windshield of the vehicle, so that the driver can view the vehicle in that virtual plane without moving his or her eyes while driving. Augmented reality can be implemented by providing information necessary for operation.

At this time, geometric characteristics, including the position of the virtual plane formed by the augmented reality device, can be determined according to the optical characteristics of individual augmented reality devices such as HMD and HUD.

Accordingly, there is an emerging need for methods and devices that can measure optical characteristics of the output of augmented reality devices.

Related prior art includes Korean Patent Publication No. 10-2017-0114375 (Title of the invention: Method and device for displaying virtual reality content, Publication date: October 16, 2017).

The present invention seeks to provide a method and device for measuring optical characteristics of a virtual image generated by an augmented reality device.

In addition, the present invention uses the optical characteristics of the virtual image generated by the augmented reality device to determine the distance to the virtual image and the lookdown/up angle of the virtual image based on the user of the augmented reality device. It is intended to provide a method and device for calculating down/up angle, horizontal/vertical field of view, static distortion, ghosting level, etc.

The problem to be solved by the present invention is not limited to the problem(s) mentioned above, and other problem(s) not mentioned will be clearly understood by those skilled in the art from the description below.

In order to achieve the above object, the method for measuring the optical properties of an augmented reality device provided by the present invention uses a plurality of cameras arranged around a predetermined measurement reference position, and multiple cameras are output on a virtual plane by the augmented reality device. Taking a test image including a pattern; Obtaining angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras; and calculating coordinates of the plurality of patterns based on the measurement reference position based on the plurality of captured images captured by the plurality of cameras, the angle of view information, and the arrangement information.

Preferably, the plurality of cameras are a central camera located at the measurement reference position, a left camera and a right camera located symmetrically around the measurement reference position, and the plurality of patterns are aligned horizontally and vertically in the test image. When arranged, the step of calculating the coordinates of the plurality of patterns includes the number of horizontal pixels of the plurality of captured images, the coordinates of the plurality of patterns in the plurality of captured images, and the plurality of cameras included in the angle of view information. The coordinates of the plurality of patterns can be calculated using the angle of view and the distance between the left camera and the right camera included in the arrangement information.

Preferably, in calculating the coordinates of the plurality of patterns, the coordinates of the plurality of patterns may be calculated using Equation 1.

[Equation 1]

Here, x _ij , y _ij , and z _ij are the x, y, and z-axis coordinates of the horizontal i-th and vertical j-th patterns based on the measurement reference position, and α is the distance between the left camera and the right camera, M is the number of horizontal pixels of the plurality of captured images, θ is the angle of view of the plurality of cameras, m ^L _ij is the horizontal coordinate of the i-th horizontal and j-th pattern in the captured images of the left camera, m ^R _ij is the horizontal coordinate of the i-th horizontal and j-th pattern vertically in the captured image of the right camera, and m ^C _ij is the horizontal coordinate of the i-th horizontal and j-th vertical pattern in the captured image of the center camera. It is a direction coordinate.

Preferably, calculating a virtual image distance between the measurement reference position and the virtual plane using the coordinates of the measurement reference position and the coordinates of at least one of the plurality of patterns on the virtual plane. More may be included.

Preferably, in calculating the virtual image distance, the virtual image distance may be calculated using Equation 2.

[Equation 2]

Here, D _VI is the virtual image distance, and x ₂₂ , y ₂₂ , and z ₂₂ are the coordinates of one of the plurality of patterns.

Preferably, a look down/up angle from the measurement reference position to the virtual plane is determined by using the coordinates of the measurement reference position and the coordinates of at least one of the plurality of patterns on the virtual plane. It may further include a step of calculating .

Preferably, in calculating the lookdown/up angle, the lookdown/up angle may be calculated using Equation 3.

[Equation 3]

Here, θ _down/up is the lookdown/up angle, and x ₂₂ , y ₂₂ , and z ₂₂ are the coordinates of one of the plurality of patterns.

Preferably, the horizontal field of view of the measurement reference position is calculated using the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the horizontal direction among the plurality of patterns on the virtual plane. Additional steps may be included.

Preferably, in calculating the horizontal viewing angle, the horizontal viewing angle may be calculated using Equation 4.

[Equation 4]

Here, θ ^H _FOV is the horizontal viewing angle, O is the coordinate of the measurement reference position, and P ₂₁ and P ₂₃ are the coordinates of the two patterns located at both ends in the horizontal direction.

Preferably, the vertical field of view of the measurement reference position is calculated using the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the vertical direction among the plurality of patterns on the virtual plane. Additional steps may be included.

Preferably, in calculating the vertical viewing angle, the vertical viewing angle may be calculated using Equation 5.

[Equation 5]

Here, θ ^V _FOV is the vertical viewing angle, O is the coordinate of the measurement reference position, and P ₁₂ and P ₃₂ are the coordinates of the two patterns located at both ends in the vertical direction.

Preferably, the step of calculating static distortion for each of the three axes based on the measurement reference position may be further included, based on the coordinates of the plurality of patterns on the virtual plane.

Preferably, the step of calculating the coordinates of the plurality of patterns further calculates the coordinates of a plurality of ghost patterns corresponding to each of the plurality of patterns, based on the coordinates of the plurality of patterns and the coordinates of the plurality of ghost patterns. Therefore, a step of calculating a ghosting level may be further included.

In addition, in order to achieve the above object, the optical characteristic measuring device of the augmented reality device provided by the present invention uses a plurality of cameras arranged around a predetermined measurement reference position, and outputs the output on a virtual plane by the augmented reality device. a photographing unit that photographs a test image including a plurality of patterns; an acquisition unit that acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras; and a calculation unit that calculates coordinates of the plurality of patterns based on the measurement reference position, based on the plurality of captured images captured by the plurality of cameras, the angle of view information, and the arrangement information.

Preferably, the plurality of cameras are a central camera located at the measurement reference position, a left camera and a right camera located symmetrically around the measurement reference position, and the plurality of patterns are aligned horizontally and vertically in the test image. When arranged, the calculation unit determines the number of horizontal pixels of the plurality of captured images, the coordinates of the plurality of patterns in the plurality of captured images, the angle of view of the plurality of cameras included in the angle of view information, and the arrangement information. Using the distance between the left camera and the right camera, the coordinates of the plurality of patterns can be calculated.

Preferably, the calculation unit may calculate the coordinates of the plurality of patterns using Equation 6.

[Equation 6]

Here, x _ij , y _ij , and z _ij are the x, y, and z-axis coordinates of the horizontal i-th and vertical j-th patterns based on the measurement reference position, and α is the distance between the left camera and the right camera, M is the number of horizontal pixels of the plurality of captured images, θ is the angle of view of the plurality of cameras, m ^L _ij is the horizontal coordinate of the i-th horizontal and j-th pattern in the captured images of the left camera, m ^R _ij is the horizontal coordinate of the i-th horizontal and j-th pattern vertically in the captured image of the right camera, and m ^C _ij is the horizontal coordinate of the i-th horizontal and j-th vertical pattern in the captured image of the center camera. It is a direction coordinate.

Preferably, the calculation unit may further calculate a virtual image distance between the measurement reference position and the virtual plane using the coordinates of the measurement reference position and the coordinates of at least one of the plurality of patterns on the virtual plane.

Preferably, the calculation unit may calculate the virtual image distance using Equation 7.

[Equation 7]

Here, D _VI is the virtual image distance, and x ₂₂ , y ₂₂ , and z ₂₂ are the coordinates of one of the plurality of patterns.

Preferably, the calculation unit further calculates the lookdown/up angle with respect to the virtual plane from the measurement reference position using the coordinates of the measurement reference position and the coordinates of at least one of the plurality of patterns on the virtual plane. You can.

Preferably, the calculation unit may calculate the lookdown/up angle using Equation 8.

[Equation 8]

Here, θ _down/up is the lookdown/up angle, and x ₂₂ , y ₂₂ , and z ₂₂ are the coordinates of one of the plurality of patterns.

Preferably, the calculation unit may further calculate the horizontal viewing angle of the measurement reference position using the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the horizontal direction among the plurality of patterns on the virtual plane. there is.

Preferably, the calculation unit may calculate the horizontal viewing angle using Equation 9.

[Equation 9]

Here, θ ^H _FOV is the horizontal viewing angle, O is the coordinate of the measurement reference position, and P ₂₁ and P ₂₃ are the coordinates of the two patterns located at both ends in the horizontal direction.

Preferably, the calculation unit may further calculate the vertical viewing angle of the measurement reference position using the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the vertical direction among the plurality of patterns on the virtual plane. there is.

Preferably, the calculation unit may calculate the vertical viewing angle using Equation 10.

[Equation 10]

Here, θ ^V _FOV is the vertical viewing angle, O is the coordinate of the measurement reference position, and P ₁₂ and P ₃₂ are the coordinates of the two patterns located at both ends in the vertical direction.

Preferably, the calculation unit may further calculate static distortion for each of the three axes based on the measurement reference position, based on the coordinates of the plurality of patterns on the virtual plane.

Preferably, the calculation unit further calculates coordinates of a plurality of ghost patterns corresponding to each of the plurality of patterns based on the plurality of captured images, the angle of view information, and the arrangement information, and calculates the coordinates of the plurality of patterns and Based on the coordinates of the plurality of ghost patterns, the ghosting level can be further calculated.

The present invention has the effect of easily measuring the optical characteristics of a virtual image generated by an augmented reality device by using a plurality of cameras.

In addition, the present invention uses the optical characteristics of the virtual image generated by the augmented reality device to determine the distance to the virtual image and the lookdown/up angle of the virtual image based on the user of the augmented reality device. It has the effect of calculating down/up angle, horizontal/vertical field of view, static distortion, ghosting level, etc.

1 is a flowchart showing a method of measuring optical characteristics of an augmented reality device according to an embodiment of the present invention.
Figure 2 is a flowchart showing a method for calculating a virtual image distance according to an embodiment of the present invention.
Figure 3 is a flowchart showing a lookdown/up angle calculation method according to an embodiment of the present invention.
Figure 4 is a flowchart showing a method for calculating a horizontal viewing angle according to an embodiment of the present invention.
Figure 5 is a flowchart showing a method for calculating a vertical viewing angle according to an embodiment of the present invention.
Figure 6 is a flowchart showing a method for calculating static distortion according to an embodiment of the present invention.
Figure 7 is a flowchart showing a method for calculating a ghosting level according to an embodiment of the present invention.
Figure 8 is a block diagram showing an apparatus for measuring optical characteristics of an augmented reality device according to an embodiment of the present invention.
9A and 9B are diagrams for explaining an environment for measuring optical characteristics of an augmented reality device according to an embodiment of the present invention.
Figure 10 is a diagram for explaining the results of shooting a test image on a virtual plane using a plurality of cameras according to an embodiment of the present invention.
11A and 11B are diagrams for explaining the coordinates of a plurality of patterns included in a captured image using a plurality of cameras according to an embodiment of the present invention.
Figures 12a and 12b are diagrams for explaining a method of calculating coordinates of a plurality of patterns according to an embodiment of the present invention.
Figure 13 is a diagram for explaining a method of calculating a virtual image distance according to an embodiment of the present invention.
Figures 14a and 14b are diagrams for explaining a method of calculating lookdown/up angles according to an embodiment of the present invention.
15A and 15B are diagrams for explaining a method of calculating the horizontal viewing angle and vertical viewing angle according to an embodiment of the present invention.
Figure 16 is a diagram for explaining a method of calculating static distortion according to an embodiment of the present invention.
Figure 17 is a diagram for explaining a method of calculating a ghosting level according to an embodiment of the present invention.
Figure 18 shows a method of measuring optical properties according to embodiments.
Figure 19 shows an example of a multi-viewpoint image according to embodiments.
Figure 20 shows a method for displaying the location of a virtual image.
Figure 21 shows a process for generating points and depth according to embodiments.
Figure 22 shows equal depth grouping according to embodiments.
Figure 23 shows HUD virtual object projection according to embodiments.
Figure 24 shows an optical specific device according to embodiments.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. While describing each drawing, similar reference numerals are used for similar components.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings.

The present invention relates to a method and device for measuring the optical characteristics of a virtual reality device, and the measurement can be performed in the following environment. For example, referring to FIG. 9A, the user's eyes are located in the eye box, and a virtual plane generated by the output of the virtual reality device may be formed outside a transparent or translucent screen (e.g., a vehicle's windshield). . At this time, the user can see the entire virtual plane by moving only his eyes. Additionally, referring to FIG. 9B, a plurality of cameras may be arranged in the eye box centered on the measurement reference position. More specifically, cam _C may be placed at the measurement reference position, and cam _L and cam _R may be placed at positions symmetrical to both sides. Meanwhile, a plurality of patterns may be arranged horizontally and vertically (e.g., 3x3) in the test image.

However, the present invention is not limited to being practiced only in such environments, and of course can be practiced in many different environments. For example, the location and size of the eye box, the number and arrangement of cameras, the number and arrangement of patterns included in the test image, etc. may vary depending on the measurement environment.

1 is a flowchart showing a method of measuring optical characteristics of an augmented reality device according to an embodiment of the present invention.

In step S110, the optical properties measurement device uses a plurality of cameras arranged around a predetermined measurement reference position to capture a test image including a plurality of patterns output on a virtual plane by the augmented reality device.

For example, referring to FIG. 9B, one camera may be placed at the measurement reference position located in the center of the eye box, and the remaining cameras may be placed symmetrically toward the front at the same height on both sides of the camera.

At this time, the optical characteristic measurement device can be connected to a plurality of cameras wirelessly or wired and transmit a command to photograph a test image on a virtual plane.

In step S120, the optical characteristic measuring device acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

For example, the optical characteristic measuring device may receive information about the angle of view of the camera and information about the arrangement of the camera from the user, and obtain angle of view information and arrangement information. Preferably, the information regarding the camera's angle of view may be a horizontal angle of view, and the information regarding the arrangement of the camera may be the separation distance between cameras symmetrically arranged on both sides of the measurement reference position.

Finally, in step S130, the optical characteristic measurement device calculates the coordinates of a plurality of patterns based on the measurement reference position based on the plurality of captured images, view angle information, and arrangement information captured by the plurality of cameras.

At this time, the optical characteristic measuring device includes information about the size of the plurality of captured images, information about the coordinates within the image of the plurality of patterns included in the plurality of captured images, information about the angle of view of the plurality of cameras, and information about the angle of view of the plurality of cameras. Using information about the arrangement, the three-dimensional coordinates of a plurality of patterns on a virtual plane with the measurement reference position as the origin (0, 0, 0) can be calculated.

Meanwhile, a detailed method of calculating the coordinates of a plurality of patterns will be described in detail later in the examples below.

In another embodiment, the optical property measurement device includes a central camera located at the measurement reference position, a left camera and a right camera located symmetrically around the measurement reference position, and a plurality of patterns are displayed horizontally and vertically in the test image. When aligned and arranged, the number of horizontal pixels in the plurality of captured images, the coordinates of the plurality of patterns in the plurality of captured images, the angle of view of the plurality of cameras included in the angle of view information, and the left camera and right camera included in the arrangement information. The coordinates of a plurality of patterns can be calculated using the distance between them.

For example, referring to FIG. 9B, the plurality of cameras may be a central camera (cam _C ) located at the measurement reference position, and a left camera (cam _L ) and a right camera (cam _R ) located symmetrically around the measurement reference position. Additionally, nine patterns can be arranged horizontally and vertically in the test image.

At this time, the optical characteristic measurement device includes the number of horizontal pixels of the plurality of captured images, the coordinates of the plurality of patterns in the plurality of captured images, the angle of view of the plurality of cameras included in the angle of view information, and the left and right cameras included in the arrangement information. Using the distance between them, the three-dimensional coordinates of each of the nine patterns on a virtual plane with the measurement reference position as the origin (0, 0, 0) can be calculated.

In another embodiment, an optical property measuring device may calculate the coordinates of a plurality of patterns using Equation 1.

[Equation 1]

Here, x _ij , y _ij , and z _ij are the x, y, and z-axis coordinates of the horizontal i-th and vertical j-th patterns based on the measurement reference position, α is the distance between the left and right cameras, and M is a plurality of is the number of horizontal pixels of the captured image, θ is the angle of view of multiple cameras, m ^L _ij is the horizontal coordinate of the i-th horizontal and j-th pattern in the captured image of the left camera, and m ^R _ij is the horizontal coordinate of the j-th pattern in the captured image of the left camera. These are the horizontal coordinates of the horizontal i-th and vertical j-th patterns in the captured image, and m ^C _ij are the horizontal coordinates of the horizontal i-th and vertical j-th patterns in the captured image of the central camera.

At this time, referring to Figure 10, the central camera (cam _C ) is placed at the measurement reference position, which is the center of the eye box, and the left camera (cam _L ) and right camera (cam _R ) are placed spaced apart at a distance of α. You can. In addition, the optical characteristic measurement device can shoot a test image on the virtual plane using the central camera (cam _C ), left camera (cam _L ), and right camera (cam _R ) arranged to face the front. As a result, the test image captured using the left camera (cam _L ) (captured image by cam _L ) is biased to the right, and the test image captured using the central camera (cam _C ) (captured image by cam _C ) is biased to the right. ), the test image is not biased, and the test image captured using the right camera (cam _R ) (captured image by cam _R ) may be biased to the left.

Meanwhile, referring to Figure 11a, the three-dimensional coordinates of nine patterns appearing on the virtual plane can be expressed as P _{ij =} (x _ij , y _ij , z _ij ), where i is the horizontal index of the pattern (i = 1, 2,3), j may be the vertical index of the pattern (j=1,2,3). In other words, P _ij may be the three-dimensional coordinate of the center of the horizontal i-th and vertical j-th pattern.

Additionally, referring to FIG. 11b, the pixel coordinates of the nine patterns appearing in the captured image can be expressed as P ^L _ij , P ^C _ij , and P ^R _ij , respectively, for the left camera (cam _L ), central camera (cam _C ), and It may refer to the coordinates of the pattern that appears in the image captured by the right camera (cam _R ). At this time, P ^L _ij = (m ^L _ij , n ^L _ij ), P ^C _ij = (m ^C _ij , n ^C _ij ), P ^R _ij = (m ^R _ij , n ^R _ij ). At this time, P ^L _ij , P ^C _ij , and P ^R _ij may be the pixel coordinates of the center of the i-th horizontal and j-th pattern.

Meanwhile, referring to FIG. 12a, it can be seen that the proportional relationship shown in Equation 2 below is established.

[Equation 2]

Here, z is the distance along the z-axis from the measurement reference position to the virtual plane, θ is the camera's angle of view, α is the distance between the left and right cameras, and m ^L _ij is the horizontal i in the captured image of the left camera. are the horizontal coordinates of the th and vertical j-th patterns, m ^R _ij are the horizontal coordinates of the i-th horizontal and j-th vertical patterns in the image captured by the right camera, and M is the number of horizontal pixels in the captured image.

At this time, it is obvious that equation 1 can be obtained by modifying equation 2.

For example, referring to Figure 12b, the optical characteristic measurement device takes pictures of the same pattern (i=1, j=1) using the central camera (cam _C ), left camera (cam _L ), and right camera (cam _R ). Using one result, x ₁₁ , y ₁₁ , and z ₁₁ can be calculated through Equation 1.

Figure 2 is a flowchart showing a method for calculating a virtual image distance according to an embodiment of the present invention.

*112 In step S210, the optical properties measurement device uses a plurality of cameras arranged around a predetermined measurement reference position to capture a test image containing a plurality of patterns output on a virtual plane by the augmented reality device. .

In step S220, the optical characteristic measuring device acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

In step S230, the optical characteristic measurement device calculates the coordinates of a plurality of patterns based on the measurement reference position based on the plurality of captured images captured by the plurality of cameras, the angle of view information, and the arrangement information.

Finally, in step S240, the optical property measurement device calculates a virtual image distance between the measurement reference position and the virtual plane using the coordinates of the measurement reference position and the coordinates of at least one of a plurality of patterns on the virtual plane. .

For example, referring to Figure 13, the optical properties measurement device calculates the distance to the coordinates (x ₂₂ , y ₂₂ , z ₂₂ ) of P ₂₂ based on the measurement reference position (0, 0, 0) to obtain a virtual image distance. can be calculated.

In another embodiment, the optical characteristic measurement device may calculate the virtual image distance using Equation 3.

[Equation 3]

Here, D _VI is the virtual image distance, and x ₂₂ , y ₂₂ , and z ₂₂ are the three-dimensional coordinates of the pattern with i=2 and j=2.

Figure 3 is a flowchart showing a lookdown/up angle calculation method according to an embodiment of the present invention.

In step S310, the optical properties measurement device uses a plurality of cameras arranged around a predetermined measurement reference position to capture a test image including a plurality of patterns output on a virtual plane by the augmented reality device.

In step S320, the optical characteristic measuring device acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

In step S330, the optical characteristic measurement device calculates the coordinates of a plurality of patterns based on the measurement reference position based on the plurality of captured images, view angle information, and arrangement information captured by the plurality of cameras.

Finally, in step S340, the optical property measurement device determines a look down/up angle from the measurement reference position to the virtual plane using the coordinates of the measurement reference position and at least one coordinate of a plurality of patterns on the virtual plane. angle) is calculated.

At this time, the lookdown/up angle is an angle representing the difference between the height of the eyebox and the virtual plane, and indicates whether the user is looking up or down at the virtual plane.

For example, when the coordinates (x ₂₂ , y ₂₂ , z ₂₂ ) of P ₂₂ are calculated based on (0, 0, 0), which is the measurement reference position where the user's eyes are located, if y ₂₂ < 0, as shown in Figure 14a It can be a look down situation, and if y ₂₂ > 0, it can be a look up situation as shown in FIG. 14b.

In another embodiment, the optical property measurement device may calculate the lookdown/up angle using Equation 4.

[Equation 4]

Here, θ _down/up is the lookdown/up angle, and x ₂₂ , y ₂₂ , and z ₂₂ are the three-dimensional coordinates of the pattern with i=2 and j=2.

Figure 4 is a flowchart showing a method for calculating a horizontal viewing angle according to an embodiment of the present invention.

In step S410, the optical properties measurement device uses a plurality of cameras arranged around a predetermined measurement reference position to capture a test image including a plurality of patterns output on a virtual plane by the augmented reality device.

In step S420, the optical characteristic measuring device acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

In step S430, the optical characteristic measurement device calculates the coordinates of a plurality of patterns based on the measurement reference position based on the plurality of captured images captured by the plurality of cameras, view angle information, and arrangement information.

Finally, in step S440, the optical properties measurement device uses the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the horizontal direction among a plurality of patterns on the virtual plane to determine the horizontal field of view of the measurement reference position. ) is calculated.

For example, referring to Figure 15a, the optical properties measurement device has three-dimensional coordinates O = (0, 0, 0) of the measurement reference position, and P ₂₁ = two patterns located at both ends in the horizontal direction among a plurality of patterns on the virtual plane. Using the three-dimensional coordinates of (x ₂₁ , y ₂₁ , z ₂₁ ) and P ₂₃ = (x ₂₃ , y ₂₃ , z ₂₃ ), the angle ∠P ₂₁ OP ₂₃ can be calculated as the horizontal viewing angle.

In another embodiment, the optical characteristic measuring device may calculate the horizontal viewing angle using Equation 5.

[Equation 5]

Here, θ ^H _FOV is the horizontal viewing angle, O is the coordinate of the measurement reference position, and P ₂₁ and P ₂₃ are the coordinates of the two patterns located at both ends in the horizontal direction among the plurality of patterns.

Figure 5 is a flowchart showing a method for calculating a vertical viewing angle according to an embodiment of the present invention.

In step S510, the optical properties measurement device uses a plurality of cameras arranged around a predetermined measurement reference position to capture a test image including a plurality of patterns output on a virtual plane by the augmented reality device.

In step S520, the optical characteristic measuring device acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

In step S530, the optical characteristic measurement device calculates the coordinates of a plurality of patterns based on the measurement reference position based on the plurality of captured images captured by the plurality of cameras, the angle of view information, and the arrangement information.

Finally, in step S540, the optical property measurement device uses the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the vertical direction among a plurality of patterns on the virtual plane to determine the vertical field of view of the measurement reference position. ) is calculated.

For example, referring to FIG. 15b, the optical properties measurement device has three-dimensional coordinates O = (0, 0, 0) of the measurement reference position, and P ₁₂ = two patterns located at both ends in the vertical direction among a plurality of patterns on the virtual plane. Using the three-dimensional coordinates of (x ₁₂ , y ₁₂ , z ₁₂ ) and P ₃₂ = (x ₃₂ , y ₃₂ , z ₃₂ ), the angle ∠P ₁₂ OP ₃₂ can be calculated as the vertical viewing angle.

In another embodiment, the optical property measurement device may calculate the vertical viewing angle using Equation 6.

[Equation 6]

Here, θ ^V _FOV is the vertical viewing angle, O is the coordinate of the measurement reference position, and P ₁₂ and P ₃₂ are the coordinates of the two patterns located at both ends in the vertical direction.

Figure 6 is a flowchart showing a method for calculating static distortion according to an embodiment of the present invention.

In step S610, the optical properties measurement device uses a plurality of cameras arranged around a predetermined measurement reference position to capture a test image including a plurality of patterns output on a virtual plane by the augmented reality device.

In step S620, the optical characteristic measuring device acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

In step S630, the optical characteristic measurement device calculates the coordinates of a plurality of patterns based on the measurement reference position based on the plurality of captured images captured by the plurality of cameras, the angle of view information, and the arrangement information.

Finally, in step S640, the optical properties measurement device calculates static distortion for each of the three axes based on the measurement reference position, based on the coordinates of a plurality of patterns on the virtual plane.

At this time, static distortion is caused by the projection of the virtual reality device. Referring to FIG. 16, the deviation of the three-dimensional coordinates of a plurality of patterns based on the line corresponding to each of the three axes (x, y, z) Indicates deviation degree.

Meanwhile, the optical properties measurement device can calculate the static distortion for each of the three axes using Equation 7.

[Equation 7]

^{Here ,} _{DT _} ^_ _{_ _ _ _} These are the x, y, and z coordinates of the vertical b (b=1,2,3)th pattern.

Figure 7 is a flowchart showing a method for calculating a ghosting level according to an embodiment of the present invention.

In step S710, the optical property measurement device uses a plurality of cameras arranged around a predetermined measurement reference position to capture a test image including a plurality of patterns output on a virtual plane by the augmented reality device.

In step S720, the optical characteristic measuring device acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

In step S730, the optical characteristic measurement device determines the coordinates of a plurality of patterns and the coordinates of a plurality of ghost patterns based on the measurement reference position based on the plurality of captured images, view angle information, and arrangement information captured by the plurality of cameras. Calculate .

For example, a ghost pattern may appear on a vehicle's windshield, which transmits half of the incoming light and reflects the other half. More specifically, referring to FIG. 17, the two physical layers of the windshield cause a ghost phenomenon, so that the pattern on the virtual plane and the ghost pattern corresponding to the pattern appear to the user as double images overlapping or blurred. It can be seen (blurred).

At this time, the optical characteristic measuring device may calculate the coordinates of a plurality of ghost patterns corresponding to each of the plurality of patterns using the same method as the method of calculating the coordinates of the plurality of patterns.

Finally, in step S740, the optical characteristic measuring device may calculate a ghosting level based on the coordinates of a plurality of patterns and the coordinates of a plurality of ghost patterns.

At this time, the optical characteristic measuring device can calculate the ghosting level from the gap between the original pattern and the corresponding ghost pattern.

More specifically, the optical property measurement device can calculate the ghosting level using Equation 8.

[Equation 8]

Here, Ghost is the ghosting level, and x _ij , y _ij , and z _ij are the x, y, z of the horizontal i (i=1,2,3)th and vertical j(j=1,2,3)th patterns. These are the coordinates, and x _Gij , y _Gij , and z _Gij are the x, y, z coordinates of the ith horizontal and jth ghost pattern vertically.

Figure 8 is a block diagram showing an apparatus for measuring optical characteristics of an augmented reality device according to an embodiment of the present invention.

Referring to FIG. 8, an apparatus 800 for measuring optical characteristics of an augmented reality device according to an embodiment of the present invention includes a photographing unit 810, an acquisition unit 820, and a calculation unit 830.

The photographing unit 810 uses a plurality of cameras arranged around a predetermined measurement reference position to photograph a test image including a plurality of patterns output on a virtual plane by an augmented reality device.

The acquisition unit 820 acquires angle of view information including information about the angle of view of the plurality of cameras and arrangement information including information about the arrangement of the plurality of cameras.

Finally, the calculation unit 830 calculates the coordinates of a plurality of patterns based on the measurement reference position based on the plurality of captured images, view angle information, and arrangement information captured by the plurality of cameras.

In another embodiment, the plurality of cameras are a central camera located at the measurement reference position, a left camera and a right camera located symmetrically around the measurement reference position, and when a plurality of patterns are arranged horizontally and vertically in the test image. , the calculation unit 830 calculates the number of horizontal pixels of the plurality of captured images, the coordinates of the plurality of patterns in the plurality of captured images, the angle of view of the plurality of cameras included in the angle of view information, and the distance between the left and right cameras included in the arrangement information. Using the distance, the coordinates of a plurality of patterns can be calculated.

In another embodiment, the calculation unit 830 may calculate the coordinates of a plurality of patterns using Equation 9.

[Equation 9]

Here, x _ij , y _ij , and z _ij are the x, y, and z-axis coordinates of the horizontal i-th and vertical j-th patterns based on the measurement reference position, α is the distance between the left and right cameras, and M is a plurality of is the number of horizontal pixels of the captured image, θ is the angle of view of multiple cameras, m ^L _ij is the horizontal coordinate of the i-th horizontal and j-th pattern in the captured image of the left camera, and m ^R _ij is the horizontal coordinate of the j-th pattern in the captured image of the left camera. These are the horizontal coordinates of the horizontal i-th and vertical j-th patterns in the captured image, and m ^C _ij are the horizontal coordinates of the horizontal i-th and vertical j-th patterns in the captured image of the central camera.

In another embodiment, the calculation unit 830 may further calculate the virtual image distance between the measurement reference position and the virtual plane using the coordinates of the measurement reference position and the coordinates of at least one of a plurality of patterns on the virtual plane.

In another embodiment, the calculation unit 830 may calculate the virtual image distance using Equation 10.

[Equation 10]

Here, D _VI is the virtual image distance, and x ₂₂ , y ₂₂ , and z ₂₂ are the coordinates of one pattern among a plurality of patterns.

In another embodiment, the calculation unit 830 may further calculate the lookdown/up angle with respect to the virtual plane from the measurement reference position using the coordinates of the measurement reference position and at least one coordinate of a plurality of patterns on the virtual plane. You can.

In another embodiment, the calculation unit 830 may calculate the lookdown/up angle using Equation 11.

[Equation 11]

Here, θ _down/up is the lookdown/up angle, and x ₂₂ , y ₂₂ , and z ₂₂ are the coordinates of one pattern among a plurality of patterns.

In another embodiment, the calculation unit 830 further calculates the horizontal viewing angle of the measurement reference position using the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the horizontal direction among a plurality of patterns on the virtual plane. You can.

In another embodiment, the calculation unit 830 may calculate the horizontal viewing angle using Equation 12.

[Equation 12]

Here, θ ^H _FOV is the horizontal viewing angle, O is the coordinate of the measurement reference position, and P ₂₁ and P ₂₃ are the coordinates of the two patterns located at both ends in the horizontal direction.

In another embodiment, the calculation unit 830 further calculates the vertical viewing angle of the measurement reference position using the coordinates of the measurement reference position and the coordinates of two patterns located at both ends in the vertical direction among a plurality of patterns on the virtual plane. You can.

In another embodiment, the calculation unit 830 may calculate the vertical viewing angle using Equation 13.

[Equation 13]

Here, θ ^V _FOV is the vertical viewing angle, O is the coordinate of the measurement reference position, and P ₁₂ and P ₃₂ are the coordinates of the two patterns located at both ends in the vertical direction.

In another embodiment, the calculation unit 830 may further calculate static distortion for each of the three axes based on the measurement reference position, based on the coordinates of a plurality of patterns on the virtual plane.

In another embodiment, the calculation unit 830 further calculates the coordinates of a plurality of ghost patterns corresponding to each of the plurality of patterns based on the plurality of captured images, view angle information, and arrangement information, and calculates the coordinates of the plurality of patterns and Based on the coordinates of a plurality of ghost patterns, the ghosting level can be further calculated.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium.

The computer-readable recording media includes magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical read media (eg, CD-ROM, DVD, etc.).

So far, the present invention has been examined focusing on its preferred embodiments. A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a restrictive perspective. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the equivalent scope should be construed as being included in the present invention.

9-10, cameras according to embodiments may correspond to a light measuring device (LMD). An optical measurement device according to embodiments may generate a virtual image plane and generate images including patterns at different positions. Methods for measuring optical properties according to embodiments include generating images including points in each pattern for a virtual image plane using one or more optical measurement devices, each image comprising one or more Captured based on light measurement devices, each image corresponding to at least one of a left image, a central image, and a right image; and generating positions of points based on one or more light measurement devices and each pattern; and the position may be obtained based on a field of view of one or more light measurement devices and a gap between the left light measurement device and the right light measurement device.

Referring to Figure 10, one LDM can capture three images for the virtual plane at the center, left, and right positions, and multiple LDMs can capture three images for the virtual plane at the center, left, and right positions. can do.

Referring to Figure 12, the positions of the points may be estimated based on, for example, the capture angle for the left LDM location and the capture angle for the right LDM location.

Based on Equation 1, the coordinate value of the position of the pattern of the image can be calculated.

Referring to Equation 1, by the method/apparatus for measuring optical properties according to embodiments, a left image including points in the pattern is captured based on the left light measurement device of one or more light measurement devices, and The central image containing the points is captured based on the central optical measurement device of one or more optical measuring devices, and the right image containing points in the pattern is captured based on the right optical measuring device of the one or more optical measuring devices. It can be. The index according to embodiments may indicate the number of positions, and the field of view may correspond to the angle of view.

Referring to Figure 13 and Equation 2, the position coordinate values are the horizontal pixel index of the left light measurement device, the horizontal pixel index of the right light measurement device, the horizontal pixel index of the center light measurement device, and the field of view of the left light measurement device. It can be calculated based on .

Referring to FIG. 14 and Equation 4, the method of measuring optical properties according to embodiments may further include measuring a virtual image distance for a virtual image plane based on the position in the pattern at the center and an optical measurement device. there is.

Referring to FIG. 15, the step of measuring the look down angle and look up angle for the virtual image plane may be further included based on the position within the pattern and the virtual image distance.

16, measuring a horizontal field of view for the virtual image plane based on a distance to a point left of center and a distance to a point right of center; and measuring a vertical field of view for the virtual image plane based on the distance to a central top point and the distance to the central bottom point; It may further include.

A method for measuring optical properties according to embodiments includes measuring horizontal distortion for a virtual image plane based on the center and a line between the center top point and the center bottom point; and measuring vertical distortion for the virtual image plane based on a line between center and a point left of center and a point right of center; may include.

An optical specific measurement method/device according to embodiments may be referred to as a method/device according to embodiments.

Figure 18 shows a method of measuring optical properties according to embodiments.

The optical properties measuring device according to the embodiments of FIGS. 8 and 25 can measure optical properties based on FIGS. 1 to 7, 18, etc.

1800, a method of measuring optical properties according to embodiments may include generating a multi-viewpoint image (i=1).

The photographing unit 810 may generate at least one image. The optical properties measuring apparatus according to the embodiments of FIG. 25 may generate at least one multi-viewpoint image. For example, n multi-view images may be generated.

The devices shown in FIGS. 8 and 25 may be referred to as a light measuring device (LMD), a camera, a processor equipped with a camera, an optical characteristic measuring device, etc.

Multi-view images according to embodiments are acquired by an optical properties measurement device according to embodiments mounted on a vehicle, an optical properties measurement device detachable from the vehicle, a light measuring device (LMD), a camera, a processor equipped with a camera, etc. It can be. The vehicle may include an optical characteristic measuring device near the dashboard, black box, and rearview mirror, an optical characteristic measuring device detachable from the vehicle, a light measuring device (LMD), a camera, and a processor equipped with a camera. One or more cameras can acquire multi-view images and can be installed at various locations in the vehicle.

1801 to 1802, the method for measuring optical properties according to embodiments may perform the following operations on one or more multi-viewpoint images. For example, when n multi-viewpoint images are captured, the following calibration operation can be applied to each n images.

1803, the method of measuring optical properties according to embodiments may further include displaying the position of the virtual image.

1804, the method of measuring optical properties according to embodiments may further include calculating the depth of the position of the virtual image.

1805, the method of measuring optical properties according to embodiments may further include estimating the depth of the virtual image.

1806, the method of measuring optical properties according to embodiments may further include matching the depth of the position and the depth of the virtual image.

1807, the method of measuring optical properties according to embodiments may further include a calibration step.

The multi-view image acquired in step 1800 may be a left image 1900, a center image 1901, and a right image 1902 captured by a camera or an optical measurement device.

In step 1803, positions 1 to 9 (changeable) of the HUD virtual image can be displayed on the multi-view image (see FIG. 20).

You can project HUD virtual objects on VID. The virtual object may be an image visible to the user's eyes. The HUD virtual object may be projected as circles 1 to 9 in FIG. 20.

In step 1804, the user can calculate the depth of the positions corresponding to circles 1 to 9 of FIG. 20. The depth of circles in patterns 1 to 9 may be calculated by at least one camera or light measurement device.

In step 1805, the frame and depth of the multi-view image can be acquired (see Figure 21).

In step 1805, the coordinate value of the geometric position of the virtual image can be obtained. For the process of obtaining coordinate values, refer to the explanations of Equation 1 to Equation 4.

Depth according to embodiments may mean the z value of a point of a virtual plane (image), such as in FIG. 11.

In step 1806, matching can be displayed as shown in FIG. 22.

In step 1807, storing positions with the same depth (for example, number 5, see FIG. 22) means calibration. Vehicles according to embodiments use the same location (e.g., No. 5) for the depth by the camera and the depth by the HUD when implementing the HUD, and use locations with different depths (e.g., No. 6 and No. 8, Figure 2). By ignoring (see 22), an accurate HUD can be implemented.

1803 indicates the step of displaying a depth i-phase virtual image by the HUD.

1804 shows the process of calculating depth i by the camera.

1805 shows the process of estimating depth i by HUD.

Therefore, in order to measure the optical characteristics of the HUD and perform calibration to improve HUD quality on a driver basis, the driver can select whether to match the depth of the virtual image and perform HUD correction.

Figure 19 shows an example of a multi-viewpoint image according to embodiments.

The multi-view image generated in Figure 18 is the same as Figure 19.

Based on the user, three images may be generated: a left image 1900, a center image 1901, and a right image 1902.

An optical specific measurement device according to embodiments may include at least one camera. An optical specific measurement device according to embodiments may include an optical measurement unit. The position of the light measurement unit of the specific optical measurement device may change or the light measurement unit may move in the position mounted on the vehicle. For example, as the light measurement unit moves to the center, left, right, etc., a multi-view image can be generated (acquired) as shown in FIG. 19.

Figure 20 shows a method for displaying the location of a virtual image.

Figure 20 shows step 1803 of Figure 19. An optical specific measurement method and device can display multiple positions of a HUD virtual image in a multi-view image. Positions according to embodiments may include positions 1 to 9. The virtual image is an image for the HUD and can have depth i.

The multi-view image 2000 is the multi-view image (left, center, right) described in FIGS. 18-19.

Virtual Image (2001) is a virtual image of the HUD. A virtual image can have a certain depth. The virtual image may be the box area 2001 in Figure 20.

Location (2002) is the location for virtual image (2001). Location (2002) is a changeable value. Location 2002 may have one or more location values. For example, there may be 9 locations for the virtual image, and may be referred to as first location, second location, ..., ninth location, etc.

A vehicle can project a HUD virtual object on a VID (Vehicle Information Device). The driver and user of the vehicle can visually see the virtual object.

The driver can select the location of the virtual image based on the UI. Some of the first to ninth positions may be displayed on the camera screen based on touch method, signal input transmission method, etc.

The method/device for measuring optical properties according to embodiments may display positions of the virtual image based on numbers, such as 2001.

Virtual images and locations are displayed on the HUD, and the driver can select one or more locations based on the UI. 2001 shows an example of the displayed results.

Figure 21 shows a process for generating points and depth according to embodiments.

The multi-view images 2100, 2101, and 2102 are the above-described multi-view images. Points 2103 are corresponding points of each multi-view image.

By step 1805, the coordinate value of point 2103 can be calculated. And, you can obtain a depth of point 2103.

To estimate the depth of the virtual image, the value of point 2103 of the virtual image can be used.

The depth i of point 2103 may be the depth of the virtual image. Depth i may be HUD depth i.

Through SW processing, corresponding points and depth can be obtained by changing the positions of multiple cameras or cameras (LDM).

Figure 22 shows equal depth grouping according to embodiments.

As shown in FIG. 22, the method/device according to embodiments can calculate the depth on the HUD virtual image and group positions with the same calculated depth using RGB colors.

For example, numbers 1, 2, 4, and 5 correspond to the same calculated depth location and are displayed in red color, numbers 3 and 6 are grouped in green color, and numbers 7 to 9 are grouped in blue color. Can be grouped into .

Drivers can select locations based on the UI. By touching the camera screen or HUD screen, you can select the same depth as the HUD's virtual object among red, green, and blue.

For example, if the depths at position 5 are the same, O can be displayed, and if the depths are different, X can be displayed.

Through SW processing, the same depth can be grouped using color information. By displaying it in the UI, the driver can select positions 1 to 9.

HUD calibration can be performed from selected colors. VID can display information based on the calculated depth.

By comparing the depths of positions 1 to 9 with the depth of the HUD virtual image (object), only the same depth can be selected. If the depth matches, the HUD quality may be better.

Figure 23 shows HUD virtual object projection according to embodiments.

The method/device according to embodiments may project a HUD virtual object on the VID as shown in FIG. 23 after the calibration process. When projecting an object, there may be an error tolerance range (e.g., +X meters, -meters).

In the above-described process, the depth can be calculated, and the HUD virtual object can be projected based on the corrected and selected depth.

By performing the process of Figure 18 at least once, HUD calibration optimized for the driver can be performed.

Figure 24 shows an optical specific device according to embodiments.

The method of Figure 18 can be performed by the device of Figure 24. Each component in Figure 24 may correspond to hardware, software, processor, and/or a combination thereof. Figure 24 may correspond to the device of Figure 8.

The light measurement unit may be a camera. It may be a light measurement unit that acquires virtual images in FIGS. 1 to 7 and 18.

The processor may be a processor that performs the operations of Figures 1 to 7, Figure 18, etc.

Memory can store data and information related to the operation of the processor. Memory can provide the necessary data to the processor. The memory may be connected to an optical measurement unit, processor, etc.

The optical measurement unit may capture multi-view images as shown in step 1800 and FIG. 19.

The processor may display positions 1 to 9 of the HUD virtual image on the multi-view image as in step 1803.

The processor may calculate the depth of positions 1 to 9 of the virtual image as in step 1804. The process of calculating the depth (coordinate Z value) of the pattern of the image can refer to Equations 1 to 4.

The calculated depth at positions 1 to 9 may be the camera depth i.

The processor may estimate the depth of the HUD virtual image as in step 1805. Corresponding points can be acquired at various locations by a camera or a light measurement device LDM, the coordinates of the points can be calculated, and the depth can be calculated.

The processor compares the camera depth i calculated (estimated) from the multi-view image (positions 1 to 9: some positions may be selected by the driver) and the virtual image HUD depth i (2103), and selects the same position. It can be displayed on the UI and grouped.

A calibrated HUD virtual object can be projected by grouping and selecting the same depths through the depths for specific locations in a multi-view image acquired by a camera or a light measurement device LDM.

As shown in Figure 18, the processor 1) projects a HUD virtual object on the VID (image visible to the user's eyes), 1-1) allows the driver to select the object based on the UI, 1-2) signals such as a touch on the camera screen can be received to display some of the positions 1-9, 2) through SW processing, obtain corresponding points and depth based on multiple LDM acquisition information, and 3) through SW processing, color the same depth. Grouping using information, 4) Providing UI-based selection information to the driver, 4-1) Receiving information such as touches on the screen such as camera, LDM, VID, etc., and selecting a HUD virtual object of the same depth as the HUD virtual object among specific colors. You can choose a location. Information required for driving can be calibrated and displayed on the VID based on the selected location and depth of the location. 5) HUD virtual objects can be projected onto the VID. Processes 1 to 5 can be performed repeatedly.

Referring to Figure 18, the method for measuring optical properties according to embodiments includes generating a multi-viewpoint image, the multi-viewpoint image including a left image, a center image, and a right image; Displaying the position of a virtual image for a multi-view image; calculating the depth of the position of the virtual image; estimating the depth of the virtual image; matching the depth of the location and the depth of the virtual image; and calibrating the error of the multi-view image based on the matched depth; may include.

Additionally, referring to Equations 1 to 4, the virtual image includes points in each pattern for the virtual image, and the positions of the points are generated based on one or more light measurement devices and each pattern.

Additionally, the left image including points within the pattern of the virtual image is captured based on the left side of one or more light measurement devices, and the center image including points within the pattern is captured based on the center of one or more light measurement devices. A right image is captured based on a light measurement device, and a right image comprising points in the pattern is captured based on a right light measurement device of one or more light measurement devices.

Additionally, the coordinate values of the position are calculated based on the horizontal pixel index of the left light measurement device, the horizontal pixel index of the right light measurement device, the horizontal pixel index of the central light measurement device, and the field of view of the left light measurement device.

Additionally, the method of measuring optical properties may further include measuring a virtual image distance for a virtual image based on the position within the central pattern and the optical measurement device.

Additionally, the method of measuring optical properties may further include measuring the look-down angle and look-up angle for the virtual image plane based on the position within the pattern and the virtual image distance.

Additionally, the optical property measurement method includes measuring a horizontal field of view for a virtual image plane based on a distance to a point to the left of center and a distance to a point to the right of center; and measuring a vertical field of view for the virtual image plane based on the distance to the central top point and the distance to the central bottom point; It may further include.

Additionally, the optical property measurement method includes measuring horizontal distortion for a virtual image plane based on a center and a line between the center top point and the center bottom point; and measuring vertical distortion for the virtual image plane based on a line between center and a point left of center and a point right of center; You can include .

A method of measuring optical properties according to embodiments includes generating a multi-viewpoint image, the multi-viewpoint image including a left image, a center image, and a right image; Displaying positions of virtual images for multi-view images; estimating the depth of locations in the virtual image; Grouping locations based on depth; selecting some locations from the grouped locations; and projecting the corrected virtual object based on some selected positions and depths. may include.

For example, referring to FIG. 25, a device that performs a method for measuring optical properties according to embodiments includes a memory; Processor associated with memory; and an optical measurement unit connected to the processor, wherein the processor generates a multi-view image, the multi-view image includes a left image, a center image, and a right image, and displays the position of the virtual image for the multi-view image. , calculate the depth of the position of the virtual image, estimate the depth of the virtual image, match the depth of the position and the depth of the virtual image, and based on the matched depth, the error of the multi-view image can be calibrated.

Additionally, the optical property measurement device includes memory; Processor associated with memory; and an optical measurement unit connected to the processor, wherein the processor generates a multi-view image, the multi-view image includes a left image, a center image, and a right image, and displays positions of the virtual image for the multi-view image. , it is possible to estimate the depth of the positions of the virtual image, group the positions based on the depth, select some positions from the grouped positions, and project the corrected virtual object based on the selected partial positions and depth.

As a result, 3D HUD vehicle service can be improved from the perspective of driver visibility and convenience. Additionally, by mounting an optical measurement device on a vehicle, measurement parameters for measuring optical characteristics, such as points and depth for a HUD virtual image, can be efficiently obtained. Based on the obtained parameter information, a calibration process that can reduce errors from the driver's perspective can be efficiently performed. 3D HUD can be combined with autonomous driving technology to enable safe and accurate autonomous driving.

The embodiments have been described in terms of a method and/or an apparatus, and the description of the method and the description of the apparatus may be applied to complement each other.

For convenience of explanation, each drawing has been described separately, but it is also possible to design a new embodiment by merging the embodiments described in each drawing. In addition, according to the needs of those skilled in the art, designing a computer-readable recording medium on which programs for executing the previously described embodiments are recorded also falls within the scope of the rights of the embodiments. The apparatus and method according to the embodiments are not limited to the configuration and method of the embodiments described above, but all or part of the embodiments can be selectively combined so that various modifications can be made. It may be composed. Although preferred embodiments of the embodiments have been shown and described, the embodiments are not limited to the specific embodiments described above, and are within the scope of common knowledge in the technical field to which the invention pertains without departing from the gist of the embodiments claimed in the claims. Of course, various modifications are possible by those who have, and these modifications should not be understood individually from the technical ideas or perspectives of the embodiments.

The various components of the devices of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiments may be implemented with one chip, for example, one hardware circuit. Depending on the embodiments, the components according to the embodiments may be implemented with separate chips. Depending on the embodiments, at least one or more of the components of the device according to the embodiments may be composed of one or more processors capable of executing one or more programs, and the one or more programs may be executed. It may perform one or more of the operations/methods according to the examples, or may include instructions for performing them. Executable instructions for performing methods/operations of a device according to embodiments may be stored in a non-transitory CRM or other computer program product configured for execution by one or more processors, or may be stored in one or more processors. It may be stored in temporary CRM or other computer program products configured for execution by processors. Additionally, memory according to embodiments may be used as a concept that includes not only volatile memory (eg, RAM, etc.) but also non-volatile memory, flash memory, and PROM. Additionally, it may also be implemented in the form of a carrier wave, such as transmission over the Internet. Additionally, the processor-readable recording medium is distributed in a computer system connected to a network, so that the processor-readable code can be stored and executed in a distributed manner.

In this document, “/” and “,” are interpreted as “and/or.” For example, “A/B” is interpreted as “A and/or B”, and “A, B” is interpreted as “A and/or B”. Additionally, “A/B/C” means “at least one of A, B and/or C.” Additionally, “A, B, C” also means “at least one of A, B and/or C.” Additionally, in this document, “or” is interpreted as “and/or.” For example, “A or B” may mean 1) only “A”, 2) only “B”, or 3) “A and B”. In other words, “or” in this document may mean “additionally or alternatively.”

Terms such as first, second, etc. may be used to describe various components of the embodiments. However, the interpretation of various components according to the embodiments should not be limited by the above terms. These terms are used to distinguish one component from another. It's just a thing. It's just a thing. For example, a first user input signal may be referred to as a second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. Use of these terms should be interpreted without departing from the scope of the various embodiments. The first user input signal and the second user input signal are both user input signals, but do not mean the same user input signals unless clearly indicated in the context.

The terminology used to describe the embodiments is for the purpose of describing specific embodiments and is not intended to limit the embodiments. As used in the description of the embodiments and the claims, the singular is intended to include the plural unless the context clearly dictates otherwise. The expressions and/or are used in a sense that includes all possible combinations between the terms. The expression includes describes the presence of features, numbers, steps, elements, and/or components and does not imply the absence of additional features, numbers, steps, elements, and/or components. . Conditional expressions such as when, when, etc. used to describe the embodiments are not limited to optional cases. It is intended that when a specific condition is satisfied, the relevant action is performed or the relevant definition is interpreted in response to the specific condition.

Additionally, operations according to embodiments described in this document may be performed by a transmitting and receiving device including a memory and/or a processor depending on the embodiments. The memory may store programs for processing/controlling operations according to embodiments, and the processor may control various operations described in this document. The processor may be referred to as a controller, etc. In embodiments, operations may be performed by firmware, software, and/or a combination thereof, and the firmware, software, and/or combination thereof may be stored in a processor or stored in memory.

Meanwhile, operations according to the above-described embodiments may be performed by a transmitting device and/or a receiving device according to the embodiments. The transmitting and receiving device may include a transmitting and receiving unit that transmits and receives media data, a memory that stores instructions (program code, algorithm, flowchart and/or data) for the process according to embodiments, and a processor that controls the operations of the transmitting and receiving device. You can.

A processor may be referred to as a controller, etc., and may correspond to, for example, hardware, software, and/or a combination thereof. Operations according to the above-described embodiments may be performed by a processor. Additionally, the processor may be implemented as an encoder/decoder, etc. for the operations of the above-described embodiments.

Claims (23)

Step of generating a multi-view image,
The multi-view image includes a left image, a center image, and a right image;
displaying the location of a virtual image for the multi-viewpoint image;
calculating a depth of the location of the virtual image;
estimating the depth of the virtual image;
matching the depth of the location and the depth of the virtual image; and
Calibrating an error of the multi-view image based on the matched depth; Including,
Methods for measuring optical properties.
According to paragraph 1,
The virtual image includes points in each pattern for the virtual image,
The positions of the points are generated based on one or more optical measurement devices and each pattern,
Methods for measuring optical properties.
According to paragraph 2,
The left image containing the points in the pattern of the virtual image is captured based on the left optical measurement device of one or more optical measuring devices, and the central image containing the points in the pattern is captured based on the one or more optical measuring devices. wherein the right image comprising points in a pattern is captured based on the right light measurement device of the one or more light measurement devices.
Methods for measuring optical properties.
According to paragraph 2,
The coordinate values of the position are calculated based on the horizontal pixel index of the left light measurement device, the horizontal pixel index of the right light measurement device, the horizontal pixel index of the central light measurement device, and the field of view of the left light measurement device.
Methods for measuring optical properties.
The method of claim 2, wherein the method
further comprising measuring a virtual image distance for the virtual image based on the position in the pattern of a center and the optical measurement device.
Methods for measuring optical properties.
The method of claim 5, wherein the method
Based on the position within the pattern and the virtual image distance, further comprising measuring a look down angle and a look up angle for the virtual image plane,
Methods for measuring optical properties.
The method of claim 2, wherein the method
measuring a horizontal field of view for the virtual image plane based on the distance to a left-of-center point and the distance to the right-of-center point; and
measuring a vertical field of view for the virtual image plane based on the distance to the central top point and the distance to the central bottom point; Containing more,
Methods for measuring optical properties.
The method of claim 2, wherein
measuring horizontal distortion for the virtual image plane based on a center and a line between the center top point and the center bottom point; and
measuring vertical distortion for the virtual image plane based on a line between the center and a point to the left of the center and a point to the right of the center; Including,
Methods for measuring optical properties.
Step of generating a multi-view image,
The multi-view image includes a left image, a center image, and a right image;
displaying positions of virtual images for the multi-viewpoint image;
estimating depth of locations in the virtual image;
Based on the depth, grouping the locations;
selecting some locations from the grouped locations; and
Projecting a corrected virtual object based on the selected partial location and the depth; containing
Methods for measuring optical properties.
According to clause 9,
The virtual image includes points in each pattern for the virtual image,
The positions of the points are generated based on one or more optical measurement devices and each pattern,
Methods for measuring optical properties.
According to clause 10,
The left image containing the points in the pattern of the virtual image is captured based on the left optical measurement device of one or more optical measuring devices, and the central image containing the points in the pattern is captured based on the one or more optical measuring devices. wherein the right image comprising points in a pattern is captured based on the right light measurement device of the one or more light measurement devices.
Methods for measuring optical properties.
According to clause 10,
The coordinate values of the position are calculated based on the horizontal pixel index of the left light measurement device, the horizontal pixel index of the right light measurement device, the horizontal pixel index of the central light measurement device, and the field of view of the left light measurement device.
Methods for measuring optical properties.
The method of claim 10, wherein the method
further comprising measuring a virtual image distance for the virtual image based on the position in the pattern of a center and the optical measurement device.
Methods for measuring optical properties.
Memory;
a processor connected to the memory; and
It includes a light measurement unit connected to the processor,
The processor,
Create a multi-view image,
The multi-view image includes a left image, a center image, and a right image,
Displaying the location of the virtual image for the multi-view image,
Calculate the depth of the location of the virtual image,
Estimate the depth of the virtual image,
Matching the depth of the location and the depth of the virtual image, and
Based on the matched depth, calibrating the error of the multi-view image,
Optical properties measurement device.
According to clause 14,
The virtual image includes points in each pattern for the virtual image,
The positions of the points are generated based on one or more optical measurement devices and each pattern,
Optical properties measurement device.
According to clause 15,
The left image containing the points in the pattern of the virtual image is captured based on the left optical measurement device of one or more optical measuring devices, and the central image containing the points in the pattern is captured based on the one or more optical measuring devices. wherein the right image comprising points in a pattern is captured based on the right light measurement device of the one or more light measurement devices.
Optical properties measurement device.
According to clause 15,
The coordinate values of the position are calculated based on the horizontal pixel index of the left light measurement device, the horizontal pixel index of the right light measurement device, the horizontal pixel index of the central light measurement device, and the field of view of the left light measurement device.
Optical properties measurement device.
The method of claim 15, wherein the processor
measuring a virtual image distance for the virtual image based on the position in the pattern of a center and the optical measurement device,
Optical properties measurement device.
Memory;
a processor connected to the memory; and
It includes a light measurement unit connected to the processor,
The processor,
Create a multi-view image,
The multi-view image includes a left image, a center image, and a right image,
Displaying the positions of the virtual image for the multi-view image,
Estimate the depth of the locations of the virtual image,
Based on the depth, group the locations,
Select some locations from the grouped locations,
Projecting a corrected virtual object based on the selected location and the depth,
Optical properties measurement device.
According to clause 19,
The virtual image includes points in each pattern for the virtual image,
The positions of the points are generated based on one or more optical measurement devices and each pattern,
Optical properties measurement device.
According to clause 20,
The left image containing the points in the pattern of the virtual image is captured based on the left optical measurement device of one or more optical measuring devices, and the central image containing the points in the pattern is captured based on the one or more optical measuring devices. wherein the right image comprising points in a pattern is captured based on the right light measurement device of the one or more light measurement devices.
Optical properties measurement device.
According to clause 20,
The coordinate values of the position are calculated based on the horizontal pixel index of the left light measurement device, the horizontal pixel index of the right light measurement device, the horizontal pixel index of the central light measurement device, and the field of view of the left light measurement device.
Optical properties measurement device.
21. The method of claim 20, wherein the processor:
measuring a virtual image distance for the virtual image based on the position in the pattern of a center and the optical measurement device,
Optical properties measurement device.