JP2018088595A - Stereographic image measuring device, and stereographic image measuring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simply and conveniently measure the error of a position of stereographic image reproduction in an integral stereographic display method.SOLUTION: A stereographic image measuring device 10 is a device for measurement of a stereographic screen image displayed by an integral stereoscopic display device 20. The stereographic image measuring device comprises: elemental image-producing means 12 for producing an elemental image so that a marker as a stereographic screen image is reproduced at a predetermined position; a detector 11 for detecting the position and brightness of a bright spot formed corresponding to an elemental lens 25 on a display screen of the integral stereoscopic display device 20; and error value measurement means 13 for using the position and brightness of a bright spot detected when white display is performed by the integral stereoscopic display device 20 to perform a computing operation for calibrating the detector 11, using the position and brightness of a bright spot detected when an elemental image is projected by the integral stereoscopic display device 20 to perform a computing operation to determine the position of brightness of a bright spot that the marker includes, and measuring a value of error of the position of reproduction of the marker with respect to the predetermined position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a stereoscopic image measuring apparatus and a stereoscopic image measuring method, and more particularly to a stereoscopic image measuring apparatus and a stereoscopic image measuring method for measuring an error of a stereoscopic image reproduced by an integral stereoscopic display method.

  In general, in the integral stereoscopic display method, a lens array is arranged on the front surface of the display, and a corresponding element image is displayed on the display, so that an autostereoscopic image with parallax in the horizontal and vertical directions can be displayed. In particular, in the method of forming a bright spot at a predetermined position on the display screen, it is possible to improve display characteristics by increasing the number of bright spots by using a plurality of display devices (Non-Patent Document). 1 and 2).

  However, in an actual display device, an error occurs in the reproduction position of the stereoscopic image due to a positional shift between the element image and the lens array and optical distortion. Therefore, as a method for correcting and adjusting the display device in advance so as not to cause an error in the reproduction position of the stereoscopic image, for example, two types of gray code patterns (stripe patterns) in the horizontal direction and the vertical direction are projected and the viewer side Therefore, a method of correcting an elemental image by photographing with a camera (see Non-Patent Document 3) is known.

Hitoshi Watanabe and five others, “Integral 3D display using multiple high-definition projectors”, Proceedings of the Annual Conference of the Institute of Image Information and Television Engineers, August 17, 2016, 34C-5 M. Yamasaki, et al., "High-density Light Field Reproduction Using Overlaid Multiple Projection Images", Proc.SPIE vol.7237, Stereoscopic Displays and Applications XX, 723709 (February 17, 2009) H. Liao, et. Al, "Scalable High-resolution integral videography autostereoscopic display with a seamless multiprojection system", Applied Optics, Vol. 44, No. 3, 2005

  However, in the conventional technique, when the positional deviation between the element image and the lens array is larger than the diameter of the element lens, or when the positional deviation occurs nonlinearly, the element image cannot be corrected with sufficient accuracy. It was difficult to prevent an error from occurring in the image reproduction position. In view of this, it is conceivable to increase the accuracy by using the result of measuring the position error of the reproduced stereoscopic image for correction adjustment of the display device. In this technique, a marker is reproduced by generating an element image so that a position detection marker, which is a stereoscopic image, is reproduced at a desired three-dimensional position, and is photographed (detected) by a detector such as a camera. Thus, the reproduction position error of the marker is measured. However, it is not obvious how to calibrate the detector before the measurement and how to photograph the marker during the measurement.

  As a method for calibrating the position of the detector, the use of a calibration pattern is generally considered. However, it takes time to install a calibration pattern, and adverse effects such as vibration when the calibration pattern is mounted on or removed from the display itself are problematic.

  As a method for photographing the marker, a method in which a diffusing plate is installed at the image forming position of the marker and the camera is focused on the diffusing plate for photographing can be considered. However, it takes time to install the diffuser, and when changing the marker display position, it is necessary to change the focus position of the camera, and the correction accuracy deteriorates due to changes in the camera internal parameters. Is a problem.

  The present invention has been made in view of the above problems, and in an integral stereoscopic display system, a stereoscopic image measuring apparatus and a stereoscopic image capable of easily measuring an error in a position where a stereoscopic image is reproduced. It is an object to provide a measurement method.

  In order to solve the above-described problem, a stereoscopic image measurement apparatus according to the present invention includes an optical element array in which a plurality of optical elements are two-dimensionally arranged, and an element image projection unit that projects an element image onto the optical element array. A stereoscopic image measuring device for measuring a stereoscopic image displayed by an integral stereoscopic display device having an element image generating means, a detector, and an error value measuring means.

According to such a configuration, the stereoscopic image measuring device generates the element image by the element image generation means so that the marker that is a stereoscopic image is reproduced at a predetermined position by the integral stereoscopic display device.
Then, the stereoscopic image measuring device detects the position and brightness of the bright spot formed corresponding to the optical element on the display screen of the integral stereoscopic display device by the detector.
Then, the stereoscopic image measuring device performs an operation of calibrating the detector using the position and brightness of the bright spot detected when white display is performed on the integral stereoscopic display device by the error value measuring means. Performing calculation to obtain the position and brightness of the bright spot constituting the marker using the position and brightness of the bright spot detected when the element image is projected in the integral stereoscopic display device, and the predetermined position The error value of the position where the marker is reproduced is measured.

  In order to solve the above problems, a stereoscopic image measurement method according to the present invention includes an optical element array in which a plurality of optical elements are two-dimensionally arranged, and an element image projection unit that projects an element image onto the optical element array. A stereoscopic image measurement method for measuring a stereoscopic image displayed by an integral stereoscopic display device having an element image generation means, a detector, and an error value measurement means. A pre-calibration step of performing pre-calibration of the detector and a measurement step of measuring an error value at a position where a marker that is a stereoscopic image is reproduced by the stereoscopic image measuring device are included.

According to such a procedure, the stereoscopic image measuring method performs a white display by the integral stereoscopic display device in the pre-calibration step, thereby displaying a bright spot corresponding to the optical element on the display screen of the integral stereoscopic display device. A first display step to be formed; a recording step for recording the display screen by the detector; a first detection step for detecting the position and brightness of a bright spot formed on the display screen; and the error value measurement. And a calculation step of performing a calculation for calibrating the detector by means.
The stereoscopic image measuring method includes a generation step of generating the element image so that the marker is reproduced at a predetermined position by the element image generation means, and the integral stereoscopic display device. A second display step of projecting an image to form a bright spot on the display screen and displaying the marker at a predetermined position; and a position and brightness of the bright spot formed on the display screen by the detector. The second detection step to detect, and the error value measuring means performs a calculation to obtain the position and luminance of the bright spot that constitutes the marker using the detected position and luminance of the bright spot, and the predetermined position And calculating an error value of a position where the marker is reproduced.

  According to the present invention, the stereoscopic image measurement device can directly shoot a display screen without installing a calibration pattern or a diffusion plate. Therefore, the reproduction position error of the stereoscopic image can be easily measured.

1 is a schematic diagram illustrating a configuration of a stereoscopic video measurement device according to a first embodiment of the present invention. It is explanatory drawing of the bright spot formation position in side view. It is explanatory drawing of the luminescent point in front view, (a) is a lens array, (b) has shown the luminescent point formation position corresponding to FIG. It is explanatory drawing of a marker, (a) is the marker reproduced and displayed, (b) is the luminance distribution, (c) is the example of arrangement | positioning in the screen of a marker. It is explanatory drawing of the projection position of a bright spot. It is explanatory drawing of the detected bright spot, (a) is the bright spot which comprises a marker, (b) is the bright spot in a picked-up image, (c) has shown the partially expanded view of (b). It is explanatory drawing of a reproduction position error. It is a flowchart showing the flow of the stereoscopic image measuring method which concerns on 1st Embodiment of this invention. It is a flowchart showing the flow of the prior calibration process shown in FIG. It is a flowchart showing the flow of the reproduction position error calculation process shown in FIG. It is explanatory drawing for demonstrating the closest position of a vector.

  With reference to FIG. 1, a 3D image measurement apparatus and a 3D image measurement method according to an embodiment of the present invention will be described. Note that the size, positional relationship, and the like of the members shown in each drawing may be exaggerated for clarity of explanation.

[Configuration of integral 3D display device]
A system used for the stereoscopic video measurement method includes an integral stereoscopic display device and a stereoscopic video measurement device. First, the integral stereoscopic display device will be described. Here, a case where a lens array is used as the optical element array will be described. The integral stereoscopic display device 20 illustrated in FIG. 1 includes an element image projection unit 21 and a lens array (optical element array) 22.

The element image projection unit 21 projects an element image onto the lens array 22, and includes a projector 23 and a collimator lens 24 here.
The projector 23 projects an element image group corresponding to each element lens (optical element) 25 on a one-to-one basis.
The collimator lens 24 is disposed between the projector 23 and the lens array 22, and the light from the projector 23 enters, and the incident light is emitted as parallel light.

  In this stereoscopic video measurement method, an element image group for reproducing a marker for detecting a position is displayed. The element image projection unit 21 displays a plurality of markers on the screen in one display. The element image group is generated by the element image generation means 12.

  The element image projected from the projector 23 via the collimator lens 24 enters the lens array 22. Thereby, the integral stereoscopic video can be reproduced. In the lens array 22, a plurality of element lenses 25 are two-dimensionally arranged. In the lens array 22, minute element lenses 25 are densely arranged. In FIG. 1, the seven element lenses 25 are illustrated in a one-dimensional manner, but the number of element lenses is arbitrary and is typically tens of thousands or more.

  In the present embodiment, as an example, the element lens 25 is a biconvex lens. The shape of the element lens 25 is not limited to a circle but may be a square. The arrangement of the element lenses 25 is a square lattice (grid structure), but may be a so-called line offset.

  Here, a case where the integral stereoscopic display device 20 forms a bright spot at a predetermined position in the display screen will be described. As an example, FIGS. 2 and 3 show positions where bright spots are formed when the incident light to the lens array 22 is parallel light. FIG. 3A shows the lens array 22 in the front view, and FIG. 3B shows the bright spot formation position 101 in the front view.

In FIG. 2, it is assumed that the focal length of the element lenses constituting the lens array 22 is f, and the lens array 22 is installed at a position of Z = −f. At this time, the bright spot formation position 101 is located on the display screen, that is, Z = 0 (XY plane). Strictly speaking, the display screen of the integral stereoscopic display device 20 is separated from the lens array 22 by the same distance (for example, about several mm) as the focal length of the element lens 25.
Each bright spot is formed when the parallel light 102 is incident on the lens array 22 at a predetermined projection angle θ. In FIG. 2, the code | symbol 103 has shown the projection light ray of the element image (the image for one element lens 25).

  When parallel light 102 is incident on the lens array 22 at a predetermined angle θ as shown in FIG. 2, for example, the shift amount g of the bright spot formation position with respect to the optical axis 104 of the element lens 25a is projected with the focal length f of the element lens 25a. It can be calculated by the following equation (1) using the angle θ.

  g = f × tan θ (1)

[Configuration of stereoscopic image measuring device]
Next, the stereoscopic image measuring device will be described. A stereoscopic video measurement device 10 shown in FIG. 1 is a device that measures a stereoscopic video displayed by an integral stereoscopic display device 20, and includes a detector 11, element image generation means 12, and error value measurement means 13. I have.

The detector 11 detects the position and brightness of the bright spot formed corresponding to the element lens 25 on the display screen of the integral stereoscopic display device 20.
As will be described later, the detector 11 detects the position and brightness of all the bright spots formed on the display screen from a single viewpoint in the pre-calibration step (when white display is performed).
Further, as will be described later, the detector 11 detects the position and brightness of the bright spot formed on the display screen in the measurement process (when the element image is projected) from the same viewpoint as when white display is performed.

  The detector 11 is, for example, a digital camera or a photo detector. In this embodiment, the detector 11 is assumed to be a single digital camera as an example. When the camera is used for the detector 11 as described above, the focus position is controlled to be the display screen of the integral stereoscopic display device 20, that is, the position of the bright spot formed at Z = 0 in FIG. Further, as an example, it is assumed that internal parameters are known, and camera optical system distortion correction is performed on a captured image.

The element image generation means 12 generates an element image so that a marker that is a stereoscopic video is reproduced at a predetermined position by the integral stereoscopic display device 20. The element image generation means 12 is constituted by an image processing means, and the element image generation method is not particularly limited.
Here, the predetermined position is a position in a three-dimensional space separated from the display screen of the integral stereoscopic display device 20 by a predetermined depth distance. As for the predetermined position, the user of the stereoscopic image measuring apparatus 10 can determine a desired position where the marker is to be displayed. When a single detector 11 detects a marker from a single viewpoint, a two-dimensional position (position in the screen) excluding depth is detected as an approximate position of the marker.

Here, the marker to be reproduced will be described with reference to FIGS. 4 (a) to 4 (c).
FIG. 4A shows an example of the shape of a marker that is reproduced and displayed. In the present embodiment, it is assumed that the marker M is, for example, circular when viewed from the front, and has a higher luminance distribution toward the center. The black portion outside the circle represents the background of the display screen.

  For example, when −π / 2 ≦ x ≦ π / 2 and −π / 2 ≦ y ≦ π / 2 on the xy plane, the luminance of the marker can be expressed as, for example, cosx × cosy. FIG. 4B shows the luminance distribution of the marker M in the x-axis direction.

  FIG. 4C shows an arrangement example of the markers on the screen. Here, a state in which a total of 77 markers of 11 horizontal x 7 vertical are reproduced and displayed as an example through the lens array 22 is schematically shown. The number of markers and the three-dimensional position (number and arrangement) are not limited to these. By arranging the markers over the entire screen, the distribution of the entire screen can be calculated for the reproduction position error of the markers.

  Note that the reproducible depth range of the marker is determined by the adjustment state of the integral stereoscopic display device 20 and the installation position of the detector 11. If a marker is displayed at a deep position with a low adjustment accuracy, the reproduced image is distorted or missing. Therefore, in the pre-calibration step, first, the marker is displayed at a position where the depth distance is short to roughly adjust the integral stereoscopic display device 20, and then the integral stereoscopic display device is gradually changed while changing the position of the marker. It is preferable to appropriately determine the depth position according to the use application, such as fine adjustment of 20.

The error value measuring means 13 performs calculation processing other than the generation of the element image in the stereoscopic image measuring apparatus 10, and mainly measures a two-dimensional error value excluding the depth with respect to the reproduction position of the stereoscopic image.
The error value measuring means 13 performs an operation for calibrating the detector 11 using the position and brightness of the bright spot detected when white display is performed by the integral stereoscopic display device 20 in the pre-calibration step described later. In order to perform this, a correction coefficient calculation means 14, a position calibration means 15, a distortion coefficient calculation means 16, and an internal parameter calibration means 17 are provided.
Further, the error value measuring means 13 uses the position and brightness of the bright spot detected when the element image is projected on the integral stereoscopic display device 20 in the measurement process described later. An error calculating means 18 is provided for performing an operation for obtaining the position and the luminance and measuring the error value at the position where the marker for the predetermined position is reproduced.

  The correction coefficient calculation means 14 calculates a correction coefficient for correcting the luminance value variation for each bright spot based on the brightness of all bright spots detected when white display is performed in the pre-calibration step. is there. In the present embodiment, in the pre-calibration step, the correction coefficient calculation unit 14 analyzes the data captured by the detector 11 and detects the position and luminance of the bright spot. In the detector 11, since the bright spot is captured as a small point group having a luminance distribution, the correction coefficient calculating unit 14 binarizes the captured image and performs labeling processing or calculates the peak of the luminance gradient. The center position of each bright spot is detected as the bright spot forming position 101 (FIG. 2). As for the distribution of the bright spot forming positions 101 in the entire display screen of the integral stereoscopic display device 20, as in the example shown in the above-described equation (1), the arrangement of the lens array 22, the position coordinates of each element lens 25, and the like. It can be calculated from the focal length and the configuration (projection angle) of the element image projection unit 21. The correction coefficient calculation means 14 regards a detection point group having the same distribution as the distribution of bright spot formation positions as a bright spot, and removes other detection points as noise. Note that the luminance value of the bright spot can be obtained by averaging the pixel values of the bright spots in the captured image.

  The correction coefficient calculated by the correction coefficient calculation means 14 is a coefficient for correcting (normalizing) a variation in luminance value caused by a molding error of each element lens 25 constituting the lens array 22 in a measurement process described later. The correction coefficient calculation means 14 calculates the average value of all the luminance values based on the luminance value of each bright spot detected by analyzing the data captured by the detector 11 from a single viewpoint when white display is performed. Find the median and use it as the reference value. The correction coefficient calculation means 14 obtains the ratio between the brightness value and the reference value for each bright spot, and sets it as the correction coefficient for the brightness value of each bright spot. Note that a bright spot with an extremely low luminance value may be regarded as a defective element lens 25 and removed from the calculation target.

The position calibrating means 15 calibrates the position and orientation of the detector 11 based on the positions of all the bright spots detected when white display is performed in the preliminary calibration process.
The position calibration unit 15 assigns real coordinates to the position of the detected bright spot, and calculates the position and orientation of the detector 11 by using a general calibration method described in Reference Document 1 below. . Here, for the origin of the real coordinates, an arbitrary position is designated, or only one bright spot at the center of the screen is displayed in a different color, and this is set as the origin.

(Reference 1) Zhang, Zhengyou, "A flexible new technique for camera calibration" IEEE Transactions on pattern analysis and machine intelligence 22.11 (2000), 1330-1334

The distortion coefficient calculation means 16 corrects the positional distortion of the image based on the positions of all the bright spots detected from multiple viewpoints by changing the position of the detector 11 when white display is performed in the pre-calibration step. The distortion coefficient is calculated. This is a process for correcting the distortion of the bright spot forming position.
If the design of the integral stereoscopic display device 20 is ideal, distortion does not occur. However, in an actual display device, a bright spot is caused by a molding error of the element lens 25 or lens distortion of the collimator lens 24 in the front stage of the lens array 22. Distortion may occur at the formation position. The distortion of the bright spot forming position causes a decrease in measurement accuracy of the reproduction position error of the stereoscopic image.

  The distortion coefficient calculation means 16 creates a perspective projection conversion model considering the distortion model when calculating the distortion coefficient. Assuming that distortion occurs in the radial direction, an example in which a distortion model is added to a general perspective projection transformation model is shown in the following equations (2) to (7).

However, in Formula (2)-Formula (7), small x, y, z is a camera coordinate system. R is a matrix indicating rotation and t is a translation indicating the attitude of the detector 11. (U, v) is a bright spot formation position in the captured image expressed in image coordinates (projection plane coordinates). (X ′, Y ′, Z) are ideal (undistorted) bright spot formation positions expressed in world coordinates. (X, Y, Z) is a bright spot forming position including distortion expressed in world coordinates. k ₁ , k ₂ and k ₃ are radial distortion coefficients. The internal parameters of the camera is assumed to be known (already calibrated), the focal length f _x, f _y and the image center c _x, c _y is assumed to be known.

The distortion coefficient calculation means 16 applies an optimization algorithm such as the Levenberg-Markert method to the evaluation function of the reprojection error created based on the above-described equation (2), whereby the distortion coefficients k ₁ , k to calculate the unknowns, including _2, k _3. The detection information of the bright spot formation position needs to be detected from multiple viewpoints.

  The internal parameter calibration means 17 is for calibrating the internal parameters when the detector 11 is a digital camera with no internal parameters calibrated. This internal parameter calibration means 17 calibrates the internal parameters of the digital camera based on the positions of all bright spots detected from multiple viewpoints by changing the position of the digital camera when white display was performed in the pre-calibration process. To do. The internal parameter calibration means 17 calculates the internal parameter by applying a general calibration method described in the above-mentioned reference 1 or the like. Note that when the position of the camera is changed, the postures R and t in Equation (2) described above change.

  The error calculation means 18 calculates the error value of the position where the marker is reproduced by calculating the position and luminance of the bright spot constituting the reproduced marker and approximating it to the theoretical luminance distribution of the marker. is there. In the present embodiment, in the measurement process, the error calculation means 18 analyzes the data captured by the detector 11 and detects the position and brightness of the bright spot.

  The error calculation means 18 detects the position of the bright spot constituting the reproduced marker by projecting the bright spot detected when the element image is projected in the measurement step to a predetermined position of the marker. Here, the projection position of the bright spot will be described with reference to FIG. FIG. 5 is an explanatory diagram of projection positions of bright spots using the same three-dimensional space as FIG. In FIG. 5, the same components as those in FIG.

A depth position of a desired three-dimensional reproduction position of the marker M is set as a position of Z = d. That is, let d be the depth distance of the marker. In FIG. 5, for the sake of simplicity, the bright spots constituting the marker are assumed to be formed from four element lenses arranged in a one-dimensional array parallel to the Y axis. However, as shown in FIG. A bright spot of a two-dimensional array is assumed.
As shown in FIG. 5, the pixels 121, 122, 123, and 124 of the element image constituting the marker M pass through different bright spot formation positions 101, and at the bright spot projection position 111 set to Z = d, A three-dimensional image of the marker is formed. The three-dimensional image of the marker is sampled at the bright spot interval (= element lens interval), and the element image becomes luminance information corresponding to the viewpoint with respect to the bright spot. At this time, the detector 11 adjusts the focus position to the bright spot formation position 101, and the error calculation means 18 analyzes the data captured by the detector 11, and performs the following process. Detect the position and brightness of the points.

First, regarding the position of the bright spot, the bright spot formation position in the captured image expressed by image coordinates (projection plane coordinates) is (u, v), the marker depth distance is d, and the bright spot projection position is (X _d , Y _d , d), X is replaced with X _d in the above equation (2), and unknowns X _d , Y _d are obtained from the following equation (8) in which Y is replaced with Y _d. The projection position (X _d , Y _d , d) of the bright spot is calculated.

  Next, a method for detecting the brightness of the bright spot will be described with reference to FIGS. 6A to 6C (see FIG. 5 as appropriate). Here, it is assumed that the marker is composed of 4 × 4 16 bright spots. In an actual display device, when a human observes at a distance, the bright spot interval is denser than the eye resolution, and a stereoscopic image is perceived without being identified as a bright spot. .

  FIG. 6A shows 4 × 4 bright spots constituting the marker in the marker M having an outer periphery indicated by a two-dot chain line. The bright spots 131, 132, 133, and 134 are formed by the pixels 121, 122, 123, and 124 of the element image that constitutes the marker M in FIG.

FIG. 6B shows a captured image 140 captured by the detector 11. The bright spots 141, 142, 143, and 144 recorded in the photographed image 140 are bright spots that are photographed while focusing on the bright spot formation position 101 instead of the projection position 111 in FIG. 5.
FIG. 6C is an enlarged view of a region 140a including the bright spot 141 in the captured image 140 of FIG. Although different depending on the resolution of the detector 11, the bright spot 141 in the captured image 140 is formed of, for example, several tens of pixels. Therefore, the error calculating unit 18 can detect the luminance value of the bright spot as an average of the pixel values of several tens of pixels constituting the bright spot in the captured image 140.

The error calculating means 18 calculates the luminance value of the bright spot constituting the reproduced marker by multiplying the luminance value detected when the element image is projected in the measurement step by the correction coefficient for each bright spot.
In the present embodiment, the error calculation means 18 multiplies the detected luminance value by the correction coefficient for each bright spot calculated by the correction coefficient calculation means 14, thereby obtaining the brightness value of the bright spot that constitutes the marker M. Is calculated.

The error calculation means 18 calculates an error value at the position where the marker is reproduced by an operation for approximating the luminance distribution of the bright spots constituting the reproduced marker to the theoretical luminance distribution of the marker.
The error value (reproduction position error) at the position where the marker is reproduced is a three-dimensional error value, but in the present embodiment, it is handled as a two-dimensional error value excluding the depth. In this case, the reproduction position errors in the x direction and the y direction on the projection plane are denoted by e _x and e _y .

In the example shown in FIG. 7, the error calculation means 18 uses the luminance value calculated from the bright spot that constitutes the detected marker at the reproduction position 150 as the theoretical luminance value of the marker at the reproduction scheduled position 160 that is the correct position. To approximate the reproduction position errors e _x and e _y of the marker.
Specifically, the error calculation means 18 calculates the reproduction position errors e _x and e _y and the adjustment coefficient a by the least square method so that the evaluation function E expressed by the following equation (9) is minimized. To do.

_{However, (X d, Y d,} d) is the projected position, L _I of the bright points constituting the marker luminance of the bright spot constituting the marker, the L _M is the theoretical value of the brightness of the marker. As the theoretical value of the marker brightness, for example, the brightness distribution cosx × cosy exemplified in FIG. 4B can be used.

[Flow of 3D image measurement method]
Next, the flow of the stereoscopic image measurement method will be described with reference to FIG. 8 (refer to FIG. 1 as appropriate). The three-dimensional image measurement method includes a pre-calibration step (step S11) in which the three-dimensional image measurement device 10 performs pre-calibration of the detector 11, and an error in a position where a marker that is a three-dimensional image is reproduced by the three-dimensional image measurement device 10. And a measuring step (step S13 to step S16) for measuring the value. In the present embodiment, a single detector 11 is used.

  In the pre-calibration (step S11), the position calibration of the detector 11 and the calculation of the correction coefficient for the brightness value of the bright spot are mainly performed. Details of this pre-calibration process will be described later. Then, for example, the stereoscopic video measurement apparatus 10 stands by until a start instruction is input to the element image generation unit 12 (step S12: No), and when it is determined that the start instruction is input (step S12: Yes), the measurement is performed. Start the process.

  Then, the stereoscopic image measuring device 10 generates an element image group by the element image generation unit 12 so that the marker is reproduced at a predetermined position where the marker is to be displayed (step S13: generation process). ). Further, the integral stereoscopic display device 20 projects the element image group generated in step S13 to form a bright spot on the display screen and reproduce and display the marker at a predetermined position (step S14: second display step). .

  Then, the stereoscopic image measuring device 10 detects the reproduced marker by the detector 11. Then, a person visually checks the marker to confirm whether or not a reproduced image of the marker can be detected. When detection is impossible, for example, when the reproduced image of the marker is missing or the distortion of the reproduced image is large (step S15: No), the process returns to step S13, and the depth distance of the marker is reduced and then again. Generate element images.

  On the other hand, if there is no problem such as missing in the reproduced image, a reproduction position error calculation instruction is input to the apparatus. For example, when the stereoscopic image measurement device 10 determines that the reproduction position error calculation instruction has been input to the error value measurement unit 13 (step S15: Yes), the error value measurement unit 13 determines that the bright spot constituting the marker The reproduction position error is calculated from the detection information relating to the position and the luminance (step S16). Although details of the reproduction position error calculation process will be described later, the calculated reproduction position error may be applied to an existing correction adjustment method for the display device.

  When the reproduction position error is further measured (step S17: No), the process returns to step S13. If no further measurement is required, for example, if it is determined that an end instruction has been input to the element image generation unit 12 (step S17: Yes), the stereoscopic video measurement device 10 ends the process.

  Next, the pre-calibration process will be described with reference to FIG. 9 (see FIGS. 2 and 3 as appropriate). In the pre-calibration step (step S11), first, as shown in FIG. 2 and FIG. 3B, the integral stereoscopic display device 20 displays the element lens on the display screen of the integral stereoscopic display device 20 by performing white display. A bright spot corresponding to 25 is formed. At this time, the integral stereoscopic display device 20 displays white, emits all pixels, and emits all bright spots with the maximum luminance (step S21: first display step). It is assumed that the distribution of all bright spot formation positions is known. Further, for example, as shown in FIG. 2, the incident light to the lens array 22 is assumed to be parallel light 102.

Step S22 in FIG. 9 is a branching process related to distortion correction of the bright spot formation position, and step S23 is a branching process related to calibration of the internal parameters of the camera. Here, only one of the branch processes of step S22 and step S23 is performed as necessary in the series of processes.
For example, as in this embodiment, assuming that the internal parameters of the camera have already been calibrated and further correction of distortion of the bright spot formation position is necessary (step S22: No), the process proceeds to step S28, and the bright brightness is set as described later. A distortion coefficient at the point formation position is calculated. In order to calculate the distortion coefficient at the bright spot forming position, it is necessary to know the internal parameters of the camera.
On the other hand, if the internal parameters of the camera have not been calibrated (step S23: No), the process proceeds to step S29, and the internal parameters are estimated as described later. In order to calibrate the internal parameters of the camera, it is necessary to correct the distortion of the bright spot formation position.

  And when distortion correction of a bright spot formation position is unnecessary (Step S22: Yes), and when an internal parameter of a camera has been calibrated (Step S23: Yes), it progresses to Step S24, and with detector 11, The display screen of the integral stereoscopic display device 20 is photographed.

  In step S24 (recording step), the detector 11 is installed in the viewing zone of the integral stereoscopic display device 20, and the display screen is captured. When the detector 11 is a digital camera as in this embodiment, the focus position is aligned with the bright spot formation position, and the display screen is recorded by the detector 11.

  Then, the error value measuring means 13 of the stereoscopic image measuring apparatus 10 analyzes the data captured by the detector 11 when white display is performed by the correction coefficient calculating means 14, and detects the positions and luminances of all the bright spots. (Step S25: first detection step). In order to improve the detection accuracy, the background screen may be used by shooting in a state where the display screen is displayed in black.

  Then, the error value measuring means 13 calculates the correction coefficient of the luminance value by the correction coefficient calculating means 14 (step S26). The luminance value correction coefficient is a coefficient for correcting variations in luminance values in the measurement process, and is calculated for each bright spot.

  Then, the error value measuring means 13 performs an operation for calibrating the position of the detector 11 based on the positions of all the bright spots detected by the position calibrating means 15. That is, the position calibration means 15 assigns real coordinates to the detected bright spot position, and calculates the position and orientation of the detector 11 (step S27: calculation step).

In the case of No in step S22 described above, the error value measuring unit 13 calculates the distortion coefficient at the bright spot forming position based on the positions of all the bright spots detected from multiple viewpoints by the distortion coefficient calculating unit 16 (step). S28). Specifically, the distortion coefficient calculating means 16 applies an optimization algorithm such as the Levenberg-Marggart method to the evaluation function of the reprojection error created based on the above-described equation (2). Calculation of unknowns including coefficients k ₁ , k ₂ , and k ₃ is performed. At that time, the display screen photographing process (a process corresponding to step S24) and the bright spot detection process (a process corresponding to step S25) are performed a plurality of times while changing the position of the detector 11. The position of all bright spots formed on the display screen is detected from multiple viewpoints. If there is no distortion at the bright spot formation position or if the distortion coefficient has been calculated before, step S28 need not be performed.

  In the case of No in step S23 described above, the error value measurement unit 13 estimates the internal parameter based on the positions of all bright spots detected from multiple viewpoints by the internal parameter calibration unit 17 (step S29). Specifically, the internal parameter calibration means 17 is a general calibration method described in Reference 1 or the like with respect to the detected detection information of the bright spot formation position (X, Y, Z) from multiple viewpoints. To calculate the internal parameters. In addition, since the detection information of the bright spot formation position is necessary for multiple viewpoints, the display screen photographing process (process corresponding to step S24) and the bright spot detection process (process corresponding to step S25) are detected. This is performed a plurality of times while changing the position of the vessel 11.

  Next, the reproduction position error calculation process will be described with reference to FIG. 10 (refer to FIG. 1, FIG. 5 to FIG. 7 and FIG. 9 as appropriate). First, the detector 11 is installed in the viewing zone of the integral stereoscopic display device 20, and the display screen is captured. Then, the integral stereoscopic display device 20 projects the element image group to form a bright spot on the display screen and reproduce and display the marker M at a predetermined position (step S14 in FIG. 8: second display step). At this time, the installation position of the detector 11 is set to the same position so that it can be detected from the same viewpoint as in the process of step S24 of the pre-calibration step. Then, when the detector 11 is a digital camera as in the present embodiment, the focus position is not changed from the pre-calibration step and is kept at the bright spot forming position, and the detector 11 captures the display screen ( Step S31). Thereby, the position of the bright spot formed on the display screen is detected.

  Then, the error value measuring means 13 of the stereoscopic image measuring apparatus 10 analyzes the data captured by the detector 11 by the error calculating means 18 and detects the position and brightness of the bright spot constituting the marker M (step S32: Second detection step). Here, since the bright spot formation positions 101 for all the bright spots formed on the display screen are calculated in the process of step S25 in the pre-calibration step, the bright spot positions constituting the marker M are Use location information. Then, the error calculation means 18 calculates the luminance value of each bright spot (131 to 134: see FIG. 6A) constituting the marker M. Here, the luminance value of the bright spot can be detected as an average of the pixel values constituting the bright spot (see FIG. 6C). Note that bright spot groups adjacent to each other with a luminance value equal to or higher than a certain value are regarded as bright spot groups constituting the marker M.

  Then, the error value measuring unit 13 normalizes the luminance value by the error calculating unit 18 (step S33: calculation step). That is, the error calculation means 18 multiplies the luminance value detected in step S32 by the correction coefficient for each bright spot calculated in step S26 of the pre-calibration step, thereby forming the bright spot constituting the marker M. Is calculated.

  Then, the error value measuring means 13 projects the bright spot on the desired reproduction position of the marker by the error calculating means 18 (step S34 in FIG. 10: calculation step, see FIG. 5). Thereby, the error value measuring means 13 calculates the position of the bright spot constituting the marker M.

  Then, the error value measuring unit 13 calculates the reproduction position error by approximating the luminance distribution of the bright spots constituting the marker M to the theoretical value of the marker luminance distribution by the error calculating unit 18 (step S35 in FIG. 10). : Calculation process, see FIG.

According to the stereoscopic image measuring apparatus 10 according to the present embodiment, the position of the bright spot captured by the detector 11 and the luminance information are used to calibrate the position of the detector 11 and measure the reproduction position error of the stereoscopic image. be able to. Accordingly, since the display screen can be directly photographed without installing a calibration pattern or a diffusion plate, the reproduction position error of the stereoscopic image can be easily measured.
Further, since it is not necessary to install the diffuser plate at the marker reproduction position when taking a three-dimensional image, it is possible to save the trouble of moving the diffuser plate each time the marker reproduction position is changed.

(Second Embodiment)
Hereinafter, an embodiment using a plurality of detectors 11 will be described as a second embodiment. In addition, the same code | symbol is attached | subjected to the same structure as 1st Embodiment, and description is abbreviate | omitted suitably.

For example, when the desired reproduction position (theoretical value) of the marker is (C _x , C _y , d) and the reproduction position errors e _x and e _y calculated in step S35 are used together, the marker (reproduced image) is used. ) Center position P can be expressed as (C _x + e _x , C _y + e _y , d). Under this assumption, first, the error calculation means 18 calculates a vector formed by the center position of the camera and the center position of the reproduced image in each detector 11. For example, the center position A _i of the i-th (i = 1, 2,..., N) camera among the n detectors 11 is set as the starting point, and the center position P _i (C _{x of} the reproduced image detected by each camera. A vector whose end point is + e _xi , C _y + e _yi , d) is obtained for each camera.

Then, the error calculation means 18 can measure a three-dimensional reproduction position error by calculating the closest position of all vectors. Here, the reason why the closest position of the vector is calculated instead of the intersection of the vectors is that, in actual measurement, the vector does not have an intersection in the three-dimensional space due to an error in the measurement value, and is in a twisted relationship. . For example, when calculating from _two vectors (vector v ₁ , vector v ₂ ), as shown in FIG. 11, the nearest neighbor point on vector v ₁ , which is an intersection when one straight line is orthogonal to vectors v ₁ and v ₂ The midpoint m between a and the nearest point b on the vector v ₂ is calculated using the least square method.

When generalized, the error calculation means 18 obtains a point at which the total sum of the distances from the n vectors is minimum as the closest position. Specifically, the playback position is P = (C _x + e _x , C _y + e _y , d), each vector obtained for each camera is v _i , the unit vector of each vector is v _i ′, and the center of each camera When the position is A _i and the distance between P and each vector v _i is obtained by the three-square theorem, the square sum E can be expressed by the following equation (10).

In the right side of Expression (10), P and A _i are position vectors, and the symbol “·” is an inner product of vectors. By calculating P that minimizes the sum of squares E in this equation (10), it is possible to calculate reproduction position errors e _x , e _y, and d.

  According to this embodiment, a three-dimensional reproduction position error of a stereoscopic image can be calculated. In addition, although the case where a three-dimensional image was image | photographed from several viewpoints using the some detector 11 was demonstrated, not only this but a three-dimensional image is image | photographed from a several viewpoint, moving the single detector 11. FIG. Even in such a case, the three-dimensional reproduction position error of the stereoscopic image can be calculated.

  As mentioned above, although each embodiment of this invention was described, this invention is not limited to these, It can implement in the range which does not change the meaning. For example, in the integral stereoscopic display device 20, the element image projection unit 21 may be configured using a backlight and a liquid crystal panel instead of the projector 23 and the collimator lens 24. Alternatively, direct light from the projector 23 may be incident on the lens array 22 without using a collimator lens.

Further, the correction coefficient calculation means 14 and the error calculation means 18 of the error value measurement means 13 analyze the data captured by the detector 11 and detect the position of the bright spot. You may make it detect by measuring the position of a point.
Although the detector 11 is assumed to be a camera, for example, a photodetector may be used for the detector 11.

  In the stereoscopic image measuring apparatus 10, the overall processing can be integrated by the error value measuring unit 13 or the element image generating unit 12, or a dedicated integrated processing unit may be provided separately from them. In the stereoscopic image measuring apparatus 10, the error value measuring means 13 and the element image generating means 12 have been described separately. However, these two functions can be realized by a single computer.

  The present invention can also be applied to a system using a plurality of display devices. For example, in the case of a display device using a plurality of projectors, the marker is reproduced by one projector and measurement is performed while the other projectors are not displayed. As long as the viewing zones of the projectors overlap, the detector may remain installed at the same position. If the viewing zones do not overlap, it is necessary to move the detector for each projector. However, by correcting the position of the detector by sharing the origin of the world coordinates at each viewpoint, it is possible to adjust and correct the entire system. It becomes.

  Although the target is the integral three-dimensional display device 20 that uses the lens array 22 to form a bright spot at a predetermined position in the display screen, the present invention is also applicable to other integral methods that use pinholes or point light sources. it can. In the integral three-dimensional display system in which diffused light is incident on the lens array, such as a system using a direct view type panel or a system using a projector and a diffusion plate, the position of the element lens 25 in FIG. The present invention can be applied by regarding the brightness of the element lens 25 as the brightness of the bright spot, assuming the bright spot formation position 101 of 3 (b).

DESCRIPTION OF SYMBOLS 10 Stereoscopic image measuring apparatus 11 Detector 12 Element image generation means 13 Error value measurement means 14 Correction coefficient calculation means 15 Position calibration means 16 Error calculation means 17 Distortion coefficient calculation means 18 Internal parameter calibration means 20 Integral stereoscopic display device 21 Element image Projection unit 22 Lens array (optical element array)
25 Lens (optical element)
101 Bright spot formation position M Marker

Claims (11)

3D image measurement for measuring a 3D image displayed by an integral 3D display device having an optical element array in which a plurality of optical elements are two-dimensionally arranged and an element image projection unit that projects an element image onto the optical element array A device,
Element image generation means for generating the element image so that a marker that is a stereoscopic image is reproduced at a predetermined position by the integral stereoscopic display device;
A detector for detecting the position and brightness of a bright spot formed corresponding to the optical element on the display screen of the integral stereoscopic display device;
The calculation is performed to calibrate the detector using the position and brightness of the bright spot detected when white display is performed on the integral stereoscopic display device, and the element image is projected on the integral stereoscopic display device. The calculation of the position and brightness of the bright spot constituting the marker is performed using the detected bright spot position and brightness, and the error value of the position where the marker is reproduced with respect to the predetermined position is measured. Error value measuring means;
A stereoscopic image measuring apparatus comprising: The detector detects the position and brightness of all bright spots formed on the display screen when performing white display on the integral stereoscopic display device, from a single viewpoint,
The error value measuring means includes
Correction coefficient calculating means for calculating a correction coefficient for each bright spot for correcting variations in brightness values based on the brightness of all bright spots detected when performing the white display;
The stereoscopic image measurement apparatus according to claim 1, further comprising: a position calibration unit configured to calibrate the position and orientation of the detector based on the positions of all bright spots detected when the white display is performed. The detector detects the position and brightness of a bright spot formed on the display screen when the element image is projected from the single viewpoint,
The error value measuring means includes
By multiplying the brightness value detected when the element image is projected by the correction coefficient for each bright spot, the brightness value of the bright spot constituting the reproduced marker is calculated, and when the element image is projected By projecting the detected bright spot to the predetermined position of the marker, the position of the bright spot constituting the reproduced marker is calculated, and the luminance distribution of the bright spot constituting the reproduced marker is calculated as the marker. The stereoscopic image measurement device according to claim 2, further comprising error calculation means for calculating an error value at a position where the marker is reproduced by an operation that approximates the theoretical luminance distribution. The error value measuring means includes
Distortion coefficient for correcting the positional distortion of the image based on the positions of all bright spots detected from multiple viewpoints by changing the position of the detector when white display is performed on the integral stereoscopic display device The stereoscopic image measuring device according to any one of claims 1 to 3, further comprising: a distortion coefficient calculating unit that performs the calculation. The detector is a digital camera;
The error value measuring means is based on the positions of all bright spots detected from multiple viewpoints by changing the position of the digital camera when white display is performed on the integral stereoscopic display device. The stereoscopic image measuring device according to any one of claims 1 to 4, further comprising an internal parameter calibration unit that calibrates a parameter. The detector is a digital camera or a photo detector,
The stereoscopic image measurement device according to any one of claims 1 to 4, wherein the error value measurement unit measures a two-dimensional error value or a three-dimensional error value excluding depth. 3D image measurement for measuring a 3D image displayed by an integral 3D display device having an optical element array in which a plurality of optical elements are two-dimensionally arranged and an element image projection unit that projects an element image onto the optical element array A method,
A pre-calibration step of performing pre-calibration of the detector by means of a stereoscopic image measurement device comprising element image generation means, detector, and error value measurement means;
A measurement step of measuring an error value at a position where a marker that is a stereoscopic video is reproduced by the stereoscopic video measurement device,
The pre-calibration step includes
A first display step of forming a bright spot corresponding to the optical element on the display screen of the integral stereoscopic display device by performing white display by the integral stereoscopic display device;
A recording step of recording the display screen by the detector;
A first detection step of detecting a position and brightness of a bright spot formed on the display screen;
A calculation step of performing a calculation to calibrate the detector by the error value measuring means,
The measurement step includes
A generation step of generating the element image so that the marker is reproduced at a predetermined position by the element image generation means;
A second display step of forming a bright spot on the display screen by projecting the element image and displaying the marker at a predetermined position by the integral stereoscopic display device;
A second detection step of detecting a position and luminance of a bright spot formed on the display screen by the detector;
A position where the error value measuring unit calculates the position and luminance of the bright spot constituting the marker using the detected position and luminance of the bright spot, and the position where the marker is reproduced with respect to the predetermined position A calculation process for measuring the error value of
A stereoscopic image measuring method comprising: The pre-calibration step includes
The detector detects the position and brightness of all bright spots formed on the display screen by performing the white display, from a single viewpoint,
A step of calculating, for each luminescent spot, a correction coefficient for correcting variation in luminance value based on the luminance of all luminescent spots detected at the single viewpoint by the error value measuring means;
The stereoscopic image measuring method according to claim 7, further comprising a step of calibrating the position and orientation of the detector based on the positions of all bright spots detected when the white display is performed. The measurement step includes
The detector detects the position and brightness of a bright spot formed on the display screen when the element image is projected, from the single viewpoint,
By the error value measuring means,
Multiplying the detected brightness value by a correction coefficient for each bright spot to calculate the brightness value of the bright spot constituting the reproduced marker;
Projecting the detected bright spot to the predetermined position of the marker to calculate the position of the bright spot constituting the reproduced marker;
The method of calculating the error value of the position where the marker was reproduced | regenerated by the calculation which approximates the luminance distribution of the luminescent spot which comprises the said reproduced | regenerated marker to the theoretical luminance distribution of the said marker. 3D image measurement method. The pre-calibration step includes
By changing the position of the detector and performing the recording step and the detection step a plurality of times, the positions of all bright spots formed on the display screen by performing the white display are detected from multiple viewpoints,
The stereoscopic image measuring method according to claim 7, further comprising: calculating a distortion coefficient for correcting positional distortion of the image based on the positions of all bright spots detected from the multiple viewpoints by the error value measuring unit. The detector is a digital camera;
The pre-calibration step includes
By performing the recording step and the detection step a plurality of times by changing the position of the digital camera, the positions of all bright spots formed on the display screen by performing the white display are detected from multiple viewpoints,
The stereoscopic image measurement method according to claim 7, further comprising a step of calibrating internal parameters of the digital camera based on the positions of all bright spots detected from the multiple viewpoints by the error value measurement unit.