JP2006211383A - Stereoscopic image processing apparatus, stereoscopic image display apparatus, and stereoscopic image generating method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stereoscopic image processing apparatus for outputting a natural stereoscopic image of which the viewer hardly gets tired by composing a plurality of stereoscopic images. <P>SOLUTION: An image processing section 40 for receiving a plurality of stereoscopic images from an image input section 30 analyzes a change in frequency characteristics with respect to depth directions of the stereoscopic images, adjusts a degree of a change in frequency characteristics with respect to the depth direction of other stereoscopic image information on the basis of stereoscopic image information selected by an operation section 20 or the like, and thereafter composes the stereoscopic images. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a stereoscopic image processing device, a stereoscopic image display device, and a stereoscopic image generation method.

  In recent years, stereoscopic image display devices such as a display panel that displays a three-dimensional stereoscopic image that can be viewed three-dimensionally when an observer moves up, down, left, and right to view an image on a display unit are becoming popular in the market. . Various technologies relating to the display of this three-dimensional stereoscopic image have been developed. Specifically, the integral photography (hereinafter referred to as “IP”) method or a technology closely related thereto is deeply developed. A method using a light beam reproduction method is known.

  A stereoscopic image by the above-described method is mainly composed of a group of images with different parallax and a micro-hole array in which a plurality of micro-holes are arranged in an array, and images from the group of images with different parallax are the eyes of the observer. This is realized by arranging the micro-hole array so that an image with parallax is projected.

Here, an example of creating and displaying a stereoscopic image in the IP system will be described with reference to FIG. The creation (photographing) of a stereoscopic image is performed by recording a video in the video recording unit with an imaging system that is generally called a stereo multi-view system, which includes a subject, a micro-hole array, and a video recording unit. The video recording unit is divided into imaging regions corresponding to the micro through holes of the micro through hole array. The imaging areas corresponding to the respective micro through holes each constitute a camera, and a minute video is recorded in each area.
The recorded three-dimensional image is reproduced (displayed) on the video display unit provided behind the microscopic aperture array as viewed from the observer based on the microscopic image recorded on the video recording unit. This is performed by displaying an image that has been subjected to a conversion process in which the hole array is inverted in a point-symmetric manner.

  In addition to the above-described method of shooting and creating with the stereo multi-view method, a stereoscopic image is created by assuming a virtual stereoscopic subject on computer data, and light rays from the virtual stereoscopic subject are stereo-multiple. There is also a method based on computer graphics (hereinafter referred to as “CG”) which is calculated and printed when recorded in the video recording unit under the same conditions as the eye system.

  A stereoscopic image created by imaging using the stereo multi-view method described above and a stereoscopic image created by CG have advantages and disadvantages, respectively. For example, in the case of imaging, there is an advantage that the texture is excellent, but there is a drawback that the number of imaging steps increases. Conversely, CG has the advantage of being excellent in versatility, but has the disadvantage of poor texture. For this reason, when using a three-dimensional image as a street advertisement, for example, it is common to combine two or more images in order to take advantage of each advantage.

For example, Patent Document 1 proposes a technique for improving the visibility of a three-dimensional image and providing a natural three-dimensional effect with a sense of perspective when synthesizing a three-dimensional image created by the above-described method.
JP2001-223877

However, when the captured image and the image created by the CG are combined, the frequency characteristics of the images in the depth direction are different from each other. For example, when a street advertisement is created using the IP method, a product image captured by the stereo multi-view method and a surrounding image created by the CG are generally combined with a product image. Also, an image created by CG has higher contrast and higher frequency characteristics. For this reason, the surrounding image is more conspicuous than the product image, and there is a problem in using it as a street advertisement.
Also, in a stereoscopic image such as the IP method, an image is generated with a depth setting that is most easily visible before and after the gaze point distance. However, when the depth is set according to the original image, there is a disadvantage that fatigue tends to occur, and further development is desired.

  Therefore, an object of the present invention is to provide a stereoscopic image processing apparatus that synthesizes a plurality of stereoscopic images and outputs a stereoscopic image that is natural and that the observer sees and is not tired.

In order to solve the above-described problem, the invention according to claim 1 is characterized in that a stereoscopic image acquisition unit that acquires a plurality of stereoscopic image information and a frequency characteristic of an image in the depth direction of the stereoscopic image for each of the plurality of stereoscopic image information. Frequency characteristic acquisition means for acquiring the degree of change as frequency characteristic information, and other stereoscopic image information according to the acquired frequency characteristic information of any one of the plurality of stereoscopic image information. A frequency characteristic adjusting unit that adjusts the degree of change in the frequency characteristic with respect to the depth direction; and a three-dimensional image synthesizing unit that synthesizes any one of the three-dimensional image information and the other three-dimensional image information after the frequency characteristic adjustment. It is characterized by that.
Further, the invention described in claim 4 is a stereoscopic image generating method for generating the stereoscopic image described in claim 1.

  According to a second aspect of the present invention, the frequency characteristic adjusting means determines the degree of change of the frequency characteristic in the depth direction of the other stereoscopic image information in accordance with the gaze point distance in the depth direction of any one of the stereoscopic image information. It is characterized by adjusting.

  According to a third aspect of the present invention, there is provided a display for displaying a group of images with different parallax based on the synthesized three-dimensional information between the stereoscopic image processing apparatus according to the first or second aspect and an observer from a light source. A three-dimensional image is displayed to the viewer by arranging a portion and a plurality of micro-holes arranged in an array so that the image of the display portion is displayed as an image with different parallax between the eyes of the viewer And a stereoscopic image display means.

  According to the first and fourth aspects of the present invention, when one stereoscopic image is obtained by synthesizing a plurality of stereoscopic images, the frequency related to the depth direction of the other stereoscopic image information based on any one of the stereoscopic image information. Since the configuration is such that the degree of change in characteristics is adjusted, a more natural stereoscopic image can be obtained.

  According to the second aspect of the present invention, since the degree of change in the frequency characteristic of the other stereoscopic image information is adjusted in accordance with the gazing point distance of any one of the stereoscopic image information, the gazing point distance image Therefore, it is possible to obtain a stereoscopic image that is easier to see and less tiring for the observer.

  According to the third aspect of the present invention, a plurality of stereoscopic image information can be synthesized, and a stereoscopic image that is natural and difficult for an observer to see can be provided by, for example, the IP method.

[First Embodiment]
Embodiments of a stereoscopic image processing apparatus according to the present invention will be described below with reference to the drawings. The present invention is not limited to this embodiment. The embodiment of the present invention shows the most preferable mode of the present invention, and the terminology of the present invention is not limited to this.

  FIG. 1 is a diagram schematically illustrating a functional configuration of the image processing apparatus 1. As illustrated in FIG. 1, the image processing apparatus 1 includes a control unit 10, an operation unit 20, an image input unit 30, an image processing unit 40, a storage unit 50, an image output unit 60, and a communication unit 70.

  The control unit 10 includes a CPU (Central Processing Unit), an internal RAM (Random Access Memory), and a ROM (Read Only Memory) not shown. The control unit 10 controls the overall operation of the image processing apparatus 1 by sending a control signal to each unit in accordance with various control programs stored in the ROM using a predetermined area of the internal RAM as a work area in the CPU that controls the entire control. .

  The operation unit 20 includes a keyboard having various keys such as cursor keys, numeric input keys, and function keys, a pointing device such as a mouse / tablet, and the like. The signal is output to the control unit 10. The image processing apparatus 1 can smoothly perform a stereoscopic image editing operation by receiving an operation instruction from a user through the operation unit 20.

  The image input unit 30 as a stereoscopic image acquisition unit is a scanner, a video input interface, or the like that inputs the stereo multiview image 301 or the 3DCG rendering image 302, and outputs the input stereoscopic image information to the image processing unit 40. In addition to the input from the image input unit 30, the stereo multi-view image 301 and the 3DCG rendering image 302 may be input from a recording medium by the storage unit 50 described later or another information device by the communication unit 70.

  The image processing unit 40 includes a first image analysis unit 401, a second image analysis unit 402, an image adjustment unit 403, and an image synthesis unit 404, and synthesizes a plurality of stereoscopic image information input from the image input unit 30. And output to the image output unit 60.

  A first image analysis unit 401 and a second image analysis unit 402 serving as frequency characteristic acquisition units receive stereoscopic image information (image signals) of the stereo multiview image 301 and the 3DCG rendering image 302 input from the image input unit 30. Received and stored data such as shooting conditions and rendering creation conditions including information about frequency characteristics in the depth direction of the image added to the stereoscopic image information in the analysis data storage unit 501, and according to the data format of the signal as necessary The compression code is restored, the color signal expression method is converted, and the like, and the image signal converted into a data format suitable for the image processing operation is sent to the image adjustment unit 403.

  In addition, the first image analysis unit 401 and the second image analysis unit 402 each of the image information when the information regarding the frequency characteristics necessary at the time of image composition to be described later is insufficient in the shooting conditions and the rendering creation conditions. Not only the position of the pixel on the plane parallel to the screen but also the depth information and the frequency characteristics on the same plane at the depth are analyzed, and the obtained data is stored in the analysis data storage unit 501. In addition to the configuration acquired from the first image analysis unit 401 and the second image analysis unit 402, the information related to the frequency characteristics described above is stored in advance in the configuration directly input from the operation unit 20 or the storage unit 50. Alternatively, the table information related to the frequency characteristics may be referred to.

  The image adjustment unit 403 serving as a frequency characteristic adjustment unit includes a first image conversion unit 403a and a frequency characteristic conversion unit 403b. The image signal input from the first image analysis unit 401 and the second image analysis unit 402 and Then, the image processing according to the present invention is performed on the basis of the data on the frequency characteristics in the depth direction of the stereoscopic image stored in the analysis data storage unit 501 and output to the image composition unit 404.

  The first image conversion unit 403a and the frequency characteristic conversion unit 403b perform, for example, image gray balance adjustment, density adjustment, and gradation on the image signals received from the first image analysis unit 401 and the second image analysis unit 402. Image processing for improving the image quality of output images, such as tone control, hypertone processing that compresses the gradation of the ultra-low frequency luminance component of the image, and hyper sharpness processing that enhances sharpness while suppressing particles The frequency characteristic conversion processing described later is performed by the frequency characteristic conversion unit 403b.

  Further, the first image conversion unit 403a and the frequency characteristic conversion unit 403b are configured to execute image processing for intentionally changing the image tone (for example, image processing for finishing a person thin or wrinkle removal). It may be. The input of the image signal to the first image conversion unit 403a is not necessarily received from the first image analysis unit 401 and the second image analysis unit 402. For example, the image input unit 30, the storage unit 50, the communication The structure received from the part 70 may be sufficient. Further, the first image conversion unit 403a and the frequency characteristic conversion unit 403b can perform the frequency characteristic conversion process before various conversion processes as long as the effects of the present invention are not hindered. The reverse may be possible.

  An image composition unit 404 as a three-dimensional image composition unit synthesizes a plurality of pieces of stereo image information after conversion processing using the image signal sent from the image adjustment unit 403, and creates three-dimensional image information (image signal) as one stereo image. ) Is output to the image output unit 60, and image synthesis processing described later is performed.

  The storage unit 50 stores data necessary for the processing of the image adjustment unit 403 and the analysis data storage unit 501 that stores the data analyzed by the first image analysis unit 401 and the second image analysis unit 402 described above. A storage unit 502 is provided. In addition, the storage unit 50 stores various setting data and programs used during control processing by the control unit 10, and sends data stored in the storage unit 50 to an image data writing unit 606 described later to perform various recordings. It is the structure stored in a medium. Specifically, the storage unit 50 is configured by an EEPROM (Electrically Erasable and Programmable ROM) which is an electrically erasable and rewritable nonvolatile memory, and is input with an address specified by the control unit 10. When data is stored or, conversely, an address is designated, the data stored at the address is output to the control unit 10.

  The image output unit 60 includes a CRT specific processing unit 601, a CRT 602, a printer specific processing unit 603, a printer 604, an image data format creation unit 605, and an image data writing unit 606, and displays them on the CRT 602 or outputs them to the printer 604. The image is displayed as a three-dimensional image on the observer's eyes using a display panel or the like. In addition, the image output unit 60 converts the data into a predetermined data format by the image data format creation unit 605 and outputs it to the image data writing unit 606 to write to the recording medium or to the communication unit 70 to output the other data. To other information devices.

  The CRT specific processing unit 601 performs processing such as changing the number of images and color matching as necessary on the image signal received from the image processing unit 40, and information necessary for displaying control information from the control unit 10 The display signal combined with the above is sent to the CRT 602. As a result, it is possible to obtain the situation of image creation as appropriate and make necessary corrections from the operation unit 20. The CRT 602 may be an LCD (Liquid Crystal Display) or the like, and is not particularly limited.

  Further, the CRT 602 may be configured to include glasses with a liquid crystal shutter that can be worn in front of the user. In this case, two images with parallax are alternately switched at high speed and displayed on the screen, and by using a liquid crystal shutter, the right eye image for the user's right eye and the left eye for the left eye The image of is displayed.

  A printer specific processing unit 603 performs calibration processing, color matching, pixel number change, etc. specific to the printer 604 as necessary, and sends an image signal to the printer 604. The printer 604 is a developing device using a large-format ink jet printer or a silver halide photosensitive material. More preferably, a configuration in which a dedicated printer specific processing unit 603 is provided for each model of the printer 604 to be connected to perform proper calibration processing is desirable.

  The image data format creation unit 605 converts the image signal received from the image processing unit 40 to JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), Exif (Exchangeable Image File Format) or the like as necessary. Conversion into various general-purpose image formats represented is performed, and image signals are transferred to the image data writing unit 606 and the communication unit 70. The image data writing unit 606 can be written on various recording media such as a multimedia card, a memory stick (registered trademark), an MD (Mini Disk: registered trademark), a CD-R, and a DVD-RAM.

  The communication unit 70 is a communication circuit capable of data communication with other information devices externally by various communication methods such as a LAN line and Bluetooth wireless communication.

  It should be noted that the internal configuration of the image processing unit 40 described above with reference to FIG. 1, the CRT specific processing unit 601, the printer specific processing unit 603, the image data format creation unit 605, and the like are provided to assist in understanding the functions of the present invention. It is not necessary to be realized as a physically independent device, but may be realized as a type of software processing in a single CPU, for example.

  The image creation (image processing) of the present invention receives stereo multi-view images and 3DCG data, and the first image analysis unit 401 and the second image analysis unit 402 use the above shooting conditions, rendering creation conditions, and multi-viewpoint images. As the position information for each pixel, not only the position on the plane parallel to the screen but also the frequency characteristics on the same plane in the depth direction and the depth are analyzed, and the information is sent to the analysis data storage unit 501. The depth information and the image information obtained from these inputs are received by the image adjustment unit 403, and the frequency characteristic conversion which is the image processing of the present invention is performed by the frequency characteristic conversion unit 403b. Thereafter, the image composition unit 404 synthesizes the images obtained from the respective input means with the image for each viewpoint by arriving at the image of these processes. Thereafter, for example, when outputting to a printer, the printer-specific processing unit 603 performs processing specific to the printer to obtain the effects of the present invention.

  As described above, the image processing apparatus 1 creates a display device such as a display panel that finally displays a three-dimensional image. However, only 3DCG, only a stereo multiview image, or 3 can be input as an image to be combined. There may be more than one. The output of the image is not limited to a method using a photographic photosensitive material, and may be an ink jet method, an electrophotographic method, a thermal method, or a sublimation type printing apparatus.

  Next, image processing performed by the control unit 10 by controlling the image processing unit 40 to output a combined multi-eye image and a CG rendering image (3DCG) will be described with reference to FIGS. 2 and 3. 2 is a flowchart showing main steps of image processing, FIG. 3A is a flowchart showing stereo multi-view image acquisition processing, and FIG. 3B is a flowchart showing CG rendering image acquisition processing. It is.

  First, the control unit 10 performs a stereo multi-view image acquisition process for analyzing the stereo multi-view image input from the image input unit 30 (step S11). When the stereo multi-view image income process is started by inputting image data from the image input unit 30, the stereo multi-view image captured in step S111 as a stereo image input process is captured as image data, and the stereo multi-view image is obtained. Image distortion correction is performed (step S112), and corresponding points on the image are extracted (step S113). The extraction of the corresponding points may be performed automatically, or may be performed semi-automatically, for example, by automatically specifying the first few points by input from the operation unit 20 and then automatically as necessary. In this case, analysis of image data including processing such as extraction of main parts (for example, an area corresponding to a human face (face area)) in one image and calculation of various image feature amounts is performed. It doesn't matter.

  After step S113, the parallax amount is calculated from the difference between the feature points using a calculation method such as triangulation (step S114), and a plane parallel to the screen is obtained for each original image pixel using the same calculation method. The upper position (x, y) and the depth coordinate (z) are calculated (step S115).

  After step S115, a frequency characteristic for each depth coordinate is calculated in step S116 as a frequency characteristic acquisition step. In the present invention, it is possible to obtain a frequency characteristic at the forefront surface and the rearmost surface of the stereoscopic image in the depth direction, and to perform processing for adjusting the degree of change of the frequency characteristic so as to match, but preferably the frequency in the depth direction A function of the frequency characteristic in the depth direction is calculated from the degree of change of the characteristic using the least square method or the like, and processing for matching the change of the frequency characteristic is performed using the function.

  After step S116, using the parallax amounts, x, y, and z, an image for interpolating the viewpoint as necessary is created from the original image (step S117). However, in the present invention, when stereo multi-view shooting is performed, an image shot with the same number of viewpoints and parallax amount as 3DCG is combined with 3DCG, or with the same number of viewpoints and parallax amount in stereo multi-view shooting. When synthesizing images of a plurality of captured scenes, the present invention can be implemented without performing viewpoint interpolation, and this step is performed when the number of viewpoints is the same even when the amount of parallax is different. Can be omitted.

  Here, the process of obtaining the viewpoint interpolation image will be specifically described. First, a reference viewpoint image serving as a reference for the obtained multi-view stereo image is set. For the reference viewpoint image, the amount of misalignment of other image data is calculated, and the amount of misalignment in the horizontal direction varies depending on the distance between multiple cameras and the subject at the time of shooting and the distribution in the depth direction. Distortion and image position adjustment for each image data are performed. Then, the amount of positional deviation of the left and right image data is obtained by template matching so that the same portion of the subject is included in almost all areas in the image data of the comparison pair.

  Subsequently, image data of a predetermined rectangular area is cut out to create a template image. Then, the difference sum of the pixel values from the image data (RGB data, etc.) of the same rectangular area of the image is calculated by a predetermined amount of movement in each of the horizontal and vertical directions, and when the difference sum of the pixel values is the smallest Let the amount of movement be the amount of displacement. After doing so, corresponding points representing the same subject portion as point-to-point correspondence between the images of the image pair in step S113 are extracted. Corresponding points are extracted by applying template matching to each point to find corresponding points. If the image size is large, the processing time becomes enormous, so that it can be performed with the required number of extraction points.

  The position of the corresponding point is determined by performing template matching and the smallest position among the difference sum of the feature points. However, if it is determined based on a threshold value given in advance that the reliability of the corresponding point extraction process is low from the value of the difference sum at the corresponding point position, information indicating that the point is not supported is given.

  The above processing is repeated to obtain a plurality of corresponding points of the multi-viewpoint stereo image pair over the image area. In addition, some points can be obtained directly from the difference by extracting the corresponding points. However, in order to improve the accuracy of the uncorresponding points and the points that were not originally searched for the corresponding points, they are supplemented by obtaining the parallax. May be.

  Subsequently, the parallax is obtained by interpolation from the four neighboring parallaxes with respect to the obtained parallax even for the point for which the corresponding point has not been obtained from the parallax obtained so far. The parallax interpolation method can also be calculated by a weighted average based on the number of points for each corresponding point position. Through the above processing, the amount of parallax can be obtained for all the pixels of the image.

  A method in the case of performing viewpoint interpolation of a multi-viewpoint image using the above-described parallax amount analysis data will be described as an example. The image size generated here is determined by the number of viewpoint images and the resolution and size at the time of printing, the printer resolution is a (dpi: dot per inch), and the print size is SX and SY inches. The image size to be printed requires pixels of x = a × SX and y = a × SY.

  The viewpoint image number N is determined to be an integer such that N = a × f in accordance with the pitch of the minute through hole region, f inch. For example, when the resolution of the printer is 400 dpi and the pitch of the micro-hole region plate is 1/20 inch, a total of 256 viewpoint images in the horizontal and vertical directions of N = 14 is desirable. At this time, the size of the image at each viewpoint is H = x / N and V = y / N in the horizontal (H) and vertical (V) directions. Actually, the print size is determined so that H and V are integers.

  The viewpoint position of the image to be generated is determined so that a predetermined amount of parallax is generated. If the amount of parallax becomes too small, the stereoscopic effect when observing a stereoscopic image is impaired. If the amount of parallax is too large, the stereoscopic image to be observed becomes unclear. Further, in order to observe a stable stereoscopic image, the viewpoint positions are determined so that the images at the respective viewpoints are arranged symmetrically at equal intervals around the viewpoint position of the image serving as the reference viewpoint.

  A method for generating an interpolated viewpoint image will be described. As a new viewpoint image created by interpolation, a parallax amount di to be obtained or an image corresponding to the left and right of the nearest pair capable of interpolating the image corresponding to the viewpoint is selected. For example, horizontal viewpoint interpolation will be described with reference to an image corresponding to the left nearest viewpoint. Assuming that the position (xi, yi) in the new viewpoint image for mapping each pixel is the pixel position (xL, yL) of the left image, the pixel position (xR, yR) of the right image, and the parallax amounts dL, dR, xi = xL + (Di-dL) / | (dL-dR) | is obtained (yi is obtained in the same manner). Here, when (di-dL) is negative, it indicates the shift amount on the right side. When the new viewpoint image and the original image are different, the above numerical value is multiplied by the ratio of the viewpoint image size / original image size. Then, the pixel signal value at the pixel position (xL, yL) of the left image is recorded at the position (xi, yi) of the new viewpoint image. This process is repeated for all pixels of the left image.

  Next, among the pixels of the new viewpoint image, for the pixels to which no pixel is assigned from the left image, processing for finding from the right side by interpolation from the same amount of parallax, or searching for neighboring pixels adjacent to the desired pixel Perform hole filling such as assigning average values. As another method, it is also possible to estimate using the adjacent image of the generated new viewpoint pixel.

  By the method described above, a new viewpoint image in which all pixels are valid is generated. By repeating this process for the number of viewpoints, the viewpoint image is interpolated. More specifically, it may be obtained using data of a plurality of viewpoint images as well as near the left and right.

Next, a CG rendering image acquisition process (step S12) for capturing 3DCG performed after step S11 will be described. First, the control unit 10 takes in a plurality of rendered viewpoint image data (step S121). Note that 3DCG capture when combining with a stereo multi-view image may be configured to perform each of the above-described steps S111 to S117. And performing the processing shown in step S12 so as to obtain similar multi-viewpoint image data is more preferable because the number of work steps is small and the improvement in quality is expected.
In the present embodiment, the stereo multi-view image and 3DCG are both captured. However, Step S11 and Step S12 are selected according to the captured image, and only one of the images may be captured. .

  After step S121, the parallax amount is calculated using a calculation method such as a triangulation method or the like from a rendering condition or a feature point difference (step S122). After the coordinates are calculated (step S123), the frequency characteristics of the image for each coordinate in the depth direction are calculated (step S124). Note that the calculation process in step S124 may be configured to obtain a function of frequency characteristics in the depth direction using a least square method or the like, similar to step S116 described above.

  After input of a plurality of stereoscopic image information in Steps S11 and S12, that is, a stereo multiview image and 3DCG are input, the relationship between the frequency characteristics in the depth direction that is the target to be converted is based on any one of the stereoscopic image information. (Step S13).

  The determination of any one of the three-dimensional image information in step S13 may be determined in advance as either a stereo multi-view image or 3DCG, or may be selected from the operation unit 20. In addition, a configuration that targets the relationship with the frequency characteristics in the depth direction, the degree to which the frequency characteristics in the depth direction are changed, and the frequency characteristic conversion table stored in advance in the storage unit 50 are used. It may be configured.

  After step S13, a first image conversion process such as color conversion is performed within a range that does not impair the effects of the present invention (step S14), and the depth to be converted as described above is obtained in step S15 as the frequency characteristic adjustment step. Based on the relationship with the frequency characteristic in the direction, the analysis information obtained in steps S117 and S124 described above is read from the analysis data storage unit 501, and the frequency characteristic conversion process is performed for each pixel on the image at each viewpoint. Is done.

  Here, the frequency characteristic conversion in step S15 will be described with reference to FIGS. FIG. 8A is a diagram illustrating an image of a house by 3DCG, FIG. 8B is a diagram illustrating a landscape image captured with stereo multi-view, and FIG. It is a figure which illustrates the image at the time of synthesize | combining the image of Fig.8 (a), (b). FIG. 9 is a graph showing that the relationship between the frequency characteristics in the depth direction for the images shown in FIGS. 8A and 8B is corrected according to the depth position Z2 as the point of sight.

  For example, if it is determined in step S13 that the relationship between the frequency characteristics in the depth direction to be converted is based on a landscape image, in step S15, the frequency characteristics change in the depth direction in FIG. 8B. Similarly, frequency characteristic conversion is performed on the image of FIG. For example, in an imaging apparatus in which a 3DCG image is set at a distance position of 3 to 5 m in the depth direction, and a landscape image has a predetermined frequency characteristic OTF (Optical Transfer Function) or MTF (Modulaton Transfer Function) at a focus position of 10 m to infinity. In the case of a captured image, the 3DCG image is converted into a degree of blur equivalent to the 3 to 5 m position in the landscape image. For this reason, when the images are combined as shown in FIG. 8C, the 3DCG image is combined with the image drawn as the landscape image without a sense of incongruity.

  The conversion of the frequency characteristic at this time is performed so that the distance between the depth direction before the processing and the depth direction after the processing matches as shown by the conversion straight line A (Z = Za) in FIG. That is, the depth positions Z1 to Z3 before the processing are converted into a linear shape while maintaining the distance in the depth direction after the processing, as indicated by the one-dot chain line or the two-dot chain line.

However, when the gazing point of the stereoscopic image is the depth position Z2, as shown in the conversion curve B of FIG. 9, the depth position Z2 of the gazing point is the depth position Z21 and the distances in the depth direction match, and the depth position Z1 is The distance in the depth direction based on a predetermined conversion curve (conversion formula) with the depth position Z2 being the gazing point as an inflection point so that the depth position Z11 and the depth position Z3 shift to the depth position Z31. May be configured to be converted.
The conversion curve B is a conversion formula determined in advance by various conditions such as the distance between the micro-hole array and the video display unit or the video recording unit, and the resolution of the image displayed on the video display unit in the IP system, for example. is there.

  In addition, when the gazing point of the stereoscopic image is the depth position Z2, not only a configuration that converts the distance in the depth direction but also a configuration that adjusts the degree of change in the frequency characteristic in the depth direction based on the conversion curve B described above. It may be. That is, the depth direction at the time of the conversion of the frequency characteristics is handled by converting the depth Z to the depth Za around the depth position Z2 that is the gazing point. For this reason, the conversion of the frequency characteristics is performed so that the width of the high contrast spreads around the gazing point.

  In this way, in the configuration in which the distance in the depth direction is reduced or the degree of change in frequency characteristics is adjusted according to the gaze point distance, the depth direction is reduced to the gaze point of the stereoscopic image, and the image is centered on the gaze point distance. Since the conversion is easy to see (to high contrast), for example, a viewpoint jump in a stereoscopic image display device using a minute through-hole such as an IP method becomes inconspicuous, and a stereoscopic image that is easy to see (not fatigued) can be displayed. it can.

  After step S15, image synthesis is performed for each viewpoint in step S16 as a stereoscopic image synthesis process, and the processed image data is stored in the image data writing unit 606 or the data storage unit 502 (step S17). ,finish. Note that when the image is combined in step S16, if there is no object to be highlighted in the image, the foreground pixel is used in each scene, and the corresponding pixel of the corresponding viewpoint in other scenes is used. Fill. On the contrary, when the target to be highlighted is designated by the operation unit 20 or the like, it is instructed by performing judgment such as performing template matching, and the corresponding pixels of the corresponding viewpoints in other scenes are painted out. A desired composite image can be obtained.

  Next, a stereoscopic image display device such as a display panel that displays a stereoscopic image to an observer using an IP image, which is created using the above-described image processing device 1, will be described with reference to FIGS. FIG. 5 is a process principle diagram showing a method for manufacturing a stereoscopic image display device according to the present invention, FIG. 6 is a principle diagram illustrating creation of an IP image in the stereoscopic image display device, and FIG. 7 is a stereoscopic image. 1 is a schematic configuration diagram of a display device.

  The input process 101 is the process of steps S11 and S12 described above, and the multi-viewpoint image data conversion process 102 is each process of steps S13 to S17. Through this process, a plurality of stereoscopic image information by a stereo multi-view image or 3DCG-shaped 3D data input is input, and frequency characteristic conversion processing is performed based on any one of the stereoscopic image information and instruction data input from the operation unit 20. It is synthesized later and stored in the data storage unit 502 as processed stereoscopic image information.

  In the IP image processing step 103, a single IP image is created from an image group with different viewpoints based on the processed stereoscopic image information stored in the data storage unit 502. For example, the image of one viewpoint having 180 × 180 pixels is collected for 32 vertical viewpoints × 32 horizontal viewpoints = 1024 viewpoints, and the image of each viewpoint observed from one pinhole (or lens) is a position corresponding to the pinhole. A single IP image is created. Specifically, as shown in FIG. 6, when one screen size of the original image is 180 × 180 images and the number of viewpoints is 32 in the upper and lower directions and 32 in the left and right, one IP address is generated from the 1024 image groups 110 with different viewpoints. A work image 111 is created. The process shown in FIG. 6 is the IP image processing process 103.

  In the IP image output step 104, the printer 604 forms an image for IP 111 on a plastic film support (preferably a transmissive support) with a light-sensitive material such as a silver halide photographic light-sensitive material.

Examples of the photosensitive material include the following.
First, a silver halide color photographic light-sensitive material is exemplified. As the photosensitive material, any known material may be used as long as it has at least three kinds of photosensitive layers having different absorption wavelength regions from each other on a light transmissive support or a reflective support. It is preferably a transmitted light observation type silver halide color photographic light-sensitive material on which an image is formed on a light-transmitting support.

  The silver halide color photographic light-sensitive material may be a color negative film for photography or a color positive film, and may be either a printing paper for printing or a transmission printing film for display. One preferred embodiment of the silver halide light-sensitive material that can be used in the present invention is a large-size color positive film. Another aspect is a color film for creating a transmissive display, and a display film suitable for digital exposure is particularly preferable.

  In the 3D display process 105, a pinhole (silver salt photosensitive material or one formed by printing) or a fly-eye lens (printed or molded) that is a minute through hole region, and an IP image output process 104 are formed. The three-dimensional image is displayed on the viewer's eyes by combining the IP image (via an appropriate spacer).

  As shown in FIG. 7, the stereoscopic image display device is a stereoscopic image display means for displaying a stereoscopic image to an observer. As shown in FIG. 7, the upper transparent sheet 114a has a micro-hole array 113 and the lower transparent sheet 114c has an IP image. 111 are respectively printed by the image processing apparatus 1 in advance, and an intermediate transparent plate 114b is inserted between them in order to keep the interval between the two transparent sheets constant. In order to illuminate the IP image 111, a light source 112 is placed thereunder.

  The observer H looks at the light that has passed through the micro-hole array 113. At that time, it cannot be distinguished whether the light entering the right eye has come out from the point Q of the IP image 111 or from S on the three-dimensional object. Similarly, it cannot be distinguished whether the light entering the left eye has come out from the point R of the IP image 111 or from S on the virtual three-dimensional object. Therefore, the binocular parallax makes it appear as if there is an object at the point S. 3, the position of the micro-hole array 113 may be exchanged with the IP image 111, which is an IP image in FIG. 3, and if the light source 112 uses reflected light instead of transmitted light, It may be on the observer H side. In this way, the stereoscopic image display device using the IP image can easily obtain a stereoscopic image, and when a silver salt photographic printer is used, there is an advantage that the stereoscopic effect is improved.

  As described above, the image processing apparatus 1 combines a plurality of input stereoscopic image information by performing frequency characteristic conversion processing according to any one of the stereoscopic image information, thereby combining a natural and excellent texture. A stereoscopic image can be output. In addition, by creating a stereoscopic image display device using the image processing device 1, it is possible to provide a stereoscopic image display device that displays a stereoscopic image that has a more natural and excellent texture and is easy for an observer to see.

  The present invention can be freely changed and improved without departing from the spirit of the invention. For example, the stereoscopic image display device includes a plurality of micro-hole arrays 113 whose mask positions can be changed by liquid crystal shutters whose opening positions can be electrically changed, and a plurality of IP images 111 on which two-dimensional stereoscopic images are drawn. It may be configured to display a pseudo moving image with frames.

It is a principle figure which shows the principle of the imaging and display of a stereo image by an integral photography (IP) system. It is the figure which showed typically the functional structure of the image processing apparatus 1 in this invention. It is a flowchart which shows the image processing in this invention. (A) is a flowchart which shows a stereo multiview acquisition process, (b) is a flowchart which shows a CG rendering image acquisition process. It is a process principle figure which shows the manufacturing method of the stereo image display apparatus in this invention. It is a principle figure explaining preparation of an IP image. 1 is a schematic configuration diagram of a stereoscopic image display device. (A) is a figure which illustrates the image of the house by 3DCG, (b) is a figure which illustrates the landscape image imaged with the stereo multiview, (c) is a figure with Fig.8 (a), and FIG. It is a figure which illustrates the image at the time of synthesize | combining the image of 8 (c). The graph which showed correcting the relationship of the frequency characteristic in the depth direction about the image shown to Fig.8 (a), (b) according to the depth position Z2 which is a gazing point.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Control part 20 Operation part 30 Image input part 40 Image processing part 50 Storage part 60 Image output part 70 Communication part 101 Input process 102 Multiview image data conversion process 103 IP image processing process 104 IP image output process 105 3D Display process 110 Image group 110a with different viewpoints Captured image 111 IP image 112 Light source 113 Microperforation array 114a Upper transparent sheet 114b Intermediate transparent plate 114c Lower transparent sheet 301 Stereo multi-view image 302 3DCG rendering image 401 First image Analysis unit 402 Second image analysis unit 403 Image adjustment unit 403a First image conversion unit 403b Frequency characteristic conversion unit 404 Image composition unit 501 Analysis data storage unit 502 Data storage unit 601 CRT specific processing unit 602 CRT
603 Printer specific processing unit 604 Printer 605 Image data format creation unit 606 Image data writing unit A Conversion line B Conversion curve H Observer Z, Za Depth Z1-3, Z11, Z21, Z31 Depth position

Claims (4)

Stereoscopic image acquisition means for acquiring a plurality of stereoscopic image information;
Frequency characteristic acquisition means for acquiring, as frequency characteristic information, the degree of change in the frequency characteristic of the image in the depth direction of the stereoscopic image for each of the plurality of stereoscopic image information;
A frequency characteristic adjusting unit that adjusts the degree of change of the frequency characteristic related to the depth direction of other stereoscopic image information according to the acquired frequency characteristic information of any one of the plurality of stereoscopic image information; ,
Stereoscopic image synthesis means for synthesizing any one of the stereoscopic image information and the other stereoscopic image information after the frequency characteristic adjustment;
A stereoscopic image processing apparatus comprising:   The frequency characteristic adjusting means adjusts the degree of change of the frequency characteristic in the depth direction of the other stereoscopic image information according to the gaze point distance in the depth direction of the one of the stereoscopic image information. The stereoscopic image processing apparatus according to 1. The stereoscopic image processing apparatus according to claim 1 or 2,
A display unit that displays a group of images with different parallaxes based on the combined three-dimensional information between a light source and an observer, and a plurality of minute through holes are arranged in an array so that the image on the display unit A three-dimensional image display means for displaying a stereoscopic image to an observer by arranging a micro-hole array that reflects both parallax images as parallax images;
A stereoscopic image display device comprising: A stereoscopic image acquisition step of acquiring a plurality of stereoscopic image information;
A frequency characteristic acquisition step of acquiring, as frequency characteristic information, the degree of change in the frequency characteristic of the image in the depth direction of the stereoscopic image for each of the plurality of stereoscopic image information;
A frequency characteristic adjustment step of adjusting the degree of change in the frequency characteristic regarding the depth direction of the other stereoscopic image information according to the acquired frequency characteristic information of any one of the plurality of stereoscopic image information; ,
A stereoscopic image combining step of combining any one of the stereoscopic image information and the other stereoscopic image information after the frequency characteristic adjustment;
3D image generation method characterized by including.

Images