JP5838775B2 - Image processing method, image processing system, and image processing program - Google Patents

Description

  The present invention relates to an image processing method, an image processing system, and an image processing program directed to parallax adjustment between images ordered along a time axis.

  In conjunction with the development of display devices in recent years, development of image processing technology related to stereoscopic display has been promoted. As a typical method for realizing such stereoscopic display, there is a method using binocular parallax felt by humans. When such binocular parallax is used, it is necessary to generate a pair of images (hereinafter also referred to as “stereo images” or “3D images”) with parallax according to the distance from the imaging means to the subject. is there.

  Such stereoscopically displayed images include not only still images but also moving images. For example, Japanese Unexamined Patent Application Publication No. 2009-239388 (Patent Document 1) discloses a stereoscopic moving image processing apparatus for stereoscopically viewing a stereoscopic moving image. This stereoscopic moving image processing device discloses a configuration for reducing fatigue of a user who performs stereoscopic viewing at the time of scene switching of a stereoscopic moving image. More specifically, this stereoscopic moving image processing apparatus detects the scene switching position of the stereoscopic moving image and adjusts the depth feeling of the stereoscopic moving image so that the depth feeling at the scene switching position is gradually changed. To do.

  As described above, since a moving image is different from a still image in that the viewing time of the user becomes longer, it is important to reduce the user's fatigue caused by parallax.

JP 2009-239388 A

  When stereoscopically displaying a moving image as described above, it is necessary to adjust the generated parallax to a level comfortable and appropriate for viewing. When a subject having a large parallax suddenly enters as a subject included in a moving image, it is difficult to adjust the parallax. For example, if a vehicle or the like crosses the field of view during imaging of a certain landscape, the vehicle closer to the imaging means is stereoscopically displayed on the near side (outward side) of the user (stereoscopic). It is preferable to generate a certain amount of parallax so that the feeling can be reproduced).

  However, when the overall parallax adjustment amount (parallax position or parallax range) of the stereoscopically displayed image is changed in accordance with such a change in the subject (generated parallax), the entire reproduced image is viewed from the user's viewpoint. The three-dimensional effect of this will change abruptly, which can give an uncomfortable feeling.

  Therefore, the present invention has been made to solve such a problem, and the object of the present invention is to use a plurality of images ordered along the time axis and to reduce the burden on the user while leaving a stereoscopic effect. To provide an image processing method, an image processing system, and an image processing program capable of reproducing a small number of stereoscopic displays.

An image processing method according to an aspect of the present invention includes an acquisition step of acquiring a plurality of first input images ordered along a time axis and a plurality of second input images ordered along a time axis. Including. The plurality of first and second input images are acquired by imaging the subject from different viewpoints. The image processing method further includes a generation step of generating a parallax image from first and second input images corresponding to each position on the time axis, and smoothing a plurality of parallax images in the time axis direction over a first time range. And a correction step for correcting at least the magnitude of the parallax on the fly-out side to a value corresponding to the result of smoothing over the first time range for the parallax image existing in the first time range, and including.

  Preferably, the correction step corresponds to a result of smoothing the parallax range between the projection-side parallax and the depth-side parallax over the first time range for the parallax image existing in the first time range. Including the step of limiting to the value.

  Preferably, the smoothing step includes a step of smoothing a plurality of parallax images in a time axis direction over a second time range different from the first time range, and the correction step is smoothed over the first time range. And correcting the parallax image based on the result and the parallax image obtained by the smoothing over the second time range.

More preferably, the first time range is longer than the second time range.
Preferably, the image processing method uses a plurality of first input images and a plurality of images generated by giving parallax according to corrected parallax images respectively corresponding to the plurality of first input images. The method further includes the step of generating a stereoscopic image.

  Preferably, the image processing method includes a step of generating a stereoscopic image by using a plurality of first and second input images obtained by adjusting the parallax according to the corresponding corrected parallax images, respectively. In addition.

  Preferably, the correction step includes a step of determining a parallax increase / decrease coefficient that is a change amount along the time axis of the parallax generated in the parallax image.

  More preferably, the correction step includes a step of limiting the parallax increase / decrease coefficient to a predetermined range.

  Preferably, the correcting step includes a step of determining an offset of the parallax position generated in the parallax image.

  Preferably, the correcting step includes a step of limiting a minimum value of parallax according to a result of smoothing over the first time range.

  Preferably, the smoothing step compares the maximum value or the minimum value of the parallax image corresponding to a certain position on the time axis with the maximum value or the minimum value of the parallax image corresponding to an adjacent position on the time axis. For a pixel that has changed beyond a predetermined threshold, the step of limiting the maximum or minimum value of the parallax to the predetermined threshold is included.

  Preferably, the smoothing step has a predetermined value compared to the parallax indicated by the pixel included in the smoothed parallax image among the parallax indicated by the pixel included in the parallax image corresponding to a certain position on the time axis. For pixels that change beyond the threshold, the step of limiting the corresponding parallax magnitude to the predetermined threshold value is included.

Preferably, the first time range is 10 seconds or more.
Preferably, the second time range is 10 seconds or less.

  An image processing system according to another aspect of the present invention obtains a plurality of first input images ordered along a time axis and a plurality of second input images ordered along a time axis. including. The plurality of first and second input images are acquired by imaging the subject from different viewpoints. The image processing system further includes generating means for generating a parallax image from the first and second input images corresponding to each position on the time axis, and smoothing the plurality of parallax images in the time axis direction over the first time range. Smoothing means for converting, and correction means for correcting the parallax image existing in the first time range to a value corresponding to the result of smoothing at least the size of the parallax on the projecting side over the first time range; including.

  An image processing program according to another aspect of the present invention obtains a plurality of first input images ordered along a time axis and a plurality of second input images ordered along a time axis in a computer. The acquisition step to be executed is executed. The plurality of first and second input images are acquired by imaging the subject from different viewpoints. The image processing program further causes the computer to generate a parallax image from the first and second input images corresponding to each position on the time axis, and to generate a plurality of parallax images in the time axis direction over the first time range. Smoothing step for smoothing, and correction for correcting the parallax image existing in the first time range to a value corresponding to the result of smoothing the parallax on the projecting side at least over the first time range Step.

  According to the present invention, it is possible to reproduce a stereoscopic display with less burden on the user, while leaving a stereoscopic effect, using a plurality of images ordered along the time axis.

1 is a block diagram showing a basic configuration of an image processing system according to an embodiment of the present invention. It is a figure which shows the specific structural example of the imaging part shown in FIG. It is a block diagram which shows the structure of the digital camera which actualized the image processing system shown in FIG. It is a block diagram which shows the structure of the personal computer which actualized the image processing system shown in FIG. It is a flowchart which shows the whole process procedure according to Embodiment 1 of this invention. It is a figure which shows an example of the input image in a certain flame | frame imaged by the imaging part 2 shown in FIG. 7 is a diagram showing an example of a parallax image generated from a pair of input images shown in FIG. 6 according to the image processing method according to the first embodiment. FIG. It is a figure which shows an example of the averaging filter used for the spatial smoothing process according to Embodiment 1 of this invention. It is a figure for demonstrating the time smoothing process according to Embodiment 1 of this invention. It is a figure which shows an example of the smoothed parallax image obtained by the spatial and temporal smoothing process with respect to the parallax image shown in FIG. It is a figure for demonstrating the parallax whole adjustment amount determination process according to Embodiment 1 of this invention. It is a figure which shows an example of the parallax image by which parallax adjustment was carried out with respect to the parallax image after smoothing shown in FIG. It is a figure for demonstrating the stereoscopic vision image generation process according to Embodiment 1 of this invention. It is a flowchart which shows the process sequence of the stereoscopic vision image generation process shown in FIG. It is a figure which shows an example of the stereo image produced | generated using the input image 1 and the input image 2 shown in FIG. It is a figure which shows the example of application of the image processing method according to Embodiment 1 of this invention. It is a figure which shows the time change of each parameter corresponding to the example of application shown in FIG. It is a figure which shows the time change of each parameter shown in FIG. 17 when a parallax increase / decrease coefficient is restrict | limited. It is a flowchart which shows the whole process sequence according to Embodiment 2 of this invention. It is a figure which shows the example of application of the image processing method according to Embodiment 2 of this invention. It is a figure which shows an example of the stereo image output in the application example shown in FIG.

  Embodiments of the present invention will be described in detail with reference to the drawings. In addition, about the same or equivalent part in a figure, the same code | symbol is attached | subjected and the description is not repeated.

[A. Overview]
An image processing system according to an embodiment of the present invention acquires a plurality of first input images ordered along a time axis and a plurality of second input images ordered along a time axis, A parallax image necessary for stereoscopically viewing an image along a time axis (typically reproducing a moving image stereoscopically) is generated and adjusted.

  More specifically, the image processing system generates a parallax image corresponding to each position on the time axis, smooths the generated parallax image in the time axis direction, and provides the entire parallax given to the reproduced image. The parallax image corresponding to each position on the time axis is corrected so that the adjustment amount becomes a value corresponding to the parallax image obtained by the smoothing. The overall parallax adjustment amount includes, for example, the magnitude of the parallax on the flyout side, the parallax range between the parallax on the flyout side and the parallax on the depth side, and the parallax position (or position offset) generated in the parallax image. Including.

  Preferably, the image processing system also uses parallax images obtained by smoothing from several adjacent parallax images for parallax images corresponding to positions on the time axis. This smoothing process is mainly intended to remove noise components and the like that can be included in each parallax image. Therefore, smoothing is performed in a time range different from the smoothing process used for setting the above-described overall parallax adjustment amount. More specifically, this time range is shorter than the time range related to the smoothing process used to set the overall parallax adjustment amount.

  In the following description, for convenience of description, the time range related to the smoothing process for setting the former parallax overall adjustment amount is referred to as a “long-time range”, and the latter parallax image corresponding to each position on the time axis The time range related to the smoothing process for generating the “Short range” is referred to as “short time range”. In a moving image, an image for each time is referred to as a “frame”, and a cycle in which the “frame” is updated is referred to as a “frame interval”. For this reason, the time range related to the smoothing process may be expressed by the number of “frames” used for smoothing or a corresponding multiple of “frame interval”.

  Note that smoothing processing (smoothing processing in the “short-time range”) for generating a parallax image corresponding to each position on the time axis is not essential, and is not performed at all, or an alternative method is adopted. May be.

The image processing system according to the present embodiment adjusts the overall parallax adjustment amount (parallax magnitude, parallax range, and parallax position) based on the result of smoothing the parallax image over a longer period. Even if a subject having a large parallax is reflected, it is possible to prevent the parallax amount of the entire image from changing suddenly due to the subject. When a subject having such a large parallax exists for a long period of time, the parallax amount of the entire image gradually increases. In addition, for a subject that is excessively reproduced on the fly-out side, the jump-out amount is gradually reduced. On the other hand, since the parallax images at each position (each frame) on the time axis are generated by smoothing over a shorter period, even for a moving image in which the parallax changes suddenly, Since the existing parallax information remains, the stereoscopic effect in each frame can be reproduced.

  As described above, the image processing system according to the present embodiment reproduces a stereoscopic display with less burden on the user while using a result obtained by smoothing the parallax images in two types of time ranges while leaving a stereoscopic effect. To do.

[B. System configuration]
First, the configuration of the image processing system according to the embodiment of the present invention will be described.

<< b1: Basic configuration >>
FIG. 1 is a block diagram showing a basic configuration of an image processing system 1 according to the embodiment of the present invention. With reference to FIG. 1, the image processing system 1 includes an imaging unit 2, an image processing unit 3, and a 3D image output unit 4. In the image processing system 1 shown in FIG. 1, the imaging unit 2 captures a subject to acquire a pair of input images (input image 1 and input image 2), and the image processing unit 3 acquires the acquired pair of inputs. By performing image processing to be described later on the image, a stereo image (left-eye image and right-eye image of each frame) for stereoscopically displaying the subject is generated. Then, the 3D image output unit 4 outputs the stereo image (the left eye image and the right eye image) to a display device or the like.

  The imaging unit 2 captures the same object (subject) from different viewpoints and generates a pair of input images. Since the image processing system according to the present embodiment can stereoscopically display a moving image, the imaging unit 2 generates a pair of input images at a predetermined time interval (frame interval). That is, the imaging unit 2 provides the image processing unit 3 with a plurality of input images 1 ordered along the time axis and a plurality of input images 2 ordered along the time axis. Then, the input image 1 and the input image 2 are acquired by imaging the subject from different viewpoints.

  More specifically, the imaging unit 2 is connected to the first camera 21, the second camera 22, an A / D (Analog to Digital) conversion unit 23 connected to the first camera 21, and the second camera 22. A / D converter 24. The A / D conversion unit 23 outputs an input image 1 indicating the subject imaged by the first camera 21, and the A / D conversion unit 24 outputs an input image 2 indicating the subject imaged by the second camera 22. To do.

  That is, the first camera 21 and the A / D conversion unit 23 correspond to an imaging unit that captures a subject and obtains a first input image, and the second camera 22 and the A / D conversion unit 24 are the first camera. 21 corresponds to another imaging unit that captures a subject from a different viewpoint and acquires a second input image.

The first camera 21 includes a lens 21a that is an optical system for imaging a subject, and an imaging element 21b that is a device that converts light collected by the lens 21a into an electrical signal. The A / D converter 23 converts a video signal (analog electrical signal) indicating a subject output from the image sensor 21b into a digital signal and outputs the digital signal. Similarly, the second camera 22 includes a lens 22a that is an optical system for imaging a subject, and an imaging element 22b that is a device that converts light collected by the lens 22a into an electrical signal. The A / D converter 24 converts a video signal (analog electrical signal) indicating a subject output from the image sensor 22b into a digital signal and outputs the digital signal. The imaging unit 2 may further include a control processing circuit for controlling each part.

  As will be described later, in the image processing according to the present embodiment, a stereo image (a left-eye image and a right-eye image) can be generated using only the input image captured by one camera.

  FIG. 2 is a diagram illustrating a specific configuration example of the imaging unit 2 illustrated in FIG. 1. More specifically, FIG. 2 shows an example of the imaging unit 2 including lenses 21a and 22a having the same basic specifications. In this imaging unit 2, an optical zoom function may be mounted on any lens.

  In the image processing method according to the present embodiment, it is only necessary that the line-of-sight directions (viewpoints) of the respective cameras with respect to the same subject are different, so in the imaging unit 2, the arrangement of the lenses 21a and 22a (vertical arrangement or horizontal The direction array can be arbitrarily set. That is, as shown in FIG. 2A, the image may be arranged in the vertically long direction (vertical stereo) or may be picked up in the horizontally long direction (horizontal stereo) as shown in FIG. Also good.

  The image processing system according to the present embodiment stereoscopically displays a subject in time. Therefore, the first camera 21 and the second camera 22 can acquire a series of images ordered along the time axis for each camera by capturing the subject in a predetermined cycle while synchronizing with each other. it can. In the image processing method according to the present embodiment, the input image may be a color image or a monochrome image.

  Referring to FIG. 1 again, the image processing unit 3 performs stereoscopic display of the subject by performing the image processing method according to the present embodiment on the pair of input images acquired by the imaging unit 2. Stereo images (left eye image and right eye image of each frame) are generated. More specifically, the image processing unit 3 includes a corresponding point search unit 31, a parallax image generation unit 32, a smoothing processing unit 33, an overall parallax adjustment amount determination unit 34, a parallax adjustment unit 35, and a 3D image. And a generation unit 36.

  The corresponding point search unit 31 performs a corresponding point search process on a pair of input images (input image 1 and input image 2) of each frame. Typically, the corresponding point search processing includes POC (Phase-Only Correlation) calculation method, SAD (Sum of Absolute Difference) calculation method, SSD (Sum of Squared Difference) calculation method, NCC (Normalized Cross Correlation) calculation method. The method etc. can be used. That is, the corresponding point search unit 31 searches for a corresponding relationship for each point of the subject between the input image 1 and the input image 2.

  The parallax image generation unit 32 acquires distance information about the two input images. This distance information is calculated based on the difference in information about the same subject. Typically, the parallax image generation unit 32 calculates distance information from a correspondence relationship between input images for each point of the subject searched by the corresponding point search unit 31. The imaging unit 2 images the subject from different viewpoints. Therefore, between two input images, a pixel representing a certain point (attention point) of the subject is shifted by a distance corresponding to the distance between the imaging unit 2 and the point of the subject. In this specification, the difference between the coordinates on the image coordinate system of the pixel corresponding to the target point of the input image 1 and the coordinates on the image coordinate system of the pixel corresponding to the target point of the input image 2 is referred to as “parallax”. Called. The parallax image generation unit 32 calculates the parallax for each point of interest of the subject searched by the corresponding point search unit 31.

This parallax is an index value indicating the distance from the imaging unit 2 to the corresponding point of interest of the subject. The larger the parallax, the shorter the distance from the imaging unit 2 to the corresponding point of interest of the subject, that is, the closer the imaging unit 2 is. In this specification, the term “distance information” is used as a general term for the parallax and the distance from the imaging unit 2 of each point of the subject indicated by the parallax.

  Note that the direction in which the parallax occurs between the input images depends on the positional relationship between the first camera 21 and the second camera 22 in the imaging unit 2. For example, when the first camera 21 and the second camera 22 are arranged at a predetermined interval in the vertical direction, the parallax between the input image 1 and the input image 2 is generated in the vertical direction.

  The parallax image generation unit 32 calculates the distance information about each point of the subject, and generates a parallax image (also referred to as “distance image”) in which the calculated distance information is associated with coordinates on the image coordinate system. To do.

  The smoothing processing unit 33 performs a smoothing process (smoothing process) on the parallax image generated by the parallax image generation unit 32. While performing spatial smoothing, as described above, in the present embodiment, the parallax images are each smoothed in two types of time ranges. The smoothing processing unit 33 performs smoothing processing for each of the two types of frames.

  The overall parallax adjustment amount determination unit 34 determines the overall parallax adjustment amount (the size of the parallax, the parallax range, and the parallax position) based on the result smoothed by the smoothing processing unit 33.

  The parallax adjustment unit 35 corrects the parallax images (preferably parallax images smoothed in a short time range) related to each frame according to the total parallax adjustment amount determined by the total parallax adjustment amount determination unit 34.

  Details of processes executed in the smoothing processing unit 33, the overall parallax adjustment amount determination unit 34, and the parallax adjustment unit 35 will be described later.

  The 3D image generation unit 36 shifts each pixel constituting the input image by the corresponding distance information (number of pixels) based on the parallax image after adjustment by the parallax adjustment unit 35 to display the subject stereoscopically. Stereo images (left eye image and right eye image of each frame) are generated. In this manner, a stereo image for stereoscopically displaying the subject is generated by shifting the pixels included in the input image in the horizontal direction based on the distance information. When viewed between the image for the left eye and the image for the right eye, each point of the subject is separated by a distance corresponding to the distance information (number of pixels) indicated by the parallax image, that is, according to the distance information (number of pixels). It is expressed with given parallax. As a result, the subject can be stereoscopically displayed.

  The 3D image output unit 4 outputs a stereo image (the image for the left eye and the image for the right eye of each frame) generated by the image processing unit 3 to a display device or the like.

  Although the image processing system 1 shown in FIG. 1 can be configured independently of each other, in general, the image processing system 1 is often embodied as a digital camera or a personal computer described below. Therefore, an implementation example of the image processing system 1 according to the present embodiment will be described.

<< b2: Embodiment 1 >>
FIG. 3 is a block diagram showing a configuration of a digital camera 100 that embodies the image processing system 1 shown in FIG. A digital camera 100 illustrated in FIG. 3 includes two cameras (a first camera 121 and a second camera 122), and can capture a stereo image for stereoscopically displaying a subject. In FIG. 3, components corresponding to the respective blocks constituting the image processing system 1 shown in FIG. 1 are denoted by the same reference numerals as in FIG.

  In the digital camera 100, an input image acquired by imaging a subject with the first camera 121 is stored and output, and an input image acquired by imaging the subject with the second camera 122 is mainly described above. Are used for the corresponding point search process and the parallax image generation process. Therefore, it is assumed that only the first camera 121 has an optical zoom function.

  Referring to FIG. 3, a digital camera 100 includes a CPU (Central Processing Unit) 102, a digital processing circuit 104, an image display unit 108, a card interface (I / F) 110, a storage unit 112, and a zoom mechanism. 114, a first camera 121, and a second camera 122.

  The CPU 102 controls the entire digital camera 100 by executing a program (including an image processing program) stored in advance. The digital processing circuit 104 executes various digital processes including image processing according to the present embodiment. The digital processing circuit 104 is typically configured by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an LSI (Large Scale Integration), an FPGA (Field-Programmable Gate Array), or the like. The digital processing circuit 104 includes an image processing circuit 106 for realizing the functions provided by the image processing unit 3 shown in FIG.

  The image display unit 108 includes an image provided by the first camera 121 and / or the second camera 122, an image generated by the digital processing circuit 104 (image processing circuit 106), various setting information related to the digital camera 100, and A control GUI (Graphical User Interface) screen or the like is displayed. It is preferable that the image display unit 108 can stereoscopically display the subject using a stereo image generated by the image processing circuit 106. In this case, the image display unit 108 is configured by an arbitrary display device (liquid crystal display device for three-dimensional display) corresponding to the three-dimensional display method. As such a three-dimensional display method, a parallax barrier method or the like can be employed. In this parallax barrier method, by providing a parallax barrier on the liquid crystal display surface, the right eye image can be visually recognized by the user's right eye, and the left eye image can be visually recognized by the user's left eye. Alternatively, a shutter glasses method may be adopted. In this shutter glasses method, the left eye image and the right eye image are alternately switched at high speed and displayed, and the user wears special glasses equipped with a shutter that opens and closes in synchronization with the switching of the image. , You can enjoy stereoscopic display.

  A card interface (I / F) 110 is an interface for writing image data generated by the image processing circuit 106 to the storage unit 112 or reading image data from the storage unit 112. The storage unit 112 is a storage device that stores image data generated by the image processing circuit 106 and various information (setting values such as control parameters and operation modes of the digital camera 100). The storage unit 112 includes a flash memory, an optical disk, a magnetic disk, and the like, and stores data in a nonvolatile manner.

  The zoom mechanism 114 is a mechanism that changes the imaging magnification of the first camera 121 according to a user operation or the like. The zoom mechanism 114 typically includes a servo motor and the like, and drives the lens group constituting the first camera 121 to change the focal length.

The first camera 121 generates an input image for generating a stereo image by imaging a subject. The first camera 121 includes a plurality of lens groups that are driven by the zoom mechanism 114. The second camera 122 is used for corresponding point search processing and parallax image generation processing, which will be described later, and images the same subject imaged by the first camera 121 from different viewpoints.

  As described above, the digital camera 100 shown in FIG. 3 is obtained by mounting the entire image processing system 1 according to the present embodiment as a single device. That is, the user can visually recognize the subject in a three-dimensional manner on the image display unit 108 by imaging the subject using the digital camera 100.

<< b3: Implementation Example 2 >>
FIG. 4 is a block diagram showing a configuration of a personal computer 200 that embodies the image processing system 1 shown in FIG. In the personal computer 200 shown in FIG. 4, the imaging unit 2 for acquiring a pair of input images is not mounted, and a pair of input images (an input image 1 and an input image 2) acquired by an arbitrary imaging unit 2. Is input from the outside. Even such a configuration can be included in the image processing system 1 according to the embodiment. In FIG. 4 as well, components corresponding to the respective blocks constituting the image processing system 1 shown in FIG. 1 are denoted by the same reference numerals as in FIG.

  Referring to FIG. 4, personal computer 200 includes a personal computer main body 202, a monitor 206, a mouse 208, a keyboard 210, and an external storage device 212.

  The personal computer main body 202 is typically a general-purpose computer according to a general-purpose architecture, and includes a CPU, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as basic components. The personal computer main body 202 can execute an image processing program 204 for realizing a function provided by the image processing unit 3 shown in FIG. Such an image processing program 204 is stored in a storage medium such as a CD-ROM (Compact Disk-Read Only Memory) and distributed, or distributed from a server device via a network. The image processing program 204 is stored in a storage area such as a hard disk of the personal computer main body 202.

  Such an image processing program 204 implements processing by calling necessary modules among program modules provided as part of an operating system (OS) executed by the personal computer main body 202 at a predetermined timing and order. It may be configured as follows. In this case, the image processing program 204 itself does not include a module provided by the OS, and image processing is realized in cooperation with the OS. Further, the image processing program 204 may be provided by being incorporated in a part of some program instead of a single program. Even in such a case, the image processing program 204 itself does not include a module that is commonly used in the program, and image processing is realized in cooperation with the program. Even such an image processing program 204 that does not include some modules does not depart from the spirit of the image processing system 1 according to the present embodiment.

  Of course, part or all of the functions provided by the image processing program 204 may be realized by dedicated hardware.

The monitor 206 displays a GUI screen provided by an operating system (OS), an image generated by the image processing program 204, and the like. As with the image display unit 108 shown in FIG. 3, the monitor 206 is preferably capable of stereoscopically displaying a subject using a stereo image generated by the image processing program 204. In this case, the monitor 206 is configured by a display device such as a parallax barrier method or a shutter glasses method, as described in the image display unit 108.

  The mouse 208 and the keyboard 210 each accept a user operation and output the contents of the accepted user operation to the personal computer main body 202.

  The external storage device 212 stores a pair of input images (an input image 1 and an input image 2) obtained by some method, and outputs the pair of input images to the personal computer main body 202. As the external storage device 212, a device that stores data in a nonvolatile manner such as a flash memory, an optical disk, or a magnetic disk is used.

  As described above, personal computer 200 shown in FIG. 4 is obtained by mounting a part of image processing system 1 according to the present embodiment as a single device. By using such a personal computer 200, the user stereoscopically displays the subject from a pair of input images acquired by imaging the subject from different viewpoints using an arbitrary imaging unit (stereo camera). Therefore, a stereo image (a left-eye image and a right-eye image) can be generated. Furthermore, by displaying the generated stereo image on the monitor 206, stereoscopic display can be enjoyed.

[C. Embodiment 1]
A configuration for generating a stereo image (a left-eye image and a right-eye image) from one input image will be described as Embodiment 1 of the present invention. More specifically, parallax images are generated and adjusted (corrected) using a pair of input images of each frame, and the other input image is generated from one input image using the parallax image obtained as a result. . In the following description, an example in which a stereoscopic image (stereo image) is generated using the input image 1 is shown. That is, in the present embodiment, a stereoscopic image is generated using the input image 1 and an image generated by giving parallax according to the corrected parallax image corresponding to the input image 1.

  As described above, in the present embodiment, by performing smoothing processing in two types of time ranges (frame intervals), a stereoscopic display with less burden on the user is reproduced while leaving a sensation.

  That is, it is difficult to see the stereoscopic video if the parallax to be reproduced is too large, while if it is too small, the stereoscopic effect looks unsatisfactory. Therefore, the information processing system according to the present embodiment, for example, when a subject existing on the pop-out side is imaged from the middle of the scene, the amount of pop-out for the subject while retaining the parallax information as a stereoscopic effect. The overall parallax adjustment is performed so as to gradually change.

<< c1: Overall processing procedure >>
First, the overall processing procedure in the present embodiment will be described.

  FIG. 5 is a flowchart showing an overall processing procedure according to the first embodiment of the present invention. Referring to FIG. 5, input image 1 and input image 2 of each frame are acquired. Typically, the first camera 21 (the lens 21a and the image sensor 21b) captures the subject and generates the input image 1. The second camera 22 (the lens 22a and the image sensor 22b) captures the subject to generate the input image 2. That is, a plurality of input images 1 ordered along the time axis and a plurality of input images 2 ordered along the time axis are acquired.

First, a parallax image for the input image 1 is generated for each frame (step S1). That is, a parallax image is generated from the input image 1 and the input image 2 corresponding to each position on the time axis. This parallax image generation processing is mainly executed by the corresponding point search unit 31 and the parallax image generation unit 32 (FIG. 1) of the image processing unit 3.

  Subsequently, a smoothing process is performed on the generated parallax image of each frame (step S2). This smoothing process is mainly executed by the smoothing processing unit 33 (FIG. 1) of the image processing unit 3.

  The smoothing process in step S2 is performed by smoothing each frame in a parallax image (spatial smoothing), smoothing a parallax image of a plurality of frames over a short time range (time smoothing), and performing a plurality of frames over a long time range. Includes smoothing for parallax images. That is, it includes a process of smoothing a plurality of parallax images in the time axis direction over a certain time range.

  Subsequently, based on the result of smoothing the parallax images of a plurality of frames over a long time range, an adjustment process of the overall parallax adjustment amount is executed (step S3). This adjustment process of the overall parallax adjustment amount is mainly executed by the overall parallax adjustment amount determination unit 34 (FIG. 1) of the image processing unit 3.

  Subsequently, step S3 is performed on the generated parallax image by smoothing within the parallax image of each frame (spatial smoothing) and by smoothing the parallax images of a plurality of frames over a short time range (temporal smoothing). The process of adjusting the parallax is executed according to the overall parallax adjustment amount determined in (Step S4). This parallax adjustment processing is mainly executed by the parallax adjustment unit 35 (FIG. 1) of the image processing unit 3.

  Finally, a stereo image is generated from the input image 1 using the parallax image adjusted in step S4 (step S5). More specifically, the input image 1 is output as it is as a left-eye image, and the right-eye image is output by giving parallax to the input image 1 using the adjusted parallax image.

  The generated stereo image (left eye image and right eye image of each frame) is output to a monitor or the like.

Hereinafter, processing in each step will be described in detail.
<< c2: Parallax image generation process >>
Details of the parallax image generation processing in step S1 will be described.

  First, the correspondence of the positional relationship between a pair of input images (input image 1 and input image 2) in each frame is searched. This corresponding point search processing is executed by the corresponding point search unit 31 shown in FIG. More specifically, in the corresponding point search process, the other input image pixel (coordinate value) corresponding to the target point of one input image is specified. For such a corresponding point search process, a matching process using a POC calculation method, an SAD calculation method, an SSD calculation method, an NCC calculation method, or the like is used.

  In the corresponding point search process, one input image is set as a reference image, and the other input image is set as a reference image, and the images are associated with each other. Depending on which input image is mainly used, the input image set as the reference image is changed.

Subsequently, based on the correspondence between the target point identified by the corresponding point search process and the corresponding point, generation of a parallax image for generating a parallax image indicating distance information associated with the coordinates of each point of the subject Processing is executed. This parallax image generation processing is executed by the parallax image generation unit 32 shown in FIG. In this parallax image generation process, for each point of interest, the difference (parallax) between the coordinates of the point of interest in the image coordinate system of the input image 1 and the coordinates of the corresponding point in the image coordinate system of the input image 2 is calculated. .

  In the present embodiment, since the input image 1 is mainly used, the parallax calculated in association with the coordinates of the target point of the corresponding input image 1 is stored. Note that, when the input image 2 is mainly used, parallax calculated in association with the coordinates of the target point of the corresponding input image 2 may be stored. As the distance information, the coordinates on the input image 1 or the input image 2 and the corresponding parallax are associated with each attention point searched by the corresponding point search process. By arranging this distance information in association with the pixel arrangement of the input image 1 or the input image 2, a parallax image representing the parallax of each point is generated corresponding to the image coordinate system of the input image 1 or the input image 2. .

  In addition, as such a corresponding point search process and a parallax image generation process, you may employ | adopt the method described in Unexamined-Japanese-Patent No. 2008-216127. Japanese Patent Laid-Open No. 2008-216127 discloses a method for calculating parallax (distance information) with subpixel granularity, but parallax (distance information) may be calculated with pixel granularity. .

  When the input image is a color image such as RGB, the corresponding point search process may be performed after conversion to a gray image.

  FIG. 6 is a diagram illustrating an example of an input image in a certain frame imaged by the imaging unit 2 illustrated in FIG. 1. 6A shows the input image 1 captured by the first camera 21, and FIG. 6B shows the input image 2 captured by the second camera 22. The example shown in FIG. 6 is taken in a configuration in which two lenses are arranged at a predetermined interval in the lateral direction.

  FIG. 7 is a diagram showing an example of the parallax image generated from the pair of input images shown in FIG. 6 according to the image processing method according to the first embodiment. In this example, an example of a parallax image (distance image) when the input image 1 shown in FIG. 6A is set as a standard image and the input image 2 shown in FIG. 6B is set as a reference image is shown.

  As shown in FIG. 7, the magnitude of the parallax (distance information) associated with each point of the input image 1 is expressed by the shade of the corresponding point.

  In the above-described corresponding point search process and parallax image generation process, the attention point and the corresponding point are specified by performing a correlation calculation, and therefore the corresponding point is searched for each unit region having a predetermined pixel size. FIG. 7 shows an example in which the corresponding point search is executed for each unit region of 48 pixels × 48 pixels. That is, in FIG. 7, both the X axis (horizontal direction) and the Y axis (vertical direction) are searched for corresponding points for each unit region defined by a 48-pixel interval, and between the searched corresponding points. The calculation result of distance is shown. The parallax image indicating the distance between the searched corresponding points is generated so as to match the pixel size of the input image.

  Note that the region corresponding to 48 pixels (search window) on the outermost periphery of the input image may be erroneously determined that there is no corresponding point, so the corresponding point search process is not performed and the closest position is determined. The pixel distance (parallax) data is used instead. That is, for the region for 48 pixels at the outermost periphery of the input image, the value of the pixel at the position inside 48 pixels from the outermost periphery is used.

<< c3: Smoothing process >>
Next, details of the smoothing process in step S2 will be described. As described above, in the present embodiment, the smoothing processing is performed by (1) smoothing within the parallax image of each frame (spatial smoothing) and (2) smoothing of the parallax images of a plurality of frames over a short time range. (Time smoothing), (
3) It includes smoothing for parallax images of multiple frames over a long time range. Hereinafter, the content of each smoothing is demonstrated.

(C3.1: Smoothing within the parallax image of each frame (spatial smoothing))
First, spatial smoothing is performed on the parallax image of each frame. As an embodiment of such a spatial smoothing process, there is a method using a two-dimensional filter of a predetermined size.

  FIG. 8 is a diagram showing an example of the averaging filter used for the spatial smoothing process according to the first embodiment of the present invention. In this spatial smoothing process, for example, an averaging filter of 145 pixels × 145 pixels as shown in FIG. 8 is applied. In the averaging filter, an average value of the pixel values (parallax) of the parallax images included in the range of the vertical direction 145 pixels and the horizontal direction 145 pixels centered on the target pixel is calculated as a new pixel value of the target pixel. More specifically, a new pixel value of the pixel of interest is calculated by dividing the sum of the pixel values of the pixels included in the filter by the pixel size of the filter.

  Instead of taking the operation of all the pixels included in the filter, an average value of pixels extracted by thinning out at predetermined intervals (for example, 12 pixels) may be used. Even when such a thinning process is performed, a smoothing result similar to the case where the average value of all pixels is used may be obtained. In such a case, the processing is performed by performing the thinning process. The amount can be reduced.

  Further, for spatial smoothing, different means such as an averaging filter or a median filter in which the pixel of interest is weighted may be employed. In order to optimize the level of the parallax image obtained by smoothing, the filter size and the like may be adjusted including the number of frames used for smoothing.

  Note that the pixel size of the parallax image obtained by the smoothing process is preferably the same pixel size as the input image. By making the pixel sizes the same, it is possible to determine the distance of each pixel on a one-to-one basis in a stereo image generation process described later.

(C3.2: temporal smoothing)
FIG. 9 is a diagram for explaining temporal smoothing processing according to the first embodiment of the present invention. Referring to FIG. 9, in the present embodiment, results obtained by smoothing parallax images in two types of time ranges are used.

  More specifically, for a long time range (A frame) including a certain processing target frame, the parallax maximum value (MAX) and the parallax minimum value (MIN) in the parallax image of each frame included in the range are smooth. It becomes. That is, for each of the parallax images for A frames, the maximum value and the minimum value of the parallax are respectively extracted, and then an average value is calculated between all the extracted maximum values, and all the extracted An average value is calculated between the minimum values. The calculated average value of the maximum value and the minimum value of the parallax is temporarily stored in association with the processing target frame. The average value of the maximum value and the minimum value of the parallax is used to determine the overall parallax adjustment amount.

Further, a short time range (for B frames) including a certain processing target frame is smoothed between parallax images of a plurality of frames included in the range. That is, for each pixel constituting the parallax image, the average value of the parallax amounts in the plurality of parallax images for the corresponding pixel position is calculated. In other words, smoothing is performed by adding and averaging the pixel values of predetermined frames before and after in the time axis direction. This smoothing process in a short time range is performed in order to remove noise generated in the parallax image. That is, it is preferable to reflect local parallax temporal changes in the parallax image as much as possible, so the number of frames in the short-time range is set based on such intention.

In order to speed up the smoothing process, for example, thinning may be performed every 6 frames.
The smoothing over a long time range (A frame) shown in FIG. 9 is preferably 10 seconds or more experimentally. For example, when 30 frames are updated per second, the long time range can be set to a total of 1081 frames (36 seconds) including the frames to be processed. Further, the smoothing over a long range (for B frames) is preferably 10 seconds or less experimentally. More preferably, it is 5 seconds or less. For example, when 30 frames are updated per second, the short time range can be set to a total of 61 frames (2 seconds) including the frames to be processed.

  When the frame rates are different, the number of frames required for the length of the time range may be calculated.

  As described above, the smoothing process in step S2 includes a process of smoothing a plurality of parallax images in the time axis direction over a short time range (for B frames) different from the long time range (for A frames).

  FIG. 10 is a diagram illustrating an example of the smoothed parallax image obtained by the spatial and temporal smoothing processing on the parallax image illustrated in FIG. 7. In the smoothed parallax image shown in FIG. 10, it can be seen that the pixel value (parallax) does not change greatly between adjacent pixels.

  The usage form of the result obtained by smoothing over a long time range (A frame) will be described later.

  Further, for time smoothing, different means such as averaging with weights applied to frames to be processed and replacement with median values may be employed. In order to optimize the level of the parallax image obtained by smoothing, the number of frames used for smoothing may be adjusted.

<< c4: Overall parallax adjustment amount determination process >>
Next, details of the overall parallax adjustment amount determination process in step S3 will be described. The overall parallax adjustment amount is determined based on the result of smoothing over a long time range (A frame) shown in FIG. That is, the average value (AVE_MAX) of the maximum parallax value (MAX) and the average value (AVE_MIN) of the minimum parallax value (MIN) in the long-time range (A frame) including the processing target frame are used. More specifically, the average values AVE_MAX and AVE_MIN are obtained by averaging the sum of the maximum value (MAX) and the minimum value (MIN) in the long-time range (in this example, for 1081 frames before and after) as shown in FIG. It is a value obtained by (smoothing).

  This overall parallax adjustment amount is for adjusting the parallax so that stereoscopic display can be performed with a more comfortable parallax amount. In the present embodiment, as the overall parallax adjustment amount, the magnitude of the parallax on the flyout side, the parallax range between the parallax on the flyout side and the parallax on the depth side, and the parallax position (or position) generated in the parallax image Offset).

  FIG. 11 is a diagram for explaining the overall parallax adjustment amount determination process according to the first embodiment of the present invention. Referring to FIG. 11, scaling is performed so that the pixel value (parallax amount) of the parallax image after the smoothing process is within a predetermined parallax range (target parallax range). That is, the inclination and the parallax offset of the parallax adjustment function shown in FIG. 11 are determined.

  The parallax range (target parallax range) corresponds to the difference between the parallax on the flyout side and the parallax on the depth side. The target parallax range r is determined empirically based on the width of the input image. At this time, the allowable value on the flying side and the allowable value on the depth side are also determined empirically.

  The target parallax range r may be determined dynamically or a fixed value set in advance may be used. For example, for an input image of 1280 pixels × 760 pixels, for example, the value “22” determined empirically is used to calculate as follows.

Target parallax range r = horizontal image size / 22 = 1280/22 = 58
Then, the parallax increase / decrease coefficient c, which is the inclination of the parallax adjustment function, and the parallax adjustment function so that the difference between the parallax in the most projected state and the parallax in the deepest side is the target parallax range r. A parallax offset o that is an intercept is calculated. More specifically, the parallax increase / decrease coefficient c and the parallax offset o are calculated according to the following equations using the average value of the maximum parallax value (AVE_MAX) and the average value of the minimum parallax value (AVE_MIN).

Parallax increase / decrease coefficient c = target parallax range r / (average value AVE_MAX−average value AVE_MIN)
Parallax offset o = (average value AVE_MAX + average value AVE_MIN) / 2
Post-adjustment parallax (distance) amount = parallax increase / decrease coefficient c × (pre-adjustment parallax (distance) amount−parallax offset o)
For example, the allowable value on the fly-out side is “29.0”, the allowable value on the depth side is “−29.0”, and the average value (AVE_MAX) of the maximum parallax value is “12.7”. When the average value (AVE_MIN) of the minimum parallax is “−14.56”, the parallax increase / decrease coefficient c is calculated as “2.13”, and the parallax offset o is “−0.93”. Is calculated.

  As described above, the overall parallax adjustment amount determination process in step S3 is performed by setting the size of the parallax on the fly-out side for the long time range (A) for the parallax image of the processing target frame existing in the long time range (A frame). And a process of correcting to a value corresponding to the result of smoothing over the frame). Also, the overall parallax adjustment amount determination process in step S3 is performed on the parallax range between the projection-side parallax and the depth-side parallax with respect to the parallax image existing in the long-time range (A frame). A process of limiting to a value corresponding to the result of smoothing over A frame).

  The overall parallax adjustment amount determination process includes a process for determining a parallax increase / decrease coefficient that is a change amount along the time axis of the parallax generated in the parallax image, and a process for determining an offset of the parallax position generated in the parallax image. In the overall parallax adjustment amount determination process, the minimum parallax value is limited according to the result of smoothing over a long time range (for A frame).

<< c5: Parallax adjustment processing >>
Next, details of the parallax adjustment processing in step S4 will be described. As described above, when the overall parallax adjustment amount is determined, the parallax adjustment processing is performed on the parallax image for the processing target frame in accordance with the parallax adjustment function illustrated in FIG. 11. That is, the pixel value (pre-adjustment parallax amount) of each pixel constituting the parallax image is input to the parallax adjustment function, and the output value is determined as the post-adjustment parallax amount.

  FIG. 12 is a diagram illustrating an example of a parallax image that has been parallax adjusted with respect to the parallax image after smoothing illustrated in FIG. 10.

  As described above, the parallax adjustment processing in step S4 is based on the result of smoothing over a long time range (for A frame) and the parallax image obtained by smoothing over a short time range (for B frame). Including a process of correcting the parallax image.

<< c6: Stereoscopic image generation process >>
Next, details of the stereoscopic image generation processing in step S5 will be described. As described above, when a parallax image in each frame is generated by the smoothing process and the parallax adjustment process, a stereoscopic image is generated.

  In the present embodiment, as an example, the input image 1 is used as it is as a left-eye image, and a right-eye image is generated by giving parallax to the input image 1 based on the parallax image. More specifically, the right-eye image is generated by shifting the position of each pixel of the input image 1 according to the coordinate value (distance, parallax) of the corresponding pixel of the parallax image.

  In order to stereoscopically display the subject, it is only necessary that the corresponding pixels be separated by a specified distance (parallax) between the left-eye image and the right-eye image. And right eye images may be generated.

  FIG. 13 is a diagram for describing a stereoscopic image generation process according to the first embodiment of the present invention. FIG. 13 shows a processing example when the input image 1 is used proactively. FIG. 14 is a flowchart showing a processing procedure of the stereoscopic image generation processing shown in FIG.

  Referring to FIG. 13, in the stereoscopic image generation process, a stereo image (left-eye image and right-eye image) of each frame is generated from an input image that is mainly used based on a parallax image.

  More specifically, the other right-eye image or left-eye image is generated by shifting the pixel position in units of lines constituting the input image that is used mainly. FIG. 13 shows an example in which the input image 1 is used as it is as the left-eye image and a right-eye image is generated from the input image 1. FIG. 13 shows 10 pixels whose pixel positions (coordinates) are “101”, “102”,..., “110” for a line of the input image 1 used as the left-eye image. Yes. The distance (parallax) corresponding to each pixel position is “40”, “40”, “41”, “41”, “41”, “42”, “42”, “41”, “40”, “40”, respectively. ”. Using these pieces of information, the shifted pixel position (coordinates in the right-eye image) is calculated for each pixel. More specifically, the pixel position after shifting for each pixel for one line is calculated according to (pixel position after shifting) = (coordinates in the image for the left eye) − (corresponding distance (parallax)). .

  Then, a corresponding one line image of the right-eye image is generated based on each pixel value and the corresponding shifted pixel position. At this time, there may be no corresponding pixel depending on the value of the distance (parallax). In the example illustrated in FIG. 13, there is no information on the pixels at the pixel positions “66” and “68” of the right-eye image. In such a case, the pixel values of the deficient pixels are interpolated using information from adjacent pixels.

  By repeating such processing for all lines included in the input image, the right-eye image is generated.

  Note that the direction in which the pixel position is shifted is a direction in which parallax should be generated, and specifically corresponds to a direction that becomes the horizontal direction when displayed toward the user.

Such a processing procedure is shown in FIG. That is, with reference to FIG. 14, the 3D image generation unit 36 (FIG. 1) calculates each shifted pixel position for one line of pixels of the input image 1 (step S <b> 3601). Subsequently, the 3D image generation unit 36 generates an image for one line (right eye image) from the shifted pixel position calculated in step S1 (step S3602).

  Thereafter, the 3D image generation unit 36 (FIG. 1) determines whether or not there is an unprocessed line in the input image (step S3603). If there is an unprocessed line in the input image (YES in step S3603), the next line is selected, and the processes in steps S3601 and S1402 are repeated.

  If the processing has been completed for all lines of the input image (NO in step S3603), the 3D image generation unit 36 outputs the generated right-eye image together with the input image 1 (left-eye image). Then, the process ends.

  13 and 14 describe the processing when the input image 1 is determined as the output image, that is, when the input image 1 is mainly used. However, when the input image 2 is determined as the output image, That is, the same applies to the processing when the input image 2 is mainly used. However, in the above-described corresponding point search process, the relationship between the base image and the reference image is switched.

  FIG. 15 is a diagram illustrating an example of a stereo image generated using the input image 1 and the input image 2 illustrated in FIG. 6.

  As described above, the stereoscopic image generation processing in step S5 is performed by giving parallax according to the input image 1 for each frame and the corrected parallax image corresponding to the input image 1 for each frame. And a process of generating a stereoscopic image using the image.

<< c7: Application example >>
An application example of the image processing method according to the above-described embodiment will be described.

  FIG. 16 is a diagram showing an application example of the image processing method according to the first embodiment of the present invention. FIG. 17 is a diagram showing temporal changes in parameters corresponding to the application example shown in FIG.

  As shown in (1) input images in FIGS. 16A to 16C, consider a case where the imaging unit 2 is fixed and a vehicle traveling on a road is imaged over a predetermined period. In such a situation, the amount of parallax changes relatively frequently due to the relationship between the building on the far side and the vehicle passing on the near side. That is, when an object (vehicle) passes near the imaging unit 2, the amount of parallax at the position corresponding to the object is relatively large, as shown in (2) parallax images in FIGS. You can see it grows.

  On the other hand, as shown in (3) parallax images after parallax adjustment in FIGS. 16A to 16C, smoothing processing and parallax adjustment processing according to the present embodiment are performed, so that the entire image is obtained. A parallax image reflecting information on the parallax change corresponding to the passage of the vehicle can be generated while the parallax change is gently maintained.

  As shown in FIG. 17A, by performing smoothing over a long time range, the parallax maximum value and minimum value (thick solid line) in the parallax image generated from the input image vary relatively greatly. However, fluctuations in the parallax maximum value and the minimum value (thin solid line) in the parallax image after parallax adjustment are suppressed relatively moderately. Then, the parallax in the parallax image after parallax adjustment is adjusted to be within a predetermined parallax range (target parallax range) (thick broken line).

  A portion indicated by an elliptical broken line in FIG. 17A indicates a period during which the vehicle shown in FIG. 16 passes. In this way, even if the parallax included in the parallax image is changing suddenly due to the movement of the subject, the parallax is smoothed over a long period of time, so that it is output in accordance with the sudden change in parallax. The parallax of the stereo image does not increase or decrease rapidly. Therefore, there is no sense of incongruity due to a sudden change in the overall parallax of a stereoscopically displayed video. This can also be seen from the fact that the values of the parallax increase / decrease coefficient c shown in FIG. 17B and the parallax offset o shown in FIG.

  On the other hand, as shown in FIG. 16C, by smoothing the parallax image over a short time range, the parallax information (stereoscopic information) in each frame remains, and appropriate stereoscopic vision in each frame remains. Can provide.

  In addition, when the direction of the imaging unit 2 is moved, for example, when a nearby utility pole is continuously imaged, the change is reflected even if smoothed over a long period of time, so that it jumped out The parallax (electric pole) is gradually suppressed to the back side.

<< c8: Modification >>
(C8.1: Limit of parallax increase / decrease coefficient)
The parallax increase / decrease coefficient c that determines the overall parallax adjustment amount may cause side effects such as increasing the parallax adjustment error even if it is too large or too small. You may restrict | limit so that it may fit in. For example, the parallax increase / decrease coefficient c may be limited to be between 0.5 and 3.0.

  That is, the adjustment process (step S3) of the overall parallax adjustment amount may include a process of determining a parallax increase / decrease coefficient that is a change amount along the time axis of the parallax generated in the parallax image.

  FIG. 18 is a diagram showing temporal changes of the parameters shown in FIG. 17 when the parallax increase / decrease coefficient c is limited. When the parallax increase / decrease coefficient c is limited, as shown in FIG. 18B, clipping is performed with the limit value, and as shown in FIG. 18A, the parallax range is prevented from changing excessively. it can.

(C8.2: smoothing process in time direction)
The smoothing process in the time direction according to the present embodiment is performed on frames included in the long-time range and the short-time range including the processing target frame.

  FIG. 9 shows an example in which smoothing is performed using past and future frames centering on a processing target frame assuming offline processing. In the case of real-time processing, a past frame is processed from a processing target frame. May be used for smoothing.

(C8.3: Clip processing)
In the smoothing process over a long time range, clip processing may be performed in order to suppress a sudden change in parallax. That is, when the change in the time axis direction of the parallax maximum value and minimum value in the parallax image generated from the input image exceeds a predetermined threshold value, the change may be limited. For example, the maximum and minimum values of disparity in the processing target frame are compared with the maximum and minimum values of disparity one frame before, or the maximum and minimum values of disparity in the processing target frame and smoothing one frame before The maximum value and the minimum value of the later parallax are compared, and a portion that changes beyond a predetermined value (for example, 30) may be clipped to a predetermined value in order to suppress a sudden change.

That is, in the smoothing process, the maximum or minimum value of the parallax image corresponding to a certain position on the time axis is compared with the maximum value or minimum value of the parallax image corresponding to a close position on the time axis. For pixels that change beyond a predetermined threshold value, a process of limiting the maximum or minimum value of the parallax to the predetermined threshold value may be included.

(C8.4: Noise removal processing)
In smoothing processing over a short time range, noise removal processing based on the amount of change in the time axis direction may be performed. For example, the parallax image of the processing target frame is compared with the parallax image of the previous frame, or the parallax image of the processing target frame is compared with the parallax image after the smoothing of the previous frame, and the difference is a predetermined value ( For example, a portion exceeding 30) (for example, the region 300 in FIG. 7) may be determined as noise. The portion determined as noise in this way may be replaced using the corresponding portion of the parallax image one frame before the comparison or the parallax image after smoothing one frame before.

  That is, the smoothing process is performed at a predetermined threshold compared to the parallax indicated by the pixels included in the smoothed parallax image among the parallaxes indicated by the pixels included in the parallax image corresponding to a certain position on the time axis. For a pixel that has changed beyond the value, a process of limiting the corresponding parallax magnitude to the predetermined threshold value may be included.

(C8.5: smoothing only the maximum value of parallax)
As for the overall parallax adjustment amount, since the change in parallax is likely to be a big problem on the flyout side, only the parallax on the flyout side may be smoothed. That is, in the smoothing process over a long time range, only the parallax maximum value (MAX) in the parallax image may be smoothed. In this case, the minimum value (MIN) of the parallax can be a constant value assuming infinity.

(C8.6: other)
You may combine suitably the some process shown in the above-mentioned modification.

[D. Second Embodiment]
In the first embodiment described above, the processing example (one-frame mode) for generating a stereoscopic video from one input image of the pair of input images has been described. On the other hand, in the second embodiment, a processing example (two-frame mode) for generating a stereoscopic moving image by performing parallax correction on a pair of input images (stereo images) will be described.

  Since the content of the basic processing is the same as that of the first embodiment, processing different from that of the first embodiment will be mainly described. That is, in the second embodiment, a pair of images (left-eye image and right-eye image) obtained by imaging a subject with a stereo camera or the like are input as input images. The left-eye image is subjected to parallax adjustment based on the left-eye parallax image, so that the left-eye image after parallax adjustment is output, and the right-eye image is converted into a right-eye parallax image. The parallax adjustment is performed based on this, so that the right-eye image after the parallax adjustment is output.

  FIG. 19 is a flowchart showing an overall processing procedure according to the second embodiment of the present invention. Referring to FIG. 19, when input image 1 (left eye image) and input image 2 (right eye image) of each frame are acquired, a parallax image for input image 1 is generated for each frame (see FIG. 19). Along with step S11), a parallax image for the input image 2 is generated for each frame (step S12). These processes are basically the same as step S1 in FIG. 5, but the images serving as the reference for calculating the parallax are different from each other.

  Subsequently, a smoothing process is performed on each generated parallax image (steps S21 and S22). This smoothing process is mainly executed by the smoothing processing unit 33 (FIG. 1) of the image processing unit 3.

  Subsequently, an overall parallax adjustment amount determination process is executed based on the result of smoothing the parallax images of a plurality of frames over a long time range (step S30). The determination process of the overall parallax adjustment amount is basically the same as that of the first embodiment, but differs in the following points.

  That is, for the parallax increase / decrease coefficient c and the parallax offset o included in the overall parallax adjustment amount, there are two types of parallax images of the left-eye image and parallax image of the right-eye image. Use. More specifically, in the parallax image of the left-eye image, the average value of the maximum parallax value (AVE_MAX1) and the average value of the minimum parallax value (AVE_MIN1), and the parallax image in the parallax image of the right-eye image. Using the average value of the maximum value (AVE_MAX2) and the average value of the minimum value of parallax (AVE_MIN2), the parallax increase / decrease coefficient c and the parallax offset o are calculated according to the following equations.

Parallax increase / decrease coefficient c = target parallax range r × 2 / (average value AVE_MAX1−average value AVE_MIN1 + average value AVE_MAX2−average value AVE_MIN2)
Parallax offset o = (average value AVE_MAX1 + average value AVE_MIN1 + average value AVE_MAX2 + average value AVE_MIN2) / 4
Subsequently, a process for adjusting the parallax is performed on each generated parallax image according to the overall parallax adjustment amount determined in step S30 (steps S41 and S42). The processing in steps S41 and S42 is the same as that in step S4 in FIG. 5, but the directions for adjusting the parallax are different from each other.

  That is, regarding the parallax adjustment, unlike the first embodiment, the amount of parallax (1-c) to be changed is distributed equally to the left and right from the state where the parallax exists in the left-eye image and the right-eye image. Adjust the parallax. Here, the addition amount of the parallax adjustment is reversed as follows because the moving directions are opposite to each other in the left-eye image and the right-eye image.

Adjusted parallax (distance) amount of left-eye image = ((1-parallax increase / decrease coefficient c) × (pre-adjustment parallax (distance) amount of left-eye image−parallax offset o) + parallax offset o) / 2
Adjusted parallax (distance) amount of right-eye image = − ((1-parallax increase / decrease coefficient c) × (pre-adjustment parallax (distance) amount of right-eye image−parallax offset o) + parallax offset o) / 2
Then, the adjusted left-eye image and right-eye image are output as a stereo image to a monitor or the like.

  As described above, the parallax adjustment processing in S41 and S42 uses the images obtained by adjusting the parallax between the input image 1 and the input image 2 for each frame according to the corresponding corrected parallax images, respectively. Includes processing to generate an image.

  FIG. 20 is a diagram showing an application example of the image processing method according to the second embodiment of the present invention. FIG. 21 is a diagram illustrating an example of a stereo image output in the application example illustrated in FIG. 20.

  As shown in FIGS. 20A and 20B, in this embodiment as well, since smoothing is performed over a long range, even if there is a sudden change in parallax in a local frame, it is output. The parallax of the stereo image does not increase or decrease rapidly. Therefore, there is no sense of incongruity due to a sudden change in the overall parallax of a stereoscopically displayed moving image. On the other hand, by smoothing the parallax image over a short time range, the parallax information (stereoscopic information) in each frame remains, and appropriate stereoscopic vision in each frame can be provided.

FIG. 21A shows an example of an input image 1 (left eye image) and an input image 2 (right eye image). FIG. 21B shows a correspondingly output stereo image (parallax adjustment). An example of a later left-eye image and right-eye image) is shown. As described above, in the present embodiment, the directions for adjusting the parallax are reversed for each of the input image 1 (left eye image) and the input image 2 (right eye image) (broken arrows).

  Since other configurations and processes of the second embodiment are the same as those of the first embodiment, detailed description will not be repeated. Also for the second embodiment, the processing and configuration described in the modification of the first embodiment can be employed as appropriate.

[E. Other forms]
The present invention includes an embodiment according to another aspect as follows. That is, the moving image processing method according to another aspect of the present invention is image processing including at least one smoothing process in the time axis direction for adjusting the amount of parallax, and at least sets the amount of parallax on the fly-out side to a predetermined amount of parallax. Below.

Preferably, the parallax range (jumping-depth) is adjusted to a predetermined parallax or less.
Preferably, the smoothing process in the time axis direction performs smoothing in two types of time ranges, and determines a parallax amount corresponding to each pixel.

Preferably, the predetermined time range is longer than the second time range.
Preferably, the stereoscopic video is generated from one of the images (one-frame mode), or is generated by adjusting both images for parallax (two-frame mode).

  Preferably, the determination of the overall parallax adjustment amount includes determining at least one of a parallax increase / decrease coefficient and a parallax offset of the parallax position.

Preferably, the parallax increase / decrease coefficient is limited so as not to exceed a predetermined range.
Preferably, in the smoothing in a predetermined time range, the minimum parallax value is smoothed.

  Preferably, in the smoothing in the predetermined time range and the second time range, the parallax that changes beyond the threshold value compared to a frame close in time is clipped to the threshold value.

  Preferably, in the smoothing in the predetermined time range and the second time range, the parallax that changes beyond the threshold value compared with the temporally smoothed frame is clipped to the threshold value.

Preferably, the predetermined time range is 10 seconds or more.
Preferably, the second time range is 5 seconds or less.

[F. advantage]
According to the embodiment of the present invention, it is possible to adjust the overall parallax well while leaving the stereoscopic effect of each frame constituting the stereoscopic moving image.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 image processing system, 2 imaging unit, 3 image processing unit, 4 image output unit, 21, 121
First camera, 21a, 22a Lens, 21b, 22b Image sensor, 22, 122 Second camera, 23, 24 A / D conversion unit, 31 Corresponding point search unit, 32 Parallax image generation unit, 3
3 smoothing processing unit, 34 parallax overall adjustment amount determination unit, 35 parallax adjustment unit, 36 image generation unit, 100 digital camera, 102 CPU, 104 digital processing circuit, 106 image processing circuit, 108 image display unit, 112 storage unit, 114 zoom mechanism, 200 personal computer, 202 personal computer main body, 204 image processing program, 206 monitor, 208 mouse, 210 keyboard, 212 external storage device.

Claims (14)

An acquisition step of acquiring a plurality of first input images ordered along a time axis and a plurality of second input images ordered along a time axis, the plurality of first and second The input images are a pair of images acquired by imaging the subject from different viewpoints,
Generating a plurality of parallax images corresponding to respective positions on the time axis from the first and second input images;
Among the time ranges in which the first input image and the second input image are acquired, a first time range and a second time range shorter than the first time range within the first time range And smoothing the plurality of parallax images in the time axis direction over the first time range, and smoothing the plurality of parallax images in the time axis direction over the second time range; ,
An overall parallax adjustment amount determination step for determining an overall parallax adjustment amount based on a result obtained by smoothing at least the magnitude of the parallax on the fly-out side over the first time range; and
And a correction step of correcting a parallax image obtained by smoothing over the second time range according to the overall parallax adjustment amount.   The correction step corresponds to a result of smoothing the parallax range between the projection-side parallax and the depth-side parallax over the first time range for the parallax image existing in the first time range. The image processing method according to claim 1, further comprising a step of limiting to a value.   A stereoscopic image is generated using the plurality of first input images and a plurality of images generated by giving parallax according to corrected parallax images corresponding to the plurality of first input images, respectively. The image processing method according to claim 1, further comprising a step.   The method further comprises generating a stereoscopic image using images obtained by adjusting the parallax of the plurality of first and second input images according to the corresponding parallax images after correction. 4. The image processing method according to any one of items 3.   The image processing method according to claim 1, wherein the correcting step includes a step of determining a parallax increase / decrease coefficient that is a slope of a parallax adjustment function.   The image processing method according to claim 5, wherein the correcting step includes a step of limiting the parallax increase / decrease coefficient to a predetermined range.   The image processing method according to claim 1, wherein the correcting step includes a step of determining an offset of a parallax position generated in the parallax image.   The image processing method according to claim 1, wherein the correcting step includes a step of limiting a minimum value of parallax according to a result of smoothing over the first time range.   In the smoothing step, the maximum value or the minimum value of the parallax image corresponding to a certain position on the time axis is compared with the maximum value or the minimum value of the parallax image corresponding to an adjacent position on the time axis, 9. The pixel according to claim 1, comprising a step of limiting a maximum value or a minimum value of the parallax to the predetermined threshold value for a pixel that has changed beyond a predetermined threshold value. 9. Image processing method.   The smoothing step has a predetermined threshold value compared to the parallax indicated by the pixels included in the smoothed parallax image among the parallaxes indicated by the pixels included in the parallax image corresponding to a certain position on the time axis. 9. The image processing method according to claim 1, further comprising a step of limiting a corresponding parallax magnitude to the predetermined threshold value for a pixel that has changed in excess of.   The image processing method according to claim 1, wherein the first time range is 10 seconds or more.   The image processing method according to claim 1, wherein the second time range is 10 seconds or less. An acquisition unit configured to acquire a plurality of first input images ordered along a time axis and a plurality of second input images ordered along the time axis; The input images are a pair of images acquired by imaging the subject from different viewpoints,
Generating means for generating a plurality of parallax images corresponding to respective positions on the time axis from the first and second input images;
Among the time ranges in which the first input image and the second input image are acquired, a first time range and a second time range shorter than the first time range within the first time range Smoothing means for smoothing the plurality of parallax images in the time axis direction over the first time range and smoothing the plurality of parallax images in the time axis direction over the second time range; ,
An overall parallax adjustment amount determination for determining an overall parallax adjustment amount based on a result obtained by smoothing at least the magnitude of the parallax on the fly-out side over the first time range; and
An image processing system comprising: a correcting unit that corrects a parallax image obtained by smoothing over the second time range according to the overall parallax adjustment amount. An image processing program, on a computer,
An acquisition step of acquiring a plurality of first input images ordered along the time axis and a plurality of second input images ordered along the time axis is executed, and the plurality of first and second The input images are a pair of images acquired by capturing the subject from different viewpoints,
Generating a plurality of parallax images corresponding to respective positions on the time axis from the first and second input images;
Among the time ranges in which the first input image and the second input image are acquired, a first time range and a second time range shorter than the first time range within the first time range And smoothing the plurality of parallax images in the time axis direction over the first time range, and smoothing the plurality of parallax images in the time axis direction over the second time range; ,
An overall parallax adjustment amount determination step for determining an overall parallax adjustment amount based on a result obtained by smoothing at least the magnitude of the parallax on the fly-out side over the first time range; and
An image processing program for executing a correction step of correcting a parallax image obtained by smoothing over the second time range according to the overall parallax adjustment amount.