JP2013074397A - Image processing system, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system, an image processing method, and an image processing program which keeps the quality of stereoscopic vision display even if one input image includes a defect such as a blur.SOLUTION: The image processing system comprises: first imaging means which takes an image of a subject to obtain a first input image; second imaging means which takes an image of the subject from a viewpoint different from that of the first imaging means to obtain a second input image; frequency characteristic obtaining means which obtains the frequency characteristic of the first and second input images; and stereoscopic vision generation means for generating a stereoscopic image for stereoscopic vision display of the subject, from the first and second input images, mainly using an input image determined to have a relatively high quality on the basis of the obtained frequency characteristics.

Description

  The present invention relates to an image processing system, an image processing method, and an image processing program directed to image generation for stereoscopic display of a subject.

  In conjunction with the development of display devices in recent years, development of image processing techniques for stereoscopically displaying the same object (subject) has been promoted. As a typical method for realizing such stereoscopic display, there is a method using binocular parallax felt by humans. When such binocular parallax is used, it is necessary to generate a pair of images (hereinafter also referred to as “stereo images” or “3D images”) with parallax according to the distance from the imaging means to the subject. is there. Even when such a stereo image is generated, it is preferable to use an input image with good image quality.

  Conventionally, although not related to a stereo image, a technique for evaluating the image quality of an input image or a technique for removing noise or the like is known.

  For example, Japanese Patent Application Laid-Open No. 2011-055259 (Patent Document 1) discloses an electronic shake correction technique that reduces shake by performing alignment processing on a plurality of images and then performing synthesis processing. Japanese Patent Laid-Open No. 2010-279002 (Patent Document 2) discloses an effective flare correction by appropriately detecting a flare component even when a saturated portion occurs in an image and an original luminance level cannot be obtained. A technique for performing is disclosed. Japanese Patent Laid-Open No. 2009-200249 (Patent Document 3) discloses that when a ghost image is generated in a captured image, the ghost image is removed from the captured image including the ghost image, thereby obtaining a captured image without the ghost image. Disclose the technology to obtain. Japanese Patent Laying-Open No. 2010-055194 (Patent Document 4) discloses a technique for reliably evaluating whether or not a subject is clearly visible.

JP 2011-055259 A JP 2010-279002 A Japanese Unexamined Patent Publication No. 2009-200749 JP 2010-055194 A

  In a configuration in which a stereo image is generated using a plurality of imaging units that respectively image subjects from different viewpoints, the image quality of input images captured by the respective imaging units may be different from each other and may be difficult to see. More specifically, in a digital camera or a mobile phone equipped with a pair of cameras, only an input image captured by one camera may be blurred. Possible reasons for this include a difference in autofocus operation, a difference in the amount of camera shake, and contamination (in particular in the case of a mobile phone) of one lens due to unintentional touching by the user. In such a case, the degree of blur is different between the image for the left eye and the image for the right eye, and the quality of the provided stereoscopic image is deteriorated.

  In addition, even when a phenomenon such as flare or ghost occurs only in one input image, a stereoscopic image provided due to a mismatch caused by such a phenomenon between the image for the left eye and the image for the right eye The quality of

  Therefore, the present invention has been made to solve such a problem, and an object of the present invention is to provide an image that can maintain the quality of stereoscopic display even when a defect such as blur exists in one input image. A processing system, an image processing method, and an image processing program are provided.

  An image processing system according to an aspect of the present invention includes: a first imaging unit that captures a subject and obtains a first input image; and a second input that captures the subject from a different viewpoint from the first imaging unit. A second imaging unit for acquiring an image; a frequency characteristic acquiring unit for acquiring the frequency characteristics of the first and second input images; and an input determined to have relatively good image quality based on the acquired frequency characteristics Stereoscopic image generating means for generating a stereo image for stereoscopically displaying the subject from the first and second input images by using the image mainly.

  Preferably, the stereoscopic generation unit determines that an input image including more high-frequency components has a relatively high image quality.

  Preferably, the frequency characteristic acquisition unit acquires the frequency specification using at least one of extraction of an edge amount included in the input image and frequency analysis on the input image.

  Preferably, the stereoscopic generation unit mainly uses one of the first and second input images based on the acquired frequency characteristics and the positional relationship between the first and second imaging units at the time of imaging. Thus, the processing mode for generating a stereo image and the processing mode for generating a stereo image using both the first and second input images are switched.

  Preferably, the stereoscopic generation unit switches a processing mode for generating a stereo image for each partial image.

  Preferably, the stereoscopic generation unit determines a processing mode to be applied to generate a stereo image for each of a plurality of frames when the first and second imaging units capture a moving image of the subject.

  More preferably, the stereoscopic generation unit maintains the processing mode once determined for a predetermined period when the first and second imaging units capture a moving image of the subject.

  More preferably, the stereoscopic generation unit determines the processing mode based on the first and second input images corresponding to the first frame when the first and second imaging units capture a moving image of the subject.

  Preferably, when the image processing system determines that the image quality is deteriorated for an input image from the specific imaging unit a plurality of times, the image processing system warns the user of lens contamination of the corresponding imaging unit. A warning means is further included.

  Preferably, the image processing system further includes an imaging inducing unit that prompts the user to perform imaging using the first and second imaging units for evaluating the frequency characteristics.

  Preferably, the frequency characteristic acquisition unit acquires the frequency characteristic with respect to the first and second input images acquired at any time of initial setting, power-on, or focusing operation immediately before imaging.

  Preferably, an imaging unit corresponding to an input image including more high-frequency components is set as a default as an imaging unit that is mainly used when generating a stereo image.

  More preferably, the stereoscopic generation unit generates a stereo image mainly using an input image from the imaging unit set as a default.

  An image processing method according to another aspect of the present invention includes a step of capturing a subject to acquire a first input image, and capturing a subject from a viewpoint different from the viewpoint from which the first input image is captured. Mainly using the step of acquiring the input image, the step of acquiring the frequency characteristics of the first and second input images, and the input image determined to have relatively good image quality based on the acquired frequency characteristics Generating a stereo image for stereoscopically displaying the subject from the first and second input images.

  According to still another aspect of the present invention, an image processing program for causing a computer to execute image processing is provided. The image processing program causes the computer to acquire a first input image obtained by imaging a subject and a second input image obtained by imaging the subject from a viewpoint different from the viewpoint from which the first input image is captured. The second acquisition means for acquiring the frequency characteristics, the frequency characteristic acquisition means for acquiring the frequency characteristics of the first and second input images, and the input image determined to have relatively good image quality based on the acquired frequency characteristics Is used as a stereoscopic generation means for generating a stereo image for stereoscopically displaying the subject from the first and second input images.

  According to the present invention, the quality of stereoscopic display can be maintained even when a defect such as blur exists in one input image.

1 is a block diagram showing a basic configuration of an image processing system according to an embodiment of the present invention. It is a figure which shows the specific structural example of the imaging part shown in FIG. It is a block diagram which shows the structure of the digital camera which actualized the image processing system shown in FIG. It is a block diagram which shows the structure of the personal computer which actualized the image processing system shown in FIG. FIG. 2 is an external view of a mobile phone that embodies the image processing system shown in FIG. 1. It is a figure which shows the procedure of the image processing method according to 1st Embodiment. It is a figure which shows an example of a pair of input image imaged by the imaging part shown in FIG. It is a flowchart which shows the process sequence of the edge extraction process shown in FIG. It is a figure which shows an example of the averaging filter for edge extraction used in the smoothing process of FIG. It is a figure which shows an example of the distance image produced | generated from a pair of input image shown in FIG. 7 according to the image processing method according to 1st Embodiment. It is a figure which shows an example of the averaging filter used in the smoothing process of FIG. It is a figure for demonstrating the processing content of the parallax adjustment process shown in FIG. It is a figure for demonstrating the process sequence in the stereo image generation process of FIG. It is a flowchart which shows the process sequence of the stereo image generation process shown in FIG. It is a figure which shows an example of the stereo image produced | generated from a pair of input image shown in FIG. 7 according to the image processing method according to 1st Embodiment. It is a figure which shows the procedure of the image processing method according to 2nd Embodiment. It is a figure which shows the process example which evaluates an image quality for every partial image according to a 2nd modification. It is a figure for demonstrating the process which produces | generates a stereo image according to the 2nd modification. It is a figure for demonstrating the process which produces | generates a stereo image according to the 2nd modification. It is a figure which shows an example of the user interface provided in a 3rd modification.

  Embodiments of the present invention will be described in detail with reference to the drawings. In addition, about the same or equivalent part in a figure, the same code | symbol is attached | subjected and the description is not repeated.

<A. Overview>
The image processing system according to the embodiment of the present invention generates a stereo image for performing stereoscopic display from a plurality of input images obtained by imaging a subject from a plurality of viewpoints. In generating the stereo image, the image processing system acquires the frequency characteristics of the respective input images, and mainly uses the input images determined to have relatively good image quality based on the acquired frequency characteristics. A stereo image for stereoscopic display of the subject is generated from each input image.

  Thereby, even if a defect such as blur exists in one input image, the quality of the stereoscopic display can be maintained.

<B. System configuration>
First, the configuration of the image processing system according to the embodiment of the present invention will be described.

(B1: Basic configuration)
FIG. 1 is a block diagram showing a basic configuration of an image processing system 1 according to the embodiment of the present invention. With reference to FIG. 1, the image processing system 1 includes an imaging unit 2, an image processing unit 3, and a 3D image output unit 4. In the image processing system 1 shown in FIG. 1, the imaging unit 2 captures a subject to acquire a pair of input images (input image 1 and input image 2), and the image processing unit 3 acquires the acquired pair of inputs. By performing image processing as will be described later on the image, a stereo image (left-eye image and right-eye image) for stereoscopically displaying the subject is generated. Then, the 3D image output unit 4 outputs the stereo image (the left eye image and the right eye image) to a display device or the like.

  The imaging unit 2 captures the same object (subject) from different viewpoints and generates a pair of input images. More specifically, the first camera 21, the second camera 22, an A / D (Analog to Digital) conversion unit 23 connected to the first camera, and an A / D conversion connected to the second camera 22. Part 24. The A / D conversion unit 23 outputs an input image 1 indicating the subject imaged by the first camera 21, and the A / D conversion unit 24 outputs an input image 2 indicating the subject imaged by the second camera 22. To do.

  That is, the first camera 21 and the A / D conversion unit 23 correspond to a first imaging unit that captures a subject and obtains a first input image. The second camera 22 and the A / D conversion unit 24 This corresponds to a second imaging unit that captures a subject from a different viewpoint from the first imaging unit and obtains a second input image.

  The first camera 21 includes a lens 21a that is an optical system for imaging a subject, and an imaging element 21b that is a device that converts light collected by the lens 21a into an electrical signal. The A / D converter 23 converts a video signal (analog electrical signal) indicating a subject output from the image sensor 21b into a digital signal and outputs the digital signal. Similarly, the second camera 22 includes a lens 22a that is an optical system for imaging a subject, and an imaging element 22b that is a device that converts light collected by the lens 22a into an electrical signal. The A / D converter 24 converts a video signal (analog electrical signal) indicating a subject output from the image sensor 22b into a digital signal and outputs the digital signal. The imaging unit 2 may further include a control processing circuit for controlling each part.

  As will be described later, in the image processing according to the present embodiment, a stereo image (a left-eye image and a right-eye image) can be generated using only the input image captured by one camera. Therefore, even if the image quality of one input image is degraded, a stereo image can be generated by using the other input image with good image quality. That is, the difference in autofocus operation between the first camera 21 and the second camera 22, the difference in the amount of camera shake generated in the first camera 21 and the second camera 22, respectively, and the lens 21a or the lens 22a caused by the user touching unconsciously. Even when only an input image taken by one camera is blurred due to dirt or the like, a stereo image with good quality can be generated.

  FIG. 2 is a diagram illustrating a specific configuration example of the imaging unit 2 illustrated in FIG. 1. More specifically, FIG. 2 shows an example of the imaging unit 2 including lenses 21a and 22a having the same basic specifications. In this imaging unit 2, an optical zoom function may be mounted on any lens.

  In the image processing method according to the present embodiment, it is only necessary that the line-of-sight directions (viewpoints) of the respective cameras with respect to the same subject are different, so in the imaging unit 2, the arrangement of the lenses 21a and 22a (vertical arrangement or horizontal arrangement). The direction array can be arbitrarily set. That is, as shown in FIG. 2A, the image may be arranged in the vertically long direction (vertical stereo) or may be picked up in the horizontally long direction (horizontal stereo) as shown in FIG. Also good.

  In the image processing method according to the present embodiment, it is not always necessary to acquire input image 1 and input image 2 at the same time. That is, if the positional relationship of the imaging unit 2 with respect to the subject is substantially the same at the imaging timing for acquiring the input image 1 and the input image 2, the input image 1 and the input image 2 are acquired at different timings, respectively. Also good. Moreover, in the image processing method according to the present embodiment, a stereo image for performing stereoscopic display can be generated not only as a still image but also as a moving image. In this case, a series of images can be acquired for each camera by capturing the subject continuously in time while synchronizing between the first camera 21 and the second camera 22. . In the image processing method according to the present embodiment, the input image may be a color image or a monochrome image.

  Referring to FIG. 1 again, the image processing unit 3 performs stereoscopic display of the subject by performing the image processing method according to the present embodiment on the pair of input images acquired by the imaging unit 2. Stereo images (left eye image and right eye image) are generated. More specifically, the image processing unit 3 includes an edge extraction unit 30, a corresponding point search unit 31, a distance image generation unit 32, a smoothing processing unit 33, a parallax adjustment unit 34, and a 3D image generation unit 35. including.

  The edge extraction unit 30 acquires frequency characteristics of a pair of input images (input image 1 and input image 2). More specifically, the edge extraction unit 30 calculates the amount of edge included in each input image. It shows that there are many high frequency components in the target input image, so that this calculated edge amount is large. That is, an input image with relatively good image quality corresponds to an input image with higher frequency characteristics.

  In this embodiment, since it is determined that the input image including a larger number of high-frequency components has a relatively high image quality, the input image having a larger edge amount is determined to have a higher image quality. This is because when the input image is blurred, the picture becomes sluggish as a whole, so the amount of edges is small and the frequency components contained in it move relatively to the low frequency side. is there.

  As a method of acquiring the frequency characteristics of the input image, instead of the method of extracting the edge amount included in the input image, various frequency analyzes are performed on the input image, and based on the result of the frequency analysis, The image quality of the target input image may be evaluated. That is, the frequency characteristic is typically determined by edge amount or frequency analysis.

  Corresponding point search unit 31 performs corresponding point search processing on a pair of input images (input image 1 and input image 2). Typically, the corresponding point search processing includes POC (Phase-Only Correlation) calculation method, SAD (Sum of Absolute Difference) calculation method, SSD (Sum of Squared Difference) calculation method, NCC (Normalized Cross Correlation) calculation method. The method etc. can be used. That is, the corresponding point search unit 31 searches for a corresponding relationship for each point of the subject between the input image 1 and the input image 2.

  The distance image generation unit 32 acquires distance information about two input images. This distance information is calculated based on the difference in information about the same subject. Typically, the distance image generation unit 32 calculates distance information from a correspondence relationship between input images for each point of the subject searched by the corresponding point search unit 31. The imaging unit 2 images the subject from different viewpoints. Therefore, between two input images, a pixel representing a certain point (attention point) of the subject is shifted by a distance corresponding to the distance between the imaging unit 2 and the point of the subject. In this specification, the difference between the coordinates on the image coordinate system of the pixel corresponding to the target point of the input image 1 and the coordinates on the image coordinate system of the pixel corresponding to the target point of the input image 2 is referred to as “parallax”. Called. The distance image generation unit 32 calculates the parallax for each point of interest of the subject searched by the corresponding point search unit 31.

  This parallax is an index value indicating the distance from the imaging unit 2 to the corresponding point of interest of the subject. The larger the parallax, the shorter the distance from the imaging unit 2 to the corresponding point of interest of the subject, that is, the closer the imaging unit 2 is. In this specification, the term “distance information” is used as a general term for the parallax and the distance from the imaging unit 2 of each point of the subject indicated by the parallax.

  Note that the direction in which the parallax occurs between the input images depends on the positional relationship between the first camera 21 and the second camera 22 in the imaging unit 2. For example, when the first camera 21 and the second camera 22 are arranged at a predetermined interval in the vertical direction, the parallax between the input image 1 and the input image 2 is generated in the vertical direction.

  The distance image generation unit 32 calculates distance information for each point of the subject, and generates a distance image (parallax image) that expresses the calculated distance information in association with coordinates on the image coordinate system.

  The smoothing processing unit 33 performs a smoothing process on the distance image generated by the distance image generation unit 32.

  The parallax adjustment unit 34 adjusts the generated distance image so as to match an allowable parallax range (a range from the maximum protruding position to the maximum depth position). Details of the adjustment processing by the parallax adjustment unit 34 will be described later.

  The 3D image generation unit 35 shifts each pixel constituting the input image by the corresponding distance information (number of pixels) based on the distance image after adjustment by the parallax adjustment unit 34, thereby displaying the subject in a stereoscopic view. Stereo images (left eye image and right eye image) are generated. In this manner, a stereo image for stereoscopically displaying the subject is generated by shifting the pixels included in the input image in the horizontal direction based on the distance information. When viewed between the left-eye image and the right-eye image, each point of the subject is separated by a distance corresponding to the distance information (number of pixels) indicated by the distance image, that is, according to the distance information (number of pixels). It is expressed with given parallax. As a result, the subject can be stereoscopically displayed.

  In the present embodiment, the 3D image generation unit 35 determines which input image has a relatively good image quality based on the frequency characteristics acquired by the edge extraction unit 30 and the like, and the image quality is relatively high. A stereo image for stereoscopically displaying the subject is generated from the input images 1 and 2 mainly using the input image determined to be good. In other words, in the present embodiment, the frequency characteristic of each image is detected for each input image captured by at least two imaging systems, and the better image quality is prioritized based on the detection result. To generate a stereo image.

  For example, when the image quality of the input image 1 is relatively high, the 3D image generation unit 35 shifts the input image 1 in the horizontal direction by the distance information (number of pixels) corresponding to each pixel (left Eye image and right eye image). The input image 1 is used as it is as a left-eye image or a right-eye image, and the other right-eye image or left-eye is shifted by shifting each pixel of the input image 1 by the corresponding distance information (number of pixels). An ophthalmic image may be generated.

  On the other hand, when the image quality of the input image 2 is relatively good, the 3D image generation unit 35 shifts the input image 2 in the horizontal direction by the distance information (number of pixels) corresponding to each pixel. An image (an image for the left eye and an image for the right eye) is generated. Similarly to the above, the input image 2 is used as it is as the left-eye image or the right-eye image, and each pixel of the input image 2 is shifted in the horizontal direction by the corresponding distance information (number of pixels) to be used for the other right eye. An image or an image for the left eye may be generated.

  At this time, since an input image with relatively poor image quality is used only for generating a distance image, the stereo image does not include a deteriorated image quality portion of one input image. Thereby, even if a defect such as blur exists in one input image, the quality of the stereoscopic display can be maintained.

  The 3D image output unit 4 outputs the stereo image (left-eye image and right-eye image) generated by the image processing unit 3 to a display device or the like.

Details of the processing operation of each unit will be described later.
Although the image processing system 1 shown in FIG. 1 can be configured independently of each other, in general, the image processing system 1 is often embodied as a digital camera or a personal computer described below. Therefore, an implementation example of the image processing system 1 according to the present embodiment will be described.

(B2: Implementation example 1)
FIG. 3 is a block diagram showing a configuration of a digital camera 100 that embodies the image processing system 1 shown in FIG. A digital camera 100 illustrated in FIG. 3 includes two cameras (a first camera 121 and a second camera 122), and can capture a stereo image for stereoscopically displaying a subject. In FIG. 3, components corresponding to the respective blocks constituting the image processing system 1 shown in FIG. 1 are denoted by the same reference numerals as in FIG.

  In the digital camera 100, an input image acquired by imaging a subject with the first camera 121 is stored and output, and an input image acquired by imaging the subject with the second camera 122 is mainly described above. Are used for the corresponding point search process and the distance image generation process. Therefore, it is assumed that only the first camera 121 has an optical zoom function.

  Referring to FIG. 3, a digital camera 100 includes a CPU (Central Processing Unit) 102, a digital processing circuit 104, an image display unit 108, a card interface (I / F) 110, a storage unit 112, and a zoom mechanism. 114, an acceleration sensor 116, a first camera 121, and a second camera 122.

  The CPU 102 controls the entire digital camera 100 by executing a program (including an image processing program) stored in advance. The digital processing circuit 104 executes various digital processes including image processing according to the present embodiment. The digital processing circuit 104 is typically configured by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an LSI (Large Scale Integration), an FPGA (Field-Programmable Gate Array), or the like. The digital processing circuit 104 includes an image processing circuit 106 for realizing the functions provided by the image processing unit 3 shown in FIG.

  The image display unit 108 includes an image provided by the first camera 121 and / or the second camera 122, an image generated by the digital processing circuit 104 (image processing circuit 106), various setting information related to the digital camera 100, and A control GUI (Graphical User Interface) screen or the like is displayed. It is preferable that the image display unit 108 can stereoscopically display the subject using a stereo image generated by the image processing circuit 106. In this case, the image display unit 108 is configured by an arbitrary display device (liquid crystal display device for three-dimensional display) corresponding to the three-dimensional display method. As such a three-dimensional display method, a parallax barrier method or the like can be employed. In this parallax barrier method, by providing a parallax barrier on the liquid crystal display surface, the right eye image can be visually recognized by the user's right eye, and the left eye image can be visually recognized by the user's left eye. Alternatively, a shutter glasses method may be adopted. In this shutter glasses method, the left eye image and the right eye image are alternately switched at high speed and displayed, and the user wears special glasses equipped with a shutter that opens and closes in synchronization with the switching of the image. , You can enjoy stereoscopic display.

  A card interface (I / F) 110 is an interface for writing image data generated by the image processing circuit 106 to the storage unit 112 or reading image data from the storage unit 112. The storage unit 112 is a storage device that stores image data generated by the image processing circuit 106 and various information (setting values such as control parameters and operation modes of the digital camera 100). The storage unit 112 includes a flash memory, an optical disk, a magnetic disk, and the like, and stores data in a nonvolatile manner.

  The zoom mechanism 114 is a mechanism that changes the imaging magnification of the first camera 121 according to a user operation or the like. The zoom mechanism 114 typically includes a servo motor and the like, and drives the lens group constituting the first camera 121 to change the focal length.

  The acceleration sensor 116 determines the attitude of the digital camera 100 by detecting gravitational acceleration.

  The first camera 121 generates an input image for generating a stereo image by imaging a subject. The first camera 121 includes a plurality of lens groups that are driven by the zoom mechanism 114. The second camera 122 is used for corresponding point search processing and distance image generation processing, which will be described later, and images the same subject imaged by the first camera 121 from different viewpoints.

  As described above, the digital camera 100 shown in FIG. 3 is obtained by mounting the entire image processing system 1 according to the present embodiment as a single device. That is, the user can visually recognize the subject in a three-dimensional manner on the image display unit 108 by imaging the subject using the digital camera 100.

(B3: Implementation example 2)
FIG. 4 is a block diagram showing a configuration of a personal computer 200 that embodies the image processing system 1 shown in FIG. In the personal computer 200 shown in FIG. 4, the imaging unit 2 for acquiring a pair of input images is not mounted, and a pair of input images (an input image 1 and an input image 2) acquired by an arbitrary imaging unit 2. Is input from the outside. Even such a configuration can be included in the image processing system 1 according to the embodiment. In FIG. 4 as well, components corresponding to the respective blocks constituting the image processing system 1 shown in FIG. 1 are denoted by the same reference numerals as in FIG.

  Referring to FIG. 4, personal computer 200 includes a personal computer main body 202, a monitor 206, a mouse 208, a keyboard 210, and an external storage device 212.

  The personal computer main body 202 is typically a general-purpose computer according to a general-purpose architecture, and includes a CPU, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as basic components. The personal computer main body 202 can execute an image processing program 204 for realizing a function provided by the image processing unit 3 shown in FIG. Such an image processing program 204 is stored in a storage medium such as a CD-ROM (Compact Disk-Read Only Memory) and distributed, or distributed from a server device via a network. The image processing program 204 is stored in a storage area such as a hard disk of the personal computer main body 202.

  Such an image processing program 204 implements processing by calling necessary modules among program modules provided as part of an operating system (OS) executed by the personal computer main body 202 at a predetermined timing and order. It may be configured as follows. In this case, the image processing program 204 itself does not include a module provided by the OS, and image processing is realized in cooperation with the OS. Further, the image processing program 204 may be provided by being incorporated in a part of some program instead of a single program. Even in such a case, the image processing program 204 itself does not include a module that is commonly used in the program, and image processing is realized in cooperation with the program. Even such an image processing program 204 that does not include some modules does not depart from the spirit of the image processing system 1 according to the present embodiment.

  Of course, part or all of the functions provided by the image processing program 204 may be realized by dedicated hardware.

  The monitor 206 displays a GUI screen provided by an operating system (OS), an image generated by the image processing program 204, and the like. As with the image display unit 108 shown in FIG. 3, the monitor 206 is preferably capable of stereoscopically displaying a subject using a stereo image generated by the image processing program 204. In this case, the monitor 206 is configured by a display device such as a parallax barrier method or a shutter glasses method, as described in the image display unit 108.

  The mouse 208 and the keyboard 210 each accept a user operation and output the contents of the accepted user operation to the personal computer main body 202.

  The external storage device 212 stores a pair of input images (an input image 1 and an input image 2) obtained by some method, and outputs the pair of input images to the personal computer main body 202. As the external storage device 212, a device that stores data in a nonvolatile manner such as a flash memory, an optical disk, or a magnetic disk is used.

  As described above, personal computer 200 shown in FIG. 4 is obtained by mounting a part of image processing system 1 according to the present embodiment as a single device. By using such a personal computer 200, the user stereoscopically displays the subject from a pair of input images acquired by imaging the subject from different viewpoints using an arbitrary imaging unit (stereo camera). Therefore, a stereo image (a left-eye image and a right-eye image) can be generated. Furthermore, by displaying the generated stereo image on the monitor 206, stereoscopic display can be enjoyed.

(B4: Implementation example 3)
FIG. 5 is an external view of a mobile phone 300 that embodies the image processing system 1 shown in FIG. Referring to FIG. 5, imaging unit 2 including lens 21 a and imaging element 21 b is mounted on one surface of mobile phone 300. Note that the configuration relating to the stereo image generation of the mobile phone 300 is the same as the configuration of the digital camera 100 shown in FIG.

<C. First Embodiment (Vertical Stereo)>
First, as an image processing method according to the first embodiment, as shown in FIG. 2A, two lenses of the same type (no optical zoom function) are arranged at a predetermined interval in the vertical direction. (Stereo camera) is assumed.

  As described above, since a commercially available digital camera or mobile phone that can generate a stereo image outputs the captured image as a stereo image as it is, a blurred image is captured on one side and is difficult to view as a stereoscopic view. There is. This is because the autofocusing works differently and the focus position (focus position) is different, especially in the case of mobile phones, the lens surface is exposed (exposed), so one side This is because the lens is touched and dirty. In this embodiment, even in such a case, a high-quality stereo image is generated.

  FIG. 6 is a diagram showing a procedure of the image processing method according to the first embodiment. Referring to FIG. 6, the image processing method according to the first embodiment is performed on input image 1 and input image 2 acquired by first camera 21 and second camera 22 capturing the same subject, respectively. Thus, edge extraction processing is executed (steps S1 and S2). An edge amount comparison process is executed based on the edge amount extracted from each input image (step S3). Based on the result of the edge amount comparison processing, output image determination processing is executed (step S4). By this output image determination process, an input image used for generating a stereo image is determined.

  Further, when an input image used to generate a stereo image is determined, corresponding point search processing (steps S10A and S10B), parallax image generation processing (steps S11A and S11B), smoothing processing (steps S12A and S12B), parallax Adjustment processing (steps S13A and S13B) and stereo image generation processing (steps S14A and S14B) are sequentially executed to generate a stereo image. Hereinafter, each step will be described in detail.

(C1: input image)
FIG. 7 is a diagram illustrating an example of a pair of input images captured by the imaging unit 2 illustrated in FIG. 7A shows the input image 1 captured by the first camera 21, and FIG. 7B shows the input image 2 captured by the second camera 22. The input image 1 shown in FIG. 7A can be used as it is as one of the finally output stereo images (in this example, the image for the left eye), and the input image shown in FIG. 2 can be used as it is as the other image (right-eye image in this example) of the stereo image finally output. However, in the example of FIG. 7, the image is taken in a configuration in which two lenses are arranged apart from each other by a predetermined interval in the vertical direction, and only the output of both the input image 1 and the input image 2 does not become a stereo image. .

  In FIG. 7, an image coordinate system is defined for convenience in order to facilitate the explanation. More specifically, an orthogonal coordinate system is employed in which the horizontal direction of the input image is the X axis and the vertical direction of the input image is the Y axis. The origin of the X and Y axes is assumed to be the upper left corner of the input image for convenience. Further, the line-of-sight direction of the imaging unit 2 (FIG. 1) is taken as the Z axis. This orthogonal coordinate system may be used in the description of other drawings in this specification.

  As described above, since the imaging unit 2 in which the first camera 121 and the second camera 122 are arranged in the vertical direction is used, the input image 1 shown in FIG. 7A and the input image 2 shown in FIG. There is a parallax along the Y-axis direction.

(C2: edge extraction process)
First, the details of the edge extraction process (steps S1 and S2) shown in FIG. 6 will be described. FIG. 8 is a flowchart showing a processing procedure of the edge extraction processing (steps S1 and S2) shown in FIG. FIG. 9 is a diagram illustrating an example of an averaging filter for edge extraction used in the smoothing process (step S2) of FIG. This edge extraction processing is executed by the edge extraction unit 30 shown in FIG. In this edge extraction process, the frequency characteristics of the input image 1 and the input image 2 are acquired.

  Referring to FIG. 8, edge extraction unit 30 performs a smoothing process on the input image (step S101). In this smoothing process, an averaged filter for edge extraction as shown in FIG. 9 is used to calculate a value after the averaging process for each pixel of the input image. In the present embodiment, a smoothing filter having a size of 9 pixels × 9 pixels centered on one target pixel is used. An image composed of pixels after the averaging process is referred to as a smoothed image.

  Subsequently, the edge extraction unit 30 performs a difference process between the original input image and the smoothed image (step S102). More specifically, the edge extraction unit 30 calculates a pixel value difference for each pixel between the original input image and the smoothed image, and integrates the absolute value of the calculated difference of each pixel for all the pixels. To do.

  When the input image is blurred, the change from the original input image is small even if smoothing processing is executed using an averaging filter. Using this principle, it can be determined that the larger absolute value of the difference between the original input image and the smoothed image is an input image including more high-frequency components.

  Finally, the edge extraction unit 30 outputs the absolute value (edge amount) of the calculated difference (step S103). It means that more high frequency components are included, so that this edge amount is large.

(C3: edge amount comparison processing and output image determination processing)
Next, details of the edge amount comparison process (step S3) and the output image determination process (step S4) shown in FIG. 6 will be described.

  In the edge amount comparison process, the edge amount calculated by the edge extraction process (step S1) for the input image 1 is compared with the edge amount calculated by the edge extraction process (step S2) for the input image 2. That is, in the edge amount comparison process, it is determined which of the input images has a larger edge amount, that is, a higher frequency component. An input image including more high-frequency components can be determined by determining the size relationship of the edge amounts.

  In the output image determination process (step S4), an input image having a larger edge amount (absolute value of difference) is selected as an output image. In the example illustrated in FIG. 7, it is assumed that the input image 1 is determined to have relatively higher image quality. Depending on the determination result of the output image determination process (step S4), a process (steps S10A to S14A) for generating a stereo image mainly using the input image 1 or the input image 2 is mainly performed. The process (step S10B-S14B) for producing | generating a stereo image using is performed.

(C4: Corresponding point search process and distance image generation process)
Next, corresponding point search processing (steps S10A and S10B) and distance image generation processing (steps S11A and S11B) will be described. In the following description, the case where the input image 1 is mainly used will be described as an example, but the same processing can be performed when the input image 2 is mainly used.

  In the corresponding point search process, the correspondence of the positional relationship between the pair of input images (input image 1 and input image 2) is searched. This corresponding point search processing is executed by the corresponding point search unit 31 shown in FIG. More specifically, in the corresponding point search process, the other input image pixel (coordinate value) corresponding to the target point of one input image is specified. For such a corresponding point search process, a matching process using a POC calculation method, an SAD calculation method, an SSD calculation method, an NCC calculation method, or the like is used.

  In the corresponding point search process, one input image is set as a reference image, and the other input image is set as a reference image, and the images are associated with each other. Depending on which input image is mainly used, the input image set as the reference image is changed.

  Subsequently, a distance image generation for generating a distance image indicating distance information associated with the coordinates of each point of the subject based on the correspondence relationship between the target point identified by the corresponding point search process and the corresponding point Processing is executed. This distance image generation process is executed by the distance image generation unit 32 shown in FIG. In this distance image generation processing, for each attention point, a difference (parallax) between the coordinates of the attention point in the image coordinate system of the input image 1 and the coordinates of the corresponding point in the image coordinate system of the input image 2 is calculated. .

  The calculated parallax is stored in association with the coordinates of the target point of the corresponding input image 1 when the input image 1 is used proactively, and the corresponding input image when the input image 2 is used proactively. It is stored in association with the coordinates of the two points of interest. As the distance information, the coordinates on the input image 1 or the input image 2 and the corresponding parallax are associated with each attention point searched by the corresponding point search process. By arranging this distance information in association with the pixel arrangement of the input image 1 or the input image 2, a distance image representing the parallax of each point is generated corresponding to the image coordinate system of the input image 1 or the input image 2. .

  In addition, as such a corresponding point search process and a distance image generation process, you may employ | adopt the method described in Unexamined-Japanese-Patent No. 2008-216127. Japanese Patent Laid-Open No. 2008-216127 discloses a method for calculating parallax (distance information) with subpixel granularity, but parallax (distance information) may be calculated with pixel granularity. .

  When the input image is a color image such as RGB, the corresponding point search process may be performed after conversion to a gray image.

  FIG. 10 is a diagram showing an example of a distance image generated from the pair of input images shown in FIG. 7 according to the image processing method according to the first embodiment. That is, in FIG. 10 (a), the input image 1 shown in FIG. 7 (a) is set as a standard image, and the input image 2 shown in FIG. 7 (b) is set as a reference image. An example of a distance image (parallax image) obtained by performing is shown. In addition, in FIG.10 (b), an example of the distance image after the smoothing process mentioned later is shown. As shown in FIG. 10, the magnitude of the parallax (distance information) associated with each point of the input image 1 is expressed by the shade of the corresponding point.

  In the corresponding point search process and the distance image generation process described above, the attention point and the corresponding point are specified by performing a correlation operation, and therefore the corresponding point is searched for each unit region having a predetermined pixel size. FIG. 10 shows an example in which the corresponding point search is performed for each unit region of 32 pixels × 32 pixels. That is, in the example shown in FIG. 7, the corresponding points are searched for each unit region defined by an interval of 32 pixels on both the X axis (horizontal direction) and the Y axis (vertical direction). The distance between is calculated. A distance image indicating the distance between the searched corresponding points is generated so as to match the pixel size of the input image. For example, when the input image 1 has a size of 3456 pixels × 2592 pixels, the distance is calculated at search points of 108 points × 81 points, and the pixel size of the input image is calculated from the calculated distances. A corresponding distance image is generated.

  Note that the region (search window) for 32 pixels on the outermost periphery of the input image may be erroneously determined that there is no corresponding point, so the corresponding point search process is not performed and the closest position is determined. The pixel distance (parallax) data is used instead. That is, for the region for 32 pixels at the outermost periphery of the input image, the value of the pixel at the position that is 32 pixels inside from the outermost periphery is used.

(C5: smoothing process)
When the distance image is acquired, smoothing processing (steps S12A and S12B in FIG. 6) is performed on the acquired distance image. This smoothing process is executed by the smoothing processing unit 33 shown in FIG. In this smoothing process, the entire distance image is averaged.

  As an embodiment of such smoothing processing, there is a method using a two-dimensional filter of a predetermined size.

  FIG. 11 is a diagram illustrating an example of an averaging filter used in the smoothing process (steps S12A and S12B) in FIG. In the smoothing process for the distance image, for example, an 81 × 81 pixel averaging filter as shown in FIG. 11 is applied. In the averaging filter, the average value of the pixel values (parallax) of the distance image included in the range of 81 pixels in the vertical direction and 81 pixels in the horizontal direction centering on the target pixel is calculated as a new pixel value of the target pixel. More specifically, a new pixel value of the target pixel is calculated by dividing the sum of the pixel values of the pixels included in the filter by the pixel size of the filter.

  In addition, instead of taking the operation of all the pixels included in the filter, an average value of pixels extracted by thinning out at predetermined intervals (for example, 9 pixels) may be used. Even when such a thinning process is performed, a smoothing result similar to the case where the average value of all pixels is used may be obtained. In such a case, the processing is performed by performing the thinning process. The amount can be reduced.

  FIG. 10B is a diagram illustrating a result of performing the smoothing process on the distance image illustrated in FIG. In the distance image after the smoothing process shown in FIG. 10B, it can be seen that the pixel value (parallax) does not change greatly between adjacent pixels.

  Note that the pixel size of the distance image obtained by the smoothing process is preferably the same pixel size as the input image. By making the pixel size the same, the distance of each pixel can be determined on a one-to-one basis in a stereo image generation process to be described later.

(C6: parallax adjustment processing)
When the distance image after the smoothing process is acquired, the parallax adjustment process is executed so that stereoscopic display can be performed with a more comfortable parallax amount. In this parallax adjustment process, scaling is performed so that the pixel value (parallax) of the distance image after the smoothing process is within a predetermined parallax range (target parallax range).

  FIG. 12 is a diagram for explaining the processing content of the parallax adjustment processing (steps S13A and A13B) illustrated in FIG. In FIG. 12, the horizontal axis represents the amount of parallax (distance) in the distance image after smoothing processing, and the vertical axis represents the amount of parallax (distance) after parallax adjustment processing. In the parallax adjustment process, the parallax adjustment function in FIG. 12 is determined.

  As a more specific procedure, first, the size of the target parallax range r is determined. This target parallax range r is determined empirically based on the width of the input image. For example, in the case of an input image of 1920 pixels × 1080 pixels, it is determined to be 49 pixels. The target parallax range r may be determined dynamically or a fixed value set in advance may be used.

  Further, the maximum value of the parallax (distance) amount included in the distance image after the smoothing process is set as the maximum parallax Pmax, and the minimum value is set as the minimum parallax Pmin. In other words, the image of the pixel having the maximum parallax Pmax is projected out and displayed stereoscopically, and the image of the pixel having the minimum parallax Pmin is displayed stereoscopically so as to exist at the depth. In the parallax adjustment processing, between the maximum parallax Pmax (the position that protrudes most in the direction orthogonal to the display surface) and the minimum parallax Pmin (the position on the deepest side in the direction orthogonal to the display surface (Z-axis direction)). A parallax increase / decrease coefficient c that is the inclination of the parallax adjustment function and an offset o that is an intercept of the parallax adjustment function are calculated so that the range matches the target parallax range r. More specifically, the parallax increase / decrease coefficient c and the offset o are calculated according to the following equations.

Parallax increase / decrease coefficient c = target parallax range r / (maximum parallax Pmax−minimum parallax Pmin)
Offset o = (maximum parallax Pmax + minimum parallax Pmin) / 2
Post-adjustment parallax (distance) amount = parallax increase / decrease coefficient c × (pre-adjustment parallax (distance) amount−offset o)
(C7: Stereo image generation process)
When the distance image after the smoothing process is acquired, a stereo image generation process (steps S14A and S14B in FIG. 6) is executed using the acquired distance image. This stereo image generation process is executed by the 3D image generation unit 35 shown in FIG.

  In the stereo image generation process, when the input image 1 is mainly used, the input image 1 is output as it is as the left-eye image, and each pixel of the input image 1 is positioned according to the corresponding distance (parallax). The right eye image is generated by shifting. On the other hand, when the input image 2 is mainly used, the input image 2 is output as it is as an image for the right eye, and the position of each pixel of the input image 2 is shifted according to the corresponding distance (parallax). The left eye image is generated.

  Note that in order to stereoscopically display the subject, it is only necessary that the corresponding pixels be separated from the left-eye image and the right-eye image by a specified distance (parallax). A commercial image and a right eye image may be generated.

  FIG. 13 is a diagram for explaining a processing procedure in the stereo image generation processing (steps S14A and S14B) in FIG. FIG. 13 shows a processing example when the input image 1 is used proactively. FIG. 14 is a flowchart showing a processing procedure of the stereo image generation processing shown in FIG.

  Referring to FIG. 13, in the stereo image generation process, a stereo image (a left-eye image and a right-eye image) is generated from an input image used mainly based on a distance image. More specifically, the other right-eye image or left-eye image is generated by shifting the pixel position in units of lines constituting the input image that is used mainly. FIG. 13 shows an example in which the input image 1 is used as it is as the left eye image and the input image 2 is used as the right eye image. FIG. 13 shows 10 pixels whose pixel positions (coordinates) are “101”, “102”,..., “110” for a line of the input image 1 used as the left-eye image. Yes. The distance (parallax) corresponding to the pixel at each pixel position is “40”, “40”, “41”, “41”, “41”, “42”, “42”, “41”, “40” It is assumed that “40”. Using these pieces of information, the shifted pixel position (coordinates in the right-eye image) is calculated for each pixel. More specifically, the pixel position after shifting for each pixel for one line is calculated according to (pixel position after shifting) = (coordinates in the image for the left eye) − (corresponding distance (parallax)). .

  Then, a corresponding one line image of the right-eye image is generated based on each pixel value and the corresponding shifted pixel position. At this time, there may be no corresponding pixel depending on the value of the distance (parallax). In the example illustrated in FIG. 13, there is no information on the pixels at the pixel positions “66” and “68” of the right-eye image. In such a case, the pixel values of the deficient pixels are interpolated using information from adjacent pixels.

  By repeating such processing for all lines included in the input image, the right-eye image is generated.

  Note that the direction in which the pixel position is shifted is a direction in which parallax should be generated, and specifically corresponds to a direction that becomes the horizontal direction when displayed toward the user.

  Such a processing procedure is shown in FIG. That is, with reference to FIG. 14, the 3D image generation unit 35 (FIG. 1) calculates the pixel position after each shift for pixels for one line of the input image 1 (step S <b> 1401). Subsequently, the 3D image generation unit 35 generates an image for one line (image for the right eye) from the shifted pixel position calculated in step S1 (step S1402).

  Thereafter, the 3D image generation unit 35 (FIG. 1) determines whether or not there is an unprocessed line in the input image (step S1403). If there is an unprocessed line in the input image (NO in step S1403), the next line is selected, and the processes in steps S1401 and S1402 are repeated.

  If the processing has been completed for all the lines of the input image (YES in step S1403), the 3D image generation unit 35 outputs the generated right eye image together with the input image 1 (left eye image). Then, the process ends.

  13 and 14 describe the processing when the input image 1 is determined as the output image, that is, when the input image 1 is mainly used. However, when the input image 2 is determined as the output image, That is, the same applies to the processing when the input image 2 is mainly used. However, in the above-described corresponding point search process, the relationship between the base image and the reference image is switched.

  FIG. 15 is a diagram showing an example of a stereo image generated from the pair of input images shown in FIG. 7 according to the image processing method according to the first embodiment. As can be seen by comparing FIG. 7 and FIG. 15, it can be seen that a clear stereo image without blur is obtained.

  That is, by adopting a series of processes as described above, a stereo image to be output is generated using an input image with relatively good image quality (not blurred). Even when there is a defect, the quality of stereoscopic display can be maintained.

(C8: Modification)
In the image processing method described above, it is assumed that defects such as blur included in the input image are detected based on the frequency characteristics of the input image. However, overexposure in the input image is performed based on the frequency characteristics of the input image. Can also be detected. For example, if only one input image is partially whitened, there is no high frequency component. Therefore, overexposure can be detected by the same method as described above. Further, when strong light is incident on the lens, a ring called a flare or a ghost or a ball-like blur may be photographed only on one side due to reflection inside the lens. Also in this case, since high frequency components are reduced by bleeding, these flares and ghosts can be detected by the same method as described above.

  A stereo image can be generated from an input image in which such overexposure, flare, and ghost are not generated.

<D. Second Embodiment (Vertical Stereo / Horizontal Stereo)>
In the first embodiment described above, when the vertical stereo shown in FIG. 2A can be used, the processing mode that mainly uses the input image 1 and the processing mode that mainly uses the input image 2 are provided. An example of selective execution was shown. Here, when the horizontal stereo shown in FIG. 2B can also be used, the input image 1 and the input image 2 can be used as they are as a stereo image. Typically, as shown in FIG. 5, the mobile phone 300 equipped with the imaging unit 2 can be either vertical stereo or horizontal stereo depending on how the user holds (the orientation direction of the mobile phone 300). .

  Therefore, in the second embodiment, a configuration in which such three processing modes can be selected will be exemplified. That is, when a stereo image is generated, whether to use one input image or two input images is switched.

  FIG. 16 is a diagram showing the procedure of the image processing method according to the second embodiment. In FIG. 16, the same process as the image processing method shown in FIG. 6 is given the same step number. Compared with the image processing method according to the first embodiment shown in FIG. 6, the image processing method shown in FIG. 16 has newly added an output mode determination process (step S5) and a stereo image output process (step S7). Since other processes are the same as those in the first embodiment, detailed description will not be repeated.

  In the image processing method according to the present embodiment, a processing mode in which one of input image 1 and input image 2 is mainly used to generate a stereo image, and both the input image 1 and input image 2 are used to record a stereo image. Switch between processing modes to generate. The processing mode is switched based on the frequency characteristics acquired from the input image 1 and the input image 2 and the positional relationship between the first camera 21 and the second camera 22 at the time of imaging.

  More specifically, as the frequency characteristics acquired from the input image 1 and the input image 2, an absolute value and a relative difference for the edge amount of each input image calculated by the edge extraction process are used. In addition, posture information acquired by the acceleration sensor 116 (FIG. 3) is used for the positional relationship between the first camera 21 and the second camera 22 at the time of imaging. That is, it is determined whether the imaging unit 2 is captured in a vertical stereo state or a horizontal stereo state.

  Referring to FIG. 16, when input image 1 and input image 2 are acquired, edge extraction processing (steps S1 and S2) is performed on these input images, and the edge amount for each input image is determined. Calculated. Since this edge extraction processing (edge amount calculation) is the same as that in the first embodiment described above, detailed description will not be repeated.

  Subsequently, an edge amount comparison process (step S3) is executed based on the calculated edge amounts. In this edge amount comparison process, it is determined whether or not a defect such as blur exists in only one of the input image 1 and the input image 2. More specifically, it is determined whether or not the difference between the edge amount of the input image 1 and the edge amount of the input image 2 is equal to or less than a predetermined threshold value. When the difference in edge amount is equal to or less than the threshold value, it can be determined that there is no significant difference in the degree of blur between the input image 1 and the input image 2, so that the input image 1 and the input image 2 are used as they are as stereo images. Output as.

  However, the edge amount comparison process (step S3) is executed when it can be determined from the posture information that the first camera 21 and the second camera 22 are in horizontal stereo. This is because when the first camera 21 and the second camera 22 are in a vertical stereo, the input image 1 and the input image 2 cannot be used as they are as a stereo image.

  That is, only when the first camera 21 and the second camera 22 are in the horizontal stereo and the difference between the edge amount of the input image 1 and the edge amount of the input image 2 is equal to or less than the threshold value, the stereo is limited. Image output processing (step S7) is executed. In this stereo image output process (step S7), the input image 1 and the input image 2 are output as they are as a stereo image.

  On the other hand, if the difference between the edge amount of the input image 1 and the edge amount of the input image 2 exceeds the threshold value, it can be determined that one of the input images is blurred. As in the first embodiment, output image determination processing is executed (step S4).

  Further, even when the first camera 21 and the second camera 22 are in vertical stereo, the input image 1 and the input image 2 cannot be used as they are as a stereo image, so that an output image determination process is executed. (Step S4).

  In the output image determination process, an input image with a larger amount of edge is selected, and a stereo image is generated mainly using the selected input image. Since the processing after execution of the output image determination processing is the same as that in the first embodiment described above, detailed description will not be repeated.

<E. First Modification (Movie)>
The image processing methods according to the first and second embodiments described above can be applied to both still images and moving images, but when applied to moving images, the following processing may be added. .

  That is, in the image processing method according to the present modification, when the first camera 21 and the second camera 22 capture a moving image of a subject, a processing mode to be applied for generating a stereo image is determined for each of a plurality of frames. A moving image is composed of a series of frames that are temporally continuous. In such moving image capturing, processing mode for each frame ((1) generation of a stereo image mainly using the input image 1, (2) generation of a stereo image mainly using the input image 2, (3) If the generation of the stereo image using the input image 1 and the input image 2 as they are is different, the stereo image generation rule is switched in a short time. As a result, the continuity of the output stereo image is impaired, and there is a possibility that the user feels uncomfortable.

  Therefore, in the image processing method according to the present modification, when the first camera 21 and the second camera 22 capture a moving image of the subject, the processing mode once determined is set over a predetermined period (typically, a plurality of frames). maintain. As a typical example of maintaining such a processing mode, there is a method of determining a processing mode (stereo image generation rule) based on the first frame of a certain section. That is, in the image processing method according to the present modification, when the first camera 21 and the second camera 22 capture a moving image of the subject, the processing mode is determined based on the input image 1 and the input image 2 corresponding to the first frame. .

  More specifically, the frequency characteristics are acquired by performing edge extraction processing for each of the input image 1 and the input image 2 corresponding to the first frame in a predetermined section (including a plurality of frames) constituting the moving image. . Then, using the acquired frequency characteristic (edge amount), the stereo image processing mode is determined according to the image processing method according to the first embodiment or the second embodiment described above. Once the processing mode is determined, stereo images are sequentially generated from the input image 1 and the input image 2 corresponding to subsequent frames (after the second frame) according to the determined processing mode. For a series of frames, a stereo image generation process mainly including the input image 1 (steps S11A to S14A), a stereo image generation process mainly including the input image 2 (steps S11B to S14B), the input image 1 and The process of outputting the input image 2 as a stereo image (step S7) is repeatedly executed.

  In this way, by applying the same generation rule to a series of frames, it is possible to maintain the continuity of the output image and provide a natural stereoscopic display (moving image) to the user.

  Note that the processing mode determined based on the first frame may be applied to all frames included in one moving image, but using human vision, the processing mode is determined in units of scenes included in the moving image. The generation rule may be changed. For example, by using a technique disclosed in Japanese Patent Application Laid-Open No. 2002-152669, a scene change in a moving image is detected to distinguish the scene, and a processing mode (scene switching timing) is set in the first frame of each scene (scene switching timing). (Generation rule) may be determined (changed). Alternatively, the processing mode (generation rule) may be determined (changed) every certain time (for example, 600 frames at 30 fps).

  As described above, in this modification, the generation rule determination condition is switched between a still image and a moving image so that the moving image is not switched more frequently than the still image. That is, in the case of a moving image, the generation rule is maintained for a certain time. More specifically, in the case of a moving image, it is preferable that the generation rule is determined only in the first frame.

<F. Second Modification (Evaluation for Each Partial Image)>
In the first and second embodiments described above, the configuration for evaluating the overall image quality of the input image has been exemplified, but the image quality may be evaluated for a partial image of the input image. Then, the processing mode for generating a stereo image for each partial image may be switched. That is, the process is switched for each partial region of the input image.

  Basically, the configuration in which the image quality is evaluated for each partial image and the processing mode is switched is the horizontal stereo, that is, the input image 1 and the input image 2 acquired by the first camera 21 and the second camera 22 respectively. It is suitable when it can be used as a stereo image. Of course, the method of this modification can also be applied as preprocessing for an input image in vertical stereo.

  FIG. 17 is a diagram illustrating a processing example in which the image quality is evaluated for each partial image according to the second modification. 18 and 19 are diagrams for describing processing for generating a stereo image according to the second modification.

  As shown in FIG. 17A, the input image may be divided into a plurality of partial areas, and the frequency characteristics of the partial images corresponding to the partial areas may be evaluated. In the example shown in FIG. 17A, a total of 12 partial images are set in which the longer side of the input image is divided into four and the shorter side is divided into three. As described above, by evaluating the frequency characteristics for each partial image, it is possible to improve the accuracy of determination of defects such as whiteout due to overexposure, flare, and ghost that occur partially.

  As a more specific method for evaluating the frequency characteristics for each partial region, first, the edge amount for each partial image shown in FIG. 17A is determined according to the same procedure as in the first and second embodiments described above. Calculated. Then, intermediate determination is performed for each partial image based on the edge amount. More specifically, it is determined whether the absolute value of the difference regarding the edge amount for the same partial image is equal to or less than a predetermined threshold value between the input image 1 and the input image 2. That is, it is determined whether the same tendency is shown between the input image 1 and the input image 2 when the corresponding partial image is viewed. When the same tendency is shown, it is determined whether or not the partial image is reliable by evaluating the absolute value of the difference with reference to the absolute value of the difference with respect to another partial image.

  On the other hand, when the edge amount is greatly different only for the corresponding partial images between the input image 1 and the input image 2, the partial image having a relatively large edge amount is selected. When the absolute value of the difference is relatively large, the input image 1 and the input image 2 tend to be in conflict with each other. In this case, a partial image showing a larger edge amount is generated as a stereo image. Used for.

  In addition, since the viewpoint with respect to a to-be-photographed object differs between the 1st camera 21 and the 2nd camera 22, when the difference of the visual field range between both cameras, ie, the occlusion has generate | occur | produced, input image 1 And the input image 2 may reflect different subjects. In order to prevent such an adverse effect, as shown in FIG. 18, a correspondence relationship between the input image 1 and the input image 2 is acquired in advance using a method such as pattern matching, and each corresponding portion is obtained. You may make it evaluate a frequency characteristic between images.

  That is, when the first camera 21 and the second camera 22 capture the subject from different viewpoints, as shown in FIG. 18, the partial image set in the input image 1 and the partial image set in the input image 2 Therefore, it is possible to search a region corresponding to each partial image from other input images after setting a plurality of partial images for one input image.

  As described above, the procedure for generating a stereo image after evaluating the image quality of each partial image based on the frequency characteristics is as shown in FIG. In this example, in the horizontal stereo configuration, the input image 1 acquired by the first camera 21 capturing an image of the subject is used as the left-eye image as it is, and the second camera 22 is acquired by capturing the image of the subject. The input image 2 is used as a right-eye image while being appropriately corrected. FIG. 19 shows a processing example in which the partial image included in the input image 2 is interpolated using the corresponding partial image of the input image 1. That is, when it is determined that a defect such as flare or ghost exists only in a specific partial image of the input image 2, flare or ghost exists only in the partial image (region) determined to have the problem. Interpolation is performed using a partial image of the input image that is not performed. That is, the right-eye image is generated by partially transplanting the partial image of the input image 1 with respect to the input image 2.

  Thus, by evaluating the image quality for each partial image, it is possible to generate a high-quality stereo image while repairing defects such as flare and ghost that are partially generated.

<G. Third Modification (User Interface)>
In the image processing system / device in which the above-described image processing method is mounted, it is preferable to mount a user interface as described below.

  FIG. 20 is a diagram illustrating an example of a user interface provided in the third modification.

(G1: Warning to users)
As described above, in the image processing method according to the present embodiment, the image quality of the input image can be evaluated, so that the camera (lens) corresponding to the input image with degraded image quality can be specified. Therefore, equipped with warning means to warn the user of lens contamination for the corresponding camera when it is determined multiple times that the image quality has deteriorated with respect to the input image from a specific camera (imaging means) May be. That is, if it is determined that the quality of the input image from one camera (lens) is always bad (the calculated edge amount is small) continuously, the lens surface on one side may be dirty. Therefore, a warning message may be presented that prompts the user to wipe the lens. In other words, if the quality of only the input image from the same lens is poor in continuous imaging (at least twice), a warning is given that the lens surface may be dirty.

As such a warning message, the contents shown in FIG.
(G2: Arousing imaging operations to users)
In order to evaluate the first camera 21 and the second camera 22 as described above, it is necessary to actually image the subject using the first camera 21 and the second camera 22. Therefore, in order to evaluate the frequency characteristics with respect to the input image 1 and the input image 2, an imaging arousing unit that prompts the user to perform imaging using the first camera 21 and the second camera 22 may be installed. That is, the user is prompted to take an input image to determine the first camera 21 and the second camera 22 (both lenses).

  As such an arousing message, the content as shown in FIG. As the determination chart, a sample in which a specific edge exists on the entire surface of the image (for example, an image of the whole grass) is used.

  It is not always necessary to use a chart. Whether or not the actual image of the subject is picked up by the user and the edge amount of the input image obtained by picking up the subject is greater than or equal to a predetermined threshold value Judging. If the edge amount is equal to or greater than a predetermined threshold value, it is determined that the input image (subject) is suitable for the evaluation of the first camera 21 and the second camera 22 and may be used as a chart substitute. In this case, as a substitute for the chart, this subject is also displayed on the monitor on the imaging device so that it can be understood at a glance which subject was used to evaluate the first camera 21 and the second camera 22. It is preferable to do.

(G3: Default setting)
Regarding the evaluation of the first camera 21 and the second camera 22 as described above, based on the input image 1 and the input image 2 acquired at the time of initial setting, when the power is turned on, or at the focusing operation immediately before imaging. Preferably, it is done.

  By acquiring the frequency characteristics of the input image 1 and the input image 2 acquired at the time of initial setting, it is possible to easily find an initial defect such that the lens surface on one side is manufactured with a slight inclination at the time of manufacturing the lens. In addition, it is easy to find a case where the quality of the lens on one side is not good and is always slightly blurred. In addition to such initial setting, the first camera 21 and the second camera 22 may be evaluated on a regular basis such as when the power is turned on or when a focusing operation is performed immediately before imaging (shutter half-pressed state).

  Further, a default camera (lens) may be determined based on the result of evaluating the first camera 21 and the second camera 22. When the default camera (lens) has been determined in this way, it is not necessary to evaluate the first camera 21 and the second camera 22 for each imaging, and the camera is acquired with a preset default camera (lens). A stereo image is generated mainly using the input image.

  That is, a camera (lens) corresponding to an input image including more high-frequency components is set as a default as an imaging unit that is mainly used when generating a stereo image. Then, a stereo image is generated mainly using the input image from the camera (lens) set as default.

<H. Modification of frequency characteristics acquisition method)>
As described above, in the image processing method according to the present embodiment, the image quality is evaluated based on the frequency characteristics of the input image. Therefore, the present invention is not limited to the edge extraction process as described above, and various methods for acquiring frequency characteristics can be employed.

  As a process for acquiring such frequency characteristics, a method using Fourier transform as disclosed in JP 2011-128926 A may be employed. Alternatively, a primary differential filter or a secondary differential filter such as Sobel may be used.

  Further, in order to improve the determination accuracy for the frequency characteristics included in the input image, a process of using a median or the like for noise removal may be included. Alternatively, after converting the pixel component of the input image into frequency characteristics, the image quality may be evaluated after ignoring the noisy frequency band such as the highest frequency region.

<I. Advantage>
According to the embodiment of the present invention, even if the input image has a defect such as blur, the input image having the better image quality is mainly (preferential) so that the defect is not shown to the user. To generate a stereo image. Therefore, even if imaging of one input image has failed for some reason, the quality of stereoscopic display can be maintained by generating a stereo image from the one with the best image quality.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  DESCRIPTION OF SYMBOLS 1 Image processing system, 2 Imaging part, 3 Image processing part, 4 Image output part, 21,121 1st camera, 21a, 22a Lens, 21b, 22b Image sensor, 22,122 2nd camera, 23, 24 A / D Conversion unit, 30 edge extraction unit, 31 corresponding point search unit, 32 distance image generation unit, 33 smoothing processing unit, 34 parallax adjustment unit, 35 image generation unit, 100 digital camera, 102 CPU, 104 digital processing circuit, 106 image processing Circuit, 108 Image display unit, 112 Storage unit, 114 Zoom mechanism, 116 Acceleration sensor, 200 Personal computer, 202 Personal computer main body, 204 Image processing program, 206 Monitor, 208 Mouse, 210 Keyboard, 212 External storage device, 300 Mobile phone .

Claims (15)

First imaging means for imaging a subject and obtaining a first input image;
Second imaging means for capturing the subject from a different viewpoint from the first imaging means to obtain a second input image;
Frequency characteristic acquisition means for acquiring frequency characteristics of the first and second input images;
A stereo image for stereoscopically displaying the subject from the first and second input images by mainly using an input image determined to have relatively good image quality based on the acquired frequency characteristics. An image processing system comprising: a stereoscopic generation unit that generates   The image processing system according to claim 1, wherein the stereoscopic generation unit determines that an input image including more high-frequency components has a relatively high image quality.   The image processing system according to claim 1, wherein the frequency characteristic acquisition unit acquires the frequency specification using at least one of extraction of an edge amount included in the input image and frequency analysis on the input image.   The stereoscopic vision generating means mainly uses one of the first and second input images based on the acquired frequency characteristics and the positional relationship between the first and second imaging means at the time of imaging. 4. The method according to claim 1, wherein the processing mode for generating the stereo image is used to switch between the processing mode for generating the stereo image using both the first and second input images. The image processing system according to item.   The image processing system according to claim 1, wherein the stereoscopic generation unit switches a processing mode for generating the stereo image for each partial image.   The stereoscopic viewing generation unit determines a processing mode to be applied to generate the stereo image for each of a plurality of frames when the first and second imaging units capture a moving image of the subject. The image processing system according to any one of 1 to 5.   The image processing system according to claim 6, wherein the stereoscopic generation unit maintains the processing mode once determined for a predetermined period when the first and second imaging units capture the moving image of the subject. .   The stereoscopic vision generating unit determines a processing mode based on first and second input images corresponding to a first frame when the first and second imaging units capture a moving image of the subject. Item 8. The image processing system according to Item 6 or 7.   And a warning unit configured to warn the user of lens contamination of the corresponding imaging unit when it is determined a plurality of times that the image quality of the input image from the specific imaging unit is deteriorated. The image processing system according to any one of 1 to 8.   The image processing according to any one of claims 1 to 9, further comprising an imaging inducing unit that prompts a user to perform imaging using the first and second imaging units for evaluating the frequency characteristic. system.   The frequency characteristic acquisition means acquires the frequency characteristic with respect to the first and second input images acquired at any of initial setting, power-on, and focusing operation immediately before imaging. The image processing system according to any one of 1 to 10.   The image according to any one of claims 1 to 11, wherein an imaging unit corresponding to an input image including more high-frequency components is default-set as an imaging unit that is mainly used when generating the stereo image. Processing system.   The image processing system according to claim 12, wherein the stereoscopic vision generation unit generates the stereo image mainly using an input image from an imaging unit set as a default. Imaging a subject to obtain a first input image;
Capturing the subject from a viewpoint different from the viewpoint from which the first input image was captured, and obtaining a second input image;
Obtaining frequency characteristics of the first and second input images;
A stereo image for stereoscopically displaying the subject from the first and second input images by mainly using an input image determined to have relatively good image quality based on the acquired frequency characteristics. Generating an image processing method. An image processing program for causing a computer to execute image processing, wherein the image processing program
First acquisition means for acquiring a first input image obtained by imaging a subject;
Second acquisition means for acquiring a second input image obtained by imaging the subject from a different viewpoint from the viewpoint obtained by imaging the first input image;
Frequency characteristic acquisition means for acquiring frequency characteristics of the first and second input images;
A stereo image for stereoscopically displaying the subject from the first and second input images by mainly using an input image determined to have relatively good image quality based on the acquired frequency characteristics. An image processing program that functions as a stereoscopic generation unit that generates the image.