WO2013047641A1

WO2013047641A1 - Three-dimensional image processing device and three-dimensional image processing method

Info

Publication number: WO2013047641A1
Application number: PCT/JP2012/074843
Authority: WO
Inventors: 毅潘
Original assignee: 富士フイルム株式会社
Priority date: 2011-09-29
Filing date: 2012-09-27
Publication date: 2013-04-04

Abstract

In the three-dimensional image processing device relating to one embodiment of the present invention, a histogram of the distance distribution from a three-dimensional image to a subject is created, and compared to patterns of distance distribution. In the event of a determination that the histogram approximates a "pattern of distance distribution suited to three-dimensional imaging", a suggested parallax adjustment value, which has been set for the distance distribution pattern in question, is set as the parallax adjustment value. Additionally, the user can set the parallax adjustment value manually, and the amount of parallax can be learned. In the event of a determination that the created histogram "does not approximate a pattern suited to three-dimensional imaging", a parallax adjustment value can be set so that the parallax of the focal subject is zero.

Description

Stereoscopic image processing apparatus and stereoscopic image processing method

The present invention relates to a stereoscopic image processing apparatus, a stereoscopic imaging apparatus, a stereoscopic image reproduction apparatus, a stereoscopic image processing method, and a stereoscopic image processing program that perform parallax adjustment of a stereoscopic image.

In a device such as a digital camera capable of capturing and displaying a stereoscopic image, parallax is adjusted when capturing and displaying the stereoscopic image. For example, Patent Document 1 describes that a favorable stereoscopic effect can be obtained by preventing imaging conditions such as left and right in-focus positions / exposure levels from being shifted. In addition, in taking and displaying a stereoscopic image, it is possible to set the amount of parallax for each user in consideration of the fact that an appropriate amount of parallax varies from user to user. For example, Patent Document 2 includes a parallax amount holding unit that holds the parallax amount of each user, and each user selects and captures and displays his / her parallax amount. Patent Document 3 describes that parallax amount data for each user is stored and the parallax amount is specified from an answer to a question to the user.

JP 2011-39486 A Japanese Patent Laid-Open No. 2002-232913 JP 2004-289527 A

However, it is difficult to adjust the amount of parallax according to the shooting scene with the conventional technique described in Patent Documents 1-3. Even when the user sets the amount of parallax, it is difficult to set an appropriate amount of parallax only by the user's own judgment, or when a user wants to set a different amount of parallax for each shooting. In some cases, it has been difficult to cope with such a situation with the prior art.

The present invention has been made based on such circumstances, and a stereoscopic image processing apparatus, a stereoscopic imaging apparatus, a stereoscopic image reproduction apparatus, and a stereoscopic image processing method capable of setting an appropriate parallax adjustment value in consideration of a shooting scene and user preference. And a stereoscopic image processing program.

In order to achieve the above object, the stereoscopic image processing apparatus of the present invention includes a stereoscopic image acquisition unit that acquires a stereoscopic image of a subject, and a distance from the acquired stereoscopic image to one or more subjects or a value corresponding to the distance. A distance distribution calculating means for calculating the distribution of the distance as a distance distribution, a distance distribution matching means for matching the calculated distance distribution with a plurality of patterns of the distance distribution stored in advance, and a three-dimensional image obtained based on the matching result A stereoscopic image processing apparatus comprising: a parallax adjustment value setting unit that sets a parallax adjustment value; and a recording unit that records an image to which the parallax adjustment value set to the acquired stereoscopic image is applied. Set the parallax adjustment value by checking whether or not the distance to the main subject in the calculated distance distribution belongs to the distance range to the main subject in multiple patterns Stage, the result of the collation, when the calculated distance distribution is determined to be similar to any of a plurality of patterns, for the acquired three-dimensional image, sets a parallax adjustment value corresponding to the pattern determined to be similar.

The appropriate parallax adjustment value for shooting and displaying a stereoscopic image varies depending on the shooting and display conditions, but also varies depending on the number of subjects and the distance distribution to the subjects. Therefore, the stereoscopic image processing apparatus of the present invention compares the distance distribution from the acquired stereoscopic image to the calculated subject with a plurality of predetermined patterns, and sets the parallax adjustment value according to the pattern determined to be similar. Thus, it is possible to set an appropriate parallax adjustment value according to the shooting scene. Such a configuration is effective when it is difficult to set an appropriate parallax adjustment value only by the user's own judgment. The “plural patterns” can be set in consideration of whether or not the distribution of distances to the subject is suitable for capturing a stereoscopic image. Further, the recommended value of the parallax adjustment value may be set as a default setting value (default value) for each pattern of the distance distribution. In addition, examples of the “value corresponding to the distance” include parallax.

The stereoscopic image processing apparatus of the present invention further includes a parallax adjustment value input unit that receives an instruction input of a parallax adjustment value from a user, and the parallax adjustment value setting unit is configured to match the received parallax adjustment value according to a pattern that is determined to be similar. The parallax adjustment value may be set. As described above, in the present invention, the parallax adjustment value is set according to the shooting scene by setting the parallax adjustment value in consideration of the distribution of the distance to the subject. By making it possible to set a parallax amount according to the user's preference, a more appropriate parallax adjustment value can be set.

In the stereoscopic image processing apparatus of the present invention, when the parallax adjustment value input unit receives the input of the parallax adjustment value, the parallax adjustment value setting unit generates a new parallax based on the received parallax adjustment value and the setting history of the parallax adjustment value. An adjustment value may be calculated and set as a parallax adjustment value according to a pattern determined to be similar, and the setting history of the parallax adjustment value may be updated. With such a configuration, it is possible to set an appropriate parallax adjustment value that reflects the tendency of the parallax adjustment value setting by the user while considering the parallax adjustment value according to the pattern of the distance distribution.

The stereoscopic image processing apparatus of the present invention further includes user identification means for identifying a plurality of users, and the parallax adjustment value setting means sets a parallax adjustment value according to which of the plurality of users is the identified user. You may do it. With such a configuration, it is possible to set a parallax adjustment value corresponding to each user's preference difference.

In the stereoscopic image processing apparatus of the present invention, the parallax amount setting unit determines that the calculated subject is not similar to any of the plurality of patterns as a result of the collation, and the in-focus subject is obtained for the acquired stereoscopic image. The parallax adjustment value may be set so that the parallax becomes zero. Depending on the setting of the distance distribution to the subject and the distance distribution pattern in the acquired stereoscopic image, the two may not be similar, but in this mode, the parallax adjustment value is set and the stereoscopic image is recorded in such a case. can do.

In the stereoscopic image processing apparatus of the present invention, the distance distribution calculated by the distance distribution calculating means may be a histogram of distances to the subject. The “distance distribution pattern” to be compared with the histogram may be a histogram or another format. Further, the calculated histogram and “distance distribution pattern” may be converted into another format and compared. In the stereoscopic image processing apparatus of the present invention, the distance distribution calculated by the distance distribution calculating unit may be a parallax histogram of the subject.

In the stereoscopic image processing apparatus of the present invention, the parallax adjustment value may be a value for adjusting the parallax of the subject at a predetermined distance. The parallax adjustment value may be a value for adjusting the parallax of the main subject.

The stereoscopic image processing apparatus of the present invention may further include stereoscopic image display means for displaying an image to which the parallax adjustment value set for the acquired stereoscopic image is applied.

In order to achieve the above object, a stereoscopic imaging apparatus of the present invention includes a stereoscopic image acquisition unit that acquires a stereoscopic image of a subject, and a distance from the acquired stereoscopic image to one or more subjects or a distribution of values corresponding to the distance. A distance distribution calculating means for calculating the distance distribution as a distance distribution, a distance distribution matching means for matching the calculated distance distribution with a plurality of patterns of the distance distribution stored in advance, and a parallax adjustment for the acquired stereoscopic image based on the matching result A stereoscopic imaging device comprising: a parallax adjustment value setting unit that sets a value; and a recording unit that records an image to which the parallax adjustment value set to the acquired stereoscopic image is applied, wherein the distance distribution matching unit calculates the calculated distance The parallax adjustment value setting means is configured to perform collation in consideration of whether the distance to the main subject in the distribution belongs to the distance range to the main subject in a plurality of patterns. As a result, when the distribution of the calculated distance is determined to be similar to any of a plurality of patterns, for the acquired three-dimensional image, it sets a parallax adjustment value corresponding to the pattern determined to be similar.

In order to achieve the above object, a stereoscopic image reproduction device according to the present invention includes a stereoscopic image acquisition unit that acquires a stereoscopic image of a subject, and a distance from the acquired stereoscopic image to one or more subjects or a value corresponding to the distance. Distance distribution calculating means for calculating the distribution as a distance distribution, distance distribution matching means for matching the calculated distance distribution with a plurality of patterns of the distance distribution stored in advance, and the parallax for the acquired stereoscopic image based on the matching result Display parallax adjustment value setting means for setting an adjustment value, recording means for recording an image to which the parallax adjustment value set for the acquired stereoscopic image is applied, and an image to which the parallax adjustment value set for the acquired stereoscopic image is applied A distance distribution matching means, wherein the distance to the main subject in the calculated distance distribution includes a plurality of patterns. The parallax adjustment value setting means determines that the calculated distance distribution is similar to any of the plurality of patterns as a result of the collation The parallax adjustment value corresponding to the pattern determined to be similar to the acquired stereoscopic image is set.

In order to achieve the above object, a stereoscopic image processing method of the present invention includes a stereoscopic image acquisition step of acquiring a stereoscopic image of a subject, and a distance from the acquired stereoscopic image to one or more subjects or a value corresponding to the distance. A distance distribution calculating step for calculating the distribution as a distance distribution, a distance distribution matching step for matching the calculated distance distribution with a plurality of patterns of the distance distribution stored in advance, and for the acquired stereoscopic image based on the matching result A stereoscopic image processing method comprising: a parallax adjustment value setting step for setting a parallax adjustment value; and a recording step for applying and recording the parallax adjustment value set for the acquired stereoscopic image, wherein the calculation is performed in the distance distribution matching step. Check whether the distance to the main subject in the measured distance distribution belongs to the distance range to the main subject in multiple patterns. In the parallax adjustment value setting step, when it is determined that the calculated distance distribution is similar to any of the plurality of patterns as a result of the collation, the parallax adjustment according to the pattern determined to be similar to the acquired stereoscopic image Set the value.

The stereoscopic image processing method of the present invention further includes a parallax adjustment value input step for receiving an instruction input by a user for a parallax adjustment value for the acquired stereoscopic image, and the parallax adjustment value setting unit receives the parallax adjustment value input unit. The value may be set as a parallax adjustment value according to a pattern determined to be similar.

In the stereoscopic image processing method of the present invention, in the parallax adjustment value setting step, when the parallax adjustment value input unit receives the input of the parallax adjustment value, a new parallax is generated based on the received parallax adjustment value and the setting history of the parallax adjustment value. An adjustment value may be calculated and set as a parallax adjustment value according to a pattern determined to be similar, and the setting history of the parallax adjustment value may be updated.

In order to achieve the above object, the stereoscopic image processing program of the present invention causes a computer to execute the stereoscopic image processing method according to any of the above aspects. A recording medium on which a computer-readable code of such a stereoscopic image processing program is recorded is also included in one aspect of the present invention.

As described above, according to the stereoscopic image processing device, the stereoscopic imaging device, the stereoscopic image reproducing device, the stereoscopic image processing method, and the stereoscopic image processing program according to the present invention, an appropriate parallax adjustment value in consideration of the shooting scene and user preference. Can be set.

FIG. 1 is an image diagram showing an appearance of a stereoscopic image processing apparatus 2 according to the first embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of the stereoscopic image processing apparatus 2. FIG. 3 is a flowchart showing an example of parallax adjustment value setting processing according to the present invention. FIG. 4A is a diagram illustrating an example of a histogram of distance distribution to a subject. FIG. 4B is a diagram illustrating another example of a histogram of distance distribution to a subject. FIG. 5A is a conceptual diagram illustrating an example of comparison between a distance distribution histogram and a distance distribution pattern. FIG. 5B is another conceptual diagram illustrating an example of comparison between a distance distribution histogram and a distance distribution pattern. FIG. 5C is still another conceptual diagram illustrating an example of comparison between a distance distribution histogram and a distance distribution pattern. FIG. 6 is a table showing an example of a pattern of distance distribution to a subject and setting of a parallax adjustment value. FIG. 7 is a flowchart showing another example of the parallax adjustment value setting process according to the present invention. FIG. 8 is a flowchart showing still another example of the parallax adjustment value setting process according to the present invention. FIG. 9 is a flowchart showing still another example of the parallax adjustment value setting process according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments for implementing a stereoscopic image processing device, a stereoscopic imaging device, a stereoscopic image reproducing device, a stereoscopic image processing method, and a stereoscopic image processing program according to the present invention will be described in detail below with reference to the accompanying drawings.

<Configuration of stereoscopic camera>
FIG. 1 is a front perspective view of a stereoscopic camera 2 according to the present invention. In FIG. 1, a first lens barrel 4 a that holds the first imaging optical system 1 a and a second lens barrel 4 b that holds the second imaging optical system 1 b are incorporated in the front surface of the stereoscopic camera 2, and a flash 5. Etc. are exposed. Further, a shutter button 6 used for a shutter release operation is provided on the upper surface of the stereoscopic camera 2.

Although not shown, an operation unit 10 including a zoom button, a menu button, a cursor button, and the like and a monitor 11 are provided on the back of the stereoscopic camera 2. By operating the operation unit 10 as appropriate, operations for turning on / off the power, switching between various modes (such as a shooting mode and a playback mode), zooming, and setting a parallax amount are performed. The monitor 11 is a parallax barrier (or lenticular) 3D monitor that functions as an electronic viewfinder when shooting an image, as an image playback monitor during image playback, and for displaying information for setting parallax adjustment values. Also works.

FIG. 2 shows the electrical configuration of the stereoscopic camera 2. The first photographing optical system 1a includes a first variable magnification lens 21, a first focus lens 22, and a first diaphragm 23 arranged along the lens optical axis L1. The first variable magnification lens 21 is driven by a first variable magnification lens control unit 24 configured by a DC motor and a driver. The first focus lens 22 is driven by a first focus lens control unit 25 configured by a DC motor and a driver. The first diaphragm 23 is driven by a first diaphragm control unit 26 composed of a DC motor and a driver. The operations of the control units 24 to 26 are controlled by a main CPU 40 (hereinafter simply represented by the CPU 40).

The first variable magnification lens control unit 24 responds to an input operation of tele or wide zoom direction information to a zoom button of the operation unit 10 (however, a ring-shaped operation member is also possible). Is moved from the home position to the tele side (feeding side) / wide side (retracting side) along the lens optical axis L1 to change the focal length (shooting magnification). When the first variable magnification lens 21 is moved to the tele side, the focal length becomes long and the photographing range becomes narrow. When the first variable magnification lens 21 is moved to the wide side, the focal point becomes short and the photographing range becomes wide.

The focus lens control unit 25 moves the first focus lens 22 along the lens optical axis L1 to perform focus adjustment. The position of the first focus lens 22 is automatically adjusted with the movement of the first variable magnification lens 21 so as not to be out of focus. It is assumed that stepwise zoom magnifications (zoom steps) Z1, Z2,..., Zn can be input from the operation unit 10. The number n of steps is arbitrary, but Z1 corresponds to the wide end and Zn corresponds to the tele end.

The target zoom direction set from the zoom button is output to the CPU 40. The CPU 40 sets a target zoom position according to the target zoom direction. If the target zoom direction is the tele direction, the closest zoom stage from the current position of the first variable magnification lens 21 to the tele direction side is set as the target zoom position, and if the target zoom direction is the wide direction, the current first variable magnification lens. The closest zoom level from 21 to the wide direction side is set as the target zoom position. The CPU 40 converts the target zoom position into the number of pulses up to the target stop position of the first variable magnification lens 21, and causes the first variable magnification lens control unit 24 to drive according to the number of pulses. Note that the pulse number 0 corresponds to the home position.

The first image sensor 28 receives the subject light imaged by the first variable magnification lens 21 and the first focus lens 22 and accumulates photocharges corresponding to the amount of received light in the light receiving element. In the first image sensor 28, the photocharge accumulation / transfer operation is controlled by a timing signal (clock pulse) periodically input from the timing generator 20 (TG). And sequentially input to the first analog signal processing unit 27. As the first image sensor 28, a CCD type or MOS type solid-state imaging device is used.

The first analog signal processing unit 27 receives an image signal for one screen input from the first image sensor 28, and amplifies R, G, B image data corresponding to the accumulated charge amount of each light receiving element accurately. To the first A / D converter 29. The first A / D converter 29 converts the input image data from analog to digital. The imaging signal of the first image sensor 28 becomes first image data (right-eye image data) via the first analog signal processing unit 27 and the first A / D converter 29.

The second photographing optical system 1b has the same configuration as the first photographing optical system 1a, and is driven by a second variable lens 31 and a second focus lens controller 36 that are driven by a second variable lens controller 34. And a second aperture 38 driven by a second aperture controller 37. The operations of the

control units

34, 36, and 37 are controlled by the CPU 40.

In addition, the same thing as each member of the 1st imaging | photography optical system 1a is used for each member of the 2nd imaging | photography optical system 1b. In addition, the first photographing optical system 1a and the second photographing optical system 1b are basically synchronized and perform an image pickup operation in conjunction with each other. However, for the purpose of improving the control speed, the respective photographing optical systems are used. May be moved individually. Either the left or right optical system may be the first photographing optical system 1a or the second photographing optical system 1b, and both are technically interchangeable.

The second analog signal processing unit 35 and the second A / D converter 39 have the same configurations as the first analog signal processing unit and the A / D converter 29, respectively. The imaging signal of the second image sensor 33 becomes second image data (left-eye image data) via the second analog signal processing unit 35 and the second A / D converter 39.

The first and second image data output from the first and second A /

D converters

29 and 39 are input to the digital signal processing units 41 and 42 via the

image input controllers

39a and 39b, respectively. The digital signal processing units 41 and 42 perform various image processing such as gradation conversion, white balance correction, and γ correction processing on each of the first and second image data. The first image data processed by the digital signal processing unit 41 and output at predetermined intervals is input to the VRAM 43. The second image data processed by the digital signal processing unit 42 and output at predetermined intervals is input to the VRAM 43.

The VRAM 43 is a working memory that temporarily stores the first and second image data. When the first and second image data of the next period are input to the VRAM 43 in a state where the first and second image data are already stored in the VRAM 43, the first and second image data already stored are new. It is overwritten with the input first and second image data. The first and second image data that are repeatedly overwritten and updated at predetermined intervals in the VRAM 43 are called through images.

The 3D image generation unit 45 combines the first and second image data stored in the VRAM 43 with stereoscopic image data for the monitor 11 to perform stereoscopic display. When the monitor 11 is used as an electronic viewfinder in the shooting mode, the display control unit 56 causes the monitor 11 to display the stereoscopic image data synthesized by the 3D image generation unit 45 as a through image.

The recording of captured images will be described below. The images captured from the first photographing optical system 1a and the second photographing optical system 1b at the timing when the shutter button 6 is pressed are processed by

analog signal processing

27 and 35, respectively, and then converted into digital signals by A /

D

29 and 39, respectively. And input to the digital signal processing units 41 and 42 via the

image input controllers

39a and 39b, respectively. The digital signal processing units 41 and 42 perform various image processing such as gradation conversion, white balance correction, and γ correction processing on each of the first and second image data. The first and second image data processed and output by the digital signal processing units 41 and 42 are recorded in the SDRAM 52. The compression / decompression processing unit 47 performs compression processing on the stored first and second image data in a compression format such as the JPEG method. The SDRAM 52 is used as a temporary storage area necessary for this compression process. The media control unit 48 records each image data compressed by the compression / decompression processing unit 47 on the memory card 49.

When the first and second image data recorded on the memory card 49 are reproduced and displayed on the monitor 11 in this way, each image data recorded on the memory card 49 is read by the media control unit 48. Each image data subjected to the decompression processing by the compression / decompression processing unit 47 is converted into stereoscopic image data by the 3D image generation unit 45 and then reproduced and displayed on the monitor 11 via the display control unit 46. At that time, a predetermined amount of parallax set by a method described later is used.

The detailed structure of the monitor 11 is not shown, but the monitor 11 has a parallax barrier display layer on the surface thereof. The method of the monitor 11 is not limited to the parallax barrier method, and other methods such as a lenticular method, a time division method, and a polarization filter method may be used as long as the same function can be realized. The monitor 11 may be a touch panel type, and the user can perform various operations via the monitor 11.

The CPU 40 controls the overall operation of the stereoscopic camera 2 in an integrated manner. The CPU 40 is connected to a flash control unit 72 that controls light emission of the flash 5 and an operation unit 10. A flash ROM 50 is connected to the CPU 40. The flash ROM 50 is a non-volatile memory capable of electrically rewriting data, but can store any data as long as there is free space.

The ROM 51 stores a control program (including a program for performing a parallax adjustment value setting process described later) for the CPU 40 to execute various processes. The clock unit 70 counts the current time and outputs it to the main CPU 40. The posture detection sensor 71 detects the photographing posture of the stereoscopic camera 2 in the horizontal position or the vertical position at the timing instructed from the CPU 40, for example, when the shutter button is half-pressed, and outputs the detection result to the CPU 40. When the power control unit 80 detects a power on signal or an off signal issued from the CPU 40 in response to an on or off operation of a power switch included in the operation unit 10, the power control unit 80 is supplied from the battery 81 to each block of the stereoscopic camera 2. Control to turn the power on or off.

The AF detection unit 44 calculates an AF evaluation value from the first image data or the second image data stored in the VRAM 43. The AF evaluation value is calculated by accumulating high-frequency components of luminance values for a region (for example, the central portion) designated by the CPU 40 in the first image data or the second image data, and represents the sharpness of the image. The AF evaluation value increases as the AF approaches the in-focus point, and becomes maximum at the time of in-focus.

The AE / AWB detection unit 73 detects subject brightness (photometry of subject brightness) based on the first image data or the second image data stored in the VRAM 43, and uses the detected subject brightness as a photometric value. The AE / AWB detection unit 73 detects a WB (white balance) value based on the first image data or the second image data stored in the VRAM 43. The method of calculating the exposure value is arbitrary, and any of spot metering, weighted average metering, and average metering may be used. The obtained photometric value, WB value, and AF evaluation value are notified to the CPU 40, and are used to control AE, AWB, and AF of the image signals obtained from the first photographing optical system 1a and the second photographing optical system 1b. .

The CPU 40 reads the program diagram defining the photometric value, the aperture value, the sensitivity, and the correspondence relationship among the shutter speeds from the ROM 51 to the SDRAM 52 and refers to the photometric value detected by the AE / AWB detector 73. The corresponding aperture value and sensitivity are set in the

aperture controllers

26 and 37 and the

image sensors

24 and 33, respectively, and exposure control is performed.

In the above description, an AE / AF evaluation value or the like is obtained based on one of the first image data and the second image data, and thereby the AE, AF, WB are obtained by the first photographing optical system 1a and the second photographing optical system 1b. However, the AE / AF evaluation values and the like are obtained for each of the first image data and the second image data, and based on these, the first photographing optical system 1a and the second photographing optical system are described. You may make it control 1b separately.

In the above description, the stereoscopic camera 2 is a so-called “compound-eye stereoscopic imaging device” having the first imaging optical system 1a and the second imaging optical system 1b, but the parallax amount setting process described below is such a compound eye. The present invention is not limited to a camera of the type, and can also be applied to a “monocular stereoscopic imaging device” that acquires a stereoscopic image using a single optical system.

[Parallax adjustment value setting processing in the present invention]
<First Embodiment>
Next, the parallax adjustment value setting process in the present invention will be described. FIG. 3 is a flowchart showing a first embodiment of parallax adjustment value setting processing according to the present invention. This process is controlled by the CPU 40. A program for causing the CPU 40 to execute this process is stored in the ROM 51. The following processing can also be executed by an information terminal such as a camera-equipped mobile phone having a configuration equivalent to that of the stereoscopic camera 2, a personal computer (hereinafter referred to as a PC), a digital television, and the like. The program for setting the parallax adjustment value includes a computer-readable code of the program such as a magneto-optical recording medium such as CD, DVD, BD, HDD (hard disk drive), SSD (solid state drive), etc. in addition to ROM. It can be used by being stored in a recording medium such as a magnetic recording device, a memory card, or a USB memory.

When the process shown in FIG. 3 is started, a stereoscopic image is first acquired in S100. This stereoscopic image can be acquired by operating the stereoscopic camera 2 to photograph the subject, but an already captured image may be acquired via the memory card 49.

Next, a distance histogram of the subject is created from the acquired stereoscopic image (S102). This distance histogram divides the left and right viewpoint images constituting the stereoscopic image into a plurality of regions (for example, 8 × 8, 16 × 16, etc.), calculates the distance to the subject in each region, and the subject is a predetermined distance. It can be created by counting the number of areas in the range. The distance to the subject can be calculated by, for example, comparing the contrasts of a plurality of imaging signals acquired at different lens positions, but is not limited to this, and various methods can be used.

In the creation of the distance histogram, it is further detected in which distance range the main subject exists. Whether or not the subject is a main subject can be determined based on criteria such as the ratio of the area occupied by the image (the number of regions present) and the face of a person. Examples of histograms created in this way are shown in FIGS. 4A and 4B. In the example of FIG. 4A, the main subject exists in the range of distance 1.2 m to 2.5 m, and another subject exists in the range of distance 5 m to 10 m. In the example of FIG. 4B, the main subject exists in the range of distance 2.5 m to 3 m, and subjects other than the main subject exist in other distance ranges.

Next, the distance histogram created as described above is compared with a pattern of distance distribution created and recorded in advance (S104). Comparison images are shown in FIGS. At the time of comparison, first, for each distance range of the histogram as in the example of FIG. 5A, the number of areas can be set to a predetermined threshold value (for example, 5 but different values may be set according to the number of divided areas, etc. ) Is exceeded, it is determined that “the subject exists”, and each distance range is classified into “(subject / main subject) exists” or “does not exist”, and a pattern as shown in FIG. 5B is obtained. create. Then, it is compared with a pattern as shown in FIG.

A plurality of patterns as shown in FIG. 5C can be set according to the distance range in which the main subject and other subjects exist. A setting example of this distance distribution pattern is shown in the table of FIG. In the example of FIG. 6, the distance range is divided into three areas, a foreground (1.2 m to 2.5 m), a middle scene (2.5 m to 5 m), and a distant view (5 m to). Nine patterns are set based on the distance range. In the example of FIG. 6, the

patterns

1, 2, 6 and 9 are “patterns suitable for 3D (stereoscopic) shooting” with an appropriate amount of parallax, and the shaded patterns 3 to 5 and 7 are too large in parallax. However, the parallax is too small to be seen three-dimensionally. FIG. 4A described above is an example of the pattern 4 that is not suitable for 3D shooting, and FIG. 4B is an example of the pattern 2 that is suitable for 3D shooting. FIG. 5C is an example of pattern 2.

When comparing the distance distributions described above, the distance to the main subject in the acquired stereoscopic image is compared in consideration of whether or not it belongs to the distance range of the main subject determined in the distance distribution pattern. For example, in the example of FIG. 5B, the main subject exists in the range of 2.5 m to 3 m, and belongs to the distance range (2.5 m to 5 m) of the main subject defined in the pattern (pattern 2) shown in FIG. 5C. ing.

It should be noted that in the distance distribution pattern, how many areas the distance range is divided and how to set the distance range of each area in the case of division may be different from those in FIG.

As a result of the comparison in S104, if it is determined that the distance histogram of the acquired stereoscopic image is similar to any one of the distance patterns suitable for stereoscopic shooting (YES in S106), the process proceeds to S108 to recommend a similar distance pattern. The parallax adjustment value is set as the parallax adjustment value and the process proceeds to S112. If the distance histogram is not similar to any of the distance patterns suitable for stereoscopic shooting (NO in S106), the process proceeds to S110 and the focused subject has zero parallax. The parallax adjustment value is set to be (cross point). It should be noted that the condition of whether or not they are similar in S106 can be set as appropriate according to the characteristics of the stereoscopic camera 2 and the shooting scene, and “similar” may include the same case. In the example of FIG. 6, recommended parallax adjustment values for distance patterns (

patterns

1, 2, 6, 8, and 9) suitable for stereoscopic shooting are shown in units of pixels. As the parallax adjustment value, a value for adjusting the parallax of an object at a predetermined distance (for example, 2 m from the stereoscopic camera 2) may be set, or a value for adjusting the parallax of the main subject may be set.

In S110, the in-focus subject is set to zero parallax in consideration of the fact that the in-focus subject is often a main subject such as a human face, but is suitable for stereoscopic shooting. The parallax adjustment value setting when there is no similarity to the distance pattern is not necessarily limited to this, and a parallax adjustment value or a fixed parallax adjustment value such that the center of the image becomes zero parallax (cross point) is used. Alternatively, the point designated by the user in the image may be zero parallax. In the present invention, “cross point” means a point where parallax is zero in a stereoscopic image.

When the parallax adjustment value is set by the processing up to S110, in S112, the set parallax adjustment value is applied to the stereoscopic image acquired in S100. Specifically, the left and right viewpoint images constituting the stereoscopic image are shifted by the number of pixels set in S108 or S110. Thereafter, the image generated by this shifting process is recorded, and the process ends. In the first embodiment, it is possible to set an appropriate parallax adjustment value according to the shooting scene by performing such parallax adjustment value setting processing. In addition, when performing the parallax adjustment value setting process described above, information such as a distance histogram may be displayed on the monitor 11, and the parallax adjustment value to be set and a stereoscopic image with the parallax adjustment value are set in advance. It may be displayed on the monitor 11.

<Second Embodiment>
Next, another aspect of parallax adjustment value setting according to the present invention will be described. In the first embodiment described above, the parallax adjustment value is set based on the similarity between the distance histogram created from the acquired stereoscopic image and the predetermined distance distribution pattern. In the second embodiment, in addition to this, the parallax adjustment value is set. The user can manually set the parallax adjustment value. FIG. 7 is a flowchart showing processing for setting a parallax adjustment value according to the second embodiment. Since S200 to S210 in the figure are the same as S100 to S110 in the flowchart shown in FIG. 3, the description thereof is omitted.

In the flowchart of FIG. 7, after setting the parallax adjustment value obtained by comparing the distance histogram with the distance pattern as shown in the example of FIG. Is determined (S212). This determination can be made based on a user instruction input to the operation unit 10. If the determination is affirmative (YES in S212), the process proceeds to S214, the parallax adjustment value input by the user via the operation unit 10 is set as the parallax adjustment value, the process proceeds to S216, and if the determination is negative (NO in S212) ) The process proceeds to S216 without performing the process of S214. In S216, as in S112 of FIG. 3, after applying the parallax adjustment value set to the stereoscopic image acquired in S200, the generated image is recorded, and the process ends.

In the second embodiment, by performing such a parallax adjustment value setting process, an appropriate parallax adjustment value according to the shooting scene can be set, and in addition, the parallax adjustment value according to the user's preference can be set. Setting can also be performed, and a more appropriate parallax adjustment value can be set.

Next, a modification of the above-described second embodiment will be described. FIG. 8 is a flowchart illustrating processing for setting a parallax adjustment value according to the modification. In the flowchart of FIG. 8, the processes other than S215-1 and S215-2 are the same as those of the flowchart of FIG.

The processing shown in the flowchart of FIG. 8 is that the recommended parallax adjustment value in the distance distribution pattern can be updated (overwritten) with the user setting value based on the user's judgment. Is different. Specifically, after setting the parallax adjustment value up to S214, it is determined in S215-1 whether or not to update the value of the recommended parallax adjustment value in the distance distribution pattern. This determination can be made based on a user instruction input to the operation unit 10. If the determination is affirmative (YES in S215-1), the process proceeds to S215-2 and the value of the recommended parallax adjustment value for the corresponding distance distribution pattern (distance distribution pattern determined to be similar to the created distance histogram) is updated. If the determination is negative (NO in S215-1), the process proceeds to S216 without performing the process of S215-2.

The user may or may not find the recommended parallax adjustment value appropriate, but even if he / she thinks the recommended parallax adjustment value is appropriate in a certain shooting situation, he / she feels that Therefore, the same recommended parallax adjustment value may not be appropriate. The reverse case is also possible. Therefore, it may not be appropriate to uniformly update (overwrite) the recommended parallax adjustment value when the parallax adjustment value is manually set. However, the user is asked whether or not to update the recommended parallax adjustment value as in the above modification. By confirming, it is possible to deal with such a situation, and it is possible to set an appropriate parallax adjustment value in consideration of the shooting scene and user preference.

In the second embodiment described above and its modifications, information such as distance histogram / BR> may be displayed on the monitor 11 as in the case of the first embodiment, or recommended parallax adjustment may be performed. The user may confirm the value and the parallax adjustment value set by the user or the stereoscopic image with the parallax adjustment value in advance to be confirmed by the user. The second embodiment described above and its modifications can also be executed by a camera-equipped mobile phone, a personal computer, or the like, as in the first embodiment.

<Third Embodiment>
In the above-described second embodiment, the case where the recommended parallax adjustment value of each distance distribution pattern is updated (overwritten) based on the user's determination has been described. In this update process, the parallax adjustment value set by the user becomes the recommended parallax adjustment value of the pattern of each distance distribution as it is, but in the third embodiment, the parallax adjustment value set by the user is learned and the setting history is taken into account. A case where the recommended parallax adjustment value is updated will be described.

FIG. 9 is a flowchart showing the parallax adjustment value setting process according to the third embodiment. In this flowchart, the processing from S300 to S314 is the same as S200 to S214 in the flowcharts of FIGS. 7 and 8 according to the second embodiment and its modifications, and the description thereof is omitted.

In the flowchart of FIG. 9, after setting the amount parallax adjustment value to be applied to the stereoscopic image in the process up to S314, it is determined in S316 whether or not to update the recommended parallax adjustment value for the corresponding distance distribution pattern. This determination can be made based on a user instruction input to the operation unit 10. If the determination is affirmative (YES in S316), the process proceeds to S318 to update the recommended parallax adjustment value of the distance pattern based on the learning result, and then proceeds to S220. If the determination is negative (NO in S316), the distance pattern is recommended. The process proceeds to S320 without updating the parallax adjustment value.

Specifically, the process performed in S318 is to set the parallax adjustment value in consideration of the setting history of the parallax adjustment value. For example, an aspect using the latest setting value, an aspect using a value with a high setting frequency, and past setting Various methods such as an aspect using an average of values, an aspect using weighted average of past setting values, and an aspect of predicting the current setting value from past setting values can be used. Moreover, you may make it make a user select which of these several methods is used. Then, when the recommended parallax adjustment value of the corresponding distance pattern is updated with the parallax adjustment value calculated using any of these methods (S318), the process proceeds to S320, and the setting history of the parallax adjustment value is updated. Note that the recommended setting value and setting history calculated in this way may be cleared according to a user instruction so as to return to the initial setting.

As described above, in the third embodiment, an appropriate parallax adjustment value considering the shooting scene is set by learning the parallax adjustment value setting value (updating the setting history and setting the parallax adjustment value based on the setting history). In addition, the parallax adjustment value having a high degree of matching with the user's preference can be set based on the tendency of the parallax adjustment value setting as the use continues.

In the third embodiment described above, information such as a distance histogram may be displayed on the monitor 11 as in the first and second embodiments, and the recommended parallax adjustment value and the user The set parallax adjustment value and the stereoscopic image with the parallax adjustment value may be displayed in advance to allow the user to confirm. Further, the third embodiment described above can also be executed by a camera-equipped mobile phone, a personal computer, or the like, as in the first and second embodiments.

Although the number of users is not particularly mentioned in the first to third embodiments described above, the parallax adjustment value setting according to the present invention can also be used when a plurality of users use the stereoscopic camera 2. For example, a plurality of tables in which recommended parallax adjustment values are set as shown in the example of FIG. 6 may be prepared corresponding to the user and used by switching for each user. You may make it hold | maintain for every user. A plurality of such recommended parallax adjustment value tables and parallax adjustment value setting history data can be stored in the flash ROM 50. At this time, the user is identified by, for example, allowing the user to input a user number or ID via the operation unit 10 or displaying the user number on the monitor 11 to select it, and the CPU 40 determines this. It can be carried out.

In the first to third embodiments, the recommended parallax adjustment value may be changed according to the zoom position (for example, 10 positions) in order to cope with the difference in the appropriate parallax adjustment value depending on the shooting scene. .

In the first to third embodiments, the subject distance histogram is compared with the distance distribution pattern, but a parallax histogram is created from the acquired stereoscopic image and compared with the parallax distribution pattern. It may be determined whether or not it is a pattern.

The first to third embodiments have been described for the case where the stereoscopic camera 2 sets the parallax adjustment value. However, the parallax adjustment value setting processing according to the present invention is not limited to a digital camera such as a stereoscopic camera. It is also possible to use an electronic device such as a camera-equipped mobile phone, personal computer, or digital television.

As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

The order of execution of each process such as operation, procedure, step, and stage in the apparatus, system, program, and method shown in the claims, the description, and the drawings is as follows. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if it is described using “first”, “next”, etc. for the sake of convenience, it means that it is essential to carry out in this order. It is not a thing.

1a: first imaging optical system, 1b: second imaging optical system, 2: stereoscopic camera, 10: operation unit, 11: monitor, 40: CPU, 49: memory card, 50: flash ROM

Claims

Stereoscopic image acquisition means for acquiring a stereoscopic image of a subject;
Distance distribution calculating means for calculating a distance to the one or more subjects or a distribution of values corresponding to the distance from the acquired stereoscopic image as a distance distribution;
A distance distribution matching means for matching the calculated distance distribution with a plurality of patterns of distance distribution stored in advance;
Parallax adjustment value setting means for setting a parallax adjustment value for the acquired stereoscopic image based on the collation result;
Recording means for recording an image obtained by applying the set parallax adjustment value to the acquired stereoscopic image;
A stereoscopic image processing apparatus comprising:
The distance distribution collating means performs collation in consideration of whether the distance to the main subject in the calculated distance distribution belongs to the distance range to the main subject in the plurality of patterns,
When the parallax adjustment value setting means determines that the calculated distance distribution is similar to any of the plurality of patterns as a result of the collation, the pattern determined to be similar to the acquired stereoscopic image Set the parallax adjustment value according to
Stereoscopic image processing device.
It further comprises parallax adjustment value input means for receiving an instruction input of the parallax adjustment value by the user,
The parallax adjustment value setting means sets the received parallax adjustment value as a parallax adjustment value according to the pattern determined to be similar;
The stereoscopic image processing apparatus according to claim 1.
When the parallax adjustment value setting means receives the input of the parallax adjustment value, the parallax adjustment value setting means
A new parallax adjustment value is calculated based on the received parallax adjustment value and the setting history of the parallax adjustment value, set as a parallax adjustment value according to the pattern determined to be similar, and the setting of the parallax adjustment value Update history,
The stereoscopic image processing apparatus according to claim 2.
A user identification means for identifying a plurality of users;
The parallax adjustment value setting means sets a parallax adjustment value according to which of the plurality of users the identified user is;
The stereoscopic image processing apparatus according to claim 1.
When the parallax adjustment value setting means determines that the calculated distance distribution is not similar to any of the plurality of patterns as a result of the collation, the parallax adjustment value setting means determines that the focused subject has zero parallax with respect to the acquired stereoscopic image. Set the parallax adjustment value so that
The stereoscopic image processing apparatus according to claim 1.
6. The stereoscopic image processing apparatus according to claim 1, wherein the distance distribution calculated by the distance distribution calculating unit is a histogram of distances to a subject.
6. The stereoscopic image processing apparatus according to claim 1, wherein the distance distribution calculated by the distance distribution calculating means is a parallax histogram of a subject.
The stereoscopic image processing apparatus according to any one of claims 1 to 7, wherein the parallax adjustment value is a value for adjusting the parallax of a subject at a predetermined distance.
The stereoscopic image processing apparatus according to any one of claims 1 to 7, wherein the parallax adjustment value is a value for adjusting a parallax of a main subject.
The stereoscopic image processing apparatus according to claim 1, further comprising stereoscopic image display means for displaying an image obtained by applying the set parallax adjustment value to the acquired stereoscopic image.
Stereoscopic image acquisition means for acquiring a stereoscopic image of a subject;
Distance distribution calculating means for calculating a distance to the one or more subjects or a distribution of values corresponding to the distance from the acquired stereoscopic image as a distance distribution;
A distance distribution matching means for matching the calculated distance distribution with a plurality of patterns of distance distribution stored in advance;
Parallax adjustment value setting means for setting a parallax adjustment value for the acquired stereoscopic image based on the collation result;
Recording means for recording an image obtained by applying the set parallax adjustment value to the acquired stereoscopic image;
A stereoscopic imaging device comprising:
The distance distribution collating means performs collation in consideration of whether the distance to the main subject in the calculated distance distribution belongs to the distance range to the main subject in the plurality of patterns,
When it is determined that the calculated distance distribution is similar to any of the plurality of patterns, the parallax adjustment value setting unit is determined to be similar to the acquired stereoscopic image. Set the parallax adjustment value according to the pattern,
Stereo imaging device.
Stereoscopic image acquisition means for acquiring a stereoscopic image of a subject;
Distance distribution calculating means for calculating a distance to the one or more subjects or a distribution of values corresponding to the distance from the acquired stereoscopic image as a distance distribution;
A distance distribution matching means for matching the calculated distance distribution with a plurality of patterns of distance distribution stored in advance;
Parallax adjustment value setting means for setting a parallax adjustment value for the acquired stereoscopic image based on the collation result;
Recording means for recording an image obtained by applying the set parallax adjustment value to the acquired stereoscopic image;
Stereoscopic image display means for displaying an image obtained by applying the set parallax adjustment value to the acquired stereoscopic image;
A stereoscopic image playback device comprising:
The distance distribution collating means performs collation in consideration of whether the distance to the main subject in the calculated distance distribution belongs to the distance range to the main subject in the plurality of patterns,
When the parallax adjustment value setting means determines that the calculated distance distribution is similar to any of the plurality of patterns as a result of the collation, the pattern determined to be similar to the acquired stereoscopic image Set the parallax adjustment value according to
Stereoscopic image playback device.
A stereoscopic image acquisition step of acquiring a stereoscopic image of the subject;
A distance distribution calculating step of calculating a distance to the one or more subjects or a distribution of values corresponding to the distance from the acquired stereoscopic image as a distance distribution;
A distance distribution matching step of matching the calculated distance distribution with a plurality of patterns of distance distribution stored in advance;
A parallax adjustment value setting step for setting a parallax adjustment value for the acquired stereoscopic image based on the collation result;
A recording step of applying and recording the set parallax adjustment value to the acquired stereoscopic image;
A stereoscopic image processing method including:
In the distance distribution collation step, the distance to the main subject in the calculated distance distribution is collated in consideration of whether it belongs to the distance range to the main subject in the plurality of patterns,
In the parallax adjustment value setting step, when it is determined that the calculated distance distribution is similar to any of the plurality of patterns as a result of the collation, the pattern determined to be similar to the acquired stereoscopic image Set the parallax adjustment value according to
Stereoscopic image processing method.
A stereoscopic image processing program for causing a computer to execute the stereoscopic image processing method according to claim 13.
15. A recording medium on which a computer-readable code of the stereoscopic image processing program according to claim 14 is recorded.