JP4875762B2 - Image processing apparatus, image display apparatus, and image pickup apparatus - Google Patents

Description

  The present invention relates to an image processing device that generates a stereoscopic image, an image display device that includes the image processing device, and an image imaging device that includes the image processing device.

  Two cameras for the left eye and right eye are arranged side by side to photograph the same subject (hereinafter abbreviated as the same subject), and two image data (hereinafter abbreviated as images) generated by the two cameras. Has been proposed (see Patent Document 1), which performs various image processing on the image to generate a stereoscopic image (also referred to as a stereo image).

A shift between images generated when the two images are overlapped is called parallax. This parallax changes depending on the distance from the camera to the subject.
For example, it is assumed that two cameras are arranged on a straight line so that the optical axes of the cameras are parallel and the same subject is photographed. At this time, if the subject is far away, the parallax of the subject is almost zero. Then, as the distance between the camera and the subject becomes shorter, the parallax of the subject becomes larger. Therefore, when a stereoscopic image is generated based on two images including a subject with a large parallax and displayed and output, the pop-out amount of the subject becomes large (appears closer), and the stereoscopic effect is enhanced.

  However, if the parallax is increased to a certain extent, the image perceived by the left eye and the image perceived by the right eye are not fused, and a double image is seen, making stereoscopic viewing impossible (fusion limit). Furthermore, it is known that double images due to excessive parallax cause visual fatigue. Therefore, in order to comfortably view a stereoscopic image, it is necessary to limit the depth (protrusion / retraction) range of the subject when displaying the image. This limitation is disclosed in Non-Patent Document 1.

  A technique for generating a stereoscopic image without generating a double image due to excessive parallax is disclosed in, for example, Patent Document 1 described above. The technique disclosed in Patent Document 1 generates an image that is easy for the observer to stereoscopically view by controlling the depth range in a non-linear manner so that the stereoscopic image is displayed within the range in which the observer can stereoscopically view. Is.

JP 2005-91508 A

3D Consortium 3DC Safety Guidelines URL: http://www.3dc.gr.jp/jp/scmt_wg_rep/3dc_guidelineJ_200812.pdf

  By the way, when the main subject for which the stereoscopic effect is to be emphasized does not have a parallax near the middle between the maximum parallax and the minimum parallax that can be taken, there is a problem that the stereoscopic effect of the main subject cannot be sufficiently obtained. This problem will be specifically described with reference to FIGS.

FIG. 16 is a diagram schematically showing a main subject 1600, a main subject 1601, and a background 1602. The distance from a camera (not shown) is the closest to the main subject 1600, and the main subject 1601 and the background 1602 become longer in this order.
As described above, various image processing is performed on an image obtained by photographing the main subjects 1600 and 1601 and the background 1602 using two cameras arranged on a straight line, a stereoscopic image is generated, and a stereoscopic image display device (Hereinafter abbreviated as “display device”).

  Since the distance from the camera increases in the order of the main subjects 1600 and 1601, and the background 1602, assuming that the parallax of the main subject 1600 is γ and the parallax of the main subject 1601 is β, γ> β. It is assumed that the parallax γ of the main subject 1600 and the parallax β of the main subject 1601 are not parallax in the vicinity of the maximum parallax that can be photographed and the minimum parallax. Since the background 1602 is located farthest from the camera, the parallax α of the background 1602 is the smallest.

  FIG. 17 shows a case where a stereoscopic image generated by executing various image processing on an image obtained by photographing the main subjects 1600 and 1601 and the background 1602 shown in FIG. 16 is displayed on the display panel P of the display device. It is a figure which shows typically the state as which the said stereo image is perceived by the viewer. “A” on the display panel P indicates the position of the display panel.

The main subject 1600 is perceived near the position 1700, the main subject 1601 is perceived near the position 1701, and the background 1602 is perceived near the position 1702.
As described above, when the stereoscopic image of the main subjects 1600 and 1601 and the background 1602 shown in FIG. 16 is displayed on the display panel P, the main subjects 1600 and 1601 are perceived by the observer as if they are retracted in the vicinity of the background 1602. . As a result, the stereoscopic effect of the subject cannot be sufficiently obtained, and the stereoscopic effect as a stereoscopic image is weakened and perceived.

  In particular, when the parallax of the main subject substantially matches the parallax of the background or foreground, the stereoscopic effect of the main subject is greatly impaired.

  An object of the present invention is to provide an image processing apparatus that solves the above-described problems, generates a stereoscopic image that is easy for an observer to stereoscopically view and is not fatigued, and easily adjusts the stereoscopic effect of an arbitrary part in the stereoscopic image. is there.

First technical means is 1 the photographing image from viewpoint disparity images corresponding to the photographed image is input, the imaging parallax of the parallax images corresponding to the first predetermined value or less or the second predetermined value or more subjects A blur processing unit that performs a blur process that increases a blur amount in proportion to the absolute value of the parallax for a pixel of the image, a parallax correction unit that saturates the parallax outside a predetermined range of the parallax image, and A parallax conversion unit that converts parallax corresponding to at least one main subject to a predetermined value, the captured image that has been blurred by the blur processing unit, and correction by the parallax correction unit; An image processing apparatus comprising: an image generation unit configured to generate a stereoscopic image based on the parallax image in which a parallax corresponding to a subject is converted, and generating a stereoscopic image.

A second technical means is the first technical means, wherein the photographed image is one of two photographed images obtained by photographing the same subject from two viewpoint positions, and the parallax image is the two photographed images. It is a parallax image calculated from the above.

According to a third technical means, in the first or second technical means, the parallax conversion unit is configured to calculate the predetermined value based on a linear function that defines a relationship between input parallax and output parallax corresponding to a main subject. The graph representing the relationship between the input parallax and the output parallax is formed by connecting a plurality of line segments having different inclinations.

According to a fourth technical means, in any one of the first to third technical means, the predetermined value is 0 or a value within a predetermined range with reference to 0 .

A fifth technical means is any one of the first to fourth technical means, wherein the predetermined range of the parallax image has a minimum value not more than the first predetermined value and a maximum value not less than the second predetermined value. It is characterized by.

A sixth technical means is an image display device comprising the image processing device of any one of the first to fifth technical means.

A seventh technical means is an image pickup apparatus comprising the image processing apparatus according to any one of the first to fifth technical means.

  According to the image processing apparatus of the present invention, it is possible to generate a stereoscopic image that is easy for an observer to stereoscopically view and less fatigue, and to easily adjust the stereoscopic effect of an arbitrary part in the stereoscopic image by parallax conversion. It is possible to emphasize the stereoscopic effect of the subject and change the sense of depth.

1 is a functional block diagram of an image processing apparatus described in a first embodiment. It is a figure explaining a parallax image. It is a parallax correction type graph that normalizes the parallax. It is a parallax correction type graph for clipping parallax. It is a graph which shows the relationship between the parallax between the right-and-left images when a viewer looks at a display apparatus, and depth amount. It is a figure which illustrates typically how an observer will perceive when a subject with parallax increased or decreased to 0 and a positive / negative value is displayed. It is a parallax conversion type | formula graph which performs parallax conversion to the parallax correct | amended by normalizing. It is a parallax conversion type | formula graph which performs parallax conversion to the parallax correct | amended by clipping. It is a parallax conversion type | formula graph which performs parallax conversion to the parallax correct | amended by clipping. It is a figure which shows typically the state in which the viewer perceives the stereo image after parallax conversion. It is a functional block diagram of the image processing apparatus demonstrated in 2nd Example. It is a graph which shows the example of expansion of parallax It is a parallax conversion type | formula graph which performs parallax conversion to the parallax correct | amended by normalizing. It is a functional block diagram of the image processing apparatus demonstrated in 3rd Example. It is a parallax correction type graph for clipping parallax. It is the figure which showed the main subject and the background typically. It is a figure which shows typically the state in which a three-dimensional image is perceived by the viewer.

Example 1
Hereinafter, the present invention will be described in detail with reference to the drawings. The configuration in each drawing is exaggerated for easy understanding, and is different from the actual interval and size.
FIG. 1 is a functional block diagram of an image processing apparatus 100 according to the present invention described in the first embodiment.
The image processing apparatus 100 includes a parallax correction unit 101, a main subject determination unit 102, a parallax conversion unit 103, and an image generation unit 104. The captured image from one viewpoint and the parallax image corresponding to the captured image are input. A stereoscopic image is generated based on the captured image and the parallax image.

  A parallax image input from an external device (not shown) or the like is input to the parallax correction unit 101. A captured image input from an external device or the like is input to the main subject determination unit 102 and the image generation unit 104.

The parallax correction unit 101 corrects the parallax of the input parallax image within a predetermined range, and outputs the parallax image in which the corrected parallax is stored to the parallax conversion unit 103. The parallax image will be described later.
The main subject determination unit 102 determines a main subject of the input captured image and outputs position information of the main subject to the parallax conversion unit 103.

The parallax conversion unit 103 extracts the parallax (after correction) corresponding to the main subject image from the parallax image input from the parallax correction unit 101 based on the position information of the main subject input from the main subject determination unit 102. To do. Then, the extracted parallax is converted based on a predetermined conversion formula, and a parallax image in which the converted parallax is stored is output to the image generation unit 104. That is, the parallax of the parallax image is converted so that the parallax corresponding to the image of the main subject determined by the main subject determination unit 102 becomes a predetermined value. The predetermined value is a value at which the main subject is displayed near the display screen when the main subject is displayed on the stereoscopic display device. Specifically, for example, 0 or a value within a predetermined range with reference to 0. The predetermined value can be adjusted.
As a result, the parallax is converted so that the stereoscopic effect of the main subject is enhanced.

The image generation unit 104 generates an image for the left eye (left image) and an image for the right eye (right image) based on the input parallax image and the captured image, and outputs the generated image to a display device (not shown). To do.
That is, the image generation unit 104 generates a stereoscopic image based on the parallax image based on the parallax converted by the parallax conversion unit 103 and the captured image.

  Hereinafter, the contents of each functional block will be described in detail. First, a parallax image input to the parallax correction unit 101 will be described with reference to FIG.

  FIG. 2A is a diagram schematically showing a state in which the subject 201 is photographed by the cameras 202 and 203. The optical axes (dotted lines) of the cameras 202 and 203 are parallel, and the cameras 202 and 203 are arranged on a straight line.

  2B shows a photographed image of the subject 201 (hereinafter referred to as a left image) 204 photographed by the left camera 202 and a photographed image of the subject 201 (hereinafter referred to as a right image) photographed by the right camera 203. ) 205 schematically.

  2C schematically shows an image in which the right image 205 is superimposed on the left image 204, and FIG. 2D shows a parallax image 206. FIG.

  As shown in FIG. 2C, the subject 201 (rectangular dotted line) of the right image 205 is shifted to the left by X pixels with respect to the subject 201 (rectangular solid line) of the left image 204. The X pixel is a parallax indicating a shift of the same subject 201 in two captured images (left image 204 and right image 205).

  The parallax image means an image in which parallax indicating a deviation of the same subject in two captured images obtained by photographing the same subject from two viewpoint positions is stored for each pixel corresponding to the pixel of the subject. In the above-described example, the parallax of the subject is stored for each pixel of the parallax image corresponding to the subject of the left image 204.

FIG. 2D schematically shows a state in which the parallax X of the subject 201 is stored in the pixel 206 a of the parallax image 206.
The parallax of the subject of the left image 204 corresponding to the pixel 206b is also stored in the other pixel 206b of the parallax image 206.
The unit of parallax may be a pixel or a distance.

The parallax correction unit 101 corrects parallax of the parallax image in order to suppress excessive parallax. This parallax can be corrected by a well-known method described later.
There are two typical parallax correction methods. One is a method for normalizing parallax, and the other is a method for clipping parallax. When parallax is normalized, the entire depth relationship is preserved without saturating the parallax, and when the parallax is clipped, the depth amount of the subject having the parallax within a stereoscopically viewable range is maintained.

An example in which the parallax correction is performed by the normalizing method is shown in FIG. 3, and an example in which the parallax correction is performed by the clipping method is shown in FIG.
FIG. 3 shows a graph of a parallax correction formula for normalizing parallax, and the parallax correction formula is expressed by (Formula 1).

  FIG. 4 shows a graph of a parallax correction formula for clipping parallax, and the parallax correction formula is expressed by (Expression 2) to (Expression 4).

The horizontal axis represents the input parallax D _in , and the vertical axis represents the output parallax D _out . The input parallax means the parallax of the parallax image input to the parallax correction unit 101 (the parallax before correction), and the output parallax means the parallax of the parallax image output from the parallax correction unit 101 (parallax after correction). Means. Further, d _in in the equation corresponds to the input parallax D _in , and d _out in the equation corresponds to the output parallax D _out .

D _{in_min} is the minimum parallax of the input parallax image, and D _{in_max} is the maximum parallax of the input parallax image.
D _{out_min} is the minimum value of parallax that can be viewed stereoscopically, and D _{out_max} is the maximum value of parallax that can be viewed stereoscopically, and corresponds to the parallax at the fusion limit.

A line 300 represents the relationship between the input parallax and the output parallax before the parallax correction is performed, and the input parallax and the output parallax have the same value before the parallax correction. A line 301 represents the relationship between the input parallax D _in before the parallax correction and the output parallax D _out after the parallax correction.

In the method of FIG. _{3, D} in_max the (maximum value of the input disparity) assigned to _{D out - max} (the maximum value of the stereoscopically viewable _{disparity), D} in_min (minimum value of the input disparity) _{D out_min} (of the stereoscopic view possible parallax By assigning to the minimum value, the parallax is corrected within a stereoscopically viewable range while maintaining the overall depth relationship without saturating the parallax.

In the method of FIG. 4, the input parallax greater than or equal to D _{out_max} (maximum stereoscopic parallax) is corrected to D _{out_max} , and the input parallax equal to or _smaller than D _{out_min} (minimum stereoscopic parallax) is set to D _{out_min.} Make corrections to. As described above, the parallax that cannot be stereoscopically viewed is saturated with the parallax that can be stereoscopically viewed, so that the parallax is corrected within a stereoscopically visible range while maintaining the depth amount.

  The stereoscopic binocular parallax range described above depends on the screen size of the display device and the viewing distance of the observer. Non-Patent Document 1 discloses the maximum binocular parallax range (fusion limit) that can be viewed stereoscopically. It is described that the range of binocular parallax in which a stereoscopic image can be comfortably viewed about 2 degrees is 1 degree or less.

When D _{out_min} and D _{out_max} are determined based on the screen size of a display device that displays a stereoscopic image and the standard viewing distance according to the screen size, for example, 40 inches HD (1920 In the case of × 1080) resolution, −130 pixels are the minimum parallax and 130 pixels are the maximum parallax with respect to the position of the display device (display panel).

  The parallax image in which the parallax corrected by the parallax correction unit 101 is stored is output to the parallax conversion unit 103.

  The main subject determination unit 102 determines a main subject from the subjects of the input captured image, and outputs position information of the main subject, for example, coordinate information indicating the pixel position of the main subject to the parallax conversion unit 103. The main subject can be determined by a known method. For example, select a subject in focus from the captured image and determine the subject as the main subject, or determine the subject extracted by executing pattern recognition processing on the captured image as the main subject. And a method in which the user himself determines the main subject. In addition, a specific person may be determined as a main subject using face recognition technology.

  The parallax conversion unit 103 performs conversion so that the parallax corresponding to the image of the main subject determined by the main subject determination unit 102 becomes a predetermined value. Specifically, the parallax is converted so that the stereoscopic effect of the subject is enhanced. In the example of FIG. 2, when the subject 201 is a main subject, the parallax X of the subject 201 is a parallax corresponding to the image of the main subject (parallax to be converted).

Here, how a viewer perceives a subject whose parallax has been increased or decreased to 0 or a positive / negative value will be described with reference to FIGS. 5 and 6.
FIG. 5 is a graph showing the relationship between the parallax between the left and right images and the depth amount when the observer looks at the display device. The depth is the amount of protrusion and the amount of retraction. The amount of protrusion is the amount that appears closer to the observer than the actual display position, and the amount of retraction is farther from the observer than the actual display position. It is the amount that can be seen.

  That is, when a certain subject is displayed on the display device, the subject appears to pop out as the parallax of the subject increases to a positive value, and retracts as the subject increases to a negative value. In addition, since a subject with zero parallax does not shift between the left and right images, even if it is displayed on the display device, the perceived position is on the display screen, and the depth position perceived in 2D does not differ. For this reason, even when an image with zero parallax is displayed on the display device, the viewer does not perceive that there is depth, but perceives that the image is displayed in 2D at the same position as the actual display position. Displaying a subject with zero parallax on the display device is said to be displayed on the display screen. The parallax 0 or a value within a predetermined range with reference to 0 corresponds to the predetermined value.

FIG. 6 is a diagram schematically illustrating how an observer perceives a subject whose parallax is increased or decreased to 0 or a positive / negative value on a display device.
The parallax on the front side surface of the subject A shown in FIG. 6 is defined as D _in (input parallax). Here, when the value of the parallax D _in is converted to 0, that is, the converted parallax is set to 0 (hereinafter, the converted parallax is referred to as D _out ), and the subject A having the parallax of 0 is displayed on the display device P. The subject A is perceived by the observer near the position 600 that is the image display surface of the display device P.

Further, when D _out is set to a positive value, the subject A is perceived to jump out to the near side of the image display surface as a position 601. When D _out is set to a negative value, the subject A is perceived by being retracted to the back side of the image display surface as a position 602.

The parallax conversion can be executed by, for example, parallax conversion formulas shown in FIGS.
FIG. 7 shows a parallax conversion type graph for performing parallax conversion on parallax corrected by normalization shown in FIG. 3, and FIG. 8 shows parallax for performing parallax conversion on parallax corrected by clipping. The conversion formula graph is shown.

  The parallax conversion formula of FIG. 7 is expressed by (Formula 5) and (Formula 6).

  The parallax conversion formula of FIG. 8 is expressed by (Formula 7) to (Formula 10).

Here, input parallax D _in indicates parallax before parallax conversion, D _{in_max} indicates maximum parallax before parallax conversion, and D _{in_min} indicates minimum parallax before parallax conversion.
Output disparity _{D out} denotes the parallax after the parallax _{conversion, D out - max} is the maximum _{disparity, D out_min} after disparity conversion indicates the minimum disparity after the disparity conversion.
A line 302 represents the relationship between the input parallax D _in before the parallax conversion and the output parallax D _out after the parallax conversion. D _in0 and D _out0 can be set to arbitrary values.

Further, d _in in the equation corresponds to the input parallax D _in before the parallax conversion, and d _out in the equation corresponds to the output parallax D _out after the parallax conversion.

Here, the graphs of FIGS. 7 and 8 will be described with reference to FIG.
According to the graph of FIG. 7, the input parallax is reduced, and according to the graph of FIG. 8, the input parallax is increased.

When the subject A described in FIG. 6 is perceived in the vicinity of the position 600 and converted to a negative value by reducing the parallax of the subject A using the parallax conversion formula shown in the graph of FIG. A position 602 is perceived by being retracted to the back side of the image display surface.
When the subject A described in FIG. 6 is perceived in the vicinity of the position 600 and converted to a positive value by increasing the parallax of the subject A using the conversion formula shown in the graph of FIG. It is perceived as popping out to the near side of the image display surface as in 601.

7, as shown in FIG. _8, the boundary of _{D in0,} are changed less and scope of parallax than _{D in0,} the conversion formula in the range of greater disparity than _{D in0.} That is, the depth from the main subject to the background can be increased or decreased, and the stereoscopic effect can be easily emphasized.

  In the following, when the parallax of the main subject shown in FIG. 16 is converted using, for example, the parallax conversion formula described with reference to FIG. This will be described with reference to FIGS. The parallax conversion formula of FIG. 9 is the same conversion formula as the parallax conversion formula of FIG.

  The parallax γ of the main subject 1600 in FIG. 16 is increased to the parallax γ ′ by the parallax conversion formula shown in FIG. Also, the parallax β of the main subject 1601 in FIG. 16 is increased to the parallax β ′ by the parallax conversion formula shown in FIG. 9.

FIG. 10 is a diagram schematically illustrating a state in which a stereoscopic image after parallax conversion is perceived.
By increasing the parallax, the main subject 1600 is perceived near the position 1000 and the main subject 1601 is perceived near the position 1001. By doing in this way, depth amount can be expanded and a three-dimensional effect is emphasized.
Compared to FIG. 17, by increasing the parallax, the difference between the position of the background 1602 and the positions of the main subjects 1600 and 1601 can be clearly perceived. As a result, it is possible to display an image with enhanced stereoscopic effect.

  Thus, by converting the parallax of the main subject, it is possible to enhance the stereoscopic effect between the main subject and the peripheral portion of the main subject. When the main subject is thick, the average value of the maximum parallax and the minimum parallax of the main subject is set as the parallax of the main subject. Further, when there are a plurality of main subjects, the average value of the maximum parallax and the minimum parallax of the plurality of main subjects is set as the parallax of each main subject. This is preferable because the parallax (display position) of the main subject can be converted in consideration of the thickness of the main subject. In addition, it is preferable to calculate the average value of parallax in consideration of the area of the region because the display position can be converted in consideration of the center of gravity of the subject.

  As described above, the parallax conversion unit 103 extracts the parallax (after correction) of the main subject determined by the main subject determination unit 102 from the parallax images input from the parallax correction unit 101 for each pixel. The parallax for each pixel is converted based on the predetermined conversion formula shown in FIG. 7 or FIG. 8, and the converted parallax is stored for each pixel of the main subject. Then, the parallax image in which the converted parallax is stored is output to the image generation unit 104.

The image generation unit 104 generates a stereoscopic image based on the captured image input from the external device and the parallax image input from the parallax conversion unit 103.
Here, a stereoscopic image generation method will be described below. The captured image is I, the parallax image is D, and the output image is O. The pixel value at the coordinates (x, y) in the image is represented by I (x, y) and D (x, y). Using (Expression 11), an image that is horizontally moved from the captured image I to the output image O by the amount of parallax is created.

  At this time, when pixels overlap, interpolation is performed using pixels with large parallax (subject close to the camera), and when the pixels are not filled, interpolation is performed using upper, lower, left, and right pixels. The captured image I is output as a left image, and the output image O is output as a right image.

With the above configuration, even if the parallax value between the main subject and the foreground or background is small, the parallax difference between the main subject and the foreground or background can be increased by converting the parallax.
As a result, it is possible to convert the depth amount between the main subject and the foreground or the background, and to generate an image in which the stereoscopic effect is enhanced within a range in which the observer can stereoscopically view.

In this embodiment, the parallax correction unit 101 and the parallax conversion unit 103 perform parallax correction and conversion using linear transformation, but the same effect can be obtained using nonlinear transformation.
In this embodiment, one set of input parallax and output parallax D _in0 and D _out0 is used, but the same effect can be obtained even when parallax conversion is performed using two or more sets.

(Example 2)
Hereinafter, Example 2 of the present invention will be described in detail with reference to the drawings. However, portions having the same functions as those in the first embodiment are denoted by the same reference numerals.
FIG. 11 is a functional block diagram of the image processing apparatus 1100 of the present invention described in the second embodiment.
The image processing apparatus 1100 is obtained by adding a parallax calculation unit 1101 to the image processing apparatus 100 described with reference to FIG. 1. The image processing apparatus 1100 receives two captured images obtained by capturing the same subject from two viewpoint positions, and is based on the captured image. Thus, a stereoscopic image is generated.

Two captured images obtained by capturing the same subject from at least two viewpoint positions are input to the parallax calculation unit 1101. In addition, one of the two captured images is input to the main subject determination unit 102 and the image generation unit 104. One captured image of the two captured images is a captured image captured by the left camera toward the subject, and is referred to as a left image. Another photographed image is a photographed image taken by the right camera toward the subject, and this is called a right image.
The left image and the right image are input to the parallax calculation unit 1101, and the left image is input to the main subject determination unit 102 and the image generation unit 104.

  The parallax calculation unit 1101 calculates the parallax based on the left image and the right image. And based on the calculated parallax, the parallax image demonstrated in Example 1 is produced | generated. For calculating the parallax, a known technique can be used, for example, a block matching method can be used. The block matching method is a method in which the left and right images are collated in appropriately defined block units, and the most similar block between the images is determined as a corresponding block to calculate the parallax.

The parallax calculation unit 1101 outputs the generated parallax image to the parallax correction unit 101.
As described in the first embodiment, the parallax correction unit 101 corrects the parallax of the parallax image input from the parallax calculation unit 1101 within a predetermined range. That is, the parallax calculated by the parallax calculation unit 1101 is corrected within a predetermined range.

During this correction, it is also possible to perform parallax adjustment that exceeds the interval (baseline length) between the camera that captured the left image and the camera that captured the right image. When shooting a stereoscopic image, the base length is often set to 65 mm. This is because the width of the human eye is about 65 mm.
However, due to the camera installation position, photographing with a baseline length of 65 mm or less may be forced. When the same subject is photographed with the same composition, the parallax decreases as the baseline length becomes shorter, and the sense of depth decreases. The parallax correction unit 101 prevents this reduction in depth by correcting the input parallax to increase the minimum and maximum values. That is, it is possible to generate an image shot with an arbitrary baseline length from an image shot with a fixed baseline length.

FIG. 12 is a graph showing an example of parallax expansion. The horizontal axis and vertical axis in the figure are the same as those in FIG. The parallax of the parallax image generated based on the left and right images captured with a short baseline length is a line 300. For example, it is _assumed that D _{in_max} (the maximum parallax of the parallax image) is 100 and D _{out_max} (the maximum parallax that can be stereoscopically viewed) is 200. The value before the parallax correction can be displayed only within the parallax 100 range. Therefore, the parallax indicated by the line 300 is corrected to the parallax indicated by the line 301 by using (Equation 1) described above.

When the parallax is corrected using (Expression 1), D _{in_max} is corrected to D _{out_max} . With the value after the parallax correction, it is possible to display up to the maximum value of the output parallax in the display device. That is, it is possible to generate an image shot with an arbitrary baseline length from an image shot with a fixed baseline length. Furthermore, the stereoscopic effect can be enhanced in accordance with the depth amount of the display device.

The parallax correction unit 101 outputs the parallax image storing the corrected parallax to the parallax conversion unit 103.
As described above, the parallax conversion unit 103 converts the parallax of the main subject by the parallax conversion illustrated in FIG. 13 corresponding to FIG. 7, and outputs the parallax image in which the converted parallax is stored to the image generation unit 104. To do. For the parallax conversion in FIG. 13, (Equation 5) and (Equation 6) described above are used.

  That is, the parallax calculated by the parallax calculation unit 1101 (the parallax of the parallax image created by the parallax calculation unit 1101) is converted so that the parallax corresponding to the main subject determined by the main subject determination unit 102 becomes a predetermined value. The predetermined value is, for example, 0, or a value within a predetermined range with reference to 0. The predetermined value can be adjusted.

  The image generation unit 104 generates a stereoscopic image based on the left image input from an external device or the like and the parallax image input from the parallax conversion unit 103. Since the generation of the stereoscopic image has been described in the first embodiment, a description thereof will be omitted.

  With the above configuration, it is possible to easily generate an image in which the stereoscopic effect is enhanced within a range in which the observer can stereoscopically view. Furthermore, by correcting the parallax of the parallax image generated based on the left and right images shot with a short baseline length, an image as if shot with a long baseline length can be generated. This is a case where a stereoscopic image is generated based on left and right images taken in a situation where the baseline length cannot be increased, such as left and right images of an endoscope camera or a camera incorporated in a small device, and the stereoscopic effect of the subject is emphasized. It is particularly effective.

(Example 3)
Hereinafter, Example 3 of the present invention will be described in detail with reference to the drawings. However, portions having the same functions as those in the first embodiment are denoted by the same reference numerals.
FIG. 14 is a functional block diagram of the image processing apparatus 1400 of the present invention described in the third embodiment.
An image processing apparatus 1400 in FIG. 14 is obtained by adding a blurring processing unit 1401 to the image processing apparatus 100 described in FIG.

  A parallax image input from an external device or the like is input to the parallax correction unit 101 and the blur processing unit 1401. A captured image input from an external device or the like is input to the main subject determination unit 102 and the blur processing unit 1401.

  Based on the input parallax image and the captured image, the blurring processing unit 1401 performs a blurring process on the pixel value of the captured image corresponding to the subject whose parallax is equal to or less than the first predetermined value or equal to or greater than the second predetermined value. Then, the captured image that has been subjected to the blurring process is output to the image generation unit 104. Note that the first predetermined value <the second predetermined value. At this time, the blurring processing unit 1401 increases the blurring amount for the pixel value of the main subject in proportion to the absolute value of the parallax of the main subject in the captured image.

  In the example of FIG. 2, when the parallax X of the subject 201 is equal to or smaller than the first predetermined value or equal to or larger than the second predetermined value, the blurring processing unit 1401 executes blurring processing on the pixel value of the captured image corresponding to the subject 201. .

The first predetermined value and the second predetermined value will be described with reference to FIG. FIG. 15 is a graph corresponding to the graph of FIG. The first predetermined value or less is a parallax range indicated by reference numeral 1500, and the second predetermined value or more is a range indicated by reference numeral 1501.
When the blurring processing unit 1401 _selects a subject with a parallax that is equal to or smaller than a first predetermined value (range indicated by reference numeral 1500), that is, a subject with a parallax that is equal to or smaller than D _{out_min} (minimum stereoscopic parallax) as a subject to be blurred. The blur amount of the subject is increased as the parallax of the subject is smaller (the subject is moved away from the camera position).

  By this processing, an image is generated in which the perceived area is blurred from the range of the parallax that can be stereoscopically viewed. The image subjected to the blurring process is displayed as if the depth of field is shallower than when actually taken. An image with a shallow depth of field has a sense of depth because the background is blurred according to the distance between the subject and the camera.

When the blurring processing unit 1401 _selects a subject with a parallax greater than or equal to a second predetermined value (range indicated by reference numeral 1501), that is, a subject with a parallax greater than or equal to D _{out_max} (maximum stereoscopic parallax) as a subject to be blurred. The blur amount of the subject increases as the parallax of the subject increases (the subject approaches the camera position).

  By this processing, it is possible to blur a range in which stereoscopic viewing is difficult, and thus it is possible to obtain an easily viewable stereoscopic image. Furthermore, when a subject with a parallax less than or equal to a first predetermined value and a subject with a parallax greater than or equal to a second predetermined value are blurred simultaneously, a stereoscopic image with enhanced attractiveness in a stereoscopically viewable range can be obtained.

  The blurring process can be realized by a known method. Typical blur processing methods include a smoothing filter and a Gaussian filter. The smoothing filter is a method of averaging pixel values using pixel values around the target pixel to obtain a pixel value of the processed image. The pixel of interest of the subject to be blurred can be averaged using 3 × 3 pixel values near 8 or averaged using 5 × 5 pixel values near 24. When the peripheral pixel value used for averaging is increased, the blurring amount of the pixel value after processing increases.

That is, when a parallax subject having a first predetermined value or less (D _{out_min} or less) is set as a blur target, the parallax and the blur amount are inversely proportional, and a parallax subject having a second predetermined value or more (D _{out_max} or more) is set as a blur target. In this case, the parallax and the blur amount are made proportional.
That is, the blurring processing unit 1401 increases the blurring amount for the pixel value of the main subject in proportion to the absolute value of the parallax of the main subject in the captured image.

  By performing this blurring process, the depth relationship can be maintained even when the parallax included in the range in which stereoscopic viewing is difficult is clipped. Then, since blur processing is performed for each pixel of the captured image according to the parallax, the blur size of each pixel differs according to the parallax, and a blur effect similar to an image captured by the camera is obtained instead of uniform blurring. .

  The blur processing unit 1401 outputs the captured image after the blur processing is performed to the image generation unit 104.

  Further, as described with reference to FIG. 4, the parallax correction unit 101 corrects the parallax by a method of clipping the parallax, and outputs a parallax image in which the corrected parallax is stored to the parallax conversion unit 103. The parallax conversion unit 103 converts the parallax of the main subject as described above, and outputs a parallax image in which the converted parallax is stored to the image generation unit 104.

  The image generation unit 104 generates a stereoscopic image based on the captured image input from the blur processing unit 1401 and the parallax image input from the parallax conversion unit 103.

  With the above configuration, by performing the parallax conversion, it is possible to easily generate an image in which the stereoscopic effect is enhanced within a range in which the observer can stereoscopically view. Furthermore, by performing blurring processing on the captured image based on the parallax image before performing parallax correction, a stereoscopic image with enhanced stereoscopic effect is generated without the blurring amount being constant even in a range where the parallax is saturated. be able to.

  According to the image processing apparatus of the present invention, it is possible to easily generate a stereoscopic image in which the stereoscopic (depth) feeling between the subject and the background (or foreground) is emphasized within a stereoscopically viewable range.

  The embodiment described above is also applied to an integrated circuit / chip set mounted on an image processing apparatus.

  The image processing apparatus of the present invention can be applied to an image display apparatus capable of displaying a stereoscopic image. By providing the image processing apparatus of the present invention, it is possible to display an image with enhanced stereoscopic effect. It is also possible to apply the image processing apparatus of the present invention to various image display apparatuses having different display surface sizes and resolutions.

  In addition, the present invention can be applied to an image pickup apparatus that can shoot a stereoscopic image, which is preferable because it is possible to shoot while previewing a photographic result of the stereoscopic image.

DESCRIPTION OF SYMBOLS 100, 1100, 1400 ... Image processing apparatus, 101 ... Parallax correction part, 102 ... Main subject determination part, 103 ... Disparity conversion part, 104 ... Image generation part, 1101 ... Disparity calculation part, 1401 ... Blur processing part, 201 ... Subject 202, 203 ... camera, 204 ... left image, 205 ... right image, 206 ... parallax image, 206a, 206b ... pixel.

Claims (7)

A captured image from one viewpoint and a parallax image corresponding to the captured image are input ,
Blur processing for performing blur processing for increasing the blur amount in proportion to the absolute value of the parallax for pixels of the captured image corresponding to a subject whose parallax of the parallax image is less than or equal to a first predetermined value or greater than or equal to a second predetermined value And
A parallax correction unit that saturates parallax outside the predetermined range of the parallax image;
A parallax conversion unit that converts parallax corresponding to at least one main subject of the captured image into a predetermined value;
A stereoscopic image is generated based on the captured image that has been blurred by the blur processing unit and the parallax image that has been corrected by the parallax correction unit and the parallax corresponding to the main subject has been converted by the parallax conversion unit. An image processing apparatus comprising an image generation unit and generating a stereoscopic image. The photographed image is one of two photographed images obtained by photographing the same subject from two viewpoint positions, and the parallax image is a parallax image calculated from the two photographed images. Item 8. The image processing apparatus according to Item 1 . The parallax conversion unit is converted to have the predetermined value based on a linear function that defines a relationship between input parallax and output parallax corresponding to a main subject, and a relationship between the input parallax and output parallax The image processing apparatus according to claim 1, wherein the graph representing the relationship is formed by connecting a plurality of line segments having different inclinations.   4. The image processing apparatus according to claim 1, wherein the predetermined value is 0 or a value within a predetermined range with reference to 0. 5. The image processing apparatus according to claim 1, wherein the predetermined range of the parallax image has a minimum value not more than the first predetermined value and a maximum value not less than the second predetermined value . The image display apparatus comprising the image processing apparatus according to any one of claims 1 to 5. Imaging apparatus comprising the image processing apparatus according to any one of claims 1 to 5.

Images