JP5533529B2 - Image processing apparatus and image processing system - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing system.

  In recent years, the spread of 3D television incorporating 3D video technology has been expanding. However, with regard to content such as 3D video, 3D video content for some movies and televisions is provided. However, 3D video content is sufficient for general users to enjoy individually. I can not say. Therefore, in the future, it will be indispensable for general users to generate content that can easily enjoy 3D video even at home and provide a 3D video system.

  However, if images acquired by two or more cameras are simply displayed as stereoscopic images, there is no particular problem if the display is a fixed size display such as a movie theater, but as stereoscopic images for general users. Since there is a high possibility that the size of each display display owned by the user will be different, depending on the display used, the image may lack a stereoscopic effect.

  That is, in order to perform adjustment for displaying a stereoscopic effect suitable for each display, adjustment of the stereoscopic effect (parallax adjustment) is important. In general, as a method for accurately adjusting the parallax, there is a method of measuring the distance between each target region and correcting the parallax amount according to the measurement data. Here, as a representative method of the distance measurement method of the target region, there is an area-based corresponding point search method.

  For example, Patent Document 1 discloses a technique for calculating parallax using two types of corresponding point search methods, and determining that a pixel whose absolute value of the difference between them is equal to or greater than a predetermined threshold is mismatched. ing.

  Further, in Patent Document 2, when a plurality of corresponding points are searched for one pixel, a target pixel and a subject shape around the target pixel are calculated based on distance values of a plurality of pixels around the target pixel. A technique has been proposed in which a corresponding point for which target distance information having the smallest difference from the subject shape is calculated is determined as an effective corresponding point.

  Furthermore, in Patent Document 3, a reference image is divided into a plurality of small regions, and the parallax of the boundary lines of these small regions is estimated by a region-based stereo method, and the parallax of the boundary line estimated in each region is estimated. The frequency distribution is analyzed to estimate the parallax of the region. By using the image feature amount when sequentially integrating the divided small regions, it is possible to perform image division closer to the original shape of the subject.

JP 2009-239485 A JP 2009-205193 A JP 2005-165969 A

  However, in Patent Document 1, it is not possible to determine which method is appropriate although an error in the measurement point is known. Further, Patent Document 2 has a problem that although it can correct a noisy corresponding point such as an isolated point, it cannot be corrected when surrounding measurement information is incorrect. Further, in Patent Document 3, not only is it necessary to analyze the frequency distribution of the parallax of the boundary line to estimate the parallax of the region, but also a complicated process of switching between the parallax measurement method of the boundary line and other measurement methods is required.

  In addition, in the area-based corresponding point search method, information on both short distance and long distance is included in the set area (distance competing area), or one image is captured by shooting from different viewpoints. If it is not (occlusion region), there is a problem that parallax cannot be measured accurately.

  Then, when multi-viewpoint images are generated based on inaccurate parallax information and stereoscopic display is performed, the stereoscopic images are projected out of the actual stereoscopic shape, have no unevenness, or are blurred, which is difficult for the observer. Due to the lack of perspective and the change in the shape of the object, the image has a lot of discomfort.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing technique that reduces a distance error in a perspective conflict area and reduces a sense of incongruity in perspective in a stereoscopic image. .

In order to solve the above-mentioned problem, the invention of claim 1 is an image processing apparatus for stereoscopic image processing, wherein a first distance image expressing distance information from a predetermined origin position to each part of a subject is acquired. A one-distance image obtaining unit; a boundary line extracting unit that extracts a boundary line between a plurality of areas of the subject in the two-dimensional image of the subject; and the boundary line in the two-dimensional image in the first distance image A virtual boundary line is set at a position corresponding to, and at least one of areas adjacent to each other across the virtual boundary line in the first distance image is set as a correction range, and the direction approaches the virtual boundary line Correction means for generating a second distance image by performing correction for suppressing a change in distance information on the first distance image, wherein the stereoscopic image processing is performed on the second distance image. Done Zui, the correcting means, based on the position of the imaginary boundary line, a correction range setting means for setting the correction range, among the pixels within the correction range, the predetermined correction determination condition is satisfied distance Correction determination means for performing correction determination with only a pixel having information as a correction target pixel, and the correction determination condition is distance information of the determination target pixel in a direction from the determination target pixel toward the virtual boundary line. If the absolute value of the difference between the distance information of the pixel adjacent to the determination target pixel, wherein the condition der Rukoto that is equal to or greater than a predetermined value.

Further, the invention of claim 2 is an image processing apparatus for stereoscopic image processing, and a first distance image acquisition means for acquiring a first distance image expressing distance information from a predetermined origin position to each part of the subject. A boundary line extracting means for extracting a boundary line between a plurality of areas of the subject in the two-dimensional image of the subject; and a virtual position at a position corresponding to the boundary line in the two-dimensional image in the first distance image. A boundary line is set, and at least one of areas adjacent to each other across the virtual boundary line in the first distance image is set as a correction range, and a change in distance information in a direction approaching the virtual boundary line is performed. Correction means for generating a second distance image by performing correction to suppress the first distance image, the stereoscopic image processing is performed based on the second distance image, The correction means includes a correction range setting means for setting the correction range with reference to the position of the virtual boundary line, and only pixels having distance information satisfying a predetermined correction determination condition among the pixels in the correction range. Correction determination means for performing correction determination with the correction target pixel as the correction target pixel, and the correction determination condition is an absolute value of a change rate of the distance information of the determination target pixel in a direction from the determination target pixel toward the virtual boundary line but wherein the condition der Rukoto that is equal to or greater than a predetermined value.

Further, the invention of claim 3 is the image processing apparatus according to claim 1 or 2, wherein the correction range setting means uses the virtual boundary line as the correction range in the first distance image. A pixel range in which the planar distance is equal to or smaller than a predetermined value is set.

The invention according to claim 4 is the image processing apparatus according to claim 1 or 2, wherein the correction range setting means uses the virtual boundary line in the first distance image as the correction range. A pixel range whose difference from the distance information is a predetermined value or less is set.

The invention of claim 5 is the image processing apparatus according to any one of claims 1 to 4, wherein the correction range setting means, as the correction range, the regions on both sides of the said imaginary boundary line It is characterized by setting.

The invention according to claim 6 is the image processing apparatus according to claim 1 or 2, wherein the correction range setting means sets the entire first distance image as the correction range. Features.

The invention of claim 7 is the image processing apparatus according to any one of claims 1 to 6, wherein the correction range setting means, a region where the imaginary boundary line does not form a closed curve, Exception processing to exclude from the correction range is performed.

The invention according to claim 8 is the image processing apparatus according to claim 1 or 2 , wherein the correction determination unit performs the correction determination on a pixel basis.

The invention according to claim 9 is the image processing apparatus according to claim 1 or 2 , wherein the correction determination unit connects the correction determination between two points and intersects the virtual boundary line. It is characterized by being performed in units of pixel columns.

The invention according to claim 10 is an image processing apparatus for stereoscopic image processing, and first distance image acquisition means for acquiring a first distance image expressing distance information from a predetermined origin position to each part of the subject; A boundary line extracting means for extracting a boundary line between a plurality of areas of the subject in the two-dimensional image of the subject; and a virtual position at a position corresponding to the boundary line in the two-dimensional image in the first distance image. A boundary line is set, and at least one of areas adjacent to each other across the virtual boundary line in the first distance image is set as a correction range, and a change in distance information in a direction approaching the virtual boundary line is performed. and a correcting means for generating a second range image by performing suppressing correction on the first distance image, the stereoscopic image processing is performed based on the second distance image, use As a definition, regarding the correction target pixel of interest, an “intersecting pixel column” is defined by a pixel column of a line segment that passes through the correction target pixel and intersects the virtual boundary line; Are defined by pixels adjacent to the correction target pixel on the side closer to the virtual boundary line than the correction target pixel; an “inner pixel” is defined as the correction target pixel in the intersecting pixel column. Defined by a pixel adjacent to the correction target pixel on the side far from the virtual boundary line; “same side pixel column” is the same side of the crossing pixel column as the correction target pixel with respect to the virtual boundary line The “opposite pixel column” is defined by the portion of the intersecting pixel column that is on the opposite side of the correction target pixel across the virtual boundary line. It means, among the same side pixel column opposite pixel column, using the distance information of the inner pixels of at least the same side pixel column, characterized by correcting the distance information correction target pixel.

The invention according to claim 11 is the image processing apparatus according to claim 10 , wherein the correction unit uses a value of distance information of a correction target pixel belonging to the same side pixel column as an inner pixel of the correction target pixel. It is characterized in that correction is performed using the value of the distance information.

The invention according to claim 12 is the image processing apparatus according to claim 10 , wherein the correction unit calculates a change rate of distance information of the correction target pixels belonging to the same side pixel row, from the inside of the correction target pixels. It is characterized in that correction is made constant by using a change rate of distance information in a pixel.

The invention according to claim 13 is the image processing apparatus according to claim 10 , wherein the correction unit sets a reference position at a position away from the virtual boundary line, and obtains distance information at the reference position. When the first distance information is used, and each pixel to be corrected in the section from the reference position to the virtual boundary line is the distance information of a pixel that is symmetrical with respect to the reference position, the second distance information is used as the correction object. The correction target pixel so that the difference between the corrected distance information and the first distance information for the pixel is proportional to a value obtained by inverting the sign of the difference between the first distance information and the second distance information. Correction for suppressing the change of the distance information of the correction target pixel is performed.

The invention according to claim 14 is the image processing apparatus according to claim 10 , wherein the correction unit suppresses a change in distance information in a direction approaching the virtual boundary line in a stepwise manner.

The invention of claim 15 produces an image processing apparatus according to claim 1 or Re claim 14 noise, the first parallax image viewed from the origin position by using the second distance image Based on the first parallax image generation unit and the first parallax image generated by the parallax image generation unit, a second viewpoint image is generated as a distance image obtained by observing the subject from a different position from the origin position. And another viewpoint image generation means.

The invention according to claim 16 is the image processing system according to claim 15 , wherein the correction means generates a second viewpoint image by the other viewpoint image generation means, and the viewpoint is determined from the origin position. According to a direction of changing to the different position, a portion for suppressing the change of the distance information is selected.

According to the first to sixteenth aspects of the present invention, by performing a correction that suppresses a change in distance information in a direction approaching the boundary, with a boundary region in which an error in the first distance image tends to be large as a correction range, It is possible to reduce or eliminate an error caused by mixing the distance information of the other area into the distance information of the one area across the boundary. For this reason, by performing the post-process of stereoscopic image processing based on the second distance image obtained in this way, the distance error near the boundary of the image area is reduced, and the sense of discomfort in perspective viewing in the stereoscopic image is reduced. It becomes possible.

According to the invention of claims 1 to 7, with reference to the position of the imaginary boundary line, by performing the setting of the correction range, the distance information calculated weak regions, i.e., discomfort on perspective viewing the stereoscopic image The correction range can be set in the vicinity of the region boundary line that is likely to occur. For this reason, it is possible to improve the efficiency of correction.

According to the inventions of claim 1, claim 2, claim 8, and claim 9 , it is necessary to correct the distance information by correcting only the pixels that are highly necessary to perform the distance information correction by performing the correction determination. Time can be shortened.

According to the tenth to fourteenth aspects of the present invention, the distance information of the correction target pixel is corrected using at least the distance information of the inner pixel of the same-side pixel column out of the same-side pixel column and the opposite-side pixel column. By doing so, it is possible to reduce the distance error in the vicinity of the image region boundary while ensuring the continuity with the distance information of the inner pixels, and to reduce the sense of discomfort in perspective viewing in the stereoscopic image.

According to the invention of claim 16, in the generation of the second viewpoint image by the other viewpoint image generation means, the portion for suppressing the change of the distance information according to the direction in which the viewpoint changes from the origin position to another position. By selecting, correction according to the viewpoint movement direction is possible.

1 is a diagram illustrating a schematic configuration of an image processing system according to an embodiment of the present invention. It is a figure which illustrates the image image | photographed with the stereo camera, and the image of the obtained distance information. 1 is a block diagram showing a functional configuration of an image processing system according to the present embodiment. 1 is a block diagram showing a functional configuration of an image processing system according to the present embodiment. It is a figure explaining the usage pattern of a stereo camera. It is a figure which shows the imaging | photography image in the case of using a stereo camera horizontally. It is a figure which shows the image obtained with the stereo camera. It is a figure explaining the image boundary line extraction method. It is a figure explaining the image boundary line extraction method. It is a figure explaining the image boundary line extraction method. It is a conceptual diagram explaining corresponding point search processing. It is a conceptual diagram explaining the setting method of a correction range. It is a conceptual diagram explaining the setting method of a correction range. It is a figure explaining a correction judgment object position. It is a figure explaining a correction judgment object position. It is a figure explaining the correction method. It is a figure explaining the correction method. It is a figure explaining the correction method. It is a figure explaining the correction method. It is a figure explaining the correction method. It is a figure explaining the correction method. It is a flowchart explaining the basic operation | movement implement | achieved in the image processing system which concerns on this embodiment.

<1. Definition of terms>
The following terms in this invention are defined as follows.
The term “image” is used as a concept term including both still images and moving images.

In addition, regarding the correction target pixel of interest,
The term “intersecting pixel column” is used as a pixel column that passes through the pixel to be corrected and intersects a virtual boundary line (described later);
The term “outside pixel” is used as a pixel adjacent to the correction target pixel on the side closer to the virtual boundary line than the correction target pixel in the intersection pixel row;
The term “inner pixel” is used as a pixel adjacent to the correction target pixel on the side of the intersecting pixel row that is farther from the virtual boundary line than the correction target pixel;
The term “same-side pixel column” is used as a portion of the intersecting pixel column that is on the same side as the correction target pixel with respect to the virtual boundary line;
The term “opposite pixel column” is used as a portion of the intersecting pixel column that is on the opposite side of the correction target pixel across the virtual boundary line.

<2. Overview of image processing system>
FIG. 1 is a diagram showing a schematic configuration of an image processing system 1 according to an embodiment of the present invention. The image processing system 1 is configured as a multi-view camera system, and includes an image processing device 3 connected to the stereo camera VC so as to be able to transmit and receive data, in addition to the binocular stereo camera VC as the imaging unit 10. Prepare.

  The stereo camera VC is composed of two imaging systems, a main camera MC and a sub camera SC. The main camera MC and the sub camera SC are configured to image a subject (imaging target) OB in front of the camera from different viewpoints at the same timing. Two image signals (hereinafter abbreviated as “image”) obtained by imaging at the same timing by the main camera MC and the sub camera SC are transmitted to the image processing device 3 via the data line CB.

  Hereinafter, an image acquired by imaging of the main camera MC is referred to as “first captured image G1”, and an image acquired by imaging of the sub camera SC is referred to as “second captured image G2”. That is, the first and second captured images G1 and G2 form a set of images in which the same subject OB is captured from different viewpoints.

  The image processing device 3 is configured by an information processing device such as a personal computer (personal computer), for example, an operation unit 31 including a mouse and a keyboard, a display 32 configured by a liquid crystal display, a speaker 33, and a stereo camera. And an interface (I / F) 37 for receiving data from the VC. The image processing apparatus 3 includes a storage device 34 and an arithmetic control unit 36.

  The storage device 34 is composed of, for example, a hard disk and stores first and second captured images G1 and G2 obtained by imaging with the stereo camera VC. Further, the storage device 34 stores a program PG for performing distance information correction described later.

  The input / output unit 35 includes, for example, a portable disk drive, sets a portable storage medium such as an optical disk, and exchanges data with the arithmetic control unit 36.

  The arithmetic control unit 36 includes a CPU 36 a that functions as a processor and a memory 36 b that temporarily stores information, and comprehensively controls each unit of the image processing apparatus 3. In the arithmetic control unit 36, various functions, various information processing, and the like are realized by reading and executing the program PG in the storage unit 34. The memory 36b can store program data stored in the portable storage medium via the input / output unit 35. This stored program can be appropriately reflected in the operation of the image processing apparatus 3.

  The calculation control unit 36 generates a parallax image and another viewpoint image by distance information correction described later, and visually outputs an image of a specific subject OB on the display 32.

<2-1. General properties and assumptions of measurement distance information>
As a preparation for explaining the details of the image processing system 1 in this embodiment, the general nature of the measurement distance information that is the premise of this embodiment and the accompanying phenomenon, that is, the circumstances that have occurred in the prior art will be explained. deep.

  FIG. 2 is a diagram illustrating an image photographed by the stereo camera VC and an image of the obtained distance information. FIG. 2A shows a first captured image G1 acquired by imaging of the main camera MC, and FIG. 2B shows a second captured image G2 acquired by imaging of the sub camera SC. Note that, here, the first captured image G1 is a reference image SG, and the second captured image G2 is a reference image RG. FIG. 2C shows a distance expressing distance information from a predetermined origin position (the position of the stereo camera VC in this embodiment) to each part of the subject, which is generated from the standard image SG and the reference image RG. An information image G3 (first distance image D1) is shown.

  As shown in FIG. 2C, in the distance information image G3 (first distance image D1), the subject OB1 is relatively far from the stereo camera VC corresponding to a predetermined origin position compared to the subject OB2. Although the expression is close, the region MX corresponding to the region boundary between the subject OB1 corresponding to the short-distance region and the subject OB2 corresponding to the long-distance region has a phenomenon that stereoscopic viewing is difficult. This is because there is a lack of stereoscopic vision because the long distance information of the area of the subject OB2 is mixed into the short distance information of the area of the subject OB1 across the boundary due to an error that occurs when performing the corresponding point search described later. Depending on the circumstances displayed as an image.

  As described above, the distance measurement at the region boundary such as the contour portion of the subject tends to obtain an erroneous distance measurement result including distance conflict. Therefore, if a stereoscopic image is generated based on the inaccurate distance information, a discomfort in perspective in the stereoscopic image is generated.

  Under such a background, the present invention corrects distance information so as to reduce the distance error in the vicinity of the image region boundary.

<3. Functional configuration of image processing system>
Returning to the description of the embodiment of the present invention. In the image processing apparatus 3, a functional configuration realized by the arithmetic control unit 36 in order to execute the correction of distance information will be described.

  FIG. 3 is a block diagram showing a functional configuration of the image processing system 1 according to the present embodiment. As shown in FIG. 3, the distance measurement unit 20 performs distance measurement from the standard image SG and the reference image RG obtained by the standard camera MC and the reference camera SC of the stereo camera VC, and the first distance image acquisition unit 40 performs the distance measurement. A first distance image D1 expressing distance information from a predetermined origin position to each part of the subject is acquired. Subsequently, the boundary line extraction unit 30 extracts a boundary line between a plurality of regions of the subject in the two-dimensional image of the subject from the reference image SG, and the distance based on the extracted boundary line and the first distance image D1. The information correction unit 50 corrects the distance information, and the second distance image generation unit 60 generates the second distance image D2. Then, based on the corrected distance information, the parallax image generation unit 70 generates a first parallax image. Finally, the other viewpoint image generation unit 80 generates an image from an arbitrary viewpoint, thereby performing image processing that enables stereoscopic viewing, and displays the stereoscopic image on the display 32.

  FIG. 4 is a block diagram specialized in the contents performed by the distance information correction unit 50 in the image processing system 1 (see FIG. 3) according to the present embodiment. As shown in FIG. 2, the correction range setting unit 41 sets the correction range based on the boundary line extracted by the boundary line extraction unit 30 and the first distance image obtained by the first distance image acquisition unit 40. Set. The correction determination unit 42 determines whether or not correction is necessary for the correction range. If the determination result indicates that correction is necessary, the correction target of the distance information is set.

  Finally, the correction distance generation unit 43 generates a correction distance by performing the selected correction on the first distance image D1.

  Although the functional configuration of the arithmetic control unit 36 as shown in FIGS. 3 and 4 is described as being realized by executing the program PG, it may be realized by a dedicated hardware configuration.

  Thereafter, the imaging unit 10, the boundary line extraction unit 30, the distance measurement unit 20, the first distance image acquisition unit 40, the distance information correction unit 50, the second distance image generation unit 60, the parallax image generation unit 70, and other viewpoint image generation The specific contents of each process of the unit 80 will be sequentially described with reference to FIGS.

<3-1. Imaging unit 10>
FIG. 5 is a diagram illustrating a specific arrangement example of the stereo camera VC of the imaging unit 10. The stereo camera VC1 shown in FIGS. 5A and 5B has a configuration in which the main camera MC and the sub camera SC are separated from each other by the base line length L. The main camera MC and the sub camera SC are They are arranged so as to be parallel to one side of the camera casing.

  A state in which the stereo camera VC1 is arranged so that the arrangement of the main camera MC and the sub camera SC is perpendicular to the horizontal plane is referred to as vertical installation (see FIG. 5A). On the other hand, a state in which the stereo camera VC1 is arranged so that the arrangement of the main camera MC and the sub camera SC is parallel to the horizontal plane is referred to as horizontal placement (see FIG. 5B).

  Further, the stereo camera VC2 shown in FIG. 5C is the same as the stereo camera VC1 in that the main camera MC and the sub camera SC are arranged apart from each other by the base line length L. The cameras SC are arranged so as to be inclined with respect to any side of the camera casing, and this state is referred to as oblique placement. In addition, you may arrange | position so that up and down, right and left may be switched with respect to the above arrangement | positioning states.

  As a configuration example, for example, the main camera MC is a digital camera system having a so-called zoom lens with a high-resolution, variable focus, such as a high-definition broadcast lens (HDTV lens) that can capture a high-definition image, and the sub camera SC. Can be a low-resolution, single-focus digital camera system such as a small unit camera or a micro camera unit (MCU) mounted on a mobile phone or the like. Alternatively, both may be configured with a high-definition broadcast compatible lens / micro camera unit. A zoom lens may be used as the lens of the sub camera SC, but high resolution is not required. As described above, the two lens units may have any configuration. However, when generating a stereoscopic image, the image on the main camera MC side is used as the other viewpoint image. It is desirable to use a lens having the same level or a better performance than the sub camera SC.

  When an image obtained by the stereo camera VC is used as a stereoscopic image as it is, it is necessary to be configured horizontally, but in this method, distance measurement is performed, and an image for stereoscopic vision is based on the distance information. Therefore, it is sufficient that the reference image SG for generating the distance information and the other viewpoint image can be acquired. Therefore, not only the horizontal arrangement shown in FIG. 5B but also the vertical arrangement shown in FIG. Alternatively, the obliquely arranged configuration shown in FIG. However, in the horizontal configuration of FIG. 5B, another viewpoint image may be generated based on the image acquired by the sub camera SC.

  FIG. 6 is a schematic diagram illustrating a shooting situation when the stereo camera VC1 is used in a horizontal position, and corresponds to a situation in which the subject OB and the background BG are viewed from above. The vertical axis in FIG. 6 indicates the distance to the subject OB and the background BG in the optical axis direction of the main camera MC when the position of the main camera MC is the origin. The horizontal axis indicates the distance in the base line direction (horizontal direction) connecting the main camera MC and the sub camera SC, and the shootable range in this direction is the horizontal direction when shooting with the main camera MC and the sub camera SC. Corresponds to the angle of view. In the main camera MC, an example is shown in which shooting is performed at three different magnifications. An image shot at the lowest magnification (the widest angle of view) is referred to as a first main camera image, and the angle of view is referred to as the first camera image. This is indicated by line L1. Next, an image taken at a high magnification is referred to as a second main camera image, and the angle of view is indicated by a line L2. An image taken with the highest magnification (the narrowest angle of view) is referred to as a third main camera image, and the angle of view is indicated by a line L3. On the other hand, since the sub camera SC does not have a zoom function, there is only one type of angle of view, and the angle of view is indicated by a line L4.

  FIG. 7 shows an image obtained from the photographed image shown in FIG. 7A shows the first main camera image taken by the main camera MC, FIG. 7B shows the second main camera image taken by the main camera MC, and FIG. 7C shows the main image. The 3rd main camera image image | photographed with the camera MC is shown.

  FIGS. 7D to 7F show sub camera images obtained by the sub camera SC when the images of FIGS. 7A to 7C are acquired, respectively. . These sub-camera images are all images having the same magnification. Since the optical axis of the secondary camera SC is separated from the main camera MC by the baseline length L, there is a parallax between the image obtained by the secondary camera SC and the image obtained by the primary camera MC. There is a deviation in the observation direction.

  As described above, the two cameras, the main camera MC and the sub camera SC, capture the same subject OB and the background BG. However, since the line of sight and the magnification of the lens are different, the size and angle of view of the subject OB are different. Different images will be obtained.

<3-2. Boundary line extraction unit 30>
Subsequently, the boundary line extraction unit 30 extracts boundary lines between a plurality of regions in the two-dimensional image of the reference image SG acquired by the imaging unit 10. That is, a boundary line between a plurality of areas constituting the subject OB and the background BG is extracted from the reference image SG serving as a base for generating another viewpoint image. In FIG. 3, the image G1 obtained by the main camera MC is used as the reference image SG. In addition, the “region” in this case refers to a spatial range in which brightness, saturation, hue, and the like change almost continuously in the image. In the example of the image as shown in FIG. The area, the trunk area of each tree in the background, the empty area, etc. Usually, the mutual boundary between these regions is an outline. The boundary line extraction unit 30 performs processing for extracting an outline (edge) for the image region. However, the edge information extraction method does not need to use a special method, and is based on a general method. Extract. For example, the following methods are mentioned.

<3-2-1. First Method for Extracting Image Boundary Line>
FIG. 8 and FIG. 9 are diagrams for explaining the first method of image boundary line extraction. FIG. 8 illustrates the reference image SG obtained by the main camera MC as an input image, and FIG. 9 shows a diagram in which a general edge detection filter process is performed on the reference image SG. As shown in FIG. 9, the contour information can be extracted as a region candidate based on the edge information of the subject OB and the background BG.

<3-2-2. Second Method for Extracting Image Boundary Line>
FIG. 10 is a diagram for explaining a second method of image boundary line extraction. The input image is the same as the reference image SG in FIG. 8, but the image boundary is based on color information or luminance information with the periphery. It is a figure explaining the method of extracting a line. Specifically, the color information or luminance information of a certain pixel is compared with the surrounding color information or luminance information, and if the difference value is smaller than a predetermined threshold, they are integrated as the same region, The target region is extracted according to the size, and edge detection is performed on the result after extraction.

<3-2-3. Other methods of image boundary extraction>
In addition to the first and second methods, region candidate extraction can also be performed by a general region cutout method. For example, a known average shift method (Mean Shift method) disclosed in Japanese Patent Application Laid-Open No. 2010-122734 or a graph cut known in Japanese Patent Application Laid-Open No. 2010-110556 can be used.

<3-3. Distance Measurement Unit 20 and First Distance Image Acquisition Unit 40>
The distance measuring unit 20 measures the distance from the predetermined origin position to each part of the subject. Based on the measurement distance result, the first distance image acquisition unit 40 acquires the first distance image D1 expressing the distance information. In this embodiment, the first distance image D1 is acquired by the following corresponding point search process.

  In general, in the corresponding point search processing, a point (corresponding point) on the corresponding point reference image corresponding to an arbitrary target point on the corresponding point reference image is searched and obtained, and the relationship between the target point and the corresponding point obtained is obtained. After obtaining the parallax information from the information, distance information is obtained from the parallax information. The corresponding point reference image is an image corresponding to the corresponding point standard image. Specifically, in a stereo image, one of a pair of images captured at the same time is a corresponding point reference image, and the other is a corresponding point reference image. In the time-series images, among images taken by the same camera, the temporally previous image is the corresponding point reference image, and the temporally subsequent image is the corresponding point reference image. A template is set for the target point on the corresponding point reference image, a window on the corresponding point reference image corresponding to the template is searched, and a corresponding point is obtained from the searched window.

  FIG. 11 is a conceptual diagram illustrating corresponding point search processing in this embodiment. FIG. 11A shows the second captured image G2 obtained by the sub camera SC as the corresponding point reference image CRG, and FIG. 11B shows the first captured image obtained by the main camera MC as the corresponding point reference image CSG. G1 is shown. For each pixel in the corresponding point reference image CSG, the corresponding pixel in the region CP surrounded by the broken line of the corresponding point reference image CRG is obtained. Note that the entire region of the corresponding point reference image CSG corresponds to the region CP of the corresponding point reference image CRG, and the region CP is enlarged.

A specific method for searching for corresponding points is known, and examples of a method for searching for corresponding points based on phase information include the following methods.
-Corresponding point search using Phase Only Correlation (POC)-Corresponding point search using Phase Shift Analysis (PSA) As a method for searching corresponding points based on luminance information Examples of the method include the following methods.
-Corresponding point search using SAD (Sum of Absolute Difference) method-Corresponding point search using SSD (Sum of Squared intensity Difference) method
Corresponding point search using NCC (Normalized Cross Correlation) method Then, based on the result of such corresponding point search, a first distance image D1 expressing the distance from the stereo camera to each part of the subject in pixel units is generated.

  In generating the first distance image D1 using the stereo camera, when an image having a small number of pixels of the two images is used as a reference image, a corresponding point search is performed in units of subpixels.

  FIG. 11C is a diagram schematically illustrating an example in which the second captured image G2 in FIG. 11A is adopted as the corresponding point reference image CSG, and subpixels are set thereto. Further, FIG. 11D shows an example in which an image corresponding to the first captured image G1 of FIG. 11B having the larger number of pixels is used as the corresponding point reference image CSG.

  In the corresponding point search processing using the correlation method, a process of sampling the corresponding point reference image CRG pixel by pixel in order to obtain the corresponding point on the corresponding point reference image CRG for the attention point on the corresponding point reference image CSG. Is made.

  As shown in FIG. 11 (d), when the magnification of the corresponding point standard image CSG is high and the magnification of the corresponding point reference image CRG is low, that is, the side with the larger number of effective pixels is used as the corresponding point standard image CSG. In this case, the corresponding point search process can be performed up to the sub-pixel unit without performing a complicated process. On the other hand, as shown in FIG. 11C, when the magnification of the corresponding point reference image CSG is low and the magnification of the corresponding point reference image CRG is high, the sampling interval is significantly different, and the corresponding point reference is performed. The information of the image CRG falls out and it becomes difficult to obtain accurate distance information. The correction processing of distance information as described later is mainly intended to compensate for errors in distance information in the vicinity of the region boundary, but is also significant as compensation for errors caused by such subpixel search.

  FIG. 11C shows an example in which one pixel is divided into three in the horizontal direction and three subpixels are set in order to perform the corresponding point search process up to the subpixel unit. However, the present invention is not limited to this, and can be further finely divided. Thereby, even when the magnification is different between the corresponding point reference image CSG and the corresponding point reference image CRG, the sampling intervals can be matched. That is, when the attention point OP is a sub-pixel level position, the search template TP is set around the sub-pixel SP that is the attention point OP.

  As a sub-pixel estimation method, for example, in Japanese Patent Application Laid-Open No. 2001-195597, after calculating a correlation value between images, a pixel is calculated based on a positional relationship between a position having the highest correlation value and its surrounding correlation values. A method for interpolating a correlation value between a pixel and a pixel by applying a linear equation or a curve equation to estimate a peak position and a peak value of the correlation value is disclosed, and this estimation method can be used.

  In any of these corresponding point search methods, there are target regions that are not good at calculating distance information, such as near the region boundary. This is because the image information of the other region is likely to be mixed in the corresponding point search in one region across the boundary line of the region. This is not only due to the unclearness of the image near the edge, but also because it compares two images of the subject viewed from different lines of sight (or two images of the subject viewed at different times). Is also related to being easily affected by parallax.

  In addition to the generation of the distance image by such a corresponding point search, even if the distance image is generated by performing the distance measurement by another method which will be described later as a `` variation example '', the distance calculation error is still in the vicinity of the region boundary. Is likely to occur.

  Therefore, in the embodiment of the present invention, the following distance information correction processing is performed.

<3-4. Distance Information Correction Unit 50 and Second Distance Image Generation Unit 60>
Since the first distance image D1 acquired by the first distance image acquisition unit 40 tends to include a measurement error due to distance competition in the contour portion of the subject, as described above with reference to FIG. 2, the distance information correction unit 50 to correct. At that time, first, in the first distance image D1, a virtual boundary line is set at a position corresponding to the boundary line extracted by the boundary line extraction unit 30 in the reference image SG. Next, at least one of areas adjacent to each other across the virtual boundary line in the first distance image D1 is set as a correction range. Then, the first distance image D1 is corrected to suppress the change in the distance information in the direction approaching the virtual boundary line, and finally the second distance image D2 is generated by the second distance image generation unit 60. . However, “suppression” here is a concept including both a case where the change is made zero and completely flat, and a case where the absolute value of the rate of change is reduced.

  Here, the “virtual boundary” is called for the following reason. That is, since the error near the boundary is large in the first distance image D1, it is not preferable to specify the position of the boundary line from the first distance image D1 itself. Accordingly, the boundary line detected in the reference image SG, which is a normal two-dimensional image, is used to estimate the position of the boundary line in the first distance image D1, and the boundary line position is the first distance image D1 itself. This is because the first distance image D1 is a borrowed boundary line or a virtual boundary line.

  In the distance information correction unit 50, a correction range setting unit 41 that sets a correction range with reference to the position of the virtual boundary line, and a pixel having distance information that satisfies a predetermined determination condition among pixels in the correction range A correction determination unit 42 that performs correction determination using only the correction target pixel, and a correction distance generation unit 43 that generates a correction distance (see FIG. 4). That is, the correction of the distance information is performed by setting the correction range, the presence / absence of correction, and the correction method for the first distance image D1 based on the edge information calculated by the boundary line extraction unit 30. To implement. In addition, although the position where a measurement error occurs differs between the above-described corresponding point search method and another distance measurement method, the distance information correction unit 50 can correct it without any problem.

  Hereinafter, regarding the correction of the distance information, the functions of the correction range setting unit 41, the correction determination unit 42, and the correction distance generation unit 43 in FIG. 4 will be sequentially described.

<3-4-1. Correction range setting unit 41>
12 and 13 are conceptual diagrams illustrating a method for setting a correction range in the first distance image D1. As a conceptual diagram, when there is a subject portion where the short-distance area ND and the long-distance area FD overlap as shown in the lower side of FIGS. 12A and 12B, two adjacent areas ND , FD, the distance information shown on the upper side of FIG. 12A can be obtained. However, in reality, as shown in the distance measurement result shown on the upper side of FIG. 12B, in the distance information corresponding to the area boundary vicinity AB of the area AR, an erroneous distance due to the influence of perspective conflict is likely to occur. This is because information on both the long distance and the short distance is included in the vicinity of the area boundary AB of the area AR as described above. Then, when the above-described area-based corresponding point search is performed on such a region boundary vicinity AB, the possibility of an erroneous distance increases. Therefore, in order to correct the distance information in consideration of such a region boundary, the correction range setting unit 41 sets the correction range on the basis of the position of the virtual boundary line described above. Examples of the correction range setting method include the following methods.

<3-4-1-1. First method of range setting>
The correction range setting unit 41 sets a pixel range in which the planar distance from the virtual boundary line is a predetermined value or less in the first distance image D1 as the correction range. That is, the range is set based on the edge information extracted by the boundary line extraction unit 30.

  FIG. 13 is a diagram illustrating the setting of the correction range. FIG. 13A shows the region boundary line NB of the short-distance region ND and the region boundary line FB of the long-distance region FD extracted by the boundary line extraction unit 30 in the reference image SG. In FIG. 13B, in the first distance image D1, the virtual boundary line NB ′ is shown in the short distance area ND and the virtual boundary line FB ′ is shown in the far distance area FD at the position corresponding to the boundary line in the reference image SG. Show. In FIG. 13C, pixel ranges in which the planar distance (distance on the two-dimensional image) from the virtual boundary lines NB ′ and FB ′ in the first distance image D1 is equal to or smaller than a predetermined value are corrected ranges NA and FA, respectively. It is the figure which showed setting.

  As shown in FIG. 13, the edge information is used as the region boundary of each subject, and the correction processing range is set for the position on the extracted edge information or the region in the range specified in advance. Here, the width of the range to be specified may be uniform with respect to each region boundary. However, as shown in FIG. 13C, the width of the band-shaped region that defines the correction range NA of the short-range region ND. Wn is larger than the width Wf of the band-like region that defines the correction range FA of the long-distance region FD. That is, the correction range of the first distance method image D1 is a band-like area that is widened by two plane distances Wf / 2 and Wn / 2 on both sides of the virtual boundary lines FB ′ and NB ′. The planar distance from the boundary lines FB ′ and NB ′ to the outer edge of the correction range can be variably set according to the distance information of the region surrounded by the virtual boundary line.

  For example, the correction range FA of the long-distance area FD sets a window size area set when the corresponding point search is performed as a correction target range, and the correction range NA of the short-distance area ND indicates the distance information calculation result. It may be implemented in a range having a width corresponding to the maximum parallax among the parallaxes of each part of the subject obtained in step (b). This is because the parallax increases as the subject portion becomes closer, so the maximum parallax reflects the distance from the camera to the nearest portion of the subject. By making it variable by such a method, an optimal correction process according to the scene can be performed. This method is effective, for example, for searching for corresponding points after multi-resolution conversion.

<3-4-1-2. Second method of range setting>
The correction range setting unit 41 sets a pixel range having distance information whose difference from the distance information of the virtual boundary line is equal to or less than a predetermined value in the first distance image D1 as the correction range. That is, in the case of FIG. 13, the distance information of each pixel in the short distance area ND (or the long distance area FD) is compared with the distance information of the virtual boundary line NB ′ (or the virtual boundary line FB ′). A range including pixels whose distance information difference is equal to or less than a predetermined threshold is set as the correction range FA (or correction range NA).

  The “difference amount” as an index here can be expressed by a difference or a ratio. Similarly to the first method, the threshold value of the difference amount of the distance information may be uniform for each region boundary, but may be changed according to the distance information for each virtual boundary line.

<3-4-1-3. Third method of range setting>
The correction range setting unit 41 sets the entire first distance image D1 as a correction range. That is, the correction range setting method is not limited to the vicinity of the region boundary as described above, and the entire range may be set as the correction range. Also in this case, since the boundary area is included in the correction range, it is included in the concept of “correction in the boundary area” corresponding to the feature of the present invention.

  In the first and second setting methods described above, the correction range setting unit 41 sets areas on both sides of the virtual boundary line as the correction range.

  However, in the first to third methods, an exception process in which a region where the virtual boundary line does not form a closed curve is excluded from the correction range by the correction range setting unit 41 is performed. This is to prevent the linear noise in the reference image from being treated as a contour line. The “closed curve” here is a concept including a closed figure formed by a broken line.

  When determining whether or not to form a closed curve, the rectangular line corresponding to the outer frame of the screen range is also regarded as a contour line, and the closed curve is extracted, so that the subject area that is only partially within the angle of view is also subject to correction. It can be.

  Conversely, by not treating the rectangular line corresponding to the outer frame of the screen range as the contour line, only the closed curve corresponding to the original contour line can be corrected. Usually, since the main subject portion is in the center area of the screen, there is little influence even if the area where the outline is cut off at the edge of the screen is excluded from the correction target.

  The former can be selected when emphasizing the precision of correction, and the latter can be selected when emphasizing high-speed correction processing.

<3-4-2. Correction Determination Unit 42>
The correction determination unit 42 performs correction determination in which only pixels having distance information satisfying a predetermined determination condition among pixels within the correction range are set as correction target pixels. That is, the correction determination unit 42 determines whether or not to perform correction within the correction range set by the correction range setting unit 41. Examples of correction determination target positions and correction determination conditions are as follows.

<3-4-2-1. Correction determination target position: first setting method>
FIG. 14 is a diagram for explaining a first setting method of the correction determination target position. As shown in FIG. 14, the determination is performed for each pixel P within the set correction range, that is, for each pixel. That is, the correction determination unit 42 performs correction determination on a pixel basis as a correction determination target.

<3-4-2-2. Correction determination target position: second setting method>
FIG. 15 is a diagram for explaining a second setting method of the correction determination target position. As shown in FIG. 15, the determination is performed for all the pixels included between the two pixels of the pixel P1 and the pixel P2 that straddle the virtual boundary line NB ′ within the set correction range. Is done. However, the pixels P1 and P2 are each selected within the correction range on both sides of the virtual boundary line.

  That is, the correction determination unit 42 treats the correction determination as a correction determination target and treats a pixel row that connects two points and intersects the virtual boundary line as a unit, and a predetermined number of pixels in the pixel row (or a predetermined ratio or more). If the pixel) satisfies the following correction determination condition, correction is performed for each pixel in the correction range to which the pixels P1 and P2 belong.

<3-4-2-3. Correction determination target position: third setting method>
The correction determination unit 42 performs determination for all pixels in each region. That is, in the case of FIG. 14 and FIG. 15, the determination is performed on all the short distance area ND and the long distance area FD. Therefore, this third setting method is suitable for the combination with the “third method of range setting” described above.

<3-4-2-4. Correction judgment conditions>
The correction determination unit 42 determines whether to correct the above-described determination target pixel.

  A first example preferable as the determination condition is that when the distance information of the determination target pixel in the direction from the determination target pixel toward the virtual boundary line is R, the distance information R and a pixel adjacent to the determination target pixel The absolute value | ΔR | of the difference from the distance information (R + ΔR) is equal to or greater than a predetermined threshold value. That is, when the change in the distance information in the vicinity thereof is large, it is determined that the distance information in the region opposite to the virtual boundary line is affected, and is set as a correction target.

  Since the horizontal distance between adjacent pixels is fixed, the determination criterion using the difference ΔR as an index is based on the determination using the differential coefficient of the change in the distance information R as an index and the inclination angle of the change in the distance information R as an index. It is equivalent to the determination to do.

  However, for the purpose of excluding pixels whose difference is accidentally increased due to noise, the difference ΔR is not only a difference from one pixel adjacent to the determination target pixel but also adjacent to the determination target pixel. An average of difference values between a plurality of pixels may be used as an index.

  Another example of the determination condition is that the absolute value | ΔR / R | of the change rate ΔR / R of the distance information of the determination target pixel in the direction from the determination target pixel toward the virtual boundary line is a predetermined value or more. It is a condition of being. This rate of change may be a rate of change between two adjacent pixels, or may be an average rate of change between three or more pixels adjacent to the determination target pixel.

<3-4-3. Correction Distance Generation Unit 43>
Correction is performed on the basis of the determination result of the correction presence / absence performed by the correction determination unit 42 within the correction range set by the correction range setting unit 41.

  Among these, the distance information correction unit 50 generates a correction distance by using the distance information of at least the inner pixel of the same side pixel column, among the same side pixel column and the opposite side pixel column, the correction distance generation unit 43, The distance information of the correction target pixel is corrected.

  The contents of a preferable correction method employed by the distance information correction unit 50 are as follows. In each of the drawings after FIG. 16 referred to, the horizontal axis indicates the pixel position, and the vertical axis indicates the distance information. The boundary reference position QM is the position on the virtual boundary line of both ends of the pixel column of the correction target area CA, and the initial reference position Q1 is the one that is not the boundary reference position QM of both ends of the pixel column of the correction target area CA. Is the position. Furthermore, although only one area CA of the correction target areas CA and RA on both sides of the virtual boundary line will be described below, the same correction can be performed for the other area RA. The initial reference position for the other area RA exists on the opposite side of the start reference position Q1 of the one area CA across the boundary reference position QM.

  In each figure, (a) shows distance information before correction, and (b) shows distance information after correction. The symbol S (i) is a distance information value for each pixel position, and the subscript “i” is an identification code for the pixel position.

<3-4-3-1. First Correction Method>
FIG. 16 is a diagram illustrating the first correction method. FIG. 16A shows that pixels belonging to the same pixel column before correction are the correction target area CA, and that the reference target pixel column RA exists as an opposite pixel column with the boundary reference position QM as a boundary. 16 (b) shows the pixel value (distance information) after correction in FIG. 16 (a).

  As shown in FIG. 16, in the first correction method, the distance information value S (N) that is the correction target has the distance information value S (N−1) corresponding to the outer pixel, and the distance information value S (N). To correct. The distance information value S (N) may be used instead of a fixed value determined in advance.

  As described above, the distance information correction unit 50 performs correction to make the distance information value of the correction target pixel belonging to the same side pixel row constant by using the distance information value of the inner pixel of the correction target pixel.

<3-4-3-2. Second Correction Method>
FIG. 17 is a diagram illustrating the second correction method. FIG. 17A shows that the pixels belonging to the same pixel column before correction are the correction target region CA, and that the reference target region RA exists as the opposite pixel column at the boundary reference position QM. FIG. 17B shows the corrected pixel in FIG. Here, for each pixel position, an inclination angle corresponding to a change in distance information between adjacent pixels is θ (i). The suffix “i” is a pixel position identification code.

  As shown in FIG. 17, in the second correction method, the inclination angle θ (N) at the correction target position QN is changed to the inclination angle θ (N) with the inclination angle θ (N−1) corresponding to the outer pixel. Make corrections. It should be noted that the inclination angle may not be θ (N) but may be corrected by a fixed value determined in advance.

  As described above, the distance information correction unit 50 performs the correction to make the change rate of the distance information of the correction target pixels belonging to the same side pixel row constant by using the change rate of the distance information in the inner pixel of the correction target pixel. Do.

<3-4-3-3. Third Correction Method>
FIG. 18 is a diagram illustrating the third correction method. In this method, for each correction target pixel Pa in the section from the initial reference position Q1 to the virtual boundary line (boundary reference position QM), distance information of a pixel (symmetric pixel) Pb that is symmetric with respect to the initial reference position Q1. When Sb is set, the difference between the corrected distance information Sa ′ for the correction target pixel Pa and the distance information S0 at the initial reference position Q1 is proportional to the sign of the difference between the distance information S0 and the distance information Sb. As described above, the correction of the correction target pixel Pa suppresses the change in the distance information of each correction target pixel.

In particular,
Sa'-S0 = Sb-S0
Will be corrected to satisfy the relationship, equivalently,
Sa = 2 × S0−Sb
Is the distance information value after correction of the correction target pixel Pa.

For example,
Sa = 100; S0 = 40; Sb = 20
In the case of (where Sa is the distance information before correction of the pixel of interest Pa),
Sa ′ = 2 × 40−20 = 60
It is corrected to.

More generally, using a predetermined proportional coefficient k (where k> 0), Sa′−S0 = k × (Sb−S0)
In other words, Sa ′ = (1 + k) × S0−k × Sb
In this case, k = 1 corresponds to the above example.

  As described above, the distance information correction unit 50 sets the reference position at a position away from the virtual boundary line, sets the distance information at the reference position as the first distance information, and within the section from the reference position to the virtual boundary line. For each correction target pixel, when the distance information of the pixel at a symmetric position with respect to the reference position is the second distance information, the difference between the corrected distance information and the first distance information for the correction target pixel is the first distance. Correction is performed on the correction target pixel so as to be proportional to a value obtained by inverting the sign of the difference between the information and the second distance information.

<3-4-3-4. Fourth Correction Method>
FIG. 19 is a diagram illustrating the fourth correction method. Here, the correction value of the distance information is changed in accordance with the inclination (the magnitude and sign of the inclination angle) of the change in the distance information of the pixel column in the correction target area CA. For example, in the case of a change in which the distance information increases toward the boundary reference position QM as shown in FIG. 19A, correction is performed so that the inclination angle of the change is reduced (FIG. 19B). On the other hand, when the distance information changes toward the boundary reference position QM as shown in FIG. 19C, correction is performed so that the sign of the slope is reversed (FIG. 19D). In either case, the change in the distance information is suppressed as compared with the change in the distance information before correction.

  As described above, the distance information correction unit 50 performs correction by changing the change rate (including the sign of change) of the distance information of the correction target pixels belonging to the same side pixel column.

<3-4-3-5. Fifth correction method>
FIG. 20 is a diagram illustrating the fifth correction method. As shown in FIG. 20B, in the fifth correction method, the distance information value of each pixel in the correction target area CA between the reference points Q1 and QM is corrected in a stepwise manner. To do.

  Specifically, a plurality of adjacent pixels corresponding to the change rate of the distance information is set as a small group, and the distance information value is converted step by step with the small group as a unit. In a region where the change rate is relatively small, a small group is formed by a relatively large number of pixels (shown by three pixels in the illustrated example), and relatively small pixels (2 in the illustrated example) A small group is composed of pixels).

  In the fifth correction method, the change of the distance information is suppressed inside each of the small groups composed of a plurality of adjacent pixels, and the correction processing is performed as a set of “local change suppression”.

  As mentioned above, the distance information correction | amendment part 50 performs the correction | amendment which suppresses the change of the distance information about the direction approaching a virtual boundary line in steps.

<3-4-3-6. Correction method considering other viewpoint information>
The distance information correction unit 50 can also perform correction based on line-of-sight information when generating another viewpoint image described later. For the extracted virtual boundary line, two reference positions Q′1 and Q′2 are set in the correction target area CA designated in advance, and another viewpoint image generation unit (second viewpoint image generation unit) 80 described later is set. In accordance with the line-of-sight information set in (1), the correction is made by changing whether the inclination angle is corrected to be a steep inclination or to be a gentle inclination.

  FIG. 21 is a diagram for explaining a correction method in consideration of other viewpoint information. FIG. 21A shows distance information of pixels belonging to the same pixel column before correction, and FIG. 21B shows distance information after correction of FIG. 21A.

  When the viewpoint is moved to the right side (when the subject is moved to the left side) as shown in FIG. 21C, the distance information is obtained at the end A1 where the short-distance region is in contact with the right side among the long-distance regions in the subject. The distance information is corrected in a section between predetermined reference positions so that the change in the distance becomes steep, and the distance information is corrected so that the change in the distance information becomes gentle at the opposite end portion A2 of the long-distance region. When the direction of movement of the viewpoint is reversed, the correction is opposite to this.

  That is, when the viewpoint is moved to the right side, the left end portion of the short-distance area enters the end A1 of the long-distance area. Therefore, as shown in FIG. 21B, the reference positions Q′1, Q′2 In the section between, the distance information value of the end portion A1 of the long-distance region is corrected to a steep change so as to approach the distance information value of the short-distance region quickly. On the other hand, at the end A2 of the long-distance area when the viewpoint is moved to the right side, the right end portion of the short-distance area moves to the left and escapes, so that the shift is made to the distance information value of the short-distance area. Make changes gradual. The latter is based on the same idea as the principle of suppressing change in the first to fifth correction methods described above.

  In order to realize the operation as described above, information on the movement direction of the viewpoint is received from the other viewpoint image generation unit 80, and one of the left and right ends of the region surrounded by the contour line according to the movement direction of the viewpoint Correction is performed by the correction distance generation unit 43 so that the change in the distance information is steep and the other is gradual.

  The reference positions Q′1 and Q′2 are basically based on the same idea as the reference positions Q1 and QM of the first to fifth correction methods described above, and correspond to the viewpoint movement amount. Make a correction.

  From the viewpoint of “suppressing the change in distance information”, when the correction method considering other viewpoint information is adopted, the distance information depends on the direction in which the viewpoint changes from the origin position to another position. The part which suppresses the change of is selected, and the change of the distance information is suppressed for the selected part.

<3-5. Parallax image generator (first parallax image generator) 70>
The parallax image generation unit (first parallax image generation unit) 70 generates a first parallax image viewed from the origin position using the second distance image D2. That is, the generation of the first parallax image is performed using the distance information corrected by the distance information correction unit 50, and further correction processing is performed by the line-of-sight information set by the other viewpoint image generation unit 80 described later. For example, when the difference in the distance information of adjacent areas is smaller than a predetermined threshold with respect to the assumed line-of-sight direction, the average value of the distance information of each adjacent area is given to both of the adjacent areas. A first parallax image is generated after integrating these regions.

<3-6. Other viewpoint image generation unit (second viewpoint image generation unit) 80>
In the other viewpoint image generation unit (second viewpoint image generation unit) 80, based on the first parallax image generated by the parallax image generation unit (first parallax image generation unit) 70, the subject from another position different from the origin position is displayed. A second viewpoint image is generated as a distance image obtained by observing. That is, as shown in FIG. 3, another viewpoint image incorporating the external information IF is generated. Specifically, the following methods can be used to create the left and right images corresponding to the second viewpoint image.

<3-6-1. First method for creating left and right images>
The other viewpoint image generation unit 80 generates both left and right images (left eye image and right eye image) based on the first parallax image generated by the parallax image generation unit 70 and the reference image SG. Specifically, the left image and the right image are created by shifting by half the amount of parallax.

<3-6-2. Second method for creating left and right images>
The other viewpoint image generation unit 80 generates either the left or right image based on the first parallax image generated by the parallax image generation unit 70 and the reference image SG. That is, if the reference image SG is the left image, the right image is created by shifting the amount of parallax so as to create the right-side parallax information, and if the reference image SG is the right image, the left-side parallax information is created. Thus, the left image is created by shifting the amount of parallax.

  As described above, in both the first and second methods, the other viewpoint image generation unit 80 generates the left and right images using the reference image SG and the first parallax image, and generates a stereoscopic image. The obtained stereo image (left and right images) is given to a display 32 such as a liquid crystal screen, and is displayed three-dimensionally. Note that an image may be created by adding display information to the image and changing the amount of parallax.

  In addition, in the generation of the second viewpoint image in the other viewpoint image generation unit (second viewpoint image generation unit) 80, the viewpoint changes from the origin position to another position, so that the image portion on either side of the subject can be viewed. May newly appear in or disappear from the visual field. In such a case, the distance information correction unit 50 changes the degree of correction for the above-described change in distance information.

<4. Basic operation of image processing system>
FIG. 22 is a flowchart for explaining a basic operation realized in the image processing system 1 according to this embodiment. Since the individual functions of each unit have already been described (see FIGS. 3 and 4), only the overall flow will be described here.

  As shown in FIG. 22, first, in step S1, the imaging unit 10 captures the standard image SG and the reference image RG, and acquires both images. Here, using two cameras, the standard image SG is an image obtained by the standard camera MC, and the reference image RG is an image obtained by the reference camera SC, but the image of the image is obtained by two or more cameras. Photography and acquisition may be performed.

  In step S <b> 2, the boundary line extraction unit 30 extracts a region boundary line of the image with respect to the two-dimensional image of the reference image SG acquired by the imaging unit 10.

  In step S3, in parallel with step S2, the distance measuring unit 20 calculates the distance to each part of the subject by performing corresponding point search processing on the range in the standard image SG from the standard image SG and the reference image RG. Based on this, the first distance image acquisition unit 40 acquires the first distance image D1. Note that the step of extracting the region boundary line in step S2 and the step of measuring the distance in step S3 may be sequentially performed. Regarding the order, either boundary line extraction (step S2) or distance measurement (step S3) may be performed first.

  In step S4, a virtual boundary line is set at a position corresponding to the boundary line on the reference image SG extracted in step S2 on the first distance image D1, and the correction range setting unit 41 determines the position of the virtual boundary line. Is used as a reference to set a correction range in the first distance image D1.

  In step S5, the correction determination unit 42 determines whether or not the measurement distance information is corrected among the pixels within the correction range set in step S4. That is, correction determination is performed with only pixels having distance information satisfying a predetermined determination condition as correction target pixels.

  In step S6, correction is performed on the correction target pixel in the first distance image D1 determined in step S5. That is, the correction distance generation unit 43 generates a correction distance for each correction target pixel in the first distance image D1, and the second distance image generation unit 60 generates a second distance image D2.

  In step S7, the parallax image generation unit 70 generates a first parallax image viewed from the origin position using the second distance image D2 based on the distance information corrected in step S6.

  In step S8, the second viewpoint image (left and right image) as a distance image in which the other viewpoint image generation unit 80 observes the subject from a different position from the origin position based on the first parallax image generated in step S7. Is generated. The obtained left and right images are given to a display 32 such as a liquid crystal screen and displayed three-dimensionally.

<5. Modification>
* Distance measurement to each part of the subject, which is the basis for generating the first distance image, may be acquired not only by the corresponding point search method described above but also by the following method.

(1) TOF (Time-of-Flight distance image sensor)
In the TOF method (for example, a technique known in Japanese Patent Laid-Open No. 2001-304811 or the like), it is difficult to perform distance measurement on all pixels in the image area. Therefore, it is necessary to interpolate measurement points for regions other than the measured pixels. As an interpolation method, there is a method of simply performing interpolation based on peripheral information. However, not only interpolation but also measurement errors are likely to occur around the edge. For this reason, it is particularly meaningful to perform the correction of the above-described embodiment of the present invention for the first distance image obtained by this method.

(2) Distance estimation method This is a method of generating a stereoscopic image by estimation processing in order to convert a two-dimensional image into a three-dimensional image. As a method using distance estimation, for example, the following technique can be employed.
A technique for classifying shooting scenes by three scene models, estimating a stereoscopic shape by using the closest model, and generating a stereoscopic image by shifting parallax forward and backward based on luminance information of the image, etc. It is disclosed in Japanese Patent Application Laid-Open No. 2006-185033 and Japanese Patent Application Laid-Open No. 2006-186510.
A technique for estimating parallax based on luminance information, which is disclosed in Japanese Patent Application Laid-Open No. 2006-319469.
A technique for generating a parallax by estimating a distance based on time-series information, which is disclosed in Japanese Patent Laid-Open No. 2000-261828.

  In the above method, since distance is not actually measured and the distance is calculated by estimation, there is a target region that is not good at estimation. Therefore, a more correct distance image can be obtained by performing the correction processing described above.

  When the distance measurement as described above is performed, a distance image is thereby obtained. Therefore, it is not necessary to use a stereo camera to obtain distance information, and the reference image SG in the above-described embodiment is used as a two-dimensional image. Only need to get.

  * In this embodiment, the correction process is performed after the correction determination is performed. However, the present invention is not limited to this, and the correction determination and the correction process may be performed by the following method.

(Method 1)
As shown in FIG. 16, a difference value between the set distance information value S (1) of the initial reference position Q1 and the distance information value S (2) corresponding to the inner pixel is calculated. When the absolute value of the difference is equal to or greater than a predetermined threshold value, correction processing is performed.

  This process is sequentially repeated in the direction toward the position QM between adjacent pixels on the region boundary line. If a distance information value S (N) indicating that the absolute value | S (N) −S (N−1) | of the difference is equal to or greater than a predetermined threshold value appears, the distance information value S ( N−1) is used as a distance information value S (N) (this new value of S (N) is referred to as S ′ (N)). In calculating the difference between the distance information value S (N) and the distance information value S (N + 1) corresponding to the inner pixel, the absolute value of the difference is calculated using the newly replaced distance information value S ′ (N). The value | S (N + 1) −S ′ (N) | is calculated.

  As described above, the distance information correction unit 50 combines the correction determination and the correction process, and calculates the distance information value of the correction target pixel belonging to the same side pixel column as the distance information of the inner pixel of the correction target pixel. Correction that makes the value constant may be performed.

(Method 2)
As shown in FIG. 17, an inclination angle θ (1) of a straight line connecting the distance information value S (1) of the set initial reference position Q1 and the distance information value S (2) corresponding to the inner pixel is calculated. . When the inclination angle is equal to or greater than a predetermined threshold value, correction processing is performed. This process is sequentially repeated in the direction toward the boundary reference position QM between adjacent pixels on the region boundary line. If a position QN where the inclination angle θ (N) shows a value equal to or greater than a predetermined threshold appears, the inclination angle θ (N−1) corresponding to the outer pixel is set as the inclination angle θ (N) (this new angle). The tilt angle θ (N) is referred to as θ ′ (N)). The inclination angle θ ′ (N) may be corrected by a fixed value determined in advance. Then, when calculating the tilt angle θ (N + 1) from the distance information value S (N + 1) and the distance information value S (N + 1), the newly replaced distance information value S ′ (N) is used to calculate the tilt angle. θ (N + 1) is calculated.

  As described above, the distance information correction unit 50 combines the correction determination and the correction processing, and calculates the change rate of the distance information of the correction target pixels belonging to the same side pixel column, and the distance information in the inner pixels of the correction target pixels. Correction may be performed using the rate of change.

  * In the present embodiment, the correction process is performed after selecting one of the first to fifth correction methods in advance before the operation is started. However, the present invention is not limited to this. For example, when the method used to correct the first distance image D1 is switched for each image portion depending on the situation near the region boundary of the first distance image D1, a function unit that selects a correction method to be used may be provided. .

DESCRIPTION OF SYMBOLS 10 Imaging part 20 Distance measurement part 30 Boundary line extraction part 40 1st distance image acquisition part 41 Correction range setting part 42 Correction determination part 43 Correction distance generation part 50 Distance information correction part 60 2nd distance image generation part 70 Parallax image generation part 80 Other viewpoint image generator MC Main camera SC Sub camera SG Standard image RG Reference image D1 First distance image D2 Second distance image

Claims (16)

An image processing apparatus for stereoscopic image processing,
First distance image acquisition means for acquiring a first distance image expressing distance information from a predetermined origin position to each part of the subject;
Boundary line extracting means for extracting boundary lines between a plurality of areas of the subject in the two-dimensional image of the subject;
In the first distance image, a virtual boundary line is set at a position corresponding to the boundary line in the two-dimensional image, and regions of the first distance image adjacent to each other across the virtual boundary line are set. Correction means for generating a second distance image by performing correction on the first distance image to suppress a change in distance information in a direction approaching the virtual boundary line while setting at least one of them as a correction range;
With
The stereoscopic image processing is performed based on the second distance image ;
The correction means includes
Correction range setting means for setting the correction range with reference to the position of the virtual boundary line;
Correction determination means for performing correction determination with only pixels having distance information satisfying a predetermined correction determination condition as correction target pixels among the pixels within the correction range;
The correction determination condition is:
The absolute value of the difference between the distance information of the determination target pixel and the distance information of the pixel adjacent to the determination target pixel in the direction from the determination target pixel toward the virtual boundary line is greater than or equal to a predetermined value. the image processing apparatus according to claim condition der Rukoto. An image processing apparatus for stereoscopic image processing,
First distance image acquisition means for acquiring a first distance image expressing distance information from a predetermined origin position to each part of the subject;
Boundary line extracting means for extracting boundary lines between a plurality of areas of the subject in the two-dimensional image of the subject;
In the first distance image, a virtual boundary line is set at a position corresponding to the boundary line in the two-dimensional image, and regions of the first distance image adjacent to each other across the virtual boundary line are set. Correction means for generating a second distance image by performing correction on the first distance image to suppress a change in distance information in a direction approaching the virtual boundary line while setting at least one of them as a correction range;
With
The stereoscopic image processing is performed based on the second distance image;
The correction means includes
Correction range setting means for setting the correction range with reference to the position of the virtual boundary line;
Correction determination means for performing correction determination with only pixels having distance information satisfying a predetermined correction determination condition as correction target pixels among the pixels within the correction range;
The correction determination condition is:
From the determination object pixel in the direction towards the imaginary boundary line, the image processing apparatus absolute value of the change rate of the distance information of the determination target pixel, wherein the condition der Rukoto that is equal to or greater than a predetermined value. The image processing apparatus according to claim 1 or 2,
The correction range setting means includes
The image processing apparatus according to claim 1, wherein a pixel range in which a planar distance from the virtual boundary line is a predetermined value or less is set as the correction range in the first distance image. The image processing apparatus according to claim 1 or 2,
The correction range setting means includes
The image processing apparatus according to claim 1, wherein a pixel range in which the amount of difference from the distance information of the virtual boundary line is a predetermined value or less is set as the correction range in the first distance image. An image processing apparatus according to any one of claims 1 to 4,
The correction range setting means includes
An image processing apparatus, wherein regions on both sides of the virtual boundary line are set as the correction range. The image processing apparatus according to claim 1 or 2,
The correction range setting means includes
An image processing apparatus, wherein the entire first distance image is set as the correction range. An image processing apparatus according to any one of claims 1 to 6,
The correction range setting means includes
An image processing apparatus that performs exception processing for excluding a region where the virtual boundary line does not form a closed curve from the correction range. The image processing apparatus according to claim 1 or 2 ,
The correction determination means includes
An image processing apparatus , wherein the correction determination is performed in units of pixels . The image processing apparatus according to claim 1 or 2 ,
The correction determination means includes
The image processing apparatus characterized in that the correction determination is performed in units of pixel columns that connect two points and intersect the virtual boundary line . An image processing apparatus for stereoscopic image processing,
First distance image acquisition means for acquiring a first distance image expressing distance information from a predetermined origin position to each part of the subject;
Boundary line extracting means for extracting boundary lines between a plurality of areas of the subject in the two-dimensional image of the subject;
In the first distance image, a virtual boundary line is set at a position corresponding to the boundary line in the two-dimensional image, and regions of the first distance image adjacent to each other across the virtual boundary line are set. Correction means for generating a second distance image by performing correction on the first distance image to suppress a change in distance information in a direction approaching the virtual boundary line while setting at least one of them as a correction range;
With
The stereoscopic image processing is performed based on the second distance image;
As the definition of terms,
“Intersecting pixel column” is defined by a pixel column of a line segment passing through the correction target pixel and intersecting the virtual boundary line;
An “outside pixel” is defined by a pixel adjacent to the correction target pixel on the side closer to the virtual boundary line than the correction target pixel in the intersection pixel row;
“Inner pixel” is defined by a pixel adjacent to the correction target pixel on the side farther from the virtual boundary line than the correction target pixel in the intersection pixel row;
“Same side pixel column” is defined by a portion of the intersecting pixel column that is on the same side as the correction target pixel with respect to the virtual boundary line;
When the “opposite pixel column” is defined by a portion on the opposite side of the correction target pixel across the virtual boundary line in the intersecting pixel column,
The correction means corrects the distance information of the correction target pixel using at least the distance information of the inner pixel of the same side pixel column out of the same side pixel column and the opposite side pixel column. . The image processing apparatus according to claim 10 ,
The correction means includes
An image processing apparatus that performs correction to make the value of distance information of a correction target pixel belonging to the same side pixel column constant by using the value of distance information of an inner pixel of the correction target pixel . The image processing apparatus according to claim 10 ,
The correction means includes
An image processing apparatus that performs correction to make the change rate of the distance information of the correction target pixels belonging to the same side pixel column constant by using the change rate of the distance information in the inner pixel of the correction target pixel . The image processing apparatus according to claim 10 ,
The correction means includes
Set a reference position at a position away from the virtual boundary line,
The distance information at the reference position is the first distance information,
For each correction target pixel in the section from the reference position to the virtual boundary line, when distance information of a pixel at a position symmetric with respect to the reference position is the second distance information,
The difference between the corrected distance information for the correction target pixel and the first distance information is proportional to a value obtained by inverting the sign of the difference between the first distance information and the second distance information. An image processing apparatus that performs correction for suppressing a change in distance information of the correction target pixel by performing correction on the correction target pixel . The image processing apparatus according to claim 10 ,
The correction means includes
An image processing apparatus that performs correction that suppresses a change in distance information in a direction approaching the virtual boundary line in a stepwise manner . An image processing apparatus according to any one of claims 1 to 14 ,
First parallax image generation means for generating a first parallax image viewed from the origin position using the second distance image;
Another viewpoint image generation means for generating a second viewpoint image as a distance image obtained by observing the subject from a different position different from the origin position based on the first parallax image generated by the parallax image generation means;
An image processing system comprising: The image processing system according to claim 15 , wherein
The correction means includes
In the generation of the second viewpoint image in the other viewpoint image generation means, selecting a portion for suppressing the change in the distance information according to the direction in which the viewpoint changes from the origin position to the different position. A featured image processing system.

Images