JP2008078772A - Stereo image processing apparatus and stereo image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stereo image processing apparatus, and so on which can perform interpolations that are close to realistic density values. <P>SOLUTION: The stereo image processing apparatus 1 for preparing a stereo image, whose resolution is increased on the basis of at least two different images whose object is common, is provided with a distance information calculation means 12 for performing stereo image matching processing for searching a correspondence point that corresponds to an optional point on an optional image from among a plurality of images from another image; a pixel interpolation means 13 for performing pixel interpolation processing for interpolating the density values between pixels constituting the optional image by the density values of pixels constituting the other image, on the basis of the correspondence point; and a pixel interpolation means 14 for performing pixel interpolation processing, on the basis of the density values of pixels of the optical image and the density values between pixels and preparing an image obtained, by increasing the resolution of the optional image, to prepare at least two images whose resolution is increased by the processing of respective means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a stereo video processing apparatus and the like that perform processing for increasing the resolution of a stereo video to increase the definition. In particular, the present invention relates to processing for pixel complementation.

Various methods have been conventionally proposed in which a stereo camera having a two-lens imaging system is used to capture images from different left and right viewpoints and create a stereoscopic image (stereo image) from the left and right images obtained by the imaging ( For example, see Patent Document 1).
Japanese Patent Laying-Open No. 2005-210217

  At this time, there is a case where the left and right captured video data is processed to generate video data to create a new video. For example, consider a case where video data is generated by performing high-definition processing for increasing the resolution. In the above-mentioned document, when the imaging magnifications of the left and right imaging units are different, the process of increasing or decreasing the resolution in one video and adjusting the size of the imaging target in the two videos according to the resolution in the other video is described. Has been.

  However, in the above-mentioned document, processing for increasing the resolution is performed using electronic zoom. The electronic zoom is to increase the resolution by estimating the density value between the pixels based on the density values of two adjacent pixels for the pixels constituting the image. Is estimated.

  Therefore, it has been desired to realize a stereo video processing apparatus or the like that can perform complementation closer to reality than the above-described processing.

  The stereo video processing apparatus according to the present invention is a stereo video processing apparatus that creates and processes a stereo video with an increased resolution based on at least two different videos having a common subject. The distance information calculation means for performing a stereo video matching process for finding a corresponding point from another video for the arbitrary point of the image, and the density value between pixels constituting the arbitrary video based on the corresponding point Pixel complementing means for performing pixel complementation processing for complementing with the density values of the pixels constituting the pixel, and pixel interpolation processing based on the pixel density value and the density value between the pixels in an arbitrary video, and the resolution of the arbitrary video A pixel interpolating means for creating and processing an increased video, and creating a video with increased resolution for at least two videos by the processing of each means A.

  A stereo video processing method program according to the present invention is a stereo video processing method program for generating and processing stereo video data with an increased resolution based on at least two different video data common to a subject. A step of performing a stereo video matching process for finding a corresponding point from another video for an arbitrary point on an arbitrary video, and a density value between pixels constituting the arbitrary video based on the corresponding point The step of performing pixel complementation processing that complements the data of the density values of the pixels constituting the other video, and the pixel interpolation processing based on the data of the density values of the pixels and the density values between the pixels in any video, A process of generating and processing video data with an increased resolution of an arbitrary video, and at least two video data To generate image data with increased Zodo.

  As described above, according to the present invention, when the pixel complementing unit complements between pixels of a certain video, it complements based on the other video pixel as the corresponding point in the stereo image matching process. Therefore, it is possible to obtain a device that can perform complementation with pixels based on actual video obtained by imaging with a stereo camera or the like, and that can perform processing based on density values close to reality, not estimation calculation. Therefore, it is possible to obtain an apparatus capable of creating a beautiful video with an increased resolution by the density value close to reality by the complement.

Embodiment 1 FIG.
FIG. 1 is a diagram showing a system configuration centering on a stereo video processing apparatus according to Embodiment 1 of the present invention. In FIG. 1, two image pickup means of the stereo camera 100 each have one or more light receiving elements such as CCDs, and each light receiving element converts the intensity of received light into, for example, an electric signal (this light receiving element). Is basically the number of pixels of the video before processing). In the case of color imaging, red (R), green (G), and blue (B) colors are divided into light in the respective wavelength regions, and the intensity of each light is converted (the light receiving element receives the light). Ultimately, it will be the pixels that make up the video).

  Then, based on the electric signal, it is converted into digital data representing a value of light intensity in each pixel (hereinafter referred to as a density value, which is a luminance value when an image is displayed on the screen). Hereinafter, each digital data is referred to as pixel data, and pixel data for one screen arranged in a matrix is referred to as video data. Here, the video data of two pairs of videos (stereo video) captured including the same object at the same time are stored in the storage means 2 by the two imaging means (in the case of a color video, red (R ), Green (G), and blue (B), there is a pair of video data (in the case of a moving image, one frame of the moving image is the video data). Hereinafter, of the pair of video data stored in the storage unit 2, video data from the right imaging unit viewed from the stereo camera 100 side is referred to as right video data, and video data from the left imaging unit is referred to as left video data. explain.

  The stereo video processing apparatus 1 performs processing (high-definition processing) for generating left and right video data with a predetermined resolution based on video data of left and right videos captured by the stereo camera 100. Here, in the present embodiment, the stereo video processing apparatus 1 is composed of a video input unit 11, a distance calculation unit 12, a pixel complementing unit 13, a pixel interpolation unit 14, and a replacement determination unit 15. .

  The video input unit 11 processes the signal transmitted from the stereo camera 100 and stores the video data in the storage unit 2. The distance information calculation unit 12 is represented by left video (hereinafter simply referred to as left video) and right video data represented by left video data based on the left and right video data stored in the storage unit 2. The parallax of the stereo camera 100 is calculated by performing a stereo image matching process with the right video (hereinafter simply referred to as the right video). Then, distance calculation processing for calculating the distance between the corresponding position in three dimensions (actual space) and the stereo camera 100 is performed for a certain position on the image.

  The pixel complementing means 13 complements the pixels so that the right video and the left video can be referred to with a narrower inter-pixel distance during the pixel interpolation processing performed later by the pixel interpolation means 14 ( Hereinafter, pixel data related to complementation (hereinafter referred to as complementary pixel and complementary pixel data) is generated. Here, in the present embodiment, in the complementing process, processing is performed using pixel data in the video data of the left video for the right video and the video data of the right video for the left video. Also, the pixel interpolation means 14 performs, for example, half-pel calculation, performs pixel interpolation processing, generates left and right video data with increased resolution (doubled for half-pel), and newly A simple video. The replacement determination unit 15 includes a counter (not shown), and determines the number of video replacements when performing distance calculation processing and complement processing, and stores video data corresponding to the determination from the storage unit 2. Let them enter.

  Here, each means of the stereo video processing apparatus 1 can be configured by different hardware. However, the stereo video processing apparatus 1 is operated by arithmetic control means (computer) such as a CPU (Central Processing Unit), for example. On the other hand, the processing procedure can be programmed in advance and configured by software, firmware, or the like. Then, the arithmetic control unit executes the program, performs processing based on the program, and realizes processing performed by each of the above-described units. The data of these programs is stored in the storage means 2, for example. The storage unit 2 stores left and right video data captured by the stereo camera 100 and various data generated by the processing of the stereo video processing device 1. Necessary data is input / output from the storage means 2 when each means of the stereo video processing apparatus 1 performs processing.

  FIG. 2 is a diagram illustrating a flowchart relating to high-definition processing of the stereo video processing apparatus according to the first embodiment. A process performed by each unit of the stereo video processing apparatus 1 will be described with reference to FIG. Here, the processing may be performed first from either of the left and right video data. However, the stereo video processing apparatus 1 of the present embodiment performs processing by inputting the left video data first.

  The video input unit 11 performs processing for storing the left and right video data in the storage unit 2 based on the signal from the stereo camera 100 (S1). Then, when performing high-definition processing of the stored video data, the replacement determination unit 15 determines replacement (S2). If the replacement is 0 times or 1 time, the distance calculation processing is not performed on either the left or right video or any of the images. Therefore, if it is determined 0 times or once, the distance calculation processing of the input video data is performed. To the distance calculation means 12. On the other hand, if it is determined that the replacement has been performed twice, the counter is reset (S6).

FIG. 3 shows a stereo coordinate system. In Figure 3, the screen of the planar left image centered at O _l, a plane around the O _r the screen of the left image, both the optical axis OO _l and OO _r are parallel to each other. In the present embodiment, the distance information calculation means 12 performs processing based on the relationship between the actual three-dimensional position in the coordinate system as shown in FIG. 3 and the left and right images. Here, the coordinates in the Y _l- axis direction of a certain position (point) on the left image (for example, y _l ) and the coordinates in the Y _r- axis direction of the corresponding position on the right image (for example, y _r ) Y _l = y _r . Therefore, the position of the right image position in the left image (the value of the Y coordinate y _l) corresponding to the called line y = y _r on the right image (this line epipolar (epipolar (epipolar) line) ) Somewhere above (this is called epipolar constraint).

  The distance calculation means 12 first performs a stereo image matching process (S3). Stereo image matching processing is, for example, for searching which position (point) of one video corresponds to which position of the other video, and the position of the left and right video obtained by stereo image matching processing. The difference becomes the parallax in the stereo camera 100. Here, in the present embodiment, a characteristic point (hereinafter referred to as a feature point) is extracted from one video and determined, and the corresponding point from the other video (hereinafter referred to as a corresponding point). Search).

For example, a block of M rows and N columns centered on this pixel data is considered for pixel data of a feature point in the x row and y column (here, L _{x, y} ) in one (left here) video. Then, based on the following equation (1), the density value difference value MAE corresponding to each of the M rows and N columns in the other (right here) image is calculated. Then, while shifting the block in the other video, search for R _{x + d + i, y; j} where the minimum MAE is calculated (this is a corresponding point). Here, since the epipolar constraint condition works as described above, the displacement of the block only needs to be in the column (X-axis) direction. Because of this condition, the amount of calculation can be reduced. Then, the value of d (absolute value) when the minimum MAE is calculated from the calculated MAE values is the parallax.

  Here, for example, when calculating the MAE at the edge of the image, pixel data (density value) may not exist depending on the position. For example, the calculation is performed by supplementing the density value. For example, the density value of a non-existing pixel is compensated as 0, the density value of the endmost pixel is compensated as a density value of a nonexistent pixel, and the density value of a nonexistent pixel is returned as line symmetry by a line forming the endmost part There are ways to compensate.

  FIG. 4 is a diagram illustrating an image of interpolation processing between pixels. In the stereo image matching process, since the difference between the density values of one pixel and the other pixel is usually calculated, MAE is calculated in units of pixel intervals. Therefore, the parallax value is also an integer multiple of the pixel interval as a unit. Here, for example, by performing an interpolation process, the corresponding points can be searched for in a finer range. In the present embodiment, it is assumed that the stereo image matching process is performed at intervals of 1/4 pixel after performing the interpolation process. Therefore, finally, the value of the parallax d can be obtained in units of 1/4 pixel.

Based on the value of the parallax d calculated by the stereo image matching process, the distance information calculation unit 12 further performs a distance calculation process. In FIG. 3, the position of a certain point in the three-dimensional space is assumed to be (X, Y, Z). At this time, the positions (x _l , y _l ) and (x _r , y _r ) of a certain point on the left image and the right image are f, and the distance between the image capturing units is B Then, they are represented by the following expressions (2) and (3), respectively. Furthermore, since (2) and (3) and parallax d = x ₁ −x _r , (X, Y, Z) is expressed by equation (4).

Accordingly, at this time, the distance D from the stereo coordinate system origin O to (X, Y, Z) is expressed by the following equation (5).
D = (X ² + Y ² + Z ² ) ^1/2 (5)

  By performing the processing as described above, the distance information calculation unit 12 calculates the distance from the origin of the stereo coordinate system in the three-dimensional space for the feature points and the corresponding points. Here, when the video is color, it is not necessary to perform the processing on each of the red (R), green (G), and blue (B) video data, and the density value of each pixel is determined by at least one video data. As a representative, stereo video matching processing and distance calculation processing may be performed. Alternatively, the density value for calculating the MAE may be obtained using a value generally expressed as the luminance signal Y. Here, for example, Y = 0.3R + 0.59G + 0.11B (R, G, and B represent the density value of each image pixel).

  If it is determined that the distance calculation process based on the left video data has been completed, the replacement determination means 15 exchanges the video to be processed by inputting the right video data (S5). Then, the process returns to S2 to make a replacement decision, and S3 and S4 are executed for the right video data in the same manner as the left video data described above. As described above, the distance calculation processing is performed for the left and right video data. Here, theoretically, since the positions of the two image pickup means differ only by B in the X-axis direction, the distance can be calculated only by calculation based on one of the video data. In reality, however, there is a calculation error in the stereo image matching processing for each video data. For this reason, distance information is calculated for each of the left and right video data. Here, for example, a portion that can be determined to be the same portion can be subjected to processing such as adjusting the calculated distance information, but is omitted in this embodiment.

  When it is determined that the number of left / right exchanges has become two due to the end of the distance calculation processing based on the left and right video data, the replacement determination unit 15 resets the counter as described above (S6), and determines replacement again. (S7). If the replacement is 0 times or 1 time, it means that either the left or right image or any video is not complemented or interpolated. Therefore, if it is determined 0 times or once, the input video data , The pixel complementing means 13 performs the complementing process. Here, since the video exchange is performed in S5, the complementing process is also performed from the left video data. To do. On the other hand, if it is determined that the replacement has been performed twice, it is assumed that the left and right video data has been generated with higher resolution video data (in this embodiment, the video data has double resolution). Finish high-definition processing.

  FIG. 5 is a diagram illustrating an image of the complementing process performed by the pixel complementing unit 13. Usually, when complementing is performed, the above-described interpolation processing is performed to calculate a density value that complements between pixels (hereinafter, this density value is referred to as complementary pixel data). However, the complementary pixel data obtained in this way is only estimated from two adjacent pixels.

  Here, in the case of a stereo image as in the present embodiment, one or a plurality of portions (corresponding points) corresponding (matching) between, for example, the left image and the right image are searched by the above-described stereo matching process. Has been detected. At this time, as described above, in the stereo image matching process performed by the distance information calculation unit 12, a process of searching for a corresponding portion in units of ¼ pixels is performed. For example, there is one video (for example, a right video). In some cases, the pixels correspond to the pixels of the other video (for example, the left video).

Therefore, in the present embodiment, the pixel complementing unit 13 generates a right (left) video for a portion (point) to be complemented in the left (right) video, for example, in generating complementary pixel data for the left (right) video. If it is determined that there is a pixel (data) as a corresponding point, complementary processing is performed such that the pixel data is used as complementary pixel data (S8). For example, in FIG. 5, the density value of R _iq of the right image, which is a corresponding point, is used for L ′ that is a portion between the pixel L _ij and the pixel L _{ij + 1} in the left video. Since the pixel data of the captured video is not obtained by estimation but actual data obtained by imaging of the stereo camera 100, it can be expected that data with values closer to reality can be obtained. In some cases, a pixel adjacent to a corresponding pixel may be used as complementary pixel data. In this complementing process, a portion to be complemented between one pixel is complemented by a pixel of the other video, and when the pixels correspond to each other, the density value of the other pixel is replaced with one pixel. It is not a thing.

  When the pixel complementing process by the pixel complementing unit 13 ends, the pixel interpolation unit 14 executes the pixel interpolation process to generate video data with increased resolution (S9). In the present embodiment, video data with the resolution doubled by half pel is generated.

For example, in the half pel, a density value of a pixel L ″ to be interpolated between the pixel L _ij and the pixel L _{ij + 1} (hereinafter referred to as an interpolated pixel) is usually calculated based on the following equation (6). This is interpolated pixel data (hereinafter referred to as interpolated pixel data).
L ″ = (L _ij + L _{ij + 1} ) / 2 (6)

  FIG. 6 is a diagram illustrating an example of a half-pel calculation method using complementary pixel data. In the present embodiment, the complementary pixel data is generated by complementing the pixels based on the other image between the pixels by the complementing process of the pixel complementing unit 13. Therefore, half-pel calculation can be performed based on the pixel data of two pixels at a distance shorter than the normal inter-pixel distance obtained by imaging with the stereo camera 100.

In the example of FIG. 6, for example, when the distance between the pixel L _iq and the pixel L _{ij + 1} is a distance corresponding to one pixel, the complementary pixel L ′ is changed from the pixel L _ij to a ¼ pixel (pixel L _{ij +1} to 3/4 pixels). Here, in the half pel, the position where the pixel is to be interpolated is the position between the pixel L _ij (pixel L _{ij + 1} ) and a half pixel between the pixel L _ij and the pixel L _{ij + 1.} Become. Therefore, it can be expected that the distance between the complementary pixel L ′ and the pixel L _{ij + 1} is narrower than the distance between the pixel L _iq and the pixel L _{ij + 1} and a density value closer to reality can be calculated. Since the interpolation pixel L ″ is at the position of 1/3 pixel from the complementary pixel L ′ and 2/3 pixel from the pixel L _{ij + 1} , the interpolation pixel data of the interpolation pixel L ″ is expressed by the following equation (7). Can be generated based on.
L ″ = (2L ′ + L _{ij + 1} ) / 3 (7)

Here, when calculating the interpolated pixel data as in the present embodiment, the interpolated pixel L ″ is the complementary pixel L ′ in the positional relationship of the pixel L _ij , the pixel L _{ij + 1} , and the complementary pixel L ′. After determining whether it is located between the pixel L _ij or between the complementary pixels L ′ and L _{ij + 1} , the pixel data of either the pixel L _ij or the pixel L _{ij + 1} is selected Then, as shown in equation (7), pixel interpolation processing is performed based on the positional relationship to generate interpolated pixel data. The interpolated pixel data is generated by performing the above calculation between the pixels to generate the interpolated pixel data (a new pixel is interpolated between the pixels), and the video data having twice the resolution is generated. Create a new video.

  When it is determined that the distance calculation processing based on the left video data has been completed, the replacement determination unit 15 stores, for example, the left video data in the storage unit 2, inputs the right video data, and exchanges the video to be processed. (S10). When the video data is generated, the process returns to S7 to make a replacement determination, and S8 and S9 are executed for the right video data in the same manner as the left video data described above. As described above, pixel interpolation processing and pixel interpolation processing are performed on the left and right video data, respectively, to generate left and right video data having twice the resolution, thereby creating a new video.

  In the case of a color video, at least a complementing process and a pixel interpolation process are performed on each video data relating to red (R), green (G), and blue (B). In the case of a moving image, high-definition processing is performed on video data of other frames. Although the horizontal direction (X direction) has been described here, the vertical direction may be performed by interpolation processing by linear interpolation and half-pel calculation.

  As described above, according to the stereo image processing apparatus of the first embodiment, when the pixel complementation unit 13 performs interpolation between pixels of one video, in the stereo image matching processing, Since the complementary pixel data is generated based on the pixel data, the complementary pixel data can be generated from actual data obtained by imaging with the stereo camera 100, and is not obtained by estimation. It is possible to obtain an apparatus capable of performing processing for obtaining data based on close density values. In addition, in the pixel interpolation processing performed by the pixel interpolation means 14, half-pel calculation is performed using the complementary pixel, so that a device having double resolution and capable of generating clear video data is obtained. be able to.

Embodiment 2. FIG.
FIG. 7 is a diagram showing a system configuration centering on a stereo video processing apparatus according to Embodiment 2 of the present invention. In FIG. 7, the same reference numerals as those in FIG. 1 perform the same operations as the corresponding reference numerals in the first embodiment, and thus description thereof is omitted. The stereo video processing apparatus 1A according to the second embodiment is a stereo video processing apparatus 1 according to the first embodiment provided with a calculation count judging means 16 for judging the number of calculation times of processing. The calculation number judgment means 16 is a means for making a judgment for performing the high-definition processing by the stereo video processing apparatus described in the first embodiment a predetermined number of times (this is n times). It is assumed that the calculation number judgment means 16 also has a counter (not shown) for counting the number of calculations.

  FIG. 8 is a diagram illustrating a flowchart of the processing operation of the stereo image high-definition processing apparatus 1A according to the second embodiment. The video input unit 11 performs processing for storing the left and right video data in the storage unit 2 based on the signal from the stereo camera 100 (S11). Then, the number-of-calculations determination means 16 determines whether or not the apparatus has performed processing by performing predetermined calculations n times (S12). If the number of times is other than n, the counter is incremented by 1, and the same distance calculation process, complement process, and high-definition process as S2-S10 of the first embodiment are performed (S13-S21).

  In addition, when the replacement determination unit 15 determines that the two complementary processings and the high-definition processing are performed, the replacement number is reset to 0 (S22), and then the calculation number determination unit 16 sets the number of processing times. Make a decision.

As described above, according to the stereo video processing apparatus 1A of the second embodiment, the calculation frequency judging means 16 is provided, and the left and right videos are respectively processed by the stereo video processing apparatus 1A with the same high-definition processing as in the first embodiment. Since n is performed ⁿ times, it is possible to obtain video (data) having a resolution of 2 ⁿ times the original video (data) on the left and right. Therefore, it is convenient when trying to further increase the resolution.

Embodiment 3 FIG.
FIG. 9 is a diagram showing a system configuration centering on a stereo video processing apparatus 1B according to Embodiment 3 of the present invention. In FIG. 9, the same reference numerals as those in FIG. 1 perform the same operations as the corresponding reference numerals in the first embodiment, so that the description thereof is omitted. The stereo video processing device 1B according to the third embodiment further includes resolution determination means 17 in the stereo video processing device 1 according to the first embodiment. The resolution determination unit 17 performs an iterative determination process for determining whether or not it is necessary to perform a complementing process using pixel data in the other video by the pixel complementing unit 13. Further, when it is determined that it is not necessary to perform complementation, it is further determined whether or not a predetermined resolution is obtained, and if it is determined that the predetermined resolution is not obtained, the pixel interpolation unit 14 uses half-pels. A pixel interpolation process is performed.

  As described in the above embodiment, the pixel complementing means 13 performs the complementing process using the other video, thereby improving the quality of the video when the resolution is increased. However, linear interpolation is used when it is not necessary to perform pixel interpolation on the left and right images, for example, when the density value no longer changes even when complement processing using the other image is performed. Therefore, the effect of the complementing process by the pixel complementing means 13 is small. Therefore, in the present embodiment, the resolution determination unit 17 calculates a value obtained by adding the absolute values of the difference in density value of each pixel before and after the complement processing by the pixel complementing unit 13 for one screen, and compares it with a preset threshold value. To do. When it is determined that the value is smaller than the threshold value, the complement process is not performed. Further, if it is determined that the resolution is lower than the predetermined resolution, the pixel interpolation means 14 is caused to perform pixel interpolation processing until the predetermined resolution or higher is reached.

  FIG. 10 is a diagram illustrating a flowchart of the processing operation of the stereo image high-definition processing apparatus 1B according to the third embodiment. The video input unit 11 performs processing for storing the left and right video data in the storage unit 2 based on the signal from the stereo camera 100 (S31). When the video data is input from the video input unit 1, the resolution determination unit 17 calculates the absolute value of the difference between the density values of each pixel before and after the complementary processing for each corresponding pixel, and further calculates the sum thereof. Here, in the first stage, since the pixel complementing unit 13 does not perform the complementing process, for example, it is assumed that iterative is necessary, or all the values (density values) of the pixel data before the complementing process are set. The absolute value of the difference is increased as 0. Then, an iterative determination process is performed in comparison with a predetermined threshold value (S32).

  If it is determined that the calculated value is greater than or equal to the threshold value, as described in the first and second embodiments, stereo video matching processing, pixel complementation processing, and pixel interpolation processing are performed (S33 to S42).

  On the other hand, if it is determined that the calculated value is smaller than the threshold value, it is not necessary to perform at least pixel complement processing. Then, it is further determined whether or not a predetermined resolution is obtained (S43). If the predetermined resolution is obtained, it is determined that the video data has been generated, and the high-definition processing in the apparatus is terminated. If it is determined that the predetermined resolution is not obtained, pixel interpolation processing using half pels by the pixel interpolation means 14 is performed (S44), and this is performed on the left and right video data (S45).

  As described above, according to the stereo video processing apparatus 1B of the third embodiment, if the resolution determination unit 17 determines whether or not to perform the complementing process by the pixel complementing unit 13, and determines that the effect of the complementing process cannot be expected. Since the complementary processing is not performed, efficient processing can be performed. At that time, it is determined whether left and right video data having a predetermined resolution has been generated. If it is determined that the left and right video data having a predetermined resolution has not been generated, Since the insertion means 14 performs the pixel interpolation process, it is possible to generate the left and right video data having a predetermined resolution without ending the process halfway.

Embodiment 4 FIG.
In the above-described embodiment, the stereo image matching process performed by the distance calculation unit has been described. In the stereo image matching process, the parallax d is derived from the value calculated based on the equation (1) based on the minimum value. The minimum value is ideally 0, but usually includes a stereo image matching error (hereinafter referred to as a matching error), and thus often does not become 0. When the value of the matching error is large, the reliability of the corresponding part in the left and right images is lowered. Further, it does not meet the purpose of generating more beautiful video data by complementing with pixels actually captured to increase the reliability of the complementary pixels.

  Here, as described above, the complementing process in the pixel complementing unit 13 is to generate the complementary pixel data in one video based on the pixel data in the other video. If the value of the matching error is large, it is difficult to say that the other pixel data that is determined to correspond to the density value of the portion related to complementation, and there is a high possibility that they are greatly different. Therefore, in the present embodiment, the pixel complementing unit 13 makes a determination related to the matching error before performing the complementing processing described in the first, second, and third embodiments. Here, although the processing of the pixel complementing means 13 in the present embodiment is strictly different from the processing of the above-described embodiment, the apparatus configuration is the same as in FIGS. 1, 7, and 9, so This will be described with reference to these drawings.

  FIG. 11 is a flowchart illustrating the operation of the stereo video processing apparatus according to the fourth embodiment. FIG. 11 particularly describes the processing performed by the pixel complementing means 13. The pixel complementing unit 13A performs the following process as a previous stage of the pixel complementing process.

The pixel complementing unit 13 calculates a matching error value (hereinafter referred to as a matching error value) based on the MAE calculated by the distance calculating unit 12 based on the equation (1) by stereo matching processing (S51). The method for calculating the matching error based on the expression (1) is not particularly limited. For example, the sum of the minimum values of MAE calculated with respect to each pixel may be used as the value of the matching error, or calculated with the parallax d _x . The MAE value may be used as a matching error value.

  Then, it is determined whether the matching error value is greater than or equal to a predetermined threshold value (S52). If the matching error value is smaller than the threshold value, pixel interpolation processing using pixel data of the other video is performed (S53). The pixel complementing process in S53 is the same as the pixel complementing process described in the first, second, and third embodiments. On the other hand, if the value of the matching error is large, there is a high possibility that the value will be significantly different from the actual value even if it is replaced with the corresponding part of the other video. Does not perform the complementing process based on the pixel data of the other video by the pixel complementing means 13. Here, the pixel complementing unit 13 does not perform the complementing process. However, complementation by linear interpolation may be performed to generate complemented pixel data.

  As described above, in the fourth embodiment, the pixel complementing means 13 calculates the matching error value based on the MAE value calculated based on the expression (1) and compares it with the threshold value, thereby performing the complementing process. Therefore, there is a high possibility that the density value of the complementary pixel is significantly different from the density value of the complementary pixel, and the density value of the other pixel having low reliability is not used as the complementary pixel data. Therefore, it is possible to obtain beautiful video data even when the resolution is increased.

Embodiment 5. FIG.
In the above-described embodiment, the pixel complementing unit 13 performs the pixel complementing process based on the left and right video data. For example, in addition to these video data, for example, the stereo camera 100 includes an imaging unit for pixel complementation. If the image is provided and the image complementation processing is performed by adding the video data obtained by the imaging, the complement pixel data can be increased.

Embodiment 6 FIG.
In the above-described embodiment, the left and right video data are always processed first from the left video data, but the order is not particularly limited. Further, in the above-described embodiment, the determination of the necessity determination such as the number of exchanges of the video data by the replacement determination unit 1E, the number of calculations by the calculation number determination unit 16, and the complementary processing by the resolution determination unit 17 is performed, for example, Although it performed before the process, it does not specifically limit. The procedure can be set so that the determination process can be performed at any time, for example, when the process can be correctly performed, for example, when the process is completed.

  In the above-described embodiment, the left and right video data are alternately processed based on the determination by the replacement determination unit 1E. For example, the distance calculation unit 12, the pixel complementing unit 13, and the pixel interpolation unit 14 are each two. The stereo video processing apparatus may be configured so as to be provided one by one and to perform parallel processing.

Embodiment 7 FIG.
In the above-described embodiment, the pixel interpolation unit 14 performs the pixel interpolation process by performing the calculation by half pel, but the present invention is not limited to this. For example, pixel interpolation processing such as quarter-pel may be performed. Further, the interpolation method is not limited to linear interpolation, and an operation using another interpolation method can be performed.

1 is a diagram illustrating a system centered on a stereo video processing apparatus 1 according to Embodiment 1. FIG. 3 is a diagram illustrating a flowchart according to processing in the first embodiment. FIG. It is a figure showing a stereo coordinate system. It is a figure showing the image of the interpolation process between pixels. It is a figure showing the image of the complementation process which the pixel complementing means 13 performs. It is a figure for showing the example of the calculation method of the half pel using complementary pixel data. 6 is a diagram illustrating a system centering on a stereo video processing apparatus according to Embodiment 2. FIG. FIG. 10 is a diagram illustrating a flowchart according to processing in the second embodiment. FIG. 10 is a diagram illustrating a system centered on a stereo video processing apparatus according to a third embodiment. FIG. 10 is a diagram illustrating a flowchart relating to processing in the third embodiment. FIG. 10 is a diagram illustrating a flowchart according to processing in the fourth embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo image processing apparatus 11 Image | video input means 12 Distance calculation means 13 Pixel complementing means 14 Pixel interpolation means 15 Replacement determination means 16 Calculation frequency determination means 17 Resolution determination means 2 Storage means 100 Stereo camera

Claims (6)

In a stereo video processing apparatus that creates and processes a stereo video with an increased resolution based on at least two different videos having a common subject,
Distance information calculating means for performing stereo image matching processing for finding a corresponding point from another image for an arbitrary point on an arbitrary image in the plurality of images,
Pixel complementing means for performing a pixel complementing process for complementing the density value between pixels constituting the arbitrary video with the density value of the pixels constituting the other video based on the corresponding points;
Pixel interpolating means for performing pixel interpolation processing based on the density value of the pixel and the density value between the pixels in the arbitrary video, and creating a video with increased resolution of the arbitrary video, and A stereo video processing apparatus that creates a video with an increased resolution for at least two videos by processing of each means.   2. The stereo video processing apparatus according to claim 1, further comprising a processing frequency determination unit that determines whether or not the processing of each unit has been performed a predetermined number of times on the arbitrary video.   The sum of the absolute value of the change in the density value of each pixel before and after the image complementing process by the image complementing unit is calculated and compared with a predetermined threshold value, and each unit including the image complementing process is calculated. The stereo video processing apparatus according to claim 1, further comprising resolution determination means for determining whether to repeat the process again.   The resolution determining means further determines whether or not a video having a predetermined resolution or higher is created when it is determined that the repetition is not necessary. The stereo image processing apparatus according to claim 3, wherein the pixel interpolation means performs pixel interpolation processing.   The pixel complementing means calculates a matching error value representing the reliability of the corresponding point found by the stereo video matching process, and compares the threshold value with a predetermined threshold value to determine whether to perform the pixel complementing process. The stereo image processing apparatus according to claim 1, wherein pixel interpolation processing is performed. In a stereo video processing method program for generating and processing stereo video data with increased resolution based on at least two different video data with a common subject,
A step of performing a stereo image matching process for finding a corresponding point from another image for an arbitrary point on an arbitrary image in the plurality of images;
Performing a pixel complementing process for complementing density values between pixels constituting the arbitrary video with data of density values of pixels constituting the other video based on the corresponding points;
Performing a pixel interpolation process on the basis of the density value data of the pixel and the density value between the pixels in the arbitrary video, and generating a video data with an increased resolution of the arbitrary video. And generating a video data with an increased resolution for at least two video data.

Images