JP2009238076A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To acquire highly precise distance data from a color image in an image processor. <P>SOLUTION: In this image processor for generating stereo image data by calculating a distance from a pair of image data obtained by photographing an object from two points of view, each of which is configured of grey level images with a plurality of colors, to an object, a pair of image data are input, and distance calculation is carried out about a plurality of pixels with a plurality of colors of the input pair of image data, and the grey level image with one color which is the most suitable for distance calculation is selected from among the grey level images with a plurality of colors configuring the image data in at least one image data of the input pair of image data, and distance data calculated in the grey level image with the selected color are recorded. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that generates stereoscopic image data from a pair of image data obtained by imaging a subject from two viewpoints, and more particularly to an image processing method when the pair of image data is a color image. .

  In recent years, there has been proposed an image processing apparatus that generates stereoscopic image data from a pair of image data obtained by imaging a subject from two viewpoints. As a method of generating stereoscopic image data, A method is known that uses a stereo matching method that estimates a three-dimensional shape of a subject by identifying points that are commonly captured, so-called corresponding points, and calculating the distance to the corresponding point according to the principle of triangulation. .

In such an image processing apparatus using a stereo matching method, a color image composed of a plurality of gray images is used as a pair of image data, and the above stereo matching is performed on each of the same color gray images, respectively. The parallax with the color of the image is calculated, the fusion process is performed by the product-sum operation based on the color of the object for each image area, and the stereoscopic image data with high reliability is calculated by calculating the unified parallax. There is a thing (patent document 1).
JP 2003-150936 A

  Usually, when there are few characteristic patterns in a grayscale image, there is a possibility that corresponding points may be erroneously detected, and an incorrect distance may be calculated. In this case, in the image processing apparatus described in Patent Document 1, erroneously calculated data is also used for the fusion process, and thus the accuracy may be reduced.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of acquiring highly accurate distance data from a color image.

The image processing apparatus according to the present invention calculates the distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each composed of a plurality of gray images, and obtains stereoscopic image data. In the image processing device to be generated,
Image input means for inputting the pair of image data;
Distance calculating means for calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
Selecting means for selecting, in at least one of the input image data of the pair of image data, a gray image of one color most suitable for the distance calculation from the gray images of the plurality of colors constituting the image data; ,
And a recording unit that records data of the distance calculated by the distance calculating unit in the grayscale image of the color selected by the selecting unit.

  In the image processing apparatus of the present invention, the selection unit calculates a value of a luminance difference between a maximum luminance value and a minimum luminance value in each of the grayscale images of the plurality of colors, and the calculated luminance difference value is the largest. A large gray image can be selected.

  In the image processing apparatus of the present invention, the selection unit calculates the number of pixels each having a predetermined luminance value at a predetermined interval in the grayscale images of the plurality of colors, and the calculated number of pixels is the largest. Many gray images can be selected.

  In the image processing apparatus of the present invention, the selection unit extracts edges in the grayscale images of the plurality of colors, calculates the number of pixels from which the edges are extracted, and the calculated number of pixels is the largest. Many gray images can be selected.

  In the image processing apparatus of the present invention, the selection unit may calculate the number of pixels in a pixel row at every predetermined interval.

  In the image processing apparatus of the present invention, the selection unit extracts edges in the grayscale images of the plurality of colors, calculates the number of times that the edge intersects the pixel column at predetermined intervals, and the calculated number of times The grayscale image with the largest number can be selected.

  In the image processing apparatus of the present invention, the image input means may be composed of two cameras.

  In the image processing apparatus of the present invention, the image input means may be a compound eye camera including two optical systems.

  In this case, it is preferable that the compound-eye camera is provided with an imaging system having a multi-plate configuration. Here, “multi-plate configuration” has a plurality of optical systems, a plurality of imaging elements corresponding to the respective optical axes are arranged, signal processing is performed using those output signals, and finally one imaging This means a configuration for obtaining an output signal.

According to the image processing method of the present invention, stereoscopic image data is obtained by calculating the distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each composed of a plurality of gray images. In the image processing method to be generated,
Input the pair of image data,
Calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
In at least one image data of the pair of input image data, a gray image of one color most suitable for the distance calculation is selected from the gray images of the plurality of colors constituting the image data,
The distance data calculated in the grayscale image of the selected color is recorded.

The image processing program according to the present invention calculates a distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each of which is composed of a plurality of gray-scale images. In an image processing program for executing generation of image data,
Input the pair of image data,
Calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
In at least one image data of the pair of input image data, a gray image of one color most suitable for the distance calculation is selected from the gray images of the plurality of colors constituting the image data,
This is for causing the data of the distance calculated in the gray image of the selected color to be recorded.

  According to the image processing apparatus, the image processing method, and the program of the present invention, a pair of image data is input, distances are calculated for a plurality of pixels of a plurality of colors in the pair of input image data, In at least one of the image data, a gray image of one color that is most suitable for calculating the distance is selected from a plurality of gray images constituting the image data, and is calculated in the gray image of the selected color. Since the distance data is recorded, the distance data calculated from the image data of one color most suitable for the distance calculation is recorded, so that the distance data with high accuracy can be acquired.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating a functional configuration of an image processing apparatus 1 according to the first embodiment. As shown in FIG. 1, the image processing apparatus 1 according to the present embodiment includes an image input unit (image input unit) 10, an operation unit 11, a display control unit 12, a monitor 13, a reading unit 14, and an image selection unit (selection unit) 15. , A distance calculation unit (distance calculation unit) 16, a data compression unit 17, a media control unit 18, a recording medium (recording unit) 19, an internal memory 20, and the like. Send and receive data.

  The image input unit 10 inputs an image, and is obtained by imaging a subject from two different viewpoints, each of which is a reference image A of a color image composed of grayscale images of R, G, and B colors. A pair of image data consisting of the reference image B is input. In the pair of image data, either image data may be used as the reference image A. In the present embodiment, two pieces of image data are input, but a plurality of pieces of image data such as three or four pieces may be input as long as the data is so-called parallax images captured from different viewpoints. In this case, of the plurality of image data, one image data is set as the standard image A, and the other image data is set as the reference image B.

  The operation unit 11 includes an operation mode switch, a menu / OK button, an up / down lever, a left / right button, a back button, a display switching button, a release button, a power switch, and the like. Make various settings.

  The display control unit 12 displays the image input by the image input unit 10 on the monitor 13, and also generates a subject generated based on distance data to each part calculated by the distance calculation unit 16 described later, that is, depth data. 3D images can be displayed on the monitor 13. Further, the display control unit 12 causes the monitor 13 to display a stereoscopic image based on the stereoscopic image data recorded on a recording medium 19 described later and read out by the media control unit 18.

  The monitor 13 displays a planar image or a stereoscopic image via the display control unit 12, and also displays various setting menus that are set and operated by the operation unit 11.

  The reading unit 14 reads the grayscale images of the three colors R, G, and B constituting the image data input by the image input unit 10 and reads the pixel values of all the pixels. Information such as luminance values is acquired for all pixels by performing a known conversion process based on the read pixel values. In the present embodiment, the luminance value is calculated from the read pixel value. However, the present invention is not limited to this, and the pixel value can be used as the luminance value as it is.

  In the reference image A of the pair of image data read by the reading unit 14, the image selection unit 15 will be described later from among the three gray-scale images Ar, Ag, and Ab of R, G, and B constituting the reference image A. The gray image of one color most suitable for the calculation of the distance by the distance calculation unit 16 is selected. Note that the method of selecting a grayscale image by the image selection unit 15 will be described in detail later.

  The distance calculation unit 16 includes a grayscale image of the color selected by the image selection unit 15 and a grayscale image of a reference image of a color corresponding to the grayscale image, that is, from a pair of grayscale images of the selected color to each part of the subject. The known stereo matching method can be used, and a point that is commonly captured, such as an edge or a segment in a pair of gray images, so-called corresponding points are specified by correlation calculation, The parallax is obtained from the position information of the corresponding point, and the distance to the corresponding point, that is, the distance to each part of the subject is calculated based on the parallax based on the principle of triangulation.

  At this time, the corresponding points are detected by calculating a pixel block Wa composed of a plurality of pixels from the standard image A, scanning the calculated pixel block Wa in the reference image B, and evaluating the degree of coincidence of shape patterns such as shading. A value (correlation degree) is calculated to detect a block Wb corresponding to the pixel block Wa in the reference image B, so-called area-based matching, or the reference image A and the reference image B first. A feature point such as a feature-based matching is used, in which feature points such as edges and segments are found from the image, and corresponding points are detected from the degree of correlation between the feature points in the base image A and the reference image B. Can do. For calculating the coincidence degree evaluation value (correlation degree), for example, a known technique such as summing absolute values of differences of pixels or summing squares of differences of pixels can be used.

  As described above, the distance calculation unit 16 calculates the distance only for the pair of gray images of one selected color, so that the time required for calculating the distance can be reduced.

  The data compression unit 17 compresses the calculated distance data, and the compressed distance data is recorded on the recording medium 19 via the media control unit 18.

  The media control unit 18 reads the distance data stored in the recording medium 19 or writes the distance data.

  The recording medium (recording means) 19 is a recording medium capable of storing various data such as distance data, and is configured by a magnetic or optical recording medium, a semiconductor memory, or the like.

  The internal memory 20 stores various constants and programs set in the image processing apparatus 1, and stores image data input by the image input unit 10, distance data calculated by the distance calculation unit 16, and the like. It also functions as a buffer memory.

  The image processing apparatus 1 according to the present embodiment is configured as described above. Next, image processing by the image processing apparatus 1 will be described. FIG. 2 is a flowchart of a series of image processing performed by the image processing apparatus 1, and FIG. 3 is a flowchart of image selection processing performed by the image selection unit 15.

  In the image processing apparatus 1 of the present embodiment, as shown in FIG. 2, the image input unit 10 inputs a pair of image data, that is, the standard image A and the reference image B (step S1), and the reading unit 14 configures the standard image A. R gradation image Ar, G gradation image Ag, B gradation image Ab, R gradation image Br, G gradation image Bg, and B gradation image Bb constituting image data B are read respectively, and brightness values, etc. Are read out (step S2).

  Next, the image selection unit 15 is one of the most suitable for the calculation of the distance by the distance calculation unit 16 from among the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A read by the reading unit 14. An image selection process for selecting a color shading image is performed (step S3).

  In the image selection processing 1 by the image selection unit 15, as shown in FIG. 3, the minimum luminance value and the maximum luminance value are calculated for each of the R, G, and B gray images Ar, Ag, and Ab constituting the reference image A (step S11), values of brightness differences Dr, Dg, and Db between the minimum brightness value and the maximum brightness value are calculated (step S12).

  Next, the image selection unit 15 determines the one having the largest calculated luminance difference value (step S13). Then, when the largest value is the luminance difference Dr (step S13; Dr), the image selection unit 15 selects the R grayscale image Ar (step S14), and when the largest value is the luminance difference Dg (step S13). In S13; Dg), the G gray image Ag is selected (step S15). If the largest value is the luminance difference Db (step S13; Db), the B gray image Ab is selected (step S16). In this way, the image selection unit 15 performs the image selection process 1.

  In general, a gray image with a large luminance difference value is more likely to have features necessary for calculating the distance than a gray image with a small luminance difference value. In other words, the corresponding points, such as edges, used for calculating the distance are points where the luminance changes abruptly, that is, points with a large luminance difference. Therefore, a gray image with a large luminance difference value has more corresponding points that can be detected. There is a high probability that it has a distribution. Therefore, by selecting a grayscale image having a large luminance difference value as described above, the detection accuracy of corresponding points, that is, the subject accuracy can be improved, so that highly accurate distance data can be acquired.

  Note that the image selection unit 15 may select an image using another method. Next, the image selection process 2 of the second embodiment by the image selection unit 15 will be described. 4A is a flowchart of second image selection processing 2 by the image selection unit 15, FIG. 4B is a flowchart of counting processing, FIG. 5 is an example of luminance distribution of a grayscale image and a pixel counting method, and FIG. It is another example of the pixel counting method.

As shown in FIG. 4A, the image selection unit 15 predetermines a predetermined luminance value C determined in advance at predetermined intervals such as every 10 for each of the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A. Pixels having (for example, L ₁₀ , L ₂₀ , L ₃₀ ...) Are detected (step S20). For example, as shown in FIG. 5, this pixel detection is performed by determining whether or not the luminance values L of all the pixels have a predetermined luminance value C for each of the gray images Ar, Ag, and Ab, A pixel having a predetermined luminance value C indicated by is detected.

  Next, the image selection unit 15 performs a counting process of counting the number of pixels Fr, Fg, and Fb detected in step S20 for each of the R, G, and B gray images Ar, Ag, and Ab that form the reference image A. This is performed (step S21).

  As shown in FIG. 4B, when the count processing is performed for the grayscale image Ar, first, 0 is substituted for Fr and initialized (step S120), and then 0 is substituted for a and initialized (step S121). Then, it is determined whether or not the grayscale image Ar has a predetermined luminance value C (step S122). If the predetermined luminance value C is present (step S122; YES), it is determined whether or not a = c. (Step S123). If a = c is not satisfied (step S123; NO), Fr is counted up (step S124), c is substituted for a (step S125), and the process proceeds to step S126.

  On the other hand, also when there is no predetermined brightness value C at step S122 (step S122; NO) and when a = c at step S123 (step S123; YES), the process proceeds to step S126, It is determined whether or not the pixel is a pixel at the end in the horizontal direction (step S126).

  If the pixel is not the pixel at the end in the horizontal direction (step S126; NO), the pixel is moved in the horizontal direction (step S127), the process proceeds to step S122, and the subsequent processing is repeated. If there is (step S126; YES), it is determined whether or not the pixel is a pixel at the end in the vertical direction (step S128).

  If the pixel is not the end pixel in the vertical direction (step S128; NO), the pixel is moved in the vertical direction (step S129), the process proceeds to step S121, and the subsequent processing is repeated to determine the end pixel. In the case (step S128; YES), the counting process is terminated. The same counting process is performed for the gray images Ag and Ab.

  When the count process is performed, for example, in the grayscale image in FIG. 5, the pixel count is “2” for the first pixel row, “1” for the second pixel row, the third row, the fourth row. In this pixel row, “1” is obtained, and in the fifth pixel row, “1” is obtained, and these are added up to obtain “6”. Thus, the number of pixels Fr, Fg, and Fb is counted as described above for each of the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A.

  In the present embodiment, the number of pixels is counted for each pixel column as described above. However, if the above-described counting is performed for all the pixels constituting the grayscale images Ar, Ag, and Ab, It can be changed as appropriate.

  Then, the image selection unit 15 determines the number of pixels with the largest counted number of pixels (step S22). When the largest number of pixels is the number of pixels Fr (step S22; Fr), the image selection unit 15 selects the R gray image Ar (step S23), and when the largest number of pixels is the number of pixels Fg. In step S22; Fg, the G gray image Ag is selected (step S24). In the case where the largest number of pixels is the number of pixels Fb (step S22; Fb), the B gray image Ab is selected (step S22; Fg). S25). In this way, the image selection unit 15 performs the image selection process 2.

  In general, a gray image with frequent undulations in luminance values is more likely to have features necessary for calculating the distance than a gray image with few undulations in luminance values. In other words, the corresponding points used for calculating the distance, such as edges and areas with large gradients of brightness, are points that have undulations in the luminance values, so the corresponding points that can be detected by grayscale images with more frequent undulations in the luminance values There is a high possibility of having a large luminance distribution. Therefore, by selecting a gray image with frequent undulations of luminance values as described above, the detection accuracy of corresponding points, that is, the subject accuracy can be improved, so that it is possible to acquire distance data with high accuracy.

  In the present embodiment, detection of pixels having a predetermined luminance value C for all pixels is detected in each of the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A as described above. The number of detected pixels Fr, Fg, and Fb was counted. As shown in FIG. 6, for example, every other pixel column, such as every other pixel column, the pixels were detected and the detected pixels The number Fr, Fg, and Fb may be counted. In this case, as shown in FIG. 6, the count of the number of pixels is “2” in the first pixel column, “1” in the third pixel column, and “1” in the fifth pixel column. The total number of pixels is “4”.

  In this way, when the correlation calculation between the standard image A and the reference image B is performed for each pixel column by counting the number of pixels Fr, Fg, and Fb in the pixel column at every predetermined interval, pixels that can become corresponding points. Can be selected, and the speed of the image selection process can be increased.

  Next, the image selection process 3 of the third embodiment by the image selection unit 15 will be described. FIG. 7 is a flowchart of the third image selection processing 3 by the image selection unit 15, and FIG. 8 is an example of a color image.

  As shown in FIG. 7, the image selection unit 15 extracts edges (contours) for the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A (step S30). In addition, the well-known technique generally used can be used for extraction of an edge.

  Next, the image selection unit 15 counts the number of pixels from which edges are extracted in step S30 for each of the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A, that is, the number of pixels Er corresponding to the edges, Eg and Eb are counted (step S31).

  Next, the image selection unit 15 determines the number of pixels having the largest counted number of pixels (step S32). When the largest number of pixels is the pixel number Er (step S32; Er), the image selection unit 15 selects the R gray image Ar (step S33), and when the largest number of pixels is the pixel number Eg. A G gray image Ag is selected (Step S32; Eg) (Step S34), and if the largest number of pixels is the number of pixels Eb (Step S32; Eb), a B gray image Ab is selected (Step S32). S35). In this way, the image selection unit 15 performs the image selection process 3.

  For example, as shown in FIG. 8, in a color image P in which a large object is picked up in the order of a red object Pr, a green object Pg, and a blue object Pb, the density of R, G, and B is varied. When the number of pixels Er, Eg, and Eb corresponding to the edge is counted for each of the images Pr, Pg, and Pb, the number of pixels Er in the R grayscale image Pr is the largest. In other words, the larger the number of pixels, the more edges are extracted, so that more pixels can be correlated in the base image A and the reference image B, that is, the corresponding points are easily detected. Therefore, as described above, by selecting the grayscale image having the largest number of pixels corresponding to the edge, the detection accuracy of corresponding points, that is, the subject accuracy can be improved, and highly accurate distance data can be acquired. It becomes.

  In the present embodiment, as described above, in the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A, the counts of the number of pixels Er, Eg, and Eb corresponding to the edges are calculated for all the pixels. Although it was performed, as shown in FIG. 6 in substantially the same manner as in the above-described embodiment, for example, the number of pixels Er, Eg, and Eb corresponding to the edges may be counted in a pixel column at every predetermined pixel interval. . As described above, when the correlation calculation between the standard image A and the reference image B is performed for each pixel column by counting the number of pixels Er, Eg, and Eb in the pixel column at every predetermined interval, A gray image reflecting the frequency can be selected, and the speed of the image selection process can be increased.

  Next, the image selection process 4 of the fourth embodiment by the image selection unit 15 will be described. FIG. 9 is a flowchart of the fourth image selection process 4 by the image selection unit 15, and FIG. 10 is an example of counting the number of times of intersection of edges in a grayscale image from which edges are extracted.

  As shown in FIG. 9, the image selection unit 15 extracts edges (contours) for the R, G, and B grayscale images Ar, Ag, and Ab constituting the reference image A (step S40). In addition, the well-known technique generally used can be used for extraction of an edge.

  Next, the image selection unit 15 counts the number of edge intersections Nr, Ng, and Nb for each of the R, G, and B gray images Ar, Ag, and Ab that form the reference image A (step S41). The count of the number of edge crossings Nr, Ng, and Nb is based on whether or not the line and the edge intersect for each line at a constant interval in the same direction as the correlation calculation direction in the base image A and the reference image B. Is counted for each pixel, and the number of intersections is counted as the number of intersections Nr, Ng, and Nb.

  For example, when the edge is extracted as shown in FIG. 10, it is determined whether or not the edge intersects from the left to the right for every predetermined pixel column in FIG. At this time, the determination of the intersection of the edges is performed, for example, by binarizing the grayscale image by a known method so that the extracted edges become black, and the number of intersections when the edges are crossed when changing from white to black and from black to white Is counted as 1. Then, in FIG. 10, the number of intersections 4 is counted in the uppermost pixel row, and then the number of intersections 7, 10, and 11 is counted as the lower pixel row is reached. The number of times is 11. In this way, the number of crossings Nr, Ng, Nb is counted.

  As shown in FIG. 9, when the number of intersections Nr, Ng, and Nb is counted (step S41), the image selection unit 15 determines the number of intersections with the largest number of intersections Nr, Ng, and Nb counted (step S41). S42). Then, the image selection unit 15 selects the R gray image Ar (step S43) when the largest number of intersections is the number of intersections Nr (step S42; Nr), and the largest number of intersections is the number of intersections Ng. In step S42; Ng, the gray image Ag of G is selected (step S44). In the case where the largest number of intersections is the number Nb of intersections (step S42; Nb), the gray image Ab of B is selected (step S42). S45). In this way, the image selection unit 15 performs the image selection process 4.

  In general, when the number of extracted edges is large, there is a high possibility that there are many pixels that can be corresponding points between the base image A and the reference image B when performing stereo matching processing. Easy correlation calculation. Therefore, if a grayscale image with a large number of intersections Nr, Ng, and Nb is selected as described above, a large number of intersections means that the number of edges that can be corresponding points is large. Accuracy, that is, subject accuracy can be improved, high-precision distance data can be acquired, and a grayscale image reflecting the frequency of edges that can be corresponding points can be selected.

  Next, as shown in FIG. 2, when the image selection unit 15 selects one color gradation image as described above (step S <b> 3), the distance calculation unit 16 selects the reference image of the color selected by the image selection unit 15. The distances from the grayscale image of A and the grayscale image of the reference image B of the corresponding color are calculated as described above (step S4).

  When the distance calculation unit 16 calculates the distance and obtains the distance data (step S4), the distance data calculated by the data compression unit 17 is then compressed, and the media control unit 18 converts the compressed distance data into the recording medium 19. (Step S5), and the image processing by the image processing apparatus 1 is terminated.

  As described above, according to the image processing apparatus 1 of the present embodiment, in the reference image A of the pair of input image data, the RGB color grayscale images Ar, Ag, and Ab constituting the image data are the most distant. Since the gray image of one color suitable for the calculation is selected and the distance is calculated from the gray image of the selected color and the gray image of the reference image B of the color corresponding to the gray image, the distance is most suitable for calculating the distance. By calculating the distance only with a gray image of one color, it is possible to calculate distance data with high accuracy and reduce the time required for calculating the distance. At this time, if it is only to reduce the time required to calculate the distance, it is not always necessary to select a grayscale image having the optimum color for calculating the distance.

  In the present embodiment, the image selection unit 15 has the luminance difference values Dr, Dg, Db or the number of pixels having a predetermined luminance value Fr, Fg, Fb, and the pixel numbers Er, Eg corresponding to edges only for the reference image A. , Eb, and the number of edge intersections Nr, Ng, and Nb are calculated, but the present invention is not limited to this, and the calculation may be performed for the reference image B, or the reference image A and the reference image B You may perform the said calculation about both.

  In the present embodiment, an image composed of three color images of R, G, and B is used as a color image. However, the present invention is not limited to this, and a plurality of colors such as four colors are used. The present invention can also be applied to a color image composed of grayscale images.

In addition, the image processing apparatus of the present invention calculates the distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each composed of a plurality of gray images, and obtains stereoscopic image data. In the image processing apparatus for generating
Image input means for inputting the pair of image data;
Selection means for selecting at least one gray image of a plurality of color images constituting the image data from at least one of the pair of input image data; ,
Distance calculation means for calculating the distance from the grayscale image of the color selected by the selection means and the grayscale image of the other image data of the color corresponding to the grayscale image can be provided.

  Next, the image processing apparatus 1-2 according to the second embodiment will be described in detail with reference to the drawings. FIG. 11 is a block diagram showing a functional configuration of the image processing apparatus 1-2 according to the second embodiment, FIG. 12 is a flowchart of a series of image processing by the image processing apparatus 1-2, and FIG. 13 is an area dividing unit 22 and image selection. 10 is a flowchart of region division and image selection processing by a unit 15-2. For convenience, FIG. 11 is a block diagram showing the functional configuration of the image processing apparatus 1 of the above embodiment, and the same parts as those in FIG.

  The image processing apparatus 1-2 according to the present embodiment is different from the image processing apparatus 1 according to the above-described embodiment in that an image dividing unit (image dividing unit) 22 is provided as shown in FIG. The image dividing unit 22 divides the grayscale images Ar, Ag, and Ab of the R, G, and B colors constituting the reference image A into a plurality of predetermined areas.

  In addition, the image selection unit 15-2 according to the present embodiment has one color that is most suitable for calculating the distance from the grayscale images Ar, Ag, and Ab of R, G, and B colors for each predetermined region divided by the region dividing unit 22. The gray image is selected.

  The image processing apparatus 1-2 of the present embodiment is configured as described above. Next, image processing by the image processing apparatus 1-2 will be described. 12 is a flowchart of a series of image processing by the image processing device 1-2, FIG. 13 is a flowchart of image division and image selection processing by the region dividing unit 22 and the image selecting unit 15-2, and FIG. 14 is a flowchart of the region dividing unit 22 and the image. It is an example of the image division by the selection part 15-2, and an image selection method.

  In the image processing apparatus 1-2 of this embodiment, as shown in FIG. 12, the image input unit 10 inputs a pair of image data, that is, the standard image A and the reference image B (step S51), and the reading unit 14 receives the standard image A. R gray image Ar, G gray image Ag, B gray image Ab and R gray image Br, G gray image Bg, B gray image Bb constituting image data B are read respectively. Pixel values such as values are read (step S52).

  Next, the region dividing unit 22 performs a predetermined region dividing process for each region including pixels having an edge, and the image selecting unit 15-2 selects a grayscale image in which the edge is extracted in each predetermined region. Image selection processing is performed (step S53).

  In the area division and image selection processing 1, as shown in FIG. 13A, first, edges are extracted from the grayscale images Ar, Ag, Ab of the colors R, G, B constituting the reference image A (step S60). Initialization is performed by substituting 0 for f and m (step S61). Then, the image dividing unit 22 determines whether or not there is an edge in the horizontal direction for each pixel row in the order of grayscale images Ar, Ag, and Ab, for example. That is, as shown in FIG. 13A, it is first determined whether or not there is an edge in the R gray image Ar (step S62). If there is an edge (step S62; YES), the pixel number n is stored ( Step S63).

  Then, it is determined whether or not m is 0 (step S64). If m is 0 (step S64; YES), the image selection unit 15-2 selects the R grayscale image Ar for the region from 1 to n pixel numbers. Is selected (step S65), r is substituted for f (step S66), and n is substituted for m (step S67).

  Next, it is determined whether or not the pixel with the n pixel number is an end pixel in the horizontal direction (step S68). If it is not the end pixel (step S68; NO), the pixel is moved by one pixel in the horizontal direction (step S68; NO) The process proceeds to step S69) and step S62. On the other hand, if the pixel is an end pixel in step S68 (step S68; YES), the process proceeds to step S98 as shown in FIG.

  If m is not 0 in step S64 (step S64; NO), f is discriminated (step S70), and if f is r (step S70; r), the image selection unit 15-2 determines. The R grayscale image Ar is selected for the region up to the pixel number of (nm) / 2 (step S71), and when f is g (step S70; g), the image selection unit 15-2 (n -M) A G gray image Ag is selected for a region up to a pixel number of 2 (step S72), and when f is b (step S70; b), the image selection unit 15-2 selects (nm). ) / 2 is selected for the region up to the pixel number of / 2 (step S73), and the process proceeds to step S66. At this time, the number below the decimal point of (nm) / 2 is raised or lowered to an integer value.

  On the other hand, if there is no edge in the R gray image Ar in step S62 (step S62; NO), it is determined whether or not there is an edge in the G gray image Ag as shown in FIG. 13B (step S74). . If there is an edge (step S74; YES), the pixel number n is stored (step S75).

Then, it is determined whether or not m is 0 (step S76). If m is 0 (step S76; YES), the image selection unit 15-2 applies the G gray image Ag for the region from 1 to n pixel numbers. Is selected (step S77), and g is substituted for f (step S78). Note that FIG.
The processing from step S79 to step S81 in B is the same as the processing from step S67 to step S69 in FIG. 13A, and the processing from step S82 to step S85 in FIG. 13B is the same as step S70 to step S73 in FIG. Description is omitted.

  When the image selection unit 15-2 selects a grayscale image in steps S83 to S85 in FIG. 13B, the process proceeds to step S78.

  On the other hand, if there is no edge in the G gray image Ag in step S74 (step S74; NO), it is determined whether or not there is an edge in the B gray image Ab as shown in FIG. 13C (step S86). . If there is an edge (step S86; YES), the pixel number n is stored (step S87).

Then, it is determined whether or not m is 0 (step S88). If it is 0 (step S88; YES), the image selection unit 15-2 selects the B grayscale image Ab for the region from 1 to n pixel numbers. Is selected (step S89), and b is substituted for f (step S90). Note that FIG.
Steps S91 to S93 in C are the same as steps S67 to S69 in FIG. 13A, and steps S94 to S97 in FIG. 13C are the same as steps S70 to S73 in FIG. 13A, respectively. Therefore, explanation is omitted.

  When the image selection unit 15-2 selects a grayscale image in steps S95 to S97 in FIG. 13C, the process proceeds to step S90.

  Next, in step S68, step S80, and step S92, when it is determined that the pixel is the edge pixel in the horizontal direction (step S68, step S80, step S92; YES), the process proceeds to step S98 as shown in FIG. 13D. And f is discriminated (step S98). If f is r (step S98; r), the image selection unit 15-2 selects the R grayscale image Ar for the area up to the n pixel number. (Step S99) When f is g (Step S98; g), the image selection unit 15-2 selects the G gray image Ag for the region up to the n pixel number (Step S100), and f is b. If there is (step S98; b), the image selection unit 15-2 selects the B grayscale image Ab for the region up to the n pixel number (step S101), and goes to step S102. To migrate the management. At this time, the number below the decimal point of (nm) / 2 is raised or lowered to an integer value.

  In step S102, it is determined whether or not the pixel is an end pixel in the vertical direction (step S102). If the pixel is not an end pixel (step S102; NO), one pixel is moved in the vertical direction (step S103). ), The process proceeds to step S61 in FIG. 13A and the subsequent processes are repeated. On the other hand, if the pixel is an end pixel (step S102; YES), the region division and image selection process 1 is terminated, and the process proceeds to step S54 in FIG.

  That is, as shown in FIG. 14, for example, in the case of a pixel row indicated by an alternate long and short dash line in a grayscale image, when it is determined whether or not there is an edge in each grayscale image Ar, Ag, Ab in the horizontal direction, First, since an edge is detected by the A pixel number of the R grayscale image Ar, the grayscale image Ar is selected in the region of the 1 pixel number to the A pixel number. Next, since the edge is detected by the B pixel number of the G gray image Ag, it is an intermediate point between the B pixel number and the edge detected immediately before the B pixel number, that is, the A pixel number (B−A). The left or right side of the / 2 pixel number is the boundary of the region, and the gray image Ar is selected for the region up to this boundary.

  Next, in the same manner, when the gray image Ag is selected in the region up to the left or right boundary of (DC) / 2 pixel number and becomes the pixel at the end in the horizontal direction, the gray image Ab is displayed in the remaining region. Selected. Then, the above process is performed on all the pixel columns.

  In general, corresponding points are easily detected in the vicinity of the edge, and there is a high possibility that the detection accuracy of the corresponding points, that is, the subject accuracy can be improved. Therefore, by selecting the grayscale image of the color from which the edge is extracted for each region including the pixel having the edge extracted as described above, the color to be selected for each region divided according to the imaged subject is selected. Since the grayscale image can be changed, distance data with high selection accuracy can be acquired.

  Next, the second region division and image selection processing 2 by the region division unit 22 and the image selection unit 15-2 will be described. FIG. 15 is a flowchart of the second area division and image selection process 2. In FIG. 15, for the sake of convenience, the same processes as those in the flowchart of FIG.

Unlike the region division and image selection processing 1 of the above embodiment, the region division unit 22 and the image selection unit 15-2 firstly change the shades of R, G, and B colors that constitute the reference image A as shown in FIG. 15A. In each of the images Ar, Ag, and Ab, pixels having a predetermined luminance value C (for example, L ₁₀ , L ₂₀ , L ₃₀ ...) Determined in advance at predetermined intervals such as every ₁₀ are detected (step S60 ′). Then, initialization is performed by substituting 0 for each of f, m, a, p, and e (step S61 ′), and the image dividing unit 22 performs pixel array processing in the order of grayscale images Ar, Ag, Ab, for example. It is determined whether or not there is a predetermined luminance value C for each pixel in the horizontal direction. That is, as shown in FIG. 15A, first, it is determined whether or not the R grayscale image Ar has a predetermined luminance value C (step S62 ′).

  If there is a predetermined luminance value C (step S62 ′; YES), it is determined whether or not a = c (step S163), and if a = c is not satisfied (step S163; NO). Then, the process proceeds to step S63, and after steps S63 to S67 are performed as in FIG. 13A, c is substituted for a (step S168), and the process proceeds to step S68.

  On the other hand, when a = c in Step S163 (Step S163; YES) and when there is no predetermined luminance value C in Step S62 ′ (Step S62 ′; NO), as shown in FIG. It is determined whether or not the image Ag has a predetermined luminance value C (step S74 ′).

  If there is a predetermined luminance value C (step S74 ′; YES), it is determined whether or not p = c (step S175), and if p = c is not satisfied (step S175; NO). Then, the process proceeds to step S75, and after steps S75 to S79 are performed as in FIG. 13B, c is substituted for p (step S180), and the process proceeds to step S80.

  On the other hand, when p = c in step S175 (step S175; YES) and when there is no predetermined luminance value C in step S74 ′ (step S74 ′; NO), as shown in FIG. It is determined whether or not the image Ab has a predetermined luminance value C (step S86 ′).

  If there is a predetermined luminance value C (step S86 ′; YES), it is determined whether or not e = c (step S187). If e = c is not satisfied (step S187; NO), Then, the process proceeds to step S87, and after steps S87 to S91 are performed as in FIG. 13C, c is substituted for e (step S192), and the process proceeds to step S92.

  On the other hand, when e = c in step S187 (step S187; YES) and when there is no predetermined luminance value C in step S86 ′ (step S86 ′; NO), as shown in FIG. Transfer processing to.

  Then, when it is determined in step S68, step S80, or step S92 that the pixel is an edge pixel in the horizontal direction (step S68, step S80, step S92; YES), whether m is 0 or not is determined. If it is determined (step S198) and m is 0 (step S198; YES), the G gray image Ag is selected for all the pixels in the horizontal direction (step S199), and the process proceeds to step S102. At this time, the R gray image Ar may be selected for all the pixels in the horizontal direction, or the B gray image Ab may be selected.

  If m is not 0 in step S198 (step S198; NO), the process proceeds to step S98, and the processes in steps S98 to S103 are performed as in FIG. 13D. The second area division and image selection process 2 is performed as described above.

  In general, a gray image with frequent undulations in luminance values is more likely to have features necessary for calculating the distance than a gray image with few undulations in luminance values. In other words, the corresponding points used for calculating the distance, such as edges and areas with large gradients of brightness, are points that have undulations in the luminance values, so the corresponding points that can be detected by grayscale images with more frequent undulations in the luminance values There is a high possibility of having a large luminance distribution. Therefore, by selecting a gray image with frequent undulations for each region as described above for each region, the detection accuracy of corresponding points, that is, the subject accuracy can be improved, so that it is possible to acquire highly accurate distance data. It becomes.

  Next, as described above, when the image selection unit 15-2 selects a gray image of one color for each predetermined region divided by the region division unit 22 as shown in FIG. 12 (step S53), the distance calculation unit 16 Calculates the distance to each part of the subject as described above from the grayscale image of the base image A of the color selected by the image selection unit 15 and the grayscale image of the reference image B of the corresponding color for each predetermined area (step S54). ).

  When the distance calculation unit 16 calculates the distance and acquires the distance data (step S54), the distance data calculated by the data compression unit 17 is then compressed, and the media control unit 18 converts the compressed distance data into the recording medium 19. (Step S55), and the image processing by the image processing apparatus 1-2 is terminated.

  As described above, according to the image processing apparatus 1-2 of the present embodiment, in the reference image A of the pair of input image data, the RGB gray images Ar, Ag, and Ab that constitute the image data are converted into a plurality of predetermined values. The image is divided into regions, and for each divided region, a gray image of one color most suitable for calculating the distance is selected from among the RGB colors, and the gray image of the selected color and the color corresponding to this gray image Since the distance is calculated from the grayscale image of the reference image B, the distance data is calculated with high accuracy by calculating the distance only with the grayscale image of one color most suitable for calculating the distance for each predetermined region. And the time required to calculate the distance can be reduced. At this time, if it is only to reduce the time required to calculate the distance, it is not always necessary to select a grayscale image having the optimum color for calculating the distance.

  In this embodiment, the region dividing unit 22 divides the region only for the reference image A, and the image selection unit 15-2 selects a grayscale image for each divided region. However, the present invention is not limited to this. Instead, the division and the selection may be performed on the reference image B, or the division and the selection may be performed on both the standard image A and the reference image B.

  In the present embodiment, an image composed of three color images of R, G, and B is used as a color image. However, the present invention is not limited to this, and a plurality of colors such as four colors are used. The present invention can also be applied to a color image composed of grayscale images.

In addition, the image processing apparatus of the present invention calculates the distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each composed of a plurality of gray images, and obtains stereoscopic image data. In the image processing apparatus for generating
Image input means for inputting the pair of image data;
Area dividing means for dividing the gray images of the plurality of colors constituting the image data into a plurality of predetermined areas in at least one of the pair of input image data;
Selection means for selecting, for each of the predetermined areas divided by the area dividing means, a gray image of at least one color that is a part of the gray images of the plurality of colors;
Distance calculation means for calculating the distance from the grayscale image of the color selected by the selection means and the grayscale image of the other image data of the color corresponding to the grayscale image can be provided.

  Next, an image processing apparatus 1-3 according to the third embodiment will be described in detail with reference to the drawings. FIG. 16 is a block diagram illustrating a functional configuration of the image processing apparatus 1-3 according to the third embodiment, and FIG. 17 is a flowchart of a series of image processing performed by the image processing apparatus 1-3. For convenience, FIG. 16 is a block diagram illustrating the functional configuration of the image processing apparatus 1 according to the first embodiment, and the same parts as those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted. Only different parts will be described in detail.

  As shown in FIG. 16, the image processing apparatus 1-3 of the present embodiment is different from the first embodiment in the distance calculation unit 16-3, and the distance calculation unit 16-3 is read by the reading unit 14. Further, the distance to each part of the subject is calculated for all the pixels of the three-color image of R, G, and B in the reference image A and the reference image B. Although the distance calculation unit 16-3 of the present embodiment calculates the distance for all pixels, the distance calculation unit 16-3 may calculate the distance only for pixel thinning or a partial region. The image processing apparatus 1-3 according to the present embodiment is configured as described above. Next, image processing by the image processing apparatus 1-3 will be described. FIG. 17 is a flowchart of a series of image processing by the image processing apparatus 1-3.

  In the image processing apparatus 1-3 of the present embodiment, as shown in FIG. 17, the image input unit 10 inputs a pair of image data, that is, the standard image A and the reference image B (step S110), and the reading unit 14 receives the standard image A. R gray image Ar, G gray image Ag, B gray image Ab and R gray image Br, G gray image Bg, B gray image Bb constituting image data B are read respectively. A pixel value such as a value is read (step S111).

  Then, the distance calculation unit 16-3 includes a pair of R grayscale images Ar, Br, and G grayscale images Ag, Bg, and B grayscale images Ab and Bb that constitute the input standard image A and reference image B, respectively. To all the pixels, the distance to each part of the subject is calculated (step S112).

  Next, the image selection unit 15 is most suitable for the distance calculation by the distance calculation unit 16-3 from among the R, G, and B grayscale images Ar, Ag, and Ab that constitute the reference image A read by the reading unit 14. Image selection processing for selecting a gray image of one color is performed (step S113). Note that the image selection processing by the image selection unit 15 uses the same image selection processing 1, image selection processing 2, image selection processing 3, and image selection processing 4 as those of the image processing apparatus 1 of the first embodiment described above. Since it can do, description is abbreviate | omitted.

  When the image selection unit 15 selects a gray image of one color from the gray images Ar, Ag, and Ab of the R, G, and B constituting the reference image A (step S113), the reference of the selected color is selected. The data compression unit 17 compresses the distance data calculated from the grayscale images of the image A and the reference image B, and the media control unit 18 records the compressed distance data on the recording medium 19 (step S114). In this way, the image processing by the image processing apparatus 1-3 ends.

  As described above, according to the image processing apparatus 1-3 of the present embodiment, the base image A and the reference image B are input, the distances are calculated for all the pixels of the input base image A and the reference image B, and are input. In the reference image A, a gray image of one color most suitable for calculating the distance is selected from the gray images Ar, Ag, Ab of the colors R, G, B constituting the reference image A, and the selected color is selected. Since the distance data calculated in the grayscale image is recorded, the distance data calculated from the grayscale image of one color most suitable for the distance calculation is recorded, whereby highly accurate distance data can be acquired. .

  In the present embodiment, the image selection unit 15 selects the grayscale image only for the standard image A, but the present invention is not limited to this, and the selection may be performed for the reference image B, or the standard image The selection may be performed for both A and the reference image B.

  In the present embodiment, an image composed of three color images of R, G, and B is used as a color image. However, the present invention is not limited to this, and a plurality of colors such as four colors are used. The present invention can also be applied to a color image composed of grayscale images.

In addition, the image processing apparatus of the present invention calculates the distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each composed of a plurality of gray images, and obtains stereoscopic image data. In the image processing apparatus for generating
Image input means for inputting the pair of image data;
Distance calculating means for calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
Selection means for selecting at least one gray image of a plurality of color images constituting the image data from at least one of the pair of input image data; ,
Recording means for recording the distance data calculated by the distance calculation means in the grayscale image of the color selected by the selection means.

  Next, an image processing apparatus 1-4 according to a fourth embodiment will be described in detail with reference to the drawings. FIG. 18 is a block diagram illustrating a functional configuration of the image processing apparatus 1-4 according to the fourth embodiment, and FIG. 19 is a flowchart of a series of image processing performed by the image processing apparatus 1-4. 18 is a block diagram showing the functional configuration of the image processing apparatus 1-2 according to the second embodiment for the sake of convenience, the same parts as those in FIG. 18 are denoted by the same reference numerals, and the description thereof will be omitted. Only different parts will be described in detail. To do.

  As shown in FIG. 18, the image processing apparatus 1-4 according to the present embodiment is different from the second embodiment in the distance calculation unit 16-4, and the distance calculation unit 16-4 is read by the reading unit 14. Further, the distance to each part of the subject is calculated for all the pixels of the three-color image of R, G, and B in the reference image A and the reference image B. In addition, although the distance calculation part 16-4 of this embodiment shall calculate the said distance about all the pixels, you may calculate the said distance only about pixel thinning | decimation or a partial area | region. The image processing apparatus 1-4 according to the present embodiment is configured as described above. Next, image processing by the image processing apparatus 1-4 will be described. FIG. 17 is a flowchart of a series of image processing by the image processing apparatus 1-3.

  In the image processing apparatus 1-4 according to the present embodiment, as shown in FIG. 19, the image input unit 10 inputs a pair of image data, that is, the standard image A and the reference image B (step S120), and the reading unit 14 receives the standard image A. R gray image Ar, G gray image Ag, B gray image Ab and R gray image Br, G gray image Bg, B gray image Bb constituting image data B are read respectively. Pixel values such as values are read (step S121).

  Then, the distance calculation unit 16-4 includes a pair of R grayscale images Ar, Br, and G grayscale images Ag, Bg, and B grayscale images Ab and Bb that constitute the input standard image A and reference image B, respectively. To all the pixels, the distance to each part of the subject is calculated (step S122).

Next, the area dividing unit 22 divides the grayscale images Ar, Ag, and Ab of R, G, and B constituting the reference image A read by the reading unit 14 into a plurality of predetermined areas, and the image selecting unit 15- 2 performs region division and image selection processing for selecting a grayscale image having a color most suitable for calculating the distance from the grayscale images Ar, Ag, and Ab of R, G, and B colors for each divided predetermined area. This is performed (step S123). Note that the region division and image selection processing by the region division unit 22 and the image selection unit 15-2 are the same as those in the image processing apparatus 1-2 of the second embodiment described above or the region division and image selection. Since the selection process 2 can be used, description thereof is omitted.

  Then, when the image selection unit 15-2 selects one color gray image from the R, G, B gray images Ar, Ag, Ab constituting the reference image A for each predetermined region (step S123). The data compression unit 17 compresses the distance data calculated from the grayscale image of the selected standard image A and reference image B, and the media control unit 18 records the compressed distance data on the recording medium 19 ( Step S124). In this way, the image processing by the image processing apparatus 1-4 ends.

  As described above, according to the image processing apparatus 1-4 of the present embodiment, the base image A and the reference image B are input, the distance is calculated for all the pixels of the input base image A and the reference image B, and the input is performed. In the reference image A, the grayscale images Ar, Ag, Ab of the colors R, G, B constituting the reference image A are divided into a plurality of predetermined areas, and R, G, B of each of the divided predetermined areas are divided. Since the gray image of one color most suitable for the calculation of the distance is selected from the gray images of colors Ar, Ag, and Ab, and the data of the distance calculated in the gray image of the selected color is recorded. The distance data calculated from the image data of one color most suitable for calculating the distance is recorded, so that highly accurate distance data can be acquired.

  In the present embodiment, the image selection unit 15-2 selects the grayscale image only for the standard image A. However, the present invention is not limited to this, and the selection may be performed for the reference image B. The selection may be performed for both the reference image A and the reference image B.

  In the present embodiment, an image composed of three color images of R, G, and B is used as a color image. However, the present invention is not limited to this, and a plurality of colors such as four colors are used. The present invention can also be applied to a color image composed of grayscale images.

In addition, the image processing apparatus of the present invention calculates the distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each composed of a plurality of gray images, and obtains stereoscopic image data. In the image processing apparatus for generating
Image input means for inputting the pair of image data;
Distance calculating means for calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
Area dividing means for dividing the gray images of the plurality of colors constituting the image data into a plurality of predetermined areas in at least one of the pair of input image data;
Selection means for selecting, for each of the predetermined areas divided by the area dividing means, a gray image of at least one color that is a part of the gray images of the plurality of colors;
Recording means for recording the distance data calculated by the distance calculation means in the grayscale image of the color selected by the selection means.

  Next, an example of a block diagram showing the configuration of the image input unit 10 of the embodiment described above is shown in FIGS. The image input unit 10 according to the above-described embodiment inputs at least two image data as parallax images captured from different viewpoints. For example, an image obtained by one camera sequentially capturing images with different viewpoints. Data may be entered.

  Further, as the image input unit 10, as shown in FIG. 20, two optical systems 10a-1, 10a-2 constituted by photographing lenses and the like, and image sensors such as CCDs, that is, R image sensors 10b-1r, 10b-2r. G image sensors 10b-1g, 10b-2g, B image sensors 10b-1b, 10b-2b, R A / D converters 10c-1r, 10c-2r, and G A / D converters Using a compound eye camera 10 having at least two imaging systems, that is, two eyes located at different viewpoints, having units 10c-1g, 10c-2g and B A / D converters 10c-1b, 10c-2b May be. In this case, the imaging system has two optical systems, a plurality of imaging elements corresponding to the respective optical axes are arranged, signal processing is performed using those output signals, and finally one imaging output signal is obtained. A multi-plate configuration is obtained. With such a compound-eye camera, parallax images captured at substantially the same time can be obtained.

  Further, as shown in FIG. 21, the image input unit 10 includes the same optical system 10a-1, the R image sensor 10b-1r, the G image sensor 10b-1g, and the B image sensor 10b-1b. A camera 10A comprising an R A / D converter 10c-1r, a G A / D converter 10c-1g, and a B A / D converter 10c-1b, and an optical system 10a similar to the above. -2, R image sensor 10b-2r, G image sensor 10b-2g, B image sensor 10b-2b, R A / D converter 10c-2r, and G A / D converter 10c-2g and A / D conversion unit 10c-2b for B may be configured by two cameras 10B. In this case, the camera 10A and the camera 10B are arranged at positions where imaging can be performed from different viewpoints. It may be composed of two or more cameras.

  The image input unit 10 in FIGS. 20 and 21 uses the image data of the parallax image obtained by the image processing input unit 10 as step S1 in FIG. 2 or step S51 in FIG. 12, step S110 in FIG. Input in step S120.

  The image processing apparatus of the present invention is not limited to the image processing apparatus of the above-described embodiment, and can be appropriately changed in design without departing from the spirit of the present invention.

The block diagram which shows the function structure of the image processing apparatus of 1st embodiment. Flowchart of a series of image processing by the image processing apparatus of FIG. Flowchart of first image selection processing by image selection unit Flowchart of second image selection processing by image selection unit Count processing flowchart Example of brightness distribution of grayscale image and pixel counting method Another example of brightness distribution of grayscale image and pixel counting method Flowchart of third image selection process by image selection unit Example of color image Flowchart of fourth image selection process by image selection unit An example of counting the number of intersections of edges in a grayscale image from which edges are extracted The block diagram which shows the function structure of the image processing apparatus of 2nd embodiment. Flowchart of a series of image processing by the image processing apparatus of FIG. Flow chart of first image division and image selection processing (part 1) Flow chart of first image division and image selection processing (part 2) Flow chart of first image division and image selection processing (part 3) Flowchart of first image division and image selection processing (part 4) Example of image segmentation and image selection method by region segmentation unit and image selection unit Flowchart of second image division and image selection process (part 1) Flowchart of second image division and image selection process (part 2) Flow chart of second image division and image selection processing (part 3) Flowchart of second image division and image selection processing (part 4) The block diagram which shows the function structure of the image processing apparatus of 3rd embodiment. Flowchart of a series of image processing by the image processing apparatus of FIG. The block diagram which shows the function structure of the image processing apparatus of 4th embodiment. Flowchart of a series of image processing by the image processing apparatus of FIG. Example of block diagram showing configuration of image input unit Another example of block diagram showing configuration of image input unit

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1-2, 1-3, 1-4 Image processing apparatus 10 Image input part (image input means)
10A, 10B Camera 10a-1, 10a-2 Optical system 15, 15-2 Image selection unit (image selection means)
16, 16-3, 16-4 Distance calculation part (distance calculation means)
19 Recording media (recording means)
22 Region dividing unit (region dividing means)

Claims (11)

In an image processing apparatus that generates stereoscopic image data by calculating a distance to a subject from a pair of image data, each of which is obtained by imaging a subject from two viewpoints, each composed of a plurality of gray images,
Image input means for inputting the pair of image data;
Distance calculating means for calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
Selecting means for selecting, in at least one of the input image data of the pair of image data, a gray image of one color most suitable for the distance calculation from the gray images of the plurality of colors constituting the image data; ,
An image processing apparatus comprising: a recording unit that records data of the distance calculated by the distance calculation unit in a grayscale image of a color selected by the selection unit.   The selection means calculates a value of a luminance difference between a maximum luminance value and a minimum luminance value in each of the plurality of color gray images, and selects the gray image having the largest value of the calculated luminance difference. The image processing apparatus according to claim 1, wherein:   The selection means calculates the number of pixels each having a predetermined brightness value in the grayscale images of the plurality of colors and selects the grayscale image having the largest number of the calculated pixels. The image processing apparatus according to claim 1, wherein:   The selection means extracts edges in the grayscale images of the plurality of colors, calculates the number of pixels from which edges are extracted, and selects the grayscale image having the largest number of the calculated pixels. The image processing apparatus according to claim 1, wherein:   5. The image processing apparatus according to claim 3, wherein the selection unit calculates the number of pixels in a pixel row at predetermined intervals.   The selection unit extracts edges in the grayscale images of the plurality of colors, calculates the number of times that the edge intersects the pixel column at predetermined intervals, and selects the grayscale image with the largest number of calculated times. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the image input unit includes two cameras.   The image processing apparatus according to claim 1, wherein the image input unit is a compound-eye camera including two optical systems.   The image processing apparatus according to claim 8, wherein the compound eye camera includes an imaging system having a multi-plate configuration. In an image processing method for generating stereoscopic image data by calculating a distance to a subject from a pair of image data, each of which is obtained by imaging a subject from two viewpoints, each composed of a plurality of gray images,
Input the pair of image data,
Calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
In at least one image data of the pair of input image data, a gray image of one color most suitable for the distance calculation is selected from the gray images of the plurality of colors constituting the image data,
An image processing method comprising: recording the distance data calculated in the grayscale image of the selected color. Executes the computer to generate stereoscopic image data by calculating the distance to the subject from a pair of image data obtained by imaging the subject from two viewpoints, each composed of a gray image of a plurality of colors. In an image processing program for
Input the pair of image data,
Calculating the distance for a plurality of pixels of a plurality of colors of the pair of input image data;
In at least one image data of the pair of input image data, a gray image of one color most suitable for the distance calculation is selected from the gray images of the plurality of colors constituting the image data,
An image processing program for executing recording of the data of the distance calculated in the grayscale image of the selected color.

Images