JP2009110356A - Image processing device, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce stored data size without deteriorating accuracy in distance measuring in image processing devices. <P>SOLUTION: In the image processing device, which generates 3D image data by calculating distances to each portion of an object using two image data obtained by photographing the object from two view points, all pixels of the input image data are read after inputting them. Then pixels constituting image data A1, A2, which are one of two image data, are thinned using thinning rate defined according to the data size for storing the 3D image data, and distances associated with the pixels in the thinned image data are calculated. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that generates stereoscopic image data from two images obtained by imaging a subject from two viewpoints, and more particularly to an image processing apparatus, an image processing method, and a program that reduce the size of data to be stored. It is.

  In recent years, there has been proposed an image processing apparatus that generates stereoscopic image data from left and right image data obtained by imaging a subject from two viewpoints. In such an image processing apparatus, the generated image data is transmitted or recorded, etc. In view of convenience, it is desired to reduce the size of stored data.

Therefore, a compound-eye imaging device that compresses and records / reproduces left and right image data obtained by imaging according to a predetermined compression rate (Patent Document 1), and interpolates or thins out left and right images having different numbers of pixels. A three-dimensional structure estimation device (Patent Document 2) that performs distance measurement calculation according to the number, a stereoscopic image creation device (Patent Document 3) that samples by sampling the left and right images and calculates color information according to the position information of the sampling Has been proposed.
JP-A-8-32871 Japanese Patent Laid-Open No. 10-134187 JP 2003-319417 A

  However, in the image processing apparatus that generates the stereoscopic image data by reducing the size by thinning out the left and right image data as in the past or combining them, and generating the stereoscopic image data, For example, as shown in FIG. 18, the right eye C of a person can be detected as a corresponding point in the left image Pl and the right image Pr before thinning, but after the thinning, the right eye C can be detected in the left image Pl, There is a possibility that the correct distance cannot be calculated because the pixel of the right eye C is thinned out and cannot be detected, and the corresponding point is not found.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus, an image processing method, and a program for reducing the size of stored data without reducing ranging accuracy.

A first image processing apparatus according to the present invention is an image processing apparatus that generates stereoscopic image data by calculating a distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints.
Image input means for inputting the two image data;
Reading means for reading out all pixels of the input image data;
Thinning means for thinning out pixels constituting one of the two image data at a thinning rate according to the size of the saved data size set for saving the stereoscopic image data;
It is characterized by comprising distance calculation means for calculating the distance for the pixels after thinning by the thinning means.

The second image processing apparatus of the present invention is an image processing apparatus that generates stereoscopic image data by calculating a distance from each of two image data obtained by imaging a subject from two viewpoints to each part of the subject.
Image input means for inputting the two image data;
Reading means for reading out all pixels of the input image data;
Distance calculating means for calculating the distance for all the read pixels;
The data size is reduced by synthesizing or thinning out the distance data calculated by the distance calculation means at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data. And a data size reducing means.

  In the first image processing apparatus of the present invention, the thinning-out unit may thin out the pixels in the order of pixels having a larger difference value of data values from surrounding pixels among the pixels.

  Further, in the second image processing apparatus of the present invention, the data size reduction unit may calculate the distances calculated by the distance calculation unit in the order of the pixels having the largest difference value of data values from the surrounding pixels among the pixels. It may be synthesized by leaving data, or thinned.

  Further, in the second image processing apparatus of the present invention, the data size reduction unit outputs the distance data calculated by the distance calculation unit in the order of pixels having a high degree of correlation between corresponding pixels in the two image data. They may be left behind for synthesis or thinning out.

  In the second image processing apparatus of the present invention, the data size reduction unit performs weighting on the distance calculated by the distance calculation unit based on the degree of correlation between corresponding pixels in the two image data, After the weighting, the distance data may be synthesized.

  In the image processing apparatus of the present invention, the image input means may be composed of two cameras.

  In the image processing apparatus of the present invention, the image input unit may be a compound eye camera including two lenses.

A first imaging processing method of the present invention is an image processing method for generating stereoscopic image data by calculating a distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints.
Input the two image data,
Read all the pixels of the input image data,
Thinning out pixels constituting one of the two image data at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data,
The distance is calculated for the pixels after the thinning.

A second imaging processing method of the present invention is an image processing method for generating stereoscopic image data by calculating a distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints.
Input the two image data,
Read all the pixels of the input image data,
Calculating the distance for all the read pixels;
The data size is reduced by synthesizing or thinning out the distance data calculated by the distance calculation means at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data. It is characterized by this.

Further, the first image processing program of the present invention executes to generate a stereoscopic image data by calculating a distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints. In an image processing program for
Input the two image data,
Read all the pixels of the input image data,
Thinning out pixels constituting one of the two image data at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data,
This is for causing the distance to be calculated for the pixels after the thinning.

Further, the second image processing program of the present invention executes to generate a stereoscopic image data by calculating a distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints. In an image processing program for
Input the two image data,
Read all the pixels of the input image data,
Calculating the distance for all the read pixels;
The data size is reduced by combining or thinning out the distance data calculated by the distance calculation means at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data. To make things happen.

  According to the first invention, before reading all the pixels of the two input image data and calculating the distance to each part of the subject, the size of the storage data size set to store the stereoscopic image data Since the pixels constituting this image data are thinned out only for one image data at the thinning rate according to the above, and the distance is calculated only for the pixels after thinning, it is possible to use all pixel values for calculating the distance. In addition, since the distance can be calculated only for necessary pixels, the size of stored data can be reduced without lowering the distance measurement accuracy, and the time required for calculating the distance can be reduced.

  According to the second invention, all the pixels of the two input image data are read out, the distance to each part of the subject is calculated for all the pixels, and after the distance is calculated, the stereoscopic image data is stored. Since the data of the distance is synthesized at a thinning rate according to the set storage data size or thinned to reduce the data size, all pixel values can be used for calculating the distance. Since only necessary distance data can be recorded, the size of stored data can be reduced without lowering the distance measurement accuracy.

  Hereinafter, an image processing apparatus 1 according to a first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a functional configuration of the image processing apparatus 1 of the present embodiment. As shown in FIG. 1, the image processing apparatus 1 according to the present embodiment includes an image input unit (image input unit) 10, an operation unit 11, a display control unit 12, a monitor 13, a reading unit (reading unit) 14, and a storage data size setting. 15, a thinning unit (thinning unit) 16, a distance calculating unit (distance calculating unit) 17, a data compression unit 18, a media control unit 19, a recording medium 20, an internal memory 21, and the like. And various types of signals and data are transmitted and received.

  The image input unit 10 inputs an image, and two image data obtained by imaging the subject from two viewpoints are input. In the present embodiment, two pieces of image data are input. However, a plurality of pieces of image data, such as three or four, may be input as long as the data is so-called parallax images captured from different viewpoints.

  The operation unit 11 includes an operation mode switch, a menu / OK button, an up / down lever, a left / right button, a back button, a display switching button, a release button, a power switch, and the like. Make various settings.

  The display control unit 12 displays the image input by the image input unit 10 on the monitor 13, and in addition to the subject generated based on the distance data to each part calculated by the distance calculation unit 17 described later, that is, the depth data. 3D images can be displayed on the monitor 13. The display control unit 12 causes the monitor 13 to display a stereoscopic image based on the stereoscopic image data recorded on a recording medium 20 described later and read by the media control unit 19.

  The monitor 13 displays a planar image or a stereoscopic image via the display control unit 12, and also displays various setting menus that are set and operated by the operation unit 11.

  The readout unit 14 reads out the pixel values of all the pixels constituting the image data input by the image input unit 10. Information such as luminance values can be obtained for all pixels by performing a known conversion process based on the read pixel values.

  The saved data size setting unit 15 sets a saved data size of distance data calculated from two image data obtained by imaging the subject from two viewpoints. The save data size can be set by the user operating the operation unit 11.

  In the thinning-out unit 16, one of the two pieces of input image data, that is, the pixel constituting the reference image data when calculating the distance to each part of the subject is set by the storage data size unit 16. The data is thinned out at a thinning rate corresponding to the size of the stored data size.

  The distance calculation unit 17 calculates the distance from the two image data to each part of the subject for the pixels after the thinning by the thinning unit 16. The thinning method by the thinning unit 16 and the distance calculation method by the distance calculating unit 17 will be described in detail later.

  The data compression unit 18 compresses the calculated distance data, and the compressed distance data is recorded on the recording medium 20 via the media control unit 19.

  The media control unit 19 reads distance data or the like stored in the recording medium 20 or writes distance data or the like.

  The recording medium 20 is a recording medium capable of storing various data such as distance data, and is configured by a magnetic or optical recording medium, a semiconductor memory, or the like.

  The internal memory 21 stores various constants and programs set in the image processing apparatus 1, and stores image data input by the image input unit 10, distance data calculated by the distance calculation unit 17, and the like. It also functions as a buffer memory.

  The image processing apparatus 1 according to the present embodiment is configured as described above. Next, image processing by the image processing apparatus 1 will be described. FIG. 2 is a flowchart of a series of image processing by the image processing apparatus 1, and FIG. 3 is an example of thinning by the thinning unit 16.

  In the image processing apparatus 1 of the present embodiment, as shown in FIG. 2, the image input unit 10 inputs two image data A and B (step S1), and the reading unit 14 receives all of the image data A and image data B. The pixel value in the pixel is read (step S2).

  Further, the image processing apparatus 1 determines the size of the saved data size set by the saved data size setting unit 16 (step S3). At this time, the size of the stored data size is set in advance by the user operating the operation unit 11.

  When the stored data size is full size (step S3; full size), the distance calculation unit 17 does not perform thinning and performs all the pixel search as shown in the image data A1 or image data A2 shown in the left diagram of FIG. Is used to calculate the distance to each part of the subject (step S4).

  On the other hand, when the stored data size is, for example, half (step S3; half size), the thinning unit 16 has 50% of the thinning rate, and one of the two pieces of image data A and B (main book) In the embodiment, pixels constituting the image data A) are thinned out.

  As a thinning method by the thinning unit 16, when the saved data size set by the saved data size setting unit 15 is half the size, for example, as shown in the image data A1 shown in the upper middle of FIG. The pixels for which the distance calculation is performed alternately and the pixels for which the distance calculation is not performed are determined alternately, and the pixels for which the distance calculation is not performed are thinned out. Further, as shown in the image data A2 in the lower part of FIG. 3, even if a column of pixels for which distance calculation is alternately performed and a column of pixels for which distance calculation is not performed are determined for each column, a column of pixels for which distance calculation is not performed is thinned out. Good.

  In the present embodiment, the storage data size is halved, but may be another size such as a quarter size. In the case of the 1/4 size, for example, as in the image data A1 shown in the upper right diagram of FIG. 3, one pixel in the 2 × 2 pixels is a pixel for which distance calculation is performed, and the other three pixels are pixels for which distance calculation is not performed. As described above, for example, odd-numbered columns may alternately determine pixels for which distance calculation is performed and pixels for which distance calculation is not performed, and even-numbered columns may be columns of pixels for which distance calculation is not performed. Further, as in the image data A2 shown in the lower right diagram of FIG. 3, for example, 1 of the 4 columns of pixels is a column of pixels for which distance calculation is performed, and the other 3 columns are columns of pixels for which distance calculation is not performed. Only the column and multiple columns of 4 may be columns for calculating the distance. In this way, the thinning rate is changed according to the size of the stored data size.

  In the present embodiment, the pixel for calculating the distance is determined like the image data A1 or the image data A2, but the present invention is not limited to this. Here, another thinning method by the thinning unit 16 when the saved data size set by the saved data size setting unit 15 is ¼ will be described. FIG. 4 is an example of the image data A, and FIG. 5 is a flowchart of the thinning process by the thinning unit 16.

  For example, as shown in FIG. 4, the thinning-out unit 16 determines the pixel for calculating the distance from the four pixels A, B, C, and D of a2b2, a2b3, a3b2, a3, and b3, as shown in FIG. , I are sequentially substituted for i (step S10), and the processes of steps S11 to S19 are executed for the pixels A to D, respectively.

  First, a luminance difference between the pixel i and a pixel above the pixel i (for example, a1b1 in the case of the pixel A) is obtained and set as iu (step S11). Next, the luminance difference between the pixel i and the pixel below the pixel i (for example, the pixel C for the pixel A) is obtained as id (step S12), and the pixel i and the pixel to the left of the pixel i (for example, the pixel A, for example) The luminance difference of a2b1) is obtained as il (step S13), and the luminance difference between the pixel i and the pixel to the right of the pixel i (for example, the pixel B in the case of the pixel A) is obtained as ir (step S14).

  The luminance differences iu, id, il calculated in steps S11 to S14. The largest luminance difference among ir is discriminated (step S15), and when the luminance difference iu is the largest luminance difference (step S15; iu), iu is stored as Mi in the buffer memory (internal memory) 21 ( Step S16).

  When the luminance difference id is the largest luminance difference (step S15; id), id is set to Mi. When the luminance difference il is the largest luminance difference (step S15; il), il is set to Mi. When the luminance difference ir is the largest luminance difference (step S15; ir), ir is stored as Mi in the buffer (internal memory) 21 (step S17 to step S19).

  Next, in steps S16 to S19, the pixel A to D that has the largest luminance difference among the MA to MD stored in the buffer (internal memory) 21 is determined (step S20), and the luminance difference MA is the largest. When the luminance difference is large (step S20; MA), the distance calculation of the pixel A (step S21), and when the luminance difference MB is the largest luminance difference (step S20; MB), the distance calculation of the pixel B is calculated. (Step S22) When the luminance difference MC is the largest luminance difference (Step S20; MC), the distance calculation of the pixel C (Step S23), and when the luminance difference MD is the largest luminance difference (Step S22) S20; MD) and distance calculation of the pixel D are performed (step S24).

  The same processing is performed on all the pixels constituting the image data A, the pixels for which the distance calculation is performed and the pixels for which the distance calculation is not performed are determined, and the pixels which form the image data A are thinned out. Is thinned out to finish the thinning process.

  By calculating the distance for pixels with a large luminance difference in this way, it is highly possible that a pixel with a large luminance difference is an edge, that is, a contour portion of the subject, so that the subject accuracy is increased and the accuracy of the distance data is increased. it can.

  When the thinning process is performed by the thinning unit 16 as described above (step S5), as shown in FIG. 2, the distance calculation unit 17 calculates the distance using the pixels after the thinning by the thinning unit 16 (step S6). ). Here, FIG. 6A shows an example of image data A and image data B, and FIG. 6B shows an example of block Wij in FIG. 6A. In the present embodiment, the image data A in FIG. 6A will be described below assuming that pixels for which distance calculation is performed (circles in the figure) and pixels that are not to be performed (crosses in the figure) are alternately determined for each pixel.

  As shown in the left diagram of FIG. 6A, the distance calculation unit 17 determines that the Aa2Ab2 pixel is a pixel (circle) calculated by the thinning unit 16 when the reference pixel in the image data A is Aa2Ab2 pixel. Therefore, distance calculation is performed. For distance calculation, first, a block W22 of 3 × 3 pixels, for example, as shown in FIG. 6B, which is formed by Aa2Ab2 pixels and pixels around the pixels, is constructed. A matching degree evaluation value (correlation degree) of a shape pattern such as shading is calculated while sequentially shifting the block W22 in the horizontal direction, and a block Wij corresponding to the block W22 in the image data B is detected. The coincidence evaluation value can be calculated using a known technique such as summing absolute values of differences between pixels or summing squares of differences between pixels.

  Then, the parallax is calculated from the position information of the corresponding block Wij in the image data A and the image data B, and the distance to the subject is calculated based on the parallax based on the principle of triangulation. Next, as shown in the left diagram of FIG. 6A, the reference pixel is shifted by one pixel in the horizontal direction to form a block W23 based on Aa2Ab3 pixels. However, since the Aa2Ab3 pixels are pixels that are determined not to be distance-calculated by the thinning-out unit 16 (X), the distance is not calculated and the reference pixel is further shifted in the horizontal direction by one pixel in the same manner as described above. Process.

  As described above, the distance calculation is performed only for the pixels for which the distance calculation is determined by the thinning unit 16, and the distance data to the subject is obtained. Therefore, the time required to obtain the distance data can be shortened. Further, since the reading unit 14 reads out the pixel values for all the pixels constituting the image data A, when performing distance calculation, as shown in FIG. 6B, pixels for which distance calculation is not performed (cross marks in the figure), that is, thinning-out. Pixels to be taken can be used. Therefore, for example, even if the pixels to be thinned have a part of a characteristic shape pattern, they can be used for distance calculation, so that the data size can be reduced without lowering the distance measurement accuracy.

  The distance calculation method used is a so-called area-based matching method, but the present invention is not limited to this. First, feature points such as edges and segments are found from image data. The correspondence between the image data A and the image data B is detected from the correlation between the feature points in the image data A and the image data B, and the distance to the subject is calculated based on the correspondence based on the principle of triangulation. A method called feature-based matching may be used. Here, FIG. 7A shows another example of the image data A and image data B, and FIG. 7B shows an example of the block WA in FIG. 7A.

  When using the feature-based matching method, as shown in the left diagram of FIG. 7A, in the block WA composed of pixels including the feature points, as shown in FIG. 7B, for example, the reference pixel is set to the center Aa2Ab2 pixel, Only the block WA whose reference pixel is a pixel (circle) for which the distance calculation determined by the thinning unit 16 is performed is the block WA while scanning the block WB in the image data B as shown in the right diagram of FIG. 7A. The block WB ′ having a high degree of correlation is detected, the distance between the block WA and the block WB is calculated, and the distance is calculated based on the principle of triangulation.

  As a result, since all the pixels constituting the image data are read when the feature points are detected, it is possible to prevent the corresponding points from being lost because the pixels including the feature points are not read and thinned out. Therefore, it is possible to prevent a reduction in detection accuracy of feature points. Further, in the block WA composed of pixels including the feature points, for example, pixels (cross marks) thinned out such as Aa2Ab1 can be used, so that the data size can be reduced without lowering the ranging accuracy as described above. Can do.

  The distance calculation unit 17 calculates the distance to each part of the subject as described above. Then, as shown in FIG. 2, when the distance is obtained by calculating the distance (step S6), the distance data calculated by the data compression unit 18 is compressed, and the compressed distance data is recorded by the media control unit 19 on the recording medium. 20 (step S7), and the image processing by the image processing apparatus 1 is terminated.

  According to the image processing apparatus 1 of the present embodiment, all the pixels of the two input image data A and B are read and set by the storage data size setting unit 15 before calculating the distance to each part of the subject. Since the pixels constituting the image data A are thinned out only for the image data A at the thinning rate according to the size of the saved data size, and the distance is calculated only for the pixels after thinning, all pixel values are used for calculating the distance. Can be used and the distance can be calculated only for the necessary pixels, so that the size of the stored data can be reduced without lowering the distance measurement accuracy and the time required for calculating the distance can be reduced.

  Next, an image processing apparatus 1-2 according to the second embodiment of the present invention will be described in detail with reference to the drawings. FIG. 8 is a block diagram showing a functional configuration of the image processing apparatus 1-2 of the present embodiment. In this embodiment, for the sake of convenience, in FIG. 8, the same parts as those in the block diagram (FIG. 1) of the image processing apparatus 1 of the above embodiment are denoted by the same reference numerals, and description thereof is omitted.

  The image processing apparatus 1-2 according to the present embodiment is different from the above embodiment in the distance calculation unit 17 ′, and the distance calculation unit 17 ′ extends all the pixels read by the reading unit 14 up to each part of the subject. The distance is calculated.

  In this embodiment, the thinning unit 16 is not provided, but a data size reduction unit 23 is provided. The data size reduction unit 23 synthesizes or thins out the data of the distance calculated by the distance calculation unit 17 ′ at a thinning rate according to the size of the storage data size set by the storage data size setting unit 15. Is reduced. The data size reduction method by the data size reduction unit 23 will be described in detail later.

  The image processing apparatus 1-2 of the present embodiment is configured as described above. Next, image processing by the image processing apparatus 1-2 will be described. FIG. 9 is a flowchart of a series of image processing performed by the image processing apparatus 1-2, FIG. 10A is an example of thinning of distance data by the data size reduction unit 23, and FIG. 10B is distance data by the data size reduction unit 23. It is an example of the synthesis | combination of.

  In the image processing apparatus 1-2 of the present embodiment, as shown in FIG. 9, the image input unit 10 inputs two image data A and B (step S30), and the reading unit 14 stores the image data A and the image data B. Pixel values in all the pixels are read out (step S31).

  Next, the distance calculation unit 17 'calculates the distance to each part of the subject for all the pixels constituting the image data A (step S32). The distance data calculated at this time is stored in the buffer 21. As a method for calculating the distance, a method using the same principle of triangulation as in the above embodiment can be used. Then, the image processing apparatus 1-2 determines the size of the saved data size set by the saved data size setting unit 16 (step S33). At this time, the size of the stored data size is set in advance by the user operating the operation unit 11.

  If the stored data size is the full size (step S33; full size), thinning or combining is performed as in the image data A3 or the image data A4 shown in the left diagram of FIG. 10A or 10B. After the data compression unit 18 compresses the distance data calculated for all the pixels constituting the image data A by the distance calculation unit 17 ′ without performing the processing, the media control unit 19 stores the compressed recording data in the recording medium 20. Recording is performed (step S34).

  On the other hand, when the stored data size is, for example, half (step S33; half size), the data size reduction unit 23 synthesizes the distance data calculated by the distance calculation unit 17 ′ at a thinning rate of 50%. Alternatively, the data size is reduced by half (step S35).

  As a thinning method by the data size reduction unit 23, a thinning method substantially similar to the thinning unit 16 in the above embodiment can be used, and recording is performed in pixels constituting the image data A as shown in FIG. The pixels to be performed and the pixels not to be performed are determined, and the distance data calculated for the pixels not to be recorded are thinned out.

  Further, as a combining method by the data size reduction unit 23, when recording data with a thinning rate of 50%, that is, a distance of 1/2 pixel, as shown in the middle diagram of FIG. , The distance data calculated for the a1b1 pixel and the a2b1 pixel are combined, or one of them is selected as one distance data, and the data of the distance for the thinning rate is 75%, that is, 1/4 pixel is recorded. Sometimes, for example, the distance data calculated for the adjacent four pixels, a1b1 pixel, a1b2 pixel, a2b1 pixel, and a2b2 pixel are combined, or one of the four distance data is selected as one distance data. . As this distance data, a maximum value, a minimum value, or a median value may be selected, or an average value may be used.

  FIG. 11 shows a flowchart of the data size reduction process by the data size reduction unit 23. For the sake of convenience, the same processing in FIG. 11 as in FIG.

  For example, as shown in FIG. 4, the data reduction unit 23 is shown in FIG. 11 in order to determine a pixel for recording distance data from the pixels A, B, C, and D of a2b2, a2b3, a3b2, a3, and b3. As described above, A to D are sequentially substituted for i (step S10), and the processes of steps S11 to S19 are executed for the pixels A to D, respectively, and the luminance difference Mi is stored in the buffer 21.

  Next, in step S16 to step S19, the largest luminance difference among the MA to MD stored in the buffer (internal memory) 21 is determined (step S20), and when the luminance difference MA is the largest luminance difference. (Step S20; MA) The distance DA calculated for the pixel A in Step S32 of FIG. 9 is used as distance data to be recorded (Step S41).

  When the luminance difference MB is the largest luminance difference (step S20; MB), the distance DB calculated for the pixel B is recorded as distance data (step S42), and the luminance difference MC is the largest luminance difference. In this case (step S20; MC), the distance DC calculated for the pixel C is recorded as distance data (step S43). When the luminance difference MD is the largest luminance difference (step S20; MD), the pixel The distance DD calculated for D is used as distance data to be recorded (step S44).

  The same processing is performed on all the pixels constituting the image data A to determine the distance data to be recorded and the distance data not to be recorded, and the four distance data calculated for each of the four pixels are converted into one distance data. As a result, the data size is reduced and the process is terminated.

  By using distance data to record the distance calculated for pixels with a large luminance difference in this way, it is highly possible that a pixel with a large luminance difference is an edge, that is, a contour portion of the subject, and therefore subject accuracy increases and distance data Accuracy can be increased.

  Next, another reduction process by the data size reduction unit 23 will be described. FIG. 12 shows a flowchart of another data size reduction process.

  For example, as shown in FIG. 4, the data size reduction unit 23 determines the pixel for recording distance data from among the pixels A, B, C, and D of a2b2, a2b3, a3b2, a3, and b3 in FIG. As shown, A to D are sequentially substituted for i (step S50), and the correlation when the distance is calculated for the pixels A to D in step S32 of FIG. 9 is stored in the buffer 21 as CA, CB, CC, CD, that is, Ci. Store (step S51).

  Next, the largest value among the respective correlation degrees CA to CD in the pixels A to D is determined (step S52). At this time, it is assumed that the larger the correlation value, the higher the degree of coincidence of the subject in the image data A and the image data B.

  If the value of the correlation degree CA is the largest (step S52; CA), the distance data recorded for the pixel A in step S32 of FIG. 9 is recorded as distance data (step S53, the value of the correlation degree CB is the largest). When it is large (step S52; CB), distance data for recording the distance calculated for the pixel B is recorded (step S54), and when the value of the correlation CC is the largest (step S52; CC), the pixel C is recorded. And distance data for recording the distance calculated for the pixel D (step S55), and when the value of the correlation degree CD is the largest (step S52; CD), the distance data for the pixel D is recorded (step S52). S56).

  The same processing is performed on all the pixels constituting the image data A to determine the distance data to be recorded and the distance data not to be recorded, and the four distance data calculated for each of the four pixels are converted into one distance data. As a result, the data size is reduced and the process is terminated.

  Thus, by using distance data to record the distance calculated for a pixel having a high degree of correlation, that is, a pixel having a high degree of correlation, the pixel having a high degree of correlation matches the subject of image data A and image data B. Since the degree is high, the accuracy of the subject is increased and the accuracy of the distance data can be increased.

  Next, still another reduction process by the data size reduction unit 23 will be described. FIG. 13 shows a flowchart of still another data size reduction process. For convenience, the same processing as in FIG. 12 is denoted by the same step number in FIG.

  As shown in FIG. 13, the data size reduction unit 23 stores the correlation degree as Ci in step S51 (step S51), and then obtains the following expression (1) for each of the pixels A to D (step S57).

Di ′ = Di × Ci / (CA + CB + CC + CD) (1)
Here, Ci / (CA + CB + CC + CD) is a weighting coefficient, and Di ′ is a weighted distance.

  Then, the distance data obtained by substituting the value of the distance Di ′ (DA ′, DB ′, DC ′, DD ′) weighted in step S57 into the following equation (2) is used as the distance data for recording (step). S58).

D = DA ′ + DB ′ + DC ′ + DD ′ (2)
The same processing is performed on all the pixels constituting the image data A, and the four distance data calculated for the four pixels are combined into one distance data to reduce the data size and the processing is terminated.

  Thus, by performing weighting according to the degree of correlation, highly accurate distance data can be obtained.

  When the data size reduction unit 23 performs the data size reduction process as described above (step S35), the data compression unit 18 compresses the distance data after the data size reduction by the data size reduction unit 23, as shown in FIG. Then, the media control unit 19 records the compressed distance data on the recording medium 20 (step S7), and ends the image processing by the image processing device 1-2.

  According to the image processing apparatus 1-2 of the present embodiment, all the pixels of the two input image data A and B are read, the distances to the subject parts are calculated for all the pixels, and after calculating the distances, The data of the distance is synthesized with the thinning rate according to the size of the saved data size set by the saved data size setting unit 15 or the thinned rate according to the size of the saved data size, or the data size is reduced by thinning. Therefore, since all the pixel values can be used for calculating the distance and only necessary distance data can be recorded, the size of the stored data can be reduced without reducing the distance measurement accuracy.

  Next, an example of a block diagram showing the configuration of the image input unit 10 of the embodiment described above is shown in FIGS. The image input unit 10 according to the above-described embodiment inputs at least two image data as parallax images captured from different viewpoints. For example, an image obtained by one camera sequentially capturing images with different viewpoints. Data may be entered.

  As the image input unit 10, as shown in FIG. 14 (a), optical systems 10a-1, 10a-2 constituted by photographing lenses and the like, imaging elements 10b-1, 10b-2 such as CCDs, and A / The compound eye camera 10 having the D conversion units 10c-1 and 10c-2 and including at least two eyes located at different viewpoints may be used. In this case, parallax images captured at substantially the same time can be obtained.

  Further, as shown in FIG. 14B, the optical system 10a-1, 10a-2 similar to the above, one image sensor 10b, and one A / D converter 10c are provided behind the optical system. By subjecting the subject images 1 and 2 that have passed through the optical system 10a-1 and the optical system 10a-2, respectively, to the photoelectric surface of the image sensor 10b by a switch (not shown) provided, and the like, Two parallax images may be obtained.

  Further, as shown in FIG. 15, the image input unit 10 includes a camera 10A including an optical system 10a-1, an image sensor 10b-1, and an A / D conversion unit 10c-1 similar to the above, and the same as described above. The optical system 10 a-2, the image sensor 10 b-2, and the A / D conversion unit 10 c-2 may be configured by two cameras 10 </ b> B. In this case, the camera 10A and the camera 10B are arranged at positions where imaging can be performed from different viewpoints. It may be composed of two or more cameras.

  14 and 15 inputs the image data of the parallax image obtained by the image processing input unit 10 in step S1 of FIG. 2 or step 30 of FIG.

  When the image input unit 10 is configured by a compound eye camera or two or more cameras as described above, the image sensor 10b or 10b-1, 10b-2 is not illustrated before image capturing. By driving at, the pixels after thinning or combining may be read after thinning or combining in units of pixels without reading all the pixels. FIGS. 16 and 17 show an example of a flowchart of a series of image processing by the image processing apparatuses 1 and 1-2 including the image input unit 10 of FIG. 14 or FIG. For the sake of convenience, the same processing as in FIG. 2 in FIG. 16 and the same processing as in FIG. 9 and FIG. 16 in FIG.

  In the image processing apparatus 1, as shown in FIG. 16, for example, a two-eye compound camera or two cameras (image input unit 10) captures a subject (step S60), and determines a capturing mode at the time of capturing (step S61). . When the shooting mode at the time of shooting is the distance measurement mode (step S61; distance measurement mode), the reading unit 14 reads out the pixel values of all the pixels of the image data A and the image data B that have been shot (step S61). S62), processing of step S3 to step S7 is performed.

  On the other hand, when the shooting mode is other than the distance measurement mode (step S61; other), the image processing apparatus 1 determines the size of the saved data size set by the saved data size setting unit 16 (step S63). If the stored data size is full size (step S63; full size), all the pixels are read out (step S64). If the saved data size is half size (step S63; half size), the image sensor is driven. By driving the unit, the two pixels are synthesized and read (step S65), and image data A and image data B are generated. At this time, one pixel may be selected from two pixels and read.

  In addition, as a synthesis method, an average value may be used, or a maximum value and a minimum value may be used. In addition, when the set size of the stored data is a size smaller than half, such as a quarter size, a median value may be used.

  Next, the A / D converter 10c or 10c-1, 10c-2 performs A / D conversion on the image data A and the image data B (step S66), and performs various image processing by a digital signal processor (not shown). After performing (step S67), the data compression unit 18 compresses the image data A and image data B after image processing, and the media control unit 19 records the compressed image data A and image data B on the recording medium 20. (Step S68). In this way, the image processing by the image processing apparatus 1 is finished.

  Further, as shown in FIG. 17, when it is determined that the image processing device 1-2 is in the distance measurement mode in step S <b> 61 (step S <b> 61; distance measurement mode), steps S <b> 32 to S <b> 34 similar to FIG. 9 are performed. When it is determined that the mode is other than the distance measurement mode (step S61; other), the processing of step S64 to step S38 similar to FIG. 16 is performed. In this way, the image processing by the image processing apparatus 1-2 is finished.

  According to the image processing apparatus 1 or the image processing apparatus 1-2 including the image input unit 10 having a compound eye camera or two or more cameras as described above, priority is given to speed except when the distance is calculated. Thus, the data size can be reduced.

  The image processing apparatus of the present invention is not limited to the image processing apparatus of the above-described embodiment, and can be appropriately changed in design without departing from the spirit of the present invention.

Block diagram showing functional configuration of image processing apparatus Flowchart of a series of image processing by the image processing apparatus The figure which shows an example of the thinning by a thinning part The figure which shows an example of the image data A Flow chart of thinning process by thinning unit The figure which shows an example of image data A and image data B The figure which shows an example of the block in FIG. 6A The figure which shows another example of image data A and image data B The figure which shows an example of the block in FIG. 7A The block diagram which shows the function structure of the image processing apparatus of 2nd embodiment. Flowchart of a series of image processing by the image processing apparatus of the second embodiment The figure which shows an example (a) of thinning of distance data by a data size reduction part, and an example (b) of a synthesis | combination, respectively Flow chart of data size reduction processing by data size reduction unit Flow chart of another data size reduction process by the data size reduction unit Flowchart of further data size reduction processing by the data size reduction unit The figure which shows an example of the block diagram which shows the structure of an image input part The figure which shows another example of the block diagram which shows the structure of an image input part Flowchart of a series of image processing by an image processing apparatus including the image input unit of FIG. Flowchart of a series of image processing by the image processing apparatus of the second embodiment comprising the image input unit of FIG. A diagram explaining a conventional problem

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1-2 Image processing apparatus 10 Image input part (image input means)
10A, 10B Camera 10a-1, 10a-2 Optical system (lens)
14 Reading unit (reading means)
15 Saved data size setting part 16 Thinning part (thinning means)
17, 17 'distance calculation part (distance calculation means)
18 Data compression unit 23 Data size reduction unit (data size reduction means)

Claims (12)

In an image processing apparatus that generates stereoscopic image data by calculating the distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints,
Image input means for inputting the two image data;
Reading means for reading out all pixels of the input image data;
Thinning means for thinning out pixels constituting one of the two image data at a thinning rate according to the size of the saved data size set for saving the stereoscopic image data;
An image processing apparatus comprising: a distance calculating unit that calculates the distance for the pixels after the thinning by the thinning unit. In an image processing apparatus that generates stereoscopic image data by calculating the distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints,
Image input means for inputting the two image data;
Reading means for reading out all pixels of the input image data;
Distance calculating means for calculating the distance for all the read pixels;
The data size is reduced by synthesizing or thinning out the distance data calculated by the distance calculation means at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data. An image processing apparatus comprising: a data size reduction unit.   The image processing apparatus according to claim 1, wherein the thinning unit thins out the pixels in order of the pixels having the largest difference value of data values with respect to surrounding pixels among the pixels.   The data size reducing means synthesizes or thins out the data of the distance calculated by the distance calculating means in the order of the pixels having the largest difference value of the data value with the surrounding pixels among the pixels. The image processing apparatus according to claim 2, wherein the image processing apparatus is provided.   The data size reduction means combines or thins out the distance data calculated by the distance calculation means in order of pixels having a high correlation degree between corresponding pixels in the two image data. The image processing apparatus according to claim 2.   The data size reduction unit weights the distance calculated by the distance calculation unit according to the degree of correlation between corresponding pixels in the two image data, and combines the distance data after the weighting. The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus.   The image processing apparatus according to claim 1, wherein the image input unit includes two cameras.   The image processing apparatus according to claim 1, wherein the image input unit is a compound-eye camera including two lenses. In an image processing method for generating stereoscopic image data by calculating a distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints,
Input the two image data,
Read all the pixels of the input image data,
Thinning out pixels constituting one of the two image data at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data,
An image processing method, wherein the distance is calculated for the thinned pixels. In an image processing method for generating stereoscopic image data by calculating a distance to each part of the subject from two image data obtained by imaging the subject from two viewpoints,
Input the two image data,
Read all the pixels of the input image data,
Calculating the distance for all the read pixels;
The data size is reduced by synthesizing or thinning out the distance data calculated by the distance calculation means at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data. An image processing method. In an image processing program for causing a computer to calculate a distance to each part of the subject from two pieces of image data obtained by imaging the subject from two viewpoints and generate stereoscopic image data,
Input the two image data,
Read all the pixels of the input image data,
Thinning out pixels constituting one of the two image data at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data,
An image processing program for executing the calculation of the distance for the pixels after the thinning. In an image processing program for causing a computer to calculate a distance to each part of the subject from two pieces of image data obtained by imaging the subject from two viewpoints and generate stereoscopic image data,
Input the two image data,
Read all the pixels of the input image data,
Calculating the distance for all the read pixels;
The data size is reduced by combining or thinning out the distance data calculated by the distance calculation means at a thinning rate according to the size of the storage data size set for storing the stereoscopic image data. An image processing program for executing this.

Images