JP2009288893A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide technology for executing an actual correlation process at a smaller calculation cost. <P>SOLUTION: An image processor 1 is provided restricting the region for executing the correlation process by specifying the region in which a movement exists from an image. The image processor 1 is provided with an image processing section 111 for executing a distortion correction process for an original image; a difference detection section 112 for detecting a region with movement from the image to which a correction process is applied; and a correlation calculation section 113 for executing the correlation process, regarding the region in which the movement is detected. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a stereo image processing technique.

  A technique is known that correlates the correspondence relationship between a pair of images captured by a stereo camera and estimates the three-dimensional position of each point by triangulation from a deviation amount (parallax).

  However, when executing the correlation process, since all the regions (template blocks) set on the reference captured image are correlated with the other image, the size of the processing load becomes a problem.

  In this regard, Patent Document 1 describes a technique that can perform distance measurement in a short time by calculating an optical flow from image density data of both cameras and estimating a previous parallax based on the optical flow. .

JP-A-5-256610

  With the technique of Patent Document 1, although it is possible to reduce the load of image processing by predicting and complementing parallax from an optical flow, it has not been possible to reduce the calculation cost by actual correlation processing.

  Therefore, an object of the present invention is to provide a technique capable of executing actual correlation processing at a lower calculation cost.

  In order to solve the above-described problem, an image processing apparatus of the present invention is an image processing apparatus that performs processing related to a first image and a second image captured from different viewpoints, for example. Correlation calculation for performing correlation processing on a difference detection means for detecting a certain region, a region having motion in the first image, and a region on an epipolar line corresponding to the region having motion in the second image And means.

  As described above, an object of the present invention is to provide a technique capable of reducing the load caused by the correlation processing of stereo images.

The best mode for carrying out the present invention will be described below with reference to the drawings.
<First embodiment>
FIG. 1 is a block diagram showing a functional configuration of an image processing apparatus 1 according to the first embodiment of the present application.

  An imaging device 50 is connected to the image processing device 1.

  The imaging device 50 includes a left camera 50a that is a first camera and a right camera 50b that is a second camera. These are stereo cameras arranged at the same height and at a predetermined interval (base line length) such that their optical axes are parallel to each other. Note that an arbitrary image sensor such as a CCD or a CMOS can be used for each camera.

  Each camera outputs a pair of stereo images (hereinafter referred to as original images) according to the frame rate, and the original images are sent to the image processing apparatus 1 as digital data via an A / D converter (not shown). The driving of each camera may be controlled by the image processing apparatus 1 or may be independently driven by control of a camera CPU (not shown) provided in the imaging apparatus 50.

The image processing apparatus 1 includes a processor 100 such as a CPU (Central Processing Unit), a memory 20 and a graphic memory 30 for storing data handled by the processor 100,
And a storage device 40 that stores a correction table 41 relating to camera settings and correction.

  The memory 20 forms a luminance information storage area 21 and stores luminance information described later generated by the processor 100.

  The graphic memory 30 has a plurality of image buffers.

  The image buffer 31 is a frame buffer, for example, for rendering a pair of original images output from the imaging device 50. Here, the original image by the left camera 50a and the original image by the right camera 50b are stored sequentially. Therefore, the image buffer 31 has an area for storing at least two frames of image data.

  The image buffer 32 is for rendering a pair of corrected images 6 generated by performing correction processing on the original image. Here, since at least a corrected image based on the original image by the left camera 50a and a corrected image based on the original image by the right camera 50b are stored, the image buffer 32 has an area for storing at least two frames of image data. Have.

  In the distance image memory 33, a parallax map generated by a correlation calculation unit 113 described later is drawn. The parallax map will be described later.

  The storage device 40 is a storage device such as a flash memory, for example. The storage device 40 stores a correction table 41 used for correction processing described later.

  The processor 100 includes, as its functional units, an image processing unit 111 that performs distortion correction processing on the original image, a difference detection unit 112 that detects a region with motion from the image subjected to the correction processing, and motion detection And a correlation calculation unit 113 that executes correlation processing on the region that has been processed. Note that these functional units are realized by the processor 100 executing a predetermined program loaded into the memory 20.

  The image processing unit 111 acquires an image output from the imaging device 50, stores it in the image buffer 31, and executes distortion correction processing on the pair of original images. Specifically, the image processing unit 111 draws a pair of original images output from the imaging device 50 according to the frame rate in the image buffer 31 in the graphic memory 30. Then, correction processing is performed on the original image.

  The correction process is a pre-process that is performed to correct distortion caused by a difference in parameter, position, and angle of each camera and improve the recognition accuracy of the correlation process. Specifically, the image processing unit 111 performs a process such as affine transformation or shading correction according to the correction table 41 stored in the storage device 40 to generate a pair of corrected images 6. Further, the image processing unit 111 draws the generated corrected image 6 on the image buffer 32.

  The difference detection unit 112 extracts a region having motion from one corrected image (hereinafter referred to as a reference image) of the pair of corrected images 6 subjected to the correction process. Specifically, the difference detection unit 112 corrects an image captured by the left camera 50a, which is the first camera, of the generated pair of corrected images 6 (hereinafter referred to as a reference image 6a; see FIG. 2A). Then, luminance information storing luminance values for each predetermined area (block) is generated.

  Here, the luminance information will be described with reference to FIGS. 2 (a) and 2 (b). FIG. 2A is a schematic diagram of the reference image 6a and the search image 6b divided into predetermined large areas, and FIG. 2B is a schematic diagram of luminance information generated from the reference image 6a.

First, the difference detection unit 112 converts the reference image 6a read from the image buffer 32 into a large area (L _{1 to} L ₁₂ ; ₄ areas consisting of 4 small areas (S _{1 to} S ₄ ; for example, 4 × 4 pixels)). For example, it is divided into 8 × 8 pixels.

  Further, the difference detection unit 112 calculates an average luminance value of pixels existing in each large area, generates luminance information, and stores the luminance information in the luminance information storage area 21 of the memory 20.

  The luminance information is generated for each reference image 6 a by the difference detection unit 112 and stored in the luminance information storage area 21 of the memory 20. The luminance information storage area 21 stores at least luminance information based on the reference image 6a of the previous frame and luminance information based on the reference image 6a of the current frame. As shown in FIG. 2B, the luminance information includes an area ID storage area 21a, a luminance value storage area 21b, and a flag storage area 21c.

  The area ID storage area 21 a stores an identifier for identifying a predetermined large area divided by the difference detection unit 112.

  The luminance value storage area 21b stores the average luminance value of the pixels included in the large area.

  In the flag storage area 21c, a flag indicating that the movement is detected by the difference detection unit 112 is registered.

Returning to FIG. 1, the difference detection unit 112 detects an area with motion from the reference image 6 a using luminance information. Specifically, the difference detection unit 112 compares the latest luminance information and the previous luminance information (that is, based on the reference image of the previous frame) with a large region existing at a corresponding position on the image. (For example, L ₁ and R ₁ , L ₂ and R ₂ ). Then, the difference detector 112, a large area to the average luminance value is a predetermined threshold value Th ₁ or more changes to detect whether there. When there is a large area whose luminance difference from the previous frame is equal to or greater than the threshold Th ₁ , a flag (eg, “1”) is set in the flag storage area 21c of the latest luminance information that stores the area ID of the large area. It is registered and labeled as an area to be subjected to correlation processing described later.

  The correlation calculation unit 113 performs a correlation process on each small region included in the large region labeled by the difference detection unit 112 and updates the parallax map.

  First, general correlation processing (stereo matching) will be described with reference to FIG. FIG. 3 is a schematic explanatory diagram of area-based matching.

First, an image (reference image) captured by one camera is divided into predetermined template blocks (T _{1 to} T _n ). Then, for every template block, search within a predetermined area (search window) including an epipolar line of an image (search image) imaged by the other camera, determine a point that most closely matches the template block, The parallax d is obtained from the horizontal shift value.

  The correlation calculation unit 113 according to the present embodiment executes such correlation processing only for each small region included in the large region labeled by the difference detection unit 112. First, the correlation calculation unit 113 refers to the flag storage area 21c for luminance information and identifies a large labeled area.

  Next, the correlation calculation unit 113 includes a small region of the reference image 6a included in the labeled large region, a predetermined small region sequence (search window) including the epipolar line of the search image 6b corresponding to the small region, Are read from the image buffer 32. Then, the degree of coincidence (evaluation value) is calculated while shifting this template block pixel by pixel on the search window.

For example, when a large area _{L 7} of the reference image 6a shown in FIG. 2 (a) is labeled, the correlation calculation unit 113, a large area _{L 7,} the image a large area _R 5 to R ₈ containing the epipolar line Read from buffer 32.

Then, correlation calculation section 113, as shown in FIG. 4, a search window 101a which is a predetermined small area columns containing epipolar line of the small areas S ₁ and S ₂ included in the large region L _7, small areas S ₃ and a search window 101b is a predetermined small area columns containing epipolar lines S _4, were searched in each small region, it calculates the respective evaluation values.

  For example, an SAD (Sum of Absolute Difference) method is used to calculate the evaluation value. In the SAD method, the absolute value of the difference is taken for each pixel in the blocks of both images, and the sum is used as the evaluation value. The smaller this evaluation value is, the higher the matching degree between the blocks is, and the parallax d is given from the horizontal shift amount.

  Here, a reference image is defined as ImgL, a search image is defined as ImgR, a horizontal direction is defined as an i coordinate, and a vertical direction is defined as a j coordinate, and a small region is defined as i × j (i = 0 to n, j = 0). N) When the block is composed of pixels, the evaluation value is obtained by the equation (1).

S = Σ | ImgL (i, j) −ImgR (i, j) | (1)
Note that the evaluation value calculation method is not limited to the above. In addition to the SAD method, an SSD (Sum of Squared Intensity Difference) method that takes the square of the difference for each pixel in the block, and a local value from the luminance value of each pixel. Any method such as an NCC (Normalized Cross Correlation) method for subtracting the average value and calculating the similarity of the variance values may be used.

  Furthermore, the correlation calculation unit 113 updates the parallax map based on the calculated parallax between the small regions. The disparity map is a distance image created using the disparity calculated for each pixel, and the correlation calculation unit 113 re-creates the disparity map using the new disparity only for the small area for which the new disparity is calculated. Construct and use previous calculation results for other areas.

  The parallax map is generated by performing correlation processing on the entire region of the corrected image 6 in the initial operation of the image processing apparatus 1. Since such a parallax map is drawn in the distance image memory 33, the correlation calculation unit 113 can update the parallax map only for a region with motion.

  A flow of processing executed by the image processing apparatus 1 configured as described above will be described with reference to a flowchart shown in FIG.

  When a pair of original images is input from the imaging device 50, the image processing unit 111 of the image processing device 1 draws the images in the image buffer 31 (S11).

  Next, the image processing unit 111 performs a correction process on the original image to generate the corrected image 6 and draws it in the image buffer 32 (S12). Specifically, the image processing unit 111 reads an original image from the image buffer 31. Then, correction processing is performed on these using the correction table 41 stored in the storage device 40, and a pair of corrected images (reference image 6 a and search image 6 b) is stored in the image buffer 32. Thereafter, the image processing unit 111 outputs a luminance information generation request to the difference detection unit 112.

  Upon receiving the luminance information generation request from the image processing unit 111, the difference detection unit 112 divides the reference image 6a into predetermined regions and generates luminance information (S13). Specifically, the difference detection unit 112 reads the reference image 6a from the image buffer 32, divides the reference image 6a into predetermined large regions (for example, configured by 8 × 8 pixels), and detects an average luminance value for each large region. Generate luminance information. The generated luminance information is stored in the luminance information storage area 21 of the memory 20.

Next, the difference detection unit 112 detects a moving area from the reference image 6a (S14). Specifically, the difference detection unit 112 reads the latest luminance information and the luminance information of the previous frame from the luminance information storage area 21, compares the luminance information for each corresponding large area, and compares the luminance difference ( Change). Then, the difference detection unit 112 determines whether or not there is a large region whose luminance difference is equal to or greater than the predetermined threshold Th _{1. If} there is a large region (YES in S14), the region ID of the large region is stored. The flag is registered in the flag storage area 21c of the latest luminance information, and a correlation processing request is output to the correlation calculation unit 113. If it does not exist (NO in S14), the process is terminated.

  When the correlation calculation unit 113 receives the correlation processing request from the difference detection unit 112, the correlation calculation unit 113 specifies a template block to be subjected to correlation processing and a search window (S15). Specifically, the correlation calculation unit 113 first refers to the latest luminance information stored in the memory 20 to identify the area ID of the large area in which the flag is stored in the flag storage area 21c, and the large area Are read from the image buffer 32 as template blocks. Further, the correlation calculation unit 113 reads out the small region sequence of the search image 6b (including the epipolar line) corresponding to the template block from the image buffer 32, and specifies each as a search window.

  Next, the correlation calculation unit 113 executes a correlation process with each template block specified in step 15 and the search window (S16). The correlation calculation unit 113 calculates the degree of coincidence (evaluation value) while shifting each template block pixel by pixel on the corresponding search window, obtains the horizontal shift amount for the region with the highest degree of coincidence, and calculates the parallax. To do.

  Then, the correlation calculation unit 113 updates the parallax map based on the calculated parallax (S17).

  With the configuration as described above, the image processing apparatus 1 according to the present embodiment suppresses an increase in the amount of computation required for processing by executing correlation processing only for a region with motion, and generates a distance image with low load. Is possible.

  The luminance value stored in the luminance value storage area 21b may be a total luminance value obtained by adding the luminance values of each pixel instead of the average luminance value of the pixels included in the large area. In this case, the difference detection unit 112 may specify an area with movement from the difference in total luminance values of the large areas.

In addition, this invention is not limited to said embodiment, A various deformation | transformation is possible within the range of the summary.
<Modification>
For example, in the above embodiment, when a plurality of large areas with motion are detected, correlation processing is executed for all of the areas. In this modification, however, correlation processing is performed only for the large area where the maximum motion is detected. The case where it performs is demonstrated.

  The difference detection unit 112 identifies a large area in which the maximum motion is detected from the reference image 6a, and executes correlation processing only for each small area included in the large area.

In step 14 of FIG. 5, if there are a plurality of large areas in which the average luminance value has a change greater than or equal to the predetermined threshold value Th ₁ , the difference detection unit 112 identifies the large area having the maximum luminance difference (change amount). . Then, the difference detection unit 112 registers a flag in the large flag storage area 21 c and outputs a correlation processing request to the correlation calculation unit 113.

With the configuration as described above, in this modification, only the region with large motion is specified and correlation processing is executed, so that even when motion is detected in a plurality of regions, the processing cost of the correlation processing is kept low. It is possible.
<Second Embodiment>
Next, an image processing apparatus 2 according to the second embodiment of the present invention will be described.

  The image processing apparatus 2 according to the present embodiment includes a processor 200, a storage device 60, a memory 70, and a graphic memory 80.

  The storage device 60 stores the correction table 41 and the order data 62 in advance.

  The order data 62 is obtained by presetting the order of small areas to be subjected to correlation processing (to be a template block) when there is no moving area on the captured image. Therefore, when no motion is detected from the captured image, the correlation processing execution order is determined based on this order data. Details will be described later.

  The memory 70 has a luminance information storage area 71 and a difference information storage area 72.

  The luminance information storage area 71 stores luminance information A based on the reference image 6a generated by the difference detection unit 212 and luminance information B based on the search image 6b.

  The difference information storage area 72 stores difference information A for storing the luminance difference for each large area when the luminance information A is compared with the luminance information of the previous frame, and large information when the luminance information B is compared with the luminance information of the previous frame. Difference information B for storing a luminance difference for each region is stored.

  The graphic memory 80 includes an image buffer 82 in addition to the image buffer 31 and the distance image memory 33.

  The image buffer 82 is, for example, a line buffer, and is used to draw a scanning line required for correlation processing among a pair of corrected images 6 obtained by performing correction processing on the original image. The image buffer 82 has an area for storing at least two lines of image data, and sequentially stores the image data input from the difference detection unit 212.

  The processor 200 has, as its functional units, an image processing unit 211 that executes correction processing on the original image, and a difference detection unit 212 that extracts a region subjected to correlation processing from the luminance difference of the image subjected to correction processing. And a correlation calculation unit 213 that executes correlation processing on the extracted region.

  The image processing unit 211 has the same configuration as that of the first embodiment. The image processing unit 211 draws a pair of original images output from the imaging device 50 in the image buffer 31 and performs a correction process to perform a corrected image 6 (reference image 6a). And search image 6b).

  The difference detection unit 212 generates each piece of luminance information for the reference image 6a and the search image 6b, and specifies a region where the luminance difference with the previous frame is similar. Specifically, the difference detection unit 212 divides the reference image 6a and the search image 6b into large regions including predetermined small regions, and stores the average luminance value of the pixels existing in each large region of the reference image 6a. Information A and luminance information B storing the average luminance value of the pixels existing in each large area of the search image 6 b are generated and stored in the luminance information storage area 71.

The configuration of the large area and the small area is substantially the same as that of the first embodiment as shown in FIG. 7, but the number of pixels is different from that of the first embodiment. Here, it is assumed that each of the four small regions (S _{1 to} S ₄ ) included in the large region is composed of one pixel. Therefore, the large areas (L _{1 to} L ₁₂ and R _{1 to} R ₁₂ ) are composed of 2 × 2 pixels.

  The reference image 6a and the search image 6b according to the present embodiment are composed of six scanning lines (line 1 to line 6; see FIG. 7). When scanning each scanning line, the blank period B rotates at regular intervals according to the refresh rate. In the present embodiment, the correlation process that can be executed within the blank period B is a matching operation for one scanning line (8 blocks of small areas). Therefore, when executing a matching calculation for 8 blocks or less in the small area, the processing is not lost, and the correlation processing can be executed without delay with respect to the frame rate.

  As shown in FIG. 8A, the luminance information A and B have substantially the same configuration as the luminance information according to the first embodiment. However, instead of the flag storage area 21c, an area pair ID storage area 71c is provided. It is different in having. In the area pair ID storage area 71c, identifiers of large areas having similar luminance differences between the luminance information A and B are stored. Details will be described later.

  Next, the difference detection unit 212 takes the difference between the luminance information A and B and the luminance information of the previous frame, and generates difference information A and B. Specifically, the difference detection unit 212 compares the latest luminance information A and B with the previous luminance information A and B (for example, one frame before), and compares the difference for each large area with the difference information A and B. As B, the difference information storage area 72 is stored.

  As shown in FIG. 8B, the difference information A and B includes an area ID storage area 72a and a luminance difference storage area 72b. The area ID storage area 72a stores an identifier for identifying a large area. In the luminance difference storage area 72b, the luminance difference value when each luminance information is compared with the luminance information of the previous frame is stored.

Further, the difference detection unit 212 compares the luminance difference value stored in the difference information B for each luminance difference value stored in the difference information A, and the absolute value of the difference is equal to or less than a predetermined threshold Th ₂ . And the smallest similar area pair (that is, the closest brightness difference value) is detected. And the identifier of the large area which becomes a similar area pair is stored in the area pair ID storage area 71c of the luminance information. In addition, the difference detection unit 212 draws each scanning line constituting the similar region pair in the image buffer 82.

For example, as illustrated in FIG. 7, when the large region L ₂ and the large region R ₃ are specified as a similar region pair, the correlation calculation unit 213 determines whether the reference image 6 a and the search image 6 b including the similar region pair are included. The scanning lines (here, line 1 and line 2 of each image, a total of four lines) are drawn in the image buffer 82.

In the case where the absolute value of the difference does not exist the predetermined threshold value Th ₂ or less similar pair of regions, the difference detection unit 212 refers to the order data 62 to be stored in the storage device 60, the (template block used in the correlation process ) A small area is specified, and a scanning line including the small area and a small area sequence (scanning line) including the epipolar line are drawn in the image buffer 82.

  In the order data 62, the execution order of correlation processing executed when a similar area pair is not detected is set in units of small areas. Any order may be used, but in the present embodiment, as shown in FIG. 10, the small area serving as the template block moves along the scanning line from the upper left.

  For example, as shown in FIG. 10, when there is a small region that becomes a template block on the line 5, the difference detection unit 212 uses the line 5 (two in total) of the reference image 6 a and the search image 6 b as image buffers. Draw to 82.

  Next, processing executed by the correlation calculation unit 213 according to the present embodiment will be described with reference to FIGS. 7 and 9.

  When a similar area pair is stored in the area pair ID storage area 71c, that is, when a similar area pair is detected by the difference detection unit 212, the correlation calculation unit 213 includes each similar area included in the reference image 6a. For the small area, the correlation process is executed for the three blocks of the small area to be paired in the search image 6b and the small areas on both sides adjacent to the small area.

For example, if a large region L ₂ and the large area R ₃ is identified as a similar region pairs, the correlation calculation unit 213, as shown in FIG. 9, each of the small areas S ₁ to S ₄ included in the large region L ₂ as a template block, take the corresponding small regions included in the large region R _3, and small regions located on both sides of the correlation for a total of three blocks (search window).

  In addition, the correlation calculation unit 213 refers to the order data 62 stored in the storage device 60 when the similar area pair is not stored in the area pair ID storage area 71c, and specifies a small area that becomes a template block according to a predetermined order. Then, the correlation processing is executed using one scanning line (8 blocks of a small area; line 5 in FIG. 10) of the corresponding search image 6b as a search window.

  Then, the correlation calculation unit 213 updates the parallax map based on the calculated parallax between the small regions.

  A processing flow in the image processing apparatus 2 according to the present embodiment configured as described above will be described with reference to a flowchart shown in FIG.

  When a pair of original images is input from the imaging device 50, the image processing unit 211 of the image processing device 2 draws the images in the image buffer 31 (S21).

  Next, the image processing unit 211 performs correction processing on the original image to generate a corrected image 6 (a reference image 6a and a search image 6b) (S22). Then, the image processing unit 211 outputs a luminance information generation request to the difference detection unit 212.

  Upon receiving the luminance information generation request from the image processing unit 211, the difference detection unit 212 divides the reference image 6a and the search image 6b into predetermined areas and generates luminance information A and B (S23).

  Next, the difference detection unit 212 compares the luminance information A and B with the luminance information of the previous frame, and generates difference information A and B for storing the difference values (S24). Specifically, the difference detection unit 212 compares the latest luminance information A and B stored in the luminance information storage area 71 with the previous luminance information A and B, and detects a difference for each corresponding large area. Difference information A and B are generated and stored in the difference information storage area 72.

Then, the difference detection unit 212 compares the luminance difference value stored in the difference information B for each luminance difference value stored in the difference information A, and the absolute value of the difference is equal to or less than a predetermined threshold Th ₂ . A pair of similar regions having the smallest absolute value of the difference is detected (S25).

  When a similar area pair is detected in S25, the difference detection unit 212 stores an identifier for identifying the similar area pair in the area pair ID storage area 71c of the luminance information A and B. Then, the scanning line including the large area is drawn on the image buffer 82, and a correlation processing request is output to the correlation calculation unit 213.

  When the similar region pair is not detected in S25, the difference detection unit 212 refers to the order data 62 and specifies a small region to be subjected to correlation processing. Then, the scanning line of the reference image 6 a including the small area and the scanning line of the search image 6 b including the epipolar line are drawn on the image buffer 82, and a correlation processing request is output to the correlation calculation unit 213.

  When the correlation calculation unit 213 receives the correlation processing request from the difference detection unit 212, the correlation calculation unit 213 determines whether a similar region pair is detected in S25 (S26). Specifically, the correlation calculation unit 213 detects whether or not there is a record in which the region ID that is a similar region pair is stored in the region pair ID storage region 71c of the luminance information A and B. If a record exists (YES in S26), the process proceeds to step 27. If no record exists (NO in S26), the process proceeds to step 30.

  When the similar region pair exists (YES in S26), the correlation calculation unit 213 specifies a template block and a search window for the similar region pair (S27). Specifically, the correlation calculation unit 213 first refers to the latest luminance information A and B stored in the memory 20 and calculates the large area of the reference image 6a from the area ID stored in the area pair ID storage area 71c. And each small area included in the large area is read from the image buffer 82 as a template block. Furthermore, the correlation calculation unit 213 identifies the small area of the search image 6b corresponding to each small area that is a template block and the small areas on both sides of the small area as a search window, and reads out from the image buffer 82.

  When the similar region pair is not detected (NO in S26), the correlation calculation unit 213 specifies a template block and a search window according to a predetermined order (S30). Specifically, the correlation calculation unit 213 specifies a small area on the reference image 6 a serving as a template block based on the order data 62 stored in the storage device 60, and reads it from the image buffer 82. Then, the scanning line of the search image 6b corresponding to the small area serving as the search window is read from the image buffer 82.

  Then, the correlation calculation unit 213 performs correlation processing between each template block identified in step 27 or step 30 and the search window (S28), and updates the parallax map based on the calculated parallax for each small region. (S29).

  According to the configuration as described above, the image processing apparatus 2 according to the present embodiment estimates the corresponding region between the two stereo images from the luminance difference and limits the range in which the correlation processing is performed, thereby reducing the frame rate. Processing within the blank period is possible without delay.

  In addition, since the correlation process is performed at any time in a predetermined order even in an area where the change in luminance value is small, a more accurate parallax map can be obtained with a low processing cost.

  Further, by terminating the processing within the blank period, it is not necessary to buffer the corrected image for each frame, and image processing with a small memory capacity can be realized.

  The size of each region (horizontal and vertical pixel count) is an example, and the present invention is not limited to the above embodiment.

1 is a block diagram showing a functional configuration of an image processing apparatus according to a first embodiment of the present invention. (A) Schematic diagram of reference image 6a and search image 6b, (b) Schematic diagram of luminance information generated from reference image 6a. Schematic explanatory diagram of area-based matching. FIG. 5 is a schematic diagram of a template block and a search window that are objects of correlation processing. 6 is a flowchart showing a flow of processing in the image processing apparatus according to the first embodiment. The block diagram which shows the functional structure of the image processing apparatus which concerns on 2nd embodiment of this invention. Schematic of the reference image 6a and the search image 6b according to the second embodiment. (A) Schematic diagram of luminance information A and B, (b) Schematic diagram of difference information A and B. FIG. 10 is a schematic diagram of a template block and a search window that are correlation processing targets when a similar region pair is detected in the image processing apparatus according to the second embodiment of the present invention. FIG. 10 is a schematic diagram of a template block and a search window that are correlation processing targets when a similar region pair is not detected in the image processing apparatus according to the second embodiment of the present invention. 10 is a flowchart showing a flow of processing in the image processing apparatus according to the second embodiment.

Explanation of symbols

  1, 2: Image processing device, 100/200: Processor, 111/211: Image processing unit, 112/212: Difference detection unit, 113/313: Correlation calculation unit, 40/60: Storage device, 41: Correction table, 62: Order data, 20.70: Memory, 30.80: Graphic memory, 50: Imaging device.

Claims (7)

A processing unit that acquires the first image and the second image captured from different viewpoints from the imaging unit;
Difference detection means for detecting a region of motion from the first image;
Correlation calculating means for executing correlation processing on a region having motion in the first image and a predetermined region including the epipolar line of the second image corresponding to the region having motion. A featured image processing apparatus. The image processing apparatus according to claim 1,
The difference detecting means includes
Dividing the first image into a second region comprising a plurality of first regions;
When the amount of change in the luminance value of the second area included in the first image is equal to or greater than a predetermined threshold, the second area is determined as a moving area. . The image processing apparatus according to claim 2,
The difference detecting means includes
When the amount of change in the luminance value of the second area included in the first image is equal to or greater than a predetermined threshold and is the maximum, the second area is determined as a moving area. An image processing apparatus. The image processing apparatus according to claim 2 or 3,
The correlation calculation means includes
A first region included in the second region detected as the region of movement;
A predetermined region corresponding to the first region and including an epipolar line of the second image;
An image processing apparatus characterized by executing correlation processing for. A processing unit that acquires the first image and the second image captured from different viewpoints from the imaging unit;
Dividing the first image into a second region comprising a plurality of first regions;
Dividing the second image into a fourth region comprising a plurality of third regions;
Detecting a change amount of a luminance value of a second region included in the first image;
Detecting the amount of change in the luminance value of the fourth region included in the second image;
The amount of change in the luminance value of the second region included in the first image is compared with the amount of change in the luminance value of the fourth region included in the second image. Is a difference detection means for detecting a pair of regions that are equal to or less than a predetermined threshold;
An image processing apparatus comprising: correlation calculation means for executing correlation processing for the region pair. The image processing apparatus according to claim 5,
The correlation calculation means includes
A first region included in a second region detected as the region pair;
A third region corresponding to the first region, and a third region present on the left and right of the third region;
An image processing apparatus characterized by executing correlation processing for. The image processing apparatus according to claim 5, wherein:
When the region pair is not detected by the difference detection unit,
The correlation calculation means includes
A first region determined in a predetermined order; a third region row including an epipolar line corresponding to the first region;
An image processing apparatus characterized by executing correlation processing for.

Images