JP2013073306A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device which properly performs pattern-matching by suppressing variation of spatial frequency in a pair of image data to be pattern-matched.SOLUTION: An image processing device 120 comprises: a color restoring section 172 that, for a pair of image data composed of a Bayer array, restores brightness of a hue unset to each pixel on the basis of adjacent pixels; a color synthesizing section 174 that, for the pair of image data, synthesizes brightness of a hue preset for each pixel on the basis of adjacent pixels; and a matching processing section 180 that extracts blocks with a predetermined size from each of the pair of image data, performs matching, and specifies blocks which are highly correlated. Thus, the pattern-matching can be performed properly by suppressing variation of spatial frequency.

Description

  The present invention relates to an image processing apparatus that executes pattern matching based on a pair of captured image data.

  Conventionally, an object such as an obstacle such as a vehicle or a traffic light located in front of the host vehicle is detected, and control is performed so as to avoid a collision with the detected object or to keep a distance between the preceding vehicle and a safe distance. Techniques to do this are known (for example, Patent Documents 1 and 2).

  The inter-vehicle distance from the preceding vehicle can be obtained from the parallax of the object in a pair of image data captured at different positions, for example. Further, the parallax of the object in the pair of image data is derived based on pattern matching between images. As pattern matching, it is generally performed that blocks having a predetermined size are compared (matched) between images and a block having high correlation is specified. In addition, a technique for performing pattern matching using an image density histogram is also disclosed (for example, Patent Document 3).

  Furthermore, in order to increase the accuracy of pattern matching, a technique for geometrically correcting the optical position shift of a captured image in the previous stage of pattern matching has been disclosed (for example, Patent Documents 4 and 5).

Japanese Patent No. 3349060 JP-A-10-283461 JP-A-5-210737 Japanese Patent No. 3284190 Japanese Patent No. 3261115

  As a means for acquiring an image, for example, a so-called so-called so-called so-called so-called so-called so-called so-called so-called so-called image arrangement, a plurality of light receiving portions (photodiodes) corresponding to pixels arranged in a grid pattern are regularly and exclusively provided with color filters for RGB signals of three primary colors The Bayer array is known. In the Bayer array, only one hue of RGB is acquired for each pixel. Therefore, the luminance of an unset (missing) hue in each pixel is usually interpolated based on adjacent pixels to restore the color. . Then, each pixel whose color is restored is geometrically corrected as described above, and then pattern matching is performed.

  However, in the above-described color restoration, the luminance of an unset hue is obtained by linearly interpolating the luminance of adjacent pixels, so that the spatial frequency is lowered. In this case, a spatial frequency deviation occurs between a pixel that is not subjected to color restoration processing and a pixel that requires restoration processing, resulting in a situation in which the accuracy of pattern matching is not improved. Further, even in geometric position correction, a spatial frequency deviation occurs between a pixel that does not require position correction and a pixel that requires position correction, and similarly, the accuracy of pattern matching cannot be improved. It was.

  The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus capable of appropriately performing pattern matching by suppressing a spatial frequency deviation in a pair of image data to be subjected to pattern matching. It is said.

  In order to solve the above problems, an image processing apparatus according to the present invention includes a color restoration unit that restores the luminance of a hue that has not been set for each pixel based on adjacent pixels in a pair of image data configured by a Bayer array. In a pair of image data, a color composition unit that synthesizes the luminance of a hue set in advance for each pixel based on surrounding pixels, and a block of a predetermined size is extracted from each of the pair of image data and matched. And a matching processing unit that identifies highly correlated blocks.

  The color synthesizing unit may synthesize the luminance of the hue set in advance for each pixel based on the surrounding pixels so as to approach the minimum value of the spatial frequency that is reduced by being restored by the color restoring unit.

  The color synthesizing unit may synthesize the luminance of a pixel in which the luminance of the hue is set in advance and the luminance of the surrounding same hue at a predetermined ratio.

  A coordinate conversion unit that performs coordinate conversion of each pixel based on a positional deviation characteristic for each pixel determined in advance according to an imaging device that generates a pair of image data, and derives the luminance of the pixel after coordinate conversion, and a coordinate conversion unit A coordinate synthesizing unit that synthesizes the luminance of pixels determined to require no coordinate conversion based on surrounding pixels may be further included.

  In order to solve the above-described problem, another image processing apparatus of the present invention performs coordinate conversion of each pixel based on a positional deviation characteristic for each pixel determined in advance according to an imaging apparatus that generates a pair of image data. A coordinate conversion unit that derives the luminance of the pixel after conversion, a coordinate synthesis unit that combines the luminance of the pixel determined to be unnecessary for coordinate conversion by the coordinate conversion unit based on the surrounding pixels, and a pair of image data. And a matching processing unit that performs matching by extracting blocks having a size of 2 and specifies blocks having high correlation.

  The coordinate synthesizing unit may synthesize the luminance of pixels that are determined not to require coordinate transformation based on the surrounding pixels so as to approach the minimum value of the spatial frequency that is reduced by the coordinate transformation performed by the coordinate transformation unit. .

  The coordinate synthesizing unit may synthesize the luminance of a pixel determined not to require coordinate conversion and the luminance of the surrounding same hue at a predetermined ratio.

  The matching processing unit has a matching position having a higher resolution than the pixel unit based on the correlation value between the specified blocks and two correlation values when one of the specified blocks is shifted in the horizontal direction by a unit pixel. May be derived.

  ADVANTAGE OF THE INVENTION According to this invention, pattern matching can be appropriately performed by suppressing the deviation of the spatial frequency in a pair of image data used as the object of pattern matching. Therefore, the parallax information can be appropriately derived, and it is possible to appropriately perform control for avoiding a collision with an object and maintaining a safe distance between the preceding vehicle and the preceding vehicle.

It is the block diagram which showed the connection relation of the environment recognition system by 1st Embodiment. It is explanatory drawing for demonstrating a luminance image and a distance image. It is explanatory drawing for demonstrating an example of a Bayer arrangement | sequence. It is explanatory drawing for demonstrating a color restoration process. It is explanatory drawing for demonstrating a coordinate transformation process. It is explanatory drawing which shows an example of coordinate transformation. It is a functional block diagram showing a schematic function of the image processing apparatus. It is explanatory drawing which showed the example of a synthesis | combination of the color synthetic | combination part. It is explanatory drawing which showed the example of a synthesis | combination of a coordinate synthetic | combination part. It is explanatory drawing which showed transition of the correlation value (difference value) in pattern matching. It is the flowchart which showed the whole flow of the image processing method. It is the flowchart which showed the flow of the color restoration process. It is the flowchart which showed the flow of the coordinate transformation process.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

(Environment recognition system 100)
The image processing apparatus is intended to process a captured image in order to appropriately achieve an arbitrary object. For example, when the image processing device is employed in an environment recognition system for recognizing the environment around the vehicle, the image processing device processes the image captured by the imaging device, and calculates the relative distance of the object in the image. The parallax information of the image is derived for identification. Here, in order to facilitate understanding of the image processing apparatus, first, an environment recognition system as an embodiment using the image processing apparatus will be described, and then a specific configuration of the image processing apparatus will be described in detail.

  FIG. 1 is a block diagram showing a connection relationship of the environment recognition system 100. The environment recognition system 100 includes an imaging device 110, an image processing device 120, an environment recognition device 130, and a vehicle control device 140 provided in the vehicle 1.

(Imaging device 110)
The imaging device 110 includes an imaging device such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). Get the brightness. In the present embodiment, color and luminance are treated equally, and when both words are included in the same sentence, each other can be read as luminance constituting the color or a color having luminance. Here, it is assumed that a color image by a Bayer arrangement is obtained in which color filters for each of the RGB signals are regularly and exclusively arranged at a light receiving portion (photodiode) corresponding to a pixel. In addition, a color image captured by the imaging device 110 is referred to as a luminance image and is distinguished from a distance image described later.

  The imaging devices 110 are arranged in a substantially horizontal direction so that the optical axes of the two imaging devices 110 are substantially parallel on the traveling direction side of the vehicle 1. The imaging device 110 continuously generates, for example, every 1/60 seconds (60 fps), image data obtained by imaging an object existing in the detection area in front of the vehicle 1. Here, the objects include not only three-dimensional objects that exist independently such as vehicles, traffic lights, roads, and guardrails, but also objects that can be specified as three-dimensional object parts such as taillights, blinkers, and lighting parts of traffic lights. Each functional unit in the following embodiment performs each process triggered by such update of image data.

(Image processing apparatus 120)
The image processing device 120 acquires image data from each of the two imaging devices 110, evaluates the correlation between the acquired pair of image data, and stores both arbitrary blocks (collected a predetermined number of pixels) in the image. Deriving parallax information including parallax between images. The image processing apparatus 120 derives the parallax using so-called pattern matching in which a block corresponding to a block arbitrarily extracted from one image data (for example, an array of horizontal 4 pixels × vertical 4 pixels) is searched from the other image data. To do. Here, the horizontal indicates the horizontal direction of the captured image and corresponds to the horizontal in real space. The vertical indicates the screen vertical direction of the captured image and corresponds to the vertical direction in the real space. Such correlation evaluation and pattern matching will be described in detail later.

  However, the image processing apparatus 120 can derive the parallax for each block, which is a unit of detection resolution, but cannot recognize what kind of target object the block is. Accordingly, the disparity information is derived independently not in units of objects but in units of detection resolution (for example, blocks) in the detection region. Here, an image in which the parallax information derived in this way is associated with image data is referred to as a distance image.

  FIG. 2 is an explanatory diagram for explaining the luminance image 124 and the distance image 126. For example, it is assumed that a luminance image (image data) 124 as illustrated in FIG. 2A is generated for the detection region 122 through the two imaging devices 110. However, only one of the two luminance images 124 is schematically shown here for easy understanding. The image processing apparatus 120 obtains the parallax for each block from the luminance image 124 and forms a distance image 126 as shown in FIG. Each block in the distance image 126 is associated with the parallax of the block. Here, for convenience of description, blocks from which parallax is derived are represented by black dots.

(Environment recognition device 130)
The environment recognition device 130 acquires the luminance image 124 and the distance image 126 from the image processing device 120, and the luminance based on the luminance image 124 and the relative distance to the vehicle (own vehicle) 1 based on the parallax information of the distance image 126. Is used to identify which object (vehicle, traffic light, road, guardrail, tail lamp, blinker, each lighting part of the traffic light, etc.) corresponds to the object in the detection area 122. At this time, the environment recognition apparatus 130 converts the disparity information for each block in the detection region 122 in the distance image 126 into three-dimensional position information including the relative distance using a so-called stereo method. Here, the stereo method is a method of deriving a relative distance of the target object from the imaging device 110 from the parallax of the target object by using a triangulation method.

(Vehicle control device 140)
The vehicle control device 140 performs control to avoid a collision with an object specified by the environment recognition device 130 or to keep the distance between the vehicle and the preceding vehicle at a safe distance. Specifically, the vehicle control device 140 acquires the current traveling state of the vehicle 1 through the steering angle sensor 142 that detects the steering angle, the vehicle speed sensor 144 that detects the speed of the vehicle 1, and the like, and controls the actuator 146. Maintain a safe distance from the preceding vehicle. Here, the actuator 146 is an actuator for vehicle control used for controlling a brake, a throttle valve, a steering angle, and the like. Further, when a collision with an object is assumed, the vehicle control device 140 displays a warning (notification) to that effect on the display 148 installed in front of the driver, and controls the actuator 146 to control the vehicle 1. Brakes automatically. Such a vehicle control device 140 may be formed integrally with the environment recognition device 130.

(Problems and solutions in the present embodiment)
Color images can be acquired in various ways. For example, (1) the incident light to the imaging device 110 is divided into RGB hues by a prism and images are acquired for each hue by three imaging elements, or (2) RGB sensitivity cells are superimposed in the optical path direction. It is possible to acquire an image based on RGB at once using the image pickup device, and (3) to acquire one hue regularly and exclusively for each pixel by the Bayer array. In the present embodiment, an image is acquired using (3) Bayer array.

  FIG. 3 is an explanatory diagram for explaining an example of the Bayer array. As shown in FIG. 3A, the Bayer array is a system in which each hue RGB is regularly and exclusively arranged on pixels arranged in a grid pattern, and in any hue, at least the horizontal direction and the vertical direction. The same hue is arranged in the two adjacent pixels. However, the hue G has a density (occupied area) that is twice that of the hue R and the hue B. This is because, as in the spectral sensitivity characteristic shown in FIG. 3B, the sensitivity distribution of the hue G is high, so that it is easy to acquire luminance information, and human vision has high sensitivity to the hue G. Because.

  In the Bayer array, luminance can be obtained from each pixel only for one hue. Therefore, color restoration processing (interpolation processing) is performed on the other two hues that are missing using the luminance of the hue of adjacent pixels.

  FIG. 4 is an explanatory diagram for explaining the color restoration processing. In the following figures including FIG. 4, the numbers attached around the image indicate the horizontal and vertical positions of the pixels. For example, the hue R is arranged as shown in FIG. 4A in the Bayer array. Accordingly, the luminances of the pixels (2, 1), (2, 3), (4, 1), and (4, 3) whose luminance has already been acquired for the hue R are the acquired luminances a, b, c, and d. Can be used as is.

  Also, in the horizontal direction or the vertical direction, in the pixels (2, 2), (3, 1), (3, 3), and (4, 2) where the two pixels whose luminance is acquired are adjacent, The luminance of the pixel is linearly interpolated, the luminance of the pixel (2, 2) = (a + b) / 2, the luminance of the pixel (3,1) = (a + c) / 2, the luminance of the pixel (3,3) = (b + d) / 2, luminance of pixel (4, 2) = (c + d) / 2.

  Further, in the pixel (3, 2) in which the four pixels whose luminance is acquired are adjacent in the diagonal direction, the luminance of the four adjacent pixels is linearly interpolated, and the luminance of the pixel (3, 2) = (a + b + c + d) / 4. Here, for convenience of explanation, derivation of pixels in the first horizontal row and the fourth vertical row is omitted. Such pixel interpolation can also be applied to the hue B as shown in FIG. 4B, and therefore the description of the hue B is omitted here.

  Further, the hue G is arranged in the Bayer arrangement as shown in FIG. Accordingly, the luminances of the pixels (2, 2), (3, 1), (3, 3), and (4, 2) whose luminance has already been acquired for the hue G are the acquired luminances e, f, g, h. Can be used as is. Further, in the pixel (3, 2) in which the four pixels whose luminance is acquired in the horizontal direction and the vertical direction are adjacent, the luminance of the four adjacent pixels is linearly interpolated, and the luminance of the pixel (3, 2) = (e + f + g + h) ) / 4.

  Here, attention is paid to the spatial frequency in the luminance of the color-reconstructed pixel. The spatial frequency indicates the fineness of the periodic structure of the image, and can be expressed by a period per unit length. When applied to the imaging device 110, the spatial frequency represents the resolution of the lens. The spatial frequency is high when no processing is performed, and is low by averaging and synthesis.

  Accordingly, in the example of FIG. 4A, the spatial frequencies of the pixels (2, 1), (2, 3), (4, 1), and (4, 3) for which the luminance has already been acquired for the hue R are assumed. 1, the spatial frequency of the pixels (2, 2), (3, 1), (3, 3), (4, 2) obtained by linear interpolation of the luminance of the two pixels becomes 1/2, The spatial frequency of the pixel (3, 2) obtained by linearly interpolating the pixel is 1/4. However, here, in order to facilitate understanding, the value of the spatial frequency is calculated for a simple pixel, but it goes without saying that the spatial frequency varies depending on the actually acquired pixel.

  Then, in the four adjacent pixels surrounded by the broken line in FIG. 4A, the spatial frequency is different from 1, 1/2, and 1/4, and the deviation is between the maximum value and the minimum value of the spatial frequency. Arise. Normally, even if a pixel having a low spatial frequency is generated, the sharpness of the image is maintained by the pixel having a high spatial frequency. However, with regard to pattern matching described later, such a spatial frequency deviation may be harmful.

  In other words, pattern matching evaluates the correlation between pixels in a pair of image data, and even if pixels having the same luminance and corresponding to the same part of the object have different spatial frequencies, the correlation Low. Here, it is conceivable to align with a high spatial frequency, but it is difficult to restore the high spatial frequency from the information amount of the pixel that has become 1/4. Therefore, in this embodiment, an attempt is made to align the spatial frequency to a low value.

  Therefore, in the present embodiment, in FIG. 4A, the pixels (2, 2), (3, 1), (3, 3), (4, 2) having a spatial frequency of 1/2, The pixel (3, 2) whose frequency is ¼ is maintained as it is, and the spatial frequency 1 of the pixel (2, 1), (2, 3), (4, 1), (4, 3) is intentionally lowered. I will do it.

  When the spatial frequency is made uniform (suppressing the spatial frequency deviation) in this way, the lower the value, the better the pattern matching accuracy. This is because when the other region corresponding to an arbitrary pixel in one of the pair of image data straddles two pixels of the image data, if the spatial frequency is a high pixel, the correlation value of both of the two pixels is low. Because. At this time, if the spatial frequency is lowered, the pixel itself is blurred and the correlation with arbitrary pixel data is increased. In this way, pattern matching is appropriately performed. However, if the spatial frequency is lowered too much, it is necessary to be careful because matching can be performed even between different objects.

  When the color restoration processing is completed, next, coordinate conversion processing is performed in which each pixel is coordinate-converted based on a positional deviation (distortion) characteristic for each pixel that is predetermined according to the imaging device 110. The coordinate conversion is a process for returning an object that has been acquired by being distorted from the original position by passing through the imaging device 110 to the original position. For example, an image having a positional shift characteristic as shown in FIG. 5A through a fish-eye lens is rotated as shown in FIG. 5B by a simple shift in the horizontal direction or the vertical direction or rotational movement using affine transformation or the like. Correct the image so that the horizontal and vertical directions are straight.

  FIG. 6 is an explanatory diagram illustrating an example of coordinate transformation. For example, when coordinate conversion is performed to shift an arbitrary pixel 150 by x in the horizontal right direction and y in the vertical downward direction, a new pixel 152 having different coordinates is generated. The luminance of the pixel 152 can be represented by a combination of the luminances of the four pixels indicated by broken lines in FIG. For example, when one pixel indicated by a solid line is restored from the luminances of four pixels = a, b, c, and d by geometric correction, the luminance of the pixel 152 indicated by the solid line is represented by a × using the distances x and y. (1-x) (1-y) + b * x (1-y) + c * (1-x) y + d * xy

  Then, as in the color restoration process, the spatial frequency of the pixel 152 becomes 1/4 in the coordinate conversion process. However, when the pixel 152 happens to be generated over only two pixels, the spatial frequency is ½, and when the pixel 152 completely overlaps with one pixel, the spatial frequency is 1. As described above, when the spatial frequency is different from 1, 1/2, and 1/4, as described in the color restoration processing, the deviation of the spatial frequency becomes a harmful effect in the pattern matching described later.

  Therefore, in the present embodiment, the pixel having a spatial frequency of ¼ or the pixel having a spatial frequency of ½ is maintained as it is, and it is determined that coordinate conversion is unnecessary by accidentally overlapping one pixel. The spatial frequency 1 of the selected pixel is deliberately lowered. Thus, by lowering the spatial frequency, the pixels themselves are blurred and the correlation with arbitrary pixel data is increased. In this way, pattern matching is appropriately performed.

  Hereinafter, the configuration of the image processing apparatus 120 capable of improving the accuracy of pattern matching by lowering the spatial frequency and suppressing the spatial frequency deviation will be specifically described.

(Image processing apparatus 120)
FIG. 7 is a functional block diagram illustrating schematic functions of the image processing apparatus 120. As shown in FIG. 7, the image processing apparatus 120 includes an I / F unit 160, a data holding unit 162, and a central control unit 164.

  The I / F unit 160 is an interface for performing bidirectional information exchange with the imaging device 110 and the environment recognition device 130. The data holding unit 162 includes a RAM, a flash memory, an HDD, and the like. The data holding unit 162 holds various information necessary for processing of each function unit described below, and temporarily stores the luminance image 124 received from the imaging device 110. Hold.

  The central control unit 164 is configured by a semiconductor integrated circuit including a central processing unit (CPU), a ROM in which programs are stored, a RAM as a work area, and the like, and through the system bus 166, the I / F unit 160 and the data holding unit. 162 is controlled. In the present embodiment, the central control unit 164 also functions as a sensitivity correction unit 170, a color restoration unit 172, a color synthesis unit 174, a coordinate conversion unit 176, a coordinate synthesis unit 178, a matching processing unit 180, and a parallax derivation unit 182. To do.

  The sensitivity correction unit 170 performs predetermined processing such as gamma correction and knee processing on the pair of image data received from the imaging device 110.

  As described with reference to FIG. 4, the color restoration unit 172 restores the luminance of the hue not set for each pixel by linear interpolation or the like based on the adjacent pixels in the pair of image data configured by the Bayer array. To do.

  The color synthesizing unit 174 synthesizes the luminance of the hue that is set in advance for each pixel and is not originally processed in the pair of image data based on the surrounding pixels. Here, the color synthesizing unit 174 synthesizes the luminance of the hue set in advance for each pixel based on the surrounding pixels so as to approach the minimum value of the spatial frequency that is reduced by being restored by the color restoring unit 172. doing. Thus, by matching the spatial frequency to the low spatial frequency pixels restored by the color restoration unit 172, the spatial frequency deviation can be suppressed, and the accuracy of pattern matching can be improved.

  More specifically, the color composition unit 174 synthesizes the brightness of pixels whose hue is set in advance and the brightness of the surrounding same hue at a predetermined ratio. Hereinafter, a synthesis example of the color synthesis unit 174 will be described with reference to FIG.

  In the Bayer array, the hue R is set to be separated by one pixel as shown in FIG. For example, it is assumed that the luminance e with respect to the hue R is preset in the pixel (4, 3). However, in this embodiment, the luminance e is not used as it is, and the spatial frequency is intentionally lowered. That is, the pixels (2, 1), (2, 3), (2, 5), (4, 1), (4, 5), (6, 1), (6, 3), (6, 5) The luminance e of the pixel (4, 3) is converted to (a + b + c + d + 8e + f + g + h + i) / 16 where a is a, b, c, d, f, g, h, i. Here, in order to leave the information of the original luminance e of the pixel (4, 3) itself, “8” is multiplied, added with the luminance of other pixels, and finally divided by “16”. That is, the luminances a, b, c, d, e, f, g, h, i are synthesized at a ratio of 1: 1: 1: 1: 8: 1: 1: 1: 1. By doing so, the spatial frequency of the pixel (4, 3) can be lowered to any value between 1/16 and 1.

  Note that the numerical values for multiplication and division can be arbitrarily determined as long as the luminances of the plurality of pixels are weighted and added and finally uniformized, and are limited to the above numerical values “8” and “16”. Not. For example, the luminance e of the pixel (4, 3) may be converted to (b + d + 4e + f + h) / 8 by referring to only the horizontal and vertical pixels.

  As for the hue B, as in the hue R, as shown in FIG. 8B, the pixels (1, 2), (1, 4), (1, 6), (3, 2), (3, 6), ( 5,2), (5,4), and (5,6) are a, b, c, d, f, g, h, i, and the luminance e of the pixel (3,4) is (a + b + c + d + 8e + f + g + h + i). ) / 16. Thus, the spatial frequency of the pixel (3, 4) can be lowered to any value between 1/16 and 1.

  In the Bayer array, the hue G is set to be separated by one pixel as in the hues R and B. However, unlike the hues R and B, pixels whose luminance is set in advance in the diagonal direction are adjacent to each other. Therefore, the spatial frequency can be lowered by using pixels closer to the hues R and B, that is, by using a value close to the original luminance. For example, as shown in FIG. 8C, when the luminances of the pixels (2, 2), (2, 4), (4, 2), and (4, 4) are j, k, m, and n, The luminance l of (3, 3) is converted to (j + k + 4l + m + n) / 8. Thus, the spatial frequency of the pixel (3, 3) can be lowered to any value between 1/8 and 1. Again, only the original luminance of the pixel (3, 3) itself is weighted four times to increase the influence of luminance over the other pixels. That is, the luminances j, k, l, m, and n are synthesized at a ratio of 1: 1: 4: 1: 1. For the hue G, the numerical values for multiplication and division can be arbitrarily determined as long as the luminances of a plurality of pixels are weighted and added and finally uniformized.

  However, when other pixels to be referred to have already been subjected to the above conversion, the spatial frequency is too low than expected. Therefore, when the preset luminance is held in the data holding unit 162 and the color composition is performed, it is preferable to use the original luminance that is held without using the luminance after color synthesis.

  With such a configuration, the spatial frequency deviation can be suppressed, and the image can be converted into an image that can be easily subjected to pattern matching as compared with the case where color restoration processing is performed only on an arbitrary pixel. Therefore, it is possible to avoid a situation in which pattern matching is not properly executed or to derive incorrect parallax information due to mismatching, and to improve the accuracy of pattern matching.

  Here, the calculation is performed on the premise that the degree of occupation (density) of the hue G is high, but the calculation formula of each hue can be arbitrarily changed according to the degree of occupation. Here, as described above, it is only necessary to lower the spatial frequency by weighting and adding the luminances of a plurality of pixels and finally dividing the luminance.

  As described with reference to FIG. 5, the coordinate conversion unit 176 performs coordinate conversion of each pixel based on a positional deviation characteristic for each pixel that is predetermined according to the imaging device 110 that generates a pair of image data, and performs coordinate conversion. The luminance of the subsequent pixel is derived (color restoration). Such coordinate transformation can employ various existing techniques such as Japanese Patent No. 3284190, and therefore detailed description thereof is omitted here.

  The coordinate synthesizing unit 178 synthesizes the luminance of the pixels that are determined not to require coordinate transformation by the coordinate transformation unit 176 based on the surrounding pixels. Here, the coordinate synthesizing unit 178 synthesizes the luminance of the pixels that are determined not to require coordinate transformation based on the surrounding pixels so as to approach the minimum value of the spatial frequency that is lowered by the coordinate transformation unit 176. Thus, by matching the spatial frequency to the low spatial frequency pixels converted by the coordinate conversion unit 176, the spatial frequency deviation can be suppressed, and the accuracy of pattern matching can be improved.

  Specifically, the coordinate synthesizing unit 178 synthesizes the luminance of the pixel that is determined not to require coordinate conversion and the luminance of the surrounding same hue at a predetermined ratio. Below, the example of a synthesis | combination of the coordinate synthetic | combination part 178 is shown using FIG.

  For example, since the pixel 152 after coordinate conversion overlaps the pixel (2, 2), if it is determined that coordinate conversion is unnecessary, the luminance c at the pixel (2, 2) can be used as it is. However, in the present embodiment, the luminance c is not used as it is, and the spatial frequency is intentionally lowered. That is, when the luminances of the pixels (1, 2), (2, 1), (2, 3), (3, 2) are a, b, d, e, the luminance c of the pixel (2, 2) is Convert to (a + b + 4c + d + e) / 8. Here, “4” is multiplied to leave the information of the original luminance c of the pixel (2, 2) itself, added with the luminance of other pixels, and finally divided by “8”. By doing so, the spatial frequency of the pixel (2, 2) can be lowered to any value between 1/8 and 1.

  With this configuration, the spatial frequency deviation can be suppressed, and the image can be converted into an image that can be easily subjected to pattern matching as compared with the case where only the coordinate conversion process is performed. Therefore, it is possible to avoid a situation in which pattern matching is not properly executed or to derive incorrect parallax information due to mismatching, and to improve the accuracy of pattern matching.

  The matching processing unit 180 extracts blocks of a predetermined size from each of a pair of data whose luminance has been restored and combined and whose positional deviation has been corrected, and performs matching to specify blocks having high correlation.

  As this pattern matching, it is conceivable to compare luminance values (Y color difference signals) in units of blocks indicating an arbitrary image position between two pieces of image data. For example, the SAD (Sum of Absolute Difference) that takes the difference in luminance value, the SSD (Sum of Squared intensity Difference) that uses the difference squared, and the similarity of the variance value obtained by subtracting the average value from the luminance value of each pixel. There are methods such as NCC (Normalized Cross Correlation). The image processing apparatus 120 performs such a block-based parallax derivation process for all blocks displayed in the detection area (for example, 600 pixels × 200 pixels). Here, the block is 4 pixels × 4 pixels, but the number of pixels in the block can be arbitrarily set. Hereinafter, a specific operation of the matching processing unit 180 will be described with reference to FIG.

The trajectory in FIG. 10 shows the transition of the correlation value (difference value) in the pattern matching in units of blocks. Thus, if the blocks can be linearly shifted, it is possible to obtain a specific point A where the correlation value is maximized (here, the difference value is minimized). However, in actuality, the movement can only be performed in units of pixels, and the correlation values (here, difference values) in the movements in units of pixels ... _Bn-2 , _Bn-1 , _Bn , _{Bn + 1} , _{Bn + 2} (n is an integer). Can only get. Then, the point with the maximum correlation value (the point with the minimum difference value) is the point B _n .

Here, using the fact that the transition of the correlation value is symmetric with respect to a line extending vertically from the point where the correlation value becomes maximum as shown in FIG. 10, the shift amount in the horizontal direction with a finer resolution. Is derived. For example, the matching processing unit 180 shifts the correlation value between the identified blocks (correlation value at the point B _n at which the correlation value is determined to be the maximum) and one of the identified blocks by a unit pixel shift left and right in the horizontal direction. Based on the two correlation values at that time (correlation values at points B _n−1 and B _{n + 1} ), a matching position having a higher resolution than the pixel unit is derived.

Specifically, the matching processor 180, the correlation value is determined to be maximum, i.e., connection point B _n and the difference value of the difference value is minimized and third point correlation values B _n-1 from the minimum 190 Is generated. Next, a line segment 192 is generated that has the same sign as that of the connection 190 but has the same inclination and a difference value that passes through the second point B _{n + 1} from the minimum. Then, a matching position C corresponding to the intersection 194 is newly derived. Such a matching position C shows a point where the correlation value is almost maximized although it may be slightly different from the point A where the correlation value is truly maximized. Further, the points derived in this way are represented with a resolution of about 8 to 16 times that of the pixel unit. Therefore, more detailed pattern matching is possible.

  In the present embodiment, as described above, since the spatial frequency deviation in the pair of image data is suppressed, the color synthesis unit 174 and the coordinate synthesis unit 178 are particularly effective in such high-resolution pattern matching. To work.

  The parallax deriving unit 182 obtains a parallax with an increased resolution with reference to the image between the blocks specified by the matching processing unit 180 and associates the parallax with the block. Such parallax is used as parallax information for the environment recognition device 130 in the subsequent stage.

(Image processing method)
Hereinafter, specific processing of the image processing apparatus 120 will be described based on the flowcharts of FIGS. 11 to 13. FIG. 11 shows the overall flow related to the interrupt processing when the image processing apparatus 120 receives the luminance image 124, and FIGS. 12 and 13 show individual subroutines therein. Also, here, the processing by the image processing method is performed in the range of 1 to 600 pixels in the horizontal direction of the image and 1 to 200 pixels in the vertical direction with the lower left corner of the luminance image 124 of 600 × 200 pixels as the origin.

  As shown in FIG. 11, when the interruption by the environment recognition method occurs when the luminance image 124 is received, the sensitivity of the luminance image 124 is corrected (S200), the color restoration process (S202), and the coordinate conversion process (S204). ) Is executed. Then, after the pattern matching process (S206) is performed, disparity information is generated (S208). Hereinafter, among the above, the characteristic color restoration process S202 and the coordinate conversion process S204 characteristic in the present embodiment will be described in detail.

(Color restoration processing S202)
Referring to FIG. 12, the color restoration unit 172 initializes (assigns “0”) a vertical variable j for specifying a pixel (S250). Subsequently, the color restoration unit 172 adds (increments) “1” to the vertical variable j and initializes the horizontal variable i (substitutes “0”) (S252). Next, the color restoration unit 172 adds “1” to the horizontal variable i (S254). Here, the reason why the horizontal variable i and the vertical variable j are provided is to execute the color restoration processing for all 600 × 200 pixels.

  Subsequently, the color restoration unit 172 acquires the luminances of the hues R, G, and B in the pixel (i, j) from the luminance image 124 (S256). Then, the color restoration unit 172 determines whether or not the hue R is a pixel that needs to be restored, that is, whether or not the hue R is a pixel that is not set in advance (S258). If the pixel requires color restoration (YES in S258), the color restoration unit 172 performs color restoration (S260). If the pixel does not require color restoration (NO in S258), the color composition unit 174 Colors are synthesized based on the pixels (S262).

  Next, the color restoration unit 172 determines whether or not the hue G is a pixel that needs to be restored (S264). If the pixel requires color restoration (YES in S264), the color restoration unit 172 performs color restoration (S266). If the pixel does not require color restoration (NO in S264), the color composition unit 174 Colors are synthesized based on the pixels (S268). Similarly, the color restoration unit 172 determines whether the hue B is a pixel that needs to be restored (S270). If the pixel requires color restoration (YES in S270), the color restoration unit 172 performs color restoration (S272). If the pixel does not require color restoration (NO in S270), the color synthesis unit 174 Colors are synthesized based on the pixels (S274).

  Next, it is determined whether or not the horizontal variable i exceeds the maximum value (here, 600) of the horizontal pixel (S276). If the horizontal variable i does not exceed the maximum value (NO in S276), the horizontal variable in step S254 is determined. Repeat from i increment processing. If the horizontal variable i exceeds the maximum value (YES in S276), the color restoration unit 172 determines whether or not the vertical variable j exceeds the maximum value (200 in this case) of the vertical pixels (S278). If the vertical variable j does not exceed the maximum value (NO in S278), the process of incrementing the vertical variable j in step S252 is repeated. If the vertical variable j exceeds the maximum value (YES in S278), the color restoration process is terminated.

(Coordinate conversion process S204)
Referring to FIG. 13, the coordinate conversion unit 176 initializes (assigns “0”) a vertical variable j for specifying a pixel (S300). Subsequently, the coordinate conversion unit 176 adds (increments) “1” to the vertical variable j and initializes the horizontal variable i (substitutes “0”) (S302). Next, the coordinate conversion unit 176 adds “1” to the horizontal variable i (S304). Here, the reason why the horizontal variable i and the vertical variable j are provided is to execute the coordinate conversion processing on all 600 × 200 pixels.

  Subsequently, the coordinate conversion unit 176 acquires the luminances of the hues R, G, and B in the pixel (i, j) from the luminance image 124 (S306). Then, the coordinate conversion unit 176 determines whether or not to perform color restoration from the surrounding four pixels or two pixels, that is, whether or not the pixel is determined not to require coordinate conversion (S308). If it is a pixel that performs color restoration (YES in S308), the coordinate conversion unit 176 performs color restoration of each hue of the pixel after coordinate transformation (S310), and if it is a pixel that does not perform color restoration (NO in S308). The coordinate composition unit 178 composes colors based on surrounding pixels (S312).

  Next, it is determined whether or not the horizontal variable i exceeds the maximum value (here, 600) of the horizontal pixel (S314). If the horizontal variable i does not exceed the maximum value (NO in S314), the horizontal variable in step S304 is determined. Repeat from i increment processing. If the horizontal variable i exceeds the maximum value (YES in S314), the coordinate conversion unit 176 determines whether the vertical variable j exceeds the maximum value (200 in this case) of the vertical pixels (S316). If the vertical variable j does not exceed the maximum value (NO in S316), the process of incrementing the vertical variable j in step S302 is repeated. If the vertical variable j exceeds the maximum value (YES in S316), the coordinate conversion process ends.

  As described above, the image processing apparatus 120 described above can appropriately perform pattern matching by suppressing a spatial frequency deviation in a pair of image data to be subjected to pattern matching. Therefore, the parallax information can be appropriately derived, and it is possible to appropriately perform control for avoiding a collision with an object and maintaining a safe distance between the preceding vehicle and the preceding vehicle.

  Also provided are a program that causes a computer to function as the image processing apparatus 120 and a computer-readable storage medium that stores the program, such as a flexible disk, a magneto-optical disk, a ROM, a CD, a DVD, and a BD. Here, the program refers to data processing means described in an arbitrary language or description method.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this embodiment. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Is done.

  For example, in the above-described embodiment, image data acquired simultaneously using two imaging devices 110 is used, but a pair of image data corresponding to time series may be used using one imaging device 110. it can. Further, the number of imaging devices 110 is not limited to 1 or 2, and can be applied to 3 or more compound eyes.

  In the above-described embodiment, the three primary colors RGB are described as the Bayer array. However, four primary colors of RGBC, or five or more different colors can be applied.

  Furthermore, in the above-described embodiment, only the image data acquired via the imaging device 110 is targeted. However, the laser light corresponding to the specific hue described above is acquired using a radio wave laser radar with different frequency filters. It is good. Here, the radio wave laser radar projects a laser beam onto the detection region 122, receives light reflected by the laser beam hitting an object, and analyzes the reflected light. Further, even in the temperature measuring device, the embodiment can be applied to image data for each temperature measuring element.

  The present invention can be used in an image processing apparatus that performs pattern matching based on a pair of captured image data.

DESCRIPTION OF SYMBOLS 1 ... Vehicle 100 ... Environment recognition system 120 ... Image processing apparatus 170 ... Sensitivity correction part 172 ... Color restoration part 174 ... Color composition part 176 ... Coordinate conversion part 178 ... Coordinate composition part 180 ... Matching processing part

Claims (8)

In a pair of image data configured with a Bayer array, a color restoration unit that restores the luminance of a hue not set to each pixel based on adjacent pixels;
In the pair of image data, a color synthesizing unit that synthesizes luminance of a hue set in advance for each pixel based on surrounding pixels;
A matching processing unit that extracts blocks of a predetermined size from each of the pair of image data, performs matching, and identifies highly correlated blocks; and
An image processing apparatus comprising:   The color synthesizing unit synthesizes a luminance of a hue set in advance for each pixel based on surrounding pixels so as to approach a minimum value of a spatial frequency that is reduced by being restored by the color restoration unit. The image processing apparatus according to claim 1, wherein:   3. The image processing apparatus according to claim 1, wherein the color synthesis unit synthesizes the luminance of a pixel in which the luminance of the hue is set in advance and the luminance of the surrounding same hue at a predetermined ratio. . A coordinate conversion unit that performs coordinate conversion of each pixel based on a positional deviation characteristic for each pixel that is predetermined according to the imaging device that generates the pair of image data, and derives the luminance of the pixel after the coordinate conversion;
A coordinate synthesizing unit that synthesizes the luminance of pixels determined not to require coordinate transformation by the coordinate transformation unit based on surrounding pixels;
The image processing apparatus according to claim 1, further comprising: A coordinate conversion unit that performs coordinate conversion of each pixel based on a positional deviation characteristic for each pixel that is predetermined according to an imaging device that generates a pair of image data, and derives the luminance of the pixel after the coordinate conversion;
A coordinate synthesizing unit that synthesizes the luminance of pixels determined not to require coordinate transformation by the coordinate transformation unit based on surrounding pixels;
A matching processing unit that extracts blocks of a predetermined size from each of the pair of image data, performs matching, and identifies highly correlated blocks; and
An image processing apparatus comprising:   The coordinate synthesizing unit synthesizes the luminance of the pixels that are determined not to require coordinate transformation based on surrounding pixels so as to approach the minimum value of the spatial frequency that is lowered by the coordinate transformation performed by the coordinate transformation unit. The image processing apparatus according to claim 5.   The image processing apparatus according to claim 5, wherein the coordinate composition unit synthesizes the luminance of a pixel that is determined not to require coordinate conversion and the luminance of a surrounding same hue at a predetermined ratio.   Based on the correlation value between the identified blocks and the two correlation values when one of the identified blocks is shifted in the horizontal direction by a unit pixel, the matching processing unit has a resolution higher than that of the pixel unit. The image processing apparatus according to claim 1, wherein a high matching position is derived.