JP2013115739A - Inter-image difference device and inter-image difference method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robust inter-image difference technology hard to receive an effect of a disturbance.SOLUTION: An inter-image difference device includes: a gradient direction calculation unit 14a for calculating gradient directions of pixel values at pixel positions of respective pixels constituting an image; a histogram generation unit 14b for setting the respective pixels as target pixels and setting histogram calculation regions around the target pixels, to generate a frequency distribution of the gradient directions at the pixel positions of the pixels included in the histogram calculation regions as a histogram at the pixel positions of the target pixels; a pixel difference unit 14c for calculating differential values between a histogram at pixel positions of difference target pixels of a first image and a histogram at pixel positions of comparison target pixels of a second image, as pixel differential values between the difference target pixels and the comparison target pixels; and an image difference unit 14 for setting pixel values at the pixel positions of the difference target pixels on the basis of the pixel differential values, to generate a difference image.

Description

  The present invention relates to an inter-image difference technique for calculating a difference between two images.

  Difference between images is a method that is very often used in the field of image processing. For example, in the technique of Patent Literature 1, by converting an image captured by a plurality of imaging devices whose imaging ranges partially overlap into a bird's eye view image viewed from a virtual viewpoint, and obtaining a difference between the bird's eye view images. It is also used in techniques for detecting obstacles. In the technique of Patent Document 2, a difference image between a background image in which only the background is photographed and an image in which the background and the moving object are photographed is binarized, and based on this binarized image along the time series. The movement of the moving object is detected.

JP 2008-048317 A JP 2006-059252 A

  In such image processing, in order to detect moving objects and obstacles with high accuracy, a robust inter-image difference technique that is hardly affected by disturbances or the like is required. However, the above-mentioned patent document merely performs a difference between pixel values, and does not mention improvement of robustness in the difference between images.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a robust inter-image difference technique that is hardly affected by disturbance.

  In order to solve the above problems, the inter-image difference device of the present invention sets a gradient direction calculation unit that obtains a gradient direction of a pixel value at a pixel position of each pixel constituting an image, and sets each of the pixels as a target pixel. In addition, a region having a predetermined size is set as a histogram calculation region around the target pixel, and the frequency distribution in the gradient direction at the pixel position of the pixel included in the histogram calculation region is represented as a histogram at the pixel position of the target pixel. A first histogram at the pixel position of the pixel to be compared with the first image, and a second pixel position at the pixel position to be compared with the second image. The difference value from the histogram is calculated as a pixel difference value between the pixel that is the difference target and the pixel that is the comparison target. And element differential unit, and a, and an image difference unit for generating a difference image by setting based on a pixel value at the pixel position of the pixel to be the difference object to said pixel difference values.

  In this configuration, a histogram of the gradient direction of the pixel value at the pixel position (hereinafter referred to as HOG (Histgrams of Oriented Gradients)) is used as the feature amount when calculating the inter-image difference. This HOG represents a frequency distribution in the gradient direction of pixel values at pixel positions of pixels included in a region of a predetermined size including the pixel of interest. The gradient direction of the pixel value indicates the relationship between the pixel value of a certain pixel and the pixel value of a pixel adjacent to the pixel. Therefore, for example, the above-described HOG is substantially the same for an image and an image in which the luminance of the entire image is increased. Therefore, by using HOG as a feature amount, it is possible to perform an inter-image difference that is robust against changes in pixel value and the like.

  In one of the preferred embodiments of the inter-image difference device of the present invention, the image is captured by the first camera and the second camera having a shooting range that partially overlaps the shooting range of the first camera. An image acquisition unit that acquires the captured image, and a viewpoint conversion unit that generates a viewpoint-converted image obtained by converting the captured image captured by the second camera into an image captured from the viewpoint position of the first camera. The first image is the captured image captured by the first camera, and the second image is the viewpoint-converted image.

  In this configuration, an inter-image difference is performed between an image that has not been subjected to viewpoint conversion and an image that has been subjected to viewpoint conversion. When performing viewpoint conversion, pixel value interpolation processing or the like is performed, and therefore, image distortion or the like occurs around the pixels subjected to such processing. Therefore, if an inter-image difference based simply on a luminance value or the like is performed, it is difficult to obtain an accurate difference value at a location where such distortion occurs, which may cause erroneous detection. On the other hand, in the present invention, as described above, HOG is used as a feature amount, so that even when such distortion occurs, stable inter-image difference can be performed.

  In one preferred embodiment of the inter-image difference device of the present invention, an area of a predetermined size is set around the pixel to be the difference target in the second image, and the pixel difference unit By sequentially setting the pixels included in the region as the pixels to be compared, the pixel to be the difference target of the first image and each pixel included in the region of the second image An area difference unit is provided that calculates a pixel difference value and calculates the pixel difference value at the pixel position of the pixel that is the difference target based on each pixel difference value.

  In this configuration, when calculating the pixel difference value for the pixel that is the difference target, only the HOG at the pixel position of the pixel that is the difference target is compared with the HOG at the pixel position of the one pixel that is the comparison target. Instead, the difference value is calculated based on the HOG at the pixel position to be the difference target and the HOG at the pixel position of each pixel included in the area set around the pixel to be the difference target. In other words, the pixel difference value is calculated not by comparing pixel-to-pixel but by comparing pixel-to-region. As a result, even if there is a slight misalignment between two images, a stable difference between images can be performed.

  As described above, if the difference value is calculated by comparing the pixel and the region, the processing speed decreases due to an increase in the amount of calculation, which is not preferable. Therefore, in one preferred embodiment of the inter-image difference device according to the present invention, the region difference unit is configured such that the pixel difference unit causes the pixel to be the difference target of the first image and the region of the second image. When calculating the pixel difference value between each pixel included in the pixel, the calculated pixel difference value between the pixel serving as the difference target and the pixel serving as the comparison target included in the region is equal to or less than a predetermined threshold value. If so, a value indicating that there is no difference as a pixel difference value at a pixel position of the pixel to be the difference target, by canceling the calculation of the pixel difference value between the pixel to be the difference target and the pixel included in the region. Set.

  In this configuration, when the region difference unit calculates each pixel difference value from the difference target pixel and each pixel included in the region set in the second image, the calculated pixel difference value is equal to or less than the threshold value. In this case, the calculation of the subsequent pixel difference value is terminated, and a value indicating that there is no difference is set as the pixel difference value for the difference target pixel. Thereby, calculation of an unnecessary pixel difference value can be avoided, and a calculation amount and a decrease in speed can be suppressed.

  An image in which a monotonous landscape is photographed has many pixels in which the gradient direction cannot be obtained (that is, non-directional), and the non-directional frequency is dominant when HOG is calculated in such an image. In one preferred embodiment of the inter-image difference device of the present invention, in view of such image characteristics, the gradient direction includes a non-directional direction and a plurality of directional directions, and the pixel difference unit includes the first difference unit. When the difference between the non-directional frequency of the histogram and the non-directional frequency of the second histogram is calculated, and the absolute value of the difference is larger than a predetermined threshold, the difference in the directional frequency Is calculated, and a value indicating that there is a difference between the first histogram and the second histogram is set as the pixel difference value.

  In this configuration, when calculating the difference between the HOGs, the difference between the non-directional frequencies is calculated first, and when the absolute value of the difference is larger than the threshold, the subsequent calculation is aborted, and the first histogram is calculated. A value indicating that there is a difference between the second histogram and the second histogram is output as a pixel difference value. Thereby, it is possible to suppress a calculation amount and a decrease in speed.

  In order to solve the above problem, the inter-image difference method of the present invention includes a step of obtaining a gradient direction of a pixel value at each pixel position of the first image, and a gradient of the pixel value at each pixel position of the second image. A step of obtaining a direction, and each pixel constituting the first image is set as a target pixel, and an area of a predetermined size is set as a histogram calculation area around the target pixel, and is included in the histogram calculation area Generating a frequency distribution in the gradient direction at the pixel position of the pixel as a histogram at the pixel position of the target pixel, and setting each pixel constituting the second image as a target pixel, and surrounding the target pixel An area having a predetermined size is set as a histogram calculation area, and the gradient at the pixel position of the pixel included in the histogram calculation area is set. Generating a frequency distribution in the direction as a histogram at the pixel position of the target pixel, and setting each pixel of the first image as a difference target pixel, and a first position at the pixel position of the difference target pixel of the first image The difference value between the histogram and the second histogram at the pixel position of the pixel to be compared with the second image is calculated as a pixel difference value between the pixel to be compared and the pixel to be compared. And a step of generating a difference image by setting a pixel value at a pixel position of the pixel to be differenced based on the pixel difference value. Of course, the above-described additional features of the inter-image difference device can also be applied to such an inter-image difference method.

It is a top view of the vehicle carrying the vehicle periphery imaging device to which the inter-image difference device of the present invention is applied. 1 is a functional block diagram of a vehicle periphery photographing device in Embodiment 1. FIG. 3 is a flowchart illustrating a process flow of the vehicle periphery photographing device according to the first embodiment. 6 is a flowchart showing a flow of processing of difference between images in the first embodiment. It is an example of the viewpoint conversion image which changed the viewpoint of the captured image of the front camera and the captured image of the left side camera into an image captured from the viewpoint position of the front camera. It is an example of the difference image of the viewpoint conversion image of a 1st image and a 2nd image. 3 is an example of a display image in Embodiment 1. FIG. 6 is a functional block diagram of a vehicle periphery photographing device according to a second embodiment. 10 is a flowchart illustrating a flow of processing for calculating a pixel difference value in the second embodiment. It is a flowchart showing the flow of the process of calculation of the pixel difference value in Example 2 which introduced the truncation of calculation.

  Embodiments of the inter-image difference device of the present invention will be described below with reference to the drawings. In this embodiment, the inter-image difference device is applied to a vehicle periphery photographing device. FIG. 1 is a plan view of a vehicle V equipped with the vehicle periphery photographing apparatus of the present embodiment. The vehicle V of the present embodiment includes a front camera F, a rear camera R, a right side camera SR, and a left side camera SL that respectively photograph front, rear, right, and left sides. In addition, when it is not necessary to distinguish between these, the camera C is collectively referred to.

  The camera C is installed so as to be able to photograph a scene around the vehicle V, in particular, an obstacle which is a three-dimensional object existing around the vehicle V. Therefore, the camera C is installed so that its optical axis is slightly below the horizontal.

  In addition, the camera C includes a wide-angle lens, and the shooting ranges of the adjacent cameras C partially overlap each other. In FIG. 1, the photographing range and the region where the photographing range overlaps (hatched portion) are shown together with the camera C. For example, the shooting range of the front camera F and the shooting range of the right side camera SR overlap on the right front side of the vehicle V. That is, both the front camera F and the right side camera SR can photograph the right front of the vehicle V.

  Each combination of the front camera F and the right side camera SR, the right side camera SR and the rear camera R, the rear camera R and the left side camera SL, and the left side camera SL and the front camera F is the first camera ( Hereinafter, they are referred to as a first camera) and a second camera (hereinafter referred to as a second camera). Note that the first camera and the second camera may be interchanged.

  In the present embodiment, the camera C uses an image sensor such as a CCD (charge coupled device) or CIS (CMOS image sensor) to take a two-dimensional image of 15 to 30 frames per second, and converts it into digital data by converting it into digital data. The output digital camera is used.

  Further, as shown in FIG. 2, a display D for performing various displays is provided in the vehicle V. In the present embodiment, the display D also serves as a monitor device for the navigation system. Therefore, the display D has the display part Da and the touch panel Db formed on the display part Da. The display unit Da can be configured by a liquid crystal display, and the touch panel Db is formed by a pressure-sensitive or electrostatic instruction input device that is formed together with the display unit Da and can output a contact position by a finger or the like as location data. can do.

  FIG. 2 is a system configuration diagram of the vehicle periphery photographing apparatus according to the present embodiment. The vehicle periphery photographing apparatus in the present embodiment includes an ECU (electronic control unit) 1 configured with a memory (CPU) and other peripheral circuits with a central processing unit (CPU) as a core. Of course, instead of the CPU, another logic operation processor or logic circuit such as a DSP (digital signal processor) may be used as a core.

  A scene around the vehicle V photographed by the camera C is input to the ECU 1. Further, the ECU 1 has a function of displaying various information, a photographed image photographed by the camera C, an image obtained by processing the photographed image, and the like on the display unit Da, and acquiring information obtained by the driver operating the touch panel Db. ing.

  The ECU 1 is communicably connected to various systems and sensors via an in-vehicle network (not shown). A CAN (controller area network) can be used as the in-vehicle network.

  As shown in the figure, the ECU 1 arbitrarily controls a control unit 10, an image acquisition unit 11 that acquires a captured image captured by the camera C, a preprocessing unit 12 that performs preprocessing on the acquired captured image, and a captured image. A viewpoint conversion unit 13 that generates a viewpoint-converted image obtained by converting the viewpoint to an image captured from the viewpoint, an image difference unit 14 that calculates a difference between the images, and an image synthesis unit that combines the images captured by the cameras C 20. Note that these functional units are realized by cooperation of software and the CPU, but may be realized by using hardware.

  The control unit 10 performs various controls including control of the flow of processing of the entire functional units configured in the ECU 1.

  The image acquisition unit 11 acquires images from each of the front camera F, the rear camera R, the right side camera SR, and the left side camera SL. As described above, since the camera C of the present embodiment outputs a digital moving image as a captured image, the image acquisition unit 11 extracts one frame from the moving image acquired from the camera C and extracts a still image (hereinafter, referred to as a still image). Abbreviated as an image). At this time, it is desirable to extract the same time frame from the moving images of all the cameras C.

  The preprocessing unit 12 performs preprocessing on the acquired image. In the present embodiment, smoothing processing is performed as preprocessing, thereby suppressing erroneous detection of obstacles due to roughness such as asphalt on the road surface. For the smoothing process, a known method such as a Gaussian filter can be used. Further, the pre-processing unit 12 may perform other image processing as necessary.

  The viewpoint conversion unit 13 performs viewpoint conversion of an image and generates a viewpoint converted image. For example, an image photographed by the camera C is converted into an image photographed from a virtual camera having a vertically downward viewpoint above the vehicle V (hereinafter referred to as a top view image), or by one camera C. The viewpoint of the photographed image is converted from the viewpoint position of another camera C to the photographed image. In the latter case, for example, the viewpoint of the image captured by the left side camera SL is converted into an image captured from the viewpoint position of the front camera F. Note that a known method can be used for the viewpoint conversion.

  The image difference unit 14 as the inter-image difference device of the present invention calculates a difference between two images and generates a difference image. The inter-image difference unit 14 performs a difference based on the HOG at each pixel position of the image. Therefore, the image difference unit 14 includes a gradient direction calculation unit 14a that calculates the gradient direction of pixel values at each pixel position of the image, a histogram generation unit 14b that calculates HOG at each pixel position of the image, and each pixel position of the two images. The pixel difference unit 14c that calculates the difference value at the pixel position based on the HOG in FIG.

The gradient direction calculation unit 14a calculates the gradient direction of the pixel value at the pixel position of each pixel constituting the image. In the present embodiment, the luminance value is used as the pixel value, and when the camera C is a color camera, the pre-processing unit 12 performs monochrome conversion on the image. For example, if the pixel value at the pixel position (x, y) is f (x, y), the change amount Δx in the x direction at the pixel position (x, y) is Δx = 2f (x + 1, y) -2f (x− 1, y), the change amount Δy in the y direction can be expressed as Δy = 2f (x, y + 1) −2f (x, y−1), and the gradient angle θ is θ = tan− ¹ (Δy / Δx). Become. However, in this calculation method, it is unknown whether the actual gradient direction is θ or θ + π. Therefore, in this embodiment, θ (−π / 2 <θ <π / 2) when Δx is positive is directed in a gradient direction, and θ (−π when Δx is negative). A value obtained by encoding / 2 ≦ θ ≦ π / 2) in four stages is a gradient direction. Specifically, signs (gradient directions) 0 and 1 are assigned to 0 ≦ θ <π / 4 and π / 4 ≦ θ <π / 2, and −π / 2 <θ ≦ −π / 4 and Gradient directions 6 and 7 are assigned to each of −π / 4 <θ <0. When Δx is negative, (gradient direction + 4) mod 8 is set as the gradient direction. Here, “mod” is an operator for calculating a remainder. As a result, the calculated gradient angle θ is encoded in eight gradient directions. When Δx and Δy are both 0, no direction (= 8) is set. Therefore, in this embodiment, the gradient direction is one of no direction and eight directions (directional).

  The histogram generation unit 14b obtains the HOG at each pixel position of the image based on the gradient direction calculated by the gradient direction calculation unit 14a. Specifically, an area (hereinafter referred to as a histogram generation area) is set around the pixel of interest, the gradient direction at the pixel position of each pixel included in the histogram generation area is acquired, and the frequency for each gradient direction is integrated. To do. This frequency distribution becomes the HOG at the pixel position of the target pixel. By scanning the target pixel, the HOG at each pixel position of the image can be obtained. In the present embodiment, the histogram generation area has a size of 5 × 5 centered on the pixel of interest, but the setting of the histogram generation area can be changed as appropriate.

  The pixel difference unit 14c is a pixel to be compared between the HOG and the second image (hereinafter referred to as a comparison target pixel) at a pixel position of a pixel that is a difference target in the first image (hereinafter referred to as a difference target pixel). The difference value between the difference target pixel and the comparison target pixel (hereinafter referred to as a pixel difference value) is calculated based on the HOG in FIG. Specifically, the difference between the HOG at the pixel position of the difference target pixel of the first image and the HOG at the pixel position of the comparison target pixel of the second image is obtained, and the pixel difference value at the pixel position of the difference target pixel And In this embodiment, the pixel position of the difference target pixel and the pixel position of the comparison target pixel are the same.

と表すことができる。 As the difference between HOGs, the absolute value sum of the frequency differences in each gradient direction can be used. That is, the gradient direction is i (= 0 to 8), the HOG at the pixel position of the difference target pixel in the first image is HOG1, the HOG at the pixel position of the comparison target pixel in the second image is HOG2, and the gradient direction of the HOG If the frequency of i is HOG [i], the pixel difference value d at the pixel position of the difference target pixel is

It can be expressed as.

  The image difference unit 14 obtains pixel difference values at all pixel positions by sequentially setting each pixel of the image as a difference target pixel with respect to the pixel difference unit 14c, and is set according to the pixel difference value. A difference image having pixel values is generated.

  The image synthesizing unit 20 generates a synthesized image obtained by synthesizing images taken by the cameras C.

  The processing flow of the vehicle periphery photographing apparatus in the present embodiment will be described below using the flowchart of FIG. In the following description, only the case where the front camera F is the first camera and the left side camera SL is the second camera will be described, but the same processing is performed for the other cameras C.

  In the present embodiment, the following processing is executed when the driver operates the touch panel Db to explicitly instruct the operation of the vehicle periphery photographing device, but outputs from various sensors and the like mounted on the vehicle V The control unit 10 may determine whether to execute the process based on the above.

  First, when the driver operates the touch panel Db to instruct the operation of the vehicle periphery photographing apparatus, the instruction is acquired by the control unit 10, and the control unit 10 that acquired the instruction displays an image to the image acquisition unit 11. Send instructions to obtain. In contrast, the image acquisition unit 11 acquires images from the first camera and the second camera (# 01). Hereinafter, these images are referred to as a main captured image and a sub-captured image, respectively. The acquired main captured image and sub captured image are stored in a memory (not shown).

  When the image acquisition is completed, the control unit 10 sends an instruction to the viewpoint conversion unit 13 to change the viewpoint of the sub-captured image from the viewpoint of the main camera. In contrast, the viewpoint conversion unit 13 generates a viewpoint conversion image from the sub-captured image by a known method (# 02). The generated viewpoint conversion image is stored in the memory. Note that the viewpoint position of each camera C (the coordinates of the camera origin in the vehicle coordinate system and the optical axis direction) or a conversion table for viewpoint conversion is recorded in advance in a memory or the like and used for viewpoint conversion processing.

  When the generation of the viewpoint conversion image is completed, the control unit 10 sends an instruction to the image difference unit 14 so as to calculate the difference between the main captured image and the viewpoint conversion image. On the other hand, the image difference unit 14 generates a difference image between the main captured image and the viewpoint conversion image and stores it in the memory (# 03).

  FIG. 4 is a flowchart showing the flow of the inter-image difference processing in this embodiment. Here, the two images that perform inter-image differences are referred to as a first image and a second image. In the present embodiment, the main captured image is the first image, and the viewpoint conversion image is the second image. First, the gradient direction calculation unit 14a calculates the gradient direction of the luminance value at each pixel position of the first image and the second image by the method described above, and stores it in the memory in a two-dimensional array format (# 21, # 22). ).

  Next, the histogram generation unit 14b calculates the HOG at each pixel position of the first image and the second image by the above process using the gradient direction at each pixel position calculated by the above process (# 23, # 24). Since the HOG at each pixel position is a one-dimensional array having nine elements, the calculated HOGs at all pixel positions are stored in the memory in a three-dimensional array format.

  The image difference unit 14 sequentially selects one pixel from the first image as a difference target pixel, and sends an instruction to the pixel difference unit 14c to calculate a pixel difference value at the pixel position of the difference target pixel.

  Specifically, first, the image difference unit 14 selects a pixel at a position from the first image as a difference target pixel (# 25). The pixel position of the selected difference target pixel is sent from the image difference unit 14 to the pixel difference unit 14c. As described above, since the pixel position of the difference target pixel and the pixel position of the comparison target pixel are the same in this embodiment, the pixel difference unit 14c that acquired the pixel position of the difference target pixel receives the first image from the memory. The HOG at the pixel position of the difference target pixel and the HOG at the pixel position of the difference target pixel of the second image are acquired, and the pixel difference value d is calculated by the above equation (1) (# 26). The calculated pixel difference value d is sent to the image difference unit 14.

  The image difference unit 14 reserves a two-dimensional array for storing pixel difference values on the memory, and stores the acquired pixel difference value d at a position corresponding to the pixel position of the difference target pixel in the two-dimensional array. Let Thereby, a difference image is generated. The calculated pixel difference value d is not used as it is as the pixel value of the difference image, but normalization or the like may be performed. Further, by comparing the calculated pixel difference value d with a predetermined threshold value, the difference image may be a binary image having a true value and a false value. In this case, the pixel position of the pixel having the true value is different, and the pixel position of the pixel having the false value is not different.

  When the above processing is completed, the image difference unit 14 determines whether or not there is a pixel that has not been processed, that is, is not selected as a difference target pixel (# 27), and if there is an unprocessed pixel (Yes branch of # 27). The process proceeds to # 25 and the above-described process is repeated.

  5A and 5B, an image photographed by the front camera F as the main photographed image and an image photographed by the left side camera SL as the sub-photographed image are photographed from the viewpoint position of the front camera F. 2 shows an example of a viewpoint-converted image obtained by converting the viewpoint to an image. FIG. 6 is a difference image between the image of FIG. 5A and the image of FIG. Note that the viewpoint-converted image in FIG. 5B is obtained by clipping only the portion overlapping with FIG. 5A after alignment with FIG. 5A. As shown in the figure, the main photographed image and the viewpoint conversion image include a white line W and an obstacle A. Further, as is clear from FIG. 5, the white line W in the main captured image and the white line W in the viewpoint conversion image are substantially at the same position. Therefore, when the inter-image difference between the main photographed image and the viewpoint conversion image is taken, a dominant pixel difference value is not calculated in the region corresponding to the white line W as shown in FIG. On the other hand, as shown in FIG. 6, a dominant pixel difference value is calculated in the area corresponding to the obstacle A, particularly in the area corresponding to the upper part of the obstacle A. Therefore, it can be determined whether there is an obstacle around the vehicle V if it is determined whether there is a region having a dominant pixel difference value in the difference image.

  By presenting this difference image to the driver, the driver can determine the presence or absence of obstacles around the vehicle V, but in the present embodiment, the driver's determination becomes easier. A top view image is displayed on the display unit Da, and an index indicating an obstacle is superimposed on the image.

  Therefore, when the difference image is generated, the control unit 10 sends an instruction to the image synthesis unit 20 to generate a top view image and superimpose an index indicating an obstacle on the top view image. On the other hand, the image composition unit 20 uses the viewpoint conversion unit 12 to convert the captured image and the difference image of each camera C into a top view image (# 10). Note that the difference image is generated based on the main captured image, and thus undergoes the same viewpoint conversion processing as the main captured image.

  Further, the image composition unit 20 detects an obstacle from the difference image (# 11). As described above, the obstacle is a region having a dominant pixel value in the difference image. Therefore, if binarization and area division are applied to the difference image, the area of the obstacle can be detected. Further, if edge extraction and binarization are performed on the difference image, the contour of the obstacle can be extracted. The image composition unit 20 detects the obstacle in this way, and composes an index indicating the position of the obstacle with the top view image synthesized by the above-described processing (# 12). The image generated in this way is stored in the memory and displayed on the display unit Da according to an instruction from the control unit 10 (# 13).

  FIG. 7 is an example of a display image displayed on the display unit Da by the above-described processing. In FIG. 7A, an index E representing the contour of the obstacle A is synthesized with the top view image. On the other hand, in FIG.7 (b), the parameter | index E which shows the contact position with the road surface of the obstruction A is synthesize | combined with respect to the top view image. In this example, when the top view image is combined, the overlapping range of the shooting range of the front camera F and the shooting range of the left side camera SL combines both images, so the shape of the obstacle A Is different from the actual shape.

  The vehicle periphery photographing apparatus in the present embodiment is different from the first embodiment in the processing flow of the image difference unit 14. In Example 1, the pixel difference value was calculated by comparing the HOG of the difference target pixel of the first image and the comparison target pixel of the second image. That is, in Example 1, the pixel difference value was calculated by comparing the pixels with each other. In contrast, in this embodiment, the pixel difference value is calculated by comparing the pixel and the region. For this reason, the image difference unit 14 in this embodiment includes an area difference unit 14d as shown in the functional block diagram of FIG.

  The region difference unit 14d sets a region of a predetermined size (hereinafter referred to as a comparison target region) around the difference target pixel in the second image, and each included in the comparison target region with respect to the pixel difference unit 14c. Are sequentially set as comparison target pixels, thereby calculating a pixel difference value between the difference target pixel and each pixel included in the comparison target region. Furthermore, the region difference unit 14d calculates a pixel difference value for the final difference target pixel based on these pixel difference values.

  Below, the flow of the process of the image difference part 14 in a present Example is demonstrated. Note that the overall processing of the image difference unit 14 in the present embodiment is substantially the same as that in the first embodiment, and the process of # 26 for calculating the pixel difference value is replaced with the process of the flowchart of FIG. Therefore, here, the processing for calculating the pixel difference value will be mainly described.

  By the processing from # 21 to # 25, the HOG at each pixel position of the first image and the second image is calculated, and one pixel in the first image is set as the difference target pixel. At this time, the area difference unit 14d sets a comparison target area having a predetermined size around the difference target pixel of the second image (# 31). In this embodiment, the comparison target area is a 5 × 5 area centered on the difference target pixel. That is, when the pixel position of the difference target pixel is (x, y), the comparison target area is a rectangular area having a diagonal line connecting (x-2, y-2) and (x + 2, y + 2). The position of the difference target pixel in the comparison target region and the size of the comparison target region can be changed as appropriate.

  Next, the region difference unit 14d sets one pixel included in the comparison target region as a comparison target pixel (# 32), and the pixel difference unit 14c compares the difference target pixel of the first image and the second image. A pixel difference value from the pixel is calculated (# 33). This process is performed for all pixels included in the comparison target region (Yes branch of # 34).

  With the above processing, the same number of pixel difference values as the pixels included in the comparison target region are obtained. In this embodiment, 25 pixel difference values are obtained. The area difference unit 14d calculates a final pixel difference value for the difference target pixel from the plurality of pixel difference values (# 35). Specifically, if all of the calculated plurality of pixel difference values are larger than a predetermined threshold value, the true value is set as a pixel difference value for the difference target pixel, and there is at least one pixel difference value equal to or smaller than the predetermined threshold value. The false value is the pixel difference value d for the difference target pixel.

When the calculation of the pixel difference value d in the present embodiment is expressed as a formula, the formula (2) is obtained. Here, d _i is the pixel difference value for each calculated in # 33.

  Thus, by calculating the pixel difference value for the difference target pixel, even if the pixel corresponding to the difference target pixel in the second image is misaligned, the pixel is included in the comparison target region. If so, the pixel difference value for the difference target pixel is a false value. Therefore, if the pixel difference value for the difference target pixel is calculated as in the present embodiment, it is possible to perform an inter-image difference that is robust against positional deviation.

  As described above, in this embodiment, the robustness against the positional deviation is improved by calculating a plurality of pixel difference values for one difference target pixel. However, when the comparison target region is set to 5 × 5, the calculation cost for obtaining the pixel difference value for one difference target pixel is 25 times that of the first embodiment in a simple calculation. Such an increase in calculation cost leads to an unfavorable situation such as a decrease in processing speed and a need for a faster CPU. Therefore, it is necessary to increase the speed by reducing the calculation cost. Hereinafter, the speed-up method will be described.

[Speedup method 1]
Here, a description will be given of the speeding up by the truncation of calculation. FIG. 10 is a flowchart showing the flow of a pixel difference value calculation process for a difference target pixel into which a calculation truncation process is introduced. Note that the processing from # 41 to # 43 is the same as the processing from # 31 to # 33, and the description thereof will be omitted.

When the pixel difference value d _i between the difference target pixel and the comparison target pixel is calculated (# 43), the calculated pixel difference value d _i is compared with a predetermined threshold TH (# 43). At this time, when the calculated pixel difference value d _i is equal to or smaller than the threshold value TH (Yes branch of # 44), the logical expression (d _i > TH) in the expression (2) becomes false and is calculated by the expression (2). The pixel difference value d is always a false value regardless of the pixel difference value d _i thereafter. Therefore, when the calculated pixel difference value d _i is equal to or smaller than the threshold value TH (Yes branch of # 44), the pixel difference value of the difference target pixel is set to a false value (# 45), and the process ends. That is, when the threshold value or less of the pixel difference value d _i are calculated, aborting the calculation of the pixel difference values d _i for the pixel included in the comparison pixel. Thereby, unnecessary operations can be reduced and the processing speed can be increased.

On the other hand, if the pixel difference value d _i calculated is greater than the threshold TH (No branch of # 44), the above processing is repeated until the set as a comparison target pixels all the pixels of the comparison regions (# 43 to # 46). When there is no unprocessed pixel (No branch at # 46), the pixel difference value d for the difference target pixel is set to a true value (# 47).

[Another embodiment]
(1) The calculation process in equation (1) may be performed as follows. First, the absolute value | HOG1 [8] −HOG2 [8] | of the non-directional frequency difference is calculated. If this value is larger than a predetermined threshold value TH1, the pixel difference value d is set to a true value, and this value is When the value is equal to or less than the predetermined threshold value, the pixel difference value may be obtained by substituting the value d calculated by Expression (1) into the logical expression (d> TH2). Here, TH2 is a threshold value for determining whether there is a difference between pixels. This method can reduce the amount of calculation for an image having many gradient directions of pixel values that are non-directional, such as an image including many regions with small pixel value changes such as road surfaces. ,preferable.

(2) In the above-described embodiment, the HOG at all pixel positions of the image is calculated and stored in the memory in advance. However, as described above, since a three-dimensional array is required for storing the HOG, it is not preferable to store the HOG in an apparatus having a small memory capacity because it compresses the memory capacity. Therefore, when calculating the pixel difference value, the HOG may be calculated sequentially. However, with this configuration, when comparison is performed in units of regions as in the second embodiment, the calculation of HOG is performed redundantly, the amount of calculation increases, and the speed decreases. In this case, it is desirable to suppress an increase in the amount of calculation by avoiding overlapping calculations as follows.

  First, the HOG at the pixel position of one difference target pixel (hereinafter referred to as the first difference target pixel) is calculated by the processing of the second embodiment, and the comparison target region (hereinafter referred to as the first comparison target) with respect to the first difference target pixel is calculated. Area) is set. Then, the HOG at each pixel position in the first comparison target area is calculated. Each HOG calculated here is stored in the memory in association with each pixel position. When the HOG is calculated in this manner, the pixel difference value for the first difference target pixel is calculated by the method of the second embodiment.

  Next, one difference target pixel (hereinafter referred to as a second difference target pixel) included in the first comparison target region is selected, and the HOG at the pixel position of the second difference target pixel is calculated. In addition, a comparison target region (hereinafter referred to as a second comparison target region) for the second difference target pixel is set.

  Here, the HOG is calculated at each pixel position in the second comparison target region. However, the first comparison target region and the second comparison target region partially overlap, and each of the first comparison target region The HOG at the pixel position is stored in the memory. Therefore, since the HOG at each pixel position in the overlapping region between the first comparison target region and the second comparison target region can be acquired from the memory, no calculation is necessary here. That is, it is only necessary to calculate HOG at each pixel position in a region where the first comparison target region and the second comparison target region do not overlap.

  For example, if the comparison target area is 5 × 5, the pixel position of the first difference target pixel is (x, y), and the pixel position of the second difference target area is (x + 1, y), 25 of the second comparison target area. Among these pixels, 20 pixels are included in the first comparison target region. Therefore, it is sufficient to calculate five HOGs.

(3) In the above-described embodiment, the difference between images is performed using the main captured image as the first image and the viewpoint conversion image as the second image. In Example 1, the result is the same even if these relationships are interchanged. However, in Example 2, the results may be different if these relationships are interchanged. Accordingly, the first difference image is calculated using the main captured image as the second image and the viewpoint converted image as the second image, and the second difference image is calculated using the viewpoint converted image as the first image and the main captured image as the second image. If the value indicates that there is no difference in both the first difference image and the second difference image (false value), the pixel value of the pixel is set as a value indicating that there is no difference, and at least one of them is different. If the value indicates true (true value), the pixel value of the pixel is set to a value (true) indicating that there is a difference. That is, if the pixel values at the coordinate positions (x, y) of the first difference image and the second difference image are f1 (x, y) and f2 (x, y), respectively, the coordinate position (x , Y) has a pixel value of f (x, y) = f1 (x, y) or f2 (f, y)
It becomes. Thus, robustness can be improved by comparing from both directions.

  The present invention can be applied to an inter-image difference technique when performing calculations such as obstacle detection and moving object detection and optical flow detection.

C: camera C1: first camera C2: second camera F: front camera R: rear camera SR: right side camera SL: left side camera 13: viewpoint conversion unit 14: image difference unit 14a: gradient direction calculation unit 14b: histogram Generation unit 14c: pixel difference unit 14d: region difference unit

Claims (6)

A gradient direction calculation unit for obtaining a gradient direction of a pixel value at a pixel position of each pixel constituting the image;
Each of the pixels is set as a target pixel, a region having a predetermined size is set as a histogram calculation region around the target pixel, and the frequency in the gradient direction at the pixel position of the pixel included in the histogram calculation region A histogram generation unit that generates a distribution as a histogram at the pixel position of the target pixel;
The difference value between the first histogram at the pixel position of the pixel that is the difference target of the first image and the second histogram at the pixel position of the pixel that is the comparison target of the second image. A pixel difference unit that calculates a pixel difference value between the pixel to be compared and the pixel to be compared;
An image difference device comprising: an image difference unit that generates a difference image by setting a pixel value at a pixel position of the pixel to be a difference target based on the pixel difference value. An image acquisition unit that acquires a captured image captured by the first camera and a second camera having a capturing range that partially overlaps the capturing range of the first camera;
A viewpoint conversion unit that generates a viewpoint conversion image obtained by converting the captured image captured by the second camera into an image captured from the viewpoint position of the first camera;
The inter-image difference device according to claim 1, wherein the first image is the captured image captured by the first camera, and the second image is the viewpoint conversion image.   A region of a predetermined size is set around the pixel to be the difference target in the second image, and pixels included in the region are sequentially set as the pixel to be compared to the pixel difference unit. Accordingly, the pixel difference value between the pixel to be the difference target of the first image and each pixel included in the region of the second image is calculated, and the pixel difference value is calculated based on the pixel difference value. The inter-image difference apparatus according to claim 1, further comprising an area difference unit that calculates the pixel difference value at a pixel position of a pixel to be a difference target.   The region difference unit is calculated when the pixel difference unit calculates the pixel difference value between the pixel to be the difference target of the first image and each pixel included in the region of the second image. If the pixel difference value between the pixel that is the difference target and the pixel that is the comparison target included in the region is equal to or less than a predetermined threshold, the pixel that is the difference target and the pixel included in the region The inter-image difference apparatus according to claim 3, wherein the calculation of the pixel difference value is terminated and a value indicating that there is no difference is set as the pixel difference value at the pixel position of the pixel to be the difference object. The gradient direction includes a non-directional direction and a plurality of directional directions,
The pixel difference unit calculates a difference between the non-directional frequency of the first histogram and the non-directional frequency of the second histogram, and when the absolute value of the difference is larger than a predetermined threshold value 5. The pixel difference value according to claim 1, wherein a value indicating that the first histogram and the second histogram have a difference is calculated as the pixel difference value without calculating the difference in frequency in the directional direction. The inter-image difference device according to one item. Determining a gradient direction of the pixel value at each pixel position of the first image;
Determining a gradient direction of the pixel value at each pixel position of the second image;
Each pixel constituting the first image is set as a target pixel, a region having a predetermined size is set as a histogram calculation region around the target pixel, and the pixel position of the pixel included in the histogram calculation region is set at the pixel position. Generating a frequency distribution in the gradient direction as a histogram at the pixel position of the target pixel;
Each pixel constituting the second image is set as a target pixel, a region having a predetermined size is set as a histogram calculation region around the target pixel, and the pixel position of the pixel included in the histogram calculation region is set at the pixel position. Generating a frequency distribution in the gradient direction as a histogram at the pixel position of the target pixel;
Each pixel of the first image is set as a difference target pixel, and the first histogram at the pixel position of the difference target pixel of the first image and the pixel position of the pixel to be compared with the second image Calculating a difference value between the second histogram and the pixel to be compared as a pixel difference value between the pixel to be compared and the pixel to be compared;
A difference image is generated by setting a pixel value at a pixel position of the pixel to be the difference based on the pixel difference value.

Images