JP2013115738A - Vehicle periphery imaging apparatus and method of processing vehicle peripheral image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicle periphery imaging technique that can detect an obstacle in the vehicle periphery with high accuracy.SOLUTION: A first camera to image a scene in the periphery of a vehicle and a second camera to image an imaging range partially overlapped with an imaging range of the first camera are mounted on the vehicle. One of the first camera and the second camera is used as a main camera and the other is used as a sub camera. An image taken by the main camera is used as a main taken image, and an image taken by the sub camera is used as a sub taken image. A viewpoint conversion section 13 creates a viewpoint-converted image obtained by allowing the sub taken image to be viewpoint-converted into an image taken from a viewpoint position of the main camera. An image difference section 14 creates a difference image between the main taken image and the viewpoint-converted image as comparison information.

Description

  The present invention relates to a technique for photographing an image around a vehicle, and in particular, a vehicle periphery photographing device that photographs a scene around a vehicle with a plurality of cameras mounted on the vehicle such that parts of the photographing regions overlap each other, and The present invention relates to a method for processing a vehicle periphery image taken by a camera.

  In recent years, a vehicle equipped with a camera for photographing a surrounding scene has been put into practical use. In such a vehicle, the driver can check obstacles and the like around the vehicle by displaying an image photographed by the camera on a display provided in the vehicle interior.

  A technique for detecting an obstacle from an image has been proposed in order to make the driver recognize the presence of an obstacle or the like. For example, in the techniques of Patent Documents 1 to 3, images captured by a plurality of imaging devices with overlapping imaging ranges are converted into bird's-eye view images viewed from a virtual viewpoint, and differences between the bird's-eye view images are obtained. Thus, a difference area (area where an obstacle exists) is detected.

  In the technique of Patent Document 1, when a bird's-eye view image is synthesized, the pixel value of one bird's-eye view image is used for the difference area detected as described above, so that an obstacle in the synthesized bird's-eye view image is displayed. Double display and disappearance of obstacles are prevented.

  On the other hand, in the technique of Patent Document 2, when a camera that better captures an obstacle is selected based on the difference area detected as described above and the bird's-eye view image is synthesized, the selected camera is selected. The bird's-eye view image is preferentially used. Thereby, it is possible to synthesize a bird's eye view image in which the driver can easily see the obstacle.

  On the other hand, in the technique of Patent Document 3, a difference area where an obstacle exists is displayed so as to be distinguished from other areas, thereby making it easier for the driver to visually recognize the obstacle.

JP 2008-048317 A JP 2008-048345 A JP 2008-085710 A

  As described above, in the techniques of Patent Documents 1 to 3, an image captured by an imaging device having overlapping imaging ranges is converted into a bird's eye view image, and obstacles are detected by performing differential processing on the images. However, in order to generate a bird's eye view image, an interpolation process or the like is necessary, and image degradation such as a reduction in sharpness occurs in an area where the interpolation process has been performed. Therefore, if the difference between the bird's-eye view images is calculated, the difference is calculated between the deteriorated images, and there is a possibility that the obstacle detection accuracy is lowered.

  The present invention has been made in view of the above problems, and an object thereof is to provide a vehicle periphery photographing technique capable of detecting an obstacle around the vehicle with high accuracy.

  In order to solve the above-mentioned problems, a vehicle periphery photographing apparatus of the present invention is mounted on a vehicle, a first camera that captures a scene around the vehicle, and a photographing range and a part of the first camera that are mounted on the vehicle. A second camera that captures overlapping shooting ranges, and one of the first camera and the second camera as a main camera and the other as a sub camera, and an image captured by the main camera is a main captured image. A viewpoint conversion unit that generates a viewpoint conversion image obtained by converting the image captured by the sub camera as a sub captured image and converting the sub captured image into an image captured from the viewpoint position of the main camera; An image information comparison unit that generates comparison information comparing the captured image and the viewpoint-converted image.

  In this configuration, a scene around the vehicle is photographed by the first camera and the second camera that partially overlap the photographing range. Therefore, if there are obstacles in the overlapping shooting range, these images are converted into viewpoints taken from the same camera viewpoint position, and comparison information generated by comparing them is generated (for example, difference images). Then, the obstacle is represented as an area having a dominant pixel value in such a difference image. As described above, in Patent Documents 1 to 3, viewpoints of both images are converted into images taken by a virtual camera with the optical axis set vertically downward above the vehicle, and the difference between these images is calculated. Was. In contrast, in the present invention, one image (main photographed image) is not subjected to viewpoint conversion, and only the other image (sub-photographed image) is photographed from the viewpoint of the camera (main camera) that photographed the main photographed image. The viewpoints are converted into images and compared. Therefore, distortion due to viewpoint conversion occurs only in the sub-photographed image, and a decrease in obstacle detection accuracy due to distortion can be suppressed.

  In one preferred embodiment of the vehicle periphery photographing device of the present invention, an edge extraction unit that generates an edge image obtained by extracting an edge from the main photographed image, and a pixel of the difference image and the edge image as the comparison information A product image generating unit that generates a product image obtained by performing a logical product on the values and outputs the product image as an output image.

  In this configuration, an edge image obtained by extracting an edge from a main captured image that is not subjected to viewpoint conversion is generated, and a product image obtained by performing a logical product of the edge image and a difference image as comparison information is generated. Therefore, the region having the dominant pixel value (true value) in the product image is the region having the dominant pixel value in the difference image, and the contour of the region having a clear edge in the main captured image. Become. In other words, in this configuration, the obstacle area that can be detected from the difference image is verified using information called an edge. Thereby, the detection accuracy of an obstacle can be improved.

  In one preferred embodiment of the vehicle periphery photographing apparatus of the present invention, an area dividing unit that divides the main photographed image into regions, each region divided into regions, and a region having a true value in the product image. An obstacle image generated by extracting an obstacle area based on the similarity, and outputting the obstacle image as the output image instead of the product image.

  As described above, the region having a superior value (true value) in the product image is the contour of the obstacle. In this configuration, the obstacle region is extracted based on the similarity between the contour region of the obstacle and the region divided region in the main captured image. In other words, in this configuration, verification is performed on the area that may be the contour of the obstacle included in the product image by using the information about the area feature value of the area division result. Thereby, the detection accuracy of an obstacle can be improved.

  In one preferred embodiment of the vehicle periphery photographing apparatus of the present invention, the output image is generated as the first output image using the first camera as the main camera, and the output is generated using the second camera as the main camera. An image is generated as a second output image, and an image in which a pixel indicating an obstacle is set to a predetermined pixel value based on the first output image and the second output image is output as an output image.

  In this configuration, the first output image obtained by the above processing based on the image captured by the first camera that has not undergone viewpoint conversion and the image captured by the second camera that has undergone viewpoint conversion, and the first camera that has undergone viewpoint conversion. The output image is an image obtained by integrating the image output by the above processing and the second output image obtained by the above processing based on the image captured by the second camera not subjected to viewpoint conversion. In other words, in this configuration, the area of the obstacle included in the output image is double-checked by integrating the two processing results obtained by switching the main captured image and the sub captured image into an output image. Thereby, the erroneous detection of the obstacle resulting from the distortion caused by the viewpoint conversion can be reduced, and the obstacle detection accuracy can be improved.

  In order to solve the above-described problem, a vehicle peripheral image processing method according to the present invention is mounted on a vehicle, and acquires a captured image from a first camera that captures a scene around the vehicle, and is mounted on the vehicle. A step of acquiring a captured image from a second camera that captures an imaging range that partially overlaps the imaging range of the first camera, and using one of the first camera and the second camera as a main camera The other camera is the secondary camera, the captured image captured by the primary camera is the primary captured image, the captured image captured by the secondary camera is the secondary captured image, and the secondary captured image is the viewpoint position of the primary camera. Generating a viewpoint-converted image obtained by converting the viewpoint to an image photographed from, and generating comparison information comparing the main photographed image and the viewpoint-converted image. Eteiru. Of course, the above-described additional features of the vehicle periphery image capturing device can also be applied to such a vehicle periphery image processing method.

It is a top view of the vehicle carrying the vehicle periphery imaging device of this invention. 1 is a functional block diagram of a vehicle periphery photographing device in Embodiment 1. FIG. 3 is a flowchart illustrating a process flow of the vehicle periphery photographing device according to the first embodiment. It is a flowchart showing the flow of a process of the difference between images. It is an example of the viewpoint conversion image which changed the viewpoint of the captured image of the front camera and the captured image of the left side camera into an image captured from the viewpoint position of the front camera. It is an example of the difference image of the viewpoint conversion image of a 1st image and a 2nd image. 3 is an example of a display image in Embodiment 1. FIG. 6 is a functional block diagram of a vehicle periphery photographing device according to a second embodiment. 6 is a flowchart illustrating a process flow of a vehicle periphery photographing apparatus according to a second embodiment. It is an example of the product image produced | generated from the edge image of a 1st image, and the difference image of FIG. 10 is an example of a display image in Embodiment 2. FIG. 6 is a functional block diagram of a vehicle periphery photographing device in Embodiment 3. 10 is a flowchart illustrating a process flow of the vehicle periphery photographing device according to the third embodiment. 10 is a flowchart showing a flow of processing of an obstacle image generation unit in Embodiment 3. FIG. 6 schematically illustrates a process flow when the process of the fourth embodiment is applied to the first embodiment. FIG. FIG. 10 schematically illustrates a process flow when the process of the fourth embodiment is applied to the second embodiment. FIG. FIG. 10 schematically illustrates a process flow when the process of the fourth embodiment is applied to the third embodiment. FIG.

  Hereinafter, embodiments of a vehicle periphery photographing apparatus of the present invention will be described with reference to the drawings. FIG. 1 is a plan view of a vehicle V equipped with a vehicle periphery photographing apparatus of the present invention. The vehicle V of the present embodiment includes a front camera F, a rear camera R, a right side camera SR, and a left side camera SL that respectively photograph front, rear, right, and left sides. In addition, when it is not necessary to distinguish between these, the camera C is collectively referred to.

  The camera C is installed so as to be able to photograph a scene around the vehicle V, in particular, an obstacle which is a three-dimensional object existing around the vehicle V. Therefore, the camera C is installed so that its optical axis is slightly below the horizontal.

  In addition, the camera C includes a wide-angle lens, and the shooting ranges of the adjacent cameras C partially overlap each other. In FIG. 1, the photographing range and the region where the photographing range overlaps (hatched portion) are shown together with the camera C. For example, the shooting range of the front camera F and the shooting range of the right side camera SR overlap on the right front side of the vehicle V. That is, both the front camera F and the right side camera SR can photograph the right front of the vehicle V.

  Each combination of the front camera F and the right side camera SR, the right side camera SR and the rear camera R, the rear camera R and the left side camera SL, and the left side camera SL and the front camera F is the first camera and the first camera of the present invention. 2 cameras. Note that the first camera and the second camera may be interchanged.

  In the present embodiment, the camera C uses an image sensor such as a CCD (charge coupled device) or CIS (CMOS image sensor) to take a two-dimensional image of 15 to 30 frames per second, and converts it into digital data by converting it into digital data. The output digital camera is used.

  Further, as shown in FIG. 2, a display D for performing various displays is provided in the vehicle V. In the present embodiment, the display D also serves as a monitor device for the navigation system. Therefore, the display D has the display part Da and the touch panel Db formed on the display part Da. The display unit Da can be configured by a liquid crystal display, and the touch panel Db is formed by a pressure-sensitive or electrostatic instruction input device that is formed together with the display unit Da and can output a contact position by a finger or the like as location data. can do.

  FIG. 2 is a system configuration diagram of the vehicle periphery photographing apparatus according to the present embodiment. The vehicle periphery photographing apparatus in the present embodiment includes an ECU (electronic control unit) 1 configured with a memory (CPU) and other peripheral circuits with a central processing unit (CPU) as a core. Of course, instead of the CPU, another logic operation processor or logic circuit such as a DSP (digital signal processor) may be used as a core.

  A scene around the vehicle V photographed by the camera C is input to the ECU 1. Further, the ECU 1 has a function of displaying various information, a photographed image photographed by the camera C, an image obtained by processing the photographed image, and the like on the display unit Da, and acquiring information obtained by the driver operating the touch panel Db. ing.

  The ECU 1 is communicably connected to various systems and sensors via an in-vehicle network (not shown). A CAN (controller area network) can be used as the in-vehicle network.

  As shown in the figure, the ECU 1 arbitrarily controls a control unit 10, an image acquisition unit 11 that acquires a captured image captured by the camera C, a preprocessing unit 12 that performs preprocessing on the acquired captured image, and a captured image. A viewpoint conversion unit 13 that generates a viewpoint-converted image obtained by converting the viewpoint to an image captured from the viewpoint, an image difference unit 14 that calculates a difference between images (an example of the image information comparison unit of the present invention), and each camera C And an image synthesis unit 20 that synthesizes the images photographed by. Note that these functional units are realized by cooperation of software and the CPU, but may be realized by using hardware.

  The control unit 10 performs various controls including control of the flow of processing of the entire functional units configured in the ECU 1.

  The image acquisition unit 11 acquires images from each of the front camera F, the rear camera R, the right side camera SR, and the left side camera SL. As described above, since the camera C of the present embodiment outputs a digital moving image as a captured image, the image acquisition unit 11 extracts one frame from the moving image acquired from the camera C and extracts a still image (hereinafter, referred to as a still image). Abbreviated as an image). At this time, it is desirable to extract the same time frame from the moving images of all the cameras C.

  The preprocessing unit 12 performs preprocessing on the acquired image. In the present embodiment, smoothing processing is performed as preprocessing, thereby suppressing erroneous detection of obstacles due to roughness such as asphalt on the road surface. For the smoothing process, a known method such as a Gaussian filter can be used. Further, the pre-processing unit 12 may perform other image processing as necessary.

  The viewpoint conversion unit 13 performs viewpoint conversion of an image and generates a viewpoint converted image. For example, an image photographed by the camera C is converted into an image photographed from a virtual camera having a vertically downward viewpoint above the vehicle V (hereinafter referred to as a top view image), or by one camera C. The viewpoint of the photographed image is converted from the viewpoint position of another camera C to the photographed image. In the latter case, for example, the viewpoint of the image captured by the left side camera SL is converted into an image captured from the viewpoint position of the front camera F. Note that a known method can be used for the viewpoint conversion.

  The image difference unit 14 calculates a difference between two images and generates a difference image (an example of a comparative image of the present invention). The difference method can use pixel values, in particular, absolute values of luminance value differences, normalized autocorrelation, and the like. In this embodiment, a histogram of pixel value gradient directions at each pixel position (hereinafter referred to as HOG). (Referred to as Histgrams of Oriented Gradients). Therefore, the image difference unit 14 includes a gradient direction calculation unit 14a that calculates the gradient direction of pixel values at each pixel position of the image, a histogram generation unit 14b that calculates HOG at each pixel position of the image, and each pixel position of the two images. The pixel difference unit 14c that calculates the difference value at the pixel position based on the HOG in FIG. In the present embodiment, the difference image is used as the comparison information, but the comparison information is not limited to this. For example, the luminance similarity (NCC, SAD, etc.) in the surrounding area of the target pixel can be used as the comparison information.

The gradient direction calculation unit 14a calculates the gradient direction of the pixel value at the pixel position of each pixel constituting the image. In the present embodiment, the luminance value is used as the pixel value, and when the camera C is a color camera, the pre-processing unit 12 performs monochrome conversion on the image. For example, if the pixel value at the pixel position (x, y) is f (x, y), the change amount Δx in the x direction at the pixel position (x, y) is Δx = 2f (x + 1, y) -2f (x− 1, y), the change amount Δy in the y direction can be expressed as Δy = 2f (x, y + 1) −2f (x, y−1), and the gradient angle θ is θ = tan− ¹ (Δy / Δx). Become. However, in this calculation method, it is unknown whether the actual gradient direction is θ or θ + π. Therefore, in this embodiment, θ (−π / 2 <θ <π / 2) when Δx is positive is directed in a gradient direction, and θ (−π when Δx is negative). A value obtained by encoding / 2 ≦ θ ≦ π / 2) in four stages is a gradient direction. Specifically, signs (gradient directions) 0 and 1 are assigned to 0 ≦ θ <π / 4 and π / 4 ≦ θ <π / 2, and −π / 2 <θ ≦ −π / 4 and Gradient directions 6 and 7 are assigned to each of −π / 4 <θ <0. When Δx is negative, (gradient direction + 4) mod 8 is set as the gradient direction. Here, “mod” is an operator for calculating a remainder. As a result, the calculated gradient angle θ is encoded in eight gradient directions. When Δx and Δy are both 0, no direction (= 8) is set. Therefore, in this embodiment, the gradient direction is one of no direction and eight directions (directional).

  The histogram generation unit 14b obtains the HOG at each pixel position of the image based on the gradient direction calculated by the gradient direction calculation unit 14a. Specifically, an area (hereinafter referred to as a histogram generation area) is set around the pixel of interest, the gradient direction at the pixel position of each pixel included in the histogram generation area is acquired, and the frequency for each gradient direction is integrated. To do. This frequency distribution becomes the HOG at the pixel position of the target pixel. By scanning the target pixel, the HOG at each pixel position of the image can be obtained. In the present embodiment, the histogram generation area has a size of 5 × 5 centered on the pixel of interest, but the setting of the histogram generation area can be changed as appropriate.

と表すことができる。 The pixel difference unit 14c calculates a difference value (hereinafter referred to as a pixel difference value) at the pixel position based on the HOG at each pixel position of the two images. Specifically, the difference between the HOG at the pixel position of the pixel that is the difference target of the first image (hereinafter referred to as the difference target pixel) and the HOG at the pixel position of the difference target pixel of the second image is obtained. The pixel difference value at the pixel position of the difference target pixel is used. The difference between the HOGs can be the absolute value sum of the frequency differences in each gradient direction. That is, the gradient direction is i (= 0 to 8), the HOGs at the pixel positions of the difference target pixels of the first image and the second image are HOG1 and HOG2, respectively, and the frequency of the HOG gradient direction i is HOG [i]. Then, the pixel difference value d at the pixel position of the difference target pixel is

It can be expressed as.

  The image difference unit 14 sequentially sets each pixel of the image as a difference target pixel with respect to the pixel difference unit 14c, thereby acquiring pixel difference values at all pixel positions and obtaining a difference between the two images.

  The image synthesizing unit 20 generates a synthesized image obtained by synthesizing images taken by the cameras C.

  The processing flow of the vehicle periphery photographing apparatus in the present embodiment will be described below using the flowchart of FIG. In the following description, only the case where the front camera F is the first camera and the main camera of the present invention and the left side camera SL is the second camera and the sub camera of the present invention will be described. Similar processing is performed.

  In the present embodiment, the following processing is executed when the driver operates the touch panel Db to explicitly instruct the operation of the vehicle periphery photographing device, but outputs from various sensors and the like mounted on the vehicle V The control unit 10 may determine whether to execute the process based on the above.

  First, when the driver operates the touch panel Db to instruct the operation of the vehicle periphery photographing apparatus, the instruction is acquired by the control unit 10, and the control unit 10 that acquired the instruction displays an image to the image acquisition unit 11. Send instructions to obtain. On the other hand, the image acquisition unit 11 acquires images from the main camera and the sub camera (# 01). The acquired images are stored in a memory (not shown) as a main captured image and a sub captured image, respectively.

  When the image acquisition is completed, the control unit 10 sends an instruction to the viewpoint conversion unit 13 to change the viewpoint of the sub-captured image from the viewpoint of the main camera. In contrast, the viewpoint conversion unit 13 generates a viewpoint conversion image from the sub-captured image by a known method (# 02). The generated viewpoint conversion image is stored in the memory. Note that the viewpoint position of each camera C (the coordinates of the camera origin in the vehicle coordinate system and the optical axis direction) or a conversion table for viewpoint conversion is recorded in advance in a memory or the like and used for viewpoint conversion processing.

  When the generation of the viewpoint conversion image is completed, the control unit 10 sends an instruction to the image difference unit 14 so as to calculate the difference between the main captured image and the viewpoint conversion image. On the other hand, the image difference unit 14 calculates the difference between the main captured image and the viewpoint conversion image (# 03). In the present embodiment, the difference result between the main captured image and the viewpoint conversion image is output as a difference image having a pixel difference value at each pixel position as a pixel value and stored in the memory.

  FIG. 4 is a flowchart showing the flow of the inter-image difference processing in this embodiment. Here, the two images that perform the inter-image difference are referred to as a first image and a second image. First, the gradient direction calculation unit 14a calculates the gradient direction of the luminance value at each pixel position of the first image and the second image by the method described above, and stores it in the memory in a two-dimensional array format (# 21, # 22). ).

  Next, the histogram generation unit 14b calculates the HOG at each pixel position of the first image and the second image by the above process using the gradient direction at each pixel position calculated by the above process (# 23, # 24). Since the HOG at each pixel position is a one-dimensional array having nine elements, the calculated HOGs at all pixel positions are stored in the memory in a three-dimensional array format.

  The image difference unit 14 sequentially selects one pixel from the first image as a difference target pixel, and sends an instruction to the pixel difference unit 14c to calculate a pixel difference value at the pixel position of the difference target pixel.

  Specifically, first, the image difference unit 14 selects a pixel at a position from the first image as a difference target pixel (# 25). The pixel position of the selected difference target pixel is sent from the image difference unit 14 to the pixel difference unit 14c. The pixel difference unit 14c that has acquired the pixel position of the difference target pixel acquires the HOG at the pixel position of the difference target pixel of the first image and the HOG at the pixel position of the difference target pixel of the second image from the memory, and calculates the above equation. The pixel difference value d is calculated from (1) (# 26). The calculated pixel difference value d is sent to the image difference unit 14.

  The image difference unit 14 reserves a two-dimensional array for storing pixel difference values on the memory, and stores the acquired pixel difference value d at a position corresponding to the pixel position of the difference target pixel in the two-dimensional array. Let

  When the above processing is completed, the image difference unit 14 determines whether or not there is a pixel that has not been processed, that is, is not selected as a difference target pixel (# 27), and if there is an unprocessed pixel (Yes branch of # 27). The process proceeds to # 25 and the above-described process is repeated.

  5A and 5B, an image photographed by the front camera F as the main photographed image and an image photographed by the left side camera SL as the sub-photographed image are photographed from the viewpoint position of the front camera F. 2 shows an example of a viewpoint-converted image obtained by converting the viewpoint to an image. FIG. 6 is a difference image between the image of FIG. 5A and the image of FIG. Note that the viewpoint-converted image in FIG. 5B is obtained by clipping only the portion overlapping with FIG. 5A after alignment with FIG. 5A. As shown in the figure, the main photographed image and the viewpoint conversion image include a white line W and an obstacle A. Further, as is clear from FIG. 5, the white line W in the main captured image and the white line W in the viewpoint conversion image are substantially at the same position. Therefore, when the inter-image difference between the main photographed image and the viewpoint conversion image is taken, a dominant pixel difference value is not calculated in the region corresponding to the white line W as shown in FIG. On the other hand, as shown in FIG. 6, a dominant pixel difference value is calculated in the area corresponding to the obstacle A, particularly in the area corresponding to the upper part of the obstacle A. Therefore, it can be determined whether there is an obstacle around the vehicle V if it is determined whether there is a region having a dominant pixel difference value in the difference image.

  By presenting this difference image to the driver, the driver can determine the presence or absence of obstacles around the vehicle V, but in the present embodiment, the driver's determination becomes easier. A top view image is displayed on the display unit Da, and an index indicating an obstacle is superimposed on the image.

  Therefore, when the difference image is generated, the control unit 10 sends an instruction to the image synthesis unit 20 to generate a top view image and superimpose an index indicating an obstacle on the top view image. On the other hand, the image composition unit 20 uses the viewpoint conversion unit 12 to convert the captured image and the difference image of each camera C into a top view image (# 10). Note that the difference image is generated based on the main captured image, and thus undergoes the same viewpoint conversion processing as the main captured image.

  Further, the image composition unit 20 detects an obstacle from the difference image (# 11). As described above, the obstacle is a region having a dominant pixel value in the difference image. Therefore, if binarization and area division are applied to the difference image, the area of the obstacle can be detected. Further, if edge extraction and binarization are performed on the difference image, the contour of the obstacle can be extracted. The image composition unit 20 detects the obstacle in this way, and composes an index indicating the position of the obstacle with the top view image synthesized by the above-described processing (# 12). The image generated in this way is stored in the memory and displayed on the display unit Da according to an instruction from the control unit 10 (# 13).

  FIG. 7 is an example of a display image displayed on the display unit Da by the above-described processing. In FIG. 7A, an index E representing the contour of the obstacle A is synthesized with the top view image. On the other hand, in FIG.7 (b), the parameter | index E which shows the contact position with the road surface of the obstruction A is synthesize | combined with respect to the top view image. In this example, when the top view image is combined, the overlapping range of the shooting range of the front camera F and the shooting range of the left side camera SL combines both images, so the shape of the obstacle A Is different from the actual shape.

  FIG. 8 is a functional block diagram of the vehicle periphery photographing apparatus in the present embodiment. In addition, the same code | symbol as Example 1 is attached | subjected to the function part similar to Example 1, and detailed description is abbreviate | omitted. The vehicle periphery photographing apparatus according to the present embodiment is provided with an edge extraction unit 15 that extracts an edge of an image, and a product image generation unit 16 that generates a product image obtained by performing a logical product on two images. 1 and different.

  The edge extraction unit 15 performs edge extraction from the image. For extracting the edge, a known method such as the Canny algorithm or the Sobel operator can be used. Note that when the Sobel operator is used, edges are detected on both sides of the actual edge, so it is desirable to perform expansion processing and thinning processing.

  The product image generation unit 16 calculates a logical product of pixel values at corresponding pixel positions in the two images, and generates a product image having the logical product value as a pixel value. The logical product in the present invention is an extension of the general concept of logical product, and both the first pixel value and the second pixel value have superior values (for example, values larger than the threshold value). These logical products have superior values (either fixed values or larger values than the predetermined value), and in other cases, these logical products are inferior ( For example, the value is less than or equal to the threshold value. Therefore, if both images are binary images, the logical product in the present invention is a general and operator.

  On the other hand, when the image is multi-valued, the logical product can be calculated as follows. For example, if the true value is 1 (dominant value), the false value is 0 (inferior value), the first pixel value is p1, and the second pixel value is p2, the first pixel value p1 and the second pixel value Can be the result of a logical operation of (p1> TH1) & (p2> TH2). In this case, only when the first pixel value p1 is larger than the threshold value TH1 and the second pixel value p2 is larger than the threshold value TH2, that is, both the pixel values are dominant values, The result of the operation is a true value. In this case, the product image is a binary image. Note that the threshold value TH1 and the threshold value TH2 may be the same value or different values. Further, the threshold value TH1 and the threshold value TH2 may be fixed values, or may be values determined according to the distribution of pixel values of the image by a method such as Otsu's discriminant analysis method.

  Thus, by extending the concept of logical product, logical product between images can be calculated without binarizing the images in advance. This eliminates the need for a memory for storing binary images, which is advantageous when the memory capacity of the vehicle periphery photographing apparatus is small.

  FIG. 9 is a flowchart showing the processing flow of the image peripheral photographing apparatus in the present embodiment. Since the processes from # 01 to # 03 and # 10 to # 13 are the same as those in the first embodiment, detailed description thereof is omitted.

  When the generation of the difference image is completed, the control unit 10 sends an instruction to the edge extraction unit 15 to perform edge extraction of the main captured image. In response to this instruction, the edge extraction unit 15 applies the Canny algorithm to the main captured image to generate an edge image (# 04). In the present embodiment, the edge extraction unit 15 performs an expansion process on the generated edge image. The edge image (binary image) generated in this way is stored in the memory. In the edge image in this embodiment, the pixel value of the edge pixel is 1 (dominant value), and the pixel value of the non-edge pixel is 0 (inferior value).

  When the generation of the edge image is completed, the control unit 10 sends an instruction to the product image generation unit 16 to generate a product image from the difference image and the edge image of the main captured image. Upon receiving this instruction, the product image generation unit 16 generates a product image by the above-described processing (# 05). In this embodiment, since the difference image is a multi-valued image and the edge image is a binary image, if the pixel value of the difference image is p1 and the pixel value of the edge image is p2, the logical operation formula is (p1> TH1 ) & P2. The product image generated in this way is stored in the memory.

  When the generation of the product image is completed, the control unit 10 instructs the image composition unit 20 to generate a composite image, and displays the generated composite image on the display unit Da (# 10 to # 13). In the first embodiment, an obstacle is detected from the difference image in the process of # 11. However, in this embodiment, instead of extracting the obstacle area from the difference image, an area having a true value is obstructed in the product image. This is the object area.

  FIG. 10A shows an example of an edge image of the first image. The edge image includes an outline of an obstacle in the main captured image and a contour such as a white line on the road surface. FIG. 10B is an example of a product image generated from this edge image and the difference image of FIG. As is apparent from the figure, the product image does not include an area such as a white line, and the contour shape of the obstacle that has not been accurately detected in the difference image is detected more accurately. Therefore, if the contour shape of the obstacle is used, a top view image obtained by synthesizing the index E indicating the exact contour shape of the obstacle A can be synthesized as shown in FIG.

  In this embodiment, the area of the obstacle is thus extracted based on the product image of the edge of the main captured image and the difference image. Therefore, according to the present embodiment, it is possible to exclude an area of an obstacle that is erroneously detected in the first embodiment and is not near the outline of the obstacle. Therefore, the obstacle area can be extracted with higher accuracy than in the first embodiment.

  FIG. 12 is a functional block diagram of the image peripheral photographing apparatus in the present embodiment. In addition, the same code | symbol as Example 2 is attached | subjected to the function part similar to Example 2, and detailed description is abbreviate | omitted. The vehicle periphery photographing apparatus of the present embodiment includes an area dividing unit 17 and an obstacle image generating unit 18 that perform an area dividing process on an image.

  The area dividing unit 17 performs an area dividing process based on the pixel value of the image (which may be a color value such as an RGB value or a luminance value) to generate a labeled image. Note that a labeling image is an image in which the same pixel value (label value) is given to pixels that are determined to belong to the same region in the region division processing. For area division, a known method such as graph cut or Mean-Shift can be used.

  The obstacle image generation unit 18 generates an obstacle image obtained by extracting the obstacle area based on the similarity between each area divided and the area having the true value in the product image. An obstacle image is an image in which the pixel value of a pixel belonging to a region determined to be an obstacle is a true value.

  As described above, the region having the true value in the product image represents the contour shape of the obstacle. Therefore, the obstacle image generation unit 18 obtains an obstacle area based on the similarity between each area divided and the contour represented in the product image. Note that various measures of similarity between the region and the contour can be used. For example, the consistency between the contour of the comparison target and the contour of the comparison target region, the consistency between the region surrounded by the comparison target contour and the comparison target region, or the like can be used.

  FIG. 13 is a flowchart showing the processing flow of the image peripheral photographing apparatus in the present embodiment. Since the processes from # 01 to # 05 and # 10 to # 13 are the same as those in the second embodiment, detailed description thereof is omitted.

  When the generation of the product image is completed, the control unit 10 sends an instruction to the region dividing unit 17 to divide the main captured image and the product image into regions. Upon receiving the instruction, the area dividing unit 17 divides the main captured image into areas and generates a first labeling image (# 06). Further, the product image is divided into regions to generate a second labeling image (# 07). These labeling images are stored in a memory.

  When the region division is completed, the control unit 10 sends an instruction to the obstacle image generation unit 18 to generate an obstacle image. Upon receiving the instruction, the obstacle image generation unit 18 generates an obstacle image and stores it in the memory (# 08).

  FIG. 14 is a flowchart showing the flow of the obstacle image generation process. First, one area (hereinafter referred to as the first area) is selected from the first labeling image (# 31). Also, one area (hereinafter referred to as the second area) is selected from the second labeling image (# 32). Then, the similarity between the first area and the second area is calculated (# 33). At this time, the feature amounts of the first region and the second region are calculated according to the similarity measure to be used. Note that the feature amount of the area may be obtained in advance and stored in the memory.

  If the calculated similarity is larger than a predetermined threshold (Yes branch of # 34), the first area is set as the obstacle area, and the first area of the two-dimensional array for the obstacle area image secured on the memory A true value is substituted for the pixel value corresponding to (# 35).

  If there is an unprocessed area in the second labeling image (Yes branch of # 36), the process is shifted to # 32 and the above process is repeated. On the other hand, if there is no unprocessed area in the second labeling image (No branch of # 36), it is determined whether or not there is an unprocessed area in the first labeling image (# 37). If there is an area (Yes branch of # 37), the process is shifted to # 31, and the above process is repeated.

  In this embodiment, as described above, the area of the obstacle is based on the similarity between each area of the main captured image and the area having the true value in the product image, that is, the area extracted as the obstacle in the second embodiment. Is extracted. Therefore, according to the present embodiment, it is possible to exclude an area of an obstacle erroneously detected in the second embodiment and having a shape or the like that does not match the area of the main captured image. Therefore, the obstacle area can be extracted with higher accuracy than in the second embodiment.

  In the first to third embodiments, the first camera (front camera F in the above embodiment) is the main camera and the second camera (left side camera SL in the above embodiment) is the sub camera. went. In contrast, in this embodiment, the processing result (output image) in which the first camera is the main camera and the second camera is the sub camera, and the processing result in which the first camera is the sub camera and the second camera is the main camera. The result of integration with (output image) is output. Note that the integration of the processing results is excluded from the processing results, for example, if the obstacle area is not located at the corresponding position in both processing results. As a result, the likelihood of the obstacle region as an obstacle is double-checked, and the extraction accuracy can be improved. The correspondence relationship between the positions of the obstacle regions in both processing results can be verified based on the postures of the main camera and the sub camera. Further, due to errors such as viewpoint conversion, there is a possibility that the area of the obstacle in both processing results may not be a completely corresponding position. In that case, it is desirable to use the area of the obstacle on the side close to the vehicle V.

[Application to Example 1]
FIG. 15 is a diagram schematically showing the flow of processing when the processing of the present embodiment is applied to the processing of the first embodiment. First, as in the first embodiment, a difference image (hereinafter referred to as a first difference image) is generated using the first camera as a main camera and the second camera as a sub camera (# 02 to # 03). Also, a difference image (hereinafter referred to as a second difference image) is generated using the second camera as the main camera and the first camera as the sub camera (# 02 ′ to # 03 ′). Then, the first difference image and the second difference image are integrated to generate a difference image as an output image.

[Application to Example 2]
In the process of the second embodiment, the difference image (output image) generated by the process of FIG. 15 may be used instead of the difference image generated by the process of the first embodiment. Moreover, you may perform the process typically shown in FIG. First, similarly to the second embodiment, a product image (hereinafter referred to as a first product image) is generated using the first camera as a main camera and the second camera as a sub camera (# 02 to # 05). Also, product images (hereinafter referred to as second product images) are generated using the second camera as the main camera and the first camera as the sub camera (# 02 ′ to # 05 ′). Then, the first product image and the second product image are integrated to generate a product image as an output image. Further, the difference image generation process in the process of FIG. 16 may be replaced with the process of FIG.

[Application to Example 3]
In the processing of the third embodiment, the product image (output image) generated by the processing of FIG. 16 may be used instead of the product image generated by the processing of the second embodiment. Moreover, you may perform the process typically shown in FIG. First, as in the third embodiment, an obstacle image (hereinafter referred to as a first obstacle image) is generated using the first camera as a main camera and the second camera as a sub camera (# 02 to # 08). Also, an obstacle image (hereinafter referred to as a second obstacle image) is generated using the second camera as the main camera and the first camera as the sub camera (# 02 ′ to # 08 ′). Then, the first obstacle image and the second obstacle image are integrated to generate an obstacle image as an output image. Also, the difference image generation process in the process of FIG. 17 may be replaced with the process of FIG. 15, or the product image generation process may be replaced with the process of FIG.

[Another embodiment]
(1) In the above-described embodiment, the top view image obtained by synthesizing the captured images of the four cameras C is generated and displayed on the display. However, the display using the captured images of some cameras C is performed. It does not matter. In this case, the camera to be used can be selected based on the outputs of various sensors and the like mounted on the vehicle V. For example, the traveling direction of the vehicle V is determined based on the output of the steering sensor that detects the steering angle of the steering and the output of the shift lever sensor that detects the operation position of the shift lever, and the captured image of the camera C on the traveling direction side. It can be set as the structure which displays using this. Specifically, when the steering angle of the steering sensor is the left direction and the output of the shift lever sensor is reverse (= R), the traveling direction of the vehicle V is the left rear, so the left side camera SL and the rear Display using a photographed image with the camera R is performed.

(2) Moreover, you may display the image image | photographed with each camera C as it is, without combining a picked-up image. In this case as well, it is desirable to superimpose an index E indicating an obstacle.

  The present invention can be applied to a vehicle periphery photographing technique for photographing a scene around a vehicle using a plurality of cameras mounted on the vehicle such that a part of the photographing region overlaps each other.

C: Camera F: Front camera R: Rear camera SR: Right side camera SL: Left side camera 13: Viewpoint conversion unit 14: Image difference unit (image information comparison unit)
15: Edge extraction unit 16: Product image generation unit 17: Area division unit 18: Obstacle image generation unit

Claims (5)

A first camera mounted on a vehicle and photographing a scene around the vehicle;
A second camera mounted on the vehicle for photographing a photographing range partially overlapping with a photographing range of the first camera;
One of the first camera and the second camera is a main camera and the other is a sub camera. An image photographed by the main camera is a main photographed image and an image photographed by the sub camera is sub photographed. A viewpoint conversion unit that generates a viewpoint conversion image obtained by converting the sub-photographed image into an image captured from the viewpoint position of the main camera, as an image;
A vehicle periphery photographing apparatus comprising: an image information comparison unit that generates comparison information comparing the main photographed image and the viewpoint conversion image. An edge extraction unit that generates an edge image obtained by extracting an edge from the main photographed image;
The product image generation part which produces | generates the product image which performed the logical product to the pixel value of the difference image as the said comparison information, and the said edge image, and outputs the said product image as an output image. Vehicle periphery photographing device. An area dividing unit for dividing the main photographed image into areas;
An obstacle image is generated by extracting an obstacle area based on the similarity between each of the divided areas and the area having a true value in the product image, and the obstacle image is used instead of the product image. The vehicle periphery photographing apparatus according to claim 2, further comprising an obstacle image generation unit that outputs the output image. Generating the output image as a first output image with the first camera as the main camera, and generating the output image as a second output image with the second camera as the main camera;
The vehicle periphery according to any one of claims 1 to 3, wherein an image in which a pixel indicating an obstacle is set to a predetermined pixel value based on the first output image and the second output image is output as an output image. Shooting device. Acquiring a captured image from a first camera mounted on a vehicle and capturing a scene around the vehicle;
Obtaining a captured image from a second camera mounted on the vehicle and capturing an imaging range partially overlapping with the imaging range of the first camera;
One of the first camera and the second camera is a primary camera and the other is a secondary camera. The captured image captured by the primary camera is a primary captured image and the captured image is captured by the secondary camera. Generating a viewpoint-converted image obtained by converting the sub-photographed image into an image photographed from the viewpoint position of the main camera, using the image as a sub-photographed image;
Generating a comparison information comparing the main photographed image and the viewpoint conversion image, and a vehicle peripheral image processing method.

Images