JP5981857B2 - Outside environment recognition device and outside environment recognition method - Google Patents

Description

  The present invention relates to an outside environment recognition device and an outside environment recognition method for recognizing an environment outside the host vehicle, and more particularly, an outside environment recognition device and an outside environment that recognize a moving object outside the host vehicle and determine a collision with the host vehicle. It relates to the recognition method.

  Conventionally, a result of searching for a first moving area based on a difference between two images having different imaging timings and a moving area specified by an image captured earlier among the two images in an image captured later A technique for identifying one moving area by integrating the second moving area is known (for example, Patent Document 1). By applying such a technique, for example, a road environment in front of the host vehicle can be imaged by an in-vehicle camera mounted on the vehicle, and a moving object existing in front of the vehicle can be recognized.

  In addition, a technique for determining whether or not a moving area belongs to a moving area using a threshold value of pixel values in order to separate the moving area and other areas in the image is disclosed (for example, Patent Document 2).

Japanese Patent No. 3230509 Japanese Patent No. 2624117

  However, as in Patent Document 1, in the case of performing so-called pattern matching, in which a moving area specified in an image captured earlier is searched in an image captured later, a block for comparing both images is determined. Setting the size and the distance between adjacent blocks becomes difficult. The processing speed based on the block size and the like and the detection accuracy are in a trade-off relationship. For example, when the block size and the like are increased, the processing speed is improved, but the detection accuracy is lowered. On the other hand, if the block size is reduced, the detection accuracy is improved, but the processing speed is reduced. In a system that avoids collision with a moving object by real-time processing, it is desired to realize both high processing speed and high detection accuracy.

  Further, in Patent Document 2, it is determined by using a threshold value whether or not it belongs to the moving region, but since the environmental image outside the vehicle changes variously depending on the sunlight conditions, a uniform threshold value is set. It is difficult to do so, and further processing load is required to statistically derive the threshold value.

  In view of such problems, the present invention can easily set a threshold for separating a moving region, or can specify a moving object with high processing speed and high detection accuracy without setting a threshold. It is another object of the present invention to provide an outside environment recognition device and an outside environment recognition method.

  In order to solve the above-described problem, an external environment recognition device according to the present invention is based on an image data acquisition unit that acquires image data obtained by imaging an external environment through an imaging device, and two image data having different imaging timings. A flow vector deriving unit for deriving a flow vector for each block obtained by dividing the image indicated by the data, and the direction and size of the flow vector that can be regarded as a moving object in the flow vectors, and adjacent blocks A moving region candidate specifying unit for specifying a moving region candidate that is a moving region candidate indicating a moving object, and a moving region including a moving region candidate in an image indicated by one of the two image data Flow of moving area candidates in the comparison area that is larger than the candidate and the comparison area in the image indicated by the other image data A first difference image deriving unit for deriving a difference in image information from a region shifted by a Couttle and generating a first difference image, a comparison region in an image indicated by one of the two image data, and the other A second difference image deriving unit for deriving a difference of image information between the comparison area and the same area in the image indicated by the image data, and subtracting the first difference image from the second difference image And a difference image subtracting unit that generates a third difference image, and a moving region specifying unit that specifies a moving region based on a distribution tendency of image information of the third difference image.

  The moving area candidate specifying unit may be grouped in units of blocks, and the moving area specifying unit may specify moving areas in units of areas smaller than the block.

  The moving region specifying unit may generate a histogram obtained by adding each piece of image information of the third difference image in the horizontal or vertical direction of the image, and may specify the moving region based on the sign.

  The movement area specifying unit may search for the outer frame of the movement area using the outer frame of the movement area candidate as a search start position.

  In order to solve the above-described problem, an external environment recognition method of the present invention acquires image data obtained by capturing an external environment through an imaging device, and based on two image data having different imaging timings, an image indicated by the image data is displayed. A moving area that represents a moving object by deriving a flow vector for each divided block, approximating the direction and size of the flow vector that can be regarded as a moving object, and grouping adjacent blocks. A moving region candidate that is a candidate for the image is specified, and the comparison image that is a region that includes the moving region candidate and is larger than the moving region candidate in the image indicated by one of the two image data, and the other image data indicates Deriving a difference in image information from an area obtained by shifting the comparison area by the flow vector of the moving area candidate in the image, and calculating the first difference image The difference between the image information of the comparison area in the image indicated by one image data and the same area as the comparison area in the image indicated by the other image data is derived from the two image data, and the second difference An image is generated, a first difference image is subtracted from the second difference image to generate a third difference image, and a moving region is specified based on a distribution tendency of image information of the third difference image.

  According to the present invention, it is possible to easily specify a moving object with high processing speed and high detection accuracy without setting a threshold for separating a moving area or setting a threshold.

It is the block diagram which showed the connection relation of the environment recognition system. It is explanatory drawing for demonstrating the image produced | generated with an imaging device. It is explanatory drawing for demonstrating a partial image. It is explanatory drawing for demonstrating the specific process of a movement area candidate. It is explanatory drawing for demonstrating the difference process between frames. It is explanatory drawing for demonstrating the difference process between frames. It is explanatory drawing for demonstrating the difference process between frames. It is the functional block diagram which showed the schematic function of the external environment recognition apparatus. It is explanatory drawing for demonstrating the process of a flow vector derivation | leading-out part. It is explanatory drawing for demonstrating operation | movement of a 1st difference image derivation | leading-out part. It is explanatory drawing for demonstrating operation | movement of a 2nd difference image derivation | leading-out part. It is explanatory drawing for demonstrating a 3rd difference image. It is explanatory drawing for demonstrating the concrete subtraction process of the brightness | luminance of a pixel. It is explanatory drawing for demonstrating operation | movement of a movement area specific | specification part. It is the flowchart which showed the flow of the whole process of the environment recognition method outside a vehicle.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

  In the present embodiment, an optical flow is used as an image processing technique for detecting a moving object located in front of the host vehicle. Here, the moving object indicates an object that moves independently, such as a vehicle, a bicycle, or a pedestrian, and the optical flow is represented by a two-dimensional velocity vector field on the image, that is, a velocity field associated with the movement of the moving object on the image. It is a thing. In such an optical flow, for example, the movement of a feature point that can be recognized as the same part between two images that are consecutively captured in a time direction and is imaged at a predetermined cycle is calculated as a vector (hereinafter referred to as “flow vector”). doing. Then, by calculating this flow vector in the entire area in the captured image, for example, the position of the moving object in the image and the moving speed and movement of the moving object only by the presence or absence of the flow vector when the host vehicle is stopped. Information such as direction can be detected.

(Environment recognition system 100)
FIG. 1 is a block diagram showing a connection relationship of the environment recognition system 100. The environment recognition system 100 includes an imaging device 110, a vehicle exterior environment recognition device 120, and a vehicle control device (ECU: Engine Control Unit) 130 provided in the host vehicle 1.

  The imaging device 110 includes an imaging element such as a wide angle (wide field angle) lens and a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and an optical axis in front of the host vehicle 1. Is installed with the side facing down slightly from the horizontal. The imaging device 110 can capture an environment corresponding to the front of the host vehicle 1 and generate a color image or a monochrome image composed of three hues (R (red), G (green), and B (blue)). . Here, the wide-angle lens refers to a lens having a relatively wide angle of view (viewing angle) compared to a standard lens (for example, 120 degrees or more) or a relatively short focal length. For example, a fish-eye lens is used as a wide-angle lens, A hemispherical space centered on the imaging device 110 can be imaged.

  Note that the projection method of the imaging apparatus 110 according to the present embodiment uses a three-dimensional projection, but various existing projection methods such as a perspective projection can be employed. Here, an example in which a fish-eye lens is used as the wide-angle lens will be described, but various wide-angle lenses such as a conical (axicon) prism can also be used. The reason why the angle of view of the lens is set to 120 degrees or more is to allow the vehicle 1 to properly grasp the situation of the orthogonal road when the host vehicle 1 stops at the stop line on the cross road or the T-shaped road.

  FIG. 2 is an explanatory diagram for explaining an image generated by the imaging apparatus 110. As can be understood with reference to FIG. 2, an image 200 (hereinafter referred to as “environment image”) 200 generated by the imaging device 110 has a monotonically increasing angle with respect to the distance from the center of the image through a circumferential fisheye lens. (3D projection method). Therefore, it is possible to acquire a wide range of environmental images 200 at a time while maintaining the resolution of the image in the traveling direction of the host vehicle 1.

  In addition, the imaging device 110 continuously generates image data obtained by converting the environment image 200 into data, for example, every 1/10 second (10 fps). Each functional unit in the following embodiment performs each process triggered by such update of image data. Therefore, each functional unit repeats the process in units of frames.

  The outside-vehicle environment recognition device 120 acquires image data indicating the environment image 200 from the imaging device 110, and derives a flow vector based on the optical flow technique using pattern matching from two image data having different imaging timings. However, the vehicle environment recognition apparatus 120 derives a flow vector using a partial image obtained by cutting out only a necessary portion in the environmental image 200.

  FIG. 3 is an explanatory diagram for explaining the partial image 202. As described above, in the environment image 200 shown in FIG. 2, since a wide-angle image can be acquired at once, not only the front of the host vehicle 1 but also moving objects approaching from either the left or right direction (horizontal direction) It becomes possible to specify. However, in the environment image 200, since the distance from the center of the image and the angle are approximately proportional, the image is distorted as the distance from the center increases. For example, in the area close to the four corners in the environment image 200 of FIG. 2, an object that actually extends horizontally is displayed curved in the environment image 200, and its end includes a vertical component. Can be visually recognized. Here, “horizontal” indicates the screen horizontal direction of the captured environment image 200, and “vertical” indicates the screen vertical direction of the captured environment image 200.

  Therefore, the outside environment recognition apparatus 120 includes a region indicated by a broken line in FIG. 2 where there is a moving object to be specified in the traffic environment in the environment image 200 shown in FIG. A partial image 202 shown in FIG. 3 is cut out by combining the central region and the left and right regions in the horizontal direction. Then, a flow vector based on the optical flow is derived for the partial image 202.

  Subsequently, the vehicle exterior environment recognition apparatus 120 specifies a moving area corresponding to the moving object from the partial image 202 using the flow vector of the partial image 202 and the luminance (image information) of each pixel. Since this moving area specifying process is a characteristic part of this embodiment, it will be described in detail later.

  Further, when the moving area is specified in this way, the vehicle exterior environment recognition apparatus 120 determines the distance of the moving object from the own vehicle 1 based on the position of the moving area on the image, the three-dimensional position including the distance, And the size of the moving object is estimated. This uses the following relationship: That is, when the imaging device 110 is positioned at the front end of the host vehicle 1 and at the center in the vehicle width direction, the imaging direction is equal to the traveling direction, and the main shaft is horizontal or downward, Shows the ground (ground surface) in front of the host vehicle 1. Although the positional relationship between the host vehicle 1 and the ground changes instantaneously depending on the behavior of the host vehicle 1, the shape of the ground, and the like, the positional relationship is constantly constant, and the positional relationship is related to the imaging device 110 being attached to the vehicle. It can be determined from the installation conditions when installed. Therefore, if the position of the moving object, specifically, the ground contact point of the moving object can be specified, the distance and size of the moving object from the host vehicle 1 can be derived, or an approximate three-dimensional position can be specified. It becomes. Since various known techniques can be applied to the distance deriving process and the three-dimensional position specifying process, description thereof is omitted here.

  Further, when the three-dimensional position of the moving object is specified, the vehicle exterior environment recognition apparatus 120 tracks the moving object with reference to the moving area, and whether the moving object and the host vehicle 1 are highly likely to collide with each other. Judgment is made. Here, when it is determined that the possibility of the collision is high, the outside environment recognition device 120 displays a warning (notification) to the driver through the display 122 installed in front of the driver, and the vehicle control device. Information indicating that is output to 130.

  Returning to FIG. 1, the vehicle control device 130 avoids a collision with a moving object specified by the outside environment recognition device 120. Specifically, the vehicle control device 130 acquires the current traveling state of the host vehicle 1 through the rudder angle sensor 132 that detects the steering angle, the speed sensor 134 that detects the speed of the host vehicle 1, and the like. Control to keep the distance from the preceding vehicle at a safe distance. Here, the actuator 136 is a vehicle control actuator used to control a brake, a throttle valve, a steering angle, and the like. Further, when a collision with a moving object is assumed, the vehicle control device 130 controls the actuator 136 to automatically brake the host vehicle 1. Such a vehicle control device 130 can also be formed integrally with the outside environment recognition device 120.

(Identification of moving area)
As described above, when specifying a moving region using pattern matching, it is difficult to set the size of a block and the distance between adjacent blocks when comparing two images with different imaging timings. If one of these is prioritized, the other deteriorates. Moreover, since the partial image 202 outside the vehicle changes variously depending on the sunlight conditions, it is difficult to set a threshold value for determining whether the region is a moving region or other region. is there.

  Therefore, in the present embodiment, the moving area is specified through two stages of processing. First, as a first step, the partial image 202 is used to obtain an optical flow with a block having a predetermined size, and a moving area candidate that is a moving area candidate is specified.

  In the present embodiment, in order to perform pattern matching for deriving a flow vector, the image data acquired in the previous frame is temporarily held at least until the current frame. Here, the image data in the previous frame is referred to as “previous image data”, and the image data in the current frame is referred to as “current image data”. If a block having a high correlation value with an arbitrary block in the image indicated by the previous image data is included in the image indicated by the current image data, a vector connecting the blocks becomes a flow vector.

  FIG. 4 is an explanatory diagram for explaining the specifying process of the moving area candidate 212. For example, when the host vehicle 1 goes straight, as shown in FIG. 4, in a region 210 corresponding to the background (hereinafter referred to as “background region”) 210 in the partial image 202, a flow vector is generated from the center of the partial image 202 to the outside. . At this time, if there is a flow vector on the partial image 202 from the outside to the center of the image, it can be considered that an object approaching the host vehicle 1 is reflected from the front side of the imaging device 110 there. A block group having such a flow vector is set as a moving area candidate 212.

  However, in the first stage, since the flow vector is derived in units of blocks having a relatively large number of pixels (for example, horizontal 8 pixels × vertical 8 pixels), the processing speed for specifying the moving region candidate 212 is high. The outer frame of the moving area candidate 212 does not always indicate an actual moving object. That is, the detection accuracy is still low.

  Then, as a second stage, using the luminance (image information) of the partial image 202, in a limited area including the moving area candidate 212 and larger than the moving area candidate 212, an area unit smaller than the above block, for example, The moving region is specified with high accuracy in pixel units. Here, the reason why the comparison area is set larger than the moving area candidate 212 is to allow the case where the moving area candidate 212 does not include a true moving area. Here, by limiting the processing target to the comparison area, the processing speed can be increased, and the moving area can be specified with high detection accuracy such as a small area unit. Thus, high processing speed and high detection accuracy can be realized at the same time.

  Further, the problem that it is difficult to set the threshold value is handled as follows. That is, the inter-frame difference process that takes the difference between two images captured in consecutive frames is executed from two viewpoints.

  5 to 7 are explanatory diagrams for explaining the inter-frame difference processing. FIG. 5A shows an image indicated by the current image data, and FIG. 5B shows an image indicated by the previous image data. 5A, the moving area candidate 212 in the image indicated by the current image data is specified as shown in FIG. 5A. Therefore, the comparison area 214 including the moving area candidate 212 and larger than the moving area candidate 212 is also specified. .

  The first viewpoint of the inter-frame difference processing is to move the comparison area 214 in the comparison area 214 in the image indicated by the current image data shown in FIG. 6A and the image indicated in the previous image data shown in FIG. The difference of the image information from the region 216 shifted by the flow vector of the region candidate 212 is derived, and the first difference image 218 is generated as shown in FIG.

  In this case, images corresponding to the movement area candidate 212 overlap, and an image corresponding to the background area 210 is shifted. Therefore, the absolute value of the difference in luminance is small in the region corresponding to the moving region candidate 212 as shown in FIG. 6C, and the luminance in the background region 210 as shown in white in FIG. 6C. The absolute value of the difference increases.

  The second aspect of the inter-frame difference processing is that the comparison area 214 in the image indicated by the current image data shown in FIG. 7A and the image shown in the previous image data shown in FIG. A difference between image information of the comparison area 214 and the same area 222 is derived, and a second difference image 224 shown in FIG. 7C is generated.

  In this case, images corresponding to the background area 210 are overlapped, and an image corresponding to the movement area candidate 212 is shifted. Therefore, the absolute value of the luminance difference is large in the region corresponding to the moving region candidate 212 as shown in white in FIG. 7C, and the luminance in the background region 210 is cross-hatched in FIG. 7C. The absolute value of the difference becomes smaller.

  In the present embodiment, the moving region is defined by utilizing the fact that the luminance of the first difference image 218 in FIG. 6C and the second difference image 224 in FIG. Identify. That is, each luminance in the comparison area 214 is compared, and a pixel whose absolute value of luminance is larger in the second difference image 224 than in the first difference image 218 can be regarded as a moving area. A pixel having a smaller luminance absolute value in the second difference image 224 can be regarded as a background area 210 other than the moving area, for example. Therefore, whether the true moving area and the area other than the moving area are separated by using a uniform threshold value of ± 0, or which luminance has a larger absolute value without setting a threshold value. Can only be separated. Hereinafter, the vehicle environment recognition apparatus 120 for realizing such processing will be specifically described, and then the vehicle environment recognition method will be described in detail.

(Vehicle environment recognition device 120)
FIG. 8 is a functional block diagram showing a schematic function of the outside environment recognition device 120. As shown in FIG. 8, the vehicle exterior environment recognition device 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

  The I / F unit 150 is an interface for performing bidirectional information exchange with the imaging device 110 and the vehicle control device 130. The data holding unit 152 includes a RAM, a flash memory, an HDD, and the like, and holds various pieces of information necessary for processing of each functional unit described below. In addition, the data holding unit 152 temporarily stores a table in which one 3D position is associated with each block or each pixel in the image, and image data (previous image data) received from the imaging device 110. Hold on.

  The central control unit 154 is configured by a semiconductor integrated circuit including a central processing unit (CPU), a ROM storing a program and the like, a RAM as a work area, and the like, and through the system bus 156, the I / F unit 150 and the data holding unit. 152 is controlled. In the present embodiment, the central control unit 154 includes the image data acquisition unit 160, the image cutout unit 162, the flow vector derivation unit 164, the moving region candidate identification unit 166, the first difference image derivation unit 168, and the second difference image. It also functions as functional units such as a deriving unit 170, a difference image subtracting unit 172, a moving region specifying unit 174, a moving object state deriving unit 176, a collision determination unit 178, a notification unit 180, and an image holding unit 182.

  The image data acquisition unit 160 acquires image data obtained by imaging the environment outside the vehicle from the imaging device 110 through the I / F unit 150.

  The image cutout unit 162 is a partial image as shown in FIG. 3 having a predetermined position and size from the environment image 200 shown in FIG. 2 based on the current image data acquired by the image data acquisition unit 160. 202 is cut out.

  The flow vector deriving unit 164 is a block obtained by dividing the partial image 202 cut out by the image cutout unit 162 through so-called pattern matching based on two pieces of image data (previous image data and current image data) having different imaging timings. The flow vector for each is derived. Here, a predetermined block size and a distance between adjacent blocks are set. The block size and the like are determined based on the lower limit size of the moving region to be detected, the minimum necessary resolution, and the like.

  FIG. 9 is an explanatory diagram for explaining the processing of the flow vector deriving unit 164. The flow vector deriving unit 164 selects a block 230b corresponding to the block 230a arbitrarily extracted from the partial image 202 shown in FIG. 9A based on the previous image data stored in the data holding unit 152 (high correlation value). Then, detection is performed from the partial image 202 shown in FIG. 9B based on the current image data (pattern matching). At this time, the movement trajectory indicated by the arrow in FIG. 9B from the block 230a corresponding to the previous image data to the block 230b corresponding to the current image data becomes the flow vector in the optical flow.

  As such pattern matching, it is conceivable to compare the luminance in units of blocks indicating arbitrary image positions between the previous image data and the current image data. For example, SAD (Sum of Absolute Difference) that takes the difference in luminance, SSD (Sum of Squared Intensity Difference) that uses the difference squared, or NCC that takes the similarity of the variance value obtained by subtracting the average value from the luminance of each pixel There are methods such as (Normalized Cross Correlation). The vehicle exterior environment recognition apparatus 120 performs such block-based flow vector derivation processing for all blocks displayed in the partial image 202 (for example, 640 pixels × 128 pixels). Further, the number of pixels in the block can be arbitrarily set, but here, the block is assumed to be 8 horizontal pixels × 8 vertical pixels.

  In this way, the flow vector deriving unit 164 can derive the flow vector of the partial image 202 corresponding to the current image data, as shown in FIG. 9C. The length of the arrow indicates the size of the flow vector, and the direction of the arrow indicates the direction of the flow vector. In addition, the number of flow vectors is essentially the same as the number of blocks composed of a predetermined number of pixels, which is a derived unit. However, here, the number of flow vectors is limited to facilitate understanding. In addition to the above, various known techniques can be applied to the flow vector derivation method.

  The moving region candidate specifying unit 166 approximates the direction and size of a flow vector that can be regarded as a moving object among the flow vectors derived by the flow vector deriving unit 164, and groups adjacent blocks to move. A moving area candidate 212 that is an area candidate is specified.

  For example, since the flow vector is generated by the movement of the feature point in the image, as shown in FIG. 9C, not only the movement area candidate 212 but also the background area 210 as the host vehicle 1 moves. Also produces a flow vector. Further, in this embodiment, since a wide-angle lens is used, it can be seen that the flow vector of the background area 210 increases as it shifts to the side as compared with the center of the partial image 202. .

  Further, when the imaging device 110 is positioned at the front end of the host vehicle 1 and at the center in the vehicle width direction and the imaging direction is set to be equal to the traveling direction, as shown in FIG. The horizontal direction of the flow vector is reversed left and right with a central reference line 240 extending perpendicularly to the horizontal center of the partial image 202.

  At this time, if the partial image 202 shown in FIG. 9C is divided into a left region and a right region based on the central reference line 240, the flow vector of the background region 210 is generated in a direction away from the central reference line 240. . On the other hand, it can be seen that the flow vector of the moving area candidate 212 corresponding to the moving object approaching the host vehicle 1 is generated in the direction approaching the center reference line 240.

  Therefore, a flow vector that divides the region horizontally in accordance with the center reference line 240 and that faces the right direction in the left region or a flow vector that faces the left direction in the right region as shown in FIG. 9C is detected. These can be regarded as the flow vectors of the moving area candidate 212.

  Then, the moving region candidate specifying unit 166 sequentially compares the flow vectors of adjacent blocks from the flow vector that can be regarded as the moving region candidate 212, and approximates the direction and the magnitude of the flow vector (for example, the direction is ± 30 degrees). And blocks whose size is within 8 pixels). Then, the grouped block group is set as a movement area candidate 212. In addition to the above, various known techniques can be applied to such a method for specifying the moving area candidate 212.

  The first difference image deriving unit 168 sets a comparison area 214 that includes the movement area candidate 212 in the image indicated by the current image data and is larger than the movement area candidate 212. The size of the comparison area 214 only needs to be large enough to include the true movement area. For example, each of the left and right in the horizontal direction and the upper and lower in the vertical direction from the movement area candidate 212 is one block in the processing of the flow vector deriving unit 164. It may be larger. Then, the first difference image deriving unit 168 includes pixels of the comparison area 214 in the image indicated by the current image data and the area 216 obtained by shifting the comparison area 214 by the flow vector of the movement area candidate 212 in the image indicated by the previous image data. The first difference image 218 is generated. Here, the unit for deriving the difference is an area smaller than the block unit in the processing of the flow vector deriving unit 164, for example, a pixel unit.

  FIG. 10 is an explanatory diagram for explaining the operation of the first difference image deriving unit 168. The first difference image deriving unit 168 compares the comparison area 214 in the image indicated by the current image data shown in FIG. 10A with the image shown in the previous image data shown in FIG. A difference in image information from the region 216 obtained by shifting the region 214 by the flow vector of the moving region candidate 212 is derived, and a first difference image 218 is generated as shown in FIG. However, the comparison area 214 and the shifted area 216 are equal in size. In addition, the flow vector to be shifted may take an average value of all the flow vectors constituting the moving region candidate 212 or may take a median.

  Here, the images corresponding to the moving area overlap, and the image corresponding to the background area 210 is shifted. Therefore, the absolute value of the luminance difference is small in the region corresponding to the moving region as hatched in FIG. 10C, and the absolute value of the luminance difference in the background region 210 is filled in white in FIG. 10C. The value increases.

  The second difference image deriving unit 170 derives a pixel difference between the comparison area 214 in the image indicated by the current image data and the area 222 that is the same as the comparison area 214 in the image indicated by the previous image data. 224 is generated.

  FIG. 11 is an explanatory diagram for explaining the operation of the second difference image deriving unit 170. The second difference image deriving unit 170 is the same area as the comparison area 214 in the image indicated by the current image data shown in FIG. 11A and the comparison area 214 in the image indicated by the previous image data shown in FIG. A pixel difference from 222 is derived, and a second difference image 224 is generated as shown in FIG.

  Here, images corresponding to the background area 210 are overlapped, and an image corresponding to the moving area is shifted. Therefore, the absolute value of the luminance difference is large in the area corresponding to the moving area as shown in white in FIG. 11C, and the absolute luminance difference is absolute in the background area 210 as hatched in FIG. 11C. The value becomes smaller.

  However, if the imaging device 110 moves with the movement of the host vehicle 1, the background area 210 is strictly different between the previous image data and the current image data. Here, when the background is sufficiently far away and the imaging apparatus 110 is traveling straight at 10 km / h, the background change between frames is small. Further, when the imaging device 110 is faster than 10 km / h or moves with a turn, the change in the background region 210 is reduced by correcting the image cutout position of the previous image data based on the speed and the angular velocity. be able to.

  The difference image subtraction unit 172 subtracts the first difference image 218 from the second difference image 224 to generate a third difference image.

  FIG. 12 is an explanatory diagram for explaining the third difference image 250. In the third difference image 250 shown in FIG. 12, the pixels corresponding to the true moving region 252 have a larger absolute value of luminance in the second difference image 224 than in the first difference image 218, that is, in the third difference image 250. The luminance of the second difference image 224 is smaller than that of the first difference image 218 in the pixels corresponding to the true background region 210, that is, the luminance of the third difference image 250 is higher. Negative value. Therefore, the third difference image 250 can be represented by binary values of positive and negative (+1, −1) according to the sign of luminance. In this way, the moving area can be appropriately separated only by the positive / negative (absolute magnitude) of the luminance of the pixel.

  FIG. 13 is an explanatory diagram for explaining a specific subtraction process of pixel luminance. Since the difference image subtraction unit 172 subtracts the first difference image 218 from the second difference image 224 in units of pixels, it is affected by random noise simply by subtraction. Therefore, when comparing the brightness of each pixel, the influence of noise is suppressed by taking the difference between the total value and the average value of the brightness of the target pixel 254 and the neighboring 8 pixels 256.

  The moving area specifying unit 174 specifies the moving area based on the distribution tendency of the image information of the third difference image 250. Here, the distribution tendency refers to a bias or appearance tendency of image information.

  FIG. 14 is an explanatory diagram for explaining the operation of the movement area specifying unit 174. FIG. 14A shows the third difference image 250, and the outermost square is the same size as the comparison region 214. Further, the area indicated by the dotted line is the moving area candidate 212, and the area filled with white at the center is the true moving area 252. Here, the coordinate of the pixel in the horizontal direction is taken as the x axis, and the coordinate of the pixel in the vertical direction is taken as the y axis.

  The moving area specifying unit 174 adds the values (binary values) of the pixels of the third difference image 250 to specify the moving area 252, and shows the total value as a histogram. For example, when all the values of the pixels in the y-axis direction (vertical direction) in the comparison area 214 are added to each pixel on the x-axis, a histogram relating to the x-axis shown in FIG. 14B is generated. When all the values of the pixels in the x-axis direction (horizontal direction) in the comparison area 214 are added to each pixel on the y-axis, a histogram regarding the y-axis shown in FIG. 14C is generated.

  As can be understood with reference to FIG. 14B, in the x-axis histogram, in the pixel corresponding to the moving region 252, a negative value (−1) as the background region 210 is also added. Since many (+1) are added, the total value often takes a positive value. On the other hand, since only a negative value (−1) is added to the pixels corresponding to the background area 210, the total value is a negative value. Therefore, a pixel whose sign of the total value is inverted can be used as the outer frame of the moving region 252. Also in the y-axis histogram, the outer frame of the moving region 252 can be obtained by the same processing as the x-axis histogram. However, since all the pixels in the moving region 252 do not take a positive value, the total value of the histogram may be somewhat uneven. In this way, a rectangular frame indicating the true movement area 252 is obtained as shown in FIG.

  Here, since the sizes of the moving region 252 and the comparison region 214 are different, the total value of the pixel values corresponding to the moving region 252 in the x-axis histogram and the y-axis histogram does not clearly take a positive value. There is. In this case, after the movement area 252 is specified, a true area is calculated by calculating an x-axis histogram and a y-axis histogram for an area slightly larger than the movement area 252 (an area smaller than the comparison area 214). The moving area 252 can be narrowed down.

  Further, since the true moving area 252 is considered to approximate the moving area candidate 212, the moving area specifying unit 174 searches the outer frame of the moving area 252 using the outer frame of the moving area candidate 212 as a search start position. Good. In this way, the moving area 252 can be quickly identified, and the processing time can be further shortened.

  In addition, as can be understood with reference to FIG. 12, the moving region 252 in the second difference image 224 is more horizontal than the moving region 252 in the first difference image 218 by the flow vector of the moving region candidate 212. Longer in the direction. Therefore, there is a possibility that the moving region 252 cannot be sufficiently specified for such a part. However, as will be described later, since it is the vertical lower part of the moving region 252 that should be accurately specified as the moving region 252, a decrease in the specifying accuracy in the horizontal direction does not matter.

  In addition, since the moving area specifying unit 174 specifies the moving area in an area unit (in this case, a pixel) that is smaller than the block that is the processing unit of the moving area candidate specifying unit 166, the moving area specifying unit 174 is high in a limited area such as the comparison area 214. It is possible to specify the moving area with high accuracy.

  Returning to FIG. 8, the moving object state deriving unit 176 determines the distance and magnitude of the moving object indicated by the moving area 252 from the host vehicle 1 based on the ground point of the moving area 252 in the ground area indicating the ground on the image. Now, a three-dimensional position is derived. For example, the data holding unit 152 holds in advance a table in which one three-dimensional position is associated with each block or each pixel in the image, and the moving object state deriving unit 176 refers to the table. Thus, the three-dimensional position corresponding to the ground contact point of the moving object (vertically below the moving area 252) is specified. In addition, the moving object state deriving unit 176 specifies the size of the moving object based on a predetermined relational expression between the size of the moving area 252 and the distance of the moving object corresponding to the moving area 252. In the present embodiment, since the moving area 252 is appropriately specified, the position of the ground contact point is accurately specified, and the distance of the moving object from the host vehicle 1 and the size of the moving object can be accurately derived. .

  The collision determination unit 178 determines whether or not the own vehicle 1 and the moving object are highly likely to collide according to the traveling direction and speed of the own vehicle 1 and the position, traveling direction, and speed of the identified moving object. To do. Since various existing techniques can be applied to the collision determination, detailed description thereof is omitted here.

  When the collision determination unit 178 determines that the possibility of a collision is high, the notification unit 180 notifies the driver to that effect through a buzzer (voice output means) or a warning light (display 122). In addition, information indicating that is output to the vehicle control device 130 through the I / F unit 150.

  The image holding unit 182 overwrites the previous image data of the data holding unit 152 with the current image data. Thus, in the next frame, the current image data can be used as the previous image data.

(External vehicle environment recognition method)
FIG. 15 is a flowchart showing the overall processing flow of the vehicle exterior environment recognition method. FIG. 15 shows an overall flow related to an interrupt process when image data captured through a wide-angle lens is transmitted from the imaging device 110.

  When the interruption by the outside environment recognition method occurs, the image data acquisition unit 160 acquires the current image data transmitted from the imaging device 110 through the I / F unit 150 (S300). The image cutout unit 162 then selects a partial image 202 (for example, horizontal 640 pixels) having a predetermined position and size from the environment image 200 (for example, horizontal 640 pixels × vertical 480 pixels) based on the acquired current image data. X vertical 128 pixels: horizontal 80 blocks x vertical 16 blocks) are cut out (S302). Next, the flow vector deriving unit 164 derives a plurality of flow vectors in the partial image 202 (S304). Then, the moving area candidate specifying unit 166 approximates the direction and the size of the flow vector that can be regarded as a moving object among the derived flow vectors, and groups adjacent blocks to move the moving area candidate 212. Is specified (S306).

  Subsequently, the first difference image deriving unit 168 includes a comparison area 214 in the image indicated by the current image data and an area 216 obtained by shifting the comparison area 214 by the flow vector of the movement area candidate 212 in the image indicated by the previous image data. A pixel difference is derived to generate a first difference image 218 (S308). The second difference image deriving unit 170 derives a pixel difference between the comparison area 214 in the image indicated by the current image data and the area 222 identical to the comparison area 214 in the image indicated by the previous image data. A difference image 224 is generated (S310). Then, the difference image subtracting unit 172 generates the third difference image 250 by subtracting the first difference image 218 from the second difference image 224 (S312), and the moving region specifying unit 174 displays the image of the third difference image 250. The moving area 252 is specified based on the information distribution tendency (S314). Then, the moving object state deriving unit 176 determines the distance, size, and three-dimensional position of the moving object indicated by the moving area 252 from the host vehicle 1 based on the ground point of the moving area 252 in the ground area indicating the ground on the image. Is derived (S316).

  The collision determination unit 178 determines whether or not the own vehicle 1 and the moving object are highly likely to collide according to the traveling direction and speed of the own vehicle 1 and the position, traveling direction, and speed of the identified moving object. (S318). As a result, when it is determined that there is a high possibility of a collision with a moving object (YES in S318), the notification unit 180 notifies the driver that the possibility of a collision is high and notifies the vehicle control device 130 of the fact. The information shown is output (S320). If it is not determined that there is a high possibility of collision with a moving object (NO in S318), the process proceeds to step S322.

  Finally, in order to use the current image data as the previous image data in the next frame, the image holding unit 182 overwrites the previous image data in the data holding unit 152 with the current image data (S322).

  As described above, according to the vehicle environment recognition apparatus 120 and the vehicle environment recognition method as described above, as a first step, the partial image 202 is used to obtain an optical flow with a block of a predetermined size, which is a candidate for a moving region. In the limited region such as the comparison region 214 that includes the moving region candidate 212 and is larger than the moving region candidate 212, the moving region candidate 212 is identified and the luminance (image information) of the partial image 202 is used as the second step. Since the moving area 252 is specified with high accuracy in units of area smaller than the block, for example, in units of pixels, it is possible to specify the moving object with high processing speed and high detection accuracy.

  In the present embodiment, by using the fact that the luminance of the first difference image 218 and the second difference image 224 is inverted, a threshold value for separating the moving region 252 is simply set, or the threshold value is set. No need to set.

  Also provided are a program that causes a computer to function as the vehicle exterior environment recognition device 120 and a computer-readable storage medium that stores the program, such as a flexible disk, magneto-optical disk, ROM, CD, DVD, and BD. Here, the program refers to data processing means described in an arbitrary language or description method.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this embodiment. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Is done.

  For example, in the above-described embodiment, the image data acquisition unit 160, the image cutout unit 162, the flow vector derivation unit 164, the moving region candidate identification unit 166, the first difference image derivation unit 168, the second difference image derivation unit 170, the difference The image subtracting unit 172, the moving region specifying unit 174, the moving object state deriving unit 176, the collision determining unit 178, the notification unit 180, and the image holding unit 182 are configured to be operated by software by the central control unit 154. However, the functional unit described above can be configured by hardware.

  Note that each step of the vehicle environment recognition method of the present specification does not necessarily have to be processed in time series in the order described in the flowchart, and may include processing in parallel or a subroutine.

  The present invention relates to an outside environment recognition device and an outside environment recognition method for recognizing an environment outside the host vehicle, and more particularly, an outside environment recognition device and an outside environment that recognize a moving object outside the host vehicle and determine a collision with the host vehicle. It can be used as a recognition method.

DESCRIPTION OF SYMBOLS 1 ... Own vehicle 110 ... Imaging device 120 ... Outside vehicle environment recognition apparatus 160 ... Image data acquisition part 164 ... Flow vector derivation | leading-out part 166 ... Movement area candidate specific | specification part 168 ... 1st difference image derivation | leading-out part 170 ... 2nd difference image derivation | leading-out part 172 ... difference image subtraction unit 174 ... moving region specifying unit

Claims (5)

An image data acquisition unit for acquiring image data obtained by imaging the environment outside the vehicle through the imaging device;
A flow vector deriving unit for deriving a flow vector for each block obtained by dividing an image indicated by the image data based on two image data having different imaging timings;
Among the flow vectors, the direction and size of a flow vector that can be regarded as a moving object are approximate, and adjacent blocks are grouped together to identify a moving area candidate that is a moving area candidate indicating the moving object. A moving area candidate identification unit;
Of the two image data, the comparison area in the image indicated by one of the image data includes the movement area candidate and is larger than the movement area candidate, and the comparison area in the image indicated by the other image data. A first difference image deriving unit for deriving a difference of image information from an area shifted by a flow vector of a moving area candidate and generating a first difference image;
Deriving a difference in image information between the comparison area in the image indicated by one of the two image data and the same area as the comparison area in the image indicated by the other image data; A second difference image deriving unit for generating an image;
A difference image subtraction unit that subtracts the first difference image from the second difference image to generate a third difference image;
A moving area specifying unit for specifying the moving area based on a distribution tendency of image information of the third difference image;
A vehicle exterior environment recognition device comprising: The moving area candidate specifying unit is grouped in units of blocks,
The outside environment recognition device according to claim 1, wherein the moving area specifying unit specifies the moving area in units of areas smaller than the block.   The moving area specifying unit generates a histogram obtained by adding each image information of the third difference image in an image horizontal direction or a vertical direction, and specifies the moving area by a sign thereof. The outside environment recognition device described in 1.   The vehicle outside environment recognition device according to any one of claims 1 to 3, wherein the movement area specifying unit searches for an outer frame of the movement area using an outer frame of the movement area candidate as a search start position. . Obtain image data that captures the environment outside the vehicle through the imaging device,
Based on two image data having different imaging timings, a flow vector for each block obtained by dividing the image indicated by the image data is derived,
Among the flow vectors, the direction and size of a flow vector that can be regarded as a moving object are approximate, and adjacent blocks are grouped together to identify a moving area candidate that is a moving area candidate indicating the moving object. ,
Of the two image data, the comparison area in the image indicated by one of the image data includes the movement area candidate and is larger than the movement area candidate, and the comparison area in the image indicated by the other image data. Deriving a difference in image information from the region shifted by the flow vector of the moving region candidate, and generating a first difference image,
Deriving a difference in image information between the comparison area in the image indicated by one of the two image data and the same area as the comparison area in the image indicated by the other image data; Generate an image,
Subtracting the first difference image from the second difference image to generate a third difference image;
The vehicle environment recognition method characterized by identifying the moving area based on a distribution tendency of image information of the third difference image.

Images