JP4813304B2 - Vehicle periphery monitoring device - Google Patents

Description

  The present invention relates to a vehicle periphery monitoring device that monitors an object that affects driving of a vehicle.

  Conventionally, as this kind of vehicle periphery monitoring device, an object such as a pedestrian who may be in contact with a vehicle is extracted from an image around the vehicle captured by an infrared camera, and the information is obtained from the driver of the vehicle. There has been proposed a display processing device that is visually provided (see, for example, Patent Document 1). This device binarizes an infrared image to search for a high brightness area, and the area is determined based on head determination conditions such as the center of gravity position, area ratio, and actual area in the infrared image of the area. It is determined whether or not. And when it determines with it being the area | region of a pedestrian's head, the area | region which includes a pedestrian's body is displayed separately from another area | region. Thereby, pedestrian information is displayed to the driver of the vehicle, and assists the driver's vision.

  In addition, in such a vehicle periphery monitoring device, a vehicle peripheral device that visually displays an object that is likely to be a pedestrian is proposed in order to suppress unnecessary alerts to the driver of the vehicle. (See, for example, Patent Document 2). This apparatus extracts an object that may be a pedestrian from images obtained by two infrared cameras, and (1) whether the object has a characteristic that a pedestrian cannot have, and (2 ) As a determination condition whether or not it has an external feature of a predetermined artificial structure, if the object satisfies the determination condition, determine that it is an artificial structure, and perform a process of excluding it from the target of alerting Yes.

However, when it is determined whether or not the object is an artificial structure according to the above determination condition, the determination accuracy is high, but the determination of (2) involves complicated calculation processing, so There is an inconvenience that the determination time becomes longer. Therefore, it is conceivable to perform only the above (1). However, when only the above (1) is performed, the determination time as to whether or not it is an artificial structure is shortened, but there is a disadvantage that the determination accuracy is lowered.
Japanese Patent Laid-Open No. 11-328364 JP 2003-230134 A

  In view of the above disadvantages, the present invention provides a vehicle periphery monitoring device that can efficiently and accurately determine whether or not an object extracted from an image is an artificial structure. It is an issue.

Whether the vehicle periphery monitoring equipment of the present invention to solve the above problems, and the object extracting means for extracting an object from an image obtained by the onboard imaging unit in the vehicle, the object is an artificial structure A vehicle periphery monitoring device that includes an artificial structure determination means for determining whether or not, and monitors an object existing around the vehicle,
Road area identifying means for identifying a road area from the image;
Road area identification means for identifying a road area that is an area included in the road area and defined based on a position of a lane mark from the image ,
The artificial structure determination means includes
When the object is extracted in the road area identified by the road area identification means by the object extraction means, the object is an artificial structure based on the first artificial structure determination condition. If the object is extracted outside the road area, the object is artificially determined based on a second artificial structure determination condition different from the first artificial structure determination condition. Determine whether it is a structure ,
When an object is extracted in the road area identified by the road area identification means by the object extraction means, based on a third artificial structure determination condition including at least a vehicle as an artificial structure, When it is determined whether or not the object is an artificial structure and the object is extracted outside the roadway area, at least one of a vehicle, a vending machine, a streetlight transformer, and a blinker light is used as the artificial structure. It is characterized in that it is determined whether or not the object is an artificial structure based on a fourth artificial structure determination condition including one .

According to the vehicle periphery monitoring device of the present invention, the road area identifying means for identifying the road area is provided, and the artificial structure determining means is a road in which the object extracted from the image is identified by the road area identifying means. Whether the target object is an artificial structure based on different determination conditions of the first artificial structure determination condition and the second artificial structure determination condition, depending on whether or not it is extracted within the area or outside the road area Determine whether or not. Here, since the artificial structures that may exist in the road region are limited, it can be determined whether or not the object is an artificial structure based on relatively simple determination conditions. On the other hand, since various artificial structures exist outside the road region, it is considered necessary to determine whether or not the target object is an artificial structure under somewhat complicated conditions. Therefore, the object extracted from the image is changed to the artificial structure by changing the first artificial structure determination condition and the second artificial structure determination condition depending on whether or not the object is extracted in the road region. It can be determined efficiently and accurately.
Further, according to the vehicle periphery monitoring device of the present invention, the vehicle road area identifying means for identifying the road area is provided, and the artificial structure determining means is configured to identify the object extracted from the image by the road area identifying means. If it is extracted within the roadway region, it is determined whether or not the object is an artificial structure based on a new determination condition called a third artificial structure determination condition. Here, since there is a low possibility that an artificial structure exists in the roadway region, the third artificial structure determination condition can be simplified. Therefore, based on the third artificial structure determination condition, it can be determined efficiently and accurately whether or not the object extracted in the roadway region is an artificial structure.
Further, according to the vehicle periphery monitoring device of the present invention, the artificial structure determination means extracts the object extracted from the image outside the road area identified by the road area identification means in the road area. In the case where the target object is an artificial structure, it is determined based on the fourth artificial structure determination condition. Here, since the artificial structures that may exist outside the road area in the road area are limited, the fourth artificial structure determination condition can be made relatively simple. Therefore, based on the fourth artificial structure determination condition, it can be more efficiently determined whether or not the object extracted outside the road area in the road area is an artificial structure.

  Furthermore, in the vehicle periphery monitoring device of the present invention, the imaging unit includes an infrared image imaging unit that acquires an infrared image, and the road region identification unit is an infrared image acquired by the infrared image imaging unit. An edge extraction process for extracting an edge part of an image is performed, and a road region is identified by the edge part extracted by the edge extraction process.

  According to the vehicle periphery monitoring device of the present invention, the infrared image capturing unit is provided as the image capturing unit, and the road area identifying unit extracts an edge portion of the infrared image obtained by the infrared image capturing unit, and the edge unit The road area can be identified with certain certainty by distinguishing and identifying the inside and outside of the road area based on the above.

  In the vehicle periphery monitoring device of the present invention, the artificial structure determination unit detects a lower end portion of the object extracted from the image, and the object is detected when the lower end portion is located in the road area. Is extracted in the road area.

  According to the vehicle periphery monitoring device of the present invention, the lower end portion of the object extracted from the image is detected, and the object is detected from the road region depending on whether the lower end portion is located in the road region. It is determined whether or not it has been extracted. Here, the lower end portion of the target object can be detected as, for example, a pixel portion positioned at the lower end on the image in the region where the same distance portion of the target object is extracted. Whether or not the object is extracted in the road area even if the object is located near the boundary of the road area by determining the position of the object with reference to the lower end of the object. Whether or not can be determined easily and reliably.

  Furthermore, in the vehicle periphery monitoring device of the present invention, the vehicle periphery monitoring device further includes a position calculation unit that calculates a position in real space corresponding to the image portion included in the image, and the artificial structure determination unit is a target extracted from the image When the position of the object in the real space is above the road area in the real space, it is determined that the object has been extracted in the road area.

  According to the vehicle periphery monitoring apparatus of the present invention, the position calculating means calculates the position of the object extracted from the image in the real space and the area range of the road area in the real space, and the real space of the object. It is determined whether or not the object is extracted from the road area depending on whether or not the upper position is located on the area range in the real space. Thereby, for example, even when the shape of the object extracted from the image is a part lacking a part of the actual shape in the real space, the object is extracted in the region. It can be accurately determined whether or not.

  The vehicle periphery monitoring device according to the present invention includes, as the imaging unit, a visible image imaging unit that acquires a visible image, and the roadway area identifying unit adds a lane mark of a roadway from the visible image acquired by the visible image imaging unit. Detecting and identifying the road area by the lane mark.

  According to such a vehicle periphery monitoring device of the present invention, the visible image capturing unit is provided as the image capturing unit, and the road area identifying unit detects a lane mark with high identification accuracy of the road area from the visible image obtained by the visible image capturing unit. The road area is identified based on the lane mark. As described above, the road area can be identified with high accuracy by the lane mark having high identification accuracy of the road area.

  In the vehicle periphery monitoring device of the present invention, the artificial structure determination unit detects a lower end portion of the object extracted from the image, and the object is detected when the lower end portion is located in the roadway region. It is characterized by determining that it has been extracted within the roadway area.

  According to the vehicle periphery monitoring device of the present invention, the lower end of the object extracted from the image is detected, and the object is detected from the road area by determining whether the lower end is located in the road area. It is determined whether or not it has been extracted. Here, the lower end portion of the target object can be detected as, for example, a pixel portion positioned at the lower end on the image in the region where the same distance portion of the target object is extracted. Then, by determining the position of the object with reference to the lower end of the object, whether or not the object has been extracted in the road area even when the object is located near the boundary of the road area Whether or not can be determined easily and reliably.

  The vehicle periphery monitoring device of the present invention includes a position calculation unit that calculates a position in real space corresponding to an image portion included in the image, and the artificial structure determination unit is configured to detect an object extracted from the image. It is determined that the object is extracted in the road area when the position in the real space is above the road area in the real space.

  According to the vehicle periphery monitoring device of the present invention, the position calculating means calculates the position of the object extracted from the image in the real space and the area range of the roadway area in the real space, and the real space of the object. It is determined whether or not the object has been extracted from the roadway area depending on whether or not the upper position is located on the area range in the real space. Thereby, for example, even when the shape of the object extracted from the image is a part lacking a part of the actual shape in the real space, the object is extracted in the region. It can be accurately determined whether or not.

In the vehicle periphery monitoring device of the present invention,
As the first artificial structure determination condition, the approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist in the road region is equal to or higher than a predetermined level. Is set,
As the second artificial structure determination condition, the approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist outside the road region is equal to or higher than a predetermined level. Is set,
As the third artificial structure determination condition, the approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist in the roadway region is equal to or higher than a predetermined level. Is set,
As the fourth artificial structure determination condition, there is an approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist outside the road area in the road area. Set to be above a certain level,
As the feature amount of the specific artificial structure, at least one feature amount of a vehicle, a vending machine, a utility pole, a streetlight transformer, a blinker light, a wall, and a tree is registered.

According to the vehicle periphery monitoring device of the present invention, a typical artificial structure extracted from the outside of the roadway region, which is difficult to distinguish from a pedestrian, a telephone pole, a streetlight transformer, a blinker light, By registering walls and trees as images of the specific artificial structure, it is possible to improve the determination accuracy of the artificial structure determination based on the first to fourth artificial structure determination conditions.

  An embodiment of the present invention will be described below with reference to FIGS.

  First, with reference to FIG. 1 and FIG. 2, the system configuration | structure of the vehicle periphery monitoring apparatus of this embodiment is demonstrated. FIG. 1 is a block diagram showing the overall configuration of the periphery monitoring device, and FIG. 2 is a perspective view showing the appearance of a vehicle (own vehicle) equipped with the periphery monitoring device. In FIG. 2, illustration of some components of the periphery monitoring device is omitted.

  With reference to FIGS. 1 and 2, the periphery monitoring device of the present embodiment includes an image processing unit 1. The image processing unit 1 is connected with two infrared cameras 2R, 2L and one visible camera 2 'as imaging means for capturing an image in front of the host vehicle 10, and the traveling state of the host vehicle 10 is also monitored. As sensors to detect, a yaw rate sensor 3 that detects the yaw rate of the host vehicle 10, a vehicle speed sensor 4 that detects the traveling speed (vehicle speed) of the host vehicle 10, and a brake sensor 5 that detects whether or not the host vehicle 10 is braked. Is connected. Further, the image processing unit 1 has a speaker 6 for outputting auditory alert information such as sound, and a display for displaying images taken by the infrared cameras 2R and 2L and visual alert information. Device 7 is connected.

  The infrared cameras 2R and 2L correspond to the infrared image capturing unit in the present invention, and the visible camera 2 'corresponds to the visible image capturing unit in the present invention. Further, the infrared cameras 2R and 2L and the visible camera 2 ′ correspond to the imaging means of the present invention.

  Although not shown in detail, the image processing unit 1 includes electronic circuits including an A / D conversion circuit, a microcomputer (having a CPU, a RAM, a ROM, and the like), an image memory, and the like. The infrared cameras 2R, 2L, Analog signals output from the visible camera 2 ′, the yaw rate sensor 3, the vehicle speed sensor 4, and the brake sensor 5 are converted into digital data by the A / D conversion circuit and input to the microcomputer. The microcomputer detects an object such as a person (pedestrian or person riding a bicycle) based on the input data, and the speaker detects when the detected object satisfies a predetermined alerting requirement. 6 or a display device 7 is used to execute a process for alerting the driver.

The image processing unit 1 has functions as road area identification means, object extraction means, artificial structure determination means, roadway area identification means, and position calculation means in the present invention. Then, the microcomputer, by executing the car both periphery monitoring program, the computer is a road area identifying means of the present invention, the object extraction unit, functions as artificial structure determination unit.

  As shown in FIG. 2, the infrared cameras 2 </ b> R and 2 </ b> L are attached to the front portion (the front grill portion in the figure) of the host vehicle 10 in order to image the front of the host vehicle 10. In this case, the infrared cameras 2R and 2L are respectively disposed at a position on the right side and a position on the left side of the center of the host vehicle 10 in the vehicle width direction. These positions are symmetrical with respect to the center of the host vehicle 10 in the vehicle width direction. The infrared cameras 2R and 2L are fixed so that their optical axes extend in parallel in the front-rear direction of the host vehicle 10 and the heights of the optical axes from the road surface are equal to each other. . The infrared cameras 2R and 2L have sensitivity in the far-infrared region, and the higher the temperature of the object to be imaged, the higher the level of the output video signal (the luminance of the video signal increases). have.

  Further, the visible camera 2 ′ is attached in front of a room mirror (not shown) of the host vehicle 10 with the imaging direction facing the front of the host vehicle 10. The visible camera 2 ′ is configured by a CCD camera or the like that can capture an image in the visible light region, and the higher the illuminance of the object to be imaged, the higher the level of the output video signal (the brightness of the image of the video signal is (Becomes higher).

  Further, in the present embodiment, the display device 7 includes a head-up display 7a (hereinafter referred to as HUD 7a) that displays image information on the front window of the host vehicle 10. In addition, as a display device 7, instead of the HUD 7a, or together with the HUD 7a, a display provided integrally with a meter for displaying a traveling state such as the vehicle speed of the host vehicle 10 or a display provided in an in-vehicle navigation device is provided. It may be used.

  Next, the overall operation of the periphery monitoring device of this embodiment will be described with reference to the flowchart of FIG.

  With reference to FIG. 3, the image processing unit 1 according to the present embodiment first performs “roadway / road area identification processing” that identifies and sets a roadway area and a road area from a visible image or an infrared image in STEP 100. When the road area can be set, the flag (j) is set (j = 1), and when the road area cannot be set, the flag (j) is reset (j = 0).

  Next, in STEP 200, “object extraction processing” for extracting the object from the infrared image is performed.

  Then, in STEP 300, it is determined whether or not the object has been extracted in the “object extraction process”. When the object has been extracted, the process proceeds to STEP 400, where the road area is set based on the value of the flag (j). Determine whether or not.

  On the other hand, if no object is extracted in STEP 300, the process returns to STEP 100, and the processes after the “roadway / road area identification process” are executed again.

  In STEP 400, when the road area is set (when j = 1), the process proceeds to STEP 500, and the artificial structure is based on different judgment conditions depending on whether or not the object is extracted from the road area. The “artificial structure determination process (I)” for performing the determination is executed, and the process proceeds to STEP 700.

  On the other hand, in STEP 400, when the road area is not set (when j = 0), the process branches to STEP 600 to perform the artificial structure determination without distinguishing the inside and outside of the road area. (II) "is executed, and the process proceeds to STEP700.

  In addition, when it is determined in the “artificial structure determination process (I)” and the “artificial structure determination process (II)” that the object is an artificial structure, the flag (k) is set (k = 1) If it is determined that the object is not an artificial structure, the flag (k) is reset (k = 0).

  In STEP 700, based on the value of the flag (k) indicating the result of the artificial structure determination in STEP 500 or STEP 600, it is determined whether or not the object is an artificial structure.

  In STEP 700, when it is determined that the object is not an artificial structure (when k = 0), the process proceeds to STEP 800, where it is determined whether the object is a pedestrian. Note that the pedestrian determination process of STEP 800 is the same as the process of STEP 34 of FIG. 5 of Patent Document 2 and will not be described in detail. However, from the characteristics such as the shape and size of the target object and luminance dispersion, It is determined whether or not the object has a pedestrian characteristic, and it is determined whether or not the object is a pedestrian.

  On the other hand, when it is determined in STEP 700 that the target object is an artificial structure (when k = 1), it is determined that the target object is not a target for alerting, and the process returns to STEP 100 to “Waveway / Road”. The processing following the “region identification processing” is executed again.

  Furthermore, when it is determined in STEP 800 that the object has pedestrian characteristics, the process proceeds to STEP 900, and the possibility of contact between the object and the host vehicle 10 is determined. Note that the determination of the possibility of contact in STEP 900 is the same as the processing described in FIG. 5 (STEPs 31 to 33) of Patent Document 2, and thus detailed description in this specification is omitted. When it exists in the approach determination area ahead of the host vehicle 10 and when the object enters the approach determination area with a predetermined movement vector, it is determined that there is a possibility of contact.

  On the other hand, if it is determined in STEP 800 that the object does not have a pedestrian feature, it is determined that the object is not a target for alerting, and the process returns to STEP 100 to execute “roadway / road area identification processing”. The following processing is executed again.

  If it is determined in STEP 900 that there is a possibility of contact between the target object and the host vehicle 10, the process proceeds to STEP 1000 to determine whether or not actual alerting should be performed. Note that the determination as to whether or not the actual alerting in STEP 1000 should be performed is the same as the processing described in FIG. 3 (STEP 18) of Patent Document 2, and therefore detailed description in this specification is omitted. However, by determining whether or not the driver of the host vehicle 10 is performing a brake operation, it is determined whether or not a warning output is to be performed.

  On the other hand, if it is determined in STEP 900 that the possibility of contact between the target object and the host vehicle 10 is low, it is determined that the target object is not a target for alerting, and the process returns to STEP 100 and the “roadway / road area” The process following the “identification process” is executed again.

  Furthermore, if it is determined in STEP 1000 that it is necessary to call attention, the process proceeds to STP 1100 and outputs a warning. Specifically, a voice alert is output via the speaker 6 and an image of the object is highlighted in the reference image of the image display device 7 to alert the driver to the object. To do. Then, the process returns to STEP 100 and the processes after the “roadway / road area identification process” are executed again.

  On the other hand, if it is determined in STEP 1000 that there is no need to perform actual alerting, the process returns to STEP 100 without executing the alerting output, and the processing below the “roadway / road area identification process” is executed again.

  The above is the overall operation of the periphery monitoring device of the present embodiment.

  Next, specific contents of the “roadway / road area identification process” will be described with reference to the flowchart shown in FIG.

  First, the image processing unit 1 acquires a visible image by the visible camera 2 ′ (STEP 101). Here, the acquired visible image is a grayscale image obtained by A / D converting the output signal of the visible camera 2 ′, and is stored in the image memory.

  Subsequently, lane mark detection is performed to detect a travel line (lane mark) such as a white line on the visible image (STEP 102). Specifically, an edge point in the visible image is extracted by a differential filter or the like, and a straight line component corresponding to a traveling division line such as a white line is detected as a lane mark from the edge point.

  For example, in STEP 101, when the visible image shown in FIG. 5A is acquired, the visible image is subjected to a differentiation process to generate an edge image, and the edge image is subjected to a well-known Hough transformation. A straight line component corresponding to the travel line is extracted. When a straight line component corresponding to a travel line such as a white line is extracted, the lane mark is detected as indicated by a circle in FIG. 5B by detecting the straight line component as a lane mark. can do.

  Next, the image processing unit 1 determines whether or not a lane mark is detected in STEP 102 (STEP 103).

  When a lane mark is detected (when the determination result in STEP 103 is YES), the process proceeds to STEP 104, and a road area is set based on the position of the lane mark detected in STEP 102. Then, proceeding to STEP 105, the flag i is set (i = 1).

  On the other hand, when it is determined in STEP 103 that no lane mark has been detected (when the determination result in STEP 103 is NO), the process branches to STEP 106 and the flag i is reset (i = 0).

  When the flag i is set or reset (STEPs 105 and 106), the image processing unit 1 acquires an infrared image by the infrared cameras 2R and 2L (STEP 107). The infrared image is a grayscale image obtained by A / D converting the infrared image that is the output signal of each of the infrared cameras 2R and 2L, and is stored in the image memory. Hereinafter, an image obtained from the infrared camera 2R is referred to as a right image, an image obtained from the infrared camera 2L is referred to as a left image, and in particular, the right image is referred to as a reference image. In addition, the reference image is a right image in the present embodiment, but may be a left image.

  Subsequently, the image processing unit 1 detects a road area from the infrared image (STEP 108). Specifically, an edge point in the reference image is extracted by a differential filter or the like, and a road area is detected in a front area of the host vehicle 10 from the edge point.

  For example, when the infrared image shown in FIG. 6A is acquired as a reference image in STEP 107, as in the lane mark detection in STEP 102 described above, the reference image is subjected to a differentiation process to generate an edge image. A straight line component corresponding to the end of the road region is extracted from the edge image by a well-known Hough transform. Then, when a straight line component corresponding to the end of the road area is extracted, the road area is detected as indicated by a circle in FIG. 6B by detecting the straight line component as the end of the road area. Can be detected.

  Next, the image processing unit 1 determines whether or not a road area is detected in STEP 108 (STEP 109).

  When the road area is detected (when the determination result in STEP 109 is YES), the inside of the end position of the road area detected in STEP 108 is set as the road area (STEP 110). Then, the process proceeds to STEP 111, the flag j is set (j = 1), and the “roadway / road area identification process” is terminated.

  On the other hand, when the road area is not detected (when the determination result in STEP 109 is NO), the process branches to STEP 112, the flag j is reset (j = 0), and the “roadway / road area identification process” is performed. finish.

In addition, the structure which performs the process of STEP101-106 by the image processing unit 1 corresponds to the road area identification means of this invention, and the structure which performs the process of STEP107-112 corresponds to the road area identification means of this invention. .

  Next, the specific contents of the “object extraction process” (STEP 200 in FIG. 3) will be described according to the flowchart shown in FIG.

  First, the image processing unit 1 acquires an infrared image by the infrared cameras 2R and 2L (STEP 201).

  Next, the image processing unit 1 binarizes the reference image (right image) (STEP 202). In this binarization process, the luminance value of each pixel of the reference image is compared with a predetermined luminance threshold value, and an area (relatively bright area) having a luminance value higher than the predetermined luminance threshold value in the reference image. This is a process of setting “1” (white) and setting an area having a luminance value lower than the luminance threshold (a relatively dark area) to “0” (black). Hereinafter, an image (monochrome image) obtained by the binarization process is referred to as a binarized image. In the binarized image, an area “1” is referred to as a high luminance area. The binarized image is stored in the image memory separately from the grayscale image (right image and left image).

  Next, the image processing unit 1 performs the processing of STEPs 203 to 205 on the binarized image, and extracts an object (more precisely, an image portion corresponding to the object) from the binarized image. . That is, the pixel group constituting the high luminance region of the binarized image is classified into lines having a width of one pixel in the vertical direction (y direction) of the reference image and extending in the horizontal direction (x direction). Then, each line is converted into run-length data composed of coordinates and length (number of pixels) of the position (two-dimensional position on the reference image) (STEP 203). Of the lines represented by the run length data, labels (identifiers) are attached to each of the line groups that overlap in the vertical direction of the reference image (STEP 204), and each of the line groups is extracted as an object. (STEP 205).

  Note that the objects extracted by the processing of STEPs 203 to 205 include not only people (pedestrians) but also artificial structures such as other vehicles. In addition, a plurality of local parts of the same object may be extracted as the target object.

  Next, the image processing unit 1 obtains the position of the center of gravity of each object extracted as described above (position on the reference image), the area, and the aspect ratio of the circumscribed rectangle (STEP 206). The area is calculated by accumulating the length of the run length data for the same object, and the coordinates of the center of gravity are the x coordinate of the line that bisects the area in the x direction and the y of the line that bisects the y direction. Calculated as coordinates, the aspect ratio is calculated as the aspect ratio of the circumscribed rectangle of the line converted to run length data. Note that the position of the center of gravity G may be substituted by the position of the center of gravity of the circumscribed rectangle.

  Next, the image processing unit 1 tracks the target object extracted in STEP 205, that is, recognizes the same target object every calculation processing period of the image processing unit 1 (STEP 207). In this process, when the object A is extracted by the processing of STEP 205 at a time (discrete system time) k in a certain arithmetic processing cycle, and the object B is extracted by the processing of STEP 205 at time k + 1 of the next arithmetic processing cycle. The identity of these objects A and B is determined. This identity determination may be performed based on, for example, the shape and size of the objects A and B on the binarized image, the correlation of the luminance distribution on the reference image (grayscale image), and the like. . When it is determined that the objects A and B are the same, the label of the object B extracted at time k + 1 (the label attached in STEP 204) is changed to the same label as the label of the object A. .

  Next, the image processing unit 1 reads the outputs (vehicle speed detection value and yaw rate detection value) of the vehicle speed sensor 4 and the yaw rate sensor 5 (STEP 208). In STEP 208, the turning angle (azimuth angle) of the host vehicle 10 is also calculated by integrating the read detection value of the yaw rate.

  On the other hand, the image processing unit 1 executes the processing of STEP 210 in parallel with the processing of STEPs 207 and 208. The processing in STEPs 210 to 212 is processing for obtaining the distance from the subject vehicle 10 of each object extracted in STEP 205. The process is schematically described. First, an area corresponding to each object (for example, a circumscribed square area of the object) is extracted as a search image R1 from the reference image (STEP 210).

  Next, in the left image, a search area R2 that is an area for searching for the same object as the object included in the search image R1 of the right image is set as the left image, and the search image R1 is set in the search area R2. Is extracted as a corresponding image R3 that is an image corresponding to the search image R1 (an image equivalent to the search image R1) (STEP 211). In this case, an area having a luminance distribution that most closely matches the luminance distribution of the search image R1 of the right image is extracted as the corresponding image R3 from the search area R2 of the left image. Note that the processing of STEP 211 is performed using a grayscale image, not a binarized image.

  Next, the number of pixels of the difference between the lateral position (x-direction position) of the center of gravity of the search image R1 in the right image and the lateral position (x-direction position) of the center of gravity of the corresponding image R3 in the left image is defined as parallax Δd. The distance z of the object from the host vehicle 10 (the distance in the front-rear direction of the host vehicle 10) is calculated using the parallax Δd (STEP 212). The distance z is calculated by the following equation (1).

z = (f × D) / (Δd × p) (1)

Note that f is the focal length of the infrared cameras 2R and 2L, D is the base length (interval of the optical axis) of the infrared cameras 2R and 2L, and p is the pixel pitch (length of one pixel).

  The above is the outline of the processing of STEPs 210 to 212. Note that the processing in STEPs 210 to 212 is executed for each object extracted in STEP 205.

  After the processing of STEP 208 and STEP 212 is completed, the image processing unit 1 next calculates a real space position that is a position (relative position with respect to the host vehicle 10) of each object in the real space (STEP 213). Here, as shown in FIG. 2, the real space position is a position (X, Y, Z) in the real space coordinate system (XYZ coordinate system) set with the midpoint of the attachment position of the infrared cameras 2R, 2L as the origin. ). The X direction and the Y direction of the real space coordinate system are the vehicle width direction and the vertical direction of the host vehicle 10, respectively. These X direction and Y direction are the x direction (lateral direction) and y of the right image and the left image, respectively. It is the same direction as the direction (vertical direction). The Z direction in the real space coordinate system is the front-rear direction of the host vehicle 10. Then, the real space position (X, Y, Z) of the object is calculated by the following equations (2), (3), (4).

X = x × z × p / f (2)
Y = y × z × p / f (3)
Z = z (4)

Note that x and y are the x-coordinate and y-coordinate of the object on the reference image.

  Next, the image processing unit 1 compensates for the influence of the change in the turning angle of the host vehicle 10 and increases the accuracy of the real space position of the target object, so that the real space position (X, Y, Z) of the target object is increased. The position X in the X direction is corrected according to the time-series data of the turning angle obtained in STEP 208 from the value obtained by the above formula (2) (STEP 214). Thereby, the real space position of the object is finally obtained. In the following description, “the real space position of the object” means the real space position of the object subjected to this correction.

  Next, the image processing unit 1 obtains a movement vector of the object relative to the host vehicle 10 (STEP 215). Specifically, a straight line that approximates time-series data in a predetermined period (a period from the current time to a predetermined time) of the real space position of the same object is obtained, and the straight line at the time before the predetermined time is obtained. A vector from the position (point) of the object toward the position (point) of the object on the straight line at the current time is obtained as a movement vector of the object. This movement vector is proportional to the relative velocity vector of the object with respect to the host vehicle 10.

The above is the details of the “object extraction process” in STEP 200. In addition, the structure which performs the process of STEP200 by the image processing unit 1 is equivalent to the target object extraction means of this invention .

  Moreover, the structure which performs the process of STEP213 by the image processing unit 1 is equivalent to the position calculation means of this invention.

  Next, specific contents of the “artificial structure determination process (I)” (STEP 500 in FIG. 3) will be described with reference to the flowchart shown in FIG.

  First, the image processing unit 1 determines whether or not the object has been extracted from the road area set by the “roadway / road area identification process” (STEP 501).

  Specifically, for example, when the image shown in FIG. 9 is acquired as the infrared image, a part of the object Q where a certain luminance value continues or a part of the same distance is set as the range of the same object. The lower end q of Q is determined. Then, it is determined whether or not the object Q is extracted from the road area ARE based on whether or not the lower end q is located in the road area ARE. For example, in the case shown in FIG. 9, since the lower end q of the object Q is located in the road area ARE, it is determined that the object Q is extracted from the road area ARE.

  In the present embodiment, the image processing unit 1 detects the lower end portion of the object on the reference image, and determines whether the object is within the road area based on whether the lower end part is located in the road area. Although it is determined whether or not it is an extracted one, the real space position (X, Y, Z) of the target object in the road region is calculated from the above equations (2), (3), (4). It may be determined whether or not the object is extracted from the road area depending on whether it is located above the corresponding area in real space (on the XZ coordinate axis).

  Next, when it is determined in STEP 501 that the object has been extracted from the road area (when the determination result in STEP 501 is YES), the flow proceeds to STEP 502 and the value of the flag (i) (i = 1 or 0) ) To determine whether or not a road area is set.

  In STEP502, when the roadway area is not set (when i = 0), the process proceeds to STEP503, and the artificial structure determination based on the first artificial structure determination condition is performed.

  The first artificial structure determination condition is that the approximation level indicating the correlation between the image (template) of a specific artificial structure registered in a memory (not shown) and the image of the target is equal to or higher than a predetermined level. It is set to be determined as an artificial structure. Here, the image of the specific artificial structure that is registered in the memory in advance is an image of the artificial structure that may exist in the road area. For example, a vending machine, a utility pole, a streetlight transformer, and a blinker light And images of vehicles.

  Specifically, the correlation between the image of the specific artificial structure registered in advance and the image of the object is determined by template matching. In template matching, when two images are given, they are overlapped and their correlation (or difference) is detected to determine the approximate level of the two images, and the correlation between the two images varies. It is expressed with a simple scale. For example, a method using a residual absolute value sum (SAD) of luminance values can be used as a measure representing correlation.

  When calculating the approximate level with the image of the object using the vending machine image shown in FIG. 10 as a template using the sum of absolute values of residuals of luminance values (SAD), the vending machine image is obtained in advance. As shown in FIGS. 11A and 11B, luminance spectra in the x-axis direction and the y-axis direction passing through the center of gravity of the vending machine are registered on the A / D converted grayscale image. Then, for the object extracted in the object extraction process (STEP 200), luminance spectra in the x-axis direction and the y-axis direction passing through the gravity center position of the object calculated in STEP 206 are obtained, and the brightness of the template is obtained. The sum of absolute values of residuals between the spectrum and the luminance spectrum of the object can be calculated as the approximate level. Since the approximation level in this case is based on the residual, the approximation level increases as the absolute value sum of the residuals decreases.

  When the approximate level calculated in this way is equal to or greater than a predetermined value (when the sum of absolute values of residuals is equal to or less than a predetermined value), the target is determined to be an artificial structure (vending machine). .

  In addition, for utility poles, streetlight transformers, and the like, artificial structure determination based on the first artificial structure determination condition can be performed in the same manner as in the above-described vending machine.

  Note that the method of pattern matching is not limited to the above-described method based on the sum of residual absolute values of luminance values (SAD) .For example, the normalized cross-correlation at each pixel position between the template and the image of the target object. You may use the normalization cross correlation method etc. which calculate a value as an approximation level.

  In the artificial structure extraction process, the artificial structure may be determined by a method other than pattern matching.

  Furthermore, in this embodiment, the artificial structure determination is performed using the image of the artificial structure as the characteristic amount of the specific artificial structure registered in the memory in advance. For example, a value obtained by measuring an actual dimension of a specific artificial structure may be used as the feature amount. In this case, the approximate level can be calculated by comparing the size of the object in the real space calculated via the image processing unit 1 with the size of the specific artificial structure registered in advance in the memory. .

  If it is determined in STEP 503 that the object is an artificial structure based on the first artificial structure determination condition (when the determination result in STEP 503 is YES), the process proceeds to STEP 511 and the flag k is set. Set (k = 1) and end the “artificial structure determination process (I)”.

  On the other hand, if it is determined in STEP 503 that the object is not an artificial structure based on the first artificial structure determination condition (when the determination result in STEP 503 is NO), the process branches to STEP 512, and the flag k is set. Reset (k = 0) and end the “artificial structure determination process (I)”.

  In STEP 502, when the road area is set (when i = 1), the process proceeds to STEP 504, where the object is extracted from the road area set by the “road / road area identification process”. It is determined whether or not (STEP 504). Whether or not the object has been extracted from the road area is determined by the same method as STEP 501 using a visible image corresponding to the infrared image shown in FIG.

  Next, when it is determined in STEP 504 that the object has been extracted from the roadway region (when the determination result in STEP 504 is YES), the process proceeds to STEP 505 and the artificial structure based on the third artificial structure determination condition Object judgment is performed.

  The third artificial structure determination condition is that an image of an artificial structure such as a vehicle that may exist in the roadway area is registered in advance in a memory (not shown), and the image of the artificial structure and the target It is set so that it is determined as an artificial structure when the approximation level indicating the correlation with the image of the object is equal to or higher than a predetermined level.

  Note that the correlation between the image of the specific artificial structure registered in advance and the image of the target object may be determined by template matching, but in particular, when performing an artificial structure determination process for a vehicle. May perform the processing described in FIG. 8 (STEPs 41 to 54) of Patent Document 2. That is, when determining whether or not the object is a vehicle, first, a binarized object corresponding to the taillight and existing at a symmetric position is searched, and then two binarized objects. A search area for searching for the presence or absence of the window shield is set in the upper part between the objects. Then, when the binarized object and the object in the search area have the same distance, it is determined whether or not the average luminance value in the search area is smaller than the specified value, and the average luminance value in the search area is smaller than the specified value. In this case, it is determined that the object is a vehicle.

  In STEP 505, when it is determined that the target object is an artificial structure based on the third artificial structure determination condition (when the determination result in STEP 505 is YES), the process proceeds to STEP 511 and the flag k is set. Set (k = 1) and end the “artificial structure determination process (I)”.

  On the other hand, when it is determined in STEP 505 that the object is not an artificial structure based on the first artificial structure determination condition (when the determination result in STEP 505 is NO), the process branches to STEP 512, and the flag k is set. Reset (k = 0) and end the “artificial structure determination process (I)”.

  If it is determined in STEP 504 that the object is not extracted from the roadway region (when the determination result in STEP 504 is NO), the process branches to STEP 506 and the fourth artificial structure determination condition is satisfied. Based on the artificial structure determination is performed.

  In the fourth artificial structure determination condition, images of vending machines, utility poles, streetlight transformers, blinker lights, vehicles, etc. that may exist outside the roadway area in the road area are registered in the memory in advance. When the approximate level indicating the correlation between the image of the artificial structure and the image of the object is equal to or higher than a predetermined level, the artificial structure is determined to be determined. As described above, the correlation between the image of the specific artificial structure registered in advance and the image of the object is determined by template matching or the like.

  If it is determined in STEP 506 that the object is an artificial structure based on the fourth artificial structure determination condition (when the determination result in STEP 506 is YES), the process proceeds to STEP 511 and the flag k is set. Set (k = 1) and end the “artificial structure determination process (I)”.

  On the other hand, if it is determined in STEP 506 that the object is not an artificial structure based on the fourth artificial structure determination condition (when the determination result in STEP 506 is NO), the process branches to STEP 512 and flag k is set. Reset (k = 0) and end the “artificial structure determination process (I)”.

  Further, when it is determined in STEP 501 that the object is not extracted from the road area (when the determination result in STEP 501 is NO), the process branches to STEP 507 and the second artificial structure determination condition is satisfied. Based on the artificial structure determination is performed.

  The second artificial structure determination condition is an image of an artificial structure such as a vending machine, a power pole, a streetlight transformer, a blinker light, a wall, a tree, or a vehicle that may exist outside the road area in advance. (Not shown), and is set to be determined as an artificial structure when the approximate level indicating the correlation between the image of the specific artificial structure and the image of the target is equal to or higher than a predetermined level. Yes. As described above, the correlation between the image of the specific artificial structure registered in advance and the image of the object is determined by template matching or the like.

  In STEP 507, when it is determined that the target object is an artificial structure based on the second artificial structure determination condition (when the determination result in STEP 507 is YES), the process proceeds to STEP 511 and the flag k is set. Set (k = 1) and end the “artificial structure determination process (I)”.

  On the other hand, if it is determined in STEP 507 that the object is not an artificial structure based on the second artificial structure determination condition (when the determination result in STEP 507 is NO), the process branches to STEP 512, and the flag k is set. Reset (k = 0) and end the “artificial structure determination process (I)”.

  The above is the details of the “artificial structure determination process (I)”. Thus, by setting the first artificial structure determination condition to the fourth artificial structure determination condition depending on whether or not the object is extracted from within the road region, the object extracted from the image becomes the artificial structure. It is possible to determine whether or not the object is efficient and accurate.

  Next, an outline of the “artificial structure determination process (II)” (STEP 600 in FIG. 3) will be described according to the flowchart shown in FIG.

  In the “artificial structure determination process (II)”, the image processing unit 1 first determines whether or not a roadway area is set based on the value of the flag (i) (STEP 601).

  In STEP601, when the roadway area is set (when i = 1), the process proceeds to STE602 and whether or not the object is extracted from the roadway area set by the “roadway / road area identification process”. Is determined (STEP 602). Note that whether or not the object has been extracted from the road area is determined in the same manner as in STEP 504 described above.

  Next, when it is determined in STEP 602 that the object has been extracted from the roadway region (when the determination result in STEP 602 is YES), the process proceeds to STEP 603 and the artificial structure based on the third artificial structure determination condition Object judgment is performed. The third artificial structure determination condition is the same processing as in STEP 505 described above, and when it is determined that the target object is an artificial structure based on the third artificial structure determination condition (the determination result of STEP 603 is In the case of YES), the process proceeds to STEP 611, the flag k is set (k = 1), and the “artificial structure determination process (II)” is ended.

  On the other hand, when it is determined in STEP 603 that the object is not an artificial structure based on the first artificial structure determination condition (when the determination result in STEP 603 is NO), the process branches to STEP 612 and flag k is set. Reset (k = 0) to end the “artificial structure determination process (II)”.

  If it is determined in STEP 602 that the object is not extracted from the roadway area (when the determination result in STEP 602 is NO), the process branches to STEP 604 and the fifth artificial structure determination condition is satisfied. Based on the artificial structure determination is performed.

  The fifth artificial structure determination condition is an image of an artificial structure such as a vending machine, a power pole, a streetlight transformer, a blinker light, a wall, a tree, or a vehicle that may exist outside the roadway area in advance. (Not shown), and is set to be determined as an artificial structure when the approximate level indicating the correlation between the image of the specific artificial structure and the image of the target is equal to or higher than a predetermined level. Yes. As described above, the correlation between the image of the specific artificial structure registered in advance and the image of the object is determined by template matching or the like.

  If it is determined in STEP 604 that the object is an artificial structure based on the fifth artificial structure determination condition (when the determination result in STEP 604 is YES), the process proceeds to STEP 611 and the flag k is set. Set (k = 1) and end the “artificial structure determination process (II)”.

  On the other hand, if it is determined in STEP 604 that the object is not an artificial structure based on the fifth artificial structure determination condition (when the determination result in STEP 604 is NO), the process branches to STEP 612 and flag k is set. Reset (k = 0) to end the “artificial structure determination process (II)”.

  In STEP 605, when the road area is not set (when i = 0), the process proceeds to STEP 605, and the artificial structure determination based on the sixth artificial structure determination condition is performed.

  The sixth artificial structure determination condition is that an image of an artificial structure generally existing around the road is registered in the memory regardless of the inside or outside of the road area and the inside or outside of the road area. It is set so that it is determined as an artificial structure when the approximation level indicating the correlation with the image of the object is equal to or higher than a predetermined level. As described above, the correlation between the image of the specific artificial structure registered in advance and the image of the object is determined by template matching or the like. The determination condition for the sixth artificial structure may be the same as the determination condition for the second or fifth artificial structure.

  If it is determined in STEP 605 that the target object is an artificial structure based on the sixth artificial structure determination condition (when the determination result in STEP 605 is YES), the process proceeds to STEP 611 and flag k is set. Set (k = 1) and end the “artificial structure determination process (II)”.

  On the other hand, if it is determined in STEP 605 that the object is not an artificial structure based on the sixth artificial structure determination condition (when the determination result in STEP 605 is NO), the process branches to STEP 612 and flag k is set. Reset (k = 0) to end the “artificial structure determination process (II)”.

The above is the outline of the “artificial structure determination process (II)” in STEP600. The configuration by the image processing unit 1 executes the processing of STEP501~512,601~612 corresponds to an artificial structure determination means of the present invention.

  In this embodiment, the lane mark is detected from the visible image (STEPs 101 to 105), and then the road region is detected from the infrared image (STEPs 107 to 112), based on the visible images of STEPs 101 to 105. It is good also as a structure (STEP107-112) which abbreviate | omits a process and detects a road area | region from an infrared image. In this case, in the artificial structure determination process (I) and the artificial structure determination process (II), the processes (STEP 502, 504 to 506, 601 to 604) based on the presence or absence of setting of the roadway region are omitted.

  Conversely, the processing based on the infrared images in STEPs 107 to 112 may be omitted and only the configuration (STEPs 101 to 105) for detecting the road area from the visible image may be employed. In this case, the process (STEP 400, 500) based on the presence / absence of setting of the road region is omitted, and only the second artificial structure determination process (II) is executed.

  In this embodiment, the road area is detected from the visible image of the visible camera 2 ′ and the road area is detected from the infrared images of the infrared cameras 2R and 2L. However, the present invention is not limited to this. A road area or a road area may be detected from an image captured by the imaging means.

  Note that, in the present embodiment, it is configured to perform a predetermined alert based on the processing result of the image processing unit 1, but the vehicle behavior may be controlled based on the processing result.

  In the above embodiment, the two infrared cameras 2R and 2L are provided. However, when the distance to the object is detected by a radar or the like, one infrared camera 2R or 2L is connected to the host vehicle 10. You may make it mount in.

The figure which shows the whole structure of one Embodiment of the periphery monitoring apparatus of the vehicle of this invention. The perspective view of the vehicle provided with the periphery monitoring apparatus of FIG. 3 is a flowchart showing processing of an image processing unit provided in the periphery monitoring device of FIG. 1. 3 is a flowchart showing processing of an image processing unit provided in the periphery monitoring device of FIG. 1. The figure which shows the captured image in this embodiment exemplarily. The figure which shows the captured image in this embodiment exemplarily. 3 is a flowchart showing processing of an image processing unit provided in the periphery monitoring device of FIG. 1. 3 is a flowchart showing processing of an image processing unit provided in the periphery monitoring device of FIG. 1. The figure which shows the captured image in this embodiment exemplarily. The figure for demonstrating the process of an image processing unit. The figure for demonstrating the process of an image processing unit. 3 is a flowchart showing processing of an image processing unit provided in the periphery monitoring device of FIG. 1.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing unit (Road area identification means, object extraction means, artificial structure determination means, roadway area identification means, position calculation means), 2R, 2L ... Infrared camera (imaging means), 2 '... Visible camera (visible) (Image capturing means), STEP 101-106 ... Processing by roadway area identifying means, STEP107-112 ... Processing by road area identifying means, STEP200 ... Processing by object extracting means, STEP213 ... Processing by position calculating means, STEP500, 600 ... Artificial structure Processing by the object judgment means.

Claims (8)

An object extraction means for extracting an object from an image obtained by an imaging means mounted on the vehicle, and an artificial structure determination means for determining whether or not the object is an artificial structure, A vehicle periphery monitoring device for monitoring objects existing in the vicinity,
Road area identifying means for identifying a road area from the image;
Road area identification means for identifying a road area that is an area included in the road area and defined based on a position of a lane mark from the image ,
The artificial structure determination means includes
When the object is extracted in the road area identified by the road area identification means by the object extraction means, the object is an artificial structure based on the first artificial structure determination condition. If the object is extracted outside the road area, the object is artificially determined based on a second artificial structure determination condition different from the first artificial structure determination condition. Determine whether it is a structure ,
When an object is extracted in the road area identified by the road area identification means by the object extraction means, based on a third artificial structure determination condition including at least a vehicle as an artificial structure, When it is determined whether or not the object is an artificial structure and the object is extracted outside the roadway area, at least one of a vehicle, a vending machine, a streetlight transformer, and a blinker light is used as the artificial structure. A vehicle periphery monitoring device that determines whether or not the object is an artificial structure based on a fourth artificial structure determination condition including one . The vehicle periphery monitoring device according to claim 1 ,
As the imaging means, an infrared image imaging means for acquiring an infrared image,
The road area identifying means performs an edge extraction process for extracting an edge part of an infrared image acquired by the infrared image capturing means, and identifies a road area by the edge part extracted by the edge extraction process. A vehicle periphery monitoring device. In the vehicle periphery monitoring device according to claim 1 or 2 ,
The artificial structure determination unit detects a lower end portion of the object extracted from the image, and determines that the object is extracted in the road region when the lower end portion is located in the road region. The vehicle periphery monitoring apparatus characterized by the above-mentioned. In the vehicle periphery monitoring device according to claim 1 or 2 ,
A position calculating means for calculating a position in real space corresponding to an image portion included in the image;
The artificial structure determination means determines that the object is extracted in the road area when the position of the object extracted from the image is above the road area in the real space. A vehicle periphery monitoring device. In the vehicle periphery monitoring device according to any one of claims 1 to 4 ,
As the imaging means, provided with a visible image imaging means for obtaining a visible image
The vehicle roadway area identifying unit detects a roadway lane mark from the visible image acquired by the visible image capturing unit, and identifies the roadway area by the lane mark. In the vehicle periphery monitoring device according to any one of claims 1 to 5 ,
The artificial structure determination means detects a lower end portion of the object extracted from the image, and determines that the object is extracted in the road area when the lower end portion is located in the road area. The vehicle periphery monitoring apparatus characterized by the above-mentioned. In the vehicle periphery monitoring device according to any one of claims 1 to 5 ,
A position calculating means for calculating a position in real space corresponding to an image portion included in the image;
The artificial structure determination means determines that the object is extracted in the road area when the position of the object extracted from the image is above the road area in the real space. A vehicle periphery monitoring device. In the vehicle periphery monitoring device according to any one of claims 1 to 7,
As the first artificial structure determination condition, the approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist in the road region is equal to or higher than a predetermined level. Is set,
As the second artificial structure determination condition, the approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist outside the road region is equal to or higher than a predetermined level. Is set,
As the third artificial structure determination condition, the approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist in the roadway region is equal to or higher than a predetermined level. Is set,
As the fourth artificial structure determination condition, there is an approximate level between the feature amount of the image of the object and the feature amount of a specific artificial structure that may exist outside the road area in the road area. Set to be above a certain level,
As a feature amount of the specific artificial structure, at least one feature amount of a vehicle, a vending machine, a utility pole, a streetlight transformer, a blinker light, a wall, and a tree is registered. Monitoring device.