JP5885640B2 - Vehicle periphery monitoring device - Google Patents

Description

  The present invention relates to a vehicle periphery monitoring device that monitors animals existing around a vehicle based on an image captured by a camera.

  Conventionally, a vehicle periphery monitoring device has been proposed in which an animal image portion is extracted from an image around a vehicle captured by an infrared camera mounted on the vehicle to monitor animals appearing around the vehicle ( For example, see Patent Document 1).

  In the vehicle periphery monitoring device described in Patent Document 1, an image portion of the same object is extracted from a time-series captured image, tracking is performed between time points, and legs of the same object between time-series captured images are detected. It is determined whether or not the object is an animal by determining whether or not the width of the image portion is changing with periodicity.

Japanese Unexamined Patent Publication No. 2009-42941 (page 21, FIG. 12)

  Since the vehicle periphery monitoring device described in Patent Document 1 determines whether or not the image portion of the same object has a periodic change in a time-series captured image, the object is an animal. It takes some time to determine whether or not.

  The present invention has been made in view of such a background, and an object of the present invention is to provide a vehicle periphery monitoring device that can shorten the time required to determine whether or not an object is an animal.

The present invention has been made to achieve the above object, and the vehicle periphery monitoring device of the present invention comprises:
An animal candidate image part extraction unit that extracts, as an animal candidate image part, an image part of the same object that may be an animal from a time-series captured image captured by a camera mounted on a vehicle;
A specific part extraction unit for extracting an image part of a specific part of an animal from the animal candidate image part;
An image portion of the specific part is extracted by the specific part extraction unit between the pair of animal candidate image parts of the same object extracted by the animal candidate image part extraction unit between the time-series captured images. When the specific part extraction state and the specific part non-extraction state in which the image part of the specific part is not extracted by the specific part extraction unit are switched, the object in the real space corresponding to the pair of animal candidate image parts is An animal determination unit that determines that the animal is an animal is provided (first invention).

  According to the first invention, when the animal candidate image part is switched between the specific part extraction state and the specific part non-extraction state between time-series captured images, for example, the leg part is in a bush on a roadside. Legs, heads, etc., such as when a hidden animal jumps out on the road and a leg appears, or an animal that has lowered its head to eat grass by the road raises its head It is assumed that an action peculiar to an animal has been performed for a specific part.

  Therefore, when the specific part extraction state and the specific part non-extraction state are switched between the pair of animal candidate image parts of the same target object extracted between time-series captured images, the animal determination unit The object in real space corresponding to the pair of animal candidate image portions can be determined to be an animal. Since this determination can be performed based on two time-series captured images, the object is more likely to be an animal than a periodicity based on a certain number of time-series captured images. The time required for determining whether or not the animal is an animal can be shortened.

In the first invention,
The specific part extraction unit extracts an image part of a leg as an image part of the specific part,
The animal determination unit switches between the specific part extraction state and the specific part non-extraction state between the pair of animal candidate image parts, and the luminance of an area around the pair of animal candidate image parts. When the change is greater than or equal to a first predetermined level, it is determined that the object in real space corresponding to the pair of animal candidate image portions is an animal (second invention).

  In the second invention, the specific part is a leg, and when the animal candidate image part is switched between the specific part non-extraction state and the specific part extraction state, the animal candidate image part is an animal image part. , This animal is in the bush by the road, and when the leg is hidden in the grass, the leg appears by jumping to the road, or vice versa . In these cases, since the background of the animal changes from the roadside bush to the road or from the road to the roadside bush, the change in luminance around the animal candidate image portion increases.

  Therefore, the animal determination unit switches between the specific part non-extraction state and the specific part extraction state between the pair of animal candidate image parts, and brightness of the periphery of the pair of animal candidate image parts. By determining that the object in the real space corresponding to the pair of animal candidate image portions is an animal when the change is equal to or higher than the first predetermined level, it is possible to improve the accuracy of animal determination.

In the first invention,
The specific part extraction unit extracts an image part of a head as an image part of the specific part;
The specific part extraction unit switches between the specific part extraction state and the specific part non-extraction state between the pair of animal candidate image parts, and brightness of a region around the pair of animal candidate image parts. When the change in is less than a second predetermined level, it is determined that the object in the real space corresponding to the pair of animal candidate image portions is an animal (third invention).

  In the third aspect, the specific part is a head, and when the animal candidate image part is switched between the specific part non-extraction state and the specific part extraction state, the animal candidate image part is an animal image part. It is assumed that the animal has raised its head from a state where its head is lowered and eats grass, or conversely, when it has started raising its head from a state where its head is raised, it begins to eat grass. In these cases, since the place where the animal is located does not change, the change in luminance around the candidate animal image portion is small.

  Therefore, the animal determination unit switches between the specific part non-extraction state and the specific part extraction state between the pair of animal candidate image parts, and brightness of the periphery of the pair of animal candidate image parts. By determining that the object in the real space corresponding to the pair of animal candidate image portions is an animal when the change is less than the second predetermined level, it is possible to improve the accuracy of animal determination.

Explanatory drawing of the attachment aspect to the vehicle of a vehicle periphery monitoring apparatus. The block diagram of a vehicle periphery monitoring apparatus. The 1st operation | movement flowchart of a vehicle periphery monitoring apparatus. The 2nd operation | movement flowchart of a vehicle periphery monitoring apparatus. Explanatory drawing of the image part of an animal. Explanatory drawing of the captured image of the condition where the animal jumped out of the road side to the road. Explanatory drawing of the captured image of the condition where the animal is eating grass by the roadside.

  An example of an embodiment of a vehicle periphery monitoring device of the present invention will be described with reference to FIGS. Referring to FIG. 1, a vehicle periphery monitoring device 10 of this embodiment is used by being mounted on a vehicle 1, and the vehicle 1 includes an infrared camera 2 (corresponding to the camera of the present invention) capable of detecting far infrared rays. Is provided.

  The infrared camera 2 has sensitivity in the far-infrared region, and has a characteristic that the level of the output video signal increases (the brightness of the video signal increases) as the temperature of the object to be imaged increases. The infrared camera 2 is attached to the front portion of the vehicle 1 in order to image the front of the vehicle 1. The front portion of the vehicle 1 is the origin O, the left-right direction of the vehicle 1 is the X axis, the vertical direction is the Y axis, A real space coordinate system in which the front-rear direction is the Z axis is defined. Instead of the infrared camera 2, a camera having sensitivity in other wavelength regions such as visible light may be used.

  Next, referring to FIG. 2, the vehicle 1 includes a yaw rate sensor 31 that detects the yaw rate of the vehicle 1, a vehicle speed sensor 32 that detects the traveling speed of the vehicle 1, and a brake sensor that detects the amount of brake operation by the driver. 33, a head 6 for displaying an image picked up by the speaker 6 for performing alerting by voice and the like and the infrared camera 2 and for allowing the driver to visually recognize an object likely to contact the vehicle 1 An up display 7 (hereinafter referred to as HUD 7) is connected. As shown in FIG. 1, the HUD 7 is provided so that the screen 7 a is displayed at a front position on the driver side of the front window of the vehicle 1.

  The vehicle periphery monitoring device 10 is an electronic circuit unit configured by a CPU, a memory, and the like (not shown). By executing a vehicle periphery monitoring control program held in the memory by the CPU, the captured image acquisition unit 11, the target It functions as the object image partial extraction unit 12 (including the function of the animal candidate image partial extraction unit of the present invention), the specific part extraction unit 13, the animal determination unit 14, and the alarm processing unit 15.

  The vehicle periphery monitoring apparatus 10 extracts an image portion that satisfies a predetermined condition from an image captured by the infrared camera 2 as an image portion (object image portion) that may be an image portion of a pedestrian or an animal, and this image. Using the yaw rate of the vehicle 1 detected by the yaw rate sensor 31 and the traveling speed of the vehicle 1 detected by the vehicle speed sensor 32, the position (real space position) of the object (object) in the real space corresponding to the portion is used. Calculate and track the image portion of the same object between time-series captured images to calculate the movement vector of the object in real space.

  Then, the vehicle periphery monitoring device 10 determines whether or not the object is to be alerted based on the movement vector, and considers the operation state of the brake recognized from the detection signal of the brake sensor 33, and the like. It is determined whether or not to alert by voice output from 6 and highlighting on HUD7. Note that the alerting process based on the movement vector is performed using a conventional method described in, for example, Japanese Patent Application Laid-Open No. 2007-213561.

  In addition, the vehicle periphery monitoring device 10 identifies the type of object (vehicle / artificial structure, pedestrian, animal, etc.) and performs monitoring processing according to the type of object. Hereinafter, the object monitoring process (particularly, the animal determination process) by the vehicle periphery monitoring apparatus 10 will be described with reference to the flowchart shown in FIG. The vehicle periphery monitoring apparatus 10 repeatedly performs the process according to the flowchart shown in FIG. 3 for each predetermined control cycle, and monitors objects existing around the vehicle 1.

  STEP 1 in FIG. 3 is processing by the captured image acquisition unit 11. The captured image acquisition unit 11 inputs a video signal output from the infrared camera 2, and takes in a grayscale image obtained by converting the video signal into digital gradation (luminance) data into an image memory.

  The subsequent STEP 2 is processing by the object image partial extraction unit 12. The object image partial extraction unit 12 sets, for each pixel of the grayscale image, a pixel whose luminance is equal to or higher than a predetermined threshold as “1” (white) and a pixel whose luminance is lower than the threshold as “0” (black). A binarization process is performed to generate a binary image.

  Then, the object image portion extraction unit 12 calculates run length data of each white region in the binary image, performs a labeling process or the like, and converts the white image portion into the object image portion (animal candidate image of the present invention). Extract as part). The object image portion extraction unit 12 basically extracts an area where white pixels are adjacent to each other as one object image portion. However, one object image portion is also detected for the approaching white pixels. May be extracted in batches. In this way, the configuration in which the object image part extraction unit 12 extracts an object image part that may be an animal image part corresponds to the animal candidate image part extraction unit of the present invention.

  In subsequent STEP3 to STEP6, the vehicle periphery monitoring device 10 determines whether the object (the object in the real space corresponding to the object image portion) is a structure (another vehicle, a utility pole, etc.) or a pedestrian. To do. First, in STEP 3, the vehicle periphery monitoring device 10 determines whether or not the object is a structure. Here, the determination of whether or not the object is a structure can be performed using a known method described in, for example, Japanese Patent Application Laid-Open No. 2007-310705.

  When it is determined that the object is a structure, the process branches to STEP 9 in the next STEP 4 and the alarm processing by the alarm processing unit 15 is not executed. On the other hand, when it is determined that the object is not a structure, the process proceeds from STEP4 to STEP5.

  In STEP 5, the vehicle periphery monitoring device 10 determines whether or not the object is a pedestrian. Here, the determination of whether or not the object is a pedestrian can be performed using a known method described in the above-mentioned Japanese Patent Application Laid-Open No. 2007-310705.

  And when it determines with a target object being a pedestrian, it branches to STEP20 at the next STEP6, and it determines that there exists a high possibility that a target object and the vehicle 1 will contact from the movement vector of the target object in real space. When it is done, the alarm processing unit 15 performs alarm processing. On the other hand, when it is determined that the object is not a pedestrian, the process proceeds from STEP6 to STEP7.

  Here, the alarm processing unit 15 performs a process for displaying an emphasis frame superimposed on the image portion of the pedestrian on the screen 7a of the captured image displayed by the HUD 7 as an alarm process, and a sound for prompting attention to the pedestrian 6 Process to output from.

  The “animal recognition” in STEP 7 is processing by the specific part extraction unit 13 and the animal determination unit 14. Details of the “animal recognition” process will be described later.

  If it is determined that the object is an animal, the process branches to STEP 30 in the next STEP 8. STEP 30 is processing by the alarm processing unit 15. In the same manner as in STEP 20 above, the alarm processing unit 15 performs alarm processing for prompting the driver to call attention to animals.

  On the other hand, when it is determined that the object is not an animal, the process proceeds to STEP 9, and the vehicle periphery monitoring apparatus 10 ends the process of one control cycle according to the flowchart shown in FIG.

  Next, the “animal recognition” process in STEP 7 of FIG. 3 will be described according to the flowchart shown in FIG.

  STEP 50 in FIG. 4 is processing by the specific part extraction unit 13. The specific part extracting unit 13 determines the image portion (animal candidate image portion) of the target object that is determined not to be a structure in STEP 4 in FIG. 3 and is determined not to be a pedestrian in STEP 6 in FIG. As shown in FIG. 4, the head image portion 21, the torso image portion 22, and the leg image portion (the front leg image portion 23, the rear leg image portion 24) of the animal candidate image portion 20 are extracted. Execute. Note that the head and legs correspond to specific parts of the present invention.

  Here, the extraction of the image portion of the head, the trunk, and the leg is, for example, publicly known based on the relative positional relationship of each part, the shape of each part, and the like described in Japanese Patent Application Laid-Open No. 2007-310705. This can be done using techniques.

  Subsequent STEP 51 and subsequent steps are processing by the animal determination unit 14. In STEP 51, the animal determination unit 14 determines whether or not all the image portions of the head, torso, and legs have been extracted from the candidate animal image portion by the specific part extraction unit 13. When all the parts have been extracted, the process proceeds to STEP 52, where it is determined that this animal candidate image part is an animal image part (the corresponding object in the real space is an animal).

  On the other hand, in STEP 50, when the specific part extraction unit 13 does not extract all the image parts of the head, torso, and legs from the animal candidate image part (for example, as shown in FIG. 5B, the animal candidate image When only the image portions of the head portion 31 and the trunk portion 32 of the portion 30 are extracted and the image portions of the legs 33 and 34 are not extracted), the process branches from STEP 51 to STEP 60. In STEP 60, the animal determination unit 14 determines whether the image portion of the head and torso or torso and leg has been extracted by the specific part extraction unit 13 in a predetermined number of control cycles (for example, three or more consecutive control cycles). Judge whether or not.

  If the head and torso or torso and leg image portions are extracted by the specific part extraction unit 13 in a predetermined number of control cycles or more, the process proceeds to STEP 52 described above, and if not extracted, the process branches to STEP 70.

  In STEP 70, the animal determination unit 14 determines an animal candidate image portion between a pair of image portions (animal candidate image portions) of the same object extracted between the captured image in the previous control cycle and the captured image in the current control cycle. It is determined whether or not the state in which the leg image portion is extracted from (the specific part extraction state) and the state in which the leg image portion is not extracted from the animal candidate image portion (the specific part non-extraction state) have been switched.

  Here, FIG. 6A illustrates the captured image Im1 of the infrared camera 2 in a certain control cycle, and includes an image portion 40a of an animal (for example, a wild animal such as a deer) in a bush by the road. It is. The leg of this animal is hidden by bushes, and only the image part 42a of the trunk is extracted from the image part 40a, and the image part of the head and legs is not extracted.

  FIG. 6B illustrates the captured image Im2 of the infrared camera 2 in the next control cycle of FIG. 6A. An image portion 40b of an animal that has jumped into the road from a bush by the road is shown in FIG. include. The image portion 40b includes leg image portions 43b and 44b that are hidden in the image portion 40a of FIG. 6A in addition to the body image portion 42b.

  The animal determination unit 14 sets a right luminance determination area 50a on the right side of the image portion 40a in FIG. 6A and sets a left luminance determination area 51a on the left side of the image portion 40a. Similarly, the animal determination unit 14 sets the right luminance determination area 50b on the right side of the image portion 40b in FIG. 6B and sets the left luminance determination area 51b on the left side of the image portion 40b.

  As shown in FIG. 6A and FIG. 6B, for the image portions 40a and 40b of the same object extracted from the time-series captured images Im1 and Im2, the leg image portion is not extracted. When the part non-extraction state (FIG. 6A) is switched to the specific part extraction state in which the leg image portions 43b and 44b are extracted (FIG. 6B), the image portions 40a and 40b are images of animals. It can be determined that there is a high possibility of being a part.

  Therefore, if there is a change in the presence or absence of extraction of the leg image part in STEP 70, the process proceeds to STEP 71, and if there is no change in the presence or absence of extraction of the leg image part, the process branches to STEP 80. In STEP 71, the animal determination unit 14 determines whether or not the surrounding luminance change is equal to or higher than the first predetermined level between the pair of animal candidate image portions.

  6 (a) and 6 (b), the right luminance determination area 50a and the left luminance determination area 51a set in FIG. 6 (a) are bush image portions, whereas FIG. The right luminance determination area 50b and the left luminance determination area 51b set in (2) are road image portions. Therefore, the luminance change due to the temperature difference between the right luminance determination area 50a and the left luminance determination area 51a in FIG. 6A and the right luminance determination area 50b and the left luminance determination area 51b in FIG. 6B is large. Become.

  Therefore, the luminance of the right luminance determination area 50a and the left luminance determination area 51a in FIG. 6A (for example, the average luminance of both the areas 50a and 51a), and the right luminance determination area 50b and the left luminance determination in FIG. A change in luminance due to temperature between the luminance of the area 51b (for example, the average luminance of both the areas 50b and 51b) increases.

  Therefore, in STEP 71, when the luminance change around the pair of animal candidate image portions of the same object between time-series captured images is equal to or higher than the first predetermined level, the pair of animal candidate image portions is an animal image portion. It can be determined that there is a higher possibility. Therefore, in this case, the process proceeds to STEP 52, and the animal determination unit 14 determines that the object in the real space corresponding to the pair of animal candidate image portions is an animal.

  On the other hand, when the brightness change around the pair of candidate animal images of the same object between the time-series captured images is less than the first predetermined level in STEP 71, the process branches to STEP 90, and the animal determination unit 14 It is determined that the object in the real space corresponding to the animal candidate image portion is not an animal.

  Next, in STEP 80, the animal determination unit 14 starts from the animal candidate image portion between the pair of animal candidate image portions of the same object extracted between the captured image in the previous control cycle and the captured image in the current control cycle. It is determined whether or not the state in which the head image part is extracted (specific part extraction state) and the state in which the head image part is not extracted from the animal candidate image part (specific part non-extraction state) have been switched.

  Here, FIG. 7A illustrates a captured image Im3 of the infrared camera 2 in a certain control cycle, and includes an image portion 60a of an animal (for example, a wild animal such as a deer) in a bush by the road. It is. The leg of this animal is hidden by bushes, and only the image part 62a of the trunk is extracted from the image part 60a, and the image part of the head and legs is not extracted.

  FIG. 7B shows a captured image Im4 of the infrared camera 2 in the next control cycle of FIG. 7A, and includes an animal image portion 60b. An image portion 60a in FIG. 7A and an image portion 60b in FIG. 7B are image portions of the same object. The image portion 60b includes an image portion 61b of the head that is hidden in the image portion 60a of FIG. 7A in addition to the image portion 62b of the trunk.

  Further, the animal determination unit 14 sets the right luminance determination area 70a on the right side of the image portion 60a in FIG. 7A and sets the left luminance determination area 71a on the left side of the image portion 60a. Similarly, the animal determination unit 14 sets a right luminance determination area 70b on the right side of the image portion 60b in FIG. 7B and sets a left luminance determination area 71b on the left side of the image portion 60b.

  As shown in FIGS. 7A and 7B, the head image portion is not extracted for the image portions 60a and 60b of the same object extracted from the time-series captured images Im3 and Im4. When the part non-extraction state (FIG. 7A) is switched to the specific part extraction state in which the head image portion is extracted (FIG. 7B), the image portions 60a and 60b are animal image portions. It can be determined that the possibility is high.

  Therefore, if there is a change in the presence or absence of the extraction of the head image part in STEP 80, the process proceeds to STEP 81, and if there is no change in the presence or absence of the extraction of the leg image part, the process branches to STEP 90. In STEP 81, the animal determination unit 14 determines whether or not the surrounding luminance change is less than the second predetermined level between the pair of animal candidate image portions.

  In FIGS. 7A and 7B, the positions of the image portion 60a and the image portion 60b have not changed, so the right luminance determination area 70a and the left luminance determination area 71a set in FIG. 7A. The positions of the right luminance determination area 70b and the left luminance determination area 71b set in FIG. 7B are not changed.

  Therefore, the luminance of the right luminance determination area 70a and the left luminance determination area 71a in FIG. 7A (for example, the average luminance of both areas 70a and 71a), and the right luminance determination area 70b and the left luminance determination of FIG. A change in luminance due to temperature between the luminance of the area 71b (for example, the average luminance of both the areas 70b and 71b) becomes small.

  Therefore, in STEP 81, when the luminance change around the pair of animal candidate image portions of the same object between the time-series captured images is less than the second predetermined level, the pair of animal candidate image portions are the animal image portions. It can be determined that there is a higher possibility. In this case, the process proceeds to STEP 52, where the animal determination unit 14 determines that the object in the real space corresponding to the pair of animal image candidate portions is an animal.

  On the other hand, in STEP 81, when the luminance change around the image portion of the same object between time-series images is equal to or higher than the second predetermined level, the process branches to STEP 90, and the animal determination unit 14 corresponds to the pair of animal candidate image portions. It is determined that the object in the real space is not an animal.

  In this embodiment, in STEP 70 and STEP 71, for the image part of the same object between time-series captured images, the object is an animal based on the change in the image part of the leg and the surrounding luminance change. Whether or not the object is an animal may be determined only by the change in the image portion of the leg portion in STEP 70.

  In this embodiment, in STEP80 and STEP81, for the image portion of the same object between time-series captured images, the object is an animal based on the change in the image portion of the head and the surrounding luminance change. Whether or not the object is an animal may be determined only by the change in the image portion of the head in STEP80.

  In the present embodiment, as shown in FIGS. 6A, 6B, 7A, and 7B, the right luminance determination area set on the right side of the animal candidate image portion and Although the change in luminance of the area around the animal candidate image portion is determined based on the average luminance of the left luminance determination area set on the left side, the luminance determination area may be set above and below the animal candidate image portion or diagonally up and down. . Further, instead of the average luminance in the luminance determination area, a change in luminance may be determined using a sum of luminances of all pixels in the luminance determination area, an intermediate value of luminances of all pixels, or the like.

  Further, in this embodiment, in STEP 70 and STEP 71 in FIG. 4, it is determined whether or not the animal candidate image part is extracted, and in STEP 80 and STEP 81, whether or not the animal candidate image part is extracted is determined. However, only one of them may be determined.

  Further, in the present embodiment, the configuration using the captured image of one camera (infrared camera 2) is shown, but the captured image of the same object by the stereo camera using the captured images of the two cameras (stereo camera). A configuration may be adopted in which the distance between the vehicle and the object is calculated from the parallax.

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2 ... Camera, 10 ... Vehicle periphery monitoring apparatus, 11 ... Captured image acquisition part, 12 ... Object image extraction part, 13 ... Specific site | part extraction part, 14 ... Animal determination part, 20, 30, 40a, 40b , 60a, 60b ... animal candidate image part, 22, 32, 42a, 42b, 62a, 62b ... image part of the trunk part, 21, 31, 41a, 41b, 61b ... image part of the head part, 23, 24, 33, 34, 43b, 44b... Image portions of legs, 50a, 50b, 70a, 70b... Right luminance determination area, 51a, 51b, 71a, 71b.

Claims (3)

An animal candidate image part extraction unit that extracts, as an animal candidate image part, an image part of the same object that may be an animal from a time-series captured image captured by a camera mounted on a vehicle;
A specific part extraction unit for extracting an image part of a specific part of an animal from the animal candidate image part;
An image portion of the specific part is extracted by the specific part extraction unit between the pair of animal candidate image parts of the same object extracted by the animal candidate image part extraction unit between the time-series captured images. When the specific part extraction state and the specific part non-extraction state in which the image part of the specific part is not extracted by the specific part extraction unit are switched, the object in the real space corresponding to the pair of animal candidate image parts is A vehicle periphery monitoring device comprising an animal determination unit that determines that the animal is an animal. In the vehicle periphery monitoring device according to claim 1,
The specific part extraction unit extracts an image part of a leg as an image part of the specific part,
The animal determination unit switches between the specific part extraction state and the specific part non-extraction state between the pair of animal candidate image parts, and the luminance of an area around the pair of animal candidate image parts. A vehicle periphery monitoring device, wherein when the change is equal to or higher than a first predetermined level, it is determined that an object in real space corresponding to the pair of animal candidate image portions is an animal. In the vehicle periphery monitoring device according to claim 1,
The specific part extraction unit extracts an image part of a head as an image part of the specific part;
The specific part extraction unit switches between the specific part extraction state and the specific part non-extraction state between the pair of animal candidate image parts, and brightness of a region around the pair of animal candidate image parts. The vehicle periphery monitoring device, wherein when the change in the value is less than a second predetermined level, it is determined that the object in the real space corresponding to the pair of animal candidate image portions is an animal.

Images