JP2014056295A - Vehicle periphery monitoring equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide target object monitoring equipment capable of performing processing of detecting a target object such as a biological body while suppressing excessive increase of calculation load according to an image taken by an infrared camera.SOLUTION: Target object monitoring equipment comprises: means 32 for extracting an image of a type-discrimination target from a high-intensity region of an image taken by an infrared camera 11 to discriminate a type thereof; means 33 for extracting an image of the type-discrimination target from a low-intensity region of the taken image to discriminate a type thereof. At least one of the means 32, 33 is configured to perform type-discrimination processing in a region other than a region-part containing an image of a type-discrimination target whose type is already determined by the other of the means 32, 33.

Description

  The present invention relates to a vehicle periphery monitoring device that monitors the periphery of a vehicle using an image captured by an infrared camera.

  A captured image (grayscale image) around the vehicle is acquired by an infrared camera mounted on the vehicle, and a person (pedestrian) as a monitoring target existing in the imaging region of the infrared camera is detected based on the captured image. Conventionally, a vehicle periphery monitoring device is known (see, for example, Patent Document 1).

  In the vehicle periphery monitoring device found in Patent Document 1, the binarization is configured from an image captured by an infrared camera, and a high-luminance image portion that is equal to or higher than a predetermined luminance threshold value and an image portion that is lower than the luminance threshold value. An image is generated. Whether or not the object is a person based on the shape or size characteristics of the image of the object (object that may be a person) extracted from the high-luminance area of the binarized image Determined.

JP 2003-284057 A

  By the way, as an image captured by an infrared camera for detecting the presence of a living body such as a person, a luminance change occurs depending on a temperature difference between an arbitrary subject existing in the imaging region of the infrared camera and its background, and the subject The higher the temperature of the object is, the higher the luminance of the image part of the object is higher than the luminance of the image part of the background, and the lower the temperature of the object is the background temperature, the image part of the object It is preferable to use a captured image having a characteristic that the luminance of the image becomes lower than the luminance of the background image portion (hereinafter also referred to as AC characteristic).

  When the captured image having the AC characteristic is used, under normal circumstances, the temperature of a living body such as a person shown in the captured image is generally the temperature of a background portion (road surface, structure, etc.) around the living body. Relatively higher temperature. For this reason, an image of a living body such as a person existing in the imaging region of the infrared camera is a high-luminance image in the captured image having the AC characteristics.

  However, when the outside air temperature is high, the temperature of the living body may be relatively lower than the temperature of the surrounding background portion. In such a case, in the binarized image generated from the captured image, the image portion of the living body cannot be obtained as a high-luminance image portion as compared to the surrounding background portion image. . As a result, a person existing in the imaging area cannot be detected.

  In order to eliminate such inconvenience, in addition to detecting the presence of a living body such as a person from a high brightness area of an image captured by an infrared camera, it is also possible to detect the presence of a living body from a low brightness area. Conceivable. However, in that case, the calculation processing load of the monitoring device becomes excessive, and there is a risk of delaying detection of the living body.

  The present invention has been made in view of such a background, and it is difficult to detect an object such as a living body based on a captured image of an infrared camera due to an environmental condition, and an excessive processing load is imposed. It aims at providing the target object monitoring apparatus which can perform the process which detects objects, such as a biological body, so that it may not become.

The object monitoring apparatus of the present invention is an object monitoring apparatus having a function of determining the type of an object existing in the imaging area of an infrared camera in order to achieve such an object,
The brightness of an image of the object changes according to a temperature difference between an arbitrary object existing in the imaging region of the infrared camera and the background of the object, and the object is more than the background. Using the infrared camera, a captured image having characteristics that the higher the temperature of the object, the higher the brightness of the image of the object, and the lower the temperature of the object than the background, the lower the brightness of the image of the object. Captured image acquisition means for acquiring
In the captured image, an image of a type determination target object, which is a target for performing type determination, is extracted from a high luminance region having a luminance equal to or higher than a predetermined first threshold, and based on the image of the type determination target First object type determination means for executing a first determination process for determining the type of the type determination object;
An image of the type determination target object is extracted from a low luminance region having a luminance equal to or lower than a predetermined second threshold in the captured image, and the type of the type determination target object is determined based on the image of the type determination target object Second object type determination means for executing a second determination process to perform,
At least one of the first object type determination unit and the second object type determination unit is an area corresponding to the one of the high luminance region and the low luminance region in the captured image. The first determination process or the second determination process is performed in an area excluding a partial area in which an image of a type determination target object whose type has already been determined by the other means is present. (First invention).

  The background of the object in the present invention means an object behind the object (a wall surface of a building, a road surface, etc.) that appears in an image captured by an infrared camera.

  According to the first aspect of the invention, the first object type determination unit can extract the image of the type determination object having a temperature higher than that of the background from the high luminance region of the captured image of the infrared camera. As a result, the type of the type determination target can be determined based on the image of the type determination target.

  In addition, the second object type determination unit can extract an image of the type determination object having a temperature lower than that of the background from the low luminance region of the image captured by the infrared camera. As a result, the type of the type determination target can be determined based on the image of the type determination target.

  Therefore, in both cases where the temperature of the type determination target is higher or lower than the background temperature, the type determination target image in the captured image is recognized, and the type of the type determination target is determined. Can be determined. As a result, when there is an object such as a living body in the imaging region of the infrared camera, it can be detected regardless of the outside air temperature.

  Further, at least one of the first object type determination unit and the second object type determination unit corresponds to the one of the high luminance region and the low luminance region in the captured image. The first determination process or the second determination process is executed in an area excluding a partial area in which an image of the type determination object whose type has already been determined by the other means is present.

  For this reason, said one means performs classification judgment with respect to the classification judgment target object whose classification has already been judged by said other means out of the first object type judgment means and the second object type judgment means. There is nothing. For this reason, it is possible to prevent the total calculation processing load of the first object type determination means and the second object type determination means from becoming excessive.

  Therefore, according to the first aspect of the present invention, it is possible to prevent an object such as a living body from being difficult to detect due to environmental conditions based on a captured image of an infrared camera, and to prevent an excessive calculation processing load. The process which detects the target object of this can be performed.

  In the first invention, both the first determination process and the second determination process determine whether the type determination target is a predetermined type of living body based on the image of the type determination target. Each of the second object type determination means includes the process of determining whether the outside air temperature is a high temperature equal to or higher than a predetermined temperature or that an environmental condition is met when it is raining. It is configured to execute a process for determining whether or not an object is a predetermined type of living body, and the first object type determination unit is configured to determine the type regardless of whether or not the environmental condition is satisfied. It is preferable that a process for determining whether or not the object is a predetermined type of living body is executed (second invention).

  According to the second aspect of the invention, the process of determining whether or not the type determination target is a predetermined type of living body by the second target type determination unit is not executed when the environmental condition is not satisfied.

  Here, when the environmental condition is not satisfied, the temperature of the living body is usually higher than the background temperature. Therefore, when the living body exists in the imaging region of the infrared camera, the second object type determination is performed. Even if the processing of the means is not performed, the existence of the living body can be recognized by the first object type determination means.

  In this case, since the process for determining whether or not the type determination target is a predetermined type of living body is not executed by the second target type determination unit, the calculation processing load by the second target type determination unit is reduced. it can.

  In addition, when the environmental condition is satisfied, the processing of the second object type determination unit is performed, so that the presence of the living body is recognized even when the temperature of the living body is lower than the background temperature. Can do.

  In the second invention, both the first determination process and the second determination process determine whether the type determination object is a person as the predetermined type of living body in the image of the type determination object. Each of the first determination process and the second determination process of the first determination process and the second determination process is performed when at least a predetermined operating condition is satisfied. An animal recognition process for determining whether the target object is an animal other than a person based on an image of the type determination target object, wherein the first target type determination unit and the second target type determination unit Among these, each determination means for executing the determination process including the animal recognition process is a person by executing the person recognition process by both determination means of the first object type determination means and the second object type determination means. Specified species When the total number of determination objects is less than a predetermined number, or when it is determined that the traveling place of the vehicle equipped with the infrared camera is not a city area, the predetermined operation condition is satisfied, When the animal recognition process is performed and the total number of type determination objects determined to be the person is a predetermined number or more, or when it is determined that the place where the vehicle is traveling is an urban area, It is determined whether or not the type determination object is an animal other than a human by performing processing that reduces the amount of calculation compared to when the predetermined operation condition is satisfied, or omitting performing the animal recognition process (3rd invention).

  Here, in an environment where there are many people or in an urban area, it is unlikely that animals other than humans will be present in the imaging region of the infrared camera. Therefore, in the third invention, among the first object type determination means and the second object type determination means, each determination means for executing the determination process including the animal recognition process is determined to be the person. When the total number of type determination objects is a predetermined number or more, or when it is determined that the place where the vehicle is running is an urban area, omitting the animal recognition process, or It is determined whether or not the type determination object is an animal other than a human by a process in which the amount of calculation is reduced as compared with a case where a predetermined operating condition is satisfied.

  Thereby, when recognizing a person and an animal other than a person existing in a captured image of the infrared camera, a situation where there are many persons (by the first object type determination unit and the second object type determination unit) In a situation where the load of human recognition processing is likely to increase, the load of the animal recognition processing in each determination means that executes the determination processing including the animal recognition processing is eliminated or reduced, and the calculation processing load of the object monitoring device is excessive. Can be prevented.

  In the second aspect of the invention, both of the first determination process and the second determination process are performed when the type determination object is a human being of the predetermined type when at least a predetermined operation condition is satisfied. Each of which includes a person recognition process for determining whether or not is based on an image of the type determination target object, and the first determination process or the both determination processes of the first determination process and the second determination process include: An animal recognition process for determining whether or not the type determination target object is an animal other than a person based on an image of the type determination target object, wherein the first target type determination unit and the second target object Both determination means of the type determination means are executed by execution of the animal detection process by each determination means that executes the determination process including the animal detection process among the first object type determination means and the second object type determination means. The animal When the total number of type determination objects determined to be less than a predetermined number, it is determined that the predetermined operation condition is satisfied, the person recognition process is executed, and the type determination target determined to be the animal If the total number of objects is equal to or greater than a predetermined number, the execution of the person recognition process is omitted, or the type determination is performed by a process in which the amount of calculation is reduced as compared with the case where the predetermined operation condition is satisfied. You may be comprised so that it may be determined whether a target object is a person (4th invention).

  Here, in an environment where there are many animals other than humans, it is unlikely that people will be present in the imaging area of the infrared camera. Therefore, in the third invention, both of the first object type determination means and the second object type determination means are the animal of the first object type determination means and the second object type determination means. The person recognition process is executed when the total number of type determination objects determined to be the animal by the execution of the animal detection process by each determination unit that executes a determination process including a detection process is a predetermined number or more. It is determined whether or not the type determination target is a person by a process that reduces the amount of calculation compared to the case where the predetermined operation condition is satisfied.

  Thereby, when recognizing a person and an animal other than a person existing in an image captured by an infrared camera, a situation where there are many animals other than a person (the first object type determination means and the second object type). Among the determination means, the load of the animal recognition process by each determination means that executes the determination process including the animal detection process is likely to increase). It is possible to prevent the calculation processing load of the object monitoring apparatus from becoming excessive.

The figure which shows the vehicle carrying the target object monitoring apparatus in 1st Embodiment of this invention. The block diagram which shows the structure regarding the target object monitoring apparatus shown in FIG. The flowchart which shows the process of the target object monitoring apparatus shown in FIG. The flowchart which shows the process (process in 1st Embodiment) of STEP2 shown in FIG. The flowchart which shows the process of STEP4 shown in FIG. The graph for demonstrating the process of STEP2-1 shown in FIG. 4, and the process of STEP4-2 shown in FIG. The figure which shows the example of the captured image of an infrared camera. The figure for demonstrating the process of STEP2-3 shown in FIG. FIG. 4 is a flowchart showing a part of the processing of STEP 2 shown in FIG. 3 (processing in the second embodiment).

[First Embodiment]
A second embodiment of the present invention will be described with reference to FIGS.

  With reference to FIG. 1, an object monitoring apparatus 10 of this embodiment is mounted on a vehicle 1. In addition to the object monitoring device 10, the vehicle 1 includes an infrared camera 11 that captures an image of a predetermined monitoring area AR 0 (an angle of view area between the straight lines L 1 and L 2 in FIG. 1) around the vehicle 1, and an infrared camera 11. A display 12 that displays display information such as a captured image so that the driver of the vehicle 1 can visually recognize, and a speaker 13 that outputs acoustic information (sound, warning sound, etc.) to notify the driver of the vehicle 1 are mounted. Yes.

  The indicator 12 is configured by a liquid crystal display installed in the passenger compartment in front of the driver's seat of the vehicle 1 or a head-up display that projects and displays an image on the windshield of the vehicle 1. In addition, the display device 12 may display navigation information (such as a map), audio information, and the like as appropriate in addition to the image captured by the infrared camera 11.

  The infrared camera 11 is a camera having sensitivity in an infrared wavelength band, and outputs a video signal indicating the luminance of each pixel constituting the captured image. The monitoring area AR0 (hereinafter sometimes referred to as the imaging area AR0) as the imaging area of the infrared camera 11 is an area on the front side of the vehicle 1 in the present embodiment. In order to image the monitoring area AR0, the infrared camera 11 is mounted on the front portion of the vehicle 1.

  The infrared camera 11 is a camera in which the brightness of a captured image defined by the video signal has the following characteristics with respect to the temperature distribution state of the entire subject in the imaging area AR0 reflected in the captured image. The characteristic is that the brightness of the image of an arbitrary object in the imaging area AR0 (the brightness of the image in the vicinity of the projection area of the object) has a brightness corresponding to the temperature itself of the object. Rather than the size of the object according to the relative temperature difference between the object and its background (subject (the wall surface of the building, road surface, etc.) existing behind the object as viewed from the infrared camera 11) This is a characteristic of luminance (hereinafter referred to as AC characteristic).

  In this AC characteristic, as the temperature of an arbitrary object in the imaging area AR0 is higher than the background temperature, the brightness of the image of the object is higher, and the temperature of the object is lower than the background temperature. , The brightness of the image of the object is lowered.

  In other words, the captured image of the infrared camera 11 having such AC characteristics is an image portion in which a relatively remarkable temperature change (spatial temperature change) occurs in the entire subject in the imaging area AR0. While the brightness change is emphasized, the brightness of the image part where the part with the uniform temperature (the part where the temperature of each part is almost the same) is reflected, regardless of the magnitude of the temperature (absolute temperature) The captured images have substantially the same brightness.

  The infrared camera 11 of the present embodiment is a camera having such AC characteristics.

  Supplementally, the infrared camera 11 does not have to have AC characteristics. That is, in the infrared camera 11, the luminance of each pixel defined by the video signal output from the infrared camera 11 is a luminance corresponding to the magnitude of the temperature of the subject projected onto the pixel ( A camera having such a characteristic that the higher the temperature is, the higher the luminance is) may be used. In that case, a captured image having the above-mentioned AC characteristics can be obtained by performing video filter processing on the captured image defined by the video signal of the infrared camera.

  The object monitoring apparatus 10 is an electronic circuit unit including a CPU, a memory, an interface circuit, and the like (not shown). The object monitoring apparatus 10 performs a predetermined control process by executing a program to be mounted by the CPU.

  Specifically, the object monitoring apparatus 10 monitors the imaging area AR0 based on the captured image of the infrared camera 11. Then, when it is detected that there is a predetermined type of object that may contact the host vehicle 1 in the imaging region AR0, contact for avoiding contact between the target object and the host vehicle 1 Execute avoidance processing.

  In the present embodiment, the predetermined type of object is a predetermined type of living body, specifically, a person (pedestrian) and an animal other than a person (deer or the like, a quadruped moving on the ground). Hereinafter, the predetermined type of living body is referred to as a specific type living body, and the specific type of living body that may be in contact with the vehicle 1 is referred to as a detection target living body.

  Moreover, the said contact avoidance process is a process (attention call process) which alerts a driver | operator's attention with respect to a detection target biological body in this embodiment. In this contact avoidance process, the object monitoring device 10 displays a captured image of the infrared camera 11 on the display 12 and gives a visual warning and an audible warning to the display 12 and the speaker 13 respectively. Let it be done.

  The object monitoring apparatus 10 will be further described with reference to FIG. The object monitoring device 10 is supplied with video signals from the infrared camera 11 and detection signals from various sensors mounted on the vehicle 1.

  In the present embodiment, the object monitoring device 10 includes a yaw rate sensor 21 that detects the yaw rate of the vehicle 1, a vehicle speed sensor 22 that detects the vehicle speed of the vehicle 1, and a brake that detects a brake operation (depression of the brake pedal) by the driver. Detection signals (or wiper operation command signals) of the sensor 23, the outside air temperature sensor 24 that detects the outside air temperature, and the wiper sensor 25 that detects the operating state of the windshield wiper (not shown) of the vehicle 1 are input.

  Further, the display 12 and the speaker 13 are connected to the object monitoring apparatus 10, and the display of the display 12 and the sound output of the speaker 13 are controlled by the object monitoring apparatus 10. ing.

  The object monitoring apparatus 10 is realized by a function (a function realized by a software configuration) or a hardware configuration (an input / output circuit, an arithmetic circuit, or the like) realized by executing a mounted program by the CPU. As main functions, a captured image acquisition unit 31 that acquires a captured image (captured image of the AC characteristic) of the infrared camera 11 and a process of detecting a detection target living body existing in the imaging region AR0 based on the captured image, respectively. The detection target living body is detected by any one of the first object detection processing unit 32 and the second object detection processing unit 33 and the first object detection processing unit 32 and the second object detection processing unit 33 to be executed. A contact avoidance processing unit 34 for executing the contact avoidance process (attention calling process).

  The first object detection processing unit 32 and the second object detection processing unit 33 include functions as first object type determination means and second object type determination means in the present invention, respectively.

  Note that the object monitoring apparatus 10 may be configured such that a plurality of CPUs or processors can perform different processes in parallel (can perform parallel processes). Moreover, the object monitoring apparatus 10 does not need to be comprised by the single electronic circuit unit, and may be comprised by the some electronic circuit unit which can communicate mutually.

  Next, more detailed processing of the object monitoring apparatus 10 will be described with reference to the flowcharts of FIGS.

  The object monitoring apparatus 10 executes the process shown in the flowchart of FIG. 3 at a predetermined control process cycle. The object monitoring apparatus 10 first performs the processing of STEP 1 by the captured image acquisition unit 31. In this process, the captured image acquisition unit 31 acquires a captured image of the infrared camera 11.

  More specifically, the captured image acquisition unit 31 causes the infrared camera 11 to capture the monitoring area AR0, and performs A / D conversion on the video signal output from the infrared camera 11 in accordance with the captured image, whereby each pixel is captured. A captured image (captured image with AC characteristics) representing a luminance value as a digital value is acquired. The captured image acquisition unit 31 stores and holds the acquired captured image in an image memory (not shown).

  The captured image acquisition unit 31 stores and holds a plurality of captured images for a period up to a predetermined time including the latest captured image in the image memory.

  Supplementally, when the infrared camera 11 does not have AC characteristics, the captured image acquisition unit 31 performs a video filter process on the captured image defined by the video signal of the infrared camera, thereby converting the captured image with AC characteristics. You only have to get it.

  Next, the object monitoring apparatus 10 executes a process of detecting the detection target living body by the first object detection processing unit 32 in STEP2. Further, the object monitoring apparatus 10 determines in STEP 3 whether or not a predetermined operation condition for executing the process of the second object detection processing unit 33 is satisfied.

  And the object monitoring apparatus 10 performs the process which detects a detection target biological body by the 2nd object detection process part 33 in STEP4, when this operating condition is satisfied. In addition, when the said operation condition is not satisfied, the process of the 2nd target object detection process part 33 is abbreviate | omitted.

  The details of the processing in STEP 2 to 4 will be described later. However, when the living body to be detected exists in the imaging area AR0 by the processing in STEP 2 and the processing in STEP 4 when the determination result in STEP 3 is affirmative, the presence is detected. Detected.

  Note that the processing in STEP 2 and the processing in STEP 3 and 4 may be executed in parallel processing by different CPUs, for example. Or you may make it perform the process of STEP2 and the process of STEP3, 4 in series by a time division process, for example.

  Next, the object monitoring apparatus 10 determines in STEP 5 whether or not a detection target living body has been detected by the processing in STEP 2 or STEP 4. If the determination result is affirmative (when a detection target living body is detected), the process of STEP 6 is executed by the contact avoidance processing unit 34. In STEP 6, the contact avoidance processing unit 34 executes a contact avoidance process (attention calling process) for avoiding future contact between the living body to be detected and the host vehicle 1.

  Specifically, the contact avoidance processing unit 34 controls the display 12 to display the captured image of the infrared camera 11 on the display 12 while highlighting the image of the detection target living body in the captured image. For example, the contact avoidance processing unit 34 surrounds and displays an image of the detection target living body in the captured image displayed on the display device 12 with a frame of a predetermined color or blinks the frame. Highlight the image.

  In addition, the contact avoidance processing unit 34 causes the speaker 13 to output an alarm sound or sound indicating that there is a detection target living body that may be in contact with the host vehicle 1 in the imaging region (monitoring region) AR0. Control.

  By controlling the display of the indicator 12 and the control of the speaker 13, a visual alarm and an audible alarm regarding a detection target living body that may be in contact with the host vehicle 1 are given to the driver. Made. Eventually, the driver's attention is alerted to the detection target living body that may be in contact with the host vehicle 1. Accordingly, the driver performs a driving operation (brake operation or the like) that can appropriately avoid contact between the living body to be detected and the host vehicle 1, and can avoid the contact.

  Note that even if there is a living body to be detected that is determined to have a possibility of contact with the host vehicle 1 in the imaging region AR0, if the driver performs a braking operation on the vehicle 1, the driver It is considered that the presence of the detection target living body is recognized and the contact with the detection target living body is to be avoided. Therefore, in this embodiment, when the contact avoidance processing unit 34 detects that the brake operation of the vehicle 1 by the driver is performed by the output of the brake sensor 23, the contact avoidance processing (attention warning) Process) is omitted.

  However, even in a situation where the brake operation of the vehicle 1 is being performed, whether or not to perform the contact avoidance process may be selected depending on, for example, the amount of depression of the brake pedal or the degree of deceleration of the vehicle 1. Or you may make it perform a contact avoidance process irrespective of whether brake operation is made or not.

  Supplementally, in this embodiment, in order to avoid contact between the living body to be detected and the host vehicle 1, the driver's attention is alerted by visual notification by the display 12 and auditory notification by the speaker 13. However, only one notification may be performed.

  Alternatively, instead of both visual notification and / or auditory notification, the driver's attention may be drawn by performing a bodily notification such as vibrating the driver's seat. Good.

  In addition, when the brake device of the vehicle 1 has a configuration in which the braking force can be adjusted from the braking force according to the operation of the brake pedal by hydraulic control or the like, the driver's attention is to be drawn. In addition, the braking force of the brake device may be automatically increased.

  Next, the processing of STEPs 2 to 4 which will be described later will be described below.

  First, basic matters regarding the processing of the first object detection processing unit 32 and the second object detection processing unit 33 will be described.

  The temperature of the specific species living body (human or animal other than humans) to which the detection target organism belongs belongs to the road surface or the like in a normal environment (such as an environment where the outside air temperature is not so high). It is higher than the temperature of surrounding objects. For this reason, the luminance of the image of the specific living body in the captured image (captured image of AC characteristics) of the infrared camera 11 is usually higher than the luminance of the background subject (road surface, building wall surface, etc.). It will be something.

  However, when the outside air temperature is high, or when it is raining, the temperature of the specific species living body (human or animal) may be lower than the temperature of surrounding objects. In such a case, the brightness of the specific living body in the captured image of the infrared camera 11 is lower than the brightness of the image of the background subject (road surface, wall surface of the building, etc.) unlike the normal case described above. It becomes brightness.

  For example, as illustrated in the upper diagram of FIG. 7, the image of the specific species living body 51 (in the illustrated example, a person) in the image captured by the infrared camera 11 is lower in brightness (darker) than the background image of the subject. It becomes.

  In view of such a phenomenon, the first object detection processing unit 32, when a specific species living body is present in the imaging region AR0, the brightness of the image of the specific species living body in the captured image is that of the image of the subject in the background. On the premise that the brightness is higher than the brightness, the target object that may be a detection target living body (the target that may be in contact with the host vehicle 1 and may be a specific kind of living body) The object is extracted, and a process of determining whether or not the type of the target object is a specific species living body (a person or an animal other than a person) is executed. And the 1st target object detection process part 32 detects a detection target biological body by the process.

On the other hand, the second object detection processing unit 33, when a specific species living body exists in the imaging region AR0, the luminance of the image of the specific species living body in the captured image is lower than the luminance of the background subject image. Assuming that the target object is a target object, a target object that may be a detection target living body is extracted, and processing for determining whether or not the type of the target object is a specific species living body is executed.
And the 2nd target object detection process part 33 detects a detection target biological body by the process.

  In this case, the brightness of the image of the specific species in the captured image of the infrared camera 11 is basically lower than the brightness of the background subject image. It is a phenomenon that occurs in a high temperature environment or in an environment during rain.

  For this reason, the object monitoring apparatus 10 performs the process of the second object detection processing unit 33 (STEP 4) when a predetermined operation condition for performing the process of the second object detection processing unit 33 is satisfied in STEP3. Process).

  The predetermined operating condition is established when the outside air temperature is higher than the predetermined temperature or when it is raining, and when the outside air temperature is lower than the predetermined temperature and is not raining. The predetermined operating condition is not satisfied.

  Then, the object monitoring device 10 determines whether or not the outside air temperature is a high temperature equal to or higher than a predetermined temperature based on the detected value of the outside air temperature by the outside air temperature sensor 24.

  In addition, the object monitoring apparatus 10 determines whether or not it is raining based on the operation state of the wiper indicated by the output of the wiper sensor 25 (or the wiper operation command signal). Specifically, the object monitoring apparatus 10 determines that it is raining when the wiper is operating, and determines that it is not raining when the wiper is not operating.

  Whether or not it is raining may be detected using a raindrop sensor. Alternatively, weather information may be received by communication to recognize whether it is raining or not.

  Further, even when the predetermined operation condition is not satisfied (when the outside air temperature is lower than the predetermined temperature or when it is not raining), the process of the second object detection processing unit 33 is appropriately executed. Good.

  Based on the above, the processing of the first object detection processing unit 32 and the second object detection processing unit 33 will be specifically described.

  First, the processing of the first object detection processing unit 32 in STEP 2 is executed as shown in the flowchart of FIG.

  First, the first object detection processing unit 32 executes binarization extraction processing in STEP 2-1. In the binarization extraction process, the first object detection processing unit 32 generates a binarized image from the captured image of the infrared camera 11. And the 1st target object detection process part 32 extracts the image of the target object (candidate object of a detection target biological body) which may be a detection target biological body from the binarized image.

  The binarized image is an image composed of pixels of two types of high and low luminance values, for example, two types of luminance values, a white luminance value (maximum luminance value) and a black luminance value (minimum luminance value). It is the image comprised by this pixel.

  This binarized image is a pixel having a luminance equal to or higher than a predetermined first threshold in a captured image of the infrared camera 11, a pixel having a high luminance value (white luminance value), and a pixel having a luminance smaller than the first threshold. , A pixel having a low luminance value (black luminance value) is generated.

  In the present embodiment, the first threshold is determined based on a distribution state of luminance in the captured image, for example, a histogram representing the relationship between the luminance of each pixel of the captured image and the number of pixels (appearance frequency). FIG. 6 shows an example of the histogram graph. Yth1 in the figure is an example of the first threshold value determined in correspondence with this histogram.

  As shown in the figure, the first threshold Yth1 is a luminance in the vicinity of the luminance value Y1 where the number of pixels reaches the peak (maximum value) on the high luminance side in the luminance distribution histogram of the captured image (for example, within a predetermined range centered on Y1). Is determined to be a value smaller than Y1 so that the luminance of Y is equal to or greater than Yth1.

  Note that the first threshold Yth1 may be a predetermined value (constant value) determined in advance. Alternatively, the first threshold Yth1 may be set according to the outside air temperature or the like. The first threshold value Yth1 is obtained when the specific species living body such as a person is imaged by the infrared camera 11 in a general environment where the temperature is higher than the temperature of the subject in the background. The luminance of the image of the specific living body in the image may be set to be equal to or higher than the first threshold value Yth1.

  In the binarized image generated as described above, when a specific species living body exists in the imaging region AR0, the image of the specific species living organism is usually obtained as an image having a high luminance value (white luminance value). However, an image of an object that is not a specific species such as a structure or a part of another vehicle (a part where the temperature is relatively high) may be obtained as an image having a high luminance value in the binarized image.

  In STEP 2-1, the first object detection processing unit 32, after generating a binarized image as described above, is composed of a high-luminance region (pixels with high luminance values (white pixels)) of the binarized image. Image of a target object that is a candidate for a detection target living body (this corresponds to a type determination target object in the present invention).

  Specifically, the first object detection processing unit 32 selects a high-luminance portion formed by a set of lines of run-length data that overlap each other in the vertical direction in the high-luminance region of the binarized image. , Extracted as an image of the object.

  The line of the run length data is an individual line composed of pixels having high luminance values (white luminance values) continuous in the horizontal direction of the binarized image in the high luminance area of the binarized image. . Then, data composed of the pixel position of the start point (or end point) of each line and the length (number of pixels) of the line (or data that can define the pixel position and length) is run-length data. It is.

  As described above, the processing of STEP 2-1 (binarization extraction processing) is executed.

  Next, the first object detection processing unit 32 executes tracking processing in STEP2-2. This tracking process is a process for tracking an object image extracted from a binarized image for a time period.

  More specifically, in the tracking process, an image of each object extracted from the binarized image at time k (current control processing cycle time) is time k−1 (time of the previous control processing cycle). ) To determine whether or not the image is the same object as the object extracted from the binarized image, and the image of each object extracted at time k (current control processing cycle time) is determined. This is a process of attaching a label (identifier).

  In this tracking process, an image is obtained at time k and time k−1 based on the position of the center of gravity of the image of the object (or the circumscribed rectangle of the image) in the binarized image, the area, and the ratio of the length and width. The identity of the object from which is extracted is determined. For example, if the amount of change (the amount of change between time k and time k−1) of the center of gravity position, area, and length / width ratio is within a predetermined allowable range, both times k, k It is determined that the image of the object extracted at -1 is the same object image.

  And about the image of the same target object as the target object extracted at the time k-1, the same label as the time k-1 is attached, and about the new target object image not extracted at the time k-1. A new label is attached.

  Next, the first object detection processing unit 32 executes a vehicle recognition process in STEP 2-3. This vehicle recognition processing is processing for recognizing an image of an object corresponding to a component of another vehicle based on an arrangement pattern of the image of the object.

  For example, when a front vehicle is present in the imaging area AR0 in front of the host vehicle 1, in the captured image of the infrared camera 11, the images of the left and right taillight portions of the front vehicle and the left and right rear wheel tires are displayed. In many cases, the image of the portion and the image of the portion of the exhaust port have relatively high luminance. In this case, in the binarization extraction process, as schematically shown in FIG. 8, the left and right taillight image portions A1 and A2 of the front running vehicle, the left and right rear wheel tire image portions A4 and A5, The image portion A3 of the exhaust port is extracted as an image of a high-luminance object.

  The arrangement pattern of these image portions A1 to A5 is such that image portions A1 and A2 of taillights having a similar luminance distribution or outer shape are aligned in the horizontal direction of the binarized image, and these The image portion A3 of the exhaust port exists below the image portions A1 and A2, and the image portions A4 and A5 of tires having similar luminance distributions or outer shapes are paired below the image portions A1 and A2. It becomes a characteristic arrangement pattern in which the binarized images are arranged in the horizontal direction.

  Therefore, in STEP 2-3, the first object detection processing unit 32 searches for a set of images of a plurality of objects (five objects) having the above characteristic arrangement pattern in the binarized image. To do. Then, when there is a set of images of a plurality of objects having the arrangement pattern, the first object detection processing unit 32 is an image of a component of another vehicle. It is determined that it is present (not an image of a living body to be detected).

  Even if the image portion A3 of the exhaust port does not exist, for example, when four image portions A1, A2, A4, A5 exist in the above arrangement pattern, those four image portions A1, A2, A4 , A5 may be determined to be an image portion of a component of an oncoming vehicle as another vehicle. In that case, the image portions A1 and A2 correspond to the image of the headlight of the oncoming vehicle.

  Next, the first object detection processing unit 32 executes a crossing person recognition process in STEP 2-4. Note that the processing after STEP 2-4 (including the processing of STEP 2-4) is the above-mentioned vehicle recognition processing among the objects extracted as having a high-luminance image in the binarization extraction processing. It is a process performed regarding the target object which excluded the target object determined to be a component.

  The crossing person recognition process is a process of determining whether or not the object is a person (crossing person) crossing the road on which the vehicle 1 is traveling.

  When a person crosses a road on which the host vehicle 1 is traveling, the stride direction of the crosser is the vehicle width direction of the host vehicle 1. For this reason, the images of the legs of the crossing person in the image captured by the infrared camera 11 often have a divergent arrangement pattern (an arrangement pattern in the form of “c”).

  Therefore, in the crossing person recognition process of STEP2-4, the first object detection processing unit 32 detects the object image in the captured image of the infrared camera 11 (the object from which the high-intensity portion has been extracted by the binarization extraction process). (The image in the captured image) has a pair of image portions (image portions corresponding to both legs) arranged in the horizontal direction in the lower end arrangement pattern at the bottom thereof, the object is a crossing person to decide.

  In the present embodiment, the object determined to be a crossing person in the crossing recognition process is not determined to be a person, and whether or not the object is a person is determined through a process described later. Will be fixed. Therefore, the determination in the crossing person recognition process is provisional.

  Next, the first object detection processing unit 32 executes a distance calculation process in STEP 2-5. This distance calculation process is a process of calculating an estimated value of the distance of the object from the own vehicle 1 (distance in the Z-axis direction in FIG. 1) and estimating the real space position of the object with respect to the own vehicle 1. .

  In this distance calculation process, the first object detection processing unit 32 performs a separate process between the object determined to be a crossing person in the crossing person recognition process in STEP 2-4 and the other objects. The distance of the target object from the own vehicle 1 is estimated.

  Specifically, for an object determined to be a crossing person in the crossing person recognition process, the first object detection processing unit 32 first assumes that the object is a person. Approximate height H is determined. In this case, the first object detection processing unit 32 determines whether the object is an adult or a child based on the size characteristics of the image of the object determined to be a crossing person.

  For example, the ratio w / h between the horizontal width w of the image corresponding to the head portion of the object in the image captured by the infrared camera 11 and the total length h in the vertical direction of the image of the object If it is greater than the predetermined threshold value, it is determined that the object is a child. When the ratio w / h is equal to or less than the predetermined threshold, it is determined that the object is an adult.

  If the first object detection processing unit 32 determines that the object is an adult, the first object detection processing unit 32 determines the standard height (for example, 170 cm) of the adult as the approximate height H of the object, and the object Is determined to be a child, a standard height (eg, 120 cm) of the child is determined as the approximate height H of the object.

  Further, the first object detection processing unit 32 is stored in advance in a memory (not shown) in advance, the height H of the object determined, the total length h in the vertical direction of the image of the object in the captured image. From the focal length F of the infrared camera 11, an estimated value of the distance Z between the host vehicle 1 and the object (crossing person) is calculated by the following equation (1).

Z = F · H / h (1)

In addition, the first object detection processing unit 32 geometrically calculates the position of the object (crossing person) in the captured image (the position in the horizontal and vertical directions with respect to the image center), the distance Z, and the focal length F. With this calculation, an estimated value of the position (position in real space) of the object (crossing person) in the vehicle width direction and height direction of the vehicle 1 is calculated.

  More specifically, Z / in the horizontal position x (position in the unit of length) and vertical position y (position in the unit of length) of the object (crossing person) in the captured image, respectively. By multiplying F, the position of the object in the vehicle width direction of the vehicle 1 (position in the X-axis direction in FIG. 1) and the position of the object in the height direction of the vehicle 1 (in the X-axis direction and Y in FIG. 1). Position in a direction orthogonal to the axial direction).

  Accordingly, the distance Z and the real space position of the object determined as a crossing person in the crossing person recognition process in STEP 2-4 are estimated.

  On the other hand, for objects other than the object determined to be a crossing person, the first object detection processing unit 32 determines from the own vehicle 1 based on the temporal change rate of the image size of the object in the captured image. The distance of the object is estimated.

  Specifically, for example, the length h (k) in the vertical direction of the image of the object extracted at time k (current time) and the same object (same label) extracted at time k−1 immediately before that Of the object) and the ratio h (k) / h (k-1) (the enlargement ratio of the image of the object) to the length h (k-1) in the vertical direction of the image and the current vehicle 1 From the vehicle speed V and the time difference Δt (control processing cycle) between the times k and k−1, the estimated value of the distance Z between the host vehicle 1 and the object is calculated by the following equation (2).

Z = V · Δt / ((h (k) / h (k-1))-1) (2)

Further, the first object detection processing unit 32, as in the case where the object is a crossing person, the position of the object in the captured image (the position in the horizontal and vertical directions with respect to the image center), the distance Z, From the focal length F, an estimated value of the position of the object (position in real space) in the vehicle width direction and height direction of the vehicle 1 is calculated by geometric calculation.

  Thereby, the distance Z and real space position of objects other than the object determined to be a crossing person by the crossing person recognition process of STEP2-4 are estimated.

  Supplementally, two infrared cameras constituting a stereo camera may be mounted on the vehicle 1. In that case, the distance from the own vehicle 1 of an arbitrary object can be estimated by a stereo distance measuring method. In addition to the infrared camera 11, a ranging device such as a radar may be mounted on the vehicle 1, and the distance of the object from the own vehicle 1 may be measured by the ranging device.

  Next, the first object detection processing unit 32 executes region determination processing in STEP 2-6. This area determination process is a process of determining whether or not the object shown in the captured image is an object that is likely to contact the host vehicle 1.

  Specifically, the first object detection processing unit 32 regards the position of the object (real space) for each object (object that may be a specific species of living body) whose distance and real space position are estimated. If the position is a position in the contact determination area AR1 (stipple area in FIG. 1) set as shown in FIG. 1 in the imaging area AR0, the object (for example, an object indicated by P1 in FIG. 1) It is determined that there is a possibility of future contact between the vehicle and the vehicle 1.

  The contact determination area AR1 is a distance value Z1 (for example, a value obtained by multiplying the vehicle speed by a predetermined proportionality constant) determined in accordance with the vehicle speed (detection value) of the own vehicle 1 in the imaging area AR0. ) And a region having a width (= α + 2β) obtained by adding a predetermined margin width β to each of both sides of the vehicle width α of the host vehicle 1 in front of the host vehicle 1 in front of the vehicle.

  In addition, in the first object detection processing unit 32, the presence position (real space position) of the detection target living body is set as illustrated in FIG. 1 in the left and right outside of the contact determination area AR1 in the imaging area AR0. Even when the position is within the entry determination areas AR2 and AR3 (lane areas in FIG. 1) and the direction of the movement vector of the object is the direction to enter the contact determination area AR1, the object (for example, FIG. 1 is determined that there is a possibility of future contact between the vehicle 1 and the subject vehicle 1.

  The entry determination areas AR2 and AR3 are set as areas in the imaging area AR0 excluding the contact determination area AR1 from the area where the distance from the host vehicle 1 is equal to or less than the distance value Z1.

  Further, the direction of the movement vector of the object is specified from, for example, a time series of real space positions up to a predetermined time before the object.

  Note that the contact determination area AR1 and the entry determination areas AR2 and AR3 are areas having a range in the height direction of the host vehicle 1 (a region having a predetermined height higher than the vehicle height of the host vehicle 1). Then, it is determined that an object present at a position higher than the predetermined height has no possibility of future contact with the host vehicle 1.

  In the region determination process of STEP 2-6, the presence or absence of the possibility of contact between the object and the host vehicle 1 is determined as described above.

  Next, the first object detection processing unit 32 executes a structure / vehicle recognition process in STEP 2-7. Note that the processing after STEP 2-7 (including the processing of STEP 2-7) is an object that excludes the object that is determined not to be in contact with the host vehicle 1 in the region determination process (the host vehicle). 1 is a process executed with respect to an object having a possibility of contact with 1).

  The structure / vehicle recognition process is a process for determining whether the object is a structure or a vehicle.

  In this structure / vehicle recognition process, the first object detection processing unit 32 includes an image area (mask area) for determining whether or not the object is a structure or a vehicle in the captured image of the infrared camera 11. ) Is set in the vicinity of the image (high brightness portion) of the object. The size of the image area is determined according to the distance calculated in STEP 2-5 for the object.

  Then, the first object detection processing unit 32 determines whether the object is a structure or a vehicle based on the shape or size characteristics of the image of the object in the image area.

  For example, when the image of the object has a straight portion or a right-angled corner, or has a pre-registered structure or vehicle shape, or the distance from the host vehicle 1 and the shape of the image are the same. An image of a certain object (a pair of objects corresponding to a pair of taillights or headlights of another vehicle) is within a predetermined range in the horizontal direction of the captured image (within a predetermined range corresponding to the distance from the host vehicle 1) If the objects are arranged at intervals, it is determined that the object is a structure or a vehicle.

  Next, the first object detection processing unit 32 determines in STEP 2-8 whether or not the object is determined to be a structure or a vehicle for each object that is the object of the structure / vehicle recognition process. Check. The first object detection processing unit 32 then performs the following processing (processing from STEP 2-9) for the object determined to be a structure or a vehicle because the object is not a detection target living body. Is omitted.

  On the other hand, the first object detection processing unit 32 executes a person recognition process in STEP 2-9 for an object that has not been determined to be a structure or a vehicle. This person recognition process is a process for determining whether or not the object is a person (pedestrian).

  In this person recognition processing, the first object detection processing unit 32 sets an image area (mask area) for determining whether or not the object is a person in the captured image of the infrared camera 11 of the object. Set near the image (high brightness area). The image area is, for example, a vertically long rectangular area, and the size (vertical width and horizontal width) of the image area includes the whole of the person's image when the object is assumed to be a person. Thus, it determines according to the distance calculated by STEP2-5 regarding this target object.

  Then, the first object detection processing unit 32 determines whether the object is a person (including a crossing person) based on characteristics such as the shape, size, and luminance distribution of the image of the object in the image region. Determine whether. As a specific method of the determination, for example, a method proposed by the applicant of the present application in Japanese Patent Application Laid-Open No. 2003-284057 can be employed.

  Next, the first object detection processing unit 32 confirms in STEP 2-10 whether or not the object is determined to be a person for each object that is the object of the person recognition process. And about the target object determined to be a person, the 1st target object detection process part 32 will determine the said target object as a detection target biological body in STEP2-14, and will complete | finish the process regarding the said target object.

  On the other hand, the 1st target object detection processing part 32 performs the process from STEP2-11 regarding the target object which was not determined to be a person.

  In STEP 2-11, the first object detection processing unit 32 determines whether or not a predetermined operation condition for animal detection is satisfied. And the 1st target object detection processing part 32 performs the animal recognition process which is a process which determines whether a target object is an animal other than a person in STEP2-12, when the operation conditions of STEP2-11 are satisfied. If the operation condition of STEP2-11 is not satisfied, the execution of this animal recognition process is omitted.

  Here, when there are many people around the own vehicle 1 or when the own vehicle 1 is traveling in an urban area, there is no animal around the own vehicle 1 or the own vehicle 1 However, it is considered very unlikely that animals will appear from the roadside on the running road.

  Therefore, in the present embodiment, the first object detection processing unit 32 includes the total number of persons detected by the person recognition process in STEP 2-9 (the total number of objects determined to be persons) and the second object. When the total number (hereinafter referred to as the total number of human detections) with the total number of people detected up to the present time by the human recognition process of STEP 4-10 described later of the detection processing unit 33 is a predetermined number or more, or When it is determined that the host vehicle 1 is traveling in an urban area, it is determined in STEP 2-11 that the animal detection operation condition is not satisfied.

  Further, the first object detection processing unit 32 determines that when the total number of human detections is less than a predetermined number (including zero) and the area where the vehicle 1 is traveling is not an urban area, STEP2- 11, it is determined that an animal detection operation condition is satisfied.

  In this case, whether or not the host vehicle 1 is traveling in an urban area is determined based on, for example, a captured image of the infrared camera 11 or map information of the navigation device.

  Supplementally, when the total number of human detections is less than the predetermined number, it is determined that the operation condition for animal detection is satisfied regardless of whether or not the area where the vehicle 1 is traveling is an urban area. It may be. In that case, it is not necessary to determine whether or not the host vehicle 1 is traveling in an urban area.

  Alternatively, when it is determined that the area in which the host vehicle 1 is traveling is not an urban area, it may be determined that the animal detection operation condition is satisfied regardless of the total number of human detections. In that case, it is not necessary to determine whether the total number of human detections is equal to or greater than a predetermined number.

  If the operation condition for animal detection is not established in STEP 2-11, the processing in the current (current) control processing cycle of the first object detection processing unit 32 ends.

  On the other hand, when the animal detection operation condition is satisfied in STEP 2-11, the first object detection processing unit 32 executes the animal recognition process in STEP 2-12.

  In this animal recognition process, the first object detection processing unit 32 applies to an object other than a person (in this embodiment, a deer or the like) for an object that has not been determined to be a person in the person recognition process. Or not).

  Specifically, the first object detection processing unit 32 displays an image area (mask area) for determining whether or not the object is an animal (quadruped animal) in the captured image of the infrared camera 11. It is set in the vicinity of the image (high brightness portion) of the object. The image area is, for example, a horizontally long rectangular area, and the size (vertical width and horizontal width) of the image area assumes that the object is a quadruped animal such as a deer. It is determined according to the distance calculated in STEP 2-5 for the object so that the whole is included.

  Then, the first object detection processing unit 32 determines whether or not the object is an animal (quadruped animal) based on characteristics such as the shape and size of the image of the object in the image region. . For example, there is a horizontally long high-luminance portion corresponding to the animal's torso, and there is a high-luminance portion corresponding to the head at the upper portion near the lateral end of the horizontally long high-luminance portion. If there are high-luminance portions corresponding to the front and rear leg portions of the animal at the lower part of the high-luminance portion, the target portion is determined to be an animal.

  Next, the first object detection processing unit 32 confirms in STEP 2-13 whether or not the object is determined to be an animal for each object that is the object of the animal recognition process.

  Then, when there is an object determined to be an animal, the first object detection processing unit 32 determines the object as a detection target living body in STEP 2-14 for the object. Then, the processing in the current control processing cycle is terminated. Moreover, the 1st target object detection process part 32 complete | finishes the process in this control process period, when the target object determined to be an animal does not exist.

  The details of the processing of the first object detection processing unit 32 have been described above.

  Next, the second object detection processing unit 33 will be described. The processing of the first object detection processing unit 32 in TEP4 that is executed when the operation condition of STEP3 is satisfied is executed as shown in the flowchart of FIG.

  First, the second object detection processing unit 33 generates a reverse captured image in STEP 4-1. The inverted captured image is an image obtained by inverting the brightness (gradation value) of each pixel of the captured image of the infrared camera 11 (hereinafter sometimes referred to as an original captured image), that is, each pixel in the inverted captured image. The image is such that the higher the luminance of the corresponding pixel in the original captured image is, the lower the luminance is, and the lower the luminance of the corresponding pixel in the original captured image is.

  Specifically, the luminance value of each pixel of the inverted captured image is obtained by subtracting the luminance value of the corresponding pixel of the original captured image (the pixel at the same position as the pixel of the inverted captured image) from the maximum luminance value (upper limit luminance value). Determined by. For example, it is assumed that the luminance of each pixel of the original captured image is expressed in a range of 8-bit gradation values (gradation values from 0 to 255). In this case, if it is the luminance value Yc (any value of 0 to 255) of any one pixel of the original captured image, the luminance value of the corresponding pixel in the inverted captured image is a gradation value of (255-Yc). It is said.

  By generating the inverted captured image in this manner, as shown in the upper diagram of FIG. 7, the brightness of the image of the person 51 as the specific species living body that appears in the original captured image is lower than the brightness of the background image. In the inverted captured image, the luminance of the image of the person 61 can be made higher than the luminance of the background image, as shown in the lower diagram of FIG.

  Next, the second object detection processing unit 33 executes binarization extraction processing in STEP 4-2. In this binarization extraction process, the second object detection processing unit 33 generates a binarized image composed of pixels of two types of high and low luminance values from the inverted captured image.

  This binarized image is a reverse captured image in which pixels having a luminance equal to or higher than a predetermined second threshold are pixels having a high luminance value (white luminance value) and pixels having a luminance smaller than the second threshold are low. It is generated by setting a pixel having a luminance value (black luminance value).

  The second threshold value is determined based on a luminance distribution state in the original captured image or the inverted captured image, for example, a histogram representing the relationship between the luminance of each pixel of the original captured image or the inverted captured image and the number of pixels (appearance frequency). The

  For example, when the histogram of the original captured image is as shown in FIG. 6, the luminance value (= maximum luminance value−Yth2) obtained by inverting the predetermined luminance value Yth2 in the drawing is the second threshold value Yth2 ′. Set as Hereinafter, the luminance value Yth2 is referred to as a pre-inversion second threshold Yth2.

  The pre-inversion second threshold Yth2 is a luminance in the vicinity of the luminance value Y2 where the number of pixels reaches a peak (maximum value) on the low luminance side in the luminance distribution histogram of the original captured image (for example, within a predetermined range centered on Y2). Is determined to be larger than Y2 so that the luminance of Y2 is less than Yth2.

  Thus, the second threshold value Yth2 ′ is determined by the luminance value (= inverted luminance of Y2 in FIG. 6) at which the number of pixels has a peak (maximum value) on the high luminance side in the histogram of the luminance distribution of the inverted captured image. The second threshold value Yth2 ′ is set to a value smaller than the inversion luminance value of Y2, so that the luminance in the vicinity of (value) (the luminance within a predetermined range centered on the inversion luminance value of Y2) is equal to or greater than Yth2 ′. Is equivalent to

  The second threshold Yth2 ′ may be a predetermined value (constant value) determined in advance. Alternatively, the second threshold Yth2 ′ may be set according to the outside air temperature or the like. The second threshold Yth2 ′ may be the same value as the first threshold Yth1.

  The second threshold value Yth2 ′ is the value of the original captured image when the specific species living body such as a person is imaged by the infrared camera 11 in an environment where the temperature is lower than the temperature of the subject in the background. The brightness of the image of the specific species in the inverted captured image with the brightness inverted may be set to be equal to or higher than the second threshold Yth2 ′.

  In STEP 4-2, the second object detection processing unit 33 generates a binarized image corresponding to the inverted captured image as described above, and then generates a high luminance area (pixels of high luminance values (white pixels) of the binarized image. The image of the target object (which corresponds to the type determination target object in the present invention) is extracted from the region). This extraction process is performed in the same manner as the extraction process in STEP 2-1.

  As described above, the binarization extraction process of STEP4-2 is executed. In this case, the image of the target object extracted from the high luminance region of the binarized image corresponding to the inverted captured image is an image included in the low luminance region of the original captured image.

  Next, the 2nd target object detection process part 33 performs a tracking process in STEP4-3. This tracking process is executed in the same manner as the tracking process of STEP2-2.

  Next, the second object detection processing unit 33 executes the processing of STEP 4-4. In this process, the second object detection processing unit 33 performs the process of the first object detection processing unit 32 (specifically, STEP2- FIG. 4 in FIG. 4) of the entire area of the inverted captured image and the binarized image thereof. 3 or the processing of STEP2-7), the existence area of the image of the object determined to be a structure or a vehicle is excluded from the objects of the subsequent processing (processing from STEP4-5).

  Then, the second object detection processing unit 33 stores and holds the arrangement position and size of the excluded area in a memory (not shown).

  Note that excluding the existence area of the image of the target object that is determined to be a structure or a vehicle means that the detection target living body candidate image does not exist in the existence area.

  Thereafter, the second object detection processing unit 33 uses the inverted captured image or the binarized image thereof (except for the area excluded in STEP 4-4), the crossing person recognition process in STEP 4-5, STEP 4-6. Distance calculation processing in STEP4-7, region determination processing in STEP4-7, structure / vehicle recognition processing in STEP4-8, determination processing in STEP4-9 (confirmation processing whether or not the object is determined to be a structure or a vehicle) The person recognition process in STEP 4-10, the determination process in STEP 4-11 (confirmation process as to whether or not the object is determined to be a person), and the process in STEP 4-12 (process for determining the detection target living body) STEP2-4, STEP2-5, STEP2-6, STEP2-7, STEP2-8, STEP2-9, STEP2-10, respectively To run in the same manner as STEP2-14.

  In this case, in the present embodiment, in the process of the second object detection processing unit 33, a process for determining whether or not the object is an animal (quadruped animal) is performed in order to reduce a calculation processing load or the like. Not done. Specifically, for an object that is determined not to be a person by STEP 4-10 human recognition processing (an object for which the determination result in STEP 4-11 is negative), the object is an animal (quadruped animal). It is not determined whether or not.

  However, in the case where an operation condition for animal detection is established by performing the same process as in STEPs 2-11 to 2-13 for an object that is determined not to be human by the human recognition process in STEP2-9, You may make it determine whether a target object is an animal (quadruped animal).

  The details of the processing of the second object detection processing unit 33 have been described above.

  In addition, if it supplements about the correspondence with this invention, the said captured image acquisition part 31 is corresponded to the captured image acquisition means in this invention.

  Moreover, the 1st target object detection process part 32 contains the function as a 1st target object classification determination means in this invention. In this case, the processes of STEPs 2-3, 2-7, 2-9, and 2-12 of the first object detection processing unit 32 correspond to the first determination process in the present invention.

  Moreover, the 2nd target object detection process part 33 contains the function as a 2nd target object classification determination means in this invention. In this case, the processes of STEPs 4-8 and 4-10 of the second object detection processing unit 33 correspond to the second determination process in the present invention.

  According to the present embodiment described above, the first object detection processing unit 32 that executes the process of detecting the detection target living body from the high luminance region of the captured image (original captured image) of the infrared camera 11, and the original captured image. By providing the second object detection processing unit 33 that executes the process of detecting the detection target living body from the low brightness area (high brightness area of the reverse captured image), the temperature of the detection target living body is higher than the temperature of the background subject. It is possible to detect the detection target living body in both cases of high temperature and low temperature.

  In addition, the processing from STEP 4-5 of the second object detection processing unit 33 is performed based on the existence region of the image of the object that has already been determined to be a structure or a vehicle by the processing of the first object detection processing unit 32. Excluded.

  For this reason, the calculation load of the process of the 2nd object detection process part 33 can be reduced by keeping the object which performs classification determination by the process of the 2nd object detection process part 33 to the minimum necessary. . As a result, the total time required for the processing of both the first object detection processing unit 32 and the second object detection processing unit 33 can be made as short as possible, and the detection of the detection target target living body is delayed. Can be prevented.

  Further, in the processing of the first object detection processing unit 32, when the operation condition of STEP2-11 is not satisfied (when it is very unlikely that a quadruped animal such as a deer is present around the host vehicle 1), The recognition process is not executed, and the animal recognition process is executed only when the operation condition is satisfied.

  For this reason, even if there are many people around the host vehicle 1 and the calculation load of the human recognition process increases, the total calculation load of the first object detection processing unit 32 is suppressed from being excessive. The

  Even if the operation condition of STEP 2-11 is established and the animal recognition process is executed, in this case, it is unlikely that there are many people around the own vehicle 1, so STEP 2- The calculation load in the human recognition process 9 is relatively small. Accordingly, even in this case, the total calculation load of the first object detection processing unit 32 is suppressed from becoming excessive.

  Therefore, it is possible to prevent the calculation load of the first object detection processing unit 32 from becoming excessive, and consequently to prevent the detection of any detection target living body existing in the imaging region AR0 from being delayed. be able to.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG. Note that the present embodiment is different from the first embodiment only in a part of the processing of the first object detection processing unit 32. For this reason, the description of the present embodiment will be centered on matters that are different from the first embodiment, and description of the same matters as those of the first embodiment will be omitted.

  Referring to FIG. 9, in the present embodiment, the first object detection processing unit 32 replaces the processing of STEP 2-9 to 2-14 after STEP 2-8 of FIG. 4 with the flowchart of FIG. 9. Steps 2-15 to 2-20 shown are executed.

  In other words, the first object detection processing unit 32 in the present embodiment is an object for which the determination result in STEP 2-8 described in the first embodiment is negative (in the structure / vehicle recognition process in STEP 2-7, In STEP 2-15, an animal recognition process for determining whether or not the target object is an animal other than a human (quadruped animal in the present embodiment) is executed for the target object determined not to be a structure or a vehicle. This animal recognition process of STEP2-15 is executed in the same manner as STEP2-12 in the first embodiment.

  Next, the first object detection processing unit 32 confirms in STEP 2-16 whether or not the object is determined to be an animal (quadruped animal) for each object that is the object of the animal recognition process. To do. And about the target determined to be an animal, the 1st target detection process part 32 will determine the said target as a detection target biological body in STEP2-20, and will complete | finish the process regarding the said target.

  On the other hand, the 1st target object detection processing part 32 performs the process from STEP2-17 regarding the target object which was not determined to be an animal.

  In STEP 2-17, the first object detection processing unit 32 determines whether or not a predetermined operation condition for human detection is satisfied. Then, when the operation condition of STEP2-17 is satisfied, the first object detection processing unit 32 executes a person recognition process for determining whether or not the object is a person in STEP2-18, and STEP2- When the operation condition 17 is not satisfied, the execution of the person recognition process is omitted.

  Here, when many animals exist around the own vehicle 1, it is considered that the possibility that there is a person around the own vehicle 1 is extremely low.

  Therefore, in the present embodiment, the first object detection processing unit 32 determines the total number of animals detected by the animal recognition process in STEP 2-15 (the total number of objects determined to be animals. Hereinafter, the total number of animal detections). ) Is more than a predetermined number, it is determined in STEP 2-17 that the operating condition for human detection is not satisfied.

  Moreover, the 1st target object detection process part 32 judges that the operating condition for person detection is materialized in STEP2-17, when the said animal detection total is less than predetermined number (including zero).

  If the operation condition for human detection is not satisfied in STEP 2-17, the processing in the current (current) control processing cycle of the first object detection processing unit 32 ends.

  On the other hand, when the operation condition for animal detection is established in STEP 2-17, the first object detection processing unit 32 executes the person recognition process in STEP 2-18.

  This person recognition process is executed in the same manner as in STEP 2-9 in the first embodiment.

  Next, the first object detection processing unit 32 confirms in STEP 2-19 whether or not it is determined that the object is a person for each object that is the object of the human recognition process in STEP 2-18.

  Then, if there is an object determined to be a person, the first object detection processing unit 32 determines the object as a detection target living body in STEP 2-20 for the object. Then, the processing in the current control processing cycle is terminated. Moreover, the 1st target object detection process part 32 complete | finishes the process in this control process period, when the target object determined to be a person does not exist.

  The above is the processing of the first object detection processing unit 32 in the present embodiment. The items other than those described above are the same as those in the first embodiment.

  In addition, if it supplements about the correspondence with this invention, the said captured image acquisition part 31 is corresponded to the captured image acquisition means in this invention.

  Moreover, the 1st target object detection process part 32 contains the function as a 1st target object classification determination means in this invention. In this case, the processes of STEPs 2-3, 2-7, 2-15, and 2-18 of the first object detection processing unit 32 correspond to the first determination process in the present invention.

  Moreover, the 2nd target object detection process part 33 in this embodiment contains the function as a 2nd target object classification determination means in this invention similarly to the case of the said 1st Embodiment. In this case, the processes of STEPs 4-8 and 4-10 of the second object detection processing unit 33 correspond to the second determination process in the present invention.

  According to the present embodiment described above, as in the first embodiment, the detection target living body is detected regardless of whether the temperature of the detection target living body is higher or lower than the temperature of the background subject. It can be made to detect.

  Similarly to the first embodiment, the process from STEP 4-5 of the second object detection processing unit 33 has already been determined to be a structure or a vehicle by the process of the first object detection processing unit 32. This is performed by excluding the existence area of the image of the object.

  For this reason, the calculation load of the process of the 2nd object detection process part 33 can be reduced by keeping the object which performs classification determination by the process of the 2nd object detection process part 33 to the minimum necessary. .

  Further, in the present embodiment, in the process of the first object detection processing unit 32, when the operation condition of STEP2-17 is not satisfied (when it is very unlikely that there is a person around the host vehicle 1), The recognition process is not executed, and the person recognition process is executed only when the operation condition is satisfied.

  For this reason, even if there are many animals around the host vehicle 1 and the calculation load of the animal recognition process increases, the total calculation load of the first object detection processing unit 32 is suppressed from being excessive. The

  Even if the operation condition of STEP2-17 is established and the person recognition process is executed, in this case, it is unlikely that there are many animals around the host vehicle 1, so STEP2- The calculation load in the 15 animal recognition processing is relatively small. Accordingly, even in this case, the total calculation load of the first object detection processing unit 32 is suppressed from becoming excessive.

  Therefore, it is possible to prevent the calculation load of the first object detection processing unit 32 from becoming excessive, and consequently to prevent the detection of any detection target living body existing in the imaging region AR0 from being delayed. be able to.

[Modification]
Next, a modification of the first embodiment or the second embodiment will be described.

  In the first embodiment, in the processing of the first object detection processing unit 32, when the operation condition for animal detection of STEP2-11 is not satisfied, the animal recognition processing of STEP2-12 is not performed. However, when the operation condition for animal detection is not satisfied, the processing with a smaller amount of calculation than when the operation condition is satisfied (for example, the condition for determining that the object is an animal is determined from the animal recognition processing in STEP 2-12). Whether or not the object is an animal by a process that has been relaxed, or a process in which the position of the object in the captured image for determining whether or not it is an animal is more limited than the animal recognition process in STEP 2-12). It may be determined whether or not.

  Further, in the processing of the first object detection processing unit 32 in the second embodiment, when the operation condition for human detection in STEP 2-17 is not satisfied, the amount of calculation is smaller than when the operation condition is satisfied. Processing (for example, processing in which the condition for determining that the object is a person is more relaxed than the human recognition process in STEP 2-18, or the position of the object in the captured image for determining whether the object is a person is determined. It may be determined whether or not the object is a person by a process limited to the person recognition process of STEP 2-18).

  Further, in the processing of the second object detection processing unit 33 in the first embodiment or the second embodiment, the first object in the first embodiment is related to the object that is determined not to be a person in the person recognition process in STEP 4-10. Similar to the object detection processing unit 32, the animal recognition process may be executed when the animal detection operation condition is satisfied.

  Further, for example, in the process of the second object detection processing unit 33 in the first embodiment or the second embodiment, it is possible to determine whether the object is a structure or a vehicle, and the first object When one of the detection processing unit 32 and the second object detection processing unit 33 determines that a certain object is a structure or a vehicle, the other processing unit Processing for determining the type of the target object may be performed by excluding the existence area of the image.

  Further, in the processing of the second object detection processing unit 33 in the first embodiment or the second embodiment, the inverted captured image is used. However, the original captured image is used as it is, and the target is detected from the low luminance region of the original captured image. An object may be extracted and the type of the object may be determined. However, by using the reverse captured image as in the first embodiment or the second embodiment, a program for many processes of the first object detection processing unit 32 and the second object detection processing unit 33 is shared. can do. As a result, the design cost of the object monitoring apparatus 10 can be reduced.

  In the first embodiment or the second embodiment, the first object detection processing unit 32 and the second object detection processing unit 33 may determine different types of objects.

  Further, in the first embodiment or the second embodiment, the case where the infrared camera 11 is mounted on the vehicle 1 has been described as an example. However, the infrared camera 11 may be a road or an entrance of a facility (such as a living body using an infrared camera). It may be installed at any location to be monitored).

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 10 ... Object monitoring apparatus, 11 ... Infrared camera, 31 ... Captured image acquisition part (captured image acquisition means), 32 ... 1st object detection part (1st object type determination means), 33 ... 1st 2 object detection part (2nd object classification determination means).

Claims (4)

An object monitoring apparatus having a function of determining the type of an object existing in an imaging area of an infrared camera,
The brightness of an image of the object changes according to a temperature difference between an arbitrary object existing in the imaging region of the infrared camera and the background of the object, and the object is more than the background. Using the infrared camera, a captured image having characteristics that the higher the temperature of the object, the higher the brightness of the image of the object, and the lower the temperature of the object than the background, the lower the brightness of the image of the object. Captured image acquisition means for acquiring
In the captured image, an image of a type determination target object, which is a target for performing type determination, is extracted from a high luminance region having a luminance equal to or higher than a predetermined first threshold, and based on the image of the type determination target First object type determination means for executing a first determination process for determining the type of the type determination object;
An image of the type determination target object is extracted from a low luminance region having a luminance equal to or lower than a predetermined second threshold in the captured image, and the type of the type determination target object is determined based on the image of the type determination target object Second object type determination means for executing a second determination process to perform,
At least one of the first object type determination unit and the second object type determination unit is an area corresponding to the one of the high luminance region and the low luminance region in the captured image. The first determination process or the second determination process is performed in an area excluding a partial area in which an image of a type determination target object whose type has already been determined by the other means is present. Object monitoring device. The object monitoring device according to claim 1,
Both the determination processes of the first determination process and the second determination process include a process of determining whether or not the type determination object is a predetermined type of living body based on an image of the type determination object. The second object type determining means determines that the type determining object is a predetermined type of living body on the condition that an environmental condition that the outside air temperature is higher than a predetermined temperature or is raining is satisfied. The first object type determination means is configured to execute a process for determining whether or not the type determination object is a predetermined type regardless of whether or not the environmental condition is satisfied. An object monitoring apparatus configured to execute a process of determining whether or not a living body. The object monitoring device according to claim 2,
In both the first determination process and the second determination process, the person recognition that determines whether or not the type determination object is a person as the predetermined type of living body based on the image of the type determination object. Each includes processing,
The first determination process or both determination processes of the first determination process and the second determination process determine whether or not the type determination object is an animal other than a human when at least a predetermined operation condition is satisfied. An animal recognition process for determining based on an image of the type determination object,
Of the first object type determination means and the second object type determination means, each determination means for executing a determination process including the animal recognition process includes the first object type determination means and the second object type determination When the total number of type determination objects determined to be a person by execution of the person recognition process by both determination means of the means is less than a predetermined number, or the place where the vehicle equipped with the infrared camera is running is When it is determined that the vehicle is not an urban area, the animal recognition process is executed and the total number of type determination objects determined to be the person is greater than or equal to a predetermined number, assuming that the predetermined operation condition is satisfied Or, when it is determined that the place where the vehicle is traveling is an urban area, the execution of the animal recognition process is omitted, or the calculation amount is reduced as compared with the case where the predetermined operation condition is satisfied. Where Object monitoring apparatus characterized by being configured such that the type determination object to determine whether an animal other than humans by. The object monitoring device according to claim 2,
Both the first determination process and the second determination process determine whether the type determination target is a person as the predetermined type of living body at least when a predetermined operation condition is satisfied. Each includes human recognition processing to determine based on the image of the object,
The first determination process or both determination processes of the first determination process and the second determination process determine whether the type determination target object is an animal other than a person based on the image of the type determination target object. Further comprising animal recognition processing to
Both determination means of the first object type determination means and the second object type determination means include a determination process including the animal detection process among the first object type determination means and the second object type determination means. When the total number of type determination objects determined to be the animal by execution of the animal detection process by each determination means to be executed is less than a predetermined number, the predetermined operation condition is satisfied, If the total number of type determination objects determined to be animals is a predetermined number or more, execution of the recognition process is omitted, or the predetermined operating condition is An object monitoring apparatus configured to determine whether or not the type determination object is a person by a process in which the amount of calculation is reduced as compared with the case where the above is established.