JP2017215743A - Image processing device, and external world recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect objects such as other vehicle and the like approaching to a vicinity from a far direction of an own vehicle in a shot image.SOLUTION: An imaging processing device 10 comprises: a first detection unit 101; and a second detection unit 104. The first detection unit 101 is configured to extract candidates of other vehicles existing in a far area corresponding to a point away by a first distance with respect to a camera 20 from a first photograph image shot by a camera at first time and obtained. The second detection unit 104 is configured to detect other vehicles existing in a near area corresponding to a point away by a second distance closer than the first distance with respect to the camera 20 from a second photograph image shot by the camera 20 at second time later than the first time and obtained. In this instance, the second detection unit 104 is configured to determine whether the candidate of the other vehicle extracted by the first detection unit 101 is other vehicle on the basis of the second photograph image, and thereby detect the other vehicle as an object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus and an external environment recognition apparatus.

  In recent years, in order to avoid accidents such as collisions between vehicles and human-vehicle collisions, the situation around the vehicle is monitored with an in-vehicle camera, and when a danger is detected, an alarm is output to the driver and Technology to automatically control behavior is advancing. In such a technique, when an object located far away from the host vehicle is photographed by the in-vehicle camera, the object has a small reflection range on the camera, so that the resolution of the image area corresponding to the object is lowered. As a result, characteristic information such as the vehicle shape is difficult to obtain, and non-detection and erroneous detection are likely to occur, so that stable control is difficult.

  For example, a technique described in Patent Document 1 has been proposed as a technique for solving the problems in the detection of a distant vehicle as described above. Patent Document 1 discloses a method for recognizing an object such as a vehicle around a vehicle by comparing two images taken at a predetermined time interval with a predetermined template image.

JP 2005-318546 A

  In the technique described in Patent Document 1, the first photographed image photographed when an object such as a vehicle is far away from the host vehicle is also compared with the template image. However, as described above, when the object to be recognized exists far from the host vehicle, the resolution of the image area corresponding to the object is low, so the first captured image accurately reflects the feature of the object. It is hard to be a thing. Therefore, even if the method of Patent Document 1 is used, it is difficult to accurately detect an object such as another vehicle that approaches the vicinity from a distance of the own vehicle in the captured image.

An image processing apparatus according to the present invention extracts a candidate for an object existing at a point away from the camera by a first distance from a first photographed image obtained by photographing with a camera at a first time. And a first captured image obtained by photographing with the camera at a second time after the first time, and closer to the camera than the first distance A second detection unit that detects the object existing at a point separated by a second distance, and the second detection unit is extracted by the first detection unit based on the second captured image. The target object is detected by identifying whether or not the candidate for the target object is the target object.
An external environment recognition apparatus according to the present invention includes an image processing device, and based on a detection result of the other vehicle by the second detection unit, an alarm signal for warning the driver of the own vehicle and an operation of the own vehicle At least one of vehicle control signals for controlling the vehicle is output.

  According to the present invention, it is possible to accurately detect an object such as another vehicle approaching the vicinity from a distance of the host vehicle in a captured image.

It is a block diagram which shows the function structure of the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of the picked-up image output from a camera. It is a figure which shows the example of the image for learning used when producing | generating a detector. It is a figure which shows the example of a setting of a distant area | region and a near region in a picked-up image. It is a figure explaining a motion vector verification process. It is a figure explaining a spatial frequency verification process. It is a figure which shows the example of the characteristic curve which shows the relationship between the total verification score TSC and the sensitivity setting value CC. It is a figure which shows the processing flow of the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the external field recognition apparatus which concerns on the 2nd Embodiment of this invention.

-First embodiment-
Hereinafter, an image processing apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a functional configuration of an image processing apparatus 10 according to the first embodiment of the present invention. An image processing apparatus 10 shown in FIG. 1 is mounted on a vehicle and used. In the following description, a vehicle on which the image processing apparatus 10 is mounted is referred to as “own vehicle”, and other vehicles existing around the own vehicle are referred to as “other vehicles”.

  The image processing apparatus 10 is connected to a predetermined position of the host vehicle corresponding to the shooting area, for example, a camera 20 attached to the body of the host vehicle. The image processing apparatus 10 includes a first detection unit 101, a verification unit 102, a determination unit 103, a second detection unit 104, and a setting change unit 105. Each function of the image processing apparatus 10 shown in FIG. 1 can be realized by appropriately combining hardware such as a microcomputer and a memory, and various programs executed on the microcomputer.

  The camera 20 shoots other vehicles existing around the host vehicle with moving images or still images at predetermined time intervals, and each frame or each still image of the acquired moving images is captured as a captured image at predetermined time intervals as a first detection unit. 101. In addition, in order to make it easy to recognize another vehicle in the image processing apparatus 10, the camera 20 can be installed at an arbitrary location of the host vehicle. For example, the camera 20 is installed in a part such as a front bumper, a rear bumper, and left and right side mirrors of the host vehicle. Or you may install the camera 20 in the vehicle of the own vehicle. Furthermore, for the purpose of recognizing other vehicles existing only in a specific area around the own vehicle, the camera 20 may be installed alone, or the other vehicles can be recognized in all areas around the own vehicle. As described above, a plurality of cameras 20 may be installed.

  FIG. 2 is a diagram illustrating an example of a captured image output from the camera 20 to the first detection unit 101. FIG. 2A is an example of a photographed image obtained by photographing the other vehicles 203 and 204 running behind the host vehicle when the camera 20 is installed in the rear bumper portion of the host vehicle. is there. In the captured image of FIG. 2A, on the road 202 extending behind the host vehicle, the other vehicle 203 existing in a distant area near the horizon 201 with respect to the host vehicle, and the host vehicle The other vehicle 204 existing in the nearby area is reflected.

  FIG. 2B is an example of a captured image in which only the other vehicle 203 existing in a distant area is reflected. In the captured image of FIG. 2B, a cutout area 205 is set in a predetermined range around the other vehicle 203. The first detection unit 101 can cut out the image of the cutout area 205 and use it for machine learning for recognizing the other vehicle 203 from the captured image.

  FIG. 2C is an example of a photographed image obtained by photographing another vehicle 203 existing in a distant area when a fisheye lens is used for the camera 20. The captured image in FIG. 2C shows a situation in which the other vehicle 203 approaches from the rear right side of the host vehicle using a fisheye lens to the camera 20 installed in the right side mirror portion of the host vehicle. As in FIG. 2B, a cutout area 205 for cutting out an image used for machine learning in the first detection unit 101 is set in the captured image. That is, the photographed image by the fisheye lens illustrated in FIG. 2C has an advantage that a wide range in the vertical and horizontal directions can be photographed, unlike a photographed image by a general lens. An image used for machine learning can also be cut out from an image taken with such a fisheye lens.

  The 1st detection part 101 detects the candidate of the other vehicle which exists in the area | region far from the own vehicle from the picked-up image input from the camera 20, and extracts it as another vehicle candidate. The first detection unit 101 detects a candidate for another vehicle from the photographed image using, for example, a detector to which a machine learning technique is applied.

  FIG. 3 is a diagram illustrating an example of a learning image used when generating the detector of the first detection unit 101. FIG. 3A is an example of a learning image when the other vehicle 203 exists in a distant area, and FIG. 3B is an example of a learning image when the other vehicle 203 does not exist. In the learning image of FIG. 3A, the other vehicle 203 is reflected on the road 202 in a distant region near the horizon 201, whereas in the learning image of FIG. There is no reflection. Note that the learning image in FIG. 3A is an image obtained by cutting out the cutout area 205 from the captured image in FIG. 2B as described above.

  The first detection unit 101 uses a machine learning type detector configured by prior learning using the two types of learning images as described above to photograph other vehicles existing in a region far from the host vehicle. It can be recognized on the image and extracted as another vehicle candidate. In actual operation, the form of the other vehicle that appears in the captured image varies depending on the type and direction of the other vehicle. Therefore, in order to obtain a practical detection performance, it is preferable to prepare a plurality of learning images, for example, several thousand to several tens of thousands, respectively, in the case where there is another vehicle and the case where there is no other vehicle.

  Here, when a fish-eye lens is used for the camera 20 as shown in FIG. 2C, the fish-eye lens has a lens condensing characteristic that is wider than that of a general lens to widen the imaging range. Therefore, the captured image obtained is distorted, and the resolution of the image tends to decrease as the region approaches the upper, lower, left, and right ends of the image, that is, the region farther from the host vehicle. However, in the image processing apparatus 10 of the present embodiment, it is only necessary to recognize that there is another vehicle on the road in the first detection unit 101, and it is not necessary to recognize the shape of the other vehicle. Therefore, a stable detection result can be obtained even when the resolution is low.

  By the way, in general, the machine learning used in the first detection unit 101 is to input a plurality of images of an object to be detected, extract an image feature representing the object from the image, and input an unknown input. For images, this is a processing method that automatically sets the parameters of the classifier so that the learned image features can be detected and identified. As a specific example of such a processing method, for example, Deep Learning is known. Deep learning can automatically extract feature parameters of images that are shared by multiple input images. As an example of a feature parameter extraction method, a feature extraction method using a neural network structure is known. In the neural network structure, a large number of input / output functions (activation functions) called neuron units that react only when matching the image features common to the input image group are combined for each small image area. It is stacked in multiple layers to form a pyramid structure. According to this method, classifier parameters are extracted for each layer of the neuron unit so that the object can be identified step by step while changing the position and image size of the object to be detected. Thus, a discriminator parameter capable of discriminating the entire object can be obtained.

  FIG. 4 is a diagram illustrating an example of setting a far area and a near area in a captured image. As illustrated in FIG. 4, the first detection unit 101 sets a boundary line 401 at a predetermined position on the captured image, for example, and sets a range from the boundary line 401 to the horizon 201 on the road 202 as the far region R1 and the boundary line. A range from 401 to the lower end of the image is set in the vicinity region R2. Then, by performing the above processing for the far region R1, the other vehicle candidate is extracted, and the information on the position, shape, size, and the like is output to the verification unit 102 as information indicating the extraction result of the other vehicle candidate. To do.

  Note that the ranges of the distant area and the near area in the captured image may be basically set arbitrarily. However, in reality, the maximum detection distance of the second detection unit 104, that is, the maximum value of the distance between the own vehicle and the other vehicle in which the second detection unit 104 can detect the other vehicle from the captured image by a method described later. Therefore, it is desirable to set these area ranges. Specifically, a boundary line 401 is set at a position corresponding to the maximum detection distance of the second detection unit 104 on the captured image, and the image region above this, that is, the second detection unit 104 cannot detect another vehicle. The region may be set as the far region R1, and the lower image region may be set as the near region R2.

  The verification unit 102 verifies the possibility that the other vehicle candidate extracted by the first detection unit 101 is actually another vehicle as a detection target. The verification unit 102 verifies whether or not the other vehicle candidate extracted by the first detection unit 101 is actually another vehicle, using a plurality of photographed images input from the camera 20 every predetermined time. Specifically, the verification unit 102 obtains a motion vector indicating the motion of another vehicle candidate from a plurality of captured images, and verifies the possibility that the other vehicle candidate is another vehicle based on the feature of the motion vector. Perform verification processing. Furthermore, spatial frequency verification processing is performed for obtaining a spatial frequency distribution of an image region including other vehicle candidates in a plurality of captured images and verifying the possibility that the other vehicle candidate is another vehicle based on the feature of the spatial frequency distribution.

  FIG. 5 is a diagram for explaining motion vector verification processing in the verification unit 102. In the motion vector verification process, when another vehicle candidate is extracted by the first detection unit 101 and information indicating the extraction result is output from the first detection unit 101, the verification unit 102 is input from the camera 20 thereafter. By tracking other vehicle candidates based on information from the first detection unit 101, a motion vector of the other vehicle candidates is extracted. For example, as shown in FIG. 5A, it is assumed that another vehicle candidate 501 is detected in the far region R1 at time T1. In this case, the motion vector 503 can be extracted by tracking the other vehicle candidate 501 from the captured images obtained at the times T2, T3, and T4 starting from the position of the other vehicle candidate 501 at the time T1.

  If the motion vector of the other vehicle candidate can be extracted as described above, the verification unit 102 next determines whether or not the motion vector locus corresponds to a realistic vehicle motion. For example, as shown in FIG. 5B, when a motion vector 503 having a linear trajectory is extracted at an interval substantially equivalent to an equal interval in real space, or as shown in FIG. In the real space, when the motion vectors 504 of the trajectory gently curved at intervals equivalent to approximately equal intervals are extracted, it is assumed that these motion vectors indicate the movements that are likely to be vehicles, and the detected other vehicle candidates are It can be determined that there is a high possibility that the vehicle is actually another vehicle. On the other hand, as shown in FIG. 5 (d), when a motion vector 505 of a trajectory whose direction changes abruptly at an interval substantially equivalent to an equal interval in the real space is extracted, or in FIG. 5 (e). As shown, when motion vectors 506 of a locus whose direction is substantially constant but whose interval changes rapidly are extracted, it is determined that these motion vectors indicate motions that are not likely to be vehicles, and other vehicles detected are detected. It can be determined that the possibility that the candidate is actually another vehicle is low. In the motion vector verification process executed by the verification unit 102, the vehicle-likeness is determined from the degree of coincidence with the actual vehicle motion for various motion vector trajectories. As a result, it is possible to verify whether the other vehicle candidate extracted by the first detection unit 101 is actually another vehicle to be detected, or whether another object has been erroneously detected.

  In addition, in FIG. 5, although the example which extracts the motion vector of the other vehicle candidate 501 from four captured images from the time T1 to the time T4 has been described, the number of captured images to be referred to when extracting the motion vector is this. However, the number of sheets can be any number.

  FIG. 6 is a diagram for explaining the spatial frequency verification processing in the verification unit 102. In the spatial frequency verification process, when another vehicle candidate is extracted by the first detection unit 101, the verification unit 102 includes an image including the other vehicle candidate for each of a plurality of photographed images input from the camera 20 thereafter. Identify the area. At this time, it is preferable to use the motion vector of the other vehicle candidate obtained in the motion vector verification process described above, but other methods that do not use the motion vector may be used. Then, by analyzing the spatial frequency of the image area specified in each captured image, it is determined whether or not a vehicle actually exists in the image area. Spatial frequency is information indicating the nature of a wavelength structure having a spatial period in a two-dimensional image, and can be obtained using, for example, Fourier transform. Specifically, as shown in FIG. 6, the spatial frequency is obtained by converting the value of each pixel in the image area specified in the captured image into a frequency component in the horizontal direction and a frequency component in the vertical direction and digitizing the value. Can be visualized.

  Here, FIG. 6 shows an example of a spatial frequency image obtained for an image region having 20 horizontal pixels and 14 vertical pixels. That is, by performing Fourier transform, an image having the same size as the original image region can be obtained as an image showing a spatial frequency component. In this spatial frequency image, the closer to the center of the image, the lower the horizontal and vertical spatial frequencies, and the farther away from the center of the image, the higher the horizontal and vertical spatial frequencies. The smaller the content of each frequency component, the more black it is displayed. Due to the characteristics of the Fourier transform, as shown in FIG. 6, in the spatial frequency image, quadrant images that are point-symmetric in four quadrants with respect to the image center are obtained. However, since these quadrant images represent almost the same spatial frequency component results, it is only necessary to focus on any one quadrant image, for example, the quadrant image in the first quadrant at the upper right.

  FIG. 6A is an example in which a spatial frequency component is visualized in a captured image in which another vehicle on the road is reflected, such as the learning image illustrated in FIG. When there is another vehicle in the image, the geometric shape of the other vehicle exists in a certain range in the image. Therefore, as shown in FIG. 6A, the spatial frequency ranges from a low frequency to a high frequency. Widely distributed. Therefore, when the state of the spatial frequency of the image region including the other vehicle candidate shows the same tendency as in FIG. 6A, it can be determined that there is a high possibility that the other vehicle actually exists.

  FIG. 6B is an example in which a spatial frequency component is visualized for a captured image in which no other vehicle is reflected, such as the learning image illustrated in FIG. When there is no other vehicle in the image, a uniform image pattern of only the road is obtained. Therefore, as shown in FIG. 6B, the spatial frequency has almost no high frequency component, and the image center is low. The distribution is concentrated only in the vicinity of the frequency component. Therefore, when the state of the spatial frequency of the image area including the other vehicle candidate shows the same tendency as in FIG. 6B, it can be determined that the possibility that the other vehicle actually exists is low.

  The verification unit 102 executes the two types of processing described above, and is extracted by the first detection unit 101 based on a plurality of captured images obtained by the camera 20 at different times. The possibility that the other vehicle candidate is another vehicle can be verified. Then, based on the verification results of the motion vector verification process and the spatial frequency verification process, a verification score relating to the probability that the other vehicle candidate is another vehicle is calculated and output to the determination unit 103 and the setting change unit 105. . Specifically, the verification unit 102 determines the verification score SC1 according to the degree of coincidence with the actual movement of the other vehicle by the motion vector verification process, and the verification score according to the existence possibility of the other vehicle by the spatial frequency verification process. SC2 is calculated, and these are added together to calculate a total verification score TSC, which is output to the determination unit 103 and the setting change unit 105.

  The determination unit 103 determines whether or not the second detection unit 104 detects another vehicle from the latest captured image based on the verification result by the verification unit 102. Specifically, the determination unit 103 compares the value of the total verification score TSC calculated by the verification unit 102 with a preset threshold value TH. As a result, when the total verification score TSC is larger than the threshold value TH and the position of the other vehicle candidate in the latest photographed image is within the detection region of the second detection unit 104, that is, the above-described neighboring region R2, In order to determine whether or not the other vehicle candidate is actually another vehicle by the vehicle detection process executed by the second detection unit 104, an operation instruction is given to the second detection unit 104. On the other hand, when at least one of these conditions is not satisfied, it is determined that it is not necessary to determine whether the other vehicle candidate is actually another vehicle, and an operation instruction is issued to the second detection unit 104. Absent.

  The second detection unit 104 identifies whether the other vehicle candidate is actually another vehicle based on the latest captured image acquired after the other vehicle candidate is extracted by the first detection unit 101. Detects other vehicles around the host vehicle. The second detection unit 104 identifies whether or not the other vehicle candidate is another vehicle by executing a predetermined vehicle detection process on the latest photographed image. Specifically, the second detection unit 104 uses, for example, a machine learning type detector similar to that described in the first detection unit 101 to determine whether the other vehicle candidate has characteristics as an actual vehicle. By determining, it is possible to identify whether the vehicle is another vehicle. At this time, the actual vehicle is not determined based on the characteristics such as the color and shape of the image portion extracted as the other vehicle candidate, instead of simply identifying that there is another vehicle on the road as in the first detection unit 101. It is preferable to identify whether or not this is true. When the other vehicle is detected from the latest photographed image, the second detection unit 104 outputs a vehicle approach signal indicating that the other vehicle is approaching. Further, when it is difficult to detect other vehicles, for example, the captured image is entirely dark, a detection FAIL signal is output.

  Based on the value of the total verification score TSC calculated by the verification unit 102, the setting change unit 105 dynamically changes the sensitivity setting value when the second detection unit 104 detects another vehicle from the latest captured image. To do. Specifically, the sensitivity setting value CC in the vehicle detection process executed by the second detection unit 104 is changed such that the higher the total verification score TSC value, the easier it is for the second detection unit 104 to detect other vehicles. .

  FIG. 7 is a diagram illustrating an example of a characteristic curve indicating the relationship between the total verification score TSC and the sensitivity setting value CC. As shown by the characteristic curve 700 in FIG. 7, when the total verification score TSC is low, it is considered that the probability that the other vehicle candidate is actually another vehicle is low, so the sensitivity setting in the second detection unit 104 is performed. The value CC is set low to prevent erroneous detection of the second detection unit 104. On the other hand, when the value of the total verification score TSC is high, it is considered that the probability that the other vehicle candidate is actually another vehicle is high. Therefore, the sensitivity setting value CC in the second detection unit 104 is A high setting is made to prevent detection failure of the second detection unit 104. In the example of the characteristic curve 700 in FIG. 7, the sensitivity setting value CC is set to CCm when the total verification score TSC is a predetermined value TSCn. The change of the sensitivity setting value by the setting changing unit 105 is preferably executed dynamically while the image processing apparatus 10 is detecting another vehicle.

  FIG. 8 is a diagram showing a processing flow of the image processing apparatus 10 according to the first embodiment of the present invention. In step 800, it is determined whether or not the ignition of the host vehicle is turned on. If the ignition is not turned on, the standby state is maintained until the ignition is turned on. If the ignition is on, the process proceeds to step 801.

  In step 801, it is determined whether the initialization of the image processing apparatus 10 has already been executed. If it has been initialized, the process proceeds to step 804; otherwise, the process proceeds to step 802. In step 802, a predetermined initialization process is executed. In the next step 803, the far region R1 and the near region R2 illustrated in FIG. 4 are set on the captured image of the camera 20.

  Subsequently, in step 804, the captured image acquired by the camera 20 is input to the first detection unit 101. In step 805, the first detection unit 101 is within the far region R1 from the captured image input in step 804. Other vehicle candidates are detected and extracted. In step 806, it is determined whether or not another vehicle candidate is extracted in step 805. If another vehicle candidate is extracted, the process proceeds to step 807. If not extracted, the process returns to step 804 to input the next photographed image.

  In step 807, the verification unit 102 performs motion vector verification processing, and calculates a verification score SC1 according to the processing result. In step 808, the position of the other vehicle candidate extracted in step 805 is calculated and tracked using the motion vector calculated in step 807. In step 809, the verification unit 102 performs spatial frequency verification processing, and calculates a verification score SC2 according to the processing result. In step 810, the value of the verification score SC1 calculated in step 807 and the value of the verification score SC2 calculated in step 809 are summed to calculate a total verification score TSC.

  In step 811, based on the position of the other vehicle candidate obtained in step 808 and the total verification score TSC obtained in step 810 by the determining unit 103, the other vehicle candidate extracted in step 805 is the latest captured image. Then, it is determined whether or not it exists in the vicinity region R2. Here, the value of the total verification score TSC is compared with a preset threshold value TH, the total verification score TSC is larger than the threshold value TH, and the position of the other vehicle candidate is within the vicinity region R2. In this case, it is determined that another vehicle candidate exists in the vicinity region R2, and the process proceeds to Step 812. On the other hand, when one or both of these conditions are not satisfied, it is determined that the other vehicle candidate does not exist in the vicinity region R2, and the process returns to step 804 to input the next photographed image.

  In step 812, the second detection unit 104 identifies whether the other vehicle candidate extracted in step 805 is an actual other vehicle based on the photographed image input in step 804, thereby identifying the other vehicle. Detect. In step 813, it is determined whether or not another vehicle is detected in step 812. If another vehicle is detected, the process proceeds to step 814. If not extracted, the process returns to step 804 to input the next photographed image.

  If the determination in step 813 is affirmative and the presence of the other vehicle is confirmed, in step 814, a predetermined warning process is performed to give a warning to the user. Here, for example, the distance between the host vehicle and the other vehicle is calculated, and based on the distance, the user who is the driver of the host vehicle is notified that the other vehicle is approaching the host vehicle. In step 815, it is determined whether or not to end the operation of the image processing apparatus 10 based on a predetermined operation end condition. As a result, when it is determined that the operation is to be continued, the processing is returned to step 804, and when it is determined that the operation is to be ended, the processing flow shown in FIG. 8 is ended.

  According to the 1st Embodiment of this invention demonstrated above, there exist the following effects.

(1) The image processing apparatus 10 includes a first detection unit 101 and a second detection unit 104. The first detection unit 101 exists in a distant region R1 corresponding to a point separated from the camera 20 by a first distance from the first captured image obtained by capturing with the camera at the first time. A candidate for an object, that is, another vehicle is extracted (step 805). The second detection unit 104 uses a second captured image obtained by photographing with the camera 20 at a second time after the first time, and is closer to the camera 20 than the first distance. An object existing in the vicinity region R2 corresponding to a point separated by a distance, that is, another vehicle is detected (step 812). At this time, the second detection unit 104 identifies whether or not the other vehicle candidate extracted by the first detection unit 101 is another vehicle based on the second captured image. Detect other vehicles. Since it did in this way, the other vehicle which approaches the vicinity from the distant place of the own vehicle can be detected correctly in the captured image.

(2) The image processing apparatus 10 further includes a verification unit 102 and a determination unit 103. The verification unit 102 verifies the possibility that the candidate for the other vehicle, which is the object, is another vehicle, based on a plurality of captured images obtained by shooting with the camera 20 at different times (steps 807, 809, 810). The determination unit 103 determines whether or not the second detection unit 104 detects another vehicle based on the verification result by the verification unit 102 (step 811). Since it did in this way, since the detection of the other vehicle by the 2nd detection part 104 can be performed at an appropriate timing, an exact detection result can be obtained.

(3) In steps 807 and 809, the verification unit 102 calculates verification scores SC1 and SC2 related to the probability that the other vehicle candidate is the other vehicle, respectively. In step 811, the determination unit 103 determines whether or not the second detection unit 104 detects another vehicle based on the total verification score TSC obtained by adding the verification scores SC <b> 1 and SC <b> 2 calculated by the verification unit 102. . Since it did in this way, in the determination part 103, the determination which reflected the verification result by the verification part 102 correctly can be performed.

(4) In Steps 807 and 809, the verification unit 102 is based on the motion vector indicating the motion of the other vehicle candidate in the plurality of captured images and the spatial frequency of the image area including the other vehicle candidate in the plurality of captured images. The verification scores SC1 and SC2 are calculated respectively. Since it did in this way, verification score SC1, SC2 which reflected the probability that the candidate of another vehicle is an other vehicle correctly can be obtained.

(5) The image processing apparatus 10 further includes a setting change unit 105 that dynamically changes the sensitivity setting value when the second detection unit 104 detects another vehicle based on the total verification score TSC. Since it did in this way, the misdetection and detection omission of the 2nd detection part 104 can be prevented appropriately according to a situation.

-Second Embodiment-
FIG. 9 is a diagram showing an example of an external environment recognition apparatus according to the second embodiment of the present invention. As shown in FIG. 9, the external environment recognition device 900 of this embodiment includes the image processing device 10 described in the first embodiment, a periphery recognition unit 901, a signal processing unit 902, and a driver notification unit 903. The external environment recognition device 900 is connected to the camera 20 mounted on the host vehicle, similarly to the image processing device 10, and also includes a control unit 911, a memory 912, a host vehicle control unit 913, an LED 914, and a speaker 915 mounted on the host vehicle. The display 916 and the car navigation device 917 are also connected. Note that the image processing apparatus 10 and other devices are connected to each other via a signal bus in the host vehicle.

  The camera 20 acquires a captured image around the host vehicle and outputs the captured image to the image processing device 10 in the external environment recognition device 900. The memory 912 temporarily holds a captured image acquired by the camera 20. The control unit 911 controls input / output of captured images between the camera 20 and the external environment recognition device 900 and input / output of vehicle control signals between the external environment recognition device 900 and the host vehicle control unit 913.

  As described in the first embodiment, the image processing apparatus 10 detects other vehicles around the host vehicle, and outputs a vehicle approach signal based on the detection result to the periphery recognition unit 901. Further, when it is difficult to detect other vehicles, a detection FAIL signal is output to the periphery recognition unit 901.

  When a vehicle approach signal is output from the image processing apparatus 10, the periphery recognition unit 901 executes a periphery recognition process for recognizing the surrounding environment of the host vehicle based on the signal. For example, the vicinity of the host vehicle and the surrounding space of the host vehicle are analyzed using the captured image of the camera 20 to recognize the presence or absence of other vehicles and pedestrians including motorcycles and bicycles, and hinders the driving and parking of the host vehicle. Recognize the presence or absence of obstacles. Further, when another vehicle or a pedestrian is approaching the own vehicle suddenly, this is detected and a collision with the own vehicle is predicted, or a collision between the own vehicle and an obstacle is predicted. In addition, lane departure warning processing that issues a warning when the vehicle departs while driving, and blind spot warning processing that issues a warning when a person or other vehicle enters the blind spot of the driver of the vehicle, etc. It may be included in the processing. The periphery recognition unit 901 outputs detection results and alarm information based on the execution result of the periphery recognition processing to the signal processing unit 902, and outputs notification information for the driver of the host vehicle to the driver notification unit 903 as necessary.

  The signal processing unit 902 generates a vehicle control signal for controlling the operation of the host vehicle based on the detection result and the alarm information output from the periphery recognition unit 901 and transmits the vehicle control signal to the host vehicle control unit 913. The own vehicle control unit 913 controls the operation of the own vehicle based on the vehicle control signal received from the signal processing unit 902 to stop the own vehicle in order to avoid a collision with another vehicle or a pedestrian, In order to avoid a collision with an obstacle, the traveling direction of the host vehicle is changed.

  Based on the notification information output from the periphery recognition unit 901, the driver notification unit 903 generates an alarm signal for warning the driver of the vehicle, and any of the LED 914, the speaker 915, the display 916, and the car navigation device 917 Send to. When the LED 914, the speaker 915, the display 916, and the car navigation device 917 receive the warning signal received from the driver notification unit 903, the device performs a predetermined display and audio output based on the warning signal, thereby enabling the driver of the host vehicle. On the other hand, the presence of other vehicles approaching the host vehicle, pedestrians, obstacles, etc. is warned.

  Note that when the detection FAIL signal is output from the image processing apparatus 10, it is determined that it is difficult for the image processing apparatus 10 to detect another vehicle, so the periphery recognition unit 901 performs the operation of the image processing apparatus 10. It is preferable to stop temporarily or continuously. The periphery recognition unit 901 can start or stop the operation of the image processing apparatus 10 by outputting an ON / OFF control signal to the image processing apparatus 10. Further, at this time, the notification information is output from the periphery recognition unit 901 to the driver notification unit 903, and based on this, the driver notification unit 903 generates an alarm signal and any one of the LED 914, the speaker 915, the display 916, and the car navigation device 917. To the driver of the own vehicle that the operation of the image processing apparatus 10 is stopped.

  According to the second embodiment of the present invention described above, the external environment recognition apparatus 900 includes the image processing apparatus 10. Further, the periphery recognition unit 901, the signal processing unit 902, and the driver notification unit 903 are used to issue a warning to the driver of the host vehicle based on the detection result of the other vehicle by the second detection unit 104 in the image processing apparatus 10. At least one of an alarm signal and a vehicle control signal for controlling the operation of the host vehicle is output. Since it did in this way, the surrounding environment of the own vehicle can be recognized correctly.

  In each of the embodiments described above, the target object detected from the captured image is the other vehicle around the host vehicle. However, the target object is not limited to this, and another object may be the target object. Moreover, although the example which detects a target object using the picked-up image acquired with the camera 20 mounted in the vehicle was demonstrated, the camera which acquires a picked-up image is not restricted to what was mounted in the vehicle. For example, it is possible to detect an object using captured images acquired by cameras for various uses other than in-vehicle use such as a camera used for street monitoring.

  The embodiment and various examples described above are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. The present invention is not limited to the above-described embodiments and modifications, and various modifications can be made without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 20 Camera 101 1st detection part 102 Verification part 103 Determination part 104 2nd detection part 105 Setting change part 900 External field recognition apparatus 901 Perimeter recognition part 902 Signal processing part 903 Driver notification part

Claims (7)

A first detection unit for extracting a candidate for an object existing at a point away from the camera by a first distance from a first photographed image obtained by photographing with a camera at a first time;
A second distance closer to the camera than the first distance is separated from a second photographed image obtained by photographing with the camera at a second time after the first time. A second detector for detecting the object present at the point,
The second detection unit identifies the target by identifying whether the candidate for the target extracted by the first detection unit is the target based on the second captured image. An image processing device to detect. The image processing apparatus according to claim 1.
A verification unit that verifies the possibility that the candidate for the target is the target based on a plurality of captured images captured by the camera at different times; and
An image processing apparatus further comprising: a determination unit that determines whether or not the second detection unit detects the object based on a verification result by the verification unit. The image processing apparatus according to claim 2,
The verification unit calculates a verification score related to the probability that the candidate for the target is the target,
The said determination part is an image processing apparatus which determines whether the said 2nd detection part detects the said target object based on the said verification score calculated by the said verification part. The image processing apparatus according to claim 3.
The verification unit is configured to determine the verification score based on a motion vector indicating a motion of the target candidate in the plurality of captured images and a spatial frequency of an image region including the target candidate in the plurality of captured images. An image processing apparatus for calculating The image processing apparatus according to claim 3 or 4,
An image processing apparatus, further comprising: a setting changing unit that dynamically changes a sensitivity setting value when the second detection unit detects the object based on the verification score. In the image processing apparatus according to any one of claims 1 to 5,
The image processing apparatus is mounted on the host vehicle,
The image processing apparatus, wherein the object is another vehicle existing around the host vehicle. An image processing apparatus according to claim 6,
At least one of an alarm signal for warning the driver of the host vehicle and a vehicle control signal for controlling the operation of the host vehicle based on the detection result of the other vehicle by the second detector. A device that recognizes the outside world.

Images