JP7400599B2 - object detection device - Google Patents

Description

The present invention relates to an object detection device.

Patent Document 1 describes the generation of a shifted image by level shifting the brightness values of a far-infrared image, the generation of an inverted image of the image, and the generation of the brightness value of each pixel of the shifted image by the brightness value of the corresponding pixel of the inverted image. A technique has been disclosed in which the contrast of an image is expanded by subtracting or dividing by , thereby making it easier to distinguish objects such as pedestrians from the background.

Patent Document 2 describes that when a far-infrared image is used as a detection target image, the brightness value of a region of a person with a high temperature increases, and the brightness value of a surrounding region with a low temperature compared to a person decreases. An invention of a person detection device for detecting a person or the like from a far-infrared image is disclosed.

Patent No. 5664551 Patent No. 6627680

However, in a far-infrared image, the contrast between a pedestrian and a background such as a road is reversed between summer and winter. Therefore, in order to ensure the accuracy of identifying pedestrians and backgrounds from far-infrared images throughout the year using the technology disclosed in Patent Document 2, learning data according to the season should be prepared and the device should be trained using the learning data. However, there is a problem in that preparation of the learning data and learning of the device require a great deal of cost and effort.

Furthermore, the technique disclosed in Patent Document 1 is effective if the background is an image with little texture and falls within a certain brightness range, such as the sea or the sky, but When surrounding buildings, forests, etc. coexist as a background, there is a problem in that it is difficult to effectively emphasize the contrast between the background and the object.

The present invention has been made in consideration of the above facts, and an object of the present invention is to obtain an object detection device that can accurately identify objects such as pedestrians from the background in image data.

An object detection device according to a first aspect includes a photographing unit that acquires far-infrared image data of a surrounding environment, and an image processing unit that generates converted image data by performing gradation conversion of the far-infrared image data acquired by the photographing unit. and the far-infrared image data and the converted image data by performing object identification processing on each of the far-infrared image data and the converted image data based on results learned in advance using learning data. Object identification that performs identification processing that outputs an identification result including information on identification of the object type in each of the far-infrared image data and the converted image data, and accuracy information representing the reliability of the identification in each of the far-infrared image data and the converted image data. and an object determining section that ultimately identifies an object in the far-infrared image data based on the identification result by the object identifying section.

In far-infrared images, the brightness of objects such as pedestrians and the background (mainly roads) are reversed between summer and winter, but in the first aspect, the captured image data and the captured image data are converted into gradations. By performing object identification based on the same learning data for each of the converted image data obtained by the method, it is possible to obtain good object identification results for either the original data or the converted image data.

In a second aspect, in the first aspect, the image processing unit generates the converted image data by inverting the brightness of the brightness value of the far-infrared image data acquired by the imaging unit.

This makes it possible to identify objects in accordance with the characteristics of far-infrared image data in which the contrast between the object and the background is reversed between summer and winter.

In a third aspect, based on the first aspect, the image processing unit may process the far-infrared image data based on a predetermined correspondence relationship between the gradation of the far-infrared image data and the gradation of the converted image data. The converted image data is generated from the external image data.

Thereby, by changing the characteristic of inverting the brightness and darkness of the gradation of far-infrared image data, it becomes possible to generate inverted image data that is optimal for object identification.

In a fourth aspect, in any one of the first to third aspects, the object identification unit identifies the position of the object in each of the far-infrared image data and the converted image data, and , the object determination unit outputs the identification result including position information of the identified object, and the object determination unit outputs the object position information, the accuracy information of each of the far-infrared image data and the converted image data, which the identification result includes. Final object identification is performed based on the

This makes it easy to locate the object in the far-infrared image data and identify the object.

In a fifth aspect, in any one of the first to third aspects, the object identification unit identifies the position of the object in each of the far-infrared image data and the converted image data, and , generating a mapping image as the identification result in which accuracy information in each of the far-infrared image data and the converted image data is associated with the position of the object in each of the identified far-infrared image data and the converted image data; The object determination section superimposes mapping images generated from each of the far-infrared image data and the converted image data, thereby determining the identification result in the far-infrared image data and the identification result in the converted image data. The final object identification is performed by integrating the two.

By integrating the identification results in the far-infrared image data and the identification results in the converted image data, the accuracy of object identification is improved.

In a sixth aspect, in any one of the first to fifth aspects, the learning data includes the far-infrared image data for learning, the converted image data for learning, and the converted image data for learning. At least one of visible light image data.

In a seventh aspect, in any one of the first to third aspects, the imaging unit further acquires visible light image data of the surrounding environment, and the object identification unit further acquires visible light image data of the surrounding environment. Information on the identification of the object type in each of the image data, the converted image data, and the visible light image data, and the reliability of the identification in each of the far-infrared image data, the converted image data, and the visible light image data. , and the object determining unit finally identifies the object in the far-infrared image data based on the identification result by the object identifying unit. .

This integrates the identification results of the captured image data and converted image data of the far-infrared image with the identification results of the visible light image data, and determines the reliability of the final object identification. Even if the object identification unit is trained using data, it is possible to perform object identification with high accuracy.

In an eighth aspect, in the seventh aspect, the learning data is the visible light image data for learning.

This has the effect that objects such as pedestrians can be accurately identified from the background in image data.

FIG. 1 is a block diagram showing an example of an object detection device according to a first embodiment of the present invention. It is a flow chart showing an example of processing of the object detection device concerning a 1st embodiment of the present invention. (A) shows an example of object identification in captured image data by an object classifier, (B) shows an example of object identification in inverted image data by an object classifier, and (C) shows an example of object identification in captured image data. (D) shows an example of a mapping image of captured image data created by extracting reliability, and (D) shows an example of a mapping image of reversed image data created by extracting identification reliability in reversed image data. , (E) is a schematic diagram illustrating an example of integrating the identification results of photographed image data and inverted image data. FIG. 7 is an explanatory diagram of inversion of brightness and darkness of gradation of captured image data in a second embodiment of the present invention. It is a block diagram showing an example of an object detection device concerning a 3rd embodiment of the present invention. It is a functional block diagram showing an example of an object detection device concerning a 4th embodiment of the present invention.

[First embodiment]
Hereinafter, an example of the first embodiment of the present invention will be described in detail with reference to the drawings.

As shown in FIG. 1, an object detection device 100 according to this embodiment includes a far-infrared camera 10 and a computer 20. The far-infrared camera 10 is mounted on a vehicle, has a detection range around the vehicle, acquires far-infrared image data within the detection range according to set measurement parameters and processing parameters, and uses the acquired far-infrared image data to It is configured so that it can be outputted to the computer 20 as a video signal, and the settings of measurement parameters and processing parameters can be changed from the outside. The far-infrared camera 10 is not only a general far-infrared camera equipped with an optical system such as a lens and an image sensor whose photosensitive characteristics are specialized for far-infrared rays, but also a single photodiode or the like that is compatible with far-infrared rays. It may also be an arrayed device or the like. Therefore, the far-infrared camera 10 is capable of acquiring one-dimensional and two-dimensional far-infrared image data that can detect and classify objects, and includes measurement parameters and parameters that affect the acquisition of the far-infrared image data. Includes all equipment that can change processing parameters.

As an example, the far-infrared camera 10 includes a sensor device including an optical system such as a lens and an image sensor that outputs a signal according to the intensity of received light, and a so-called image file that converts the signal output by the sensor device into an image file. It includes a pre-processor which is an engine. Measurement parameters of the far-infrared camera 10 include shutter speed (exposure time), gain (sensitivity), angle of view, aperture (F number), focus, frame rate (measurement cycle), etc. for the sensor device.

Processing parameters for the preprocessor include white balance, contrast, spatial resolution (number of pixels), and the like.

When the sensor device and the preprocessor are integrated, it is equivalent to performing processing such as white balance using the sensor device. Furthermore, since it is sometimes possible to change the frame rate by frame dropping, which is thinning out of frames in the sensor device, it becomes difficult to distinguish between measurement parameters and processing parameters. Therefore, measurement parameters and processing parameters may be treated without any particular distinction.

The computer 20 includes a CPU, a memory, a nonvolatile storage unit storing programs for executing various processes, a network interface, and the like. Functionally, the computer 20 receives far-infrared image data (hereinafter referred to as "captured image data") acquired by the far-infrared camera 10 and generates inverted image data of the far-infrared image data. An image preprocessing unit 22 that performs image processing, an object classifier 24 that performs object identification that estimates the type of object in the photographed image data and the inverted image data, and an object classification unit 24 that performs object identification in each of the photographed image data and the inverted image data. It is configured to include an image post-processing section 26 that outputs the integrated identification results.

The image preprocessing unit 22 includes an image memory that is a storage device, and stores photographed image data as it is in the image memory, and also creates inverted image data by inverting the brightness and darkness of the gradation of the photographed image data and stores it in the image memory. Remember.

The object discriminator 24 performs object discrimination for each of the photographed image data and the inverted image data stored in the image memory of the image preprocessing unit 22 to identify objects such as pedestrians from the background such as the road surface.

The object classifier 24 is trained in advance using learning data, and performs signal processing based on DNN (Deep Neural Network), for example. Further, as learning data for learning the object classifier 24, photographed image data and inverted image data to which the positions of objects such as pedestrians are assigned in advance are used. The learning data for learning the object classifier 24 may be visible light image data.

The image post-processing unit 26 integrates the object identification results for each of the captured image data and the inverted image data by the object identifier 24 and outputs the result as an object identification result. As will be described later, the integration of the identification results is performed by, for example, superimposing the identification results of the captured image data and the identification results of the inverted image data, and adding the reliability of identification linked to each identification result. Based on the numerical values obtained, the reliability of object identification is determined and the final object identification is performed.

FIG. 2 is a flowchart showing an example of processing of the object detection device 100 according to the present embodiment. It is assumed that the object discriminator 24 has been trained in advance using captured image data and inverted image data to which the positions of objects such as pedestrians are previously assigned as learning data.

In step 200, far-infrared image data around the vehicle is acquired by, for example, photographing the traveling path in front of the vehicle using the far-infrared camera 10. The acquired far-infrared image data is stored in the image memory of the image preprocessing section 22 as photographed image data.

In step 204, the image preprocessing unit 22 creates inverted image data by inverting the brightness and darkness of the gradations of the captured image data, and stores it in the image memory.

In step 206, each of the photographed image data and the inverted image data stored in the image memory of the image preprocessing unit 22 is input to the object classifier 24, and the object classifier 24 outputs a classification result for each image data. .

In step 208, the image post-processing unit 26 integrates the identification results for each of the captured image data and the inverted image data to obtain a final identification result. Then, in step 210, the final identification result obtained in step 208 is output, and the process ends.

FIG. 3 is an explanatory diagram showing an example of object identification by the object identifier 24 and an example of integration of identification results by the image post-processing unit 26. 3(A) shows an example of object identification in captured image data by the object identifier 24, and FIG. 3(B) shows an example of object identification in inverted image data by the object identifier 24. As shown in FIG. 3A, in the object identification in the captured image data, a car is identified with an identification reliability of 70%, and a person is identified with an identification reliability of 40%. Further, as shown in FIG. 3B, in the object identification in the inverted image data, a car is identified with an identification reliability of 50%, and a person is identified with an identification reliability of 80%.

FIG. 3(C) shows an example of a mapping image of captured image data created by extracting the identification reliability from the captured image data, and FIG. 3(D) shows an example of a mapping image created by extracting the identification reliability from the inverted image data. FIG. 4 is a schematic diagram illustrating an example of a mapping image of inverted image data. The mapping images shown in FIGS. 3(C) and 3(D) are created, for example, by the object classifier 24, and the areas corresponding to the positions of the objects identified in each of the photographed image and the inverted image are marked with identification reliability. are mapped.

As shown in FIG. 3(C), in the object identification in the captured image data, cars can be identified with high reliability, but the identification reliability of humans is low. Further, as shown in FIG. 3(D), in the object identification in the inverted image data, although humans can be identified with high reliability, the identification reliability of cars is low. In this embodiment, as described above, objects such as cars and people are accurately identified from the background by integrating the identification results for each of the photographed image data and the inverted image data to obtain the final identification result.

FIG. 3E is a schematic diagram illustrating an example of integrating the identification results of photographed image data and inverted image data. In this embodiment, as an example, by superimposing the mapping images generated from each of the captured image data and the inverted image data, the identification result of the captured image data and the identification result of the inverted image data are superimposed, and each The reliability of object identification is determined based on the numerical value obtained by adding the reliability of identification linked to the identification result of . In FIG. 3E, the overlapped portions are shown with hatching. In FIG. 3E, the identification reliability of the hatched area is the sum of the identification reliability of each of the photographed image data and the inverted image data. The total recognition reliability is 120% for cars and 120% for people, indicating that the reliability of object identification is quite high, and objects such as pedestrians can be accurately identified from the background of far-infrared images. can do.

As explained above, in this embodiment, in addition to the brightness image data of the normal far-infrared image, by adding inverted image data in which the brightness and darkness of the brightness value is reversed, it is possible to Even when detection is not possible, it becomes possible to detect an object.

As mentioned above, a characteristic of far-infrared images is that the contrast between pedestrians and the background (mainly the road) is reversed between summer and winter. Therefore, when a device is trained using far-infrared image data, it is required to prepare learning data that corresponds to such a wide range of temperature environments. However, preparing learning data compatible with a wide range of temperature environments requires a huge amount of time, cost, and effort, and is considered difficult in practice.

In this embodiment, inverted image data of photographed image data is generated, and both the captured image data and the inverted image data are input to the object discriminator 24. The object discriminator 24 has been trained using learning data regardless of the season, and performs object identification based on the same learning data for each of the photographed image data and the inverted image data. As mentioned above, in far-infrared images, the brightness of the pedestrian and the background are reversed between summer and winter, but object identification is performed based on the same learning data for each of the photographed image data and the reversed image data. If so, there is a possibility that good object identification results can be expected in either the photographed image data or the inverted image data. Therefore, in this embodiment, it is not necessary to re-train the object classifier 24 using different learning data depending on the season.

[Second embodiment]
Next, a second embodiment of the present invention will be described. FIG. 4 is an explanatory diagram of inversion of the brightness and darkness of the gradation of photographed image data in this embodiment. This embodiment differs from the first embodiment in the generation of inverted image data from photographed image data in the image preprocessing section 22, but is otherwise similar to the first embodiment, so the same configurations are denoted by the same reference numerals. Detailed explanation will be omitted.

In the first embodiment, the image preprocessing unit 22 simply inverts the brightness and darkness of the gradations of the captured image data to generate inverted image data. The straight line shown with slope = -1 in Fig. 4 corresponds to the gradation of the captured image data and the gradation of the inverted image data when the inverted image data is generated by simply reversing the brightness and darkness of the gradation of the captured image data. It shows the correspondence. When the slope is -1, for example, the brightest pixel in the captured image data becomes the darkest pixel in the inverted image data, and the darkest pixel in the captured image data becomes the brightest pixel in the inverted image data.

In the present embodiment, inverted image data is generated from photographed image data with a straight line-like characteristic represented by slope≦-1 or slope≧-1. When generating inverted image data from photographed image data with characteristics such as a straight line with slope ≦-1 or slope ≧-1, the contrast of the pixels of the reversed image data corresponding to the dark pixels of the photographed image data is This can be improved over the case where inverted image data is generated by simply inverting the brightness and darkness of the gradation of the data. For example, when generating inverted image data from photographed image data with the characteristic of slope ≦-1 as shown in FIG. The brightness value of each pixel having the maximum brightness value is converted to the pixel having the smallest brightness value, that is, the darkest pixel, in the inverted image data. As a result, the generated inverted image data shows an image in which dark areas are more prominent than the inverted image data generated with the characteristic of slope=-1. Furthermore, when inverted image data is generated from photographed image data with the characteristic of slope ≧ -1, the generated inverted image data is an image in which bright areas are more pronounced than the inverted image data generated with the characteristic of slope = -1. shows.

The characteristic of reversing the brightness and darkness of the gradation of the captured image data may be an upwardly convex curve or a downwardly convex curve instead of a straight line as shown in FIG. In this embodiment, photographed image data is converted to inverted image data based on a predetermined correspondence relationship between the gradation of the captured image data and the gradation of the inverted image data, regardless of whether it is linear or curved. You can.

As described above, the identification results between the photographed image and the inverted image may differ depending on the object, such as a car or a person. As shown in FIG. 4, by changing the characteristic of inverting the brightness and darkness of the gradations of the captured image data, it is possible to improve the accuracy of object identification.

In order to improve the accuracy of object identification, methods such as improving the learning data of the object detection device 100 can be considered, but preparing new learning data and learning the object detection device 100 using the image data is complicated and costly. It takes. However, it is relatively easy to change the characteristic of reversing the brightness and darkness of the gradations of photographed image data as shown in FIG. Therefore, according to the present embodiment, the accuracy of object identification can be improved more easily than when the device is retrained using new learning data.

[Third embodiment]
Next, a third embodiment of the present invention will be described. FIG. 5 is a block diagram showing an example of the object detection device 200 according to this embodiment. This embodiment differs from the first embodiment in that it includes a visible light camera 12 that acquires visible light images and a computer 120 that includes an object classifier 124 trained using visible light images, but in other respects. Since this embodiment is similar to the first embodiment, the same components are given the same reference numerals and detailed explanations will be omitted.

The image preprocessing unit 122 generates inverted image data from the captured image data acquired by the far-infrared camera 10, and the generated inverted image data and captured image data are input to the object discriminator 124 in the first This is similar to the embodiment. In this embodiment, the object discriminator 124 is input with not only photographed image data acquired by a far-infrared camera and inverted image data generated from the photographed image data, but also visible light image data acquired by the visible light camera 12. be done.

The object classifier 124 identifies each of the captured image data of the far-infrared camera 10 input from the image preprocessing unit 122, the inverted image data of the far-infrared image, and the visible light image data input from the visible light camera 12. Perform object identification.

The object classifier 124 is trained in advance using learning data, and performs signal processing based on DNN, for example. Further, as learning data for learning the object classifier 124, visible light image data to which the positions of objects such as pedestrians are assigned in advance is used. Note that, as the learning data, photographed image data and inverted image data may also be used.

Then, the image post-processing unit 126 superimposes the identification results based on the captured image data, the identification results based on the inverted image data, and the identification results based on the visible light image data, and calculates the reliability of identification associated with each identification result. The reliability of object identification is determined based on the numerical value obtained by adding the degrees.

Generally, in a recognition system for automatic driving of a vehicle, an object classifier is trained using visible light image data. If an object classifier trained using visible light image data can be used to recognize far-infrared image data, there will be no need to prepare a new object recognizer specialized for far-infrared image data, and the cost will be reduced. It is advantageous. An object recognizer trained using visible light image data cannot predict which of far-infrared image data and inverted image data is more suitable for recognition, but it is difficult to predict which is more suitable for recognition. By integrating each classification result with the classification result in visible light image data and determining the reliability of the final object identification, even an object classifier trained with visible light image data can identify objects. It becomes possible to do this with high precision. Furthermore, according to the present embodiment, object identification using far-infrared image data allows objects such as pedestrians at night or objects such as pedestrians that exist in the distance to be detected by object identification using far-infrared image data. can also be identified more clearly.

[Fourth embodiment]
Next, a fourth embodiment of the present invention will be described. FIG. 6 is a functional block diagram showing an example of the object detection device according to this embodiment. In this embodiment, in addition to photographed image data and inverted image data, image data of a rectangular area of a predetermined number of pixels cut out from the center of the image of the inverted image data is inputted to the object discriminator 224 to perform object identification. .

There is a high possibility that an object that is far away from the far-infrared camera 10 or an object that is small exists in the image center of the captured image data and the inverted image data. In this embodiment, image data cut out from a rectangular area near the center of the image of inverted image data as shown in FIG. Small objects such as pedestrians can be identified with high accuracy.

In this embodiment, each of the identification results (1) based on photographed image data, the identification results (2) based on inverted image data, and the identification results (3) based on image data of a rectangular area of the inverted image data , the classification of the object, the position of the object, and the likelihood of identification of the object are calculated respectively, and the identification results (1), (2), and (3) are integrated to perform final object identification.

Each of the image data of the rectangular area of the photographed image data, the inverted image data, and the inverted image data includes an overlapping area. The accuracy of object identification can be ensured by comprehensively determining the identification results based on each image in such overlapping areas.

In this embodiment, objects are detected from image data cut out from a rectangular area near the center of the image of inverted image data as image data of an area corresponding to the traveling direction of the own vehicle. etc. can be detected with higher accuracy.

As described above, in the first embodiment, the inversion image data is generated by inverting the brightness of the brightness value of the captured image data, and in the second embodiment, the gradation of the captured image data determined in advance and the inverted image are generated. Inverted image data was generated from the photographed image data based on the correspondence with the gradation of the data. However, the present invention is not limited thereto. For example, preprocessing such as contrast enhancement and contour extraction may be performed on each of the captured image data and the inverted image data, or in addition to such preprocessing, other known image processing methods may be combined. Further, the resolution of each of the photographed image data and the inverted image data may be expanded by super-resolution processing or the like.

In the above, a mode has been described in which a neural network model as an example of a model is learned by deep learning, but the present invention is not limited to this. For example, a model different from the neural network model may be trained using a learning method different from deep learning.

10 Far-infrared camera 12 Visible light camera 20 Computer 22 Image pre-processing unit 24 Object classifier 26 Image post-processing unit 100 Object detection device 120 Computer 122 Image pre-processing unit 124 Object classifier 126 Image post-processing unit 200 Object detection device

Claims (8)

an imaging unit that acquires far-infrared image data of the surrounding environment;
an image processing unit that generates converted image data by gradation-converting the far-infrared image data acquired by the imaging unit;
By performing object identification processing on each of the far-infrared image data and the converted image data based on results learned in advance using learning data, an object identification unit that performs identification processing to output an identification result including information on identification of the object type and accuracy information representing reliability of the identification in each of the far-infrared image data and the converted image data;
an object determining unit that finally identifies an object in the far-infrared image data based on the identification result by the object identifying unit;
An object detection device including: The object detection device according to claim 1, wherein the image processing unit generates the converted image data by inverting the brightness of the brightness value of the far-infrared image data acquired by the imaging unit. The image processing unit generates the converted image data from the far-infrared image data based on a predetermined correspondence relationship between the gradation of the far-infrared image data and the gradation of the converted image data. Object detection device according to item 1. The object identification unit identifies the position of the object in each of the far-infrared image data and the converted image data, and outputs the identification result including position information of the identified object; 4. The method according to claim 1, wherein final object identification is performed based on object position information included in the identification result and accuracy information in each of the far-infrared image data and the converted image data. Object detection device. The object identification unit identifies the position of the object in each of the far-infrared image data and the converted image data, and also applies accuracy information in each of the far-infrared image data and the converted image data to the identified far-infrared image data. generating a mapping image associated with the position of the object in each of the infrared image data and the converted image data as the identification result;
The object determination unit is configured to superimpose mapping images generated from each of the far-infrared image data and the converted image data, thereby distinguishing between the identification result in the far-infrared image data and the identification result in the converted image data. The object detection device according to any one of claims 1 to 3, wherein the object detection device performs final object identification by integrating. 6. The learning data is at least one of the far-infrared image data for learning, the converted image data for learning, and the visible light image data for learning. Object detection device. The imaging unit further acquires visible light image data of the surrounding environment,
The object identification unit includes information on identifying the type of object in each of the far-infrared image data, the converted image data, and the visible light image data, and information on identifying the type of object in each of the far-infrared image data, the converted image data, and the visible light image data. Executing an identification process that outputs an identification result including accuracy information representing the reliability of the identification in each of the image data,
The object detection device according to any one of claims 1 to 3, wherein the object determination section finally identifies the object in the far-infrared image data based on the identification result by the object identification section. The object detection device according to claim 7, wherein the learning data is the visible light image data for learning.