JP2020087233A - Recognition processing apparatus, recognition processing method, and program - Google Patents

Abstract

To provide a recognition processing apparatus configured to estimate the size of animals accurately.SOLUTION: A recognition processing apparatus 100 includes a video acquisition unit 112, a recognition processing unit 120, and an estimation unit 130. The video acquisition unit 112 acquires a video formed by imaging the surroundings. The recognition processing unit 120 recognizes an animal and reflection of two eyes of the animal. The estimation unit 130 estimates the size of the animal on the basis of a distance between the two eyes obtained from the recognized reflection of two eyes.SELECTED DRAWING: Figure 2

Description

The present invention relates to a recognition processing device, a recognition processing method, and a program.

Patent Document 1 discloses an image recognition device that clearly distinguishes between a pedestrian and an animal other than a pedestrian. The image recognition device according to Patent Document 1 determines whether or not an object that is a candidate for determination detected from an image is a predetermined animal other than a pedestrian, and is a pedestrian or an animal other than a pedestrian. Clearly distinguish. Further, the image recognition device according to Patent Document 1 makes a determination regarding a predetermined part that constitutes a predetermined animal other than a pedestrian, with respect to an object that is a candidate for determination. Then, the image recognition device according to Patent Document 1 determines that the image recognition device is a predetermined animal other than the pedestrian when the determination value of the predetermined region obtained as a result exceeds a predetermined threshold.

JP, 2008-021035, A

There are various sizes of animals depending on the types of animals. For example, a deer and a cat are completely different in size. Here, in the technique according to Patent Document 1, the distance to the object that is a candidate for discrimination detected from the input image is measured from the position on the image, and the object that is a candidate for discrimination is determined based on the measured distance. Measure the size. However, the method of measuring the distance to the object in the technique according to Patent Document 1 may not be able to measure the distance accurately. Therefore, the technique according to Patent Document 1 may not be able to accurately estimate the size of the animal.

In view of the above problems, the present invention has an object to provide a recognition processing device, a recognition processing method, and a program capable of accurately estimating the size of an animal.

A recognition processing device according to the present invention includes a video acquisition unit that acquires a video image captured by a camera that captures the surroundings, a recognition processing unit that recognizes an animal and the reflection of two eyes of the animal from the video image, And an estimation unit that estimates the size of the animal based on the distance between the two eyes obtained from the reflected two eyes.

Further, the recognition processing method according to the present invention obtains a video image of the surroundings, recognizes an animal and the reflection of the two eyes of the animal from the video, and obtains from the recognized reflection of the two eyes. Estimate the size of the animal based on the distance between the two eyes that are taken.

Further, the program according to the present invention includes a step of acquiring a video image of the surroundings, a step of recognizing an animal and a reflection of two eyes of the animal from the video image, and a recognized reflection of the two eyes of the animal. And estimating the size of the animal based on the distance between the two eyes obtained from the computer.

According to the present invention, it is possible to provide a recognition processing device, a recognition processing method, and a program capable of accurately estimating the size of an animal.

FIG. 1 is a diagram showing a vehicle according to a first exemplary embodiment. FIG. 1 is a diagram showing a configuration of a recognition processing device and a recognition system having the recognition processing device according to the first exemplary embodiment. 3 is a flowchart showing a recognition processing method according to the first exemplary embodiment. It is a figure for demonstrating the estimation process of the size of an animal.

(Embodiment 1)
Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the substantially same components are assigned the same reference numerals.

FIG. 1 is a diagram showing a vehicle 1 including a recognition system 50 according to the first embodiment. In FIG. 1, an example in which a vehicle 1 has one infrared camera 2, one visible light camera 4, and a recognition processing device 100 is shown. Two or more infrared cameras 2 and visible light cameras 4 may be provided, respectively. A plurality of infrared cameras 2 and visible light cameras 4 may photograph the front and rear and right and left of the vehicle 1. Further, in the following description, an example in which the recognition processing device 100 is mounted on the vehicle 1 is shown, but the recognition processing device 100 according to the present embodiment does not have to be mounted on the vehicle 1.

The infrared camera 2 captures an image of the surroundings of the vehicle 1 with infrared rays, particularly far infrared rays. Then, the infrared camera 2 generates infrared video data which is video data of a thermal image by detecting the thermal energy of the object. That is, the infrared video data is video data showing the infrared video taken by the infrared camera 2.

The visible light camera 4 is a normal camera that receives visible light and forms an image. The visible light camera 4 captures an image around the vehicle 1 by receiving the light emitted by the object and the reflected light from the object. Then, the visible light camera 4 generates visible light image data which is image data of visible light. That is, the visible light image data is image data indicating a visible light image captured by the visible light camera 4.

The recognition processing device 100 can be installed at any position of the vehicle 1. The recognition processing device 100 can be connected to a CAN (Control Area Network) included in the vehicle 1. The recognition processing device 100 performs person recognition for recognizing a person such as a pedestrian or the like in a video image captured by the infrared camera 2 by using a recognition dictionary. Details will be described later. Note that, hereinafter, the term “video” also means “video data representing a video” as a processing target in information processing. Similarly, hereinafter, the term "image" also means "image data representing an image" as a processing target in information processing. The term "voice" also means "voice data indicating voice" as a processing target in information processing.

FIG. 2 is a diagram showing a configuration of the recognition processing device 100 and the recognition system 50 including the recognition processing device 100 according to the first embodiment. The recognition system 50 according to the first embodiment includes an infrared camera 2, a visible light camera 4, a recognition dictionary storage unit 20, a display unit 30, and a recognition processing device 100. The recognition processing device 100 may include at least one of the infrared camera 2, the visible light camera 4, the recognition dictionary storage unit 20, and the display unit 30.

The recognition dictionary storage unit 20 stores dictionary data regarding an animal recognition dictionary, which is a recognition dictionary used by the recognition processing device 100 to recognize animals other than humans. Note that animals other than humans will be simply referred to as “animals” hereinafter. The recognition dictionary is, for example, a learned model generated by machine learning. The animal recognition dictionary can be generated by machine learning using a large number of images (infrared images) of various animals. The recognition dictionary is used by the recognition processing device 100 when detecting an animal in a video. The term "recognition dictionary" also means "dictionary data indicating a recognition dictionary" as a processing target in information processing.

The display unit 30 is, for example, a display or the like. The display unit 30 displays an image under the control of the recognition processing device 100. The display unit 30 may include a speaker. In this case, the display unit 30 may output sound from the speaker. Further, when the recognition processing device 100 is mounted on the vehicle 1, the display unit 30 is provided inside the vehicle 1 at a position where the driver of the vehicle can visually recognize it while driving. In this case, the display unit 30 may be realized by the interface unit 108 described later.

The recognition processing device 100 includes a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 (IF; Interface) as a main hardware configuration. The control unit 102, the storage unit 104, the communication unit 106, and the interface unit 108 are connected to each other via a data bus or the like. That is, the recognition processing device 100 has a function as a computer.

The control unit 102 is a processor such as a CPU (Central Processing Unit). The control unit 102 has a function as an arithmetic device that performs control processing, arithmetic processing, and the like. The storage unit 104 is a storage device such as a memory or a hard disk. The storage unit 104 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 104 has a function of storing a control program and a calculation program executed by the control unit 102. The storage unit 104 also has a function of temporarily storing processed data and the like.

The communication unit 106 performs a process necessary for communicating with an external device. The communication unit 106 may include a communication port. The interface unit 108 (IF; Interface) is, for example, a user interface (UI). The interface unit 108 has an input device such as a keyboard, a touch panel or a mouse, and an output device such as a display or a speaker.

The recognition processing device 100 also includes a video acquisition unit 112, a recognition processing unit 120, a calculation unit 122, an estimation unit 130, and a display control unit 140. The estimation unit 130 includes a size estimation unit 132 and a distance estimation unit 134. The image acquisition unit 112, the recognition processing unit 120, the calculation unit 122, the estimation unit 130, and the display control unit 140 can be realized by executing a program under the control of the control unit 102 or as a function of the control unit 102. More specifically, the image acquisition unit 112, the recognition processing unit 120, the calculation unit 122, the estimation unit 130, and the display control unit 140 are realized by the control unit 102 executing the programs stored in the storage unit 104. obtain. In addition, the necessary program is recorded in an arbitrary non-volatile recording medium and installed as needed, so that the video acquisition unit 112, the recognition processing unit 120, the calculation unit 122, the estimation unit 130, and the display control unit 140 can be stored. It may be realized.

Moreover, the image acquisition unit 112, the recognition processing unit 120, the calculation unit 122, the estimation unit 130, and the display control unit 140 are not limited to being realized by software according to a program. The image acquisition unit 112, the recognition processing unit 120, the calculation unit 122, the estimation unit 130, and the display control unit 140 may be realized by any combination of hardware, firmware, and software. In addition, the image acquisition unit 112, the recognition processing unit 120, the calculation unit 122, the estimation unit 130, and the display control unit 140 use a user programmable integrated circuit such as an FPGA (field-programmable gate array) or a microcomputer. May be realized. In this case, the integrated circuit may be used to realize a program including the above-described components.

The image acquisition unit 112 acquires an image of the surroundings. Specifically, the image acquisition unit 112 acquires an infrared image from the infrared camera 2. Further, the image acquisition unit 112 acquires a visible light image from the visible light camera 4.

The recognition processing unit 120 recognizes the animal and the reflection of the two eyes of the animal from the image. Specifically, the recognition processing unit 120 recognizes an animal from an infrared image using an animal recognition dictionary. More specifically, the recognition processing unit 120 detects an object that may be an animal from the infrared image and recognizes that the object is an animal. In addition, the recognition processing unit 120 detects, when two lights are detected in a range corresponding to an object recognized as an animal using the animal recognition dictionary in the visible light image, these two lights are the two eyes of the animal. Recognized as a reflection of. Details will be described later.

It should be noted that at night or the like, when the animal is exposed to the headlight of the vehicle 1 or the like, the animal tends to face the light source side of the headlight, that is, the direction of the vehicle 1. In this case, the eyes of the animal appear to shine by reflecting light from the headlights of the vehicle 1 or the like. Therefore, in the visible light image, the reflected light of two eyes of the animal may appear. The reflected light of the eyes of the animal is the light reflected by the bright plate behind the eyes when the light of the headlight of the vehicle 1 enters the eyes of the animal.

Further, the recognition processing unit 120 performs the animal recognition process on the video data acquired by the video acquisition unit 112 for each frame forming the video data or for each frame at a predetermined interval. Here, it is assumed that the angle of view of the infrared camera 2 is substantially the same as the angle of view of the visible light camera 4. The image acquisition unit 112 associates frames of infrared image data and frames of visible light image data, which are captured at the same timing with each other. Then, the image acquisition unit 112 calibrates the positional relationship between the infrared image frame and the visible light image frame in accordance with the displacement between the position of the infrared camera 2 and the position of the visible light camera 4. As a result, the position coordinates of the infrared image and the visible light image captured at the same timing are substantially the same. Therefore, when the infrared image and the visible light image are superposed, the positions of the objects shown in the two images may coincide with each other. The recognition processing unit 120 may perform the above-described calibration process performed by the image acquisition unit 112.

The calculation unit 122 calculates the distance between two eyes of the animal, that is, the distance in the visible light image. The estimation unit 130 estimates the animal. The size estimation unit 132 estimates the size of the animal based on the distance between the two eyes obtained from the reflection of the two eyes. The distance estimation unit 134 estimates the distance to the animal based on the estimated size of the animal or the distance between the two eyes. Details will be described later.

It should be noted that the distance between the eyes of the animal may be approximately the same distance regardless of the type or size of the animal. That is, the individual difference in the eye distance of the animal is much smaller than the individual difference in the body size (body length or body height) of the animal. For example, the eye distance of an animal can be approximately around 15 cm to 20 cm. In the present embodiment, such a property of animals is utilized.

The display control unit 140 performs control for displaying an image on the display unit 30 according to the processing results of the recognition processing unit 120 and the estimation unit 130. For example, the display control unit 140 may cause the display unit 30 to display a warning that the animal exists in front of the vehicle 1 or the like when the recognition processing unit 120 recognizes the animal. The display control unit 140 may also cause the display unit 30 to display the size of the animal and the distance from the vehicle 1 to the animal estimated by the estimation unit 130.

FIG. 3 is a flowchart showing the recognition processing method according to the first embodiment. The image acquisition unit 112 acquires an infrared image from the infrared camera 2 (step S102). Further, the image acquisition unit 112 acquires a visible light image from the visible light camera 4 (step S104). As described above, the recognition processing unit 120 recognizes an animal from the infrared image using the animal recognition dictionary (step S106).

When the animal is recognized (YES in step S108), the recognition processing unit 120 detects the reflection of the second eye of the animal in the visible light image (step S110). Specifically, the recognition processing unit 120 corresponds to an object recognized as an animal by using the animal recognition dictionary in the processing of S106 in the superimposed image in which the frame of the infrared video data and the frame of the visible light video data are superimposed. It is determined whether or not two lights are detected within the range. Then, when the two lights are detected in this range, the recognition processing unit 120 recognizes these two lights as the reflection of the two eyes of the animal. The process of step S108 may be performed by determining whether or not two lights are detected in the range in which the animal is recognized from the infrared image and is determined to be the head of the animal.

When the reflection of the two eyes of the animal is detected (YES in step S112), the calculation unit 122 calculates the distance between the two eyes of the animal in the visible light image or the superimposed image (step S120). Specifically, the calculation unit 122 calculates the number of pixels between the pixels corresponding to the reflection of the two eyes in the visible light image or the superimposed image. Alternatively, the calculation unit 122 may calculate the distance between the pixels corresponding to the reflection of the two eyes in the visible light image or the superimposed image.

The size estimation unit 132 estimates the size of the animal recognized in the process of S106 based on the calculated distance between the two eyes (step S122). Specifically, as mentioned above, the eye distance of an animal can be generally constant independent of the animal. Therefore, the distance between the eyes of this animal is D (cm). Here, D may be a predetermined constant value. This will be described below with reference to FIG.

FIG. 4 is a diagram for explaining the process of estimating the size of an animal. FIG. 4 is a superimposed image in which a visible light image in which the reflection of the eye 92 of the animal is captured is superimposed on an infrared image in which the animal 90 is captured. The distance between the two eyes 92 in this superimposed image is d. Further, the distance corresponding to the lateral width of the body of the animal 90 in the superimposed image is w. In addition, the distance corresponding to the body height of the animal 90 in the superimposed image is h. In this case, the size estimating unit 132 calculates the estimated value W of the actual width of the body of the animal 90 as W=D*w/d. Further, the size estimation unit 132 calculates the estimated value H of the actual body height of the animal 90 as H=D*h/d.

The distance estimation unit 134 estimates the distance from the infrared camera 2 to the animal based on the estimated size of the animal or the calculated distance between the two eyes (step S124). The distance from the infrared camera 2 to the animal may be the distance from the recognition system 50 to the animal, or when the recognition system 50 is mounted on the vehicle 1, it may be the distance from the vehicle 1 to the animal. Specifically, the distance estimation unit 134 calculates the distance from the infrared camera 2 to the animal by the following two methods, for example. However, the method of calculating the distance to the animal is not limited to the following method.

As illustrated in FIG. 4, when an animal is reflected from the upper end to the lower end of the infrared image, the frame height of the infrared image is h, and the actual height corresponding to h is H. The angle of view of the infrared camera 2 is θ, and the distance from the infrared camera 2 to the animal is L. In this case, the following expression (1) is established.
tan (θ/2)=H/2L (1)
Therefore, the distance estimation unit 134 calculates the distance L from the infrared camera 2 to the animal by the following formula (2).
L=H/{2*tan(θ/2)} (2)

Alternatively, assuming that the focal lengths of the infrared camera 2 and the visible light camera 4 are f, the following formula (3) is established using the distance d between the two eyes in the superimposed image.
f:d=L:D (3)
Therefore, the distance estimation unit 134 calculates the distance L from the infrared camera 2 to the animal by the following formula (4).
L=D*(f/d) (4)

As described above, the recognition processing device 100 according to the present embodiment recognizes the animal and the reflection of the two eyes of the animal from the image, and determines the size of the animal based on the distance between the two eyes. Is configured to estimate. Here, as described above, the distance between the two eyes of the animal may be substantially constant regardless of individual differences. This makes it possible to accurately estimate the size of the animal.

Further, the recognition processing device 100 according to the present embodiment is configured to estimate the distance from the infrared camera 2 to the animal based on the estimated size of the animal or the distance between the two eyes. .. As mentioned above, the estimated animal size generally corresponds to the actual animal size of the animal. Further, as described above, the distance between the two eyes of the animal can be substantially constant regardless of individual differences. Therefore, with the configuration as described above, it is possible to accurately estimate the distance from the infrared camera 2 to the animal.

Further, the recognition processing device 100 according to the present embodiment recognizes an animal from an infrared image, and when two lights are detected in a range corresponding to an object recognized as an animal in the visible light image, two lights are detected. The light is configured to be recognized as a reflection of the animal's two eyes. This makes it possible to reliably recognize that the two lights are reflections of the animal's eyes.

Further, in the above embodiment, the recognition processing unit 120 may recognize the type of animal from the infrared image by using the animal recognition dictionary. In this case, since the size estimation unit 132 estimates the size of the animal using the distance between the two eyes of the animal corresponding to the recognized type of animal, the size of the animal can be calculated more accurately. Can be estimated.

(Modification) The present invention is not limited to the above-described embodiment, but can be modified as appropriate without departing from the spirit of the present invention. For example, the order of each process in the above-described flowchart can be changed as appropriate. Also, one or more of the processes in the above-described flowchart may be omitted.

For example, in the flowchart of FIG. 3, the process of S104 may be executed after the process of S106, such as after the process of S108. The order of the processing of S122 and the processing of S124 may be reversed. That is, when estimating the distance to the animal from the second distance of the animal instead of the size of the animal, the process of S124 may be performed before the process of S122.

Further, the program described above can be stored using various types of non-transitory computer readable media and can be supplied to the computer. Non-transitory computer readable media include various types of tangible storage media. Examples of the non-transitory computer-readable medium include a magnetic recording medium (for example, flexible disk, magnetic tape, hard disk drive), magneto-optical recording medium (for example, magneto-optical disk), CD-ROM (Read Only Memory), CD-R, It includes a CD-R/W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)). In addition, the program may be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

1 vehicle 2 infrared camera 4 visible light camera 20 recognition dictionary storage unit 30 display unit 50 recognition system 100 recognition processing device 112 image acquisition unit 120 recognition processing unit 122 calculation unit 130 estimation unit 132 size estimation unit 134 distance estimation unit 140 display control Department

Claims (5)

An image acquisition unit that acquires the image captured by the camera that captures the surroundings,
A recognition processing unit that recognizes the animal and the reflection of the two eyes of the animal from the image;
An estimation unit that estimates the size of the animal based on the distance between the two eyes obtained from the recognized reflection of the two eyes. The recognition processing device according to claim 1, wherein the estimation unit estimates the distance from the camera to the animal based on the estimated size of the animal or the estimated distance between the two eyes. The image acquisition unit acquires an infrared image by an infrared camera and a visible light image by a visible light camera,
The recognition processing unit,
Recognizing the animal from the infrared image,
In the visible light image, when two lights are detected in a range corresponding to an object recognized as the animal, the two lights are recognized as reflection of two eyes of the animal. The recognition processing device described. Get a video of the surroundings,
Recognizing the animal and the two eye reflections of the animal from the image,
A recognition processing method for estimating the size of the animal based on the distance between the two eyes obtained from the recognized reflection of the two eyes. A step of acquiring a video image of the surrounding area,
Recognizing the animal and the reflexes of the animal's two eyes from the image;
A step of estimating the size of the animal based on the distance between the two eyes obtained from the recognized reflection of the two eyes.