JP7468207B2 - IMAGE RECOGNITION DEVICE, IMAGE RECOGNITION METHOD, AND IMAGE RECOGNITION PROGRAM - Google Patents

Description

The present invention relates to an image recognition device, an image recognition method, and an image recognition program for recognizing objects within a captured image.

There is a driving assistance system that recognizes people such as pedestrians from images captured by a camera mounted on a vehicle and alerts the driver. The distance from the camera mounted on the vehicle to a person can be estimated based on the bottom end position of the area recognized as a person in the image (in the case of a person, the position where the person's toes or heels touch the ground) (see, for example, Patent Document 1). Similarly, in the case of a surveillance camera installed indoors, the distance from the surveillance camera to a person can be estimated based on the bottom end position of the area recognized as a person in the image.

JP 2018-018234 A

In the above-mentioned system, if the surface under a person's feet (e.g., the road surface or floor) is wet, the lower half of the person's body is reflected by the surface and captured in an image. The same is true when the surface is made of a material that easily reflects light (e.g., marble). In such cases, the image recognition system recognizes the feet reflected on the surface as one with the actual feet, and detects the person's feet as longer than they actually are. In other words, it recognizes the range of the person as an area expanded downward from the actual range. In this case, it may be determined that the recognized person is closer than they actually are, and unnecessary warnings may be issued. Similarly, for objects other than people (e.g., animals such as dogs, oncoming vehicles, and other obstacles), it may be recognized as an area expanded downward from the actual detection range.

This embodiment has been made in consideration of these circumstances, and its purpose is to provide a technology that detects the range of an object within an image with high accuracy.

In order to solve the above problems, an image recognition device of one aspect of this embodiment includes an image acquisition unit that acquires an image from an imaging device, an object recognition unit that recognizes a person as an object from the image acquired by the image acquisition unit and identifies the head of the recognized person , a symmetrical part detection unit that detects upper and lower symmetrical parts at the lower part of the range of the person based on the vertical ratio between the range of the person recognized by the object recognition unit and the head of the person , an object range correction unit that, when the symmetrical part detection unit detects upper and lower symmetrical parts, corrects the range of the person so that the position of the symmetrical center line of the detected upper and lower symmetrical parts corresponds to the lower limit position of the range of the person , and a distance estimation unit that estimates the distance from the imaging device to the person based on the position of the range of the person .

Another aspect of this embodiment is an image recognition method, which includes the steps of acquiring an image from an imaging device, recognizing a person as an object from the acquired image and identifying the head of the recognized person , detecting a vertically symmetrical portion at a lower portion of the recognized person's range based on a vertical ratio between the recognized person's range and the person's head, correcting the range of the person when the vertically symmetrical portion is detected so that the position of the symmetrical center line of the detected vertically symmetrical portion corresponds to the lower limit position of the range of the person , and estimating a distance from the imaging device to the person based on the position of the range of the person .

In addition, any combination of the above components, and conversions of the expressions of this embodiment between methods, devices, systems, recording media, computer programs, etc., are also valid aspects of this embodiment.

According to this embodiment, the range of an object within an image can be detected with high accuracy.

1 is a diagram showing a configuration of an image recognition system according to an embodiment of the present invention. 10A to 10C are diagrams for explaining a specific example of a detection frame correction process according to the first embodiment; FIG. 2 is a diagram for explaining a specific example of the detection frame correction process in the first embodiment (part 2); FIG. 3 is a diagram for explaining a specific example of the detection frame correction process in the first embodiment; FIG. 4 is a diagram for explaining a specific example of the detection frame correction process in the first embodiment; FIG. 11 is a diagram for explaining a specific example of a correction process for a detection frame according to the second embodiment; FIG. 2 is a diagram for explaining a specific example of a correction process of a detection frame according to the second embodiment; FIG. 11 is a diagram for explaining a specific example of the detection frame correction process according to the second embodiment (part 3).

FIG. 1 is a diagram showing the configuration of an image recognition system 1 according to an embodiment of the present invention. The image recognition system 1 according to the embodiment includes an imaging device 10, an image recognition device 20, a display device 30, and an audio output device 40. In the following, in this embodiment, an example is assumed in which the image recognition system 1 is mounted on a vehicle.

The imaging device 10 captures an image of the area ahead in the direction of travel of the vehicle, and is attached, for example, to the front grille or to the rearview mirror inside the vehicle. The imaging device 10 includes a solid-state imaging element and a signal processing circuit. For example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Devices) image sensor can be used as the solid-state imaging element. The solid-state imaging element converts incident light into an electrical image signal and outputs it to the signal processing circuit. The signal processing circuit performs signal processing such as A/D conversion and noise removal on the image signal input from the solid-state imaging element, and outputs it to the image recognition device 20.

The solid-state imaging element may be an element capable of capturing images in the visible light range, an element capable of capturing images in the infrared range, or an element capable of capturing images in both wavelength ranges. That is, the imaging device 10 may be a general visible light camera, a camera capable of capturing images in both the visible light range and the near-infrared range, or a far-infrared (FIR) camera.

For example, a solid-state imaging element of a camera capable of capturing images in the visible light and near-infrared light ranges may use a color filter in which one of the two G filters that make up a normal Bayer array is replaced with an IR filter.

The solid-state image sensor of a far-infrared camera converts far-infrared radiation emitted by objects such as people that act as heat sources into a thermal image. In automotive applications, far-infrared cameras are used in night vision systems to assist the driver's nighttime vision. Note that when a near-infrared camera is used instead of a far-infrared camera, it is necessary to shine near-infrared radiation at the object.

The image recognition device 20 includes an image acquisition unit 21, an object recognition unit 22, a symmetrical part detection unit 23, an object tracking unit 24, an object range correction unit 25, a distance estimation unit 26, a notification control unit 27, and a ground surface information acquisition unit 28. These components can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As hardware resources, a CPU, ROM, RAM, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and other LSIs can be used. As software resources, programs such as firmware can be used.

The imaging device 10 and the image recognition device 20 are connected by a cable. However, a wireless connection is also possible as long as the requirements for real-time performance and noise resistance are met.

The image acquisition unit 21 acquires an image from the imaging device 10. The object recognition unit 22 recognizes a specific object from the image acquired by the image acquisition unit 21. In the following, in this embodiment, a person is assumed to be the object.

The object recognition unit 22 has a person classifier that has been generated by learning a large number of images that include people as dictionary data. The object recognition unit 22 searches the input image acquired by the image acquisition unit 21 using the person classifier. For example, HOG (Histograms of Oriented Gradients) features can be used to detect people. Note that Haar-like features and LBP (Local Binary Patterns) features may also be used. When a person is present in the input image, the object recognition unit 22 detects the person using a rectangular detection frame.

If the floor under a person's feet is made of a reflective material (e.g., marble) or if there is a puddle under the person's feet, the image of the person will be mirror-reflected on the floor or in the puddle. In this case, the image captured by the imaging device 10 will also show an upside-down image of the person under the person's feet. In this case, it will be difficult for the object recognition unit 22 to identify the bottom end of the person's legs. Depending on the accuracy of the feature points of the person's legs extracted for each frame, the bottom edge of the person's detection frame will fluctuate up and down on a frame-by-frame basis. The object recognition unit 22 will erroneously recognize the person's legs as longer than when the person's image is not mirror-reflected.

The person detection frame is the basis for distance estimation, which will be described later, so it is desirable for the upper edge of the detection frame to accurately capture the top of the person's head, and the lower edge of the detection frame to accurately capture the bottom of the person's legs. However, if there is specular reflection at the person's feet, there are cases in which the bottom edge of the person's detection frame is significantly misaligned with the bottom of the person's legs. Therefore, in this embodiment, a mechanism is introduced to correct the range of the detection frame if there is a top-bottom symmetrical area at the bottom of the detection frame.

First, a method for correcting the detection frame according to the first embodiment will be described. The symmetrical part detection unit 23 detects vertically symmetrical parts in the lower part of the detection frame of the person recognized by the object recognition unit 22. For example, vertically symmetrical parts are detected as follows. The symmetrical part detection unit 23 cuts out an image from a part of the upper part of the search area for vertically symmetrical parts. The search area for vertically symmetrical parts may be set to the lowermost area when the detection frame is divided into three equal parts in the Y-axis direction, for example.

The symmetrical part detection unit 23 flips upside down an image that is cut out from a portion of the upper part of a search area for vertically symmetrical parts. The symmetrical part detection unit 23 matches the upside-down flipped image with the image in the search area while sliding it up and down in the search area for vertically symmetrical parts. The symmetrical part detection unit 23 identifies the matching position that provides the highest degree of matching. If the highest degree of matching is equal to or greater than a preset threshold, the symmetrical part detection unit 23 determines that a vertically symmetrical part exists, and if the degree is less than the threshold, determines that a vertically symmetrical part does not exist.

If it is determined that there are upper and lower symmetrical parts, the symmetrical part detection unit 23 sets the midline between the upper cut-out position and the matching position within the search range to the position of the symmetrical center line of the upper symmetrical part. The position of the symmetrical center line of the upper symmetrical part can be estimated to be the person's grounded position. The object range correction unit 25 corrects the range of the detection frame so that the position of the symmetrical center line of the upper and lower symmetrical parts corresponds to the lower limit position of the person's detection frame. If the symmetrical part detection unit 23 determines that there are no upper and lower symmetrical parts, the object range correction unit 25 does not correct the range of the person's detection frame.

The distance estimation unit 26 estimates the distance from the imaging device 10 installed in the vehicle to the person based on the position of the detection frame of the detected person. Specifically, the distance estimation unit 26 estimates the distance from the imaging device 10 (i.e., the vehicle) to the person based on the y coordinate of the ground position of the person detected in the captured image (i.e., the bottom side of the person's detection frame) and a previously derived table or function.

The table or function describes the relationship between the y coordinate of the ground contact position of a person in an image, which is determined according to the installation position (height from the road surface) of the imaging device 10 on the vehicle and the angle of view of the imaging device 10, and the distance from the imaging device 10 to the person.

The notification control unit 27 outputs a warning to at least one of the display device 30 and the audio output device 40 depending on the distance from the vehicle to the person estimated by the distance estimation unit 26.

The display device 30 may be a HUD (Head-up Display) installed in the vehicle. The HUD may be a windshield-illuminated type or a combiner type. The display device 30 may be a display (e.g., a liquid crystal display or an organic EL display) of a car navigation system or a display audio system installed in the vehicle. The display device 30 may be a display of a smartphone or a tablet linked to a car navigation system or a display audio system. The display device 30 may be a display in a meter panel installed in the vehicle. The display device 30 may be an electronic rearview mirror that constitutes a camera monitoring system. The display device 30 may be a dedicated display that displays an image in front of the vehicle.

The audio output device 40 may be a speaker of a car navigation system or a display audio system installed in a vehicle. The audio output device 40 may also be a speaker of a smartphone or a tablet connected to a car navigation system or a display audio system. The audio output device 40 may also be a speaker dedicated to outputting a warning sound or a warning message.

When a person is detected in an image captured by the imaging device 10, the notification control unit 27 superimposes a detection frame of the person on the screen displayed on the display device 30. The notification control unit 27 changes the display mode of the detection frame according to the distance from the vehicle to the person estimated by the distance estimation unit 26. When the distance from the vehicle to the person is equal to or greater than a preset threshold, the notification control unit 27 displays the detection frame in a relatively inconspicuous color (e.g., white or yellow). When the distance from the vehicle to the person is less than the threshold, the notification control unit 27 displays the detection frame in a relatively conspicuous color (e.g., yellow or red). When the distance from the vehicle to the person is less than the threshold, the notification control unit 27 may blink the detection frame. When the distance from the vehicle to the person is less than the threshold, the notification control unit 27 may thicken the detection frame.

The notification control unit 27 causes the audio output device 40 to output a warning sound or warning message when a person is detected in an image captured by the imaging device 10 and the distance from the vehicle to the person is less than the threshold. The notification control unit 27 may also cause the audio output device 40 to output a warning sound or warning message with a relatively low volume when the distance from the vehicle to the person is equal to or greater than the threshold, and may cause the audio output device 40 to output a warning sound or warning message with a relatively high volume when the distance is less than the threshold.

It is also possible to omit either the notification by the display device 30 or the notification by the audio output device 40. In addition, if a vibrator is installed on the steering wheel, the notification control unit 27 may vibrate the vibrator when a person is detected in the image captured by the imaging device 10 and the distance from the vehicle to the person is less than the threshold value.

Figures 2 to 5 are diagrams for explaining a specific example of the correction process for the detection frame D1 according to the first embodiment. Figure 2 shows a state in which a reflected image R1 of a person image O1 is reflected in a puddle P1 on the road surface, and the detection frame D1 extends downward beyond the actual range of the person in the frame image F1. Figure 3 shows an area As that includes upper and lower symmetrical parts detected at the bottom of the detection frame D1.

As shown in FIG. 4, the symmetrical part detection unit 23 detects the symmetrical center line Lc of the upper symmetrical part. As shown in FIG. 5, the object range correction unit 25 shifts the bottom side of the detection frame D1 upward to the position of the detected symmetrical center line Lc. This causes the detection frame D1 to be corrected to a state that reflects the actual range of the person. The object range correction unit 25 may correct the position of the bottom side of the detection frame by changing the aspect ratio of the detection frame, or may correct the position of the bottom side of the detection frame without changing the aspect ratio of the detection frame.

Returning to FIG. 1. Next, a method of correcting a detection frame according to Example 2 will be described. In Example 2, the object recognition unit 22 includes a head recognition unit 22a. The head recognition unit 22a detects the area of the head of a detected person within the detection frame of the person. The head recognition unit 22a has a head classifier generated by learning a large number of images containing the person's head as dictionary data. The head recognition unit 22a searches within the image area defined by the person's detection frame using the person's head classifier. The head recognition unit 22a detects the person's head within the person's detection frame using a rectangular detection frame.

The object tracking unit 24 tracks the range of the person recognized by the object recognition unit 22 in the subsequent frame image. For example, a particle filter or a mean shift method can be used to track the range of the person. This makes it possible to track each person individually even if multiple people are detected in a frame image.

In the second embodiment, the symmetrical part detection unit 23 calculates the head-to-body ratio of the person based on the ratio between the vertical length of the person's detection frame and the vertical length of the detection frame for the person's head. The symmetrical part detection unit 23 stores the head-to-body ratio of the same person being tracked by the object tracking unit 24 in chronological order. As described above, if specular reflection occurs at the feet of the person, the bottom side of the person's detection frame extends further downward than it actually is.

If the calculated head-to-body ratio is within a set range (e.g., 7-8 head-to-body ratio for adults, 4-7 head-to-body ratio for children), the symmetrical part detection unit 23 determines that the person was photographed without specular reflection. If the head-to-body ratio of a person photographed without specular reflection changes to a value that exceeds the upper limit of the set range, the symmetrical part detection unit 23 determines that the person has changed to a state with specular reflection. As described above, the range of a person photographed with specular reflection includes upper and lower symmetrical parts. The symmetrical part detection unit 23 detects the occurrence of upper and lower symmetrical parts based on the calculated change in the head-to-body ratio.

When a vertically symmetrical part occurs, the object range correction unit 25 corrects the detection frame of the person captured with specular reflection based on the head-to-body ratio value of the person captured with no specular reflection. Specifically, the object range correction unit 25 shifts the bottom edge of the detection frame upward based on the position of the head within the detection frame to the foot position corresponding to the head-to-body ratio when the image was captured with no specular reflection.

Figures 6 to 8 are diagrams for explaining a specific example of the correction process of the detection frame D1 according to the second embodiment. Figure 6 shows a person's detection frame D1 and the person's head detection frame D2 in a frame image F0 captured without specular reflection. The symmetrical part detection unit 23 calculates the person's head-to-body ratio based on the ratio between the vertical length h1 of the person's detection frame D1 and the vertical length h2 of the person's head detection frame D2. In the example shown in Figure 6, the head-to-body ratio (h1/h2) falls within the set range, and the symmetrical part detection unit 23 determines that the person was captured without specular reflection.

Figure 7 shows a person detection frame D1 and the person's head detection frame D2 in a frame image F1 captured with specular reflection. The symmetrical part detection unit 23 calculates the person's head-to-body ratio based on the ratio between the vertical length h1 of the person's detection frame D1 and the vertical length h2 of the person's head detection frame D2. In the example shown in Figure 7, the head-to-body ratio (h1/h2) exceeds the upper limit of the set range, and the symmetrical part detection unit 23 determines that the person was captured with specular reflection.

If it is determined that the person was captured with specular reflection, as shown in FIG. 8, the object range correction unit 25 shifts the bottom edge of the detection frame D1 of the person in the frame image F1 captured with specular reflection upward so that the head-to-body ratio matches the head-to-body ratio calculated in the frame image F0 captured without specular reflection.

Returning to FIG. 1, the ground surface information acquisition unit 28 acquires information for estimating the state of the ground surface at the location where a person detected in an image captured by the imaging device 10 is located. Based on the acquired information, the ground surface information acquisition unit determines whether or not the state of the ground surface is likely to reflect the image of a person (i.e., whether or not specular reflection is likely to occur).

The ground contact information acquisition unit 28 acquires operation information of the vehicle's wipers as information for estimating the state of the ground contact. The wiper operation information can be acquired via an in-vehicle network (e.g., a Controller Area Network (CAN) or a Local Interconnect Network (LIN)). If the wipers are operating, the ground contact information acquisition unit 28 can estimate that it is raining, and estimates that the state of the ground contact is one in which specular reflection is likely to occur.

The ground contact information acquisition unit 28 may acquire weather information at the current location of the vehicle via the Internet as information for estimating the state of the ground contact. In this case, a wireless communication function for accessing an external server is required. The current location of the vehicle can be acquired from a GPS (Global Positioning System) sensor mounted on the car navigation system. If the weather at the current location of the vehicle is raining, the ground contact information acquisition unit 28 estimates that the state of the ground contact is one in which specular reflection is likely to occur. Furthermore, the ground contact information acquisition unit 28 estimates that the state is one in which specular reflection is likely to occur not only when it is raining, but also for a predetermined period after it rains.

When the state of the ground surface is such that specular reflection is likely to occur, the functions of the symmetrical part detection unit 23 and the object range correction unit 25 are enabled, and when the state is such that specular reflection is unlikely to occur, the functions of the symmetrical part detection unit 23 and the object range correction unit 25 are disabled. For example, when the road surface is dry, the image of a person is not reflected on the road surface, and there is little need to operate the symmetrical part detection unit 23 and the object range correction unit 25. By stopping the symmetrical part detection unit 23 and the object range correction unit 25, the processing load of the image recognition device 20 can be reduced.

The state of the ground surface can also be estimated by analyzing the image captured by the imaging device 10. Regardless of the state of the ground surface, the symmetrical part detection unit 23 and the object range correction unit 25 may be operated at all times. In that case, the ground surface information acquisition unit 28 is not required.

As described above, according to this embodiment, even if an image of a person is captured by specular reflection on the ground, the lower edge position of the person detection frame can be corrected upward to prevent the range of the person from being erroneously detected as being longer downward than it actually is. In other words, the range of the person in the image can be detected with high accuracy. This makes it possible to prevent an erroneous estimation of the distance from the vehicle to the person and the issuance of unnecessary warnings.

The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention.

In the above-described embodiment, an example of recognizing a person as an example of an object has been described. In this regard, the object is not limited to a person, but also includes animals such as dogs and cats, oncoming vehicles, preceding vehicles, bicycles, and motorcycles. The object recognition unit 22 detects each object in the captured image based on the classifier of each object. As in the case of a person, these objects are also mirror-reflected on the ground surface, resulting in upper and lower symmetrical parts. In this case, the object range correction unit 25 corrects the range of the object so that the position of the symmetric center line of the upper and lower symmetrical parts corresponds to the lower limit position of the range of the object.

In the above-described embodiment, an example is assumed in which the image recognition system 1 is mounted on a vehicle. In this regard, the image recognition system 1 may be mounted on a moving object other than a vehicle. For example, the image recognition system 1 may be mounted on a motorcycle, bicycle, railroad vehicle, ship, airplane, multicopter (drone), etc. Furthermore, the image recognition system 1 may be mounted on a surveillance camera. In this case, too, it is possible to prevent the distance from the surveillance camera to an intruder from being erroneously estimated and unnecessary warnings being issued.

1 Image recognition system, 10 Imaging device, 20 Image recognition device, 21 Image acquisition unit, 22 Object recognition unit, 22a Head recognition unit, 23 Symmetrical part detection unit, 24 Object tracking unit, 25 Object range correction unit, 26 Distance estimation unit, 27 Notification control unit, 28 Ground surface information acquisition unit, 30 Display device, 40 Audio output device.

Claims (4)

an image acquisition unit that acquires an image from an imaging device;
an object recognition unit that recognizes a person as an object from the image acquired by the image acquisition unit and identifies a head of the recognized person ;
a symmetrical part detection unit that detects upper and lower symmetrical parts in a lower part of the range of the person recognized by the object recognition unit based on a vertical ratio between the range of the person and the head of the person;
an object range correction unit that corrects the range of the person when the symmetrical part detection unit detects a vertically symmetrical part so that a position of a symmetrical center line of the detected vertically symmetrical part corresponds to a lower limit position of the range of the person ;
a distance estimation unit that estimates a distance from the imaging device to the person based on a position of the range of the person ;
An image recognition device comprising: A ground surface information acquisition unit that acquires information for estimating a state of a ground surface at a location where the person is located,
When the state of the ground surface is such that specular reflection of the image of the person is likely to occur , the functions of the symmetrical part detection unit and the object range correction unit are enabled.
The image recognition device according to claim 1 . acquiring an image from an imaging device;
A step of recognizing a person as an object from the acquired image and identifying a head of the recognized person ;
detecting a vertically symmetrical portion in a lower portion of the recognized person's area based on a vertical ratio between the recognized person's area and the person's head ;
when detecting vertically symmetrical parts, correcting the range of the person so that the position of the symmetrical center line of the detected vertically symmetrical parts corresponds to the lower limit position of the range of the person ;
estimating a distance from the imaging device to the person based on a position of the range of the person ;
The image recognition method includes: Acquiring an image from an imaging device;
A process of recognizing a person as an object from the acquired image and identifying the head of the recognized person ;
A process of detecting a vertically symmetrical portion in a lower portion of the recognized person's range based on a vertical ratio between the recognized person's range and the person's head ;
When vertically symmetrical parts are detected, a process of correcting the range of the person so that the position of the symmetrical center line of the detected vertically symmetrical parts corresponds to the lower limit position of the range of the person ;
A process of estimating a distance from the imaging device to the person based on a position of the range of the person ;
An image recognition program that causes a computer to execute the following:

Images