JP2020195140A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To detect a person appearing in an image more accurately.SOLUTION: In a visible light image obtained by shooting while using visible light, and an infrared image obtained by shooting in a shooting range including the shooting range of the visible light image, by using infrared light, an image processing apparatus detects a person, and judges a person not detected in the infrared light image, out of the persons detected in the visible light image, as a person who exists actually.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for detecting a person from an image taken by an imaging device.

Conventionally, there is a person detection device that detects a person area (hereinafter, simply referred to as "person") reflected in a surveillance image using a surveillance image. Such a person detection device is generally used by being incorporated in a monitoring system. In addition, a network camera is generally used for shooting surveillance images.

As an example of the person detection method implemented by the person detection device, there is a method of detecting a face portion of a person (for example, Patent Document 1). In such a method for detecting a face portion, a method using image brightness and color information is common.

Japanese Unexamined Patent Publication No. 2014-64083

The person detection method using image brightness and color information as described in Patent Document 1 has the following problems. That is, there is a problem that it is difficult to distinguish a face on a poster, a monitor, or the like from an actual person (real person) imaged by an imaging device. Here, the face in the poster is a face in a printed matter such as a poster photographed by an imaging device. The face on the monitor is a face displayed on the monitor taken by the imaging device.

The present invention has been made in view of the above problems, and an object of the present invention is to more accurately detect a person appearing in an image.

As one means for achieving the above-mentioned object, the image processing apparatus of the present invention has the following configuration. That is, in the visible light image obtained by shooting with visible light, it corresponds to the first detection means for detecting the face of the person by using the image feature of the face of the person and the shooting range of the visible light image. In the infrared light image obtained by photographing the imaging range to be performed using infrared light, the second detection means for detecting the face of a person using the image features and the first in the visible light image. When the face is not detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the detection means, it is detected by the first detection means in the visible light image. A predetermined image process is executed on the region of the face, and the second detection means is performed at a position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image. When the face is detected by the above, the first detecting means in the visible light image has an image processing means that does not execute the predetermined image processing on the detected area of the face. ..

According to the present invention, it is possible to more accurately detect a person appearing in an image.

The figure which shows an example of the structure of the image pickup apparatus and the image processing apparatus in Embodiment 1. FIG. The figure for demonstrating the face detection process in Embodiment 1. The flowchart of an example of the person detection process in Embodiment 1. The figure which shows the general monitoring system configured by using the person detection device. The figure for demonstrating an example of the conventional person detection method. The figure which shows an example of the structure of the image pickup apparatus and the image processing apparatus in Embodiment 2. FIG.

Hereinafter, the present invention will be described in detail based on the embodiments with reference to the accompanying drawings. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration.

First, FIG. 4 shows a general surveillance system configured by using a network camera and a person detection device. In FIG. 4, the network camera 401 is connected to the person detection device 403 and various control devices 404 via a network 402 such as a LAN (Local Area Network). Although only one network camera is shown in FIG. 4, a plurality of network cameras may be connected to the person detection device 403 and various control devices 404. The person detection device 403 receives the surveillance image from the network camera 401 and detects the person appearing in the surveillance image. The various control devices 404 may, for example, receive a surveillance image from the network camera 401 and display an image reflecting the person detection result by the person detection device 403 on the surveillance image on a display device (not shown) separately connected. it can. Further, the various control devices 404 can store the surveillance video received from the network camera 401 in a separately connected data storage device (not shown) according to the person detection result by the person detection device 403. In the example of FIG. 4, the person detection device 403 and various control devices 404 are divided into two devices, but it is also possible to arrange both functions in the same device.

Here, a problem in a person detection method using image brightness and color information as described in Patent Document 1 will be described in more detail with reference to FIG. FIG. 5 is a diagram for explaining an example of a conventional person detection method.

In FIG. 5A, image 501 is an image taken at a certain angle of view. In image 501, poster 503, poster 504, person 505, and person 506 are shown. The poster 503 and the poster 504 are posters on which a person is printed, and the person 505 and the person 506 are real people who actually exist. FIG. 5B shows an image 502 in which a person's face 507 to 510 is detected using the brightness and color information of the image with respect to the image 501. From the viewpoint of image brightness and color information, the person (face 507 and face 508) and the real person (face 509 and face 510) appearing on the poster look the same. Therefore, as shown in image 502, all faces 507 to 510 are detected as "human faces". However, in reality, the faces 507 and 508 do not correspond to real people because they are the faces reflected in the poster (the faces of the printed matter on the poster). Such a person detection technique is not suitable for a monitoring system that detects only a real person.

Hereinafter, embodiments for more accurately detecting a person appearing in an image will be described.

[Embodiment 1]
FIG. 1 is a diagram showing an example of the configuration of the image pickup apparatus 101 and the image processing apparatus 111 according to the first embodiment. In FIG. 1, some components of the image pickup apparatus 101 are omitted. In addition, the image pickup device 101 can capture moving images and still images. The image processing device 111 functions as a person detection device in this embodiment.

First, the configuration of the image pickup apparatus 101 will be described. In the image pickup apparatus 101, the objective lens 102, the photographing aperture 103, the focus lens 104, and the image pickup lens 105 are sequentially arranged on the optical axis 11, and the dichroic mirror 106 is arranged behind the objective lens 102, the focus lens 104, and the image pickup lens 105. When light is incident, it is condensed by the objective lens 102 and incident on the photographing diaphragm 103. The light that has passed through the shooting aperture 103 is incident on the lens system including the focus lens 104 and the image pickup lens 105. The light that has passed through the lens system is incident on the dichroic mirror 106 for wavelength separation.

The dichroic mirror 106 is configured to transmit visible light and reflect infrared light. As an example, visible light is light having a wavelength of less than 800 nm, and infrared light is light having a wavelength of 800 nm or more. Therefore, in the present embodiment, the dichroic mirror 106 is configured to transmit light having a wavelength of less than 800 nm and reflect light having a wavelength of 800 nm or more. In this embodiment, the visible light may include ultraviolet light. Further, the dichroic mirror 106 may reflect only light having a wavelength in a specific range. For example, only light having a wavelength of 900 nm to 1300 nm may be reflected.

The visible light transmitted through the dichroic mirror 106 is incident on the visible light image sensor 107 arranged behind the dichroic mirror 106. The visible light image sensor 107 receives the incident visible light and converts the received visible light into an electric charge to generate an image pickup signal (visible light image pickup signal).

On the other hand, the infrared light component reflected by the dichroic mirror 106 passes through the optical axis 12 and is incident on the infrared light image sensor 108. The infrared light image sensor 108 receives the incident infrared light and converts the received infrared visual light into an electric charge to generate an image pickup signal (infrared light image pickup signal).

For the visible light image sensor 107 and the infrared light image sensor 108, for example, a CMOS image sensor (Complementary Metal Oxide SemicONductor Image Sensor) can be used. Further, a CCD image sensor (Charge Coupled Device Image Sensor) may be used for the visible light image sensor 107 and the infrared light image sensor 108. Further, the visible light imaging element 107 and the infrared light imaging element 108 are arranged at substantially conjugate positions in order to form an image in the same manner, but the focusing position shifts due to the wavelength difference between the infrared light and the visible light. The infrared light imaging element 108 is arranged so as to be displaced rearward on the optical path. Further, the imaging device 101 can also have an illumination light source including an LED (Light Emitting Diode) light source that emits near-infrared light (not shown).

The outputs of the visible light image sensor 107 and the infrared light image sensor 108 are processed into image data by the image processing unit 109. Such image data is, for example, color space information such as YCbCr, RGB, and YUV that can be used by a general image processing device, or compressed image data such as Motion JPEG compression or h.264 compression. The image processing unit 109 generates data obtained by subjecting the outputs from the visible light image sensor 107 and the infrared light image sensor 108 to such processing as a visible light image and an infrared light image.

The visible light image and the infrared light image generated by the image processing unit 109 are sent to the control unit 110. The control unit 110 can transfer the visible light image and the infrared light image to an external device such as an external storage or an external computer connected via a network such as a LAN. In addition, the control unit 110 receives instructions from an external device to start or end shooting to the image pickup device 101, and the image pickup device 101 can perform control corresponding to the instructions.

As described above, the image pickup device 101 includes the visible light image sensor 107 and the infrared light image sensor 108, so that the visible light image obtained by imaging the received visible light and the infrared ray obtained by imaging the received infrared light are imaged. It is possible to generate an optical image. Moreover, these images have the same angle of view.

Here, the infrared light image generated by the image pickup apparatus 101 will be described. In general, human skin has high absorption from around 1000 nm of wavelength, and shows higher absorption than clothes at wavelengths of 1400 to 1600 nm. Therefore, in the infrared light image, the person appears black (see FIG. 2 described later). On the other hand, black ink (carbon black, etc.) for producing shades and black color of printed matter appears in an infrared light image in the same manner as a visible image. Therefore, in the infrared light image, the person on the printed matter becomes an image similar to the visible image even though the contour, the contrast, and the like are coordinated.

Subsequently, the configuration of the image processing device 111 will be described. The communication unit 112 acquires the visible light image and the infrared light image generated by the image pickup apparatus 101. In the present embodiment, the visible light image and the infrared light image are images based on shooting at the same angle of view. Therefore, the visible light image and the infrared light image are images in the same shooting range. Further, in the present embodiment, the visible light image and the infrared light image are images based on shooting at the same time.

The image input unit 113 performs data expansion processing on the visible light image and the infrared light image acquired by the communication unit 112 as necessary, and sends them to the face detection unit 114. The face detection unit 114 executes a face detection process as a person detection process on the image received from the image input unit 113.

As an example of the face detection process executed by the face detection unit 114, there is a method using an eigenface by principal component analysis. This method is described, for example, in "M.A.Turk and A.P.Pentland," Face Recognition Using Eigenfaces ", Proc. Of I3 Conf. On Computer Vision and Pattern Recognition, pp.586-591, 1991." Further, as shown in JP-A-9-251534, the face detection process may be performed by a method utilizing feature points such as eyes, nose, and mouth. In these methods, it is determined whether or not the input image is a person's face by a pattern matching method of the input image and a plurality of standard patterns. As a result of face detection in the face detection process, information on feature points of a face image (nose / mouth / face contour shape, size of each, position of feature points, etc.) can be mentioned. Since the face detection process itself can be realized by a known technique as described above, detailed description thereof will be omitted here. In addition, different face detection methods and parameters may be used for the visible light image and the infrared light image. For example, since the infrared light image is often monochrome, a face detection method or parameter suitable for the monochrome image may be used.

Here, the face detection process by the face detection unit 114 will be described with reference to FIG. FIG. 2 is a diagram for explaining the face detection result in the present embodiment. FIG. 2A shows a visible light image 201. As a result of face detection processing by the face detection unit 114 for the visible light image 201, faces 203 to 206 are output as detection results. On the other hand, FIG. 2B shows an infrared light image 202. As a result of the face detection process by the face detection unit 114 on the infrared light image 202, the face 207 and the face 208 are output as the detection results. Face 209 and face 210 are not output as detection results because the skin of a real person appears black.

The detection result for the visible light image and the detection result for the infrared light image output from the face detection unit 114 are compared by the comparison unit 115. By comparing the two detection results, the comparison unit 115 determines that the person (face) detected in the visible light image but not detected in the infrared light image is a real person.

Referring to FIG. 2, for example, the comparison unit 115 subtracts the regions of the face 207 and the face 208 detected in the infrared light image 202 from the regions of the faces 203 to 206 detected in the visible light image 201 (the region (the region of the face 207 and the face 208 detected in the infrared light image 202). The areas of faces 207 and 208) are determined to be the faces of real people.

Further, the comparison unit 115 may determine whether or not the face is detected in the infrared light image 202 at the same position as the position of the face detected in the visible light image. In this case, if the face is detected at the same position, the comparison unit 115 determines that the face detected in the visible light image is not the face of a real person. However, the positions of the faces detected in the visible light image and the infrared light image may be slightly different from each other. Therefore, if it is within a predetermined deviation amount, it may be determined as "the same position". On the other hand, the comparison unit 115 determines that the face detected in the visible light image is the face of a real person when the face is not detected at the same position, that is, when the face is detected only in the visible image. judge. Referring to FIG. 2, for example, in the comparison unit 115, the face 207 was detected in the infrared light image 202 at the position of the face 203 detected in the visible light image 201, so that the face 203 is not the face of a real person. Is determined. Further, the comparison unit 115 determines that the face 205 is the face of a real person because the face is not detected in the infrared light image 202 at the position of the face 205 detected in the visible light image 201.

The information of the person determined by the comparison unit 115 is sent to the output unit 116. The output unit 116 aggregates the information of the person and outputs the aggregated result to a separately connected display device (not shown), and the output unit 116 outputs the aggregated result to the separately connected external storage (not shown). ) Can also be accumulated.

Next, the person detection process by the image processing device 111 will be described with reference to FIG. FIG. 3 is a flowchart of an example of the person detection process in the present embodiment.

First, the image processing device 111. The visible light image A is acquired from the image pickup apparatus 101 (s301). Subsequently, the image processing device 111 acquires an infrared light image B from the image pickup device 101 (s302). In these image acquisition processes, images are acquired sequentially or simultaneously from the image pickup apparatus via the network. The processing order of s301 and s302 may be reversed. Subsequently, the image processing device 111 performs the face detection process as described above on the acquired visible light image A and infrared light image B (s303).

After the face detection process (s303), the image processing device 111 compares whether or not all the faces detected in the visible light image A are detected in the infrared light image B (s304). For example, the image processing device 111 compares whether or not the face detected in the visible light image A is also detected at the same position in the infrared light image B. If they are detected at the same position, the image processing device 111 determines that the face detected in the visible light image A is not the face of a real person. On the other hand, if it is not detected at the same position, that is, if it is detected only in the visible image, the image processing device 111 determines that the face detected in the visible light image A is the face of a real person. .. If it is determined that the face is a real person according to this determination result, the count of the number of people is incremented by 1. Through the above processing, the image processing device 111 obtains the result of counting the number of real people existing in the same angle of view (s305).

As described above, according to the present embodiment, even when a person printed on a printed matter such as a poster and a real person who actually exists are mixed in the same image, it is possible to detect only the real person. Therefore, the imaging device and the image processing device in the present embodiment can realize a monitoring system that detects and counts only a real person from the captured monitoring images.

[Embodiment 2]
FIG. 6 is a diagram showing an example of the configuration of the image pickup apparatus 101 and the image processing apparatus 111 according to the second embodiment. The difference is that the image processing unit 117 is added as compared with FIG. 1 described in the first embodiment. The image processing unit 117 performs various image processing on the region where the real person (the face of the real person) is determined to exist in the visible light image. For example, the image processing unit 117 may superimpose a mask image for privacy protection on an area where it is determined that a real person exists. Superimposing a mask image is effective in a situation where a real person does not want to be viewed. Further, the image processing unit 117 may perform image processing so that the image quality (for example, resolution, gradation, compression rate) of the region where the real person is determined to exist is higher than the image quality of the other regions. By increasing the image quality, it is possible to browse the detected real person with high accuracy. The image processed by the image processing unit 117 can be output to an external device via the output unit 116.

As described above, according to the present embodiment, in addition to the effects described in the first embodiment, it is possible to generate an image according to the treatment of the detected real person.

[Other Embodiments]
In the above embodiment, an example of detecting a person has been described, but the present embodiment can also be applied to an example of detecting a subject whose appearance differs between a visible light image and an infrared light image. Further, although the above embodiment has described an example in which a person is detected by detecting a face, the present invention is not limited to the face, and the person may be detected by detecting another part. Further, in the above embodiment, the example of using visible light and infrared light has been described, but it is also possible to use invisible light other than infrared light instead of infrared light. Further, in the above embodiment, the visible light image and the infrared light image have the same shooting range, but they do not have to be exactly the same. For example, the shooting range of the visible light image may be included in the shooting range of the infrared light image.

Further, the image processing device 111 may store the information of the face detected in the visible light image that is determined not to be the face of a real person. In this case, for example, the image processing device 111 does not require the processing of the comparison unit 115, and can determine a face other than the face of a real person in the newly obtained visible light image. The image processing device 111 can also output face information determined to be not the face of such a real person via the output unit 116 for all purposes.

Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device implement the program. It can also be realized by the process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 imager, 11 optical axis. 12 optical axis, 102 objective lens, 103 shooting aperture, 104 focus lens, 105 image sensor, 106 dichroic mirror, 107 visible light image sensor, 108 infrared light image sensor, 109 image processing unit, 110 control unit, 111 image processing device , 112 Communication section, 113 Image input section, 114 Face detection section, 115 Comparison section, 116 Output section

Claims (13)

In the visible light image obtained by photographing with visible light, the first detection means for detecting the face of the person by using the image feature of the face of the person and
A second detection means for detecting a person's face by using the image features in an infrared light image obtained by photographing an imaging range corresponding to the imaging range of the visible light image using infrared light. When,
When the face is not detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the face is detected in the visible light image. A predetermined image process is executed on the face area detected by the first detection means,
When the face is detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the first in the visible light image. (1) An image processing means that does not perform the predetermined image processing on the face area detected by the detection means, and
An image processing device characterized by having. The image processing means
When the face is not detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the face in the visible light image A mask process for masking the face area detected by the first detection means is executed as the predetermined image process.
When the face is detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the first in the visible light image. 1. The image processing apparatus according to claim 1, wherein the mask processing is not executed on the face region detected by the detection means. The image processing means
When the face is not detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the face is detected in the visible light image. An image quality enhancement process for increasing the image quality of the face region detected by the first detection means is executed as the predetermined image process.
When the face is detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the first in the visible light image. 1. The image processing apparatus according to claim 1, wherein the image quality improving process is not executed on the face region detected by the detection means. An image processing device that executes detection processing to detect a person's face.
In the visible light image obtained by photographing with visible light, the first detection means for detecting the face of the person by using the image feature of the face of the person and
A second detection means for detecting a person's face by using the image features in an infrared light image obtained by photographing an imaging range corresponding to the imaging range of the visible light image using infrared light. When,
It has an output means for outputting information on the detection result of the detection process.
When the face is not detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the face is not detected in the visible light image. The face detected by the first detection means is included in the detection result, and is included in the detection result.
When the face is detected by the second detection means at the position in the infrared light image corresponding to the position where the face is detected by the first detection means in the visible light image, the first in the visible light image. (1) An image processing apparatus characterized in that the face detected by the detection means is not included in the detection result. The information regarding the detection result output by the output means is obtained by the first detection means at a position in the visible light image corresponding to a position where the face is not detected by the second detection means in the infrared light image. The image processing apparatus according to claim 4, wherein the information indicates the number of detected faces. The image processing apparatus according to claim 4, wherein the output means causes the display means to display information regarding the detection result. In the visible light image obtained by taking a picture using visible light, the first detection step of detecting the person's face by using the image feature of the person's face and
A second detection step of detecting a person's face using the image features in an infrared light image obtained by photographing an imaging range corresponding to the imaging range of the visible light image using infrared light. When,
When the face is not detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the face is not detected in the visible light image. A predetermined image process is executed on the face area detected in the first detection step, and the image processing is performed.
When the face is detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the first in the visible light image. 1 An image processing step in which the predetermined image processing is not performed on the face region detected in the detection step, and
An image processing method characterized by having. In the image processing step
When the face is not detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the face is not detected in the visible light image. A mask process for masking the face area detected in the first detection step is executed as the predetermined image process.
When the face is detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the first in the visible light image. 1. The image processing method according to claim 7, wherein the mask processing is not performed on the face region detected in the detection step. In the image processing step
When the face is not detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the face is not detected in the visible light image. An image quality enhancement process for increasing the image quality of the face region detected in the first detection step is executed as the predetermined image process.
When the face is detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the first in the visible light image. 1. The image processing method according to claim 7, wherein the image quality improving process is not executed on the face region detected in the detection step. An image processing method that executes detection processing to detect a person's face.
In the visible light image obtained by taking a picture using visible light, the first detection step of detecting the face of the person using the image feature of the face of the person and
A second detection step of detecting a person's face using the image features in an infrared light image obtained by photographing an imaging range corresponding to the imaging range of the visible light image using infrared light. When,
It has an output process that outputs information about the detection result of the detection process.
When the face is not detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the face is not detected in the visible light image. The face detected in the first detection step is included in the detection result, and is included in the detection result.
When the face is detected in the second detection step at the position in the infrared light image corresponding to the position where the face is detected in the first detection step in the visible light image, the first in the visible light image. (1) An image processing method characterized in that the face detected in the detection step is not included in the detection result. The information regarding the detection result output in the output step is obtained in the first detection step at the position in the visible light image corresponding to the position where the face was not detected in the second detection step in the infrared light image. The image processing method according to claim 10, wherein the information indicates the number of detected faces. The image processing method according to claim 10 or 11, wherein in the output step, information on the detection result is displayed on a display means. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 6.

Images