JP6789601B2 - A learning video selection device, program, and method for selecting a captured video masking a predetermined image area as a learning video. - Google Patents

Description

The present invention relates to a technique for collecting captured images suitable as learning images when constructing a learning model.

Conventionally, there is a technique of recognizing a person or an object from a captured image by using a learning model based on a deep learning technique. Here, there is a problem that it is not preferable that the learning video used for constructing the learning model includes a face image that can identify an individual and various privacy images. For this reason, it is common to use a photographed video that can be published and does not include a face image or a privacy image as the learning video.

For example, there is a technique that enables a photographed image obtained by processing a face image to be searched according to a searcher (see, for example, Patent Document 1). According to this technique, after the identity verification of the searcher is executed, the face image of the person for whom the searcher has obtained permission in advance is registered as the search key. Then, the face image other than the registered person is subjected to privacy protection processing.

Further, there is also a technique of considering a privacy image by photographing the subject with infrared rays (see, for example, Patent Document 2). According to this technique, the photographed data obtained by photographing the subject by infrared rays and the external shape data of the subject stored in advance are compared. The presentation information is associated with the external shape data, and the presentation information of the external shape data that matches the shooting data is output.

Further, there is also a technique for editing a photographed image to be disclosed to an unspecified number of third parties in order to protect privacy and portrait rights without impairing the image quality (see, for example, Patent Document 3). According to this technique, a specific subject is extracted from a moving image stream, and an image of the subject is masked. At this time, it is determined whether or not to apply the mask processing to the specific subject according to the output condition based on the resolution of the moving image stream.

FIG. 1 is a system configuration diagram having a behavior analysis device.

According to the system of FIG. 1, the behavior analysis device analyzes the behavior of a person reflected in the image captured by the camera, and functions as a server by connecting to the Internet. The behavior analysis device has, for example, a behavior estimation engine, and this learning model is constructed by learning images accumulated in a learning image storage unit. The learning video is an association between a captured video in which a person's behavior is reflected and the action target.

The behavior estimation engine may be, for example, ActivityNet using deep learning technology (see, for example, Non-Patent Document 1). According to this technology, a learning model created from a learning video that reflects a wide variety of human behaviors (for example, "walking", "speaking", and "holding") is used to "behavior" of the person reflected in the captured video. Can be analyzed.
Further, the behavior estimation engine may be, for example, Two-stream ConvNets (see, for example, Non-Patent Document 2). According to this technology, using CNN (Spatial stream ConvNet) in the spatial direction and CNN (Temporal stream ConvNet) in the time series direction, the characteristics of the appearance of objects and backgrounds in the image and the horizontal component of the optical flow By extracting both the movement characteristics in the series of vertical components, the behavior is recognized with high accuracy.

According to the system of FIG. 1, each terminal is equipped with a camera, and a photographed image of a person's behavior is transmitted to the behavior analysis device 1. The terminal is a smartphone or mobile terminal owned by each user and connects to an access network such as a mobile phone network or a wireless LAN.
Of course, the terminal is not limited to a smartphone or the like, and may be, for example, a Web camera installed in the house. Further, the video data captured by the Web camera may be recorded on the SD card, and the recorded video data may be input to the behavior analysis device 1.

In actual operation, for example, a user who participated in a monitor test is asked to take a picture of his / her behavior with the camera of his / her smartphone. The smartphone transmits the video to the behavior analyzer. The behavior analysis device estimates the user's behavior from the video, and uses the estimation result in various applications.

Japanese Unexamined Patent Publication No. 2014-89625 Japanese Unexamined Patent Publication No. 2016-169990 Japanese Unexamined Patent Publication No. 2014-42234

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem and Juan Carlos Niebles, “ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding,” CVPR2015., [Online], [Search October 19, 2017], Internet <URL : http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.pdf> Karen Simonyan and Andrew Zisserman, “Two-Stream Convolutional Networks for Action Recognition in Videos,” in NIPS 2014, [online], [Search October 19, 2017], Internet <URL: https://arxiv.org/ abs / 1406.2199> FaceNet: A Unified Embedding for Face Recognition and Clustering, [online], [Search on October 19, 2017], Internet <URL: https://arxiv.org/abs/1503.03832> "Let's judge" regulars "from face images using AI! , [Online], [Search on October 19, 2017], Internet <URL: https://future-architect.github.io/articles/20170526/> "How far can you tell !? Digital camera face recognition confrontation", [online], [Search on October 19, 2017], Internet <URL: http://news.mynavi.jp/articles/2007/08/07/face/ 001.html ＞ "World-recognized NEC face recognition technology", [online], [Search on October 19, 2017], Internet <URL: http://jpn.nec.com/rd/research/DataAcquition/face.html> OpenPose, [online], [Searched October 19, 2017], Internet <URL: https://github.com/CMU-Perceptual-Computing-Lab/openpose> "I tried OpenPose, which can detect bones from videos and photos", [online], [Searched on October 19, 2017], Internet <URL: http://hackist.jp/?p=8285> "OpenPose has been upgraded so that you can try 3d pose estimation", [online], [Search on October 19, 2017], Internet <URL: http://izm-11.hatenablog.com / entry / 2017/08/01/140945 ＞

It is preferable that the behavior analyzer uses a large amount of captured images from which the privacy images have been removed according to the above-mentioned Patent Documents 1 to 3 as appropriate learning images for constructing the learning model.
However, even though the context (for example, a person or an object) in the learning model could be estimated from the initially captured image, it is often impossible to estimate the context by removing the privacy image from the captured image. When such a captured image is used as a learning image, the recognition accuracy of the learning model based on the learning image is lowered.

Further, in order to improve the recognition accuracy of the learning model, a large amount of captured images are required, but collecting only the captured images that have cleared the privacy problem requires cost and technical labor.

Therefore, an object of the present invention is to provide a learning image selection device, a program, and a method capable of selecting whether or not a photographed image masking a predetermined image area can be used as a learning image.

According to the present invention, it is a learning image selection device that inputs a captured image and selects a learning image that makes it possible to recognize the first context.
A first context recognition means for determining whether or not the first context can be recognized for the captured image, and
It has a captured image masking means for masking a predetermined image area for a captured image determined to be true by the first context recognition means.
The first context recognition means recursively inputs the masked shot video, determines whether or not the first context can be recognized, and selects only the shot video determined to be true as the learning video. It is a feature.

According to another embodiment of the learning video selection device of the present invention.
It is also preferable that the first context recognition means sequentially estimates a person's action target.

According to the present invention, it is a learning image selection device that inputs a captured image and selects a learning image that makes it possible to recognize the first context.
A first context recognition means for determining whether or not the first context can be recognized for the captured image, and
With respect to the photographed image determined to be true by the first context recognition means, the photographed image masking means for masking a predetermined image area and the photographed image masking means.
It has a second context recognition means for determining whether or not the second context can be recognized for the masked shot image.
The first context recognizing means recursively inputs the masked captured image determined to be true by the second context recognizing means, determines whether or not the first context can be recognized, and determines that it is true. It is characterized in that only the captured image is selected as the learning image.

According to another embodiment of the learning video selection device of the present invention.
The first context recognition means is to sequentially estimate a person's action target.
It is also preferable that the second context recognition means sequentially estimates the joint region of a person and / or sequentially estimates an object.

According to another embodiment of the learning video selection device of the present invention.
The captured image masking means represents an image area to be masked by a rectangular area, and from each of the outer frame sides of the rectangular area toward the outer frame side of the captured image, on the unmasked upper side, lower side, left side, and right side. It is also preferable to output each of the divided captured images.

According to another embodiment of the learning video selection device of the present invention.
The first context recognition means recursively inputs the photographed image determined to be false into the photographed image masking means.
It is also preferable that the captured image masking means narrows the image area to be masked under predetermined conditions.

According to another embodiment of the learning video selection device of the present invention.
It is also preferable that the captured image masking means masks an image area based on face detection from the captured image.

According to another embodiment of the learning video selection device of the present invention.
The captured image masking means stores a privacy image in advance, and it is also preferable to mask an image region similar to the privacy image from the captured image under predetermined conditions or more.

According to the present invention, it is a learning image selection program for operating a computer mounted on a device for inputting a captured image and selecting a learning image capable of recognizing a first context.
A first context recognition means for determining whether or not the first context can be recognized for the captured image, and
A computer is made to function as a photographed image masking means for masking a predetermined image area for a photographed image determined to be true by the first context recognition means.
The first context recognizing means recursively inputs the masked captured image, determines whether or not the first context can be recognized, and selects only the captured image determined to be true as the learning image. It is characterized by having a computer function.

According to the present invention, it is a learning image selection program for operating a computer mounted on a device for inputting a captured image and selecting a learning image capable of recognizing a first context.
A first context recognition means for determining whether or not the first context can be recognized for the captured image, and
With respect to the photographed image determined to be true by the first context recognition means, the photographed image masking means for masking a predetermined image area and the photographed image masking means.
A computer is made to function as a second context recognition means for determining whether or not the second context can be recognized for the masked shot image.
The first context recognizing means recursively inputs the masked captured image determined to be true by the second context recognizing means, determines whether or not the first context can be recognized, and determines that it is true. It is characterized in that the computer functions to select only the captured video as the learning video.

According to the present invention, there is a learning image selection method of a device for inputting a captured image and selecting a learning image that makes it possible to recognize the first context.
The device is
With respect to the captured image, the first step of determining whether or not the first context can be recognized, and
The second step of masking a predetermined image area for the captured image determined to be true by the first step, and
The masked captured image in the second step is recursively determined whether or not the first context can be recognized, and only the captured image determined to be true is selected as the learning image. It is characterized by doing.

According to the present invention, there is a learning image selection method of a device for inputting a captured image and selecting a learning image that makes it possible to recognize the first context.
The device is
With respect to the captured image, the first step of determining whether or not the first context can be recognized, and
The second step of masking a predetermined image area for the captured image determined to be true by the first step, and
The third step of determining whether or not the second context can be recognized for the masked shot image, and
A fourth step, which recursively determines whether or not the first context can be recognized for the masked shot image determined to be true by the third step, and selects only the captured image determined to be true as the learning image. It is characterized by performing the steps of.

According to the learning video selection device, program, and method of the present invention, it is possible to select whether or not a captured video masking a predetermined image area can be used as a learning video.

It is a system block diagram which has a behavior analysis apparatus. It is a 1st functional block diagram of the learning image selection apparatus in this invention. It is explanatory drawing which shows the processing of each function of FIG. 1 with respect to the captured image. It is a 2nd functional block diagram of the learning image selection apparatus in this invention. It is explanatory drawing which shows the context recognized by the 2nd context recognition part. It is explanatory drawing which shows the photographed image which the mask area was narrowed by the photographed image mask part. It is explanatory drawing which shows the plurality of photographed images divided by the mask area by the photographed image mask part.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 2 is a first functional configuration diagram of the learning video selection device according to the present invention.
FIG. 3 is an explanatory diagram showing processing of each function of FIG. 1 with respect to the captured image.

The behavior analysis device of FIG. 1 described above generates a learning model from a large amount of learning videos in advance on the premise of estimating human behavior from the captured video.
Here, the behavior analysis device of the present invention has a learning video selection function (device) that selects a large amount of learning video suitable for the learning video. The learning video selection function inputs the captured video from the captured video storage unit 101 and outputs the selected learning video to the learning video storage unit 102. As a result, the learning video storage unit 102 stores only the learning video selected by the learning video selection function.
Then, the existing behavior estimation engine builds a learning model from the learning video accumulated in the learning video storage unit 102.
It should be noted that the captured image storage unit 101 may acquire and store a large amount of captured images in which human behavior is reflected in advance from the terminal via the communication network.

According to FIG. 2, the learning video selection function of the behavior analysis device 1 includes a first context recognition unit 11 and a captured video mask unit 12. These functional components are realized by executing a program that makes the computer mounted on the device function. Further, the processing flow of these functional components can be understood as a learning video selection method of the device.

[First context recognition unit 11]
The first context recognition unit 11 determines whether or not the first context can be recognized with respect to the captured image. The captured image determined to be recognizable (true) is output to the captured image mask unit 12, and the captured image determined to be unrecognizable (false) is discarded.
Here, the first context recognition unit 11 is the same as the behavior estimation engine described above. Specifically, as described above, it may be ActivityNet or Two-stream ConvNets that sequentially estimate a person's action target. Further, the moving region may be extracted by an optical flow in which a portion where the same feature point is moving between frames is extracted and the movement of an object in the captured image is represented by a “vector”.

According to FIG. 3A, the captured image input to the first context recognition unit 11 is shown.
According to FIG. 3B, human behavior is estimated from the captured image of FIG. 3A. Here, the specific behavior of "folding the laundry" is presumed.

Further, the first context recognition unit 11 recursively inputs the masked photographed image from the photographed image mask unit 12 described later, and determines whether or not the first context can be recognized. Then, the captured image determined to be recognizable (true) is output to the learning image storage unit 102, and the captured image determined to be unrecognizable (false) is discarded. That is, by masking a predetermined image area of the captured image, if the first context cannot be recognized, the captured image is not used as a learning image.

[Captured image mask unit 12]
The captured image mask unit 12 masks a predetermined image area for the captured image determined to be recognizable (true) by the first context recognition unit 11.

Here, the predetermined image area may be a face area. Examples of face detection technology include Facenet of Google (registered trademark) (see, for example, Non-Patent Documents 3 and 4), face detection function of a digital camera (see, for example, Non-Patent Document 5), and face detection function of NEC (registered trademark). (See, for example, Non-Patent Document 6). For face detection, a generalized learning vector quantization method based on the minimum classification error is used, and a rectangular region matching the face is extracted by searching the rectangular region in order from the edge of the captured image.

Of course, the image area to be masked by the present invention is not limited to face detection. For example, it may be a privacy area such as a car license plate or a front cover.
The captured image mask unit 12 stores a predetermined image area to be masked in advance as a local feature amount of the reference image, and extracts an image area similar to the reference image. Specifically, algorithms such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features) can also be used. The image area extracted here is masked in the captured image.

The mask means that the area is filled with a predetermined color (for example, black), an opaque pattern, or the like. This masking process protects privacy, such as making it impossible to identify an individual.

According to FIG. 3C, the face region detected by the captured image mask unit 12 is shown.
According to FIG. 3D, the detected face region is masked in the captured image.

Then, the captured image mask unit 12 feeds back the masked photographed image to the first context recognition unit 11 (behavior estimation engine).

According to FIG. 2 described above, the learning video selection function includes a test video storage unit 103 that inputs a test video to the first context recognition unit 11, and an estimation result determination unit 14 that determines an estimation result for the test video. And further.
Here, as a data set, the captured video to which the action target is assigned in advance is classified into a learning video (learning data) and a test video (test data). For example, 90% of the captured video is selected as the learning video, and the remaining 10% of the captured video is assigned to the test video. The estimation result determination unit 14 compares the action estimation result for the test video with the action target assigned to the test video, and determines whether the test video is correct or not. By inputting a large number of test images, the recognition accuracy of the first context recognition unit 11 based on the learning model can be calculated.

FIG. 4 is a second functional configuration diagram of the learning video selection device according to the present invention.
FIG. 5 is an explanatory diagram showing a context recognized by the second context recognition unit.

According to FIG. 4, as compared with FIG. 2, the learning video selection device further includes a second context recognition unit 13.

[Second context recognition unit 13]
The second context recognition unit 13 determines whether or not the second context can be recognized with respect to the masked photographed image output from the photographed image mask unit 12. The second context recognition unit 13 recognizes a context different from that of the first context recognition unit 11. For example, it differs as follows.
First context recognition unit 11 = one that sequentially estimates "human behavior target" Second context recognition unit 13 = one that sequentially estimates "human joint area",
And / or those that sequentially estimate the "object"

<Estimation of human joint area>
The second context recognition unit 13 specifically extracts feature points of human joints using a skeleton model such as OpenPose (registered trademark) (see, for example, Non-Patent Documents 7 to 9).
OpenPose is software that can detect multiple human body / hand / face key points in real time from images, and is published by GitHub. For example, 15 key points can be detected for the entire human body shown in the captured image.
According to FIG. 5A, the human joint region is estimated from the captured image.

<Estimation of object area>
Specifically, the second context recognition unit 13 can estimate the object to be reflected in the captured image by using a neural network such as CNN (Convolutional Neural Network).
According to FIG. 5B, the object is estimated from the captured image. Specifically, the "towel" is recognized as an object.

Then, the second context recognition unit 13 feeds back the recognition result (recognizable / unrecognizable) of the second context to the first context recognition unit 11 with respect to the masked captured image.

According to FIG. 4, the masked captured image determined by the first context recognition unit 11 to be true (recognizable in the second context) by the second context recognition unit 13 is recursively the first. Determine if the context is recognizable. Then, only the captured video determined to be recognizable (true) in the first context is output as a learning video to the learning video storage unit 102.

<Recursive repetition of context recognition for masked footage>
FIG. 6 is an explanatory diagram showing a captured image in which the mask area is narrowed by the captured image mask portion.
Regarding the above-described embodiments of FIGS. 2 and 4, when the first context recognition unit 11 determines that the masked shot image input from the shot image mask unit 12 is false (the first context cannot be recognized), The masked photographed image may be recursively output to the photographed image mask unit 12.
Here, the captured image mask unit 12 narrows the image area to be masked under predetermined conditions. Specifically, the predetermined condition narrows the rectangular range of the image area to be masked by a predetermined ratio. That is, the area where the context should be recognized is expanded. Even if the image area of the mask is narrowed, it is necessary to protect the privacy, such as making it impossible to identify an individual.
In the case of FIG. 2, the masked photographed image whose mask area is narrowed by the photographed image mask unit 12 is recursively input to the first context recognition unit 11.

<Mask method of another embodiment in the captured image mask unit 12>
FIG. 7 is an explanatory diagram showing a plurality of captured images divided by a masked area by the captured image mask unit.
According to the embodiment of FIG. 3D described above, the predetermined image area from the captured image is simply masked as, for example, blackened.
Here, according to FIG. 7, the captured image mask unit 12 represents an image area to be masked by a rectangular area, and is masked from each outer frame side of the rectangular area toward the outer frame side of the captured image. Outputs each shot image divided into upper side, lower side, left side and right side. Each of these captured images is input to the first context recognition unit 11 in the case of FIG. 2 and to the second context recognition unit 13 in the case of FIG. As a result, only the masked photographed video whose context is recognized by the first context recognition unit 11 is stored in the learning video storage unit 102.

As described in detail above, according to the learning video selection device, program, and method of the present invention, it is possible to mask a predetermined image area from the captured video and determine whether or not the captured video can be used as the learning video. You can choose.
In particular, according to the present invention, even if the face region and the privacy region are removed from the captured image, the recognition of the context originally recognized by the initially captured image can be maintained. In particular, for captured images based on labeled action objects that can be recognized for deep learning, the context does not become unrecognizable, and re-learning is possible to improve recognition accuracy.

With respect to the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Behavior analysis device 101 Captured video storage unit 102 Learning video storage unit 103 Test video storage unit 11 First context recognition unit 12 Captured video mask unit 13 Second context recognition unit 14 Estimated result judgment unit

Claims (12)

It is a learning image selection device that inputs a captured image and selects a learning image that makes it possible to recognize the first context.
With respect to the captured image, a first context recognition means for determining whether or not the first context can be recognized, and
It has a captured image masking means for masking a predetermined image area for a captured image determined to be true by the first context recognition means.
The first context recognition means recursively inputs the masked shot video, determines whether or not the first context can be recognized, and selects only the shot video determined to be true as the learning video. A featured learning video selection device. The learning video selection device according to claim 1, wherein the first context recognition means sequentially estimates a person's action target. It is a learning image selection device that inputs a captured image and selects a learning image that makes it possible to recognize the first context.
With respect to the captured image, a first context recognition means for determining whether or not the first context can be recognized, and
With respect to the photographed image determined to be true by the first context recognition means, the photographed image masking means for masking a predetermined image area and the photographed image masking means.
It has a second context recognition means for determining whether or not the second context can be recognized for the masked shot image.
The first context recognizing means recursively inputs the masked captured image determined to be true by the second context recognizing means, determines whether or not the first context can be recognized, and determines that it is true. A learning video selection device characterized in that only captured video is selected as a learning video. The first context recognition means is to sequentially estimate a person's action target.
The learning image selection device according to claim 3, wherein the second context recognition means sequentially estimates a human joint region and / or sequentially estimates an object. The captured image masking means represents an image area to be masked by a rectangular area, and the unmasked upper side, lower side, left side, and right side are directed from each outer frame side of the rectangular area toward the outer frame side of the captured image. The learning video selection device according to any one of claims 1 to 4, wherein each shot video classified into the above is output. The first context recognition means recursively inputs the photographed image determined to be false into the photographed image masking means.
The learning video selection device according to any one of claims 1 to 5, wherein the captured image masking means narrows the image area to be masked under a predetermined condition. The learning video selection device according to any one of claims 1 to 6, wherein the captured video masking means masks an image region based on face detection from the captured video. Any one of claims 1 to 7, wherein the captured image masking means stores a privacy image in advance and masks an image region similar to the captured image under predetermined conditions or more from the captured image. The learning video selection device described in. It is a learning video selection program that operates a computer mounted on a device that inputs a shot video and selects a learning video that makes it possible to recognize the first context.
With respect to the captured image, a first context recognition means for determining whether or not the first context can be recognized, and
A computer is made to function as a captured image masking means for masking a predetermined image area for a captured image determined to be true by the first context recognition means.
The first context recognition means recursively inputs the masked shot video, determines whether or not the first context can be recognized, and selects only the shot video determined to be true as the learning video. A learning video selection program characterized by having a computer function. It is a learning video selection program that operates a computer mounted on a device that inputs a shot video and selects a learning video that makes it possible to recognize the first context.
With respect to the captured image, a first context recognition means for determining whether or not the first context can be recognized, and
With respect to the photographed image determined to be true by the first context recognition means, the photographed image masking means for masking a predetermined image area and the photographed image masking means.
A computer is made to function as a second context recognition means for determining whether or not the second context can be recognized for the masked shot image.
The first context recognizing means recursively inputs the masked captured image determined to be true by the second context recognizing means, determines whether or not the first context can be recognized, and determines that it is true. A learning video selection program characterized by having a computer function to select only captured video as a learning video. It is a learning image selection method of a device that inputs a captured image and selects a learning image that makes it possible to recognize the first context.
The device
With respect to the captured image, the first step of determining whether or not the first context can be recognized, and
The second step of masking a predetermined image area for the captured image determined to be true by the first step, and
The masked captured image in the second step is recursively determined whether or not the first context can be recognized, and only the captured image determined to be true is selected as the learning image. A learning video selection method for a device characterized by It is a learning image selection method of a device that inputs a captured image and selects a learning image that makes it possible to recognize the first context.
The device
With respect to the captured image, the first step of determining whether or not the first context can be recognized, and
The second step of masking a predetermined image area for the captured image determined to be true by the first step, and
The third step of determining whether or not the second context can be recognized for the masked shot image, and
A fourth step, which recursively determines whether or not the first context can be recognized for the masked shot image determined to be true by the third step, and selects only the captured image determined to be true as the learning image. A learning video selection method for a device characterized by performing the steps of.

Images