JP2021196755A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

To provide an appropriate image to be used for analysis, for improving accuracy of image analysis.SOLUTION: An image processing apparatus 10 configured to process an image to be used for analysis of whether an image captured by a monitoring camera includes a desired subject or not includes a conversion section 132 which converts a background and/or a subject in the image to be used for analysis. The conversion section 132 converts, when converting a background of an image, the background of the image to a background captured by the monitoring camera or a background of the same kind as the background captured by the monitoring camera, or converts, when converting a subject of the image, a property of the subject to a property that the subject is likely to have in an area captured by the monitoring camera.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Conventionally, in image analysis, a part in which an object (for example, a person) appears is cut out from an image to be analyzed, a feature amount of the cut out part (cut out image) is extracted, and a cutout part is cut out based on the extracted feature amount. There is a technology to analyze. It has also been proposed to perform image analysis using a model composed of a deep neural network. In training this model, a public data set containing a large number of images is often used as training data (see Non-Patent Document 1).

Shohei Hido, Yukino Baba et al., “Data Scientist Training Reader: Introduction to Machine Learning”, Gijutsu-Hyoronsha, September 2015, p. 15

However, even if the model is trained using a large number of images in the public data set, the model may not satisfy the desired image analysis accuracy.

The present invention has been made in view of the above, and is an image processing apparatus, an image processing method, and an image processing program capable of providing an appropriate image used for analysis in order to improve the accuracy of image analysis. The purpose is to provide.

In order to solve the above-mentioned problems and achieve the object, the image processing apparatus of the present invention processes an image used for analyzing whether or not a desired subject is captured in an image captured by a surveillance camera. An image processing device that converts the background and / or subject of an image used for analysis, and the conversion unit uses a surveillance camera to convert the background of the image when converting the background of the image. When converting to a background that is being shot or a background that is similar to the background that is being shot by a surveillance camera and converting the subject of the image, the nature of the subject is likely to be possessed by the subject in the area shot by the surveillance camera. It is characterized by converting into a property.

Further, the image processing method of the present invention is an image processing method executed by an image processing apparatus that processes an image used for analyzing whether or not a desired subject is captured in an image captured by a surveillance camera. Including a conversion step of converting the background and / or subject of the image used for analysis, the conversion step includes, when converting the background of the image, the background of the image, the background taken by the surveillance camera, or. , When converting to a background of the same type as the background shot by the surveillance camera and converting the subject of the image, it is characterized by converting the properties of the subject to the properties that the subject is likely to have in the area shot by the surveillance camera. And.

Further, the image processing program of the present invention provides a computer with a conversion step of converting an image background and / or a subject used for analyzing whether or not a desired subject is captured in an image captured by a surveillance camera. When converting the background of the image, the conversion step is executed to convert the background of the image to the background taken by the surveillance camera or the same kind of background as the background taken by the surveillance camera, and the subject of the image is converted. When converting, the property of the subject is converted into a property that the subject is likely to have in the area photographed by the surveillance camera.

According to the present invention, it is possible to provide an appropriate image used for analysis in order to improve the accuracy of image analysis.

FIG. 1 is a block diagram showing an example of the configuration of the analysis system according to the embodiment. FIG. 2 is a diagram illustrating the content of analysis processing by the analysis device. FIG. 3 is a block diagram showing an example of the configuration of the image processing device. FIG. 4 is a diagram illustrating the processing content of the conversion unit. FIG. 5 is a diagram illustrating the processing content of the conversion unit. FIG. 6 is a block diagram showing an example of the configuration of the learning device. FIG. 7 is a block diagram showing an example of the configuration of the analysis device. FIG. 8 is a flowchart showing a processing procedure of image processing according to the first embodiment. FIG. 9 is a diagram illustrating an outline of the learning device of the second embodiment. FIG. 10 is a diagram showing a configuration example of the learning device of the second embodiment. FIG. 11 is a diagram showing an example of automatically assigning the coordinates of the sub-object in the second embodiment. FIG. 12 is a diagram showing an example of a method for creating a removed image in the second embodiment. FIG. 13 is a diagram showing an example of a region picked up by the local branch of the second embodiment. FIG. 14 is a block diagram showing an example of the configuration of the analysis device according to the second embodiment. FIG. 15 is a diagram showing an example of a processing procedure of the learning device of the second embodiment. FIG. 16 is a diagram showing an example of a processing procedure of the analysis device of the second embodiment. FIG. 17 is a diagram showing an example of an analysis result by a deep neural network learned by the learning device of the second embodiment. FIG. 18 is a diagram showing an example of a computer in which an image processing device, a learning device, and an analysis device are realized by executing a program.

Hereinafter, embodiments of an image processing apparatus, an image processing method, and an image processing program according to the present application will be described in detail with reference to the drawings. Further, the present invention is not limited to the embodiments described below.

[Embodiment 1]
First, the first embodiment will be described. The present embodiment relates to an analysis system that performs image analysis using a model configured by a deep neural network. The model extracts the feature amount in the image in which the desired subject or the candidate of the desired subject is captured, and uses the extracted feature amount to estimate the attribute to which the subject or the candidate of the subject belongs in the image, or to estimate the subject or the subject. This is a model that collates the candidate with the detection target. Further, in the first embodiment, the image to be analyzed is an image taken by a surveillance camera.

[Analysis system configuration]
First, the configuration of the analysis system according to the first embodiment will be described. FIG. 1 is a block diagram showing an example of the configuration of the analysis system according to the first embodiment.

As shown in FIG. 1, the analysis system 1 according to the first embodiment is an analysis device 30 that performs image analysis using a model configured by a deep neural network, and a learning device that executes learning of the model of the analysis device 30. Has 20 and. The analysis system 1 has an image processing device 10 that processes an image to be learned in front of the learning device 20.

FIG. 2 is a diagram illustrating the content of analysis processing by the analysis device 30. The model used by the analysis device 30 includes a feature extraction module that extracts the feature amount (feature amount vector) of the image x to be analyzed, and a feature amount extracted by the feature extraction module, to which the subject in the cut-out image belongs. It has an analysis module that estimates attributes and collates the subject with the subject to be detected. The cut-out image is an image obtained by cutting out a portion including a subject from the original image.

Specifically, in the analysis device 30, the feature extraction module in the model performs a feature amount extraction step of extracting the feature amount (step S1 in FIG. 2). Subsequently, the model uses the feature amount extracted by the feature extraction module to perform an attribute estimation step (step S2 in FIG. 2) for estimating the attribute to which the object in the image belongs, or a collation for matching the object with the object to be detected. A step (step S3 in FIG. 2) is performed, and the analysis result is output. Attributes include the gender and age of the person, as well as the skeleton and gait. Further, the attribute is not limited to humans, and may be a type of animal other than humans, or may be an object such as a vehicle or a robot.

In the example of FIG. 2, the model estimates that the attribute of the person in the input image x is "male". Further, the model collates the input person of the image x with the person to be detected, and analyzes that the person of the image x and the person to be detected are "others".

The learning device 20 learns a model using image data for learning. The learning device 20 extracts a feature amount of an image and analyzes whether or not a desired subject is captured in the image based on the extracted feature amount, and executes learning of a model configured by a deep neural network.

Then, the image processing device 10 processes an image used for analyzing whether or not a desired subject is captured. The image processing device 10 provides the learning device 20 with image data for learning (learning data) used for learning the model. The image processing device executes predetermined image processing on the image used for learning, and outputs the image after the image processing to the learning device 20. The image processing device 10 may provide the analysis device 30 with image data for analysis (image for analysis) to be analyzed.

Specifically, the image processing device 10 generates a learning image used by the learning device 20 for learning the model. The image processing device 10 acquires cut-out images of a publicly available image data set, and extracts an image having a desired attribute from these cut-out images. For example, the image processing apparatus 10 extracts a background of an image captured by a surveillance camera that captures an image to be analyzed, or an image having a background similar to the background captured by the surveillance camera. Further, for example, the image processing device 10 extracts an image including a subject having a property that the subject tends to have in a region photographed by a surveillance camera that captures an image to be analyzed. In the image data set, object information including object attributes and object identification information is added to each image.

Then, when the number of extracted images does not reach the target number, the image processing device 10 determines the properties of the background or the subject of the image other than the desired attribute among the cut out images of the published image data set. The conversion produces an image with the desired attributes. The target number of sheets is set, for example, according to the analysis accuracy of the model to be analyzed.

Specifically, when the background of the image is converted, the image processing device 10 converts the background of the image into a background taken by the surveillance camera or a background of the same type as the background taken by the surveillance camera. When converting the subject of an image, the image processing device 10 converts the property of the subject into a property that the subject is likely to have in the area photographed by the surveillance camera. The image processing device 10 outputs an extracted or converted image having a desired attribute to the learning device 20 or the analysis device 30 as learning data.

As described above, in the first embodiment, in the image processing device 10 in the previous stage of the learning device 20, an image including a desired background or a subject having a desired property is obtained from the cut-out image of the image data set. Extract or generate. Therefore, in the first embodiment, the nature of the background or the subject of each image of the learning data is unified, the elements to be estimated at the time of machine learning are reduced, the subject to be originally estimated is appropriately learned, and the analysis accuracy is improved. To improve.

[Image processing device]
Next, the configuration of the image processing device 10 will be described. FIG. 3 is a block diagram showing an example of the configuration of the image processing device 10. As shown in FIG. 3, the image processing device 10 has an input / output unit 11, a storage unit 12, and a control unit 13.

The input / output unit 11 accepts the input of information and outputs the information. The input / output unit 11 is a communication interface for transmitting / receiving various information to / from another device connected via a network or the like. The input / output unit 11 communicates between another device (for example, the learning device 20 or the analysis device 30) and the control unit 13 (described later) via a telecommunication line such as a LAN (Local Area Network) or the Internet. .. Further, the input / output unit 11 is a device device such as a mouse or a keyboard that accepts input of various instruction information to the image processing device 10 in response to an input operation by the user. The input / output unit 11 is realized by, for example, a liquid crystal display, and a screen whose display is controlled by the image processing device 10 is displayed and output.

The storage unit 12 is realized by semiconductor memory elements such as RAM (Random Access Memory) and flash memory (Flash Memory), and a processing program for operating the image processing device 10 and data used during execution of the processing program can be stored. It will be remembered. The storage unit 12 has image data 121, conversion data 122, extracted image 123, and converted image 124.

The image data 121 is, for example, a data set of published images. The image data 121 may be a data set of a plurality of images.

The conversion data 122 is data required for converting the background of an image into a desired background or a background of the same type as the desired background in the conversion process by the conversion unit 132 (described later), and is a desired background or a desired background. It is image data etc. that shows the same kind of background as. The conversion data 122 is data required for converting the property of the subject of the image into the property that the subject is likely to have in the area photographed by the surveillance camera in the conversion process of the conversion unit 132. It is image data or the like in which a subject having a property that the subject tends to have in the area to be photographed is captured.

The extracted image 123 is an image extracted from the image data 121 by an extraction process by the extraction unit 131 (described later). The extracted image 123 is an image having a desired attribute. For example, the extracted image 123 is an image having a background of an image captured by a surveillance camera or a background of the same type as the background captured by the surveillance camera. Alternatively, the extracted image 123 is an image including a subject having a property that the subject tends to have in the area photographed by the surveillance camera.

The converted image 124 is an image generated by conversion by the conversion unit 132. The converted image is an image in which the background is converted into the background of the image captured by the surveillance camera or the background of the same type as the background captured by the surveillance camera. Alternatively, the converted image 124 is an image obtained by converting the properties of the subject into properties that the subject tends to have in the area photographed by the surveillance camera.

The control unit 13 controls the entire image processing device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Further, the control unit 13 has an internal memory for storing programs and control data that specify various processing procedures, and executes each process using the internal memory. Further, the control unit 13 functions as various processing units by operating various programs. The control unit 13 has an extraction unit 131 and a conversion unit 132.

The extraction unit 131 extracts an image having a desired attribute from the image data 121. The extraction unit 131 extracts the background of the image captured by the surveillance camera that captures the image to be analyzed, or the image having the same type of background as the background captured by the surveillance camera. Extract as.

For example, the background of the image is an escalator, an elevator, or the like. When a surveillance camera shoots an escalator or elevator, the posture of the subject is almost always upright. When the area where the surveillance camera is imaged is an escalator or an elevator, the extraction unit 131 uses skeleton estimation, an attention method for picking up an area where a predetermined background is captured, and the like, and the background is an escalator or an elevator and is upright. The image of the subject in the state is extracted.

The nature of the subject is the appearance property of the subject, specifically, the posture, clothes, facial expressions, hairstyle, or belongings. The image processing device 10 extracts an image including a subject having a property that the subject tends to have in the area photographed by the surveillance camera that captures the image to be analyzed. For example, when a surveillance camera is installed in a wedding hall, the subject's dress is formal wear. When the area where the surveillance camera is imaged is the ceremony hall, the extraction unit 131 extracts the image of the subject wearing the formal wear.

The conversion unit 132 converts the background and / or subject of the image used for analysis. When the number of the extracted images 123 extracted by the extraction unit 131 does not reach the target number, the conversion unit 132 generates the converted image 124 in which the background and / or the subject of the image is converted.

When converting the background of the image, the conversion unit 132 converts the background of the image into a background taken by the surveillance camera or a background of the same type as the background taken by the surveillance camera. FIG. 4 is a diagram illustrating the processing of the conversion unit 132.

For example, when a surveillance camera shoots an escalator, the background of the image is the escalator. In this case, the posture of the subject is in an upright state. Therefore, as shown in FIG. 4, the conversion unit 132 acquires an image G1 in which the subject H1 in an upright state is captured from the image data 121, although the background is not an escalator. Then, the conversion unit 132 generates an image G1 ′ in which the background of the image G1 is converted into a background in which the escalator B1 appears. If the conversion data does not include a background in which the same escalator as the escalator captured by the surveillance camera appears, the conversion unit 132 may convert the background using the background of a similar escalator.

Then, when converting the subject of the image, the conversion unit 132 converts the property of the subject into a property that the subject is likely to have in the area photographed by the surveillance camera. The nature of the subject is the appearance property of the subject, specifically, the posture, clothes, facial expressions, hairstyle, or belongings. FIG. 5 is a diagram illustrating the processing of the conversion unit 132.

For example, when a surveillance camera is installed in a wedding hall, the subject's dress is formal wear. Therefore, as shown in FIG. 5, the conversion unit 132 acquires the image G2 in which the subject H2, which is a plain dress, is captured from the image data 121. Then, the conversion unit 132 generates an image G2'that is obtained by converting the clothes of the subject H2 from plain dress to formal wear. Further, for example, the conversion unit 132 may acquire an image in which a subject with a real face is captured from the image data 121, and generate an image in which the face of the subject is converted from the real face to a face with makeup applied.

When the conversion unit 132 converts the object attribute of the image and the object identification information into another attribute or identification information, the conversion unit 132 changes the object information of the converted image 124 according to the conversion content. The image processing device 10 outputs the extracted image 123 and the converted image 124 to the learning device 20 as learning data.

[Learning device]
Next, the configuration of the learning device 20 will be described. FIG. 6 is a block diagram showing an example of the configuration of the learning device 20. As shown in FIG. 6, the learning device 20 has an input / output unit 21, a storage unit 22, and a control unit 23.

The input / output unit 21 has the same function as the input / output unit 11 shown in FIG. 3, and performs input / output of information and communication with other devices (for example, the image processing device 10 and the analysis device 30).

The storage unit 22 is realized by semiconductor memory elements such as RAM (Random Access Memory) and flash memory (Flash Memory), and stores a processing program for operating the learning device 20, data used during execution of the processing program, and the like. Will be done. In the storage unit 22, the image processing device 10 stores the extracted image 123 and the converted image 124 as learning data 221. The extracted image 123 is an image having desired attributes extracted from the image of the public data set by the image processing apparatus 10. Further, the converted image 124 is an image in which the properties of the background and / or the subject are converted to match the desired attributes. Further, the storage unit 22 has a model 222.

The model 222 is a model in which the feature amount of the image is extracted by the feature extraction module, and whether or not a desired subject is captured in the image is analyzed by the analysis module based on the extracted feature amount. Model 222 is composed of a deep neural network. The model 222 uses the extracted features to estimate the attributes to which the subject in the image belongs and to collate the subject with the subject to be detected. Various parameters of the model 222 are adjusted by learning using the learning data 221 by the learning unit 231 described later.

The control unit 23 has the same function as the control unit 13 shown in FIG. 3 and controls the entire learning device 20. The control unit 23 functions as various processing units by operating various programs. The control unit 23 has a learning unit 231.

The learning unit 231 learns image analysis of an image based on the feature amount extracted from the learning data 221 in the feature extraction module.

As described above, the learning device 20 is an image of the extracted image 123 or the image data 121 including a subject having a desired background or desired properties, which is previously extracted from the image data 121 to be learned by the image processing device 10. The learning is performed using the converted image 124 converted into a desired background and the converted image 124 obtained by converting the properties of the subject so as to have the properties of the desired subject. Since the extracted image 123 and the converted image 124 have the same characteristics of the background or the subject of each image, the learning device 20 reduces the elements to be estimated at the time of machine learning, and appropriately learns the subject to be estimated from the model. It is possible to improve the analysis accuracy of the model.

[Analyzer]
Next, the configuration of the analysis device 30 will be described. FIG. 7 is a block diagram showing an example of the configuration of the analysis device 30. As shown in FIG. 7, the analysis device 30 includes an input / output unit 31, a storage unit 32, and a control unit 33.

The input / output unit 31 has the same function as the input / output unit 11 shown in FIG. 3, and performs input / output of information and communication with other devices (for example, the image processing device 10 and the learning device 20).

The storage unit 32 is realized by semiconductor memory elements such as RAM (Random Access Memory) and flash memory (Flash Memory), and stores a processing program for operating the analysis device 30, data used during execution of the processing program, and the like. Will be done. The storage unit 32 has an analysis image 321 and a model 322, and an analysis result 323 showing a classification result of the object captured in the image or an estimation result of the attribute of the object captured in the image.

The control unit 33 has the same function as the control unit 13 shown in FIG. 3 and controls the entire analysis device 30. The control unit 33 functions as various processing units by operating various programs. The control unit 33 has an analysis unit 331.

Using the model 322, the analysis unit 331 uses the feature amount extraction process in the feature extraction module and the feature amount extracted by the feature extraction module to estimate the attributes to which the subject belongs in the analysis image and to estimate the subject and the detection target. Match with the subject.

As described above, since the analysis device 30 uses the model 322 which has improved the accuracy by learning the extracted image 123 and the converted image 124 in which the properties of the background or the subject of each image are unified at the time of analysis, the analysis is highly accurate. Can be executed.

[Image processing procedure]
Next, a processing procedure for image processing executed by the image processing device 10 will be described. FIG. 8 is a flowchart showing a processing procedure of image processing according to the first embodiment.

As shown in FIG. 8, the image processing apparatus 10 extracts an image having a desired attribute from the image data 121 (step S11). When the number of extracted images reaches the target number (step S12: Yes), the image processing device 10 outputs the extracted images 123 as learning data to the learning device 20 (step S15).

On the other hand, when the number of extracted images has not reached the target number (step S12: No), the image processing apparatus 10 extracts an image other than the desired attribute from the image data (step S13), and the extracted image is referred to. A conversion process for converting the background and / or the subject of the image is performed (step S14).

Then, when the extracted image and the converted image reach the target number (step S12: Yes), the image processing device 10 outputs the extracted image 123 and the converted image 124 to the learning device 20 as learning data (step). S15). Further, when the extracted image and the converted image do not reach the target number of images (step S12: No), the image processing apparatus 10 processes the extracted images and the converted images in steps S13 and S14 until the target number of images is reached. repeat.

[Effect of Embodiment 1]
The analysis accuracy when the present embodiment was applied to the learning image of the model 222 and the analysis accuracy when the present embodiment was not applied were evaluated. Table 1 shows the evaluation results. Rank-1 and mAP both take a value of 0 to 100%, and the higher the value, the better the collation accuracy.

In Table 1, the collation accuracy when all the images of the image data 121 are adopted and learned, and the extracted image 123 and the converted image 124 whose background is only the escalator among the image data 121 are adopted and learned. Indicates the collation accuracy. As shown in Table 1, the case where the learning is performed by adopting the extracted image 123 and the converted image 124 whose background is only the escalator among the image data 121 is the case where all the images of the image data 121 are adopted and learned. In comparison, both Rank-1 and mAP were the best.

In the first embodiment, an extracted image including a subject having a desired background or a desired property, which is previously extracted from the image data 121 to be learned by the image processing device 10 as learning data by the image processing device 10. Learning is performed using the converted image 124 in which the image of 123 or the image data 121 is converted into a desired background and the converted image 124 in which the properties of the subject are converted so as to have the properties of the desired subject. .. As described above, in the first embodiment, the extracted image 123 and the converted image 124 in which the properties of the background or the subject of the image are unified are used at the time of learning. Therefore, according to the first embodiment, the elements to be estimated at the time of machine learning are reduced, the subject to be originally estimated can be appropriately learned by the model, and the improvement of the analysis accuracy of the model can be improved.

[Embodiment 2]
Next, the second embodiment will be described. In the second embodiment, a case where an attention model is used as a model will be described. The attention model is to pick up multiple areas from the clipped image, extract the feature amount for each of the picked up areas, integrate each extracted feature amount, and estimate the attribute to which the subject in the image belongs. This is a model that collates the subject with the subject to be detected. In the second embodiment, model learning or analysis using the model is performed using the extracted image 123 and the converted image 124 processed by the image processing device 10.

[Overview]
First, the outline of the learning device 2 of the present embodiment will be described with reference to FIG. The target of learning here is assumed to be a deep neural network that performs image analysis. This deep neural network has a cutout module (not shown in FIG. 14) that cuts out the part where the object is reflected from the image to be analyzed, a feature amount extraction module that extracts the feature amount of the cut out part, and the extracted features. It shall be equipped with an analysis module that analyzes the cut out portion based on the quantity.

The feature amount extraction module is composed of a plurality of modules for extracting feature amounts from an image. This module is, for example, a local branch used in HA-CNN and the like. Hereinafter, the case where the module constituting the feature quantity extraction module is a local branch will be described as an example. This feature extraction module may include a global branch. Further, a case where the analysis module includes a classification module for classifying objects reflected in an image and an attribute estimation module for estimating the attributes of the object will be described as an example.

The learning device assigns a sub-object in charge of (to be picked up) to each local branch of the feature quantity extraction module. This sub-object is the object that makes up the object.

For example, when an object is a person, the sub-objects of the object are the upper body, the lower body, and the like. For example, the learning device allocates the upper body of the person as the sub-object in charge of the local branch shown by reference numeral 401 in FIG. 14, and the lower body of the person as the sub-object in charge of the local branch shown in reference numeral 402.

After that, the learning device learns the area to be picked up by each local branch of the feature quantity extraction module. For example, the learning device uses information indicating an area (area to be picked up by the local branch) in which a sub-object to be picked up by each local branch exists for each of the extracted image 123 and the converted image 124. Then, each local branch learns the area of the sub-object to be picked up.

For example, if there is an error between the area of the sub-object in charge of the local branch shown by reference numeral 401 in FIG. 14 and the area picked up by the local branch, the learning device causes the learning device to reduce the error. Adjust the parameter value of the local branch. If there is an error between the area of the sub-object in charge of the local branch indicated by reference numeral 402 and the area picked up by the local branch, the learning device adjusts the parameter value of the local branch so as to reduce the error. I do. By repeating such adjustments, each local branch can accurately pick up the area of the sub-object assigned to it. Such adjustment (learning) is called a direct reflection of the feature quantity extraction module for convenience of explanation. In addition, the learning device also learns (indirect reflection) which area should be picked up by each local branch in order to further improve the analysis accuracy by the analysis module.

In this way, when learning the feature quantity extraction module, the learning device is necessary for learning rather than learning only by the above indirect reflection by performing direct reflection in addition to the above indirect reflection. The number of images and the number of epochs can be significantly reduced.

[Learning device]
Next, a configuration example of the learning device will be described with reference to FIG. The learning device 220 includes an input / output unit 21, a storage unit 22, and a control unit 223.

The storage unit 22 stores the learning data 221 including the extracted image 123 and the converted image 124 input via the input / output unit 21, and the model 2222 of the deep neural network obtained by learning by the control unit 223. The model 2222 is information showing, for example, parameter values of various modules (allocation module, feature amount extraction module, analysis module) used in the above deep neural network. The information of this model is appropriately updated by the learning process by the control unit 223.

In the above-mentioned extracted image 123 and converted image 124, for example, information on the area where the sub-object exists in the image (that is, the area to be picked up by the local branch) is added to each image. Information (for example, coordinates) of the area where this sub-object exists may be given manually or automatically.

For example, when the learning device 220 determines that the feature quantity extraction module picks up two sub-objects, the upper body and the lower body of a person, and automatically adds information (for example, coordinates) of the area where these sub-objects exist in the image. think of.

In this case, for example, an image showing the whole body of a person (see reference numeral 601 in FIG. 16), an image showing the upper body (see reference numeral 602 in FIG. 16), and an image showing the lower body (see reference numeral 603 in FIG. 16). And be prepared in advance.

Then, the learning device 220 describes these images as "the upper half of the image is the upper half of the image and the lower half of the image is the lower half of the image" for the image showing the whole body of the person, and "for the image showing the upper half of the image". It is judged that "the whole image is the upper body and the lower body does not exist", and for the image in which the lower body is reflected, "the whole image is the lower body and the upper body does not exist". After that, based on the above determination result, the learning device 220 assigns the region where the upper body exists and the region where the lower body exists in each image, for example, with the coordinates of the four sides of the rectangular region. Then, the learning device 220 stores the image to which the coordinates of the area where each sub-object exists are given as a partial image in the storage unit 12. When preparing an image showing the upper body and an image showing the lower body, the learning device 220 may be prepared by dividing the image showing the whole body into two upper and lower parts as shown in FIG. ..

Returning to the description of FIG. The control unit 223 includes a sub-object allocation unit 2231 and a learning unit 2232.

The sub-object allocation unit 2231 allocates the sub-object in charge of the local branch for each local branch constituting the feature amount extraction module. That is, the sub-object allocation unit 2231 assigns, for each local branch, which sub-object among the sub-object groups constituting the object is picked up and the feature amount is extracted. Here, the number and types of sub-objects to be picked up in the feature amount extraction module may be any number and type.

For example, when the object handled by the deep neural network is a person and the number of sub-objects picked up by the feature extraction module is 2, the sub-object allocation unit 2231 puts the upper body of the person in the first local branch. Assign the lower body of the person to the second local branch. Similarly, when the number of sub-objects to be picked up in the feature amount extraction module is 2, the sub-object allocation unit 2231 assigns the right half of the person to the first local branch and the person to the second local branch. You may assign the left half of the body.

Further, when the number of sub-objects to be picked up in the feature amount extraction module is 3, the sub-object allocation unit 2231 assigns a person's face to the first local branch, for example, and assigns a person's face to the second local branch. Allocate the upper body excluding, and assign the lower body of the person to the third local branch.

In addition to the indirect reflection (second learning) described above, the learning unit 2232 uses a partial image to directly reflect on the area of the sub-object assigned to each local branch by the sub-object allocation unit 2231 (first). (Learning).

That is, the learning unit 2232 learns each local branch so that each local branch can accurately pick up the area of the sub-object assigned to the local branch by using the information indicating the area of the sub-object in the image for each image. (First learning) is performed, and the learning of each local branch is performed so as to further improve the analysis accuracy of the image analysis by using the result of the image analysis using the feature amount of the sub-object picked up by each local branch. (Second learning) is performed.

The loss function when the learning unit 2232 directly reflects on each local branch (first learning) is, for example, as follows.

For example, the shape of the area picked up by each local branch is rectangular, and the coordinates of the rectangular area actually picked up by the i-th local branch are (x ₀ , x ₁ , y ₀ , y ₁ ) (see FIG. 18). ), Consider the case where the coordinates of the rectangular area to be picked up by the i-th local branch are given as follows.

In this case, the learning unit 2232 uses, for example, the following equation (1) as a loss function that propagates directly to the i-th local branch.

The learning unit 2232 updates the model in the storage unit 12 by using the parameter values of the feature amount extraction module obtained by the direct reflection and the indirect reflection.

[Analyzer]
Next, the configuration of the analysis device according to the second embodiment will be described. FIG. 19 is a block diagram showing an example of the configuration of the analysis device according to the second embodiment. The analysis device 320 includes an input / output unit 31, a storage unit 32, and a control unit 233.

The storage unit 32 includes an image 321 for analysis, a model 2322 (attention model) which is an attention model whose parameters are optimized by learning by a learning device 220, and a classification result of an object captured in the image or an object captured in the image. It has an analysis result 2323 showing the estimation result of the attribute.

The control unit 233 has the same function as the control unit 33 shown in FIG. 8 and controls the entire analysis device 230. The control unit 233 functions as various processing units by operating various programs. The control unit 233 has a sub-object allocation unit 2331 and an analysis unit 2332.

The sub-object allocation unit 2331 allocates a corresponding area to each module of the model 2322 from the analysis image 321. The area to be picked up and the allocation of the area to be picked up to each module are optimized by learning in the learning device 220, respectively.

The analysis unit 2332 uses the model 2322 to extract the feature amount for each region in each module, and uses the feature amount extracted by each module to estimate the attribute to which the subject belongs in the analysis image 321 and detect the subject. Match with the target subject.

[Processing procedure of learning process]
An example of the processing procedure of the learning apparatus 220 will be described with reference to FIG. First, the sub-object allocation unit 2231 of the learning device 220 assigns sub-objects to each local branch in the feature amount extraction module of the deep neural network to be learned (step S201). After that, the learning unit 2232 learns each local branch of the feature quantity extraction module (step S202). That is, the learning unit 2232 performs indirect reflection of each local branch using the error back-propagated from the analysis module, and direct reflection of each local branch using the image of the learning data of the storage unit 22. ..

[Processing procedure for analysis processing]
Next, an example of the processing procedure of the above-mentioned analysis device 320 will be described with reference to FIG. First, the sub-object allocation unit 2331 allocates the analysis image 321 to the area corresponding to each module of the model 2322 (step S211). The area to be picked up and the allocation of the area to be picked up to each module are optimized by learning in the learning device 220, respectively.

As an analysis process, the analysis unit 2332 uses the model 2322 to extract the feature amount for each region in each module, and uses the feature amount extracted by each module to estimate the attribute to which the subject in the analysis image 321 belongs. Or the subject and the subject to be detected are collated (step S212).

[Effect of Embodiment 2]
In the above learning device 220, the area to be picked up by each of the local branches of the feature quantity extraction module is given, and the error is recorded as a loss function, and direct reflection is also performed. As a result, each of the local branches of the feature amount extraction module can accurately pick up the target part (sub-object) of the object even for an image in which the object is incompletely reflected. As a result, since the feature amount extraction module can accurately extract the feature amount of each sub-object, it is possible to improve the accuracy when the analysis module analyzes the object (for example, classification, attribute estimation, collation, etc.). ..

For example, if the learning by the learning device 220 of the second embodiment is applied to the learning of the deep neural network that automatically analyzes the person reflected in the image with respect to the image taken by the surveillance camera, "I got lost." When there is a request "I want to find a 5-year-old girl in red clothes" or "I want to find the criminal in this photo", I had to handle it visually in the past. Automatic analysis can also be performed on "images".

When learning by the learning device 220 of the second embodiment is applied to learning of a deep neural network that automatically analyzes a person for an image taken by a surveillance camera, and learning by a conventional technique (HA-CNN) is applied. The result of comparison with the case of the above is shown in FIG.

Here, for each deep neural network, the top five images that resemble the "person in the border pants" (the person in the image shown in "Find this person" on the left side of FIG. 17) from the images. Instructed to look for.

In this case, the deep neural network learned by the conventional technique (HA-CNN), which is a comparative example, should originally search for "a person with border pants", but in the above top five images, "Border T" is included. Includes "shirt person" and "border one-piece person". It is considered that this is because only the lower half of the person is shown in the comparison source image (the image shown in "Find this person" in FIG. 22), and the deep neural network fails to link the area and the part on the image. Be done.

On the other hand, in the deep neural network learned by the learning device 220 of the second embodiment, only the "person in the border trousers" is included in the above top five images searched from the images. "Border T-shirt person" and "Border one piece person" are not included. From this, it was shown that the deep neural network learned by the learning device 220 of the second embodiment can accurately search even an incomplete image.

That is, in the prior art, it was not decided in advance which sub-object each local branch should be in charge of in the feature quantity extraction module of the deep neural network. Therefore, which sub-object each local branch should be responsible for had to rely on indirect reflection from subsequent analysis modules. As a result, in order for each local branch to learn so that features can be extracted accurately even with incomplete images, it was necessary to prepare a large amount of training data and prepare a long learning time. ..

On the other hand, the learning device 220 of the second embodiment determines in advance which sub-object each local branch in the feature quantity extraction module should be in charge of. As a result, the learning device 220 can perform direct reflection in addition to the indirect reflection of each local branch described above. As a result, each local branch of the feature quantity extraction module can learn to accurately extract the feature quantity even if the image is incomplete, by using the amount of learning data and the learning time that can be realistically secured.

[About the system configuration of the embodiment]
The components of the image processing device 10, the learning devices 20, 220, and the analysis devices 30, 230 shown above are functional conceptual ones, and do not necessarily have to be physically configured as shown in the figure. That is, the specific form of the distribution and integration of the functions of the image processing device 10, the learning devices 20, 220 and the analysis devices 30, 230 is not limited to those shown in the illustration, and all or part of them may be used in various loads or usage conditions. Depending on the unit, it can be functionally or physically distributed or integrated.

Further, each process performed by the image processing device 10, the learning devices 20, 220, and the analysis devices 30, 230 may be realized by a CPU and a program in which any part of the analysis is executed by the CPU. Further, each process performed by the image processing device 10, the learning devices 20, 220, and the analysis devices 30, 230 may be realized as hardware by wired logic.

Further, among the processes described in the first embodiment, all or a part of the processes described as being automatically performed can be manually performed. Alternatively, all or part of the process described as being performed manually can be automatically performed by a known method. In addition, the above-mentioned and illustrated processing procedures, control procedures, specific names, and information including various data and parameters can be appropriately changed unless otherwise specified.

[program]
FIG. 18 is a diagram showing an example of a computer in which the image processing device 10, the learning devices 20, 220, and the analysis devices 30, 230 are realized by executing the program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these parts is connected by a bus 1080.

Memory 1010 includes ROM 1011 and RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, the display 1130.

The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each processing of the image processing device 10, the learning devices 20, 220, and the analysis devices 30, 230 is implemented as a program module 1093 in which a code that can be executed by the computer 1000 is described. The program module 1093 is stored in, for example, the hard disk drive 1090. For example, the program module 1093 for executing the same processing as the functional configuration in the image processing device 10, the learning devices 20, 220, and the analysis devices 30, 230 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

Further, the setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, a memory 1010 or a hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 into the RAM 1012 and executes them as needed.

The program module 1093 and the program data 1094 are not limited to those stored in the hard disk drive 1090, and may be stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). Then, the program module 1093 and the program data 1094 may be read by the CPU 1020 from another computer via the network interface 1070.

Although the embodiment to which the invention made by the present inventor is applied has been described above, the present invention is not limited by the description and the drawings which form a part of the disclosure of the present invention according to the present embodiment. That is, other embodiments, examples, operational techniques, and the like made by those skilled in the art based on the present embodiment are all included in the scope of the present invention.

1 Analysis system 10 Image processing device 11,21,31 Input / output unit 12,22,32 Storage unit 13,23,33,223,233 Control unit 20,220 Learning device 30,230 Analysis device 121 Image data 122 Conversion data 123 Extracted image 124 Converted image 131 Extracted part 132 Converted part 222,322,2222,2232 Model 231,232 Learning part 321 Analysis image 323,2323 Analysis result 331,2332 Analysis part 2231,231 Sub-object allocation part

Claims (5)

An image processing device that processes an image used for analyzing whether or not a desired subject is captured in an image captured by a surveillance camera.
It has a converter that converts the background and / or subject of the image used for the analysis.
When converting the background of the image, the conversion unit converts the background of the image into a background photographed by the surveillance camera or a background of the same type as the background photographed by the surveillance camera. An image processing apparatus characterized in that when converting a subject of an image, the property of the subject is converted into a property that the subject is likely to have in an area photographed by the surveillance camera. The image processing apparatus according to claim 1, wherein the property of the subject is an appearance property. The image processing apparatus according to claim 2, wherein the appearance property is a posture, clothes, facial expressions, hairstyles, or personal belongings. It is an image processing method executed by an image processing device that processes an image used for analyzing whether or not a desired subject is captured in an image captured by a surveillance camera.
It comprises a conversion step of transforming the background and / or subject of the image used for the analysis.
In the conversion step, when converting the background of the image, the background of the image is converted into a background photographed by the surveillance camera or a background of the same type as the background photographed by the surveillance camera. An image processing method characterized in that when converting a subject of an image, the property of the subject is converted into a property that the subject is likely to have in an area photographed by the surveillance camera. The computer is made to perform a conversion step of converting the background and / or the subject of the image used for analyzing whether or not the desired subject is captured in the image captured by the surveillance camera. When converting the background, the background of the image is converted to the background taken by the surveillance camera or the background of the same type as the background photographed by the surveillance camera, and the subject of the image is converted. An image processing program that converts the properties of the subject into properties that the subject is likely to have in the area photographed by the surveillance camera.

Images