JP2021120840A - Learning method, device, and program - Google Patents

Abstract

To provide a learning method for learning an image conversion process in which an image can be obtained while a task recognition accuracy, privacy protection, and a compression efficiency are ensured.SOLUTION: The learning method is for learning a weight parameter of an image conversion unit 11 by using a neural network structure. In the method, the weight parameter of the image conversion process is learned with use of a first cost for a recognition result obtained by recognizing, at a task unit 13 which is configured to recognize a prescribed task, a privacy protection image which is obtained by conversion of a training image at the image conversion unit 11, and a second cost for an evaluation result of the similarity between a compressed image obtained by compression of the training image at a compression unit 21 and the privacy protection image, such that a total cost calculated from the first cost and the second cost is minimized or the first cost and the second cost are alternately minimized.SELECTED DRAWING: Figure 7

Description

The present invention relates to a learning method, an apparatus and a program for learning an image conversion process capable of obtaining an image in which task recognition accuracy, privacy protection and compression efficiency are ensured.

In the case where image / audio data that may include user privacy information is transmitted to the cloud and analyzed using machine learning such as a neural network, it is necessary to prevent invasion of privacy to the user. For example, regarding audio data, if the service provider views the content of the smart speaker sent to the cloud, even if it is viewed for technical purposes such as improving the accuracy of machine learning, the result will be. As a result of privacy invasion.

In such a smart speaker, the privacy of the user is generally protected by encrypting the data from eavesdropping on the communication path. However, since the encrypted data is decrypted on the cloud side, the above situation may occur.

As disclosed in the news release article "The world's first realization of secret calculation technology that can perform standard learning processing of deep learning while encrypted" at the following URL, re-learning and fine while encrypted on the cloud side There is also a method of performing processing such as tuning.
https://www.ntt.co.jp/news2019/1909/190902a.html

The first problem here is that the image and audio data cannot be confirmed by the perception of the service provider. Actually, there are some tasks that we want to do by human perception, such as investigating the cause of problems and making mistakes in machine learning. For example, the service provider may want to visually eliminate poisoning data or respond to user complaints, but such confirmation may be difficult. The second issue is that raw data is included even if it is encrypted, so users are likely to be worried that the raw data may be leaked due to attacks or operational mistakes. ..

On the other hand, with respect to image data, a method of protecting privacy by performing image processing such as blurring or replacing sensitive information considered to be privacy has been conventionally performed. Although there is a sense of security for the user not to provide the raw data, the accuracy of the image analysis task on the service provider side tends to be very low.

In recent years, there has been an attempt to generate a privacy image without lowering the analysis accuracy of a task by machine learning such as a neural network as much as possible. With such a method, the cloud administrator or the service provider can perceptually judge the image while maintaining the accuracy of the task to some extent, and the privacy of the user can be protected. Since the user does not have to send the original image, it is considered to have the effect of lowering the psychological barrier to the use of the service.

For example, in the method of Patent Document 1, by estimating the facial organs and the face orientation and replacing the face with an avatar, privacy can be protected and the behavior recognition accuracy related to driving can be maintained. Similarly, in the method of Non-Patent Document 1, privacy can be protected and the accuracy of behavior recognition can be maintained by recreating the face area with a face different from the person himself / herself by GAN (hostile generation network).

The methods of Patent Document 1 and Non-Patent Document 1 are methods for replacing a part of the privacy area of an image such as a face, and do not consider the privacy of the entire image. For example, items that are not necessary for the service, such as the clothes and skin quality that you are wearing, the state of the room, etc., have not been erased, and it is not possible to respond to requests that you want to erase the overall reality.

As a method capable of erasing / reducing the overall reality, in the method of Non-Patent Document 2, behavior recognition from a moving image is performed from a low-resolution image. Since the resolution is low, there is an advantage that the image file size can be reduced. However, it is difficult to perform tasks other than simple action recognition from a simple low-resolution video, and applicable tasks are limited.

On the other hand, in Non-Patent Document 3, a model that transforms the entire original image is trained and generated by using a framework of hostile learning so as to approach a target image in which a large amount of random noise is inserted. By using the framework of hostile learning, it is possible to learn an image conversion model that is close to the target image with random noise while maintaining the accuracy of the task. Tasks include, for example, image recognition, facial organ recognition, and the like.

With this method, elements unnecessary for task analysis are easily hidden from the entire converted image, and it is easy to respond to privacy requests such as wanting to erase the overall reality. On the other hand, with this method, deterioration of task accuracy can be suppressed to a low level.

Special Table 2018-528536 Gazette

Ren, Zhongzheng, Yong Jae Lee, and Michael S. Ryoo. "Learning to anonymize faces for privacy preserving action detection." Proceedings of the European Conference on Computer Vision (ECCV). 2018. Ryoo, Michael S., et al. "Privacy-preserving human activity recognition from extreme low resolution." Thirty-First AAAI Conference on Artificial Intelligence. 2017. Kim, Tae-hoon, et al. "Training with the Invisibles: Obfuscating Images to Share Safely for Learning Visual Recognition Models." ArXiv preprint arXiv: 1901.00098 (2019).

However, even in the method of Non-Patent Document 3 that can deal with various requirements as described above, the following problems exist.

That is, in the method of Non-Patent Document 3, random noise tends to appear in the image converted for privacy. Image compression is not taken into consideration, and the converted image converted to random noise has a large spatial frequency component in each component from the low frequency component to the high frequency component, which causes a problem that the compression efficiency is extremely deteriorated. There is. In image analysis, reducing the capacity when transmitting and storing many still images and moving images is an issue, and it is required to reduce the file size while maintaining privacy and analysis accuracy, but compression efficiency If it is bad, the request for file size reduction cannot be satisfied.

In view of the above problems of the prior art, an object of the present invention is to provide a learning method, an apparatus and a program for learning an image conversion process capable of obtaining an image in which task recognition accuracy, privacy protection and compression efficiency are ensured. do.

In order to achieve the above object, the present invention is a learning method for learning weight parameters of an image conversion process by a neural network structure, and recognizes a privacy-protected image obtained by converting a training image by the image conversion process into a predetermined task. Using the first cost for the recognition result recognized in the task process of performing the above, the second cost for the evaluation result of the similarity between the compressed image obtained by compressing the training image by the compression process, and the privacy protection image. It is characterized by learning the weight parameter of the image conversion process. Further, it is characterized in that it is a learning device corresponding to the learning method and a program that causes a computer to execute the learning method.

According to the present invention, it is possible to learn an image conversion process capable of obtaining an image in which task recognition accuracy, privacy protection and compression efficiency are ensured by learning using the first cost and the second cost. can.

It is a functional block diagram of the conventional execution configuration which is the configuration at the time of task execution (inference) by the conventional method. It is a functional block diagram of the conventional learning configuration which is the configuration when learning the weight parameter of the conventional image conversion unit. It is a flow chart of learning by the conventional learning configuration, and shows the procedure of learning using GAN which is an existing method. It is a functional block diagram of the recognition device which is the structure at the time of task execution (inference time) which concerns on one Embodiment. It is a functional block diagram of the learning apparatus which concerns on one Embodiment which is the structure at the time of learning the weight parameter of an image conversion part. It is a flowchart of learning by the learning apparatus which concerns on one Embodiment, and shows the procedure of learning using GAN which is an existing method. It is a functional block diagram of the learning apparatus which concerns on one Embodiment different from FIG. It is a flowchart of learning by the learning apparatus 20 which concerns on one Embodiment in the configuration of FIG. It is a figure which shows the hardware configuration in a general computer device.

Hereinafter, before explaining the present embodiment, the method of Non-Patent Document 3 as a inverse proportion (hereinafter, referred to as “conventional method”) will be briefly described. FIG. 1 is a functional block diagram of the conventional implementation configuration 100, which is the configuration at the time of task execution (inference) by the conventional method. The conventional implementation configuration 100 is the conventional image conversion unit 101, the conventional compression unit 102, and the conventional task unit 103. Has.

The conventional image conversion unit 101 has a role of an image obfuscation device, converts an image to be converted (an image provided by a user and is subject to privacy protection), and outputs a privacy protection image. Conventionally, the compression unit 102 outputs a compressed privacy protection image by compressing the privacy protection image by an existing method. The conventional task unit 103 decodes the compressed privacy protection image, executes a predetermined recognition task (for example, posture estimation or image recognition), and outputs a recognition result (for example, posture estimation result or image recognition result).

As described above, the privacy-protected image (or the compressed privacy-protected image for storing or transmitting the image) has been converted into a privacy-protected state (obfuscated state), and the conventional task unit. The recognition accuracy of 103 is also an image that can secure a certain accuracy. However, there is a problem that the compression efficiency is poor and the file size of the compressed privacy-protected image becomes large.

In order to execute the task according to the conventional implementation configuration 100 of FIG. 1, a conventional image conversion unit 101 composed of a predetermined convolutional neural network or a multi-layer perceptron (hereinafter, referred to as “convolutional neural network or the like”) is provided in advance. It is necessary to learn and find the weight parameter. FIG. 2 is a functional block diagram of the conventional learning configuration 200, which is a configuration for learning the weight parameter of the conventional image conversion unit 101. As shown in the figure, the conventional learning configuration 200 includes a conventional image conversion unit 101, a conventional task unit 103, a conventional first evaluation unit 104, a target image generation unit 201, a conventional identification unit 203, and a conventional second evaluation unit 204.

The conventional image conversion unit 101 and the conventional task unit 103, which are present in both FIGS. 1 and 2, respectively, have the same configuration in FIGS. 1 and 2, so that they have a common reference numeral. However, the weight parameters of the conventional image conversion unit 101 are sequentially updated by learning by the conventional learning configuration 200, and the conventional image conversion unit 101 composed of the weight parameters when the learning is completed is the conventional implementation configuration of FIG. It will be used in 100.

On the other hand, the conventional task unit 103 common to FIGS. 1 and 2 is composed of an arbitrary existing convolutional neural network or the like that executes a predetermined task (posture estimation, etc.) on the image. The weight parameters are determined as those that have already been learned at the time of learning by the conventional learning configuration 200. (That is, in the learning by the conventional learning configuration 200, the weight parameter of the conventional task unit 103 is not learned and updated.)

FIG. 3 is a flow chart of learning based on the conventional learning configuration 200, and shows a procedure for learning using the framework of hostile learning, which is an existing method. At the start of the flow, initial values are set as random values or the like in the weight parameters of the conventional image conversion unit 101 (and the conventional identification unit 203) to be learned. When the flow is started, in step S101, the conventional identification unit 203 is learned by the conventional learning configuration 200 in the GAN connection configuration, the weight parameter thereof is updated, and then the process proceeds to step S102.

The GAN connection configuration is the conventional image conversion unit 101, the target image generation unit 201, the conventional identification unit 203, and the conventional second evaluation unit, in which the conventional task unit 103 and the conventional first evaluation unit 104 are omitted in the conventional learning configuration 200. Defined as a configuration with only 204. Specifically, the weight parameter of the conventional identification unit 203 is updated in step S101 by a series of learning procedures shown in the following (101) to (104).

(101) A fake image is generated by applying a conversion process by the conventional image conversion unit 101 to the training image given as training data, and noise is superimposed on the training image by the target image generation unit 201. A real image is generated by applying processing (noise superimposition processing that randomly changes the color for each pixel neighborhood with Gaussian noise). Here, for example, half is generated as a real image and the other half is generated as a fake image to obtain these as mini-batch.

(102) The fake image and the real image obtained as the mini-batch are identified by the conventional identification unit 203, which is the fake image (output of the conventional image conversion unit 101), and which is the real image (target image generation unit 201). (Output of) is obtained. The conventional identification unit 203 is configured by a predetermined convolutional neural network or the like as a device (discriminator for discriminating authenticity) that executes a task of discriminating between a real image and a fake image. The weight parameter is learned. (Therefore, the fake image and the real image constituting the mini-batch are input to the conventional identification unit 203, but the correct answer information as to whether the input image is a real image or a fake image is not given. , Conventionally, the identification unit 203 obtains the identification result by itself.)

(103) The conventional second evaluation unit 204 receives the identification result of the conventional identification unit 203 and gives a correct answer as learning data in advance (correct answer as to which of each image in the mini-batch is a real image and which is a fake image). The cost for the identification result is calculated by evaluating with a predetermined identification cost function that gives a low cost value if the identification result is correct and a high cost value if the identification result is not correct.

The identification cost function in the conventional second evaluation unit 204 is for improving the accuracy of the conventional identification unit 203 for discriminating authenticity.

(104) After performing the above processes (101) to (103) for a plurality of training images, that is, the forward propagation calculation of the cost (error), the cost (error) is used in the reverse direction in the conventional second evaluation unit. 204 → By calculating the error backpropagation method of the conventional identification unit 203, the weight of the conventional identification unit 203 is used by using an optimizer such as the stochastic gradient descent method (hereinafter referred to as “stochastic gradient descent method”). Update the parameters. With this update, it is expected that the accuracy of the conventional identification unit 203 for discriminating authenticity will be improved.

In step S102, the weight parameter of the conventional image conversion unit 101 is updated by learning in the GAN connection configuration and the task connection configuration, and then the process proceeds to step S103. The GAN connection configuration is as described in step S101, while the task connection configuration is defined as a configuration in which the conventional learning configuration 200 includes only the conventional image conversion unit 101, the conventional task unit 103, and the conventional first evaluation unit 104. NS. Specifically, the weight parameter of the conventional image conversion unit 101 is updated in step S102 by a series of learning procedures shown in (201) to (203A) or (203B) below. As described below, the procedure of (203B) is also possible as a modification of the procedure of (203A), and any of them may be used. (203A) is a method of alternately learning the conventional image conversion unit 101 by alternately calculating the output cost of the conventional first evaluation unit 104 and the output cost of the conventional second evaluation unit 204, and (203B) is these. This is a method of learning the conventional image conversion unit 101 from the total cost calculated from the two output costs.

(201) In the GAN connection configuration, the procedures (101) to (103) described with respect to step S101 are carried out for the training image given as the learning data. However, at this time, (103) is carried out as a procedure (103') changed from the procedure in step S101, and specifically, in the procedure (103'), the cost function conventionally used by the second evaluation unit 204 is used as a procedure. Change to the identification failure cost function, which evaluates exactly the opposite of the identification cost function used in (103). That is, using a predetermined identification failure cost function that gives a high cost value if the identification result obtained by the conventional identification unit 203 is correct, and a low cost value if the identification result is not correct (if identification fails). , Conventionally, the second evaluation unit 204 evaluates and calculates the cost for the identification result.

The above-mentioned cost function for identification failure in the conventional second evaluation unit 204 is for improving the accuracy of generating a fake image in the conventional image conversion unit 101 so that the conventional identification unit 203 fails to distinguish the authenticity. It is a thing.

(202) In the task connection configuration, the training image given as learning data is converted by the conventional image conversion unit 101 to obtain a fake image, and this fake image is recognized by the conventional task unit 103 and the recognition result. The recognition result is evaluated by collating it with the correct answer given as learning data in the conventional first evaluation unit 104, and the cost for the recognition result is calculated. If the recognition result is correct, the cost is set to a low cost value, and if the recognition result is not correct, the cost is set to a high cost value. Conventionally, the first evaluation unit 104 calculates the cost using a predetermined cost function.

(203A) A plurality of training images are divided into mini-batch, and the above processing (201) or (202), that is, forward propagation calculation of cost (error) is performed for each batch, and the conventional first evaluation is performed in the connection configuration. Calculate the output cost of unit 104 or the output cost of conventional second evaluation unit 204. In the case of the GAN connection configuration, the error of the conventional second evaluation unit 204 → the conventional identification unit 203 → the conventional image conversion unit 101 is used in the reverse direction (on the GAN connection configuration) using the cost output by the conventional second evaluation unit 204. By calculating the backpropagation method, the weight parameter of the conventional image conversion unit 101 is updated by using the stochastic gradient descent method or the like. Further, in the case of the task connection configuration, the error back propagation method of the conventional first evaluation unit 104 → the conventional task unit 103 → the conventional image conversion unit 101 is used in the reverse direction using the cost output by the conventional first evaluation unit 104. By performing the calculation, the weight parameter of the conventional image conversion unit 101 is updated by using the stochastic gradient descent method or the like. The back propagation of the above GAN connection configuration and the back propagation of the task connection configuration are alternately performed, and the weight parameter of the conventional image conversion unit 101 is learned.

(203B) When the total cost is used, the processing of the above (201) and (202) for a plurality of training images (training images common to the GAN connection configuration and the task connection configuration), that is, the cost (error). ), And for each training image common to both connection configurations, as a predetermined weighted sum of the cost output by the conventional first evaluation unit 104 and the cost output by the conventional second evaluation unit 204. Calculate the total cost of, and use the total cost in the opposite direction (on the GAN connection configuration), the conventional second evaluation unit 204 → the conventional identification unit 203 → the conventional image conversion unit 101 and (on the task connection configuration). By calculating the error backpropagation method of the conventional first evaluation unit 104 → the conventional task unit 103 → the conventional image conversion unit 101 in the direction, the weight parameter of the conventional image conversion unit 101 is calculated by using the stochastic gradient descent method or the like. You may update it. (Note that the conventional method uses the total cost that takes into account some errors in the task section when configuring the GAN connection. In the case of the task connection configuration, the total cost is not used.)

Due to the update of (203A) or (203B), the fake image obtained by conversion by the conventional image conversion unit 101 causes the conventional identification unit 203 to fail to distinguish the authenticity (that is, the target image generation unit 201 obtains it). It is expected that the accuracy (similar to a real image) will be improved, and the recognition accuracy of the conventional task unit 103 will also be improved (that is, the image will be in a state suitable for recognition processing). ..

In step S103, it is determined whether or not the learning has converged, and if it has converged, the weight parameters of the conventional image conversion unit 101 (and the conventional identification unit 203) at that time are obtained as the final learning result, and then the figure is shown. By ending the flow of 3 and returning to step S101 if it has not converged, the learning (steps S101 and S102) as described above will be further continuously carried out. For the convergence test in step S103, for example, by using a test image separate from the training image, the total cost of the procedure (203B) or the conventional first and second evaluation units 104 and 204 of the procedure (203A) output respectively. The cost may be calculated to evaluate the accuracy of the learning model, and it may be determined whether or not the improvement (history of improvement) of the accuracy has converged. A predetermined number of epochs or the like may be simply set as a convergence condition.

As described above, in the method of Non-Patent Document 3, as shown in FIGS. 2 and 3, the conventional image conversion unit 101 and the conventional identification unit 203 are learned while competing with each other by using the framework of hostile learning. , The learning result of the conventional image conversion unit 101 (and the conventional identification unit 203) can be obtained.

Hereinafter, an embodiment of the present invention will be described assuming that the method of Non-Patent Document 3 has been improved in consideration of the image compression rate.

FIG. 4 is a functional block diagram of the recognition device 10 which is configured at the time of task execution (inference) according to one embodiment, and the recognition device 10 includes an image conversion unit 11, a compression unit 21, and a task unit 13.

The image conversion unit 11 has a role of an image obfuscation device, converts an image to be converted (an image provided by the user and is subject to privacy protection), and outputs a privacy protection image. The compression unit 21 outputs the compressed privacy protection image by compressing the privacy protection image by the existing method. At this time, the compression unit 21 can perform compression according to a predetermined compression setting specified by the user. The task unit 13 decodes the compressed privacy protection image, executes a predetermined recognition task (for example, posture estimation or image recognition), and outputs a recognition result (for example, posture estimation result or image recognition result).

Also in this embodiment, as in the conventional method of Non-Patent Document 3, the privacy-protected image (or the compressed privacy-protected image for storing or transmitting the image) is in a privacy-protected state (obfuscated state). The image is converted and the recognition accuracy of the task unit 13 can be ensured to a certain degree.

On the other hand, in the present embodiment, unlike the conventional method, the obtained image has excellent compression efficiency, and the privacy-protected image (uncompressed state) obtained by the image conversion unit 11 is used as the compressed privacy-protected image in the compression unit 21. By compressing (irreversibly), the file size can be kept small without significantly changing the quality from before compression.

In order to execute the task by the recognition device 10 of FIG. 4, it is necessary to learn the image conversion unit 11 configured by the convolutional neural network or the like in advance and obtain the weight parameter thereof. FIG. 5 is a functional block diagram of the learning device 20 according to the embodiment, which is a configuration for learning the weight parameter of the image conversion unit 11. As shown in the figure, the learning device 20 includes an image conversion unit 11, a task unit 13, a first evaluation unit 14, a compression unit 21, an identification unit 23, and a second evaluation unit 24.

The image conversion unit 11, the compression unit 21, and the task unit 13, which are present in both FIGS. 4 and 5, have the same configuration in FIGS. 4 and 5, so that they have a common reference numeral. However, the weight parameters of the image conversion unit 11 are sequentially updated by learning by the learning device 20, and the image conversion unit 11 composed of the weight parameters when the learning is completed is used in the recognition device 10 of FIG. It becomes a thing.

On the other hand, the task unit 13 common to FIGS. 4 and 5 is composed of an arbitrary existing convolutional neural network or the like that executes a predetermined task (posture estimation, etc.) on the image, and is composed of the learning of FIG. The weight parameter is determined as having already been learned at the time of learning by the device 20. (That is, in learning by the learning device 20, the weight parameters of the task unit 13 are not basically learned and updated, but if the task accuracy cannot be sufficiently maintained, fine tuning is performed and the task unit is performed. The 13 weight parameters may be updated. In that case, the task accuracy of the normal image is reduced, but the task accuracy is improved for the output image of the image converter.)

FIG. 6 is a flowchart of learning by the learning device 20 according to the embodiment, and shows a procedure for performing learning using GAN, which is an existing method. The procedure of FIG. 6 by the learning device 20 is composed of steps S11, S12, and S13, which correspond to steps S101, S102, and S103 of FIG. 3 by the conventional learning configuration 200, respectively, and the conventional learning configuration 200. Each part of the above is read as each part of the learning device 20 as follows, and steps S101, S102, and S103 are performed, which correspond to steps S11, S12, and S13 of FIG. 6, respectively. (Therefore, with respect to each step of FIG. 6, since each step of FIG. 3 is corresponded to by reading the functional unit which is the main processing subject from that of FIG. 2 to that of FIG. 5, duplicate description will be omitted. )

That is, when the correspondence of reading is shown in the form of "configuration of conventional learning configuration 200 before replacement-> configuration of learning device 20 after replacement", "target image generation unit 201-> compression unit 21" and "conventional identification unit 203". → Identification unit 23 ”,“ Conventional second evaluation unit 204 → Second evaluation unit 24 ”,“ Conventional image conversion unit 101 → Image conversion unit 11 ”,“ Conventional task unit 103 → Task unit 13 ”and“ Conventional first evaluation It can be read as "Part 104 → First Evaluation Department 14". GAN connection and task connection during learning are also defined by these replacements.

In the above correspondence, only the "target image generation unit 201 and the compression unit 21" are in a relationship of performing different processing from each other, and all the others are in a relationship of performing the same processing in each step during learning. In other words, in the present embodiment, the learning device 20 is prepared by replacing the target image generation unit 201 of the conventional learning configuration 200 with the compression unit 21, and each step of FIG. 6 which is the same as each step of FIG. 3 is performed. As a result, the image conversion unit 11 whose weight parameter is learned by executing the learning device 20 has excellent compression efficiency and privacy protection as described in the recognition device 10 of FIG. An image suitable for the recognition process by the task unit 13 can be output.

In the present embodiment, as described above, the provision of the compression unit 21 in the learning device 20 (as an alternative to the target image generation unit 201 used in the conventional method) is based on the following unique knowledge. Is. That is, the compression unit 21 obtains a real image from the training image by performing lossy compression such as JPEG according to the compression setting specified by the user. Here, since lossy compression is accompanied by deterioration, it is a finding that a real image that has been lossy compressed and whose data size has been reduced can be used as it is as a privacy protection image.

Therefore, according to the flow of FIG. 6 in line with the framework of hostile learning, the image conversion unit 101 is an image having high compression efficiency and privacy protection as being similar to the image compressed by the compression unit 21. Assuming that an image suitable for recognition by the task unit 13 can be output, it is possible to learn the weight parameter together with the identification unit 23 having a hostile relationship.

FIG. 7 is a functional block diagram of the learning device 20 according to the embodiment different from that of FIG. 5, and FIG. 8 is a flowchart of learning by the learning device 20 according to the embodiment in the configuration of FIG.

In the embodiments of FIGS. 5 and 6, the weight parameters of the image conversion unit 11 are learned by using the framework of hostile learning, whereas in the embodiments of FIGS. 7 and 8, the framework of hostile learning is used. The weight parameter of the image conversion unit 11 can be learned without learning. By not using the framework of hostile learning, the learning device 20 of FIG. 7 has a configuration in which the identification unit 23 is excluded from the configuration of FIG.

The second evaluation unit 24 in the learning device 20 of FIG. 7 performs the following processing as a processing different from the processing in FIG. 5 (evaluation processing of the identification result of the identification unit 23). That is, the second evaluation unit 24 of FIG. 7 reads the compressed image obtained by compressing the training image by the compression unit 21 and the privacy protection image obtained by converting the training image by the image conversion unit 11, and determines the predetermined image. By the cost function of, the cost is calculated so that the larger the difference between these two images, the larger the value. In one embodiment, the second evaluation unit 24 in FIG. 7 determines that the mean square error between the compressed image and the privacy protection image (MSE, the sum of squares of the pixel values of the difference images of the two images divided by the number of pixels). The cost can be calculated. Alternatively, the cost may be calculated by averaging the number of pixels of the sum of the absolute values of the difference images.

On the other hand, regarding the processing contents of the compression unit 21, the image conversion unit 11, the task unit 13, and the first evaluation unit 14, which are configurations other than the second evaluation unit 24 in the learning device 20 of FIG. 7, the learning device of FIG. 5 It is the same as the processing content in 20. Hereinafter, each step of the flow of FIG. 8 will be described.

At the start of the flow of FIG. 8, the initial value of the weight parameter of the image conversion unit 11 is set in advance. (Regarding the task unit 13, the weight parameters have already been learned for the task unit 13 as in the embodiments of FIGS. 5 and 6.) When the flow of FIG. 8 is started, the image conversion unit 11 is in step S21. After learning the above and updating its weight parameter, the process proceeds to step S22. In step S21, the weight parameter of the image conversion unit 11 can be updated by a series of learning procedures specifically shown by the following (21) to (22A) or (22B). As the procedure (22A) and (22B), basically any one of them may be used. (Both may be used.)

(21) The training image given as the training data is converted by the image conversion unit 11 to obtain a privacy protection image, and the training image is compressed by the compression unit 21 to obtain a compressed image. The cost is calculated by evaluating the privacy protection image and the compressed image by the second evaluation unit 24. In addition, the task unit 13 recognizes the privacy protection image to obtain a recognition result, and the first evaluation unit 14 evaluates the recognition result by collating it with the correct answer given as learning data to determine the cost for the recognition result. calculate. If the recognition result is close to the correct answer (similar to the conventional first evaluation unit 104 of FIGS. 2 and 3 which is the same as the first evaluation unit 14 of FIGS. 5 and 6), the cost is set to a low cost value and the correct answer. The first evaluation unit 14 calculates using a predetermined cost function so that the cost value is high if it is not close to.

(22A) A plurality of training images are divided into mini-batches, the cost (error) is forward-propagated for each batch, and the cost output by the first evaluation unit 14 or the output cost of the second evaluation unit 24 is performed for each batch. To calculate. In the case of the GAN connection configuration (defined as the configuration of the image conversion unit 11, the compression unit 21 and the second evaluation unit 24 (the configuration excluding the identification unit 23 in FIG. 5) with respect to FIG. 7 as corresponding to FIG. 5), By calculating the error backpropagation method of the second evaluation unit 24 → image conversion unit 11 in the opposite direction using the cost output by the second evaluation unit, the image conversion unit 11 uses the stochastic gradient descent method or the like. Update the weight parameter of. Further, in the case of the task connection configuration (defined in the same manner as in FIG. 7 with respect to FIG. 7), the cost output by the first evaluation unit 14 is used, and the first evaluation unit 14 → the task unit 13 → the image conversion unit is used in the opposite direction. By calculating the error backpropagation method of 11, the weight parameter of the image conversion unit 11 is updated by using the stochastic gradient descent method or the like. The back propagation of the above GAN connection configuration and the back propagation of the task connection configuration are alternately performed, and the weight parameter of the image conversion unit 11 is learned.

(22B) When the total cost is used, the forward propagation calculation of the cost (error) is performed for a plurality of training images (training images common to the GAN connection configuration and the task connection configuration), and in both connection configurations. For each common training image, the total cost as a predetermined weighted sum of the output cost of the first evaluation unit 14 and the output cost of the second evaluation unit 24 is calculated, and the total cost is used (GAN). Probabilistic gradient descent by calculating the error backpropagation method of the second evaluation unit 24 → image conversion unit 11 and the first evaluation unit 14 → task unit 13 → image conversion unit 11 in the opposite direction (on the connection configuration). The weight parameter of the image conversion unit 11 may be updated by using a method or the like.

By updating using the cost of the procedure (22A) or (22B), the image conversion unit 11 also in the embodiment of FIGS. 7 and 8 as in the case of using the framework of hostile learning according to FIGS. 5 and 6. The privacy-protected image obtained by converting with is similar to the image compressed by the compression unit 21, so that the requirements for privacy protection and file size reduction can be satisfied, and the image can be recognized by the task unit 13. Be expected.

In step S22, it is determined whether or not the learning has converged, and if it has converged, the weight parameter of the image conversion unit 11 at that time is obtained as the final learning result, and then the flow of FIG. 8 is terminated and the learning is converged. If not, by returning to step S21, the learning as described above (step S21) will be further continuously carried out. The convergence test in step S22 is the same as in step S103 in FIG. 3 and step S13 in FIG. 6, for example, by using a test image separate from the training image, the total cost of the procedure (22B) and the procedure (22A). ), The cost of the first evaluation unit 14 and the cost of the second evaluation unit 24 may be calculated to evaluate the accuracy of the training model, and it may be determined whether or not the improvement of the accuracy (history of improvement) has converged. Further, the learning may be simply rounded up by a predetermined number of epochs.

According to each embodiment of the present invention described with reference to FIGS. 4 to 8 and the like, the recognition result of the task unit 13 for the privacy protection image obtained by converting the training image by the image conversion unit 11 is first evaluated. Using the first cost evaluated in Part 14 and the second cost evaluated in the second evaluation part 24 for the similarity between the compressed image obtained by converting the training image in the compression part 21 and the privacy protection image, the neural network structure is constructed. By learning the weight parameters of the image conversion unit 11, the image conversion unit 11 obtained as a learning result outputs an excellent image in all three points of privacy protection, compression efficiency, and recognition performance in the task unit 13. It becomes possible. As already explained, as learning using the first cost and the second cost, it is possible to learn so that the alternate minimization of each cost and the total cost obtained as the weighted sum are minimized. be.

Hereinafter, various supplementary examples and additional examples will be described.

(1) Regarding the compression unit 21 commonly used in each of the embodiments of FIGS. 4 to 8, the following may be applied. The compression by the compression unit 21 is performed by, for example, image compression using frequency conversion (JPEG using DCT (discrete cosine transform) as the base, JPEG2000 using wavelet transform as the base, etc.), and the following (i) to (iii). It can be performed under the compression setting specified by the user from the viewpoint.

(i) In the case of JPEG or JPEG2000 compression, it may be set to obtain a low-quality compressed image compressed so that the quality value is less than half of the whole. In the case of JPEG, many high-frequency components become 0 due to quantization. This makes it easier to protect the privacy of edges such as clothing textures and increases the compression ratio.

(ii) -a The DCT component of JPEG and the minimum frequency component after wavelet transform of JPEG2000 are all set to the same value (example: intermediate value is desirable. 0.5 for 0-1 gradation). This eliminates most of the gradation and makes it easier to protect the privacy of the skin and the like. In addition, the compression rate is increased by reducing the information on the DCT coefficient.

(ii) -b Let the DCT component of JPEG and the minimum frequency component after wavelet transform be several values (example: 2 to 8 values). This eliminates fine gradations and makes it easier to protect the privacy of the skin and the like. In addition, the compression rate is increased by reducing the information on the DCT coefficient.

That is, in (ii) -a, the converted minimum frequency component (original value) is rewritten to the same value, and in (ii) -b, the converted minimum frequency component is roughly quantized. Normally, low-frequency components are finely quantized and not coarsely quantized in order to ensure image quality, but here, the feature is that they are coarsely quantized.

(iii) Select the base of the frequency to be used and / or change the intensity. That is, when selecting a basis, the conversion coefficient of a predetermined basis that has not been selected is deleted. When changing the intensity (conversion coefficient), the conversion coefficient of a predetermined basis is forcibly rewritten to a constant value, or the absolute value of the coefficient is changed. For example, except for the DC component of DCT, the intensity is weakened by reducing the absolute value of the coefficient.

Basically, it is advisable to select and combine (i)-(iii) at any time, which makes it difficult to reduce the accuracy of the task. For general behavior recognition and image recognition tasks, it is highly likely that (i) and (iii), which understand movement and gradation, are suitable, and (ii) is suitable for the task of extracting key points of the face and skeleton. Be done. If the compatibility between the task and (i)-(iii) is unknown, a plurality of frequency bases may be selected in (iii). For example, it is divided into low frequency, high frequency, and its intermediate frequency components, and the obfuscation device (image converter) is learned with only each frequency component or only the frequency components in any two regions, and the degree of deterioration of task accuracy is determined. After obtaining it, the frequency component to be used may be determined. The method of dividing the frequency components is not limited to three.

For example, when the face is photographed with the same size, the spatial frequency analysis using FFT (Fast Fourier Transform) shows that there is a high correlation between the intensity of the spatial frequency and age or gender. It is known as disclosed in ("Relationship between facial expressions and impressions in facial images and spatial frequency characteristics").
Patent No. 05827225 (Japanese Patent Application No. 2012-521378)
https://www.jstage.jst.go.jp/article/itej/69/11/69_836/_pdf/-char/ja

Using this idea, the frequency intensity used for compression may be changed to hide age and gender. For example, it is known that when the intensity of low frequency is high, it is easy for women to distinguish it, and when the intensity of high frequency is high, it is easy for men to distinguish it. Therefore, it is considered that privacy can be protected by gradually increasing the intensity of high-frequency components so that they look masculine even in female images. Such a compressed image may be created for the entire training image.

(2) From the viewpoint of privacy protection and improvement of compression rate, a dynamic range (number of gradations) reduced in advance may be used as a training image. For example, it is possible to reduce the dynamic range and set the average pixel value to around 128, or to change the number of gradations of 256 for each of RGB to 8 values. Further, color reduction, conversion of a color that is conspicuous to humans such as skin color to another color (eg, blue, purple, green, etc.) may be performed.

When learning the image conversion unit 11 while further updating the learned parameters of the task unit 13 (when performing the fine tuning described above), the generated image may be input to the task as it is, but if not, the generated image may be input to the task as it is. Before executing the task, the task may be executed after returning to the original dynamic range and the number of colors. Similarly, in order to protect the privacy of images and reduce data, before compression, color reduction, conversion of colors that are noticeable to humans such as skin color to other colors (eg blue, purple, green, etc.), reduction of dynamic range, etc. You may go. In this case, the original number of colors, colors, and dynamic range are restored before the task is executed, and then the task is executed.

(3) When the task is posture estimation, there are various application examples of protecting privacy in a state where posture estimation is possible, and for example, there are the following.
-When sending an image of exercise in the house to a server, performing image analysis by posture estimation, and receiving advice, it is possible to protect the privacy of the person, skin, clothes, room, etc. of the image taken at home.
-When transmitting the image of the outside of the vehicle taken by the drive recorder to the server, it is possible to protect the privacy of the pedestrian while maintaining the state in which the AI (artificial intelligence) can recognize the pedestrian's raising hand posture, falling posture, etc.
・ When transferring / publishing data by raising the level of disclosure of images outside the vehicle taken by the drive recorder collected on the server, the pedestrian's hand-raising posture, falling posture, etc. can be recognized by AI while maintaining the pedestrian's posture. You can protect your privacy.
・ Privacy of the driver and passengers in the car while making it possible to detect the actions of the driver and passengers (calling on a mobile phone, facing backwards, etc.) from the images inside the car taken by the drive recorder. You can protect and send / publish the video to the server.

(4) Regarding the image conversion unit 11 commonly used in each of the embodiments of FIGS. 4 to 8, the following may be applied.

An convolution layer having a frequency base to be compressed (a frequency base common to that used for compression in the compression unit 21) as a kernel is inserted into the intermediate layer or the output layer of the network constituting the image conversion unit 11. For example, let the kernel be an 8x8 DCT basis. (The convolution layer of the frequency base kernel may be included in only a part of the specific layer, not the entire layer.) By setting the stride width to the compression block size in the compression unit 21, an image that is easy to compress is converted. It is expected that learning will make it easier to generate networks. The basis may be selected in advance, or the combination of the basis selected in the network in which the task accuracy and the compression rate are experimentally increased may be specified later. Moreover, the strength may be changed. Strength adjustment is realized, for example, by multiplying the kernel value by a scalar in advance.

The kernel value may be scalar-multiplied by 1 / table value using a quantization table value such as JPEG for each quality. Since the table value has a wide range, it is advisable to convert it to 2 to 8 values. After that, by quantizing with a step-like activation function that can be converted into 2 to 8 values, it is possible to simulate the actual JPEG compression to some extent in the image generation neural network.

When learning, only the inserted frequency-based kernel does not update the weights. That is, it is set to a fixed value, and only the weights of other layers need to be updated.

In addition, an up-convolution layer (a layer whose weight is fixed and is not updated by learning) corresponding to inverse quantization and inverse DCT is inserted in the subsequent stage (not immediately after) of the inserted DCT layer (layer including the frequency basis kernel). May be good. By adding an up-convolution layer, it returns to the spatial domain instead of the frequency domain, making it easier to visually check privacy, and the image converter that simulates JPEG compression can be directly connected to the task, causing task errors during learning. Since it can be propagated back to the image converter, it can be expected that the effect of JPEG compression can be estimated correctly. Normally, JPEG compression after the image converter is not considered during learning, but it can be expected that this will make it possible to consider the effects of JPEG compression to some extent correctly.

(5) FIG. 9 is a diagram showing a hardware configuration in a general computer device 70, and the recognition device 10 and the learning device 20 of each embodiment of FIGS. 4 to 8 each have such a configuration 1. It can be realized as one or more computer devices 70. The computer device 70 is a CPU (central processing unit) 71 that executes a predetermined instruction, and a dedicated processor 72 (GPU (graphic calculation device)) that executes a part or all of the execution instructions of the CPU 71 on behalf of the CPU 71 or in cooperation with the CPU 71. And deep learning dedicated processor, etc.), RAM73 as the main storage device that provides a work area for the CPU 71 and the dedicated processor 72, ROM74 as the auxiliary storage device, communication interface 75, display 76, mouse, keyboard, touch panel, etc. It includes an input interface 77 that accepts data, and a bus BS for exchanging data between them.

Each part of the recognition device 10 and the learning device 20 can be realized by a CPU 71 and / or a dedicated processor 72 that reads and executes a predetermined program corresponding to the function of each part from the ROM 74. Further, the learning method by the learning device 20 can be carried out by the CPU 71 and / or the dedicated processor 72 that reads and executes a predetermined program corresponding to each step of FIG. 6 or FIG. 8 from the ROM 74.

10 ... recognition device, 20 ... learning device
21 ... compression unit, 23 ... identification unit, 24 ... second evaluation unit, 11 ... image conversion unit, 13 ... task unit, 14 ... first evaluation unit

Claims (11)

It is a learning method to learn the weight parameters of the image conversion process by the neural network structure.
The first cost for the recognition result of the privacy protection image obtained by converting the training image by the image conversion process and recognizing the privacy protection image by the task process for recognizing a predetermined task, and
It is characterized in that the weight parameter of the image conversion process is learned by using the compressed image obtained by compressing the training image by the compression process and the second cost for the evaluation result of the similarity between the privacy protection image and the compressed image. Learning method. The learning method according to claim 1, wherein the second cost is evaluated based on a difference between the compressed image and the privacy protection image. Further using an identification process by a neural network structure, which is learned to distinguish authenticity by identifying the compressed image as a real image and the privacy-protected image as a fake image.
It is characterized by further comprising learning by a hostile generation network so that the identification process improves the accuracy of discriminating authenticity and the image conversion process improves the accuracy of misleading authenticity with respect to the identification process. The learning method according to claim 1 or 2. The learning method according to any one of claims 1 to 3, wherein the compression process uses a discrete cosine transform or a wavelet transform. The compression process is characterized by including frequency conversion using a conversion basis and replacement or quantization of the frequency-converted lowest frequency component with a constant value. The learning method described in either. The learning method according to any one of claims 1 to 5, wherein the compression process includes frequency conversion using a conversion basis and deleting a conversion coefficient of a predetermined conversion basis. The compression process involves frequency conversion using a conversion basis.
7. The learning method described in Crab. The learning method according to claim 7, wherein the stride width of the fixed folding layer is set to match the compression block size in the compression process. Learning the weight parameter of the image conversion process is
A claim characterized in that the first cost and the second cost are alternately minimized, or the total cost calculated from the first cost and the second cost is minimized. The learning method according to any one of 1 to 8. A learning device that learns weight parameters for image conversion processing using a neural network structure.
The first cost for the recognition result of the privacy protection image obtained by converting the training image by the image conversion process and recognizing the privacy protection image by the task process for recognizing a predetermined task, and
It is characterized in that the weight parameter of the image conversion process is learned by using the compressed image obtained by compressing the training image by the compression process and the second cost for the evaluation result of the similarity between the privacy protection image and the compressed image. Learning device. A program comprising causing a computer to execute the learning method according to any one of claims 1 to 9.

Images