JP7453900B2 - Learning method, image conversion device and program - Google Patents

Description

The present invention relates to a learning method, an image conversion device, and a program for learning image conversion processing that can protect image privacy while ensuring task accuracy.

In cases where image/audio data that may include user privacy information is sent to the cloud and analyzed using machine learning such as neural networks, it is necessary to prevent invasion of user privacy. For example, regarding audio data, if a service provider were to listen to the content of a smart speaker sent to the cloud, even if the viewing was for a technical purpose such as improving the accuracy of machine learning, the result would be A violation of privacy may occur as a result.

Note that such smart speakers generally protect user privacy from eavesdropping on the communication path by encrypting data. However, since encrypted data is decrypted on the cloud side, the above situation may occur.

As disclosed in the news release article at the URL below, "Realizing the world's first secure computation technology that can perform standard training processing for deep learning while encrypted," re-learning and fine-tuning can be performed while encrypted on the cloud side. There is also a method of performing processing such as tuning.
https://www.ntt.co.jp/news2019/1909/190902a.html

The first problem here is that it is not possible to confirm the image and audio data as perceived by the service provider. In fact, there are some tasks that we would like to perform using human perception, such as investigating the cause of problems and errors in machine learning. For example, the service provider may want to respond to user complaints or visually eliminate poisoning data, but such confirmation may also be difficult. As a user complaint, it is conceivable that users may complain that, for example, a service that uses images to return posture estimation results cannot detect posture. At this time, information such as an encrypted scrambled image does not know the shape of the person, and it is not possible to visually confirm what posture has not been detected. In addition, considering the case where only the obfuscated images for which the pose could not be detected are collected and the pose estimator is tuned again, it is not possible to annotate the scrambled images visually, and the user is required to annotate the scrambled images in consideration of privacy. If you ask them to do so, incorrect annotation data is likely to be mixed in, which hinders learning. In addition, there is a desire to be able to understand what is happening using human perception, such as monitoring in a monitor room, as work that would like to be done using human perception. In the event of an unexpected problem caused by a person, it is now possible to track the shape and movement of the person so that you can visually determine what position the person is in, how much they are not moving, etc. If not, it will be difficult for the security guard to decide what to do next (such as going to the site).

The second issue is that even if it is encrypted, it still contains raw data, so users are likely to worry that the raw data may be leaked due to an attack or operational error. .

On the other hand, with regard to image data, methods have been used to protect privacy by performing image processing such as blurring or replacing sensitive information that is considered to be private. Although there is a sense of security for the user by not providing raw data, the accuracy of the image analysis task on the service provider's side is likely to deteriorate significantly.

In recent years, there have been attempts to generate privacy images without reducing the accuracy of task analysis using machine learning such as neural networks. Such a method allows cloud administrators and service providers to judge images based on perception while maintaining a certain degree of task accuracy, and also protects user privacy. Since users do not have to send original drawings, it is thought to have the effect of lowering psychological barriers to using the service.

For example, in the method of Patent Document 1, by estimating facial organs and facial orientation and replacing the face with an avatar, it is possible to protect privacy and maintain the accuracy of behavior recognition related to driving. Similarly, in the method of Non-Patent Document 1, it is possible to protect privacy and maintain the accuracy of behavior recognition by regenerating the face area with a face different from the person's own face using a generative adversarial network (GAN).

The methods disclosed in Patent Document 1 and Non-Patent Document 1 are methods for replacing a partial privacy area of an image such as a face, and do not take into account the privacy of the entire image. For example, things that are unnecessary for the service, such as the clothes worn, skin type, and the state of the room, are not erased, and requests to erase the entire reality cannot be met.

As a method capable of erasing/reducing the overall reality, the method of Non-Patent Document 2 performs action recognition from a video from a low-resolution image. The low resolution has the advantage of reducing the image file size. However, it is difficult to perform tasks other than simple action recognition from simple low-resolution videos, and the tasks to which it can be applied are limited.

On the other hand, Non-Patent Document 3 uses an adversarial learning framework to learn and generate a model that transforms the entire original image so that it approaches the target image into which a large amount of random noise has been inserted. By using an adversarial learning framework, it is possible to learn an image transformation model that is close to the target image containing random noise while maintaining task accuracy. Examples of tasks include image recognition of up to about 20 classes, recognition of facial organs, etc.

With this method, elements that are unnecessary for task analysis are easily hidden from the entire converted image, making it easy to respond to requests for privacy such as wanting to erase the overall reality. On the other hand, this method can also keep the deterioration of task accuracy to a low level.

Special Publication No. 2018-528536

Ren, Zhongzheng, Yong Jae Lee, and Michael S. Ryoo. "Learning to anonymize faces for privacy preserving action detection." Proceedings of the European Conference on Computer Vision (ECCV). 2018. Ryoo, Michael S., et al. "Privacy-preserving human activity recognition from extreme low resolution." Thirty-First AAAI Conference on Artificial Intelligence. 2017. Kim, Tae-hoon, et al. "Training with the Invisibles: Obfuscating Images to Share Safely for Learning Visual Recognition Models." arXiv preprint arXiv:1901.00098 (2019).

However, the conventional methods such as Non-Patent Document 3 still have room for improvement in terms of ensuring task accuracy and ease of handling during visual confirmation, etc., while protecting privacy. That is, since the output image after image conversion was assumed to be an RGB output, it was unclear whether it was possible to sufficiently reduce color information and gradation information and sufficiently protect privacy. Furthermore, in Non-Patent Document 3, random noise is distracting, and when a moving image is generated by converting images frame by frame, there is a problem of flickering to the eyes, so there is a problem regarding ease of use. Here, a privacy-protected image can be obtained as an information-reduced image in which the information of the original image is reduced, and can be used for purposes other than privacy protection. However, with conventional methods, it is difficult to ensure task accuracy. However, it was not possible to obtain an information-reduced image that was easy to handle.

In view of the problems of the prior art described above, the present invention provides a learning method for image conversion processing that can ensure task accuracy, protect privacy, etc., and obtain an information-reduced image that is easy to handle, an image conversion device corresponding to the method, and The purpose is to provide programs.

In order to achieve the above object, the present invention is a learning method for learning parameters of an image conversion process using a neural network structure, the learning method being a learning method for learning parameters of an image conversion process using a neural network structure, and in which a training image is converted into an information-reduced image obtained by converting a training image by the image conversion process. updating parameters based on an error calculated based on the degree of information reduction evaluated by updating the parameters based on an error calculated based on a result of recognizing the information reduced image in a task process that performs recognition processing of a predetermined task; is used to learn the parameters of the image conversion process, and the information-reduced image by the image conversion process has a gradation information channel and a contour information channel from which gradation information and contour information of the training image are respectively extracted. In the parameter update using an error calculated based on the information reduction degree, the information reduction degree in the image of the gradation information channel and the image of the contour information channel is evaluated.

The present invention also provides an image conversion device that obtains an information-reduced image by converting an input image using image conversion processing parameters learned by the learning method, and performs image conversion processing using a neural network structure. An image conversion device that converts an input image to obtain an information-reduced image, the information-reduced image having a gradation information channel and a contour information channel from which gradation information and contour information of the input image are respectively extracted. The image conversion device is a program that causes a computer to execute the learning method, or a program that causes a computer to function as the image conversion device.

According to the present invention, by learning to obtain an information-reduced image obtained by converting an original image into one having a gradation information channel and a contour information channel , it is possible to ensure task accuracy and protect privacy. It is possible to obtain an information-reduced image that is easy to handle.

FIG. 1 is a functional block diagram of a task execution device and a learning device according to an embodiment. FIG. 3 is a diagram for explaining processing by a conversion NW unit of an image conversion unit. FIG. 3 is a diagram illustrating an example of processing in a conversion NW unit. 11A and 11B are diagrams showing three example images of a processing example by the task execution device; It is a flowchart of learning by a learning device concerning one embodiment. FIG. 7 is a diagram illustrating an example of a vertical step step function during forward propagation and a slope step-like step function during back propagation, which is smoothed to have a constant slope. It is a figure which shows the example of various quantization functions, etc. It is a figure which shows the experimental example (question and answer) of the privacy protection effect by embodiment of this invention. FIG. 1 is a diagram showing the hardware configuration of a general computer.

FIG. 1 is a functional block diagram of a task execution device and a learning device according to an embodiment. As illustrated, the task execution device 10 includes an image conversion section 1 including a conversion network (NW) section 11 and a quantization section 12, and a task section 13. The task execution device 10 is configured to include a convolutional neural network and a multilayer perceptron (hereinafter referred to as "convolutional neural network etc.") having a predetermined structure in the conversion NW unit 11 and the task unit 13. By using parameters, it is possible to perform tasks on images such as image conversion and inference processing on images.

A configuration for learning parameters of the task execution device 10 is a learning device 30 that includes the task execution device 10 itself, which is subject to parameter update through learning, and a learning section 20. The learning unit 20 includes a privacy update unit 2 including a gradation update unit 21 and a contour update unit 22, and a task update unit 23.

Through the learning performed by the learning device 30, the parameters of the conversion NW unit 11 in the image conversion unit 1 are sequentially updated, and the image conversion unit 1 configured with the parameters at the time when the learning is completed is updated to the task execution device 10. It can be used as the included image conversion unit 1. In this parameter update, as shown as lines L11 and L12 in FIG. The weight parameters of the conversion NW unit 11 are updated by performing error backpropagation in the order of ``11''. Similarly, when performing fine tuning, the parameters of the task part 13 provided in the task execution device 10 are also updated sequentially, and the task part 13 configured with the parameters at the time of completion of learning is changed to the task part 13 provided in the task execution device 10. It will be available as. In this parameter update, as shown as line L23 in FIG. and update the weight parameters of the conversion NW unit 11. (If fine tuning is not performed, the weight parameters are fixed and not updated in the task unit 13, and only the weight parameters of the conversion NW unit 11 are updated.) Below, the parameters of the task execution device 10 have been learned by the learning device 30. The details of the learning process will be described later, and the process of the task execution device 10 that performs the inference process will be described.

In FIG. 1, among the arrows representing data transfer between functional blocks, those drawn with solid lines indicate inference processing by the task execution device 10 and learning processing by the learning device 30 (forward error propagation during learning). , and those drawn with dashed-dotted arrows are related to the learning process (error backpropagation during learning) by the learning device 30.

Note that the task execution device 10 may be operated as follows. That is, the image conversion unit 1 may be placed on the side of the user who acquires the image subject to privacy protection, and the task unit 13 may be placed on the side of the cloud that executes the task using the privacy protected image. That is, the image conversion unit 1 and the task unit 13 may communicate via a network, and the privacy protected image may be transmitted from the image conversion unit 1 to the task unit 13. Alternatively, both the image conversion unit 1 and the task unit 13 may be placed on the cloud side, and the image subject to privacy protection (original image) is encrypted and sent to the cloud side, converted by the image conversion unit 1, and then By deleting the original image, it may be possible to prevent it from being saved on the cloud side. In this way, tasks can be performed while protecting privacy.

The image conversion unit 1 of the task execution device 10 is composed of a convolutional neural network, etc., and converts the original image by learning and determining its parameters (neural network weight parameters, etc.) in advance. This makes it possible to output privacy-protected images. The task unit 13 of the task execution device 10 is also composed of a convolutional neural network or the like with a predetermined structure that executes a predetermined task (object recognition, pose estimation, etc.) on an image, and its parameters are learned in advance. By determining this in advance, it becomes possible to execute a task on an image and output the result. Regarding the task part 13, any existing trained neural network etc. may be used depending on the task content, but in this case, weight parameters trained with an existing data set may be used, or image transformation In order to improve the accuracy of task execution for the privacy-protected image output by the unit 1, weight parameters that have been re-learned (fine-tuned) may be used. Note that the quantization unit 12 can be used as a function to easily adjust the degree of privacy protection (which also means the degree of privacy leakage in the opposite evaluation criteria) of the image conversion unit 1.

The outline of each part will be further explained below, mainly focusing on inference.

The image conversion unit 1 has the role of an image obfuscator, and converts the image to be converted (the image provided by the user and subject to privacy protection), thereby achieving privacy protection for the entire image. Output a privacy protected image. At this time, in the image conversion unit 1, the conversion NW unit 11 and the quantization unit 12 process the input image in this order to obtain a privacy protected image.

FIG. 2 is a diagram for explaining processing by the conversion NW unit 11 of the image conversion unit 1. The image conversion unit 11 can be configured as a deep learning network, and in FIG. 2, the conversion NW unit 11 is depicted as having an input stage Sin and an output stage Sout as a schematic diagram of the network structure. In the conversion NW11, input image Pin=(Pa,Pb,Pc) configured as a normal color image (general color image) (each image Pa, Pb, Pc is R, The three channel images G, B) are read in the input stage Sin, and the output image Pout=(P1,P2) with privacy protection applied to it is obtained in the output stage Sout, and this output image is quantized. Output to section 12.

This output image Pout=(P1,P2) is an image composed of two channels: the first image P1 of the first channel CH1 and the second image P2 of the second channel CH2. For example, the first image P1 is the input image The second image P2 is configured as a gradation image obtained by extracting gradation information from the input image Pin by flattening, and the second image P2 is configured as a contour image obtained by extracting contour information from the input image Pin by edge extraction. By using the two-channel output image Pout=(P1, P2), privacy protection can be applied to the input image Pin. (Although it is possible to obtain an image with further enhanced privacy protection by quantizing the output image Pout in the subsequent quantization unit 12, a certain level of privacy protection is also possible only in the conversion NW unit 11. .)

FIG. 3 shows an example of the input image Pin, the first image P1, the second image P2, and the output image Pout as an example of the processing in the conversion NW unit 11. FIG. 6 is a diagram illustrating a case where the image is a color image. It can be seen that gradation information is extracted from the input image Pin in the first image P1, and outline information is extracted in the second image P2.

The quantization unit 12 performs quantization on the image obtained from the conversion NW unit 11 to further protect privacy and outputs the image to the task unit 13 as an output of the image conversion unit 1. As shown in FIG. 1, the number of quantizations (4-value, 8-value, 16-value, etc.) in the quantization unit 12 can be set by a user, an administrator, or the like.

Note that the quantization unit 12 may be implemented as a quantization function applied to the final layer of the network constituting the conversion NW unit 11, or such a quantization function may be used in the final layer of the network constituting the conversion NW unit 11. It is also possible to output the image without applying the quantization function and further perform quantization on this output image.

Note that details of the above-mentioned conversion NW section 11 and quantization section 12 will be described later when explaining learning.

The task unit 13 performs predetermined key point extraction tasks (for example, pose estimation and facial organ detection) and object detection on the privacy protected image obtained by the image conversion unit 1, and extracts the results (for example, pose estimation results and facial organ detection). (organ detection results, object detection results).

Regarding the task, for example, for top-down posture estimation, the one disclosed in Non-Patent Document 4 below may be used. Inside the task, a heat map is created from the coordinates of the skeleton of the correct answer data, and a deep neural network model is trained to approximate it. As mentioned above, by using the existing method, the task unit 13 can perform arbitrary tasks such as not only pose estimation but also facial organ detection and object detection on the privacy protected image, and output the results.
[Non-patent Document 4] Chen, Yilun, et al. "Cascaded pyramid network for multi-person pose estimation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

FIG. 4 is a diagram showing an example of processing by the task execution device 10 as three image examples EX1 to EX3. In contrast to the original image EX1 (an input image to the task execution device 10), the image EX2 is an example of a privacy-protected image output from the image conversion unit 11 (a privacy-protected image obtained from the original image EX1). In the original image EX1, a person is photographed posing (during soccer practice), and while the original image EX1 retains the colors and detailed gradations, the converted privacy-protected image EX2 In this case, the three-dimensional color information of RBG is deleted, and the gradation is flattened with a small number of values such as 4 values, making it obfuscated and protecting privacy from the overall perspective of the image. can be seen. Since it is difficult to maintain task accuracy with gradation alone, contour information is also retained.

When displaying a privacy protection image EX2 consisting of two channels, for example, gradation information is displayed in the R channel, contour information is displayed in the G channel, and the B channel is displayed with the maximum gradation in all pixels as a dummy. By doing so, display using a method of displaying a normal color image is possible. Example EX3 is an example of a skeleton detected from the privacy protection image EX2, and is shown as an extracted skeleton superimposed on the original image of example EX1. (According to experiments by the present inventor, the pose estimator of Non-Patent Document 4 was tuned during learning of the image conversion unit 1, and images of the dataset were converted by the image conversion unit 1 and evaluated (task unit 13). input), the skeleton can be extracted from the privacy-protected image with less than 5% accuracy degradation.)

The gradation information and contour information are extracted by the conversion NW unit 11 as described above, and the details of the extraction will be described later in the explanation of learning, but the technical significance of using them is as follows.

In other words, the privacy protection image of the prior art is based on the use of normal color spaces such as RGB, but in the embodiment of the present invention, the output from the image conversion unit 1 is converted into colors such as "R", "G", "B", etc. Rather than learning as a color space channel, it is divided into two channels: the gradation channel (CH1) and the contour channel (CH2). In key point extraction tasks such as pose estimation, the inventors found through preliminary experiments that sharp contours that maintain resolution and gradations that are sufficient to identify the area are important, so these "gradations" Two channels are set for "contour". Furthermore, preliminary experiments conducted by the inventors have shown that these two elements are very effective for tasks such as object detection (human detection, etc.). By eliminating channels that output colors in this way, it is possible not only to explicitly limit the number of gradations, but also to generate a privacy-protected image in which no color information remains explicitly. In addition, since it is only necessary to control CH1 and CH2 individually, it is easy to stably output similar images for similar input images, and it is easy to generate images with less flicker when handling video input.

As explained in Example EX2 in Figure 4, regarding the privacy protection image composed of two channels CH1 and CH2 in which normal color information (for example, color information from RGB channels and YCbCr channels) has disappeared, CH1 and CH2 are R By assigning it to the or G channel (or color channels such as Y or Cb), you can output and visually check it as a normal image. (For channel B, which is not used, enter the same values or the same information as the G channel and R channel. Similarly, for the Cr channel, enter the same values or the same information as the Y channel and Cb channel. Good luck.)

Further, the technical significance of the quantization unit 13 is as follows.

That is, since the perception of privacy differs from person to person, it is desirable to be able to easily adjust the degree of protection of a privacy protection image. From the perspective of operation, memory, and processing speed, it is important to keep the model size of the image converter (image converter 1 in the embodiment of this invention) small, and to avoid having to prepare multiple models depending on the privacy level. It is. However, with conventional methods, it is not easy to adjust privacy while maintaining task accuracy. (If you want to do this, you may need to separately learn image converters with different privacy levels and prepare multiple image converters.)

In the embodiment of the present invention, adjustment is possible by replacing the quantization function applied to the output image or the final layer of the image conversion unit 1. For example, applying a quantization function for 4-value conversion to the image output from image conversion unit 1 places emphasis on privacy, and applying a quantization function for 8-value conversion or 16-value conversion places emphasis on accuracy. It is possible to easily switch the degree of user privacy protection according to the individual user's perception of privacy without having to have multiple image converter models. By having the image conversion unit 1 learn by switching between multiple quantization functions (4-value, 8-value, 16-value, etc.), the task can be Accuracy can be maintained.

FIG. 5 is a flowchart of learning by the learning device 30 according to one embodiment. According to the embodiment of FIG. 5, as illustrated in FIG. 4 as an image of example EX2, an image which is converted by the image converting unit 1 to have excellent privacy protection and also ensures accuracy of task execution by the task unit 13 is obtained. It can be obtained by learning the parameters of the image conversion unit 1 so that it can be obtained. At this time, it is also possible to learn the parameters of the task unit 13 by fine tuning. Each step in FIG. 5 will be explained below.

Before starting the flow shown in FIG. 5, initial values are set as random values or the like for the parameters of the image conversion unit 1 (conversion NW unit 11) to be learned. When performing fine tuning on the task unit 13, parameters learned using an existing model are set as initial values of the parameters of the task unit 13. (If fine tuning is not performed, the parameters of task part 13 can be fixed at these initial values.)

When the flow of FIG. 5 is started, in step S11, the conversion NW unit 11 is trained in the learning device 30 configured for privacy protection connection, and its parameters are updated, and then the process proceeds to step S12. The privacy-preserving connection configuration is defined as a configuration in which only the image conversion unit 1 and privacy update unit 2 are used in the learning device 30, and in step S11, parameters are calculated by performing forward propagation and back propagation calculations in this configuration. Update. That is, in step S11, forward propagation calculations are performed in the order of "conversion NW unit 11 → quantization unit 12 → privacy update unit 2" using the training image, an error is obtained in the privacy update unit 2, and this error is The parameters of the conversion NW unit 11 are updated as shown by lines L11 and L12 by backpropagating in the order of “quantization unit 12 (inverse quantization described later)→conversion NW unit 11”.

Note that error is a loss (cost) whose value increases as the deviation from the correct answer increases, and the term "error" will be used here.

During forward propagation in step S11, the image conversion unit 1 converts the training images (each image forming the mini-batch) using the parameters set at that time to obtain a privacy-protected image, and updates it for privacy. Output to part 2. At this time, as shown by lines L1 and L2, the image of the first channel CH1 from which the gradation information has been extracted from the privacy protected image is output to the gradation updating unit 21, and the contour information from the privacy protected image is output to the gradation updating unit 21. The extracted image of the second channel CH2 is output to the contour updating section 22. In the privacy updating unit 2, the gradation updating unit 21 and the contour updating unit 22 each calculate an error for evaluating the privacy protection performance of the obtained privacy protected image.

Here, details of the learning method in the privacy update unit 2 will be described later, but learning can be performed by various embodiments. For example, as shown by line L10, an embodiment is possible in which learning is performed using an image obtained by processing a training image, which is an original image to be converted into a privacy-protected image by the image conversion unit 1.

At this time, in the first method, an error function is set based on a simple standard in which the higher the degree of privacy failure (privacy leakage degree), the higher the error value is, and the gradation update unit 21 and the contour update unit Calculate in each of 22.

Furthermore, in the second method, privacy image control can be performed using a GAN (Generative Adversarial Network) framework in each of the gradation updating section 21 and the contour updating section 22. In GAN, generally, during training, the generator and classifier are trained adversarially to bring the output of the generator (fake image) closer to the target image (real image). The classifier uses the output from the generator as a fake image and the target image as a real image, and learns the authenticity of the images. At the time of inference (image generation), a classifier is not required, and the generator can output an image with a style similar to the target image. This GAN framework is applied to the privacy update unit 2, and the image conversion unit 1 (conversion NW unit 11) is trained to approach the target image obtained by processing the training image. To do this, it is sufficient to use an image obtained by processing a training image as a target image, use the image conversion unit 1 as a generator, and learn the image conversion unit 1 (generator) and the classifier in an adversarial manner. Note that the image converting section 1 (generator) may be shared as one generator, and the discriminator may be provided in each of the gradation updating section 21 and the contour updating section 22. (In the case of using the second method, the detailed configuration of the discriminator (and processing device for obtaining the target image) provided in each of the gradation update section 21 and the contour update section 22 is not shown in FIG. Although omitted, it is possible to perform learning in accordance with the GAN framework assuming that each of the gradation update unit 21 and contour update unit 22 has a classifier and a processor.)

The classifier performs learning by using the output from the image conversion unit 1 as a fake and the target image as a real image. When used, it is possible to bring the output of the image converter 11 closer to the target image. By processing the target image in advance so that it has image characteristics that make it easy to protect privacy and maintain accuracy, privacy image control that makes it easy to maintain privacy even when trying to maintain task accuracy, which will be described later, can be performed. Obtaining the target image by processing the original image will be described later.

During backpropagation in step S11, the error calculated for each training image constituting the mini-batch is used to update the privacy updater 2 (as described later, both the gradation updater 21 and the contour updater 22). ) perform error backpropagation calculations on the image conversion unit 1, thereby updating the parameters of the image conversion unit 1 as shown by lines L11 and L12. Backpropagation calculations can be performed using any existing method such as an optimizer such as stochastic gradient descent, which performs calculations that trace the network that constitutes the image transformation unit 1 in the reverse direction from the output side to the input side. That's fine.

In step S11, as described above, the privacy update unit 2 updates the parameters of the image conversion unit 11 by back propagation using the error, the details of which will be described later, so that the image obtained by the conversion by the image conversion unit 1 is updated. It is expected that the privacy protection performance will be even better.

In step S12, the image conversion unit 1 is trained in the learning device 30 having the task connection configuration, and its parameters are updated, and then the process proceeds to step S13. Note that when performing fine tuning of the task unit 13, in step S12, the learning of the task unit 13 is performed as well as the learning of the image conversion unit 1, and the parameters of the task unit 13 are also updated, before proceeding to step S13. The task connection configuration is defined as a configuration in which only the image conversion unit 1, task unit 13, and task update unit 23 are used in the learning device 30, and forward propagation and back propagation calculations are performed in this configuration in step S12. Then, the weight parameters of the conversion NW unit 11 of the image conversion unit 1 (and further the task unit 13 when performing fine tuning) are updated. That is, in step S12, forward propagation calculation is performed in the order of "conversion NW unit 11 → quantization unit 12 → task unit 13 → task update unit 23" using the training image, and the error is obtained in the task update unit 23. By backpropagating this error in the order of "task unit 13 → quantization unit 12 (inverse quantization described later) → conversion NW unit 12", the image conversion unit 11 (and task unit 13).

During forward propagation in step S12, the image conversion unit 1 converts the training images (each image forming the mini-batch) using various parameters including the weights set at that time to obtain a privacy-protected image. and outputs it to the task section 13. The task unit 13 executes tasks such as recognition on the obtained privacy-protected image using the parameters set at the time (fixed parameters if fine tuning is not performed), and uses the results for the task. Output to the update unit 23. The task update unit 23 compares the results of tasks such as recognition for the obtained privacy protection image with the correct answer for the task assigned in advance to the training image, and determines that the obtained result is close to the correct answer. The error is calculated assuming that the value becomes smaller as the value increases.

During backpropagation in step S12, the task update unit 23 performs backpropagation calculations on the image conversion unit 1 and the task unit 13 using the error calculated for each training image constituting the mini-batch. By doing so, the weight parameters of the conversion NW unit 11 (and task unit 13) of the image conversion unit 1 are updated as shown by line L23. For backpropagation calculations, an optimizer can be used as any existing method as in step S11, and a network consisting of a series connection of image conversion unit 1 and task unit 13 (image conversion unit 1→task unit 13) Calculation can be performed in the reverse direction (from the task unit 13 to the image conversion unit 1) from the output side to the input side.

In step S12, the task update unit 23 updates the parameters of the image conversion unit 1 (conversion NW unit 11) by backpropagation as described above, so that the image obtained by the conversion by the image conversion unit 1 is updated to the task unit 23. It is expected that the accuracy of tasks such as recognition will improve. When performing fine tuning, the parameters of the task unit 13 are also updated, which is expected to improve the accuracy of recognition by the task unit 13 of images obtained by conversion by the image conversion unit 1.

In step S13, it is determined whether the learning has converged or not. If it has converged, the process proceeds to step S14, and the parameters of the conversion NW unit 11 (and task unit 13) of the image conversion unit 1 at that point are changed to the final After obtaining the learning result, the flow in FIG. 5 is ended, and if it has not converged, the process returns to step S11, so that the learning described above (steps S11 and S12) is further continued. . For example, by using a test image separate from the training image for the convergence determination in step S13, the error calculated by the privacy update unit 2 as in step S11 and the error calculated by the task update unit 2 as in step S12 can be used. The accuracy of the parameters as a learning model can be evaluated by determining the error calculated by , and the determination can be made based on whether the improvement in accuracy (history of improvement) has converged. A predetermined number of epochs or the like may simply be used as the convergence condition.

As described above, according to the flow of FIG. 5, in step S11, the direction is directed to minimize the privacy-related error so that the privacy protection performance of the privacy protection image obtained by the conversion by the image conversion unit 1 (conversion NW unit 11) is improved. In step S12, the error related to the task is updated so that the task accuracy for the privacy-protected image obtained by the conversion by the image conversion unit 1 (conversion NW unit 11) is improved. An adversarial learning framework that alternately minimizes (optimizes) errors related to privacy protection, GAN errors, and errors related to task accuracy by updating the weight parameters of image conversion unit 1 in the direction of minimizing them. Accordingly, the parameters of the conversion NW unit 11 (and task unit 13) of the image conversion unit 1 can be learned. When minimizing the task error, the weight parameters of the task unit 13 may also be updated. In the flow of FIG. 5, the positions of steps S11 and S12 may be interchanged. Furthermore, each time the flow in FIG. 5 is repeated, the learning and updating of step S11 is set to be performed once, and the learning and updating of step S12 is set to be performed twice. You may change the learning rate.

Hereinafter, each embodiment of the error calculation for the privacy protection image by the privacy update unit 2 in step S11, which will be described in detail later, will be described, but it will be summarized as follows (1) and (2).

(1) Image converter learning for tone reduction in CH1 The original image may be a color or grayscale image. Here, a color image is converted to grayscale to have 256 gradations, and the grayscale image and the output image of an image converter (CH1 channel) with about 4 gradations are compared using MSE minimization, GAN, etc. for learning. Suppose that In this case, the output image of the image converter becomes an image with a lot of granular noise, like a halftone image, in order to maintain the gradation of a grayscale image in a pseudo manner. In other words, a flat privacy image is difficult to generate. Therefore, the image to be compared with the image converter output image in CH1 learning is not a grayscale image with many gradations such as 256 gradations, but an image that is flattened by converting the grayscale image to simple N-values etc. . It is desirable that the N value be approximately twice the gradation of the image you want to output (8 gradations in the case of 4-gradation output). By doing so, it is possible to reduce the graininess that occurs in halftone images.

(2) Regarding image converter learning for contour expression in CH2 This generates a contour image from the original image and trains the image converter to approximate that image. The contour image may be subjected to normal filter processing (DoG, xDog, Laplacian, Sobel), etc. The output image (CH2 channel) of the image converter is learned to approximate the contour image using GAN, etc. The key point extraction accuracy is improved by the appearance of the outline compared to when only CH1 is used, but it is thought that it is possible to prevent the privacy of gradation and realism from increasing.

Details of each embodiment will be described below. In either embodiment, as shown in the image of example EX2 in FIG. As a result, privacy protection for gradations and the like in the entire image can be achieved.

In this embodiment, as shown in the flow shown as lines L1, L2, and L10 in FIG. The image (original image) is also used to calculate the error. Describes specific processing.

As a premise of this embodiment, quantization-aware training (hereinafter referred to as quantization-aware training) is applied to at least the final layer of the image conversion unit 1, so first, this quantization-aware training will be explained. do. In the case of quantization-aware learning, the activation function is used as an activation function to obtain the value of the output layer of the network that constitutes the image transformation unit 1 (i.e., each pixel value of the transformed image), or as an activation function that is applied to the final layer. A constraint is set that as many step functions as the number of gradations (a predetermined number of gradations lower than the original image) constituting this converted image are used as additional quantization functions. That is, by using a step function for the number of gradations as the quantization function located at the end of the network that constitutes the image conversion unit 1, the image is naturally quantized and constructed with a lower number of gradations than the original image. It becomes possible to obtain a flattened privacy-protected image.

Note that, as already explained, the application of the quantization function (both during learning and during inference) is shown as the quantization unit 12 as a functional block in FIG.

However, when calculating forward propagation, a step function that rises in vertical steps is used to quantize and assign each pixel value of the output image to one of the values within a predetermined number of gradations, but when calculating back propagation, a step function is used that rises in vertical steps. In order to prevent the slope from becoming 0, for example, a function that uses the STE (straight through estimator) method to change the rise of the stairs from a vertical step to a gentle slope with a constant slope. Take advantage of.

Figure 6 shows the step function of the vertical step during forward propagation, and it is smoothed to have a constant slope other than 0 (in other words, the increasing behavior of the step function is simulated so that the slope of the differential does not disappear). An example with a gradient step-like step function during backpropagation (as an example) is shown. Graphs G2F and G2B show a step function with two vertical steps and a corresponding step function with a sloped step, respectively. When using the graph G2F, the image conversion unit 1 outputs a binary image. The function used at the time of backpropagation may be a straight line like graph G2B2 by STE (Straight through estimator) instead of the stepwise step function shown in graph G2B. Graphs G3F and G3B show a step function of three vertical steps and a corresponding step function of a sloped step, respectively. When graph G3F is used, the image conversion unit 1 outputs a ternary image. The function used during backpropagation may be a straight line as shown in graph G3B2 using STE (Straight through estimator) instead of the stepwise step function shown in graph G3B.

The number of gradations output by the image conversion unit 1, which is set by the step function, is preferably set to, for example, 2 to 64 gradations based on the following considerations. (In case of 2 gradations, it will be black and white binary)

First, if quantization-aware learning is not used, calculations are usually performed using 32-bit floating points for weights and activation functions, but a privacy image of pixel values output with 32 bits for each RGB is used. It is considered undesirable to transmit this because the amount of information is large and the bit depth of the general image format is 8 bits (256 gradations). It is also possible to output 32 bits for each channel and then quantize it to a general 8 bits with 256 gradations, but post-quantization tends to result in lower accuracy. Therefore, when generating a privacy-protected image that performs flattening, it is preferable to use learning using STE in backpropagation so that the output layer has 256 gradations or less. Furthermore, it is said that 64 or more gradations can be recognized as continuous gradations due to human visual characteristics. If tone curve correction is performed to widen the range of some pixel values in a flattened privacy image, there is a risk that a gradation that was not perceptible to humans before correction may appear. To prevent this, it is more desirable to use 64 gradations or less. Furthermore, since the foreground and background can be seen to some extent with two gradations, at least two gradations are sufficient. Also, as known from the four-color theorem, it is said that four colors are sufficient to separate adjacent areas, so it is thought that a level number of about four levels would be sufficient to express a flat painting style. In summary, the flattened image should have 2 to 64 gradations, with about 4 gradations being the most desirable. Because flattening uses such a small number of gradations, post-quantization tends to result in extremely low accuracy, but this can be prevented by using STE in backpropagation.

Note that quantization-aware learning is performed not only on the output layer that outputs the privacy-protected image, but also on the entire network that makes up the image conversion unit 1 (that is, all activation functions are It is also possible to use a network replaced with a step function of numbers. (In this case, in FIG. 1, the internal configuration of the conversion NW unit 11 may be interpreted as including the quantization unit 12.) For example, the image conversion unit 1 is trained using an 8-bit integer (INT8), etc. However, it is also possible to perform quaternization using an activation function or an additional quantization function in the final layer. In this case, the capacity of the model of the image conversion unit 1 is reduced, and the amount of calculation can also be reduced. It has the advantage of being easy to operate in real time even on mobile devices.

For learning on the task part 13 side (i.e., when the task part 13 is also learned by fine tuning in the learning in step S12), quantization-considered learning is performed (i.e., the activation function of the task part 13 also uses a step function. ), or quantization-aware learning may not be performed. Here, the effect of quantization-aware learning is similar to the general effect, and is that accuracy is less likely to decrease when converting a model trained with 32-bit decimals to an 8-bit model.

Here, the meaning of reducing the number of gradations (for example, 64 gradations or less as described above) in learning that takes into account quantization in the output layer in the image conversion unit 1 will be described. Generally, the output value (pixel value) of each node output from the output layer is expressed by 256 values. When considering privacy protection through flattening, this can be achieved simply by reducing the number of gradations of this output value. For example, if the output value is two gray scales, the number of gray scales is small, so it is likely to be flattened. However, task accuracy tends to be quite low. On the other hand, as the output value increases from 4 to 8 gradations, privacy is more likely to be leaked, but task accuracy is also easily maintained. In other words, there is a trade-off relationship between privacy protection and task accuracy.

Hereinafter, the processing using the two channels CH1 and CH2 in the image conversion unit 1, which will be described in detail later, will be explained.

As described above, the image conversion unit 1 can be configured to include a conversion NW unit 11 that can be configured with a DNN network or the like that converts an image, and a quantization unit 12 that is applied thereafter. The privacy update unit 2 does not have three output channels like RGB that can express colors, but has a channel for expressing gradation (CH1) and a channel for expressing contours (CH2), and the processing of each channel is The key updating section 21 and the contour updating section 22 are responsible for this. Here, color information is explicitly removed by not having a color channel. When extracting key points such as facial organ detection and pose estimation, these two channels are used because the minimum gradation and contour information that does not reduce resolution are more important than many gradations and colors. It is also effective in detecting objects including people.

If you apply the normal R channel to channel CH1, the normal G channel to channel CH2, and set the B channel to the maximum value for all pixels or the same value as R and G, you can convert the image in the normal image format. A person can visually recognize the converted image output by the unit 1. If it is a color space such as YCbCr, CH1 may be assigned to the luminance Y channel, and CH2 may be assigned to Cb or Cr. Alternatively, after allocating the normal color space in this way, it may be further converted to gray scale to form one channel.

The image conversion unit 1 outputs a tone expression channel CH1 (also called a tone channel) and a contour expression channel CH2 (also called a contour channel). The gradation updating unit 21 receives the image of the gradation expression channel CH1 as input, and learns the image converting unit 1 by connecting only the CH1 image of the converted image. The contour update section 22 receives the image of the contour expression channel CH2 as input, and learns the image conversion section 1 by connecting only the CH2 image of the converted image. In step S11 of FIG. 5 described above, the gradation updating section 21 and the contour updating section 22 each perform learning by backpropagating the error to the image converting section 1 once or at a determined rate. It is desirable to additionally apply the quantization function at the end, but in that case, CH1 and CH2 may be quantized with a common quantization function, or may be quantized with a different quantization function. For example, both CH1 and CH2 may be 4-valued, or different quantization may be performed.

The role of the gradation updating unit 21 will be described. Here, in order to learn to bring the gradation channel CH1 image of the converted image closer to the reference flattened image Ty as the correct answer, the following first or second method can be used.

The first method is to read the converted image (CH1) output by the image conversion unit 1 and a training image that is the original image of this converted image, and use a processing device to generate a reference flattened image Ty from the training image. After that, the image transformation unit 1 is trained by back propagating the error to minimize the MSE, which is a quantification of the image difference between the reference flattened image Ty and the transformed image, and its parameters are updated. do.

The second method is to read the converted image (CH1) output by the image conversion unit 1 and a training image that is the original image of this converted image, and use a processing device to generate a reference flattened image Ty from the training image. Then, the classifier and generator (image conversion unit 1) in the GAN framework are trained using the reference flattened image Ty (target image) as a real image and the converted image as a fake image, and their parameters are updated. (As mentioned above, the processing device and the discriminator are provided in the gradation updating section 21, and the discriminator may be learned in the gradation updating section 21.)

Regarding the processing equipment, in both the first method and the second method, if the number of gradations in the reference flattened image Ty is too large, the gradation channel CH1 of the converted image is difficult to flatten, resulting in pseudo-halftone expression (halftone image). Since noise such as , etc. tends to increase, it is better to generate a flat image by simply converting it into N-values with a smaller number of gradations. On the other hand, if the N value is set to be the same as or less than the number of gradations output by CH1, problems such as flat images are broken, areas are difficult to distinguish, and task accuracy is difficult to maintain, probably due to stronger gradation constraints. Occur. Therefore, the N value is desirably about twice the gradation level output by CH1, and is preferably suppressed to three times or less. If the number of gradations output by CH1 is 4, the reference flattened image Ty is generated by simple 8-value conversion or the like. At this time, the reference flattened image is 8-valued, but it is not intended to pseudo-express the gradation with 4-valued values, but to output different styles in CH1 and CH2. It is used to express a particular style of painting.

Here, the reference flattened image Ty is generated as a flat image in which the regions are identified by applying the existing method of region segmentation, and then the regions are colored with approximately the same number of gradations as the number of gradations output by CH1. You can. Further, Ty may generate an N-value image using a discriminant analysis method or the like that dynamically determines a threshold value. (However, flickering is more likely to occur when a converted image is animated than a simple N-value image.) Also, for CH1, there is no problem with the first method from the viewpoint of the amount of calculation.

Through each of the above methods, learning progresses so that the gradation channel CH1 of the converted image by the image conversion unit 1 becomes flat similar to the reference flattened image Ty, and privacy protection is realized in the converted image. Can be done.

Next, the role of the contour updating section 22 will be described. Here, in order to learn to bring the contour channel CH2 image of the converted image closer to the reference contour image Te as the correct answer, the following first or second method can be used.

The first method is to read the converted image (CH2) output by the image converter 1 and the training image that is the original image of this converted image, and generate a reference contour image Te from the training image using a processing device. Then, the error is back-propagated to make the image conversion unit 1 learn, and its parameters are updated so as to minimize the MSE, which is a quantification of the image difference between the reference contour image Te and the converted image.

The second method is to read the converted image (CH2) output by the image converter 1 and the training image that is the original image of this converted image, and generate a reference contour image Te from the training image using a processing device. Then, the classifier and generator (image conversion unit 1) in the GAN framework are trained using the reference contour image Te (target image) as a real image and the converted image as a fake image, and their parameters are updated. (As mentioned above, the processing device and the discriminator are provided in the contour updating section 22, and the discriminator may be learned in the contour updating section 22.)

Regarding the processing tool, in both the first method and the second method, the reference contour image Te can be created using a normal contour extraction filter. For example, there are Laplacian, Sobel, Canny, DoG (Difference of Gaussian), and extended DoG filters. Edge information usually includes both positive and negative numbers around 0, so it is best to visualize it using the same method as normal visualization. For example, it is a good idea to take the absolute value of the value and process it so that the background is white instead of the line (inversion). Alternatively, it may be normalized to the range of values that the image can take (0 to 1, 0 to 255, etc.) so that 0 is in the middle gradation. Furthermore, the colors may be reduced.

Through each of the above methods, learning progresses so that the contour channel CH2 of the converted image by the image conversion unit 1 becomes similar to the reference contour image Te, and privacy protection is achieved by leaving only important information in the converted image. can do.

Note that the contour image has little information, and the first method may not necessarily represent the contour in CH2 of the image conversion unit 1. In this case, the second method may be used.

As described above, the privacy image update unit 2 calculates the error by approaching the drawing style (flatness/outline) of the flattened/outline extracted reference images Ty, Te, and learns according to the flow shown in Figure 5. , privacy protection can be realized by flattening the converted image by the image conversion unit 1, and task accuracy can be maintained. Furthermore, the following processing examples 1 and 2 may be performed.

(Example 1) Note that the task unit 13 may perform re-learning from scratch or additional learning after the training image is converted by the trained image converting unit 1. For example, task accuracy can be improved by learning additional supervised training images that are prone to mistakes. At this time, the output from the image conversion unit 1, especially its gradation channels, may contain granular noise depending on the learning mode, so even if the task unit 13 is trained after performing noise removal processing. good. The noise removal process includes, for example, expansion/contraction processing, and the noise may be removed after the output from the image conversion unit 1 is quantized by the quantization unit 12. This allows the degree of privacy protection to be further increased.

(Example 2) On the other hand, the converted image may be applied to a task different from the task used during learning. For example, consider a case where the task unit 13 is initially assigned body skeleton estimation and causes the image conversion unit 1 to learn. Once the learning of the image conversion unit 1 converges to a certain extent, the output image of the image conversion unit 1 will contain information important for body skeleton estimation (contours and region segmentation), with the condition that unnecessary information (colors and detailed gradations) will be removed. (grayscale that is easy to understand) remains in a state where the task accuracy is not degraded as much as possible. In fact, this situation is likely to be the same in object detection tasks other than skeletons. Therefore, if you connect the training image for object detection with the skeleton estimation task and transform it using a trained image converter, and then train the object detector, the object detection task will also perform reasonably well on privacy-protected images. Accuracy can be maintained. In this case, learning for another task such as an object detector may be performed by tuning from a trained model or by learning from scratch.

That is, learning may be performed in steps 1 and 2 below. At this time, in the learning device 30 having the configuration shown in FIG. 1, it is assumed that in step 1, the task unit 13 is the task unit 13-1 related to the first task, and in step 2, the task unit 13 is the task unit 13-2 related to the second task. Assuming that, learning can be performed (by replacing the task part 13 used in steps 1 and 2 with mutually different task parts 13-1 and 13-2).
<Procedure 1> Parameters of the image conversion unit 1 and the task unit 13-1 regarding the first task (eg, skeleton estimation task) are learned according to the flow shown in FIG.
<Procedure 2> Using the image conversion unit 1 configured with the fixed parameters learned in Step 1, the task unit 13-2 for a second task different from the first task (for example, an object detection task) is Learn parameters. The learning may be performed by following the flowchart of FIG. 5 with step S11 omitted, and only learning the task portion 13-2 is performed in step S12.

When performing transfer learning on a task model that has been trained to increase accuracy with respect to the converted image, such as the model that was learned by updating the task during learning in Example 1 and Example 2 above and step S12 in Figure 5, However, it is better to perform learning after converting the training images using a similar image conversion unit 1. At this time, even if the number of input sheets is reduced, it is easy to improve the accuracy of the task. This is considered to be because the trained image conversion unit 1 can easily reduce unnecessary information during learning.

When learning another task from scratch, it is easier to improve the accuracy of the task even if the number of training images is reduced than when using unconverted full-color RGB images. This is considered to be because the trained image conversion unit 1 can easily reduce unnecessary information during learning (and inference). In other words, even in learning from scratch for another task, by using the trained image conversion unit 1, it is possible to save the effort of acquiring a large amount of training images such as photographing training images with various backgrounds.

In other words, the privacy-protected image obtained as a converted image in the image conversion unit 1 can be called a privacy-protected image as an example of its suitable use, but more generally, the privacy-protected image obtained as a converted image in the image conversion unit 1 can be used to convert the original image after maintaining accuracy for various tasks. This can be called an information-reduced image in which information is reduced from the image. (In the following, based on this premise, the output image of the image conversion unit 1, which is generally an information-reduced image, will be referred to as a privacy-protected image depending on the usage example.)

Similarly, the error evaluated by the privacy updating unit 2 during learning was an error based on the degree of privacy protection, but more generally, it can also be called an error based on the degree of information reduction. Similarly, the quantization function set by the quantization unit 12 in response to user settings etc. adjusts the degree of privacy protection, but more generally, it can also be called a function that adjusts the degree of information reduction. It is.

The details of the quantization unit 12 will be explained below.

The quantization function that is finally applied can be switched depending on the privacy level desired by the user. For example, it is possible to prepare step-like quantization functions for binarization to 32-value conversion, and then quantize converted images output within 64 values at the quantization level selected by the user. be. The higher the granularity of quantization, the more privacy is leaked, but the task accuracy tends to be higher.

Even if the quantization granularity is different, if the function used during error backpropagation is the same (STE result), there is no need to switch step functions for learning, and even for switching unlearned step functions. It is easy to maintain a certain degree of accuracy, but in order to further maintain task accuracy even when the quantization granularity changes, when the image conversion unit is trained by the gradation update unit 21 and contour update unit 22, it is necessary to It is even better to learn by using quantization functions alternately.

FIG. 7 is a diagram showing examples of various quantization functions used as described above, and the example of FIG. 7 will be referred to as appropriate below. In each example in Figure 7, the horizontal axis (x-axis) is the value before quantization (0≦x≦1), and the vertical axis (y-axis) is the value after quantization (0≦y≦1). It shows the quantization function and its base function (a continuous function that is simulated by the quantization function). Note that the two examples EXa0 and EXd0 enclosed in parentheses () are base functions, and the other examples are quantization functions. The base function of example EXa0 is a linearly increasing function of y=x, and examples EXa,EXb are 4-value and 8-value quantization functions that uniformly simulate this, and Examples of quantization functions are EXc and EXe. Further, the base function of example EXd0 is a sigmoid curve-like increasing function, and example EXd is a four-value quantization function that simulates this.

For example, in even-numbered epochs, CH1 and CH2 are trained using a 4-value quantization function, and in odd-numbered epochs, they are trained using an 8-value step function. The simplest CH1 learning method (using MSE minimization) when learning alternately with 4-value and 8-value is to generate Ty by simple 8-value conversion in even-numbered epochs, and apply it to the step function at the end of image transformation. EXa (4-value linear type), Ty is generated by simple 16-value conversion at odd epochs, and EXb (8-value linear type) is applied to the step function applied at the end of image conversion for learning. In both cases, the example EXa0 base function (linear type; y (value after quantization) = x (value before quantization)) is used during error backpropagation to avoid the gradient becoming 0, and the function during backpropagation is It is better to make them the same. (In the case of switching learning, it is not desirable to use gradient step functions like G2B and G3B in Figure 6 during backpropagation.)

The step function switching learning described above was based on the assumption that the values before quantization and the gradation values after quantization were equally spaced, as in the example EXa and EXb (4-value, 8-value linear type). You can also use step functions that are not equally spaced, such as EXc (8-value non-uniform linear type).

If Ty is generated by coarsely quantizing the intermediate tones before quantization (input), as in EXc (8-value non-uniform linear type), it is easier to maintain the privacy of the intermediate tones. Example EXc (8-value non-uniform linear type) can use Example EXa0 (linear increasing function) as a function during backpropagation, just like Example EXa (4-value linear type), so inconsistency is less likely to occur. . For example, by switching between these examples EXa and EXc (4-value linear type, 8-value non-uniform linear type) and learning, even if the granularity after quantization is increased from 4 values to 8 values, the intermediate gradation Privacy is difficult to leak. On the other hand, there is also a balance with task error feedback, and changes in intermediate gradations, etc. that are minimally necessary for task analysis, are likely to be expressed by contour channels and low or high gradations that are shifted from the original intermediate gradation. . (At this time, the gradation difference on the output side needs to be separated by at least a value that divides the dynamic range into 64 equal parts so that people can clearly see the boundary.)

On the other hand, by changing the base shape of the step function used for Ty generation, learning, and inference from a linear shape, the degree of privacy leakage and task accuracy can also be changed. For example, since the distribution of natural images is biased toward intermediate tones, after correcting Ty with a tone curve that emphasizes contrast like EXd0 (sigmoid curve type) or after flattening the distribution (after flattening the distribution), EXa0 When Ty is generated using quantization based on (linear increasing type), information deletion due to quantization-aware learning is less likely to occur, making it easier to maintain task accuracy.

When using (inference), not only the quantization function used for switching learning, but also intermediate and higher-granularity quantization functions can be used without any problem to some extent, and the degree of privacy leakage can be adjusted. For example, when learning with 4-value and 8-value quantization functions, a 6-value quantization function tends to show intermediate privacy leakage and task accuracy, and a 16-value quantization function tends to show higher task accuracy at the cost of increased privacy leakage. Prone. (However, if the granularity of quantization is coarsened, it tends to be difficult to maintain accuracy, so it is a good idea to learn the switching function around 4 values.) The quantization function used for switching learning It is thought that task accuracy can be easily maintained by making the intermediate quantization function match the switching learning function as much as possible.

For example, if learning to switch between 4-value and 8-value quantization functions is performed using Examples EXa and EXb (4-value linear type, 8-value linear type), the intermediate 6-value quantization function will be changed to Example EXe. (6-value non-uniform linear type) In example EXe (the graph has 6 steps as a 6-value non-uniform linear type), the values before quantization (0~0.25, 0.75~1, i.e. 1, 2, 5, 6 out of 6 steps) The values before quantization (0.25 to 0.75, that is, the 3rd and 4th steps out of 6 steps) match the example EXa (4-value linear type). (linear type).

In addition, the function used for backpropagation is preferably a straight line as shown in the example EXa0 (linear increasing function) and graphs G2B2 and G3B2 in Figure 6, and a sloped step-like function as shown in B2B and G3B in Figure 6. It is better not to use it because it tends to cause inconsistency.

Also, when used, different quantization functions can be used for the same 4-value conversion (although it does not necessarily keep the task accuracy as high as possible). For example, if you want to emphasize task accuracy, use Example EXa (4-value linear type) for the converted image of the image conversion unit learned using Example EXa (4-value linear type), and make some adjustments to prioritize privacy. If desired, example EXd (four-level sigmoid curve type) can also be used. For example, EXd (4-level sigmoid curve type) is close to a binary image and can express intermediate gradations, and although the task accuracy is slightly lower, the degree of privacy leakage is reduced, and it can be used as a fine adjustment between 3-level and 4-level conversion. can also be used.

As described above, according to each embodiment of the present invention, the image conversion unit 1 can be configured with a small convolutional network, etc., and can also operate on a user terminal. By providing multiple quantization functions on the user terminal and allowing them to switch between them, the user can select the privacy level that is appropriate for him/her, and the accuracy of subsequent posture estimation etc. will basically remain within that range. can be kept to the maximum.

In addition, while maintaining the accuracy of key point extraction for tasks such as pose estimation and facial organ detection from privacy-protected images, color and gradation information is explicitly reduced, making it less likely to cause obtrusive noise when outputting still images and videos. Can generate privacy-protected images. Furthermore, while maintaining task accuracy, the degree of privacy leakage on an image can be easily adjusted by simply switching the quantization function for the image without switching the image converter.

Below, various supplementary examples and additional examples will be explained.

(1) When the task is pose estimation, there are various application examples for protecting privacy while pose estimation is possible, including the following.
- When you send images of yourself exercising at home to a server, perform image analysis using posture estimation, and receive advice, you can protect the privacy of people, skin, clothes, rooms, etc. in images taken at home.
- When sending images captured by drive recorders and road-driving robots to a server, it is possible to protect the privacy of pedestrians while maintaining a state in which AI (artificial intelligence) can recognize the postures of pedestrians passing each other.
・When transferring and publishing the data by increasing the disclosure level of the video footage collected on the server, it is possible to protect the privacy of pedestrians while maintaining a state in which AI can recognize the posture of pedestrians.
・Easy to protect the privacy of subjects in real-time video monitoring applications in monitor rooms and viewing by service providers.

(2) As shown in the example question in Figure 8, users' feelings about privacy protection vary greatly depending on the purpose of use of the video, the location where the video was taken, and other factors. For example, assuming that the video is captured by a camera mounted on a robot that moves around the city, we can ask question Q1, ``Is it okay to make it available to the public?'' and ``Is it okay for service providers to view it for the purpose of improving service quality?'' If you ask question Q2, “Is it?”, the receptivity of users will be very different. (In Figure 8, the answers to questions Q1 and Q2 are shown as A1 and A2, and the former is less acceptable from the viewpoint of privacy protection.) In this patent, as shown in the latter, the answers to questions Q1 and Q2 are shown as A1 and A2. It is assumed that the privacy protection of

Note that FIG. 8 shows a video of an unspecified number of people walking on a street corner (original image, original video) and a video to which the embodiment of the present invention is applied to protect privacy (converted image, privacy protection video). ) was shown to multiple subjects who assumed to be users, and answers A1 and A2 were shown as the answers to questions Q1 and Q2 above, indicating that users' feelings about privacy protection vary. However, reasonable privacy protection is achieved by embodiments of the present invention.

The details of the experiment shown in FIG. 8 are as follows. A converted image with privacy protection performed according to an embodiment of the present invention is created from a daytime street image (original image), and the results of evaluating user acceptability of the original image and the converted image are shown. (There were 9 types of videos, original images and converted images, so 18 clips, each 10 seconds long, were used.The experiment participants asked questions while imagining people who stood out in the video.In addition, the experiment participants had 20 (12 men and women in their 60's to 60's.) Regarding question Q1, "Is it okay for it to be shown to the public?", as shown in answer A1, only about 50% of the converted images answered "on the good side". In contrast, question Q2: "Can the service provider view it for the purpose of improving service quality? (Save all videos. Discard immediately after using for the purpose. Store for a maximum of 2 months.) ”, 90% of the answers were on the “good side” as in answer A2. Regarding privacy protection in question Q2, it is expected that even if area information such as people's outlines and clothing remains, it will be practical for privacy protection of daytime street images. Furthermore, in this experiment, we confirmed that the sense of privacy protection increases to 100% when faces are blurred, and such local privacy protection processing can also be used at the same time. In addition, by retaining area information such as a person's shape (outline) and clothing, it is possible to visually annotate the skeleton, making it easier for service providers to consider improving quality, such as by tuning posture estimators. Become.

(3) FIG. 9 is a diagram showing the hardware configuration of a general computer device 70, and each of the task execution device 10 and learning device 30 of each embodiment described above is one having such a configuration. This can be realized as the computer device 70 described above. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, a dedicated processor 72 (GPU (graphics processing unit)) that executes some or all of the instructions executed by the CPU 71 instead of the CPU 71 or in cooperation with the CPU 71. A RAM 73 serves as the main memory that provides a work area for the CPU 71 and the dedicated processor 72, a ROM 74 serves as an auxiliary memory, a communication interface 75, a display 76, a mouse, a keyboard, a touch panel, etc. It includes an input interface 77 for receiving data, and a bus BS for transmitting and receiving data therebetween.

Each part of the task execution device 10 and the learning device 30 can be realized by a CPU 71 and/or a dedicated processor 72 that reads a predetermined program corresponding to the function of each part from the ROM 74 and executes it. Further, the learning method by the learning device 30 can be implemented by the CPU 71 and/or the dedicated processor 72, which reads a predetermined program corresponding to each step in FIG. 5 from the ROM 74 and executes it.

10...Task execution device, 20...Learning section, 30...Learning device
1...Image conversion section, 11...Conversion NW section, 12...Quantization section, 13...Task section
2...Privacy update section, 21...gradation update section, 22...contour update section, 23...task update section

Claims (15)

A learning method for learning parameters of image conversion processing using a neural network structure,
updating parameters using an error calculated based on the degree of information reduction evaluated for the information reduction image obtained by converting the training image by the image conversion process;
learning the parameters of the image conversion process using a parameter update based on an error calculated based on a result of recognizing the information-reduced image in a task process that performs a recognition process of a predetermined task;
Through the image conversion process, the information-reduced image is converted into one having a gradation information channel and a contour information channel from which gradation information and contour information of the training image are respectively extracted,
A learning method characterized in that the degree of information reduction in the image of the gradation information channel and the image of the contour information channel is evaluated in updating parameters using an error calculated based on the degree of information reduction. A step function that outputs pixel values as discrete values is set in the output layer of the neural network that performs the image conversion process, and the information-reduced image is converted as having pixel values of the discrete values. The learning method according to claim 1. The step function is used during forward propagation of errors, and a continuous function that simulates the increasing behavior of the step function is used during back propagation of errors, thereby updating parameters. The learning method according to claim 2. 4. The learning method according to claim 2, wherein the number of gradations, which is the number of discrete values that can be output by the step function, is set to a value of 2 or more and 64 or less. The step function can be set to a plurality of step functions that differ in the number of gradations that are the number of discrete numbers that can be output and/or the continuous function that is the target to be simulated by the step function,
5. The learning method according to claim 2, wherein learning is performed while switching between the plurality of step functions. 6. The information-reduced image is obtained in the form of a general color image by assigning a gradation information channel and a contour information channel to channels in a general color image such as RGB or YCbCr. The learning method described in either. In parameter updating using an error calculated based on the information reduction degree,
Learning according to any one of claims 1 to 6, characterized in that a difference between the image of the gradation information channel and a flattened reference image generated by flattening the training image is evaluated as an error. Method. In parameter updating using an error calculated based on the information reduction degree,
The image of the gradation information channel is taken as a fake image, the flattened reference image generated by flattening the training image is taken as a real image, and a GAN (Generative Adversarial Network) is used to distinguish between the fake image and the real image. 8. The learning method according to claim 1, wherein parameters are updated for a generator that obtains an image of a gradation information channel as a fake image through the image conversion process through learning of a discriminator. 9. The flattened reference image is generated with the number of gradations of the flattened reference image being larger than the number of gradations of the image of the gradation information channel. How to learn. In parameter updating using an error calculated based on the information reduction degree,
10. The method according to claim 1, wherein a difference between the image of the contour information channel and a contour reference image generated by applying a contour extraction filter to the training image is evaluated as an error. How to learn. In parameter updating using an error calculated based on the information reduction degree,
The image of the contour information channel is a fake image, the contour reference image generated by applying a contour extraction filter to the training image is a real image, and a GAN (Generative Adversarial Network) is used to distinguish between the fake image and the real image. ) The learning method according to any one of claims 1 to 10, characterized in that parameters are updated for a generator that obtains an image of the contour information channel as a fake image by the image conversion process through learning of a discriminator of . 12. Parameter updating using an error calculated based on the information reduction degree and parameter updating using an error calculated based on a result recognized in the task processing are performed alternately. The learning method described in any of the above. Regarding the predetermined task, after learning the parameters of the image conversion process using parameter updating based on an error calculated based on the recognition result in the first task process that performs the recognition process of the first task, further:
The method is characterized in that parameters for a second task process that performs a recognition process for a second task different from the first task are learned using an information-reduced image obtained by converting a training image by the learned image conversion process. The learning method according to any one of claims 1 to 12. An image conversion device characterized in that an information-reduced image is obtained by converting an input image using parameters for image conversion processing learned by the learning method according to any one of claims 1 to 13. A program that causes a computer to execute the learning method according to any one of claims 1 to 13, or causes a computer to function as an image conversion device according to claim 14 .

Images