JP7280210B2 - Learning method, device and program - Google Patents

Description

The present invention relates to a learning method, apparatus, and program for learning image conversion processing capable of obtaining an image that ensures task recognition accuracy, privacy protection, and compression efficiency.

In cases where image/audio data that may contain user privacy information is sent to the cloud and analyzed using machine learning such as a neural network, it is necessary to prevent infringement of user privacy. For example, with respect to audio data, if the service provider views the content of the smart speaker sent to the cloud, even if it is for technical purposes such as improving the accuracy of machine learning, the result As a result, a violation of privacy may occur.

In addition, in such a smart speaker, the user's privacy is generally protected by data encryption from eavesdropping of the communication channel. However, since the cloud side decrypts the encrypted data, the situation described above can occur.

As disclosed in the news release article at the following URL, "The world's first realization of secure computation technology that enables standard deep learning processing while encrypted", re-learning and fine-tuning are performed while encrypted on the cloud side There is also a method of performing processing such as tuning.
https://www.ntt.co.jp/news2019/1909/190902a.html

The first problem here is that the service provider cannot confirm the image and audio data by perception. In fact, there are tasks that should be performed by human perception, such as investigating the cause of problems and making mistakes in machine learning. For example, there are cases where the service provider side wants to visually exclude poisoning data or respond to complaints from users, but such confirmation may be difficult. The second issue is that even if encrypted, raw data is included, so users tend to worry that raw data may be leaked due to attacks or operational errors. .

On the other hand, with respect to image data, a technique has been conventionally used to protect privacy by performing image processing such as blurring and replacement on sensitive information considered to be private. Although the user feels relieved that the raw data is not provided, the accuracy of the image analysis task on the service provider's side tends to be very low.

In recent years, attempts have been made to generate a privacy image without lowering the analysis accuracy of a task by machine learning such as a neural network as much as possible. Such a method allows cloud administrators and service providers to perceive images while maintaining task accuracy to some extent, and also protects user privacy. Since the user does not have to send the original image, it is thought to have the effect of lowering the psychological barrier to using the service.

For example, in the method of Patent Document 1, by estimating facial organs and facial orientation and replacing the face with an avatar, it is possible to protect privacy and maintain action recognition accuracy regarding driving. Similarly, in the method of Non-Patent Document 1, privacy can be protected and the accuracy of action recognition can be maintained by recreating a face area with a face different from that of the person using a GAN (hostile generation network).

The methods of Patent Document 1 and Non-Patent Document 1 are methods of replacing a partial privacy region of an image such as a face, and do not consider the privacy of the entire image. For example, it is not possible to respond to a request to erase the reality of the whole because the clothes, the skin type, the state of the room, etc., which are unnecessary for the service are not erased.

As a method capable of erasing/reducing the overall reality, the method of Non-Patent Document 2 performs action recognition from moving images from low-resolution images. Since it is a low resolution, there is an advantage that the image file size can be reduced. However, it is difficult to perform tasks other than simple action recognition from simple low-resolution videos, and applicable tasks are limited.

On the other hand, Non-Patent Document 3 learns and generates a model that transforms the entire original image using an adversarial learning framework so as to approach a target image in which a large amount of random noise is inserted. By using an adversarial learning framework, we can learn an image transformation model that is close to the target image with random noise while maintaining the accuracy of the task. Tasks include, for example, image recognition, recognition of facial features, and the like.

With this method, it is easy to hide elements unnecessary for task analysis from the entire converted image, and it is easy to respond to requests for privacy such as erasing the reality of the whole. On the other hand, this method can also suppress the deterioration of task accuracy.

Japanese Patent Publication No. 2018-528536

Ren, Zhongzheng, Yong Jae Lee, and Michael S. Ryoo. "Learning to anonymize faces for privacy preserving action detection." Proceedings of the European Conference on Computer Vision (ECCV). 2018. Ryoo, Michael S., et al. "Privacy-preserving human activity recognition from extreme low resolution." Thirty-First AAAI Conference on Artificial Intelligence. 2017. Kim, Tae-hoon, et al. "Training with the Invisibles: Obfuscating Images to Share Safely for Learning Visual Recognition Models." arXiv preprint arXiv:1901.00098 (2019).

However, even the technique of Non-Patent Document 3, which can cope with various demands as described above, has the following problems.

That is, in the technique of Non-Patent Document 3, random noise tends to appear in an image converted for privacy. Image compression is not taken into consideration, and a random noise transformed image has large spatial frequency components in each component ranging from low frequency components to high frequency components, and the compression efficiency is greatly deteriorated. There is In image analysis, reducing the amount of space required to transmit and store large numbers of still and moving images has become an issue. If it is bad, it cannot meet the demand for file size reduction.

SUMMARY OF THE INVENTION In view of the above problems of the prior art, it is an object of the present invention to provide a learning method, an apparatus, and a program for learning an image conversion process capable of obtaining an image ensuring task recognition accuracy, privacy protection, and compression efficiency. do.

In order to achieve the above object, the present invention provides a learning method for learning weight parameters for image transformation processing using a neural network structure, wherein a privacy protection image obtained by transforming a training image by the image transformation processing is used for recognition of a predetermined task. and a second cost for the similarity evaluation result between the compressed image obtained by compressing the training image and the privacy protection image. It is characterized by learning a weight parameter of the image conversion process. Further, the present invention is characterized by being a learning device corresponding to the learning method and a program for causing a computer to execute the learning method.

According to the present invention, by learning using the first cost and the second cost, it is possible to learn an image conversion process that can obtain an image that ensures task recognition accuracy, privacy protection, and compression efficiency. can.

FIG. 11 is a functional block diagram of a conventionally implemented configuration, which is a configuration when a task is executed (inference) by a conventional method; FIG. 4 is a functional block diagram of a conventional learning configuration, which is a configuration for learning weighting parameters of a conventional image conversion unit; It is a flowchart of learning by the conventional learning structure, and shows the procedure of learning using GAN which is an existing method. FIG. 4 is a functional block diagram of a recognition device that is configured when executing a task (during inference) according to an embodiment; FIG. 4 is a functional block diagram of a learning device according to an embodiment, which is a configuration for learning weighting parameters of an image conversion unit; 10 is a flowchart of learning by the learning device according to one embodiment, showing a procedure for performing learning using GAN, which is an existing method. FIG. 6 is a functional block diagram of a learning device according to another embodiment different from FIG. 5; 8 is a flow chart of learning by the learning device 20 according to one embodiment in the configuration of FIG. 7. FIG. It is a figure which shows the hardware constitutions in a common computer apparatus.

Before describing the present embodiment, the method of Non-Patent Document 3 (hereinafter referred to as the “conventional method”) will be briefly described as a comparison. FIG. 1 is a functional block diagram of a conventional implementation configuration 100, which is a configuration at the time of task execution (at the time of inference) in a conventional method. have

The conventional image conversion unit 101 serves as an image obfuscator, converts an image to be converted (an image provided by the user and subject to privacy protection), and outputs a privacy-protected image. The conventional compression unit 102 outputs a compressed privacy protection image by compressing the privacy protection image using an existing method. The conventional task unit 103 decodes the compressed privacy-protected image, performs a predetermined recognition task (for example, pose estimation or image recognition), and outputs a recognition result (for example, pose estimation result or image recognition result).

As already explained, the privacy-preserving image (or the compressed privacy-preserving image for storing or transmitting it) has been converted to a privacy-preserving state (obfuscated state), and the conventional task unit The recognition accuracy in 103 is also an image that can secure a certain accuracy. However, there is a problem that the compression efficiency is low and the file size of the compressed privacy protection image becomes large.

In order to perform the task by the conventional implementation configuration 100 of FIG. It is necessary to learn and obtain the weight parameter. FIG. 2 is a functional block diagram of a conventional learning configuration 200, which is a configuration for learning the weighting parameters of the conventional image conversion unit 101. As shown in FIG. As shown, the conventional learning configuration 200 includes a conventional image conversion unit 101 , a conventional task unit 103 , a conventional first evaluator 104 , a target image generator 201 , a conventional identification unit 203 and a conventional second evaluator 204 .

1 and 2, the conventional image conversion unit 101 and the conventional task unit 103, which are respectively present in both FIGS. 1 and 2, have the same configuration in FIGS. However, the weighting parameters of the conventional image conversion unit 101 are sequentially updated by learning by the conventional learning configuration 200, and the conventional image conversion unit 101 configured with the weighting parameters when the learning is completed uses the conventional implementation configuration shown in FIG. 100 will be used.

On the other hand, the conventional task unit 103, which is common in FIGS. At the time of learning by the conventional learning configuration 200, it is assumed that learning has already been completed, and the weighting parameters are determined. (That is, in learning by the conventional learning configuration 200, the weighting parameters of the conventional task section 103 are not learned and updated.)

FIG. 3 is a flowchart of learning by the conventional learning configuration 200, showing the procedure of learning using the framework of adversarial learning, which is an existing method. At the start of the flow, an initial value is set as a random value or the like for the weight parameter of the conventional image conversion unit 101 (and the conventional identification unit 203) to be learned. When the flow is started, in step S101, the conventional discriminating unit 203 is trained by the conventional learning configuration 200 in the GAN connection configuration, and after updating the weight parameter, the process proceeds to step S102.

The configuration of GAN connection means that the conventional task unit 103 and the conventional first evaluation unit 104 are omitted from the conventional learning configuration 200, and the conventional image conversion unit 101, the target image generation unit 201, the conventional identification unit 203, the conventional second evaluation unit 204 only. Specifically, in step S101, the weight parameter of the conventional identification unit 203 is updated by a series of learning procedures shown in (101) to (104) below.

(101) A fake image is generated by applying conversion processing by the conventional image conversion unit 101 to a training image given as learning data, and noise is superimposed on the training image by the target image generation unit 201. A real image is generated by applying a process (noise superimposition process in which Gaussian noise randomly changes color for each pixel neighborhood). Here, for example, half are generated as real images and the remaining half are generated as fake images, and these are obtained as a mini-batch.

(102) The conventional identification unit 203 identifies the fake image and the real image obtained as the mini-batch. output). Note that the conventional identification unit 203 is configured with a predetermined convolutional neural network or the like as a unit that performs the task of identifying a real image and a fake image (discriminator for discriminating authenticity). A weight parameter is what is learned. (Therefore, the fake image and the real image that make up the mini-batch are input to the conventional identification unit 203, but correct information as to whether the input image is a real image or a fake image is not given. , the conventional identification unit 203 obtains the identification result by itself.)

(103) The conventional second evaluation unit 204 receives the identification result of the conventional identification unit 203 and receives the correct answer given in advance as learning data (correct answer as to which of the images in the mini-batch is the real image and which is the fake image). and a predetermined identification cost function that gives a low cost value if the identification result is correct and a high cost value if the identification result is not correct, thereby calculating the cost for the identification result.

The cost function for identification in the conventional second evaluation unit 204 is used to improve the accuracy with which the conventional identification unit 203 discriminates authenticity.

(104) After performing the processes (101) to (103) above for a plurality of training images, that is, forward propagation calculation of the cost (error), the cost is used in the reverse direction, the conventional second evaluation unit 204→By calculating the error backpropagation of the conventional identification unit 203, the weight of the conventional identification unit 203 is obtained using an optimizer such as stochastic gradient descent method (hereinafter referred to as “stochastic gradient descent method”). Update parameters. This update is expected to improve the accuracy with which the conventional identification unit 203 distinguishes authenticity.

In step S102, the weighting parameters of the conventional image conversion unit 101 are updated by learning in the GAN connection configuration and the task connection configuration, and then the process proceeds to step S103. The GAN connection configuration is as described in step S101, while the task connection configuration is defined as a configuration that includes only the conventional image conversion unit 101, the conventional task unit 103, and the conventional first evaluation unit 104 in the conventional learning configuration 200. be. Specifically, the weighting parameters of the conventional image conversion unit 101 are updated in step S102 by a series of learning procedures indicated by (201) to (203A) or (203B) below. As described below, the procedure (203B) is also possible as a modified example of the procedure (203A), and either one may be used. (203A) is a method of alternately learning the conventional image conversion unit 101 by alternately calculating the output cost of the conventional first evaluation unit 104 and the output cost of the conventional second evaluation unit 204, and (203B) is a method of learning these This is a method of learning the conventional image conversion unit 101 from the total cost calculated from the two output costs.

(201) In the GAN connection configuration, the procedures of (101) to (103) described with respect to step S101 are performed on training images given as learning data. However, at this time, (103) is performed as a procedure (103′) changed from the procedure in step S101. The cost function for identification used in (103) is changed to a cost function for identification failure that performs the opposite evaluation. That is, using a predetermined identification failure cost function, a high cost value is given if the identification result obtained by the conventional identification unit 203 is correct, and a low cost value is given if it is not correct (identification fails) , the evaluation in the conventional second evaluation unit 204 is performed, and the cost for the identification result is calculated.

Note that the cost function for identification failure in the conventional second evaluation unit 204 is used to improve the accuracy of fake image generation in the conventional image conversion unit 101 so that the conventional identification unit 203 fails to distinguish authenticity. It is.

(202) In the task connection configuration, the conventional image conversion unit 101 converts training images given as learning data to obtain fake images, and the conventional task unit 103 recognizes the fake images to obtain recognition results. This recognition result is evaluated by collating it with the correct answer given as learning data in the conventional first evaluation unit 104, and the cost for the recognition result is calculated. The cost is calculated using a predetermined cost function in the conventional first evaluation unit 104 so that if the recognition result is correct, the cost is low, and if the recognition result is not correct, the cost is high.

(203A) Divide a plurality of training images into mini-batches, perform the processing of (201) or (202) above for each batch, that is, the forward propagation calculation of the cost (error), and in the connection configuration, the conventional first evaluation The cost output by the unit 104 or the cost output by the conventional second evaluation unit 204 is calculated. In the case of the GAN connection configuration, using the cost output from the conventional second evaluation unit 204, the error of the conventional second evaluation unit 204→conventional identification unit 203→conventional image conversion unit 101 is reversed (on the GAN connection configuration). By performing backpropagation calculation, the weighting parameters of the conventional image conversion unit 101 are updated using the stochastic gradient descent method or the like. In the case of the task connection configuration, using the cost output from the conventional first evaluation unit 104, the error backpropagation method of the conventional first evaluation unit 104→conventional task unit 103→conventional image conversion unit 101 is used in the reverse direction. By performing the calculation, the weighting parameters of the conventional image conversion unit 101 are updated using the stochastic gradient descent method or the like. The above-described backpropagation of the GAN connection configuration and backpropagation of the task connection configuration are alternately performed, and the weighting parameters of the conventional image conversion unit 101 are learned.

(203B) When using the total cost, the processing of (201) and (202) above for a plurality of training images (common training images for the GAN connection configuration and the task connection configuration), that is, the cost (error ), and for each training image common to both connection configurations, a predetermined weighted sum of the cost output by the conventional first evaluation unit 104 and the cost output by the conventional second evaluation unit 204 is , and using the total cost in the reverse direction (on the GAN connection configuration), the conventional second evaluation unit 204 → the conventional identification unit 203 → the conventional image conversion unit 101 and (on the task connection configuration) In the direction, the conventional first evaluation unit 104→conventional task unit 103→conventional image conversion unit 101 calculates the weight parameters of the conventional image conversion unit 101 using the stochastic gradient descent method or the like by performing error backpropagation calculation. You may update. (In the conventional method, when configuring GAN connection, the total cost is used that slightly considers the error of the task unit. In the case of task connection configuration, the total cost is not used.)

By updating (203A) or (203B), the fake image obtained by conversion by the conventional image conversion unit 101 causes the conventional identification unit 203 to fail to discriminate authenticity (that is, the fake image obtained by the target image generation unit 201 It is expected that the accuracy will be improved (similar to a real image) and the recognition accuracy in the conventional task unit 103 will also be improved (that is, the image will be in a state suitable for recognition processing). .

In step S103, it is determined whether or not the learning has converged. By ending the flow of 3 and returning to step S101 if not converged, the learning (steps S101 and S102) as described above will be further continued. For the determination of convergence in step S103, for example, by using a test image separate from the training image, the total cost of procedure (203B) or the conventional first and second evaluation units 104 and 204 of procedure (203A) output The cost may be calculated to evaluate the accuracy of the learning model, and determination may be made based on whether or not the improvement in accuracy (improvement history) has converged. A simple predetermined number of epochs or the like may be used as the convergence condition.

As described above, in the method of Non-Patent Document 3, as shown in FIG. 2 and FIG. , the learning result of the conventional image conversion unit 101 (and the conventional identification unit 203) can be obtained.

Hereinafter, an embodiment of the present invention will be described as a method in which the method of Non-Patent Document 3 is improved in consideration of the image compression rate.

FIG. 4 is a functional block diagram of the recognition device 10, which is a configuration when executing a task (during inference) according to one embodiment.

The image conversion unit 11 serves as an image obfuscator, converts an image to be converted (an image provided by the user and subject to privacy protection), and outputs a privacy-protected image. Compression unit 21 outputs a compressed privacy protection image by compressing the privacy protection image using an existing method. At this time, the compression unit 21 can perform compression according to predetermined compression settings designated by the user. The task unit 13 decodes the compressed privacy-protected image, performs a predetermined recognition task (for example, pose estimation or image recognition), and outputs a recognition result (for example, pose estimation result or image recognition result).

In the present embodiment, as in the conventional method of Non-Patent Document 3, a privacy-protected image (or a compressed privacy-protected image for storing or transmitting it) is converted to a privacy-protected state (obfuscated state). It is an image that has been converted and that can ensure a certain level of recognition accuracy in the task unit 13 .

On the other hand, in this embodiment, unlike the conventional method, the obtained image has excellent compression efficiency. By (irreversible) compression, the file size can be kept small without significantly changing the quality from before compression.

In order for the recognition device 10 in FIG. 4 to perform the task, it is necessary to previously learn the image transforming unit 11, which is composed of a convolutional neural network or the like, and obtain its weighting parameters. FIG. 5 is a functional block diagram of the learning device 20 according to one embodiment, which is the configuration for learning the weighting parameter of the image conversion unit 11. As shown in FIG. As illustrated, the learning device 20 has an image conversion unit 11, a task unit 13, a first evaluation unit 14, a compression unit 21, an identification unit 23 and a second evaluation unit .

4 and 5, the image conversion section 11, the compression section 21 and the task section 13, which are respectively present in both FIGS. However, the weight parameters of the image conversion unit 11 are sequentially updated by learning by the learning device 20, and the image conversion unit 11 configured with the weight parameters when learning is completed is used in the recognition device 10 of FIG. become a thing.

On the other hand, the task unit 13, which is common to FIGS. The weighting parameters are already determined when learning is performed by the device 20, assuming that learning has already been completed. (That is, in learning by the learning device 20, the weighting parameters of the task unit 13 are basically not learned and updated, but if the task accuracy cannot be sufficiently maintained, fine tuning is performed to We may update the 13 weighting parameters, which reduces task accuracy for normal images, but improves task accuracy for image transformer output images.)

FIG. 6 is a flowchart of learning by the learning device 20 according to one embodiment, and shows a procedure of learning using GAN, which is an existing method. 6 by the learning device 20 consists of steps S11, S12 and S13, which respectively correspond to steps S101, S102 and S103 of FIG. 6 are replaced with those of the learning apparatus 20 as follows, and steps S101, S102, and S103 are executed, respectively, corresponding to steps S11, S12, and S13 in FIG. (Therefore, each step in FIG. 6 corresponds to each step in FIG. 3 by changing the functional unit that is the subject of processing from that in FIG. 2 to that in FIG. 5, so redundant description will be omitted. )

That is, if the correspondence relationship of reading is shown in the form of "the configuration of the conventional learning configuration 200 before the reading change→the configuration of the learning device 20 after the reading change", the "target image generation unit 201→the compression unit 21" and the "conventional identification unit 203 → identification unit 23”, “conventional second evaluation unit 204→second evaluation unit 24”, “conventional image conversion unit 101→image conversion unit 11”, “conventional task unit 103→task unit 13”, and “conventional first evaluation Section 104→first evaluation section 14" can be read as a corresponding relationship. GAN connections and task connections during learning are similarly defined by these replacements.

In the above correspondence relationship, only the "target image generating unit 201 and the compressing unit 21" are in a relationship of performing mutually different processes, and all others are in a relationship of performing the same processes in each step during learning. In other words, in this embodiment, the learning device 20 is prepared by replacing the target image generation unit 201 of the conventional learning configuration 200 with the compression unit 21, and each step in FIG. 6, which is the same as each step in FIG. By executing in the learning device 20, as a result, the image conversion unit 11 whose weight parameter is learned has excellent compression efficiency and privacy protection as described in the recognition device 10 of FIG. 4, and An image suitable for recognition processing by the task unit 13 can be obtained as an outputtable image.

In the present embodiment, providing the compression unit 21 in the learning device 20 (in place of the target image generation unit 201 used in the conventional method) as described above is based on the following unique knowledge. is. That is, the compression unit 21 obtains a real image from the training image by performing irreversible compression such as JPEG according to the compression setting specified by the user. Here, since irreversible compression is accompanied by deterioration, it is knowledge that a real image whose data size has been reduced by irreversible compression can be used as it is as a privacy protection image.

Therefore, according to the flow of FIG. 6, which is in line with the adversarial learning framework, the image conversion unit 101 uses an image that is similar to the image compressed by the compression unit 21 and that has high compression efficiency and privacy protection. Assuming that there is an image suitable for recognition by the task unit 13 and can output an image that is suitable for recognition by the task unit 13, it is possible to learn the weight parameter together with the identification unit 23, which is in a hostile relationship.

FIG. 7 is a functional block diagram of the learning device 20 according to another embodiment different from FIG. 5, and FIG. 8 is a flow chart of learning by the learning device 20 according to the embodiment in the configuration of FIG.

In the embodiments of FIGS. 5 and 6, the adversarial learning framework is used to learn the weight parameters of the image transforming unit 11, whereas in the embodiments of FIGS. 7 and 8, the adversarial learning framework is not used. It is possible to learn the weighting parameters of the image transforming unit 11 without the need. By not using the adversarial learning framework, the learning device 20 in FIG. 7 has a configuration in which the identification unit 23 is removed from the configuration in FIG.

The second evaluation unit 24 in the learning device 20 in FIG. 7 performs the following processing as processing different from the processing in FIG. 5 (processing for evaluating the identification result of the identification unit 23). That is, the second evaluation unit 24 in FIG. 7 reads the compressed image obtained by compressing the training image by the compression unit 21 and the privacy protection image obtained by converting the training image by the image conversion unit 11, and A cost function is calculated such that the larger the difference between these two images, the larger the cost. In one embodiment, as the mean squared error (MSE, the sum of the squares of the pixel values of the difference image between the two images divided by the number of pixels) between the compressed image and the privacy protection image, the second evaluation unit 24 of FIG. Cost can be calculated. Alternatively, the cost may be calculated by averaging the number of pixels of the sum of absolute values of the difference image.

On the other hand, the processing contents of each of the compression unit 21, the image conversion unit 11, the task unit 13, and the first evaluation unit 14, which are components other than the second evaluation unit 24 in the learning device 20 of FIG. The processing content is the same as in 20. Each step of the flow in FIG. 8 will be described below.

At the start of the flow of FIG. 8, the initial values of the weight parameters of the image conversion unit 11 are set in advance. (As for the task section 13, the weighting parameters have already been learned, as in the embodiments of FIGS. 5 and 6.) When the flow of FIG. is learned, the weight parameter is updated, and then the process proceeds to step S22. In step S21, the weighting parameters of the image conversion unit 11 can be updated by a series of learning procedures specifically shown in (21) to (22A) or (22B) below. Basically, either of the procedures (22A) and (22B) may be used. (Both may be used.)

(21) A training image given as learning data is converted by the image conversion unit 11 to obtain a privacy protection image, and the training image is compressed by the compression unit 21 to obtain a compressed image. A cost is calculated by evaluating the privacy-protected image and the compressed image in the second evaluation unit 24 . Further, the task unit 13 recognizes the privacy-protected image to obtain a recognition result, and the first evaluation unit 14 compares the recognition result with the correct answer given as learning data to evaluate the cost of the recognition result. calculate. If the recognition result is close to the correct answer (similar to the conventional first evaluation unit 104 in FIGS. 2 and 3, which is the same as the first evaluation unit 14 in FIGS. 5 and 6), the cost is set to a low cost value, is calculated using a predetermined cost function in the first evaluation unit 14 so that a high cost value is obtained if it is not close to .

(22A) Divide a plurality of training images into mini-batches, perform forward propagation calculation of cost (error) for each batch, and calculate the cost output by the first evaluation unit 14 or the cost output by the second evaluation unit 24 for each batch. to calculate In the case of the GAN connection configuration (defined as the configuration of the image conversion unit 11, the compression unit 21 and the second evaluation unit 24 in FIG. 7 corresponding to FIG. 5 (the configuration excluding the identification unit 23 in FIG. 5)), Using the cost output by the second evaluation unit, the image conversion unit 11 is calculated using the stochastic gradient descent method or the like by performing error backpropagation calculation in the reverse direction from the second evaluation unit 24 to the image conversion unit 11. Update the weight parameter of . In the case of the task connection configuration (defined in the same manner as in FIG. 5 with respect to FIG. 7), the costs output by the first evaluation unit 14 are used in the opposite direction, the first evaluation unit 14→task unit 13→image conversion unit 11, the weighting parameters of the image conversion unit 11 are updated using the stochastic gradient descent method or the like. The backpropagation of the GAN connection configuration and the backpropagation of the task connection configuration are alternately performed, and the weight parameter of the image conversion unit 11 is learned.

(22B) When using the total cost, perform forward propagation calculation of the cost (error) for multiple training images (common training images for the GAN connection configuration and the task connection configuration), and in both the connection configurations For each common training image, calculate a total cost as a predetermined weighted sum of the cost output by the first evaluation unit 14 and the cost output by the second evaluation unit 24, and use the total cost (GAN In the connection configuration), the error backpropagation calculation is performed in the reverse direction of the second evaluation unit 24 → image conversion unit 11 and the first evaluation unit 14 → task unit 13 → image conversion unit 11, thereby stochastic gradient descent The weighting parameter of the image conversion unit 11 may be updated using a method or the like.

7 and 8 as well as in the case of using the adversarial learning framework according to FIGS. The privacy-protected image obtained by the conversion in is similar to the image compressed by the compression unit 21, thereby satisfying the requirements for privacy protection and file size reduction and becoming an image suitable for recognition by the task unit 13. Be expected.

In step S22, it is determined whether or not the learning has converged, and if it has converged, the weight parameter of the image conversion unit 11 at that time is obtained as the final learning result, and the flow of FIG. If not, by returning to step S21, learning as described above (step S21) is further continued. The convergence determination in step S22 is performed in the same manner as step S103 in FIG. 3 and step S13 in FIG. ), the accuracy of the learning model is evaluated by calculating the cost of the first evaluation unit 14 and the cost of the second evaluation unit 24, and determination is made based on whether or not the improvement in accuracy (improvement history) has converged. Alternatively, learning may simply be rounded up at a predetermined number of epochs.

As described above, according to each embodiment of the present invention described with reference to FIGS. Using the first cost evaluated by the unit 14 and the second cost evaluated by the second evaluation unit 24 for the similarity between the compressed image obtained by converting the training image by the compression unit 21 and the privacy protection image, the neural network structure is constructed. By learning the weighting parameters of the image conversion unit 11, the image conversion unit 11 obtained as a learning result outputs an image excellent in all three points of privacy protection, compression efficiency, and recognition performance in the task unit 13. becomes possible. As already explained, as learning using the first cost and the second cost, it is possible to alternately minimize each cost or to minimize the total cost obtained as a weighted sum. be.

Various supplementary examples and additional examples will be described below.

(1) The compressing unit 21 commonly used in each of the embodiments shown in FIGS. 4 to 8 may be modified as follows. The compression by the compression unit 21 is performed by, for example, image compression using frequency transform (JPEG using DCT (discrete cosine transform) as a base, JPEG2000 using wavelet transform as a base, etc.) to achieve the following (i) to (iii). This can be done under user-specified compression settings in terms.

(i) In the case of JPEG or JPEG2000 compression, setting may be made so as to obtain a low-quality compressed image compressed so that the quality value is less than half of the whole. In the case of JPEG, many high frequency components become 0 due to quantization. This makes it easier to preserve the privacy of edges, such as clothing textures, and provides a higher compression ratio.

(ii)-a The DCT component of JPEG and the minimum frequency component after wavelet transform of JPEG2000 are all set to the same value (eg, the intermediate value is desirable, 0.5 for gradation 0-1). This eliminates most of the gradation, making it easier to protect the privacy of the skin. In addition, the reduction of DCT coefficient information also increases the compression rate.

(ii)-b The DCT component of JPEG and the minimum frequency component after wavelet transform are set to several values (eg, 2 to 8 values). This eliminates fine gradations and makes it easier to protect the privacy of the skin. In addition, the reduction of DCT coefficient information also increases the compression ratio.

That is, in (ii)-a, the minimum frequency component after transformation (original value) is rewritten to the same value, and in (ii)-b, the minimum frequency component after transformation is roughly quantized. Normally, low-frequency components are finely quantized and not coarsely quantized in order to ensure image quality.

(iii) select the frequency basis used and/or change the intensity; That is, when a basis is selected, the transform coefficients of a predetermined basis that have not been selected are deleted. When changing the intensity (transform coefficient), the transform coefficient of the predetermined base is forcibly rewritten to a constant value, or the absolute value of the coefficient is changed. For example, other than the DC component of DCT, the strength is weakened by decreasing the absolute value of the coefficient.

Basically, it is good to select/combine (i)-(iii) as needed so that the accuracy of the task is less likely to decrease. For general action recognition and image recognition tasks, (i) and (iii) are likely to be suitable for detecting movement and gradation, and (ii) is likely to be suitable for tasks to extract key points of faces and skeletons. be done. Also, if the compatibility between the task and (i)-(iii) is unknown, multiple frequency bases may be selected in (iii). For example, divide into low frequency, high frequency, and intermediate frequency components, learn the obfuscator (image conversion part) only with each frequency component, or only the frequency component of any two areas, and measure the accuracy deterioration degree of the task After obtaining, the frequency components to be used may be determined. The number of divisions of frequency components is not limited to three.

For example, when a face is photographed at the same size, spatial frequency analysis using FFT (Fast Fourier Transform) shows that there is a high correlation between the intensity of the spatial frequency and age or gender. (“Relationship between facial expressions and impressions in facial images and spatial frequency characteristics”).
Patent No. 05827225 (Patent application 2012-521378)
https://www.jstage.jst.go.jp/article/itej/69/11/69_836/_pdf/-char/ja

Using this idea, the intensity of the frequencies used for compression may be varied to mask age and gender. For example, it is known that a high intensity of low frequency is easily discriminated by females, and a high intensity of high frequency is easily discriminated by males. Therefore, it is conceivable that the privacy can be protected by, for example, stepwise increasing the intensity of the high-frequency components in an image of a woman so that the image looks masculine. Such compressed images may be made for the entire training image.

(2) From the viewpoint of privacy protection and improvement of compression ratio, images whose dynamic range (the number of gradations) has been reduced in advance may be used as training images. For example, it is possible to set the average pixel value to around 128 while reducing the dynamic range, 256 gradation levels for each of RGB→eight values, and the like. In addition, the color may be reduced, or a color such as skin color that is conspicuous to humans may be converted to another color (eg, blue, purple, green, etc.).

When learning the image conversion unit 11 while further updating the learned parameters of the task unit 13 (when performing the fine tuning described above), the generated image may be directly input to the task. Before executing the task, the original dynamic range and number of colors can be restored before executing the task. Similarly, in order to protect image privacy and reduce data, before compression, color reduction, conversion of colors that stand out to humans such as skin tones to other colors (e.g. blue, purple, green, etc.), reduction of dynamic range, etc. you can go In this case, restore the original number of colors, colors, and dynamic range before executing the task, and then execute the task.

(3) When the task is posture estimation, there are various application examples for protecting privacy in a state where posture estimation is possible.
・In the case of transmitting images of exercising at home to a server, performing image analysis based on posture estimation, and receiving advice, it is possible to protect the privacy of the person, skin, clothes, room, etc. in the images taken at home.
・When transmitting images of the outside of the vehicle captured by the drive recorder to the server, it is possible to protect the privacy of pedestrians while maintaining a state where AI (artificial intelligence) can recognize pedestrians' postures such as raising their hands and falling.
・In raising the level of publicity of the video outside the vehicle captured by the drive recorder collected on the server and transferring/publishing the data, it is possible to maintain a state in which the AI can recognize the posture of pedestrians raising their hands, falling postures, etc. You can protect your privacy.
・It is possible to detect the actions of the driver and passengers (calling on a mobile phone, looking backwards, etc.) from the video taken by the drive recorder, while protecting the privacy of the driver and passengers in the car. You can protect it, send the video to the server, and publish it.

(4) The image converter 11 commonly used in each of the embodiments shown in FIGS. 4 to 8 may be modified as follows.

In the intermediate layer or output layer of the network that constitutes the image conversion unit 11, a convolution layer is inserted with a kernel of the frequency base to be compressed (common frequency base used for compression by the compression unit 21). For example, let an 8×8 DCT basis be the kernel. (The convolution layer of the frequency basis kernel may be included in only part of the specific layer, not all of it.) By setting the stride width to the compression block size in the compression unit 21, an image that is easy to compress is converted. It is expected that it will become easier to generate networks by learning. The bases may be selected in advance, or a combination of bases selected in a network with high task accuracy/compression ratio experimentally may be specified later. Moreover, you may change intensity|strength. Intensity adjustment is realized, for example, by scalar-multiplying the kernel value in advance.

The kernel value may be scalar-multiplied by 1/table value using a quantization table value such as JPEG for each quality. Since the range of table values is wide, it is better to convert to 2 to 8 values. After that, it is possible to simulate actual JPEG compression to some extent in a neural network for image generation by quantizing with a step-like activation function that can be converted to 2 to 8 values.

During learning, only the inserted frequency basis kernel does not update the weights. In other words, only the weights of other layers need to be updated with fixed values.

In addition, after the inserted DCT layer (the layer containing the frequency basis kernel), insert an upconvolution layer (a layer with fixed weights that is not updated by learning) that corresponds to inverse quantization and inverse DCT. good too. By adding an up-convolution layer, we can return to the spatial domain instead of the frequency domain, making it easier to visually confirm privacy. Backpropagation to the image converter is also possible, so it is expected that the effect of JPEG compression can be correctly estimated. Normally, the JPEG compression after the image converter is not taken into account during training, but it is expected that this will allow the effects of JPEG compression to be considered correctly to some extent.

(5) FIG. 9 is a diagram showing the hardware configuration of a general computer device 70. The recognition device 10 and the learning device 20 of each embodiment of FIGS. 4 to 8 each have such a configuration. It can be implemented as one or more computer devices 70. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, and a dedicated processor 72 (GPU (graphic processing unit)) that executes some or all of the execution instructions of the CPU 71 instead of the CPU 71 or in cooperation with the CPU 71. , dedicated processor for deep learning, etc.), RAM 73 as a main storage device that provides a work area to the CPU 71 and dedicated processor 72, ROM 74 as an auxiliary storage device, communication interface 75, display 76, mouse, keyboard, touch panel, etc. It has an input interface 77 for receiving data and a bus BS for exchanging data therebetween.

Each part of the recognition device 10 and the learning device 20 can be realized by a CPU 71 and/or a dedicated processor 72 that reads and executes a predetermined program corresponding to the function of each part from the ROM 74 . Also, the learning method by the learning device 20 can be implemented by the CPU 71 and/or the dedicated processor 72 that reads and executes a predetermined program corresponding to each step in FIG. 6 or 8 from the ROM 74 .

10... recognition device, 20... learning device
21 Compression unit 23 Identification unit 24 Second evaluation unit 11 Image conversion unit 13 Task unit 14 First evaluation unit

Claims (12)

A learning method for learning weight parameters for image conversion processing using a neural network structure,
a first cost for a recognition result obtained by recognizing a privacy-protected image obtained by converting a training image in the image conversion process in a task process for recognizing a predetermined task;
learning the weight parameter of the image conversion process using a second cost for the similarity evaluation result between the compressed image obtained by compressing the training image and the privacy protection image ;
The learning method of claim 1, wherein the second cost is evaluated based on a difference between the compressed image and the privacy protected image . A learning method for learning weight parameters for image conversion processing using a neural network structure,
a first cost for a recognition result obtained by recognizing a privacy-protected image obtained by converting a training image in the image conversion process in a task process for recognizing a predetermined task;
learning the weight parameter of the image conversion process using a second cost for the similarity evaluation result between the compressed image obtained by compressing the training image and the privacy protection image ;
Using a neural network structure-based identification process that is learned to distinguish authenticity by identifying the compressed image as a real image and the privacy-protected image as a fake image,
Further comprising learning by a generative adversarial network so that the identification process improves the accuracy of discerning authenticity and the image conversion process improves the accuracy of misleading the identification process. and learning method. 3. The learning method according to claim 1 , wherein said compression processing uses discrete cosine transform or wavelet transform. 4. The method according to any one of claims 1 to 3 , wherein said compression processing includes frequency transforming using a transform base, and replacing the frequency-transformed lowest frequency component with a constant value or quantizing. Any of the learning methods described. 5. The learning method according to any one of claims 1 to 4 , wherein said compression processing includes performing frequency transform using transform bases and deleting transform coefficients of a predetermined transform base. A learning method for learning weight parameters for image conversion processing using a neural network structure,
a first cost for a recognition result obtained by recognizing a privacy-protected image obtained by converting a training image in the image conversion process in a task process for recognizing a predetermined task;
learning the weight parameter of the image conversion process using a second cost for the similarity evaluation result between the compressed image obtained by compressing the training image and the privacy protection image ;
The compression process includes frequency transforming using a transform basis,
A learning method, wherein the layer in the neural network structure in the image transformation processing includes a fixed convolutional layer that has a transformation base used in the compression processing as a kernel and is not updated by learning. 7. The learning method according to claim 6 , wherein the stride width of said fixed convolutional layer is set to match the compression block size in said compression process. Learning the weight parameter of the image conversion process includes:
Alternately minimizing the first cost and the second cost, or minimizing a total cost calculated from the first cost and the second cost. 8. The learning method according to any one of 1 to 7 . A learning device for learning weight parameters for image conversion processing using a neural network structure,
a first cost for a recognition result obtained by recognizing a privacy-protected image obtained by converting a training image in the image conversion process in a task process for recognizing a predetermined task;
learning the weight parameter of the image conversion process using a second cost for the similarity evaluation result between the compressed image obtained by compressing the training image and the privacy protection image ;
The learning device , wherein the second cost is evaluated based on a difference between the compressed image and the privacy protection image . A learning device for learning weight parameters for image conversion processing using a neural network structure,
a first cost for a recognition result obtained by recognizing a privacy-protected image obtained by converting a training image in the image conversion process in a task process for recognizing a predetermined task;
learning the weight parameter of the image conversion process using a second cost for the similarity evaluation result between the compressed image obtained by compressing the training image and the privacy protection image ;
Using a neural network structure-based identification process that is learned to distinguish authenticity by identifying the compressed image as a real image and the privacy-protected image as a fake image,
Further comprising learning by a generative adversarial network so that the identification process improves the accuracy of discerning authenticity and the image conversion process improves the accuracy of misleading the identification process. and learning device. A learning device for learning weight parameters for image conversion processing using a neural network structure,
a first cost for a recognition result obtained by recognizing a privacy-protected image obtained by converting a training image in the image conversion process in a task process for recognizing a predetermined task;
learning the weight parameter of the image conversion process using a second cost for the similarity evaluation result between the compressed image obtained by compressing the training image and the privacy protection image ;
The compression process includes frequency transforming using a transform basis,
A learning device, wherein a layer in a neural network structure in the image transformation processing includes a fixed convolutional layer that has a transformation base used in the compression processing as a kernel and is not updated by learning. A program for causing a computer to execute the learning method according to any one of claims 1 to 8 .

Images