JP7208940B2 - Image processing device, server, image processing method, attitude estimation method, and program - Google Patents

Description

The present invention relates to an image processing device, a server, an image processing method, a posture estimation method, and a program.

In recent years, a service has been put into practical use in which a user transmits an image including himself/herself as a subject to a server, and the server executes image recognition processing. For example, in order for a user to receive guidance on yoga poses at home, a service is being considered in which a video of the user himself/herself taken with a smartphone is sent to a server, the server estimates the pose, and the result is sent to the user. .

In order to implement such a service, it is required to prevent the user's personal information from being leaked from the image that the user transmits to the server. For example, Non-Patent Literature 1 proposes a technique for obtaining, by learning, an image transformation that achieves both target orientation estimation and privacy violation estimation.

Haotao Wang, Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin, “Privacy-Preserving Deep Visual Recognition: An Adversarial Learning Framework and A New Dataset,” July 29, 2019, [accessed January 30, 2020], Internet <URL: https://arxiv.org/pdf/1906.05675.pdf>

The image transformation in the above technique is unique, and the same processing is performed for any input image. However, there are various factors in how a person perceives privacy concerns, and they differ from person to person, and even the same person can change from time to time. For this reason, simply performing a uniform privacy anonymization process does not always meet the user's desire to anonymize images.

SUMMARY OF THE INVENTION The present invention has been made in view of these points, and an object of the present invention is to provide a technique for changing an image anonymization pattern in accordance with a user's selection.

A first aspect of the present invention is an image processing apparatus. This device includes an image acquisition unit that acquires an image to be processed that includes a subject to be anonymized, a selection reception unit that receives a selection of an anonymization processing pattern to be applied to the subject, and an anonymization processing that is different for each image. The selection receiving unit receives from an anonymization model storage unit that stores a plurality of different anonymization models learned to execute the anonymization models determined for each of the plurality of anonymization processing patterns. a masking model acquisition unit that reads a masking model corresponding to a masking process pattern; a model applicator for generating.

Each of the different masking models may share a plurality of different masking sub-models that perform different types of masking processes on images, and the model application unit outputs the plurality of different masking sub-models. An image obtained by superimposing images based on a weighting coefficient determined according to the pattern of the anonymization process may be output as the anonymized image.

Each of the plurality of anonymization sub-models has at least one of a partial area set in advance for the subject and a background area set as an area other than the subject as an anonymization target area. It may be learned.

The subject may be a person, and each of the plurality of anonymization sub-models includes: (1) an input image and an output image generated by applying the anonymization sub-model to the input image; and (2) a posture learned to estimate the posture of a person included in the anonymized image obtained by superimposing each of the plurality of anonymization sub-models. Learning may be performed such that the degree of divergence increases and the estimation accuracy increases based on two evaluation functions: a posture evaluation function that indicates the level of estimation accuracy of the estimation model.

The divergence evaluation function is the degree of divergence in the anonymization target region, which is the degree of divergence in the anonymization target region between the input image and the output image generated by applying the anonymization submodel to the input image. , and the size of a non-anonymization target region deviation degree, which is a degree of deviation from the input image in a region other than the anonymization target region in the output image, and the plurality of encryption Based on the divergence evaluation function and the posture evaluation function, each of the sub-models has a large anonymization target area deviation, a small anonymization non-anonymization area deviation, and an estimation accuracy of The model application unit subtracts pixel values of areas other than the anonymization target area of each anonymization submodel from the pixel values of the output images of each of a plurality of different anonymization submodels. An image obtained by superimposing the obtained images may be output as the anonymized image.

A second aspect of the present invention is a server. The server includes an anonymized image acquisition unit that acquires, via a communication network, an anonymized image generated by the image processing apparatus and pattern information indicating a pattern of anonymization processing applied to the anonymized image. , a plurality of different sharpening models trained to perform different sharpening processes on an image, the sharpening models storing sharpening models determined for each of the plurality of sharpening process patterns. a sharpening model acquisition unit that reads a sharpening model corresponding to the pattern information from a model storage unit; A sharpening unit that generates an image, and a pose estimation unit that estimates a pose of a person included in the sharpened image by applying the pose estimation model to the sharpened image.

A third aspect of the present invention is an image processing method. In this method, the processor acquires an image to be processed that includes a subject to be anonymized, receives a selection of a pattern of anonymization processing to be applied to the subject, and performs different anonymization processing on the image. from an anonymization model storage unit that stores a plurality of different anonymization models that have been learned to be executed and that are determined for each of the plurality of anonymization processing patterns; reading out a concealment model corresponding to the pattern; and applying the read concealment model to the processing target image to generate a concealment image.

A fourth aspect of the present invention is a program. This program provides a computer with a function of acquiring an image to be processed that includes a subject to be anonymized, a function of accepting selection of a pattern of anonymization processing to be applied to the subject, and performing different anonymization processing on each image. from an anonymization model storage unit that stores a plurality of different anonymization models that have been learned to be executed and that are determined for each of the plurality of anonymization processing patterns; A function of reading an anonymization model corresponding to a pattern and a function of generating an anonymization image by applying the read anonymization model to the image to be processed are realized.

A fifth aspect of the present invention is a pose estimation method. In this method, a processor of a server connected to the above-described image processing device via a communication network generates a masked image generated by the image processing device and a pattern indicating a pattern of masking processing applied to the masked image. and a plurality of different sharpening models each trained to perform a different sharpening process on an image, the sharpening processes of the plurality of sharpening processes. reading out a sharpening model corresponding to the pattern information from a sharpening model storage unit that stores a sharpening model determined for each pattern; and applying the read out sharpening model to the concealed image. generating a sharpened image; and estimating the pose of a person included in the sharpened image by applying the pose estimation model to the sharpened image.

A sixth aspect of the present invention is a program. This program stores an anonymized image generated by the image processing device and pattern information indicating a pattern of anonymization processing applied to the anonymized image in a computer connected to the image processing device via a communication network. via the communication network, and a plurality of different sharpening models trained to perform different sharpening processes on the image, each of the plurality of sharpening process patterns A function of reading out a sharpening model corresponding to the pattern information from a sharpening model storage unit storing a sharpening model determined in and applying the read out sharpening model to the anonymized image, A function of generating a sharpened image and a function of estimating the pose of a person included in the sharpened image by applying the pose estimation model to the sharpened image are realized.

In order to provide the program of the fourth aspect and the program of the sixth aspect of the present invention, or to update a part of these programs, a computer-readable recording medium recording these programs is provided. Alternatively, these programs may be transmitted over a communication line.

Any combination of the above-described components, and expressions of the present invention converted into methods, devices, systems, computer programs, data structures, recording media, etc. are also effective as aspects of the present invention.

ADVANTAGE OF THE INVENTION According to this invention, the technique which changes the pattern of anonymization of an image according to a user's selection can be provided.

1 is a schematic diagram for explaining an outline of image processing executed by an image processing apparatus according to an embodiment; FIG. 1 is a diagram schematically showing functional configurations of an image processing apparatus and a server according to an embodiment; FIG. FIG. 4 is a diagram schematically showing the internal structure of a pattern database that stores anonymization processing patterns; FIG. 10 is a diagram for explaining a mask image used for learning an anonymization sub-model; FIG. FIG. 4 is a diagram schematically showing the structure of a learning network for learning anonymization submodels; FIG. 4 is a diagram for explaining an example of information about a subject according to the embodiment; FIG. FIG. 4 is a schematic diagram for explaining posture estimation loss; 4 is a flowchart for explaining the flow of image processing executed by the image processing apparatus according to the embodiment; FIG. 10 is a diagram schematically showing changes in an image during posture estimation processing by the server according to the embodiment; FIG. 5 is a schematic diagram for explaining the internal configuration of an image processor according to a first modified example of the embodiment;

(Overview of Embodiment)
An image processing apparatus according to an embodiment performs a plurality of different image processes on an image to be processed, and synthesizes the processing results of each image process to generate one output image. Here, the plurality of different image processes execute different types of anonymization processes on the images to be processed. Therefore, the output image generated by synthesizing the processing results of each image processing is an image obtained by subjecting the image to be processed to a plurality of different anonymization processes. Here, the image processing apparatus according to the embodiment can change the strength of the anonymization processing for each image processing, and can accept the user's selection regarding the pattern of the strength of each image processing. Thereby, the image processing apparatus according to the embodiment can change the image anonymization pattern according to the user's selection.

FIG. 1 is a schematic diagram for explaining an outline of image processing executed by an image processing apparatus according to an embodiment. FIG. 1 shows an example in which an image I to be processed includes a woman taking a yoga pose in a room.

In the example shown in FIG. 1, the image processing apparatus performs processing on an input image by an image processor F including a first image processor F1, a second image processor F2, and a third image processor F3, A first intermediate image B1, a second intermediate image B2, and a third intermediate image B3 are output, respectively. After that, the image processing device outputs the anonymized image H by weighting and superimposing the first intermediate image B1, the second intermediate image B2, and the third intermediate image B3.

In the example shown in FIG. 1, the first image processor F1 is an image processor for concealing the face area of a person captured in the image I to be processed. The second image processor F2 is an image processor for concealing the outline of the person captured in the image I to be processed. Furthermore, the third image processor F3 is an image processor for anonymizing a background area, which is an area other than the person, in the image I to be processed.

Here, the image processing apparatus prepares a plurality of different patterns P as weightings used when superimposing the first intermediate image B1, the second intermediate image B2, and the third intermediate image B3. A user's selection as to whether to adopt the pattern P is accepted. FIG. 1 shows an example in which the weight of the second intermediate image B2 is relatively large, the weight of the third intermediate image B3 is relatively small, and the weight of the first intermediate image B1 is intermediate. there is

By changing the weighting pattern P of the intermediate image B, the user changes the pattern P of the anonymization process according to his/her preference, such as emphasizing the anonymization of the face or emphasizing the anonymization of the background. be able to.

(Functional configuration of image processing apparatus 1 according to embodiment)
FIG. 2 is a diagram schematically showing functional configurations of the image processing apparatus 1 and the server 2 according to the embodiment. The image processing device 1 includes a storage unit 10 , an imaging unit 11 and a control unit 12 . The server 2 also includes a storage unit 20 and a control unit 21 . The image processing apparatus 1 and the server 2 are connected in a communicable manner via a communication network N such as the Internet.

In FIG. 2, arrows indicate main data flows, and there may be data flows not shown in FIG. In FIG. 2, each functional block does not show the configuration in units of hardware (apparatus), but the configuration in units of functions. Therefore, the functional blocks shown in FIG. 2 may be implemented within a single device, or may be implemented separately within a plurality of devices. Data exchange between functional blocks may be performed via any means such as a data bus, network, or portable storage medium.

The storage unit 10 includes a ROM (Read Only Memory) that stores a BIOS (Basic Input Output System) of a computer that implements the image processing apparatus 1, a RAM (Random Access Memory) that serves as a work area of the image processing apparatus 1, an OS ( (Operating System), application programs, and a large-capacity storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various information referred to when the application program is executed.

The imaging unit 11 is an imaging device for generating the processing target image I, and is realized using a known solid-state imaging device such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. be. The control unit 12 is a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of the image processing apparatus 1, and executes a program stored in the storage unit 10 to perform an image acquisition unit 120, a selection acceptance unit 121 , an anonymization model acquisition unit 122 , and a model application unit 123 .

The storage unit 20 includes a ROM storing the BIOS and the like of the computer that implements the server 2, a RAM serving as a work area for the server 2, an OS and application programs, and an HDD storing various information referred to when the application programs are executed. A large-capacity storage device such as an SSD. The control unit 21 is a processor such as a CPU or GPU of the server 2, and by executing a program stored in the storage unit 20, an anonymized image acquisition unit 210, a sharpening model acquisition unit 211, a sharpening unit 212, and It functions as posture estimation section 213 .

Note that FIG. 2 shows an example in which the image processing apparatus 1 and the server 2 are configured by a single apparatus. However, at least one of the image processing device 1 and the server 2 may be implemented by computational resources such as multiple processors and memories, such as a cloud computing system. In this case, each unit constituting the control unit 12 or the control unit 21 is realized by executing a program by at least one of a plurality of different processors.

The image acquisition unit 120 acquires a processing target image I including a subject to be concealed. The image acquiring unit 120 may acquire the image captured by the imaging unit 11 as the processing target image I, read it from a memory card (not shown), or acquire it from another PC by wire or wirelessly. good too.

The selection accepting unit 121 accepts selection of the anonymization processing pattern P to be applied to the subject. Specifically, the selection accepting unit 121 accepts a selection from the user via an input interface (not shown) of the image processing apparatus 1 such as a mouse, keyboard, or touch panel.

The anonymization model acquisition unit 122 reads the anonymization model corresponding to the anonymization processing pattern P accepted by the selection reception unit 121 from the anonymization model storage unit that stores a plurality of different anonymization models. Here, each anonymization model is determined for each of a plurality of anonymization processing patterns P, and is learned so as to execute different anonymization processing for each image. Note that in the example shown in FIG. 2, the storage unit 10 also serves as an anonymization model storage unit.

The model application unit 123 generates a masked image H by applying the masked model read by the masked model acquisition unit 122 to the image I to be processed. Accordingly, the image processing apparatus 1 can anonymize the processing target image I using the anonymization processing of the pattern P according to the user's selection. Details of the server 2 will be described later.

(Details of masking model)
As described with reference to FIG. 1, each masking model has a plurality of different masking sub-models (image processor F in FIG. 1) that perform different types of masking processes on images. there is Specifically, each anonymization model has an anonymization model that is not different from each other, and each anonymization model shares an anonymization sub-model.

Hereinafter, in this specification, the model for anonymizing the face area of the person imaged in the processing target image I is a "first anonymization submodel F1", and the contour of the person imaged in the processing target image I is A model for anonymization is a "second anonymization sub-model F2", a model for anonymization of a background region other than a person in the image I to be processed is a "third anonymization sub-model F3", and each When the anonymization sub-model is not particularly distinguished, it is described as "anonymization sub-model F". In this sense, the anonymization submodel F and the first to third anonymization submodels F1 to F3 respectively correspond to the image processor F and the first to third image processors F1 to F3 in FIG. Note that the processing content of each anonymization submodel is an example, and other processing may be performed.

The model application unit 123 anonymizes an image generated by superimposing an intermediate image B, which is an output image of a plurality of different anonymization submodels F, based on a weighting coefficient determined according to an anonymization processing pattern P. Output as image H. As a result, even if each anonymization model has the same anonymization sub-model F, the image processing apparatus 1 can generate different anonymization images H by changing the weighting pattern P.

FIG. 3 is a diagram schematically showing the internal structure of a pattern database that stores anonymization processing patterns P. As shown in FIG. The pattern database is managed by the model application unit 123 and stored in the storage unit 10. FIG. FIG. 3 exemplifies the weights of four patterns P of anonymization processing, namely, "face-oriented type", "outline-oriented type", "background-oriented type", and "balanced type".

For example, in the "face-oriented" pattern P, the weight of the first intermediate image B1 is 0.7, the weight of the second intermediate image B2 (0.15), and the weight of the third intermediate image B3 (0.15). bigger than As described above, the first intermediate image B1 is an image that has undergone image processing for concealing the face region of the person captured in the image I to be processed. Therefore, the fact that the weight of the first intermediate image B1 is higher than the other weights means that in the anonymized image H, the anonymization of the person's face area is emphasized. The same applies to other patterns P as well. In this way, by storing a plurality of anonymization processing patterns P in the pattern database in advance, the user can change the anonymization processing pattern P to be applied to the processing target image I simply by performing a selection operation. can be done.

(Learning of anonymous submodel F)
Each anonymization sub-model F used in the anonymization process according to the embodiment is created by a machine learning technique using known Generative Adversarial Networks (GAN), which is a type of neural network. The method of learning the anonymization sub-model F will be described below. Since the GAN is already known, a detailed description of the GAN itself is omitted, and data and evaluation functions used mainly for learning will be described below.

FIGS. 4(a) to 4(d) are diagrams for explaining the mask image M used for learning the anonymization submodel. Specifically, FIG. 4A is a diagram showing an example of teacher data T for generating anonymization submodels. FIGS. 4(b)-(d) are mask images M used for learning the first anonymization sub-model F1, the second anonymization sub-model F2, and the third anonymization sub-model F3, respectively.

As shown in FIG. 4B, the first mask image M1 is an image obtained by masking the area other than the face area of the person appearing in the teacher data T. As shown in FIG. Also, as shown in FIG. 4C, the second mask image M2 is an image obtained by masking the background area, which is the area other than the person appearing in the teacher data T. As shown in FIG. Furthermore, as shown in FIG. 4D, the third mask image M3 is an image obtained by masking the area of the person appearing in the teacher data T. As shown in FIG. These mask images are prepared in advance by the learner for each teacher data T used for learning. Alternatively, mask processing may be implemented as pre-processing of the anonymized sub-model F using a conventional technique.

FIG. 5 is a diagram schematically showing the structure of a learning network Nt for learning the anonymization submodel F. As shown in FIG. The learning network Nt includes an image processor F, a posture estimator E, and a divergence calculator D. The teacher data T used for learning the anonymized sub-model F includes image data T1 and information T2 regarding the posture of the person appearing in the image data T1. Here, the image data T1 also includes the mask image M illustrated in FIG. "Information T2 about the posture of the person" is information indicating the parts of the body of the person captured in each T1.

FIGS. 6A and 6B are diagrams for explaining an example of information about a subject according to the embodiment. Specifically, FIG. 6(a) is a diagram schematically showing the body part positions set for the female subject imaged in the image data T1, and FIG. 6(b) is shown in FIG. 6(a). FIG. 10 is a diagram showing position coordinates of each part position obtained in a tabular format;

As shown in FIG. 6A, the information T2 about the posture of the person includes 15 part positions including the top of the subject's head, wrists, elbows, shoulders, neck, waist, base of feet, knees, and toes. is set. In addition, as the coordinates indicating the position of each part of the subject, in a two-dimensional coordinate system in which the upper left of the information T2 on the posture of the person is the origin O, the horizontal direction of the information T2 on the posture of the person is the X axis, and the vertical direction is the Y axis. Coordinates are set. As shown in FIG. 6B, numbers from 1 to 15 are assigned to the 15 site positions as site numbers, and the coordinates of the site corresponding to each site number are associated with the image data T1. is set.

Returning to the description of FIG. In the learning network, image data T1 is first input to an image processor F. The image processor F includes a first concealment sub-model F1, a second concealment sub-model F2, and a third concealment sub-model F3. input to each of the encryption sub-model F2 and the third anonymization sub-model F3.

Here, each anonymizing sub-model F processes an area excluding the area masked by the corresponding mask image M. FIG. For example, the first mask image M1 corresponding to the first anonymization sub-model F1 is an image obtained by masking areas other than the face area of the person appearing in the teacher data T. FIG. Therefore, the first anonymization sub-model F1 performs processing on the area excluding the area masked by the first mask image M1, that is, the person's face area. Similarly, the second anonymization sub-model F2 performs processing on the person region, and the third anonymization sub-model F3 performs processing on the background region.

The first intermediate image B1, the second intermediate image B2, and the third intermediate image B3, which are the outputs of the first anonymization sub-model F1, the second anonymization sub-model F2, and the third anonymization sub-model F3, respectively, are synthesized. output as The synthesized output of each image processor F is input to the attitude estimator E and the divergence calculator D. FIG. Note that the intermediate images B1 to B3, which are the outputs of the respective image processors F, are combined with different weightings for each of the above-described anonymization processing patterns P to form an anonymized image H. FIG. That is, the anonymization sub-model F is generated by different learning for each anonymization processing pattern P. FIG.

The divergence calculator D includes a first divergence calculator D1, a second divergence calculator D2, and a third divergence calculator D3. The divergence calculator D calculates the divergence between the image input to the image processor F and the image output by the image processor F. FIG.

Specifically, the first divergence calculator D1 calculates the divergence between the input image and the output image of the image processor F in an area excluding the area masked by the first mask image M1. That is, the first divergence calculator D1 calculates the divergence in the person's face area. Similarly, the second degree of divergence calculator D2 calculates the degree of divergence in the person region, and the third degree of divergence calculator D3 calculates the degree of divergence in the background region.

Here, the degree of divergence calculated by the degree of divergence calculator D can be obtained by using any measurement method if it is possible to measure the degree of divergence between the image input to the image processor F and the image output by the image processor F. However, as an example, it is measured using Mean Squared Error (MSE). Hereinafter, the MSE calculated for the person's face area by the first divergence calculator D1 is the first original image divergence loss L _d1 , the MSE calculated for the person area by the second divergence calculator D2 is the second original image divergence loss L _d2 , The MSE calculated for the background area by the third deviation calculator D3 is referred to as a third original image deviation loss _Ld3 . One of the purposes of the image processing apparatus 1 according to the embodiment is to conceal the processing target image I. For this purpose, it is preferable that each original image divergence loss L is as large as possible. This is because the greater the original image divergence loss L, the greater the divergence between the input and the output of the image processor F.

The pose estimator E is implemented by a known pose estimation model trained to estimate the pose of a person included in an anonymized image H obtained by superimposing each of the plurality of anonymized submodels F. More specifically, the posture estimator E detects 15 points including the top of the head, wrists, elbows, shoulders, neck, waist, groin, knees, and toes of the person included in the image output by the image processor F. Estimate the position of the part of Here, for each part position to be estimated by the posture estimator E, the correct answer is determined as the information T2 regarding the posture of the person. Therefore, the posture estimator E uses the error between each part position estimated from the image output by the image processor F and the correct position of each part position determined as the information T2 regarding the posture of the person as a posture estimation loss _Lp . calculate.

FIG. 7 is a schematic diagram for explaining the posture estimation loss _Lp . In FIG. 7 , white circles indicate body part positions estimated by the posture estimator E. In FIG. A white square indicates the position of the left elbow of the person, which is set in advance in the information T2 regarding the posture of the person as the correct position for each part position.

In order to avoid complication, FIG. 7 shows only the position of the left elbow as the correct position. In FIG. 7, the dashed circle C is an enlarged view of the subject's left elbow. As shown in FIG. 7, the position of the left elbow estimated by the posture estimator E and the correct position of the left elbow are shifted by a distance Q. In FIG. The posture estimator E calculates the mean square of the deviation amount at each part position as the posture estimation loss _Lp .

In this case, the smaller the pose estimation loss _Lp , the higher the recognition accuracy of the pose estimator E. One purpose of the image processing apparatus 1 according to the embodiment is to estimate the orientation from the anonymized image H. For this purpose, the orientation estimation loss _Lp is preferably as small as possible.

Therefore, each anonymization sub-model F includes a deviation evaluation function G1 (a first deviation evaluation function G11, a second deviation evaluation function G12, and a third deviation evaluation function G13) and a posture evaluation function G2 shown below. is learned in the GAN framework such that . in particular,
G11=1/λ ₁ L _d1
G12=1/λ ₂ L _d2
G13=1/λ ₃ L _d3
G2 = Attitude estimation loss L _p
Here, λ ₁ , λ ₂ and λ ₃ are each positive real numbers.

Learning based on the divergence evaluation function G1 is learning related to the anonymization process of the image I to be processed. By using the mask image M, each of the plurality of anonymization sub-models F has at least one of a partial area set in advance as the subject and a background area set as an area other than the subject. is learned as an anonymization target area. More specifically, first, the first image processor F1 is trained using the first divergence evaluation function G11, then the second image processor F2 is trained using the second divergence evaluation function G12, Subsequently, the third image processor F3 is trained using the third deviation evaluation function G13, and then each image processor F (first image processor F1, second image processor F2 , and the third image processor F3) are trained. By repeating this, each image processor F is learned. In learning using the posture loss Lp, errors weighted by the corresponding pattern P are used for error back propagation (back propagation) to each image processor F. FIG.

A decrease in the deviation evaluation function G1 indicates an increase in the deviation between the input image of the image processor F and the output image generated by applying the anonymization submodel F to the input image. there is As a result, the image processor F learns such that the degree of divergence between the processing target image I and the anonymized image H increases.

On the other hand, the smaller posture evaluation function G2 indicates the higher estimation accuracy of the posture estimation model. As a result, the image processor F is trained so that the orientation estimation loss between the processing target image I and the anonymized image H is small. Here, learning about the anonymization processing increases the degree of divergence between the processing target image I and the anonymized image H, and thus the degree of divergence of the region including the person between the processing target image I and the anonymized image H also increases. means that In this case, the accuracy of person estimation using the anonymized image H as input may deteriorate. By learning each anonymization sub-model F based on not only the divergence evaluation function G1 but also the orientation evaluation function G2, both anonymization of the processing target image I and accuracy of orientation estimation can be achieved.

(Processing Flow of Image Processing Method Executed by Image Processing Apparatus 1)
FIG. 8 is a flowchart for explaining the flow of image processing executed by the image processing apparatus 1 according to the embodiment. The processing in this flowchart starts, for example, when the image processing apparatus 1 is activated.

The image acquisition unit 120 acquires the processing target image I including the subject to be concealed (S2). The selection accepting unit 121 accepts from the user a selection of the anonymization process pattern P to be applied to the subject (S4).

The anonymization model acquisition unit 122 reads and acquires the anonymization model corresponding to the anonymization processing pattern P accepted by the selection acceptance unit 121 from the anonymization model storage unit (S6). The model application unit 123 applies the anonymization model read by the anonymization model acquisition unit 122 to the processing target image I to generate an anonymization image H (S8). When the model application unit 123 generates the anonymized image H, the processing in this flowchart ends.

(Posture estimation server)
Returning to the description of FIG. 2 again. The anonymized image acquisition unit 210 of the server 2 transmits the anonymized image H generated by the image processing apparatus 1 and the pattern information indicating the anonymization process pattern P applied to the anonymized image H via the communication network N. to get.

The sharpening model acquisition unit 211 reads a sharpening model corresponding to pattern information from a sharpening model storage unit that stores a sharpening model determined for each pattern P of a plurality of sharpening processes. Here, a "sharpening model" is a plurality of different models, each trained to perform a different sharpening process on an image.

For example, the contour of a person is important for estimating the pose of the person. When the contour-oriented type is selected as the pattern P of the anonymization processing, the contour of the person region is blurred more than the other patterns P. Therefore, the sharpening model in the contour-oriented pattern P is , is trained to perform strong sharpening processing. Note that in the example shown in FIG. 2, the storage unit 20 also serves as the sharpening model storage unit.

The sharpening unit 212 generates a sharpened image by applying the sharpening model read by the sharpening model acquisition unit 211 to the concealed image H. FIG. The posture estimation unit 213 estimates the posture of the person included in the sharpened image by applying a posture estimation model to the sharpened image.

FIG. 9 is a diagram schematically showing how an image changes during posture estimation processing by the server 2 according to the embodiment. As shown in FIG. 9, the anonymized image H is an image in which the face region and background region of the person and the outline of the person are blurred. When the sharpening unit 212 applies the sharpening model to the anonymized image H, a sharpened image S in which the edges of the human region are emphasized is output.

When the posture estimation unit 213 applies the posture estimation model to the sharpened image S, the posture of the person appearing in the sharpened image S is estimated. Specifically, as shown in FIG. 9, a processing result O obtained by estimating each part of the person appearing in the sharpened image S is output. In this manner, the server 2 performs the sharpening process on the anonymized image H before executing the pose estimation process, thereby improving the accuracy of estimating the pose of the person appearing in the anonymized image H. can.

(Effects of Image Processing Apparatus 1 and Server 2 According to Embodiment)
As described above, according to the image processing apparatus 1 according to the embodiment, it is possible to provide a technique for changing the image anonymization pattern P according to the user's selection. Further, according to the server 2 according to the embodiment, it is possible to estimate the posture of the person appearing in the anonymized image H generated by the image processing device 1 .

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. be. For example, all or part of the device can be functionally or physically distributed and integrated in arbitrary units. In addition, new embodiments resulting from arbitrary combinations of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment caused by the combination has the effect of the original embodiment. Such modifications will be described below.

(First modification)
A case has been described above in which the divergence evaluation function G1 indicates the degree of divergence in the anonymization target region between the input image and the output image generated by applying the anonymization submodel F to the input image. . Instead, the divergence evaluation function G1 according to the first modification is the divergence in the anonymization target region between the input image and the output image generated by applying the anonymization submodel F to the input image. and the magnitude of the anonymization target area divergence degree, which is the degree of anonymization target area deviation, and the magnitude of the anonymization target area deviation degree, which is the degree of deviation from the input image in the area of the output image other than the anonymization target area. ing.
G1=MSE other than processing target area/MSE in processing target area

For example, a first divergence evaluation function G11, which is an evaluation function used for learning of the first image processor F1, is G11=(λ ₂ L _d2 +λ ₃ L _d3 )/(λ ₁ L _d1 ). Similarly, the second divergence evaluation function G12, which is the evaluation function used for learning of the second image processor F2, is G12=(λ ₁ L _d1 + λ ₃ L _d3 )/(λ ₂ L _d2 ), and the third image A third deviation evaluation function G13, which is an evaluation function used for learning by the processor F3, is G13=(λ ₁ L _d1 +λ ₂ L _d2 )/(λ ₃ L _d3 ). Each anonymization sub-model F according to the first modification is learned independently using the corresponding deviation evaluation function G1.

Each of the plurality of anonymization sub-models F has a large anonymization target region deviation, a small anonymization non-target region deviation, and an estimation accuracy is learned to become higher. As a result, in the intermediate image B output by the anonymization submodel F according to the first modification, the area masked by the corresponding mask image M (that is, the area other than the anonymization target area) is close to the original image. is learned as

Specifically, in the learning network according to the first modification, a plurality of different anonymization sub-models F are obtained from the pixel values of the output image in areas other than the anonymization target area of each anonymization sub-model (that is, mask An image obtained by subtracting the pixel values of the area masked by the image M) is output as an intermediate image B, and an image obtained by superimposing the intermediate image B is output as an anonymized image H.

FIG. 10 is a schematic diagram for explaining an image processor F according to the first modification of the embodiment. Specifically, FIG. 10 is a diagram for explaining the operation of the third image processor F3 according to the first modification of the embodiment. As shown in FIG. 10, the third image processor F3 according to the first modification anonymizes the background area, which is not masked by the third mask image M3, in the teacher image data T1. The pixel values of the pixels in the person region masked by the image M3 maintain the pixel values of the image data T1.

As shown in FIG. 10, the output image of the third image processor F3 is subtracted from the image data T1. As a result, the pixel value of the person area masked by the third mask image M3 becomes 0 in the third intermediate image B3. In other words, the third intermediate image B3 according to the first modified example is an image in which only the background image, which is the anonymization target area, is captured.

The same applies to the first intermediate image B1 output by the first image processor F1 according to the first modification. The image is an image that includes only the face area, which is the face area. Also, the second intermediate image B2 output by the second image processor F2 is an image including only the person area. As a result, the anonymized image H output by the model application unit 123 according to the first modification is an image in which the anonymization target regions of the anonymization submodels F are superimposed like a patchwork, and the anonymized image H can be composed only of pixels output by any of the anonymization sub-models F.

(Second modification)
In the above description, the case where the anonymization sub-model F is learned for each anonymization processing pattern P has been described. Instead, during learning, learning is performed with a fixed weighting pattern P (for example, a pattern in which all anonymization submodels have equal weights), and during anonymization processing, the weight is changed for each anonymization processing pattern P. good too. As a result, the calculation cost associated with the generation of the image processor F can be reduced.

(Third modification)
In the above description, the case where the aspect ratios of the processing target image I and the anonymized image H are the same has been mainly described. However, the aspect ratio of the anonymized image H may differ from the aspect ratio of the image I to be processed. This can be done by changing the aspect ratio of the anonymized image H by the aspect ratio changer in the control unit 12 to create a new anonymized image H. FIG. If the aspect ratio of the anonymized image H is changed, the body shape of the person in the anonymized image H is also changed.

1... Image processing apparatus 10... Storage unit 11... Imaging unit 12... Control unit 120... Image acquisition unit 121... Selection reception unit 122... Anonymized model acquisition unit 123. Model application unit 2 Server 20 Storage unit 21 Control unit 210 Anonymized image acquisition unit 211 Sharpening model acquisition unit 212 Sharpening unit 213 Posture estimating unit D Deviation calculator E Posture estimator F Image processor N Communication network Nt Learning network

Claims (10)

an image acquisition unit that acquires an image to be processed that includes a subject to be anonymized;
a selection reception unit that receives selection of a pattern of anonymization processing to be applied to the subject;
a plurality of different masking models trained to perform different masking processes on an image, the masking model storing a masking model determined for each of the plurality of masking process patterns an anonymization model acquisition unit that reads, from a storage unit, an anonymization model corresponding to the anonymization processing pattern accepted by the selection acceptance unit;
a model application unit that generates a masked image by applying the masked model read by the masked model acquisition unit to the image to be processed;
An image processing device comprising: each of the different masking models shares a plurality of different masking sub-models that perform different types of masking processes on images;
The model application unit outputs an image obtained by superimposing output images of a plurality of different anonymization sub-models based on a weighting coefficient determined according to the pattern of the anonymization process, as the anonymization image.
The image processing apparatus according to claim 1. Each of the plurality of anonymization sub-models has at least one of a partial area set in advance for the subject and a background area set as an area other than the subject as an anonymization target area. being learned,
The image processing apparatus according to claim 2. the subject is a person,
Each of the plurality of anonymization sub-models (1) indicates the degree of divergence in the anonymization target region between an input image and an output image generated by applying the anonymization sub-model to the input image. and (2) a posture indicating the level of estimation accuracy of a posture estimation model trained to estimate the posture of a person included in the anonymized image obtained by superimposing each of the plurality of anonymization sub-models. Based on the two evaluation functions, the evaluation function and the learning is performed so that the deviation is increased and the estimation accuracy is increased.
The image processing apparatus according to claim 3. The divergence evaluation function is the degree of divergence in the anonymization target region, which is the degree of divergence in the anonymization target region between the input image and the output image generated by applying the anonymization submodel to the input image. , and the size of a non-anonymization target region deviation degree, which is a degree of deviation from the input image in a region other than the anonymization target region in the output image, and
Each of the plurality of anonymization sub-models increases the anonymization target area deviation and decreases the anonymization non-target area deviation based on the deviation evaluation function and the posture evaluation function, and It is learned so that the estimation accuracy is high,
The model application unit is configured to anonymize an image obtained by superimposing an image obtained by subtracting a pixel value of an area other than an anonymization target area of each anonymization submodel from a pixel value of an output image of each of a plurality of different anonymization submodels. output as an image,
The image processing apparatus according to claim 4. Anonymized image acquisition for acquiring an anonymized image generated by the image processing apparatus according to claim 4 or 5 and pattern information indicating a pattern of anonymization processing applied to the anonymized image via a communication network. Department and
a plurality of different sharpening models trained to perform different sharpening processes on an image, the sharpening models storing sharpening models determined for each of the plurality of sharpening process patterns; a sharpening model acquisition unit that reads a sharpening model corresponding to the pattern information from a storage unit;
a sharpening unit that generates a sharpened image by applying the sharpening model read by the sharpening model acquisition unit to the anonymized image;
a posture estimation unit that estimates a posture of a person included in the sharpened image by applying the posture estimation model to the sharpened image;
A server with the processor
a step of acquiring a processing target image including a subject to be concealed;
a step of receiving selection of a pattern of anonymization processing to be applied to the subject;
a plurality of different masking models trained to perform different masking processes on an image, the masking model storing a masking model determined for each of the plurality of masking process patterns a step of reading an anonymization model corresponding to the accepted anonymization process pattern from a storage unit;
generating a masked image by applying the read masking model to the image to be processed;
An image processing method that performs to the computer,
A function of acquiring a processing target image including a subject to be concealed;
a function of accepting selection of a pattern of anonymization processing to be applied to the subject;
a plurality of different masking models trained to perform different masking processes on an image, the masking model storing a masking model determined for each of the plurality of masking process patterns a function of reading an anonymization model corresponding to the accepted anonymization process pattern from a storage unit;
a function of generating an anonymized image by applying the read anonymization model to the image to be processed;
program to realize A processor of a server connected to the image processing apparatus according to claim 4 or 5 via a communication network,
a step of acquiring, via the communication network, an anonymized image generated by the image processing device and pattern information indicating a pattern of anonymization processing applied to the anonymized image;
a plurality of different sharpening models trained to perform different sharpening processes on an image, the sharpening models storing sharpening models determined for each of the plurality of sharpening process patterns; reading a sharpening model corresponding to the pattern information from a storage unit;
generating a sharpened image by applying the read sharpening model to the concealed image;
estimating a pose of a person included in the sharpened image by applying the pose estimation model to the sharpened image;
A pose estimation method that performs A computer connected to the image processing apparatus according to claim 4 or 5 via a communication network,
a function of acquiring, via the communication network, an anonymized image generated by the image processing apparatus and pattern information indicating a pattern of anonymization processing applied to the anonymized image;
a plurality of different sharpening models trained to perform different sharpening processes on an image, the sharpening models storing sharpening models determined for each of the plurality of sharpening process patterns; a function of reading a sharpening model corresponding to the pattern information from a storage unit;
a function of generating a sharpened image by applying the read sharpening model to the concealed image;
a function of estimating the pose of a person included in the sharpened image by applying the pose estimation model to the sharpened image;
program to realize

Images