JP7423310B2 - Data processing device, data processing method and trained model - Google Patents

Description

The present invention relates to a data processing device and a learned model, and more particularly to a data processing device, a data processing method, and a learned model that perform a predetermined data processing task on input data.

Patent Document 1 discloses that instead of transmitting the camera image as it is, the local device that acquired the image adds noise that does not affect matching to the face image to generate a noise-added image that is difficult to visually identify. A network-type authentication system has been disclosed that takes privacy into consideration by transmitting a noise-added image to a server and performing person verification using the noise-added image on the server side.

In particular, when a data processing task such as person matching is performed using a multilayer neural network on a server, the local device uses noise to the extent that the product of the filter component of the convolutional layer included in the multilayer neural network and the noise component can be considered to be 0. An example is disclosed in which a noise-added image is generated based on conditions for setting the components of .

Japanese Patent Application Publication No. 2018-129750

However, in order to generate a noise-added image that meets the above conditions, for example, if a method is used to find the filter components through learning and then find the noise components that meet the above conditions, noise that satisfies the above conditions will be found. That is not guaranteed. That is, in a nonlinear classifier such as a multilayer neural network, it is sometimes difficult to analytically find noise that satisfies the above conditions.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a data processing device, a data processing method, and a trained model that can perform data processing tasks using a multilayer neural network with high accuracy even when noise is added.

In order to achieve the above object, a data processing device according to the present invention includes a filter used in each convolution layer in a multilayer neural network that performs a predetermined data processing task on input data, and a noise added to the input data. a learning unit that learns the data, and inputs noise-added data obtained by adding noise data learned by the learning unit to the input data to the multilayer neural network, and inputs the noise-added data obtained by adding the noise data learned by the learning unit to the input data to An input data processing unit that obtains a result of the data processing task based on an output. input to a multilayer neural network, the noise data and the filter used in the previous convolution layer included in the multilayer neural network have orthogonality, and the obtained result of the data processing task and the It is characterized in that learning is performed so that the results of the data processing task given in advance to the learning data match.

According to the data processing device according to the present invention, the learning unit inputs learning data to which the results of the data processing task are assigned in advance to the multilayer neural network, and includes the noise data and the multilayer neural network. filters used in the convolutional layer in the previous stage that are used have orthogonality, and the obtained result of the data processing task matches the result of the data processing task given in advance to the learning data. Learn how. Then, the input data processing section inputs noise-added data obtained by adding the noise data learned by the learning section to the input data to the multilayer neural network. Based on the output, a result of the data processing task is determined.

In this way, the noise data and the filter used in the previous convolutional layer included in the multilayer neural network have orthogonality, and the results of the predetermined data processing task and the results of the predetermined data processing task are By learning to match the results of a predetermined data processing task, it is possible to perform a data processing task using a multilayer neural network with high accuracy even when noise is added.

Further, the learning unit is configured to convolve a filter group used in a previous convolution layer included in the multilayer neural network and a filter of the filter group used in the previous convolution layer with a predetermined stride, in each region to be convolved. The filter and the noise data can be learned such that the noise data region corresponding to the filter has orthogonality.

The input data is an image, and the learning unit is configured to generate a noise block group consisting of a predetermined reference noise block and a derived noise block obtained by shifting the reference noise block according to a predetermined shift pattern. It is possible to learn to generate the noise data by arranging any one of them side by side.

Further, the learning unit can learn such that all noise blocks of the noise block group and each of the filters have orthogonality.

Further, the learning section can obtain the derived noise block by shifting the noise block using the shift pattern according to the predetermined stride.

Further, the input data is an image, the predetermined stride is an integral multiple of the size of the filter, and the learning unit learns so that the predetermined noise block and the filter have orthogonality. , the noise data may be one or more noise blocks obtained through learning arranged side by side.

Further, the input data is an image, and the noise-added data is obtained by adding the noise data after dropping randomly determined pixels among the pixels of the input data. , the learning unit can perform learning by omitting randomly determined pixels from among the pixels of the learning data and inputting the data to the multilayer neural network.

Further, the input data is an image, and the noise-added data is obtained by adding random noise to the input data and then adding the noise data,
The learning section can perform learning by adding random noise to the learning data and inputting the data to the multilayer neural network.

In the data processing method according to the present invention, the learning unit learns filters used in each convolution layer in a multilayer neural network that performs a predetermined data processing task on input data, and noise data to be added to the input data. Then, the input data processing unit inputs noise-added data obtained by adding noise data learned by the learning unit to the input data to the multilayer neural network, and inputs the noise-added data obtained by adding the noise data learned by the learning unit to the input data to A data processing method for obtaining a result of the data processing task based on an output, wherein the learning unit inputs learning data to which the result of the data processing task has been assigned in advance to the multilayer neural network; The noise data and the filter used in the previous convolutional layer included in the multilayer neural network have orthogonality, and the obtained result of the data processing task and the It is characterized by learning to match the results of data processing tasks.

The trained model according to the present invention is a multilayer neural network for performing a predetermined data processing task on input data, and the trained model is a multilayer neural network that performs a predetermined data processing task on input data, and uses noise-added data obtained by adding noise data to the input data. A trained model that is a multilayer neural network for determining the result of the data processing task based on the output when input to the multilayer neural network, the training data being provided with the result of the data processing task in advance. input to the multilayer neural network, the noise data and a filter used in a previous convolutional layer included in the multilayer neural network have orthogonality, and the obtained result of the data processing task; The learning data is characterized in that the learning data is trained in advance so that the result of the data processing task given in advance matches the result of the data processing task.

According to the trained model according to the present invention, training data to which the results of a predetermined data processing task have been assigned in advance is input to the multilayer neural network, and the noise data and the previous convolution layer included in the multilayer neural network are The filters used are orthogonal and are trained in advance so that the obtained result of the predetermined data processing task matches the result of the predetermined data processing task given in advance to the learning data. Then, the result of a predetermined data processing task is determined based on the output when the noise-added data obtained by adding noise data to the input data is input to the multilayer neural network.

In this way, the filter group used in the previous convolutional layer included in the multilayer neural network is trained to have orthogonality with the noise data, so even if noise is added, the filter group used in the previous convolutional layer Data processing tasks can be performed with precision.

As explained above, according to the data processing device, data processing method, and trained model of the present invention, even if noise is added, data processing tasks using a multilayer neural network can be performed with high accuracy. is obtained.

1 is a schematic diagram showing the configuration of a face authentication system according to an embodiment of the present invention. FIG. 3 is a diagram showing the structure of input face data. FIG. 3 is a diagram showing the structure of noise-added input face data. FIG. 2 is a schematic diagram showing a noise block and a filter of a multilayer neural network. FIG. 3 is a schematic diagram showing a face image before noise is added and a face image after noise is added. FIG. 3 is a schematic diagram showing characteristics of convolution between a filter and noise-added face data. FIG. 3 is a diagram showing an example of a noise image. FIG. 3 is a diagram showing an example of a derived noise block. FIG. 3 is a diagram showing the configuration of registered face data. FIG. 3 is a diagram showing the configuration of authenticated registered face data. FIG. 6 is a diagram for explaining a method for simultaneously proceeding with image processing task learning and orthogonal constraint learning. FIG. 3 is a diagram showing the configuration of authentication history data. FIG. 3 is a diagram showing an example of a notification image. 3 is a flowchart showing the operation of learning processing using face authentication according to an embodiment of the present invention. 7 is a flowchart showing the operation of detection processing by the image processing device according to the embodiment of the present invention. 2 is a flowchart showing the operation of matching processing by the face authentication device according to the embodiment of the present invention. 2 is a flowchart showing the operation of notification processing by the notification device according to the embodiment of the present invention.

Embodiments of the present invention will be described in detail below with reference to the drawings. Note that in this embodiment, a case where the present invention is applied to a network type face authentication system will be described as an example.

<System configuration>
Hereinafter, the configuration of an embodiment of the present invention will be described with reference to FIG. 1 showing a schematic configuration of a face authentication system 1000 to which the present invention is applied.

(Face recognition system 1000)
The face authentication system 1000 includes an imaging device 1100, a network 1200, an image processing device 1300, a face authentication device 1400, and a notification device 1500. Note that the face authentication device 1400 is an example of a data processing device.

(Imaging device 1100)
The imaging device 1100 is a surveillance camera installed for the purpose of monitoring a predetermined area, and is installed at a position where it can photograph the face of a person staying within the area to be monitored. A monitoring image captured by the imaging device 1100 is transmitted to the image processing device 1300.

(Network 1200)
The network 1200 is a line used for transmitting and receiving data between the image processing device 1300, the face authentication device 1400, and the notification device 1500. A LAN (Local Area Network) or a public line such as the Internet can be used as the network 1200 of the present invention. Regarding the messages on the network 1200, it is desirable to take security measures such as encrypting the messages using a known VPN technology or the like.

(Image processing device 1300)
The image processing device 1300 is composed of a CPU, an MPU, peripheral circuits, terminals, various types of memories, etc., and performs image processing on images taken by the imaging device 1100, and transmits the results to the face authentication device 1400 and the like via the network 1200. It is transmitted to the notification device 1500. Each of the image processing section 1310, storage section 1320, and transmission/reception section 1330 that constitute the image processing apparatus 1300 will be described in detail below.

(Image processing unit 1310)
The image processing section 1310 includes a face image acquisition means 1311 and a noise addition means 1312.

(Face image acquisition means 1311)
The face image acquisition unit 1311 extracts a face image of a person from the monitoring image taken by the imaging device 1100, and uses it as an input face image. Furthermore, a unique face image identifier and photographing time are added to the input face image, and the input face image is transmitted to the noise adding means 1312 as input face data 200 having the configuration shown in FIG. 2, and is stored in the storage unit 1320.

The face image identifier 210 is an identifier for uniquely identifying the input face image 220. For example, a 128-bit integer is used as the face image identifier 210, the initial value is set to 0, and the face image identifier 210 is assigned to the input face image 220. There is a method such as incrementing the value of the face image identifier 210 for each time. A checksum or the like may be added to the facial image identifier 210 to prevent the facial image identifier 210 from being fraudulently estimated. If a plurality of people exist in the monitoring image, the face images of each person are extracted, assigned different face image identifiers 210, and transmitted to the noise addition means 1312 and the storage unit 1320.

Many methods for extracting facial images have been proposed in the past, and any known method may be adopted as appropriate. For example, a method may be adopted in which a face image is extracted using a filter called a learned classifier, or a method in which a binarized edge image of the input image is generated and an elliptical shape, which is the shape of the face, is detected in the edge image. Good.

(Noise addition means 1312)
The noise addition means 1312 generates noise-added face data 320 by adding noise to the face image of the person extracted by the face image acquisition means 1311 so as to make it difficult to visually identify the subject, and also performs face image acquisition. The same face image identifier 310 as the face image output by the means 1311 is given to the noise-added face data 320, and the noise-added face data 320 is transmitted to the transmitting/receiving unit 1330 as the noise-added input face data 300 having the configuration shown in FIG. Note that the noise-added face data 320 is an example of noise-added data.

Regarding the method for creating the noise-added face data 320, a face matching method using a multilayer neural network 1440 including a convolutional layer, specifically a convolutional neural network (CNN), which receives the noise-added face data 320 as input, is used. An application case will be explained as an example. As will be described later, the multilayer neural network 1440 has a feature for receiving noise-added face data 320 that is difficult to visually identify from the input data processing unit 1430 and comparing it with face image data registered in the storage unit 1420. Output the amount. Note that the multilayer neural network 1440 that has completed learning is an example of a trained model.

In this case, as shown in FIG. 4, a noise block 3020 having the same structure (dimensions, number of elements) as the filter 3010 used in the convolutional layer in the previous stage of the multilayer neural network 1440 is set to have orthogonality with the filter 3010. Then, a noise-added image 3030 is generated by arranging and adding it to the face image extracted by the face image acquisition means 1311. That is, a noise-added image 3030 is generated by adding a noise image in which noise blocks 3020 are arranged side by side to the input face image 220. The noise-added image 3030 can be noise-added face data 320 that makes it difficult to visually identify the subject. Here, "the filter and the noise block have orthogonality" means that when the elements of both are expressed as vectors, the element vector of the filter and the element vector of the noise block are orthogonal in the vector space. When the filter element vector and the noise block element vector are orthogonal, their inner product becomes zero. Note that the noise image is an example of noise data.

FIG. 5 shows a schematic diagram of a face image 400 before noise is added and a noise-added image 410. The noise-added image 410 is an image that is difficult to visually identify compared to the face image 400 before noise is added.

Here, a principle will be described in which the noise-added face data 320 that is difficult to visually identify generated in this way does not reduce the accuracy of face matching processing using the multilayer neural network 1440.

The convolution process in the multilayer neural network 1440 is performed using the filter 3010 in the convolution layer and the image block elements of the input image (here, the noise-added image 3030, that is, the noise-added face data 320 that is difficult to visually identify) that is input to the multilayer neural network 1440. Outputs the dot product map. An image block is an area in an input image that includes a target element for inner product calculation at a specific convolution position. In convolution processing, the filter position is sequentially scanned over the input image with a predetermined stride (shift width), and the inner product with the image block at each position (convolution position) is calculated, and finally an inner product map for the entire input image is calculated. Output.

As shown in FIG. 6, since the inner product has the characteristic of "linearity," the convolution of the filter 3010 and the noise-added face data 3120 that is difficult to visually identify is the "convolution of the filter 3010 and the face image 3130 before adding noise." ” and “convolution of the filter 3010 and the noise image 3140 in which noise blocks are arranged.” When the stride is an integer multiple of the size (width, height) of the filter 3010, if the "noise image 3140 in which the filter 3010 and the noise block are arranged" has orthogonality, "the filter 3010 and the noise image 3140 in which the noise block is arranged Since the inner product of "convolution with noise image 3140" becomes zero at any convolution position, only the output of "convolution with filter 3010 and face image 3130 before noise addition" remains. Since this is a convolution with a face image to which no noise block is added, the multilayer neural network 1440 outputs the same convolution result (feature for matching) regardless of the presence or absence of noise. That is, even if the matching process is performed using the noise-added face data 320 that is difficult to visually identify due to the addition of noise as described herein, there is no reduction in matching accuracy.

In this way, when the stride is an integer multiple of the size of the filter 3010, the noise pattern in the region of the noise image where the filter 3010 is convolved is constant, so there is orthogonality between the noise block itself and the filter 3010. All you have to do is do it. However, if the stride of the filter 3010 is not an integral multiple of the size of the filter 3010, the convolution position in the noise image 3140 may cross the boundary of the noise block 3020. Merely ensuring orthogonality is not sufficient.

This situation will be explained with reference to FIG. FIG. 7A shows a noise image 3200 in which four types of noise blocks Na, Nb, Nc, and Nd are arranged side by side as an example of the arrangement of the noise blocks 3020. The noise blocks Na, Nb, Nc, and Nd are orthogonal to the filter 3010. When convolution is performed on this noise image 3200 using a filter 3010 with a stride of 1, for example, as shown in FIG. 7(B), the noise blocks Na, Nb, Nc, obtained by shifting the noise blocks, Convolution is also performed on the derived noise block Ne consisting of some of the elements of Nd. At this time, although the noise blocks Na, Nb, Nc, and Nd each ensure orthogonality with the filter 3010 as a single noise block, this alone does not guarantee orthogonality between the derived noise block Ne and the filter 3010. It doesn't mean that it is.

In such a case, each region in which the filter is convolved in the noise image 3200 in which the noise blocks 3020 are arranged is determined based on the size and stride of the filter, and each region to be convolved and the filter 3010 are set to have orthogonality. There is a need. Here, in the case of a noise image 3310 in which only noise blocks Na are arranged side by side as shown in FIG. 8(A), blocks corresponding to each area to be convolved in the noise image can be obtained by shifting the noise blocks Na. Can be done. For example, if the stride of the filter is 1, by scanning the filter, each area in which the filter is convolved in the noise image 3310 in which noise blocks Na are arranged is obtained by shifting (cyclically shifting) the elements of the noise block Na in the row and column directions. Become something. Therefore, by configuring a plurality of derived noise blocks in which the elements of noise block Na are shifted in the row and column directions, in the noise image 3310 in which noise blocks Na are arranged, there is a region where the filter is convolved at any convolution position. , matches any of the derived noise blocks. Therefore, the derived noise block obtained by shifting the elements of the noise block Na in the row and column directions may be set to be orthogonal to the filter 3010. Note that the noise block Na that serves as a reference for creating derived noise blocks is an example of a reference noise block. Further, a group consisting of a noise block Na and a plurality of derived noise blocks obtained by shifting the noise block Na is an example of a noise block group. Hereinafter, the reference noise block and the derived noise block that constitute the noise block group may be referred to as noise blocks.

In FIG. 8A, the size of the noise block Na is 3×3. In FIG. 8B, each element of the noise block Na is represented by Na1 to Na9, and convolution with the filter 3010 is performed with a stride of 1. At this time, a variation 3330 of the derived noise block corresponding to each region in which the filter is convolved in the noise image 3310 is obtained by shifting the elements of the noise block Na in the row and column directions, as shown in FIG. 8(C). This results in a pattern of two shifts. Therefore, based on one of the single noise blocks 3020, derived noise blocks are obtained by shifting its elements in the row and column directions, and each derived noise block is set to be orthogonal to the filter 3010. .

Note that the stride may be 2 or more, and in that case, a derived noise block in which the elements of the noise block 3020 are shifted in the row direction and column direction according to the stride may be used.

In addition, in order to set the filter 3010 and the noise block 3020 to be orthogonal, it is necessary to set the filter 3010 of the multilayer neural network 1440 and the noise added to the training image of the multilayer neural network 1440 during learning of the multilayer neural network 1440. There is a method in which learning is performed with constraints such that block 3020 is orthogonal, and the learning results are applied.

For example, if the coefficient of the filter 3010 is wf and the value of the noise block 3020 is wn, then an orthogonality constraint loss value lossC representing the degree of non-orthogonality as expressed by the following equation (1) can be defined. Here, i represents a variation of the filter, and j represents a variation of the noise block 3020 depending on the stride position. "・" represents the inner product. By minimizing the degree of non-orthogonality during the learning process of the multilayer neural network 1440, a noise block 3020 that is orthogonal to the filter 3010 can be obtained.

(1)

Note that in the method using derived noise blocks, the size of the noise block 3020 does not necessarily have to match the size of the filter 3010, but the noise block 3020 is assumed to be smaller or larger than the filter 3010, and the noise block 3020 is It is also possible to create a derived noise block by shifting the elements of . However, since the unit of convolution processing is a region unit corresponding to the size of the filter 3010, an arbitrary region of the same size as the filter 3010 cut out from the noise image created by arranging the noise blocks 3020 is orthogonal to the filter 3010. It is necessary to learn as follows.

For example, if the size of the filter shown in FIGS. 8(A) and 8(B) is 4×4, one of the areas corresponding to the area to be convolved with the filter in the noise image 3310 is 4×4 from the upper left corner of FIG. 8(B). ×4 elements are cut out, and the area is learned to have orthogonality with each filter in the filter group. At this time, learning is performed by imposing a constraint so that elements having the same identifier in the region have the same value.

(Transmission/reception section 1330)
The transmitter/receiver 1330 transmits the noise-added input face data 300 created by the noise adder 1312 to the transmitter/receiver 1410 of the face authentication device 1400 via the network 1200 .

Further, as described later, information on an authenticated face image identifier transmitted from the transmitting/receiving unit 1410 of the face authentication device 1400 is received, and input face data 200 corresponding to the authenticated face image identifier is read from the storage unit 1320. It is transmitted to the receiving unit 1510 of the notification device 1500 via the network 1200.

(Face recognition device 1400)
The face authentication device 1400 is composed of a CPU, an MPU, peripheral circuits, terminals, various types of memories, etc., and receives the noise-added input face data 300 transmitted by the image processing device 1300, and uses the noise-added input face data 300 to perform face registration. The input data processing unit 1430 determines whether the face data is of a registered person by referring to the registered face data stored in the storage unit 1420 in advance. Note that a face authentication task that determines whether the face data is of a person whose face has been registered is an example of a data processing task.

The input data processing unit 1430 is equipped with a multilayer neural network 1440 that has been trained in advance, and inputs the noise-added face data 320 of the noise-added input face data 300 to the multi-layer neural network 1440 and outputs the feature amount. The noise-added input face data 300 is determined to be the registered face data by inputting the registered face data pre-stored in the storage unit 1420 into the multilayer neural network 1440 and comparing the similarity with the feature amount output. Determine if they match.

The input data processing unit 1430 uses the face image identifier linked to the noise-added input face data 300 determined to be a match in the face authentication task as an authenticated face image identifier via the transmitting/receiving unit 1410 and the network 1200. It is transmitted to the receiving section 1510 of the notification device 1500.

Hereinafter, each of the transmitting/receiving section 1410, input data processing section 1430, and storage section 1420 that constitute the face authentication device 1400 will be described in detail.

(Transmission/reception section 1410)
The transmitting/receiving unit 1410 receives the noise-added input face data 300 transmitted by the image processing device 1300 via the network 1200 and outputs it to the input data processing unit 1430.

Further, the authenticated face image identifier output by the input data processing unit 1430 is transmitted to the transmitting/receiving unit 1330 of the image processing device 1300 via the network 1200.

(Storage unit 1420)
The storage unit 1420 stores face data of a person whose face has been registered in advance as registered face data to which a registered face image identifier has been added. As shown in FIG. 9, registered face data 600 includes a registered face image identifier 610, a registered face image 620, and registered attribute information 630. At least one piece of registered face data 600 is stored for one registered person. When a plurality of registered persons exist, the storage unit 1420 stores a plurality of pieces of registered face data 600 to which different registered face image identifiers 610 are assigned.

The registered face image identifier 610 is an identifier for uniquely identifying the registered face image 620, and uses, for example, a 128-bit integer. For example, there is a method in which the initial value is set to 0 and the value of the registered face image identifier 610 is incremented every time new registered face data 600 is created. A checksum or the like may be added to the registered facial image identifier 610 to prevent the registered facial image identifier 610 from being fraudulently estimated.

The registered face image 620 is a face image of a registered person, and is used as input data to the multilayer neural network 1440 in the face authentication task and as a notification image in the notification device 1500.

The registered attribute information 630 represents attribute information associated with a registered person, such as name, gender, age, and affiliated organization.

(Input data processing unit 1430)
The input data processing unit 1430 compares the noise-added input face data 300 outputted by the image processing device 1300 with the registered face data 600 stored in advance in the storage unit 1420, and determines whether the noise-added input face data 300 is a registered face. It is determined whether or not it matches any of the data 600. If registered face data 600 that matches the noise-added input face data 300 exists, an authenticated face image identifier 710 is added to the registered face image 620 and registered attribute information 630 of the matched registered face data 600. , creates authenticated registered face data 700 shown in FIG.

The input data processing unit 1430 determines whether the noise-added input face data 300 matches the registered face data 600 by comparing the noise-added face data 320 of the noise-added input face data 300 with the registered face image 620 of the registered face data 600. , by determining a threshold value of the similarity of the feature amounts output from the multilayer neural network 1440.

The input data processing unit 1430 inputs the noise-added face data 320 of the noise-added input face data 300 outputted by the image processing device 1300 to the trained multilayer neural network 1440, and extracts input image features for matching. , Furthermore, the registered face image 620 of the registered face data 600 stored in the storage unit 1420 is input to the multilayer neural network 1440 to extract the registered image feature amount for comparison, and the input image feature amount and the registered image are extracted. The feature values are compared to determine whether they match.

A large amount of training face image data prepared in advance is input to the multilayer neural network 1440 to calculate feature quantities, and if the pair of feature quantities belong to the same person, the degree of similarity increases, and if the pair of feature quantities belong to different people. If so, the multilayer neural network 1440 is trained so that the degree of similarity decreases (learning of data processing tasks).

Specifically, the input unit 1610 receives a large amount of learning face image data to which a person identifier for identifying which person the face image belongs to is received in advance, and the learning unit 1630 uses the multilayer neural network 1440. The learning result is stored in the storage unit 1420. Note that the person identifier is an example of the learning result of the data processing task.

In addition to learning the data processing task, the learning unit 1630 also performs training so that the filter 3010 of the convolution layer in the previous stage of the multilayer neural network 1440 and the noise block 3020 are orthogonal, as described in the explanation of the noise adding means 1312. Also performs learning (orthogonal constraint learning).

In learning data processing tasks, for example, a fully connected layer with the same number of outputs as person identifiers included in the training face image data is connected to a general deep learning model such as VGG16, and outputs corresponding to the person identifiers are connected to a general deep learning model such as VGG16. It is sufficient to calculate the loss value lossM for the data processing task using the one-hot vector set to 1 as training data, and proceed with learning so that lossM becomes smaller.

Specifically, the output of the fully connected layer is converted into a probability output using a softmax function, and the cross entropy with the training data is defined as lossM. Then, for each filter 3010, the values of the elements of the filter 3010 may be learned so that the lossM becomes smaller based on the differential value obtained by differentiating the lossM with the value of the element of the filter 3010.

In orthogonal constraint learning, the orthogonal constraint loss value lossC representing the degree of non-orthogonality between the weight vector of the filter 3010 in the convolutional layer at the previous stage of the multilayer neural network 1440 and the weight vector of the noise block 3020 is calculated according to the above formula (1). Then, learning should be performed so that lossC becomes smaller.

Alternatively, the cosine value (cos θ) of the angle θ formed by the weight vector of the filter 3010 in the convolution layer in the previous stage of the multilayer neural network 1440 and the weight vector of the noise block 3020 is used as an index of non-orthogonality, and the absolute value of the cosine value The absolute value of the difference between the correct cosine value (zero) in the case of a right angle may be calculated as the orthogonality constraint loss value lossC, and learning may be performed so that lossC becomes smaller.

Specifically, for each noise block 3020, based on the differential value obtained by differentiating the sum of lossC for all filters 3010 by the value of the element of the noise block 3020, the noise block is adjusted such that the sum of lossC becomes smaller. It is sufficient to learn the value of the element 3020.

Note that the data processing task loss value lossM and the orthogonal constraint loss value lossC are integrated using a balance coefficient α to obtain an overall loss value lossT, and by performing learning so that the overall loss value lossT is small, the data Processing task learning and orthogonal constraint learning can proceed simultaneously.

(2)

FIG. 11 shows the above-described learning with constraints.

Specifically, as shown in FIG. 11, the learning unit 1630 calculates a loss value for a data processing task, which is calculated using feature quantities calculated by inputting a large amount of face image data for learning into a multilayer neural network 1440. The overall loss value lossT, which is the integration of lossM and the orthogonality constraint loss value lossC calculated using the weight vector of the filter 3010 in the convolution layer in the previous stage of the multilayer neural network 1440 and the weight vector of the noise block 3020, becomes smaller. As such, the multilayer neural network 1440 and the noise block 3020 are trained.

For example, for each filter 3010, based on the differential value obtained by differentiating the sum of the total loss values lossT for all the noise blocks 3020 by the values of the elements of the filter 3010, the calculation is performed so that the sum of the total loss values lossT becomes smaller. By learning the values of the elements of the filter 3010 (see 1111 in FIG. 11), and for each noise block 3020, based on the differential value obtained by differentiating the sum of lossT for all filters 3010 by the value of the element of the noise block 3020. Then, the value of the element of the noise block 3020 is learned so that the sum of lossT becomes small, and for each filter 3010, the sum of lossT for all the noise blocks 3020 is differentiated by the value of the element of the filter 3010. Based on the values, learning the values of the elements of the filter 3010 (see 1112 in FIG. 11) is alternately repeated so that the sum of lossT becomes smaller.

(Notification device 1500)
The notification device 1500 is composed of a CPU, an MPU, peripheral circuits, terminals, various memories, a display monitor, etc., and receives input face data 200 sent from the image processing device 1300 and authenticated registration sent from the face authentication device 1400. Authentication history data is created from the face data 700 by the authentication history creation section 1520 and stored in the storage section 1530. The authentication history data stored in the storage section 1530 is displayed on the notification section 1540.

(Receiving unit 1510)
The receiving unit 1510 receives the input face data 200 transmitted by the image processing device 1300 and the authenticated registered face data 700 transmitted by the face authentication device 1400, and outputs them to the authentication history creation unit 1520.

(Authentication history creation unit 1520)
Authentication history creation section 1520 creates authentication history data from input face data 200 received via reception section 1510 and authenticated registered face data 700, and stores it in storage section 1530.

FIG. 12 shows the configuration of authentication history data 800. When creating the authentication history data 800, first, the face image identifier 210 of the input face data 200 is compared with the authenticated face image identifier 710 of the authenticated registered face data 700, and the input face data 200 having the same face image identifier is Select authenticated registered face data 700. Then, the input face image 220 of the selected input face data 200 and the photographing time 230, the registered face image 620 of the authenticated registered face data 700 and the registered attribute information 630 are connected, and the input face image 810, the photographing time 820, the registered face image 830 and registered attribute information 840 is created.

(Storage unit 1530)
Storage unit 1530 stores authentication history data 800 created by authentication history creation unit 1520. The upper limit number of authentication history data 800 to be stored is determined based on the capacity of the storage unit 1530, and if the number of stored authentication history data 800 exceeds the upper limit, refer to the photographing time 820 of the authentication history data 800. Then, the authentication history data 800 is deleted in order from the oldest history until the number returns to the upper limit number.

(Notification Department 1540)
The notification unit 1540 creates a notification image from the authentication history data 800 stored in the storage unit 1530, and displays the image on a display monitor or the like that constitutes the notification unit 1540.

FIG. 13 shows an example of creating a notification image 900. In the example of FIG. 13, input face image 810, registered face image 830, photographing time 820, and registered attribute information 840 included in authentication history data 800 are arranged side by side as notification information 910, 920, 930, and 940, respectively, and are converted into images. ing. Thereby, the operator visually confirms whether or not the registered person has been detected.

<Operation of face recognition system>
The operation of the face authentication system 1000 to which the present invention is applied will be described below with reference to the flowcharts shown in FIGS. 14 to 17. Note that an example will be described in which the registered face data 600 is stored in advance in the storage unit 1420 of the face authentication device 1400.

The learning process of the face authentication device 1400 shown in FIG. 14 is executed in advance. In the learning process, first, the input unit 1610 selects the registered face data 600 that is a face image that is extracted from a monitoring image showing a monitoring target area and that is obtained from the imaging device 1100 of the face authentication system 1000. A large amount of learning face image data to which a registered face image identifier 610 has been assigned in advance is received (step S1010).

Then, for each face image data for learning, the face image is input to the multilayer neural network 1440, and image feature amounts for learning are extracted (step S1030).

Then, lossT is calculated using the image feature amount for learning and the noise block (step S1040). Based on the calculated lossT, the noise block and the filter group of the multilayer neural network 1440 are learned to optimize the lossT (step S1050).

Then, it is determined whether the learning of the noise block and the filter group of the multilayer neural network 1440 has converged (step S1060). For example, if the number of repetitions of steps S1030 to S1050 has reached the upper limit, the learning is It is determined that the process has converged, and the process moves to step S1070. On the other hand, if the number of repetitions of steps S1030 to S1050 has not reached the upper limit, it is determined that the learning has not converged, and the process returns to step S1030.

Then, the finally learned noise block and the filter group of the multilayer neural network 1440 are stored in the storage unit 1420, and the learning process is ended.

Then, the noise block data stored in the storage unit 1420 is transmitted to the image processing device 1300 via the network 1200 by the transmission/reception unit 1410, and the noise block data is stored in the storage unit 1320 of the image processing device 1300. be done.

The detection process of the image processing apparatus 1300 shown in FIG. 15 is executed every time one monitoring image is acquired. In the detection process, first, a monitoring image showing a monitoring target area is acquired from the imaging device 1100 (step S1110). Then, the face image acquisition unit 1311 of the image processing unit 1310 extracts a face image from the monitoring image (step S1120). The face image acquisition unit 1311 determines whether one or more face images have been extracted (step S1130), and if no face images have been extracted, the detection process is ended without performing subsequent processing. . On the other hand, if one or more face images have been extracted, the face image acquisition unit 1311 creates input face data 200 from the detected face images and stores it in the storage unit 1320, and then moves the process to step S1140.

The following steps S1140 to S1150 are performed for each face image extracted by the face image acquisition means 1311.

The noise addition unit 1312 adds noise to the input face data 200 created by the face image acquisition unit 1311 using the noise block data stored in the storage unit 1320 to create noise-added input face data 300 ( Step S1140). At this time, by adding a noise image in which noise blocks are arranged side by side to the input face image 220 of the input face data 200, the noise-added input face data 300 including the noise-added face data 320 that is difficult to visually identify is generated. create. Next, the generated noise-added input face data 300 is transmitted to the face authentication device 1400 (step S1150).

When the processing of steps S1140 to S1150 is completed for all face images, the image processing unit 1310 ends the detection processing.

The face authentication device 1400 that has received the noise-added input face data 300 performs the matching process shown in FIG. 16. Note that the operation of the face authentication device 1400 in FIG. 16 described below is executed every time one piece of noise-added input face data 300 is received.

The face authentication device 1400 compares all registered face data 600 registered in advance in the storage unit 1420 with the received noise-added input face data 300 (step S1210). For matching, the trained multilayer neural network 1440 is used to compare the features of the received noise-added input face data 300 and the features of the registered face data 600, and calculate the degree of similarity between the two. It is determined whether the maximum similarity obtained as a result of the matching is equal to or greater than a predetermined authentication threshold (step S1220), and if it is equal to or greater than the authentication threshold, the maximum similarity is indicated with the received noise-added input face data 300. It is determined that the registered face data 600 originated from the same person, and the registered face data 600 is transmitted to the notification device 1500 (step S1230), and the matching process ends.

Further, the notification device 1500 that has received the input face data 200 and the registered face data 600 performs the notification process shown in FIG. 17. Note that the operation of the notification device 1500 in FIG. 17 described below is executed every time one pair of input face data 200 and registered face data 600 is received.

The notification device 1500 that has received the input face data 200 and the registered face data 600 creates authentication history data 800 using the input face data 200 and the registered face data 600 (S1310), and detects a specific person related to the authentication history data 800. is notified (S1320), and the notification process is ended. For example, the notification is made by displaying and outputting the notification image 900 related to the authentication history data 800 from a display unit (not shown) included in the notification device 1500.

As described above, in the face recognition system 1000 according to the embodiment of the present invention, the image processing device 1300 uses noise-added face data obtained by adding a noise image to an input face image for face recognition. Send to device 1400. The face authentication device 1400 matches the noise-added face data and the registered face image using a multilayer neural network. Here, the face authentication device 1400 inputs the learning face image data into the convolutional neural network in advance, and inputs the filter used in the previous convolutional layer included in the multilayer neural network and the noise image (filter with a predetermined stride). In the case of convolution, the regions of the noise image corresponding to each region to be convolved are orthogonal, and only the probability output corresponding to the registered person obtained by the multilayer neural network and the output element corresponding to the same person are 1 Learning is performed so that the loss value calculated as the cross entropy of the one-hot vector with the teacher data becomes small. As a result, even if noise is added to the input facial data, face recognition tasks using a multilayer neural network can be performed with high accuracy.

Furthermore, the image processing device 1300 is trained in advance to have orthogonality with the filter 3010 of the previous convolution layer included in the multilayer neural network 1440 that performs matching in the face authentication device 1400 on a person image extracted from a captured image. The noise-added input face data 300, which is generated by adding a noise image based on the noise block 3020 and which makes it difficult to visually identify the subject of the face image, is transmitted to the face authentication device 1400. As a result, the matching process using the multilayer neural network 1440 can be executed without reducing accuracy, and when an authenticated face image identifier corresponding to the noise-added input face data 300 is received from the face authentication device 1400, the corresponding By transmitting the input face data 200 corresponding to the authenticated face image identifier to the notification device 1500, it is possible to acquire the face image data of the detection target person while considering the privacy of unrelated persons. In other words, it is determined whether or not a person appearing in a surveillance image is a person whose face has been registered in advance, using noise-added information that is difficult to distinguish visually, and only information about persons whose faces have been registered is used to determine whether or not the person is a person whose face has been registered in advance. It becomes possible to record and broadcast highly identifiable information. Note that in this embodiment, face recognition is performed using a face image; however, the present invention is not limited to this, and a person image indicating a person area is used to perform face recognition, including not only the face but also the degree of similarity in physique, clothing, etc. Authentication may also be performed.

Furthermore, regarding the network-type face authentication system 1000 that transmits a person image taken by a locally installed imaging device 1100 to the face authentication device 1400 via a network to authenticate or detect a specific person, it is important to protect the privacy of the person being photographed. This makes it possible to realize a data transmission method that takes into account consideration.

<Modified example>
Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, in this embodiment, the case where the image processing device 1300 transmits the input face data 200 corresponding to the authenticated face image identifier to the notification device 1500 has been described as an example, but the present invention is not limited to this, and other methods may be used. The information may be transmitted to a device such as the face authentication device 1400, for example.

Furthermore, in the present embodiment, the noise addition means 1312 adds a noise image based on the noise block 3020 to the input face image 220 created by the face image acquisition means 1311. The noise block 3020 may be added after random pixel deletion or random noise addition. For example, a randomly determined pixel of the input face image 220 is deleted, and then a noise image in which noise blocks 3020 are arranged is added, or random noise is added to the input face image 220, and then the noise block 3020 is added. 3020 noise images may be added.

However, it is preferable to ensure matching accuracy for an image obtained by performing random pixel deletion or random noise addition to the input face image 220 during learning of the multilayer neural network 1440. That is, after applying random pixel deletion and random noise addition to the learning face image data, the learning face image data is input to the multilayer neural network 1440, and the matching probability corresponding to the registered person is outputted. It is sufficient to learn so that the loss value calculated as the cross entropy with the teacher data of the one-hot vector in which only the output elements corresponding to the same person becomes 1 is small. At this time, the rate of random pixel missing (ratio of missing pixels to the total number of pixels) applied to the input face image 220 or the strength of random noise addition is the same as that applied to the training image data. It is preferable that In this way, by performing the processing of random pixel deletion and random noise addition to the input face image 220, in addition to adding noise to the input face image 220, it also hides characteristics such as periodicity of the noise block 3020 itself. This makes it possible to make it difficult to analyze the value of the noise block.

Furthermore, in the present embodiment, the noise adding unit 1312 adds the noise block 3020 to the input face image 220 created by the face image acquiring unit 1311, but it adds a plurality of noise blocks 3020 of different sizes simultaneously and in the same way. After learning, each noise image in which each noise block 3020 is arranged may be superimposed and added to the input face image 220. Since both of the plurality of noise blocks of different sizes can be learned to be orthogonal to the filter 3010 of the multilayer neural network 1440, even if they are superimposed, the influence of the noise can be eliminated and the data processing task will not be affected. In this way, variations in noise images other than each noise block included in one noise block group can be increased. Furthermore, by combining noise images created from noise blocks of different sizes, the apparent noise period can be largely camouflaged, making it difficult to visually estimate the characteristics of the noise. For example, consider two variations of the size of the noise block: 5×5 and 3×3. Then, as a single noise block, the noise period is 5 and 3, respectively, but in the composite noise image obtained by superimposing and adding each noise block, the noise period is 5 × 3 = 15, and the noise period is 5 × 3 = 15. The number of variations of pairs can also be increased to 25 patterns (5 x 5) x 9 patterns (3 x 3) = 225 patterns.

Alternatively, the entire noise image may be multiplied by an arbitrary weight and then added to the input face image 220. The region of the noise image corresponding to the convolution region of the filter has been learned to have orthogonality with the filter 3010 of the multilayer neural network 1440, so even if the entire noise image is multiplied by an arbitrary weight, it will not have orthogonality. Because it remains the same, the effects of that noise can be eliminated and it does not affect data processing tasks. Similarly, when superimposing each noise image generated based on each of multiple types of noise blocks and adding it to the input face image 220, each noise image as a whole is multiplied by an arbitrary weight and then superimposed. may be added to the input face image 220. The difficulty of visual identification due to addition of noise can be adjusted by the weight multiplied by the noise image. Therefore, an arbitrary weight value may be set based on a predetermined S/N ratio, or an arbitrary weight value may be determined based on the S/N ratio. Furthermore, the weight values may be set randomly, or different weight values may be set for each extracted facial image area.

Furthermore, in the present embodiment, the noise addition means 1312 adds the noise block 3020 to the input face image 220 created by the face image acquisition means 1311, but when learning the noise block 3020, each element of the noise block By adding a constraint such that the size of the value of and spatial variation fall within a predetermined range, specific elements of the noise block may be made less noticeable, that is, the periodicity of the noise block may be made less noticeable.

In addition, in this embodiment, face recognition using a face image is performed as a data processing task, but the data processing task is not limited to this. It is also possible to estimate attribute information such as.

Furthermore, in this embodiment, a face image is detected from a surveillance camera image, noise is added to the face image, and image processing is performed. Image processing such as posture estimation and person density estimation may be performed on the noise-added image.

Further, the learning unit 1630 may be provided in a learning device other than the face authentication device 1400, and the multilayer neural network 1440 learned by the learning device may be stored in the face authentication device 1400.

Further, although the case where the input data is an image has been described as an example, the present invention is not limited to this, and the input data may be data other than images. For example, the input data may be voice data and the data processing task may be voice recognition or speaker recognition. In this case, noise is added by adding noise data obtained by arranging noise blocks to spectrum data obtained from speech data, and a multilayer neural network including a convolution layer is used to perform speech recognition or speaker recognition. That's fine. Alternatively, noise is added to the audio signal by adding noise data in which noise blocks are arranged in a time-series direction, features are obtained using a multilayer neural network including convolutional layers, and RNN ( Speech recognition may be performed using Recurrent Neural Networks).

Further, although an example has been described in which noise is added by adding noise images in which noise blocks are arranged side by side from the upper left side without gaps, the present invention is not limited to this. For example, if the stride of the filter group used in the previous convolutional layer included in the multilayer neural network exceeds the filter size, a gap of "stride value - filter size" will occur between the filters before and after shifting the filters. For example, if the filter size is 3×3 and the stride is 4, there will be a gap of 4-3=1 between the filters applied during the convolution operation. Since the convolution calculation is skipped in this gap, noise may be added by adding noise images in which noise blocks are placed apart by this gap. In this case, the value of the gap between the noise blocks arranged in the noise image is a predetermined value (for example, 0) or a randomly set pixel value. In addition, in the case of a stride that allows noise blocks to be placed apart, it is possible to avoid convolution operations that span multiple noise blocks, so when learning noise blocks, it is not necessary to shift the elements of the noise block to calculate the derived noise. There is no need to generate blocks.

Further, although the case where the input image and the noise image are aligned and added together has been described as an example, the present invention is not limited to this. For example, the noise image may be shifted in at least one of the X-axis direction and the Y-axis direction by an offset value before being added to the input image. Thereby, even if the input image and the noise image are the same pair, variations in the noise-added image can be increased. However, in the noise-added image, in a portion corresponding to the offset, an area where noise is not added (noise-unapplied area) occurs, so orthogonality with the filter is no longer ensured. Therefore, when inputting the noise-added image to a multilayer neural network and performing a convolution operation in the previous convolution layer included in the multilayer neural network, the convolution operation result of this area where noise-unapplied areas and noise-applied areas are mixed is can reduce the performance of data processing tasks. Therefore, for example, the application of the filter of the convolution layer may be started from a position shifted by the offset, and the convolution process may be executed only in the area to which the noise image has been added. To this end, it is desirable to perform learning on the data processing task by shifting the amount of the offset to improve the accuracy of the data processing task under the same offset conditions. The above processing makes it possible to eliminate the influence of this offset.

As described above, those skilled in the art can make various changes within the scope of the present invention according to the embodiment.

1000 Face recognition system 1100 Imaging device 1110 Face image 1120 Person 1200 Network 1300 Image processing device 1310 Image processing section 1311 Face image acquisition means 1312 Noise addition means 1320 Storage section 1330 Transmission/reception section 1400 Face authentication device 1410 Transmission/reception section 1420 Storage section 1430 For input Data processing unit 1440 Multilayer neural network 1610 Input unit 1630 Learning unit 3010 Filter 3020 Noise block 3030 Noise added images 3200, 3310 Noise image

Claims (9)

a learning unit that learns filters used in each convolution layer in a multilayer neural network that performs a predetermined data processing task on input data, and noise data that is added to the input data;
Noise-added data obtained by adding noise data learned by the learning unit to the input data is input to the multilayer neural network, and the data processing task is performed based on the output of the multilayer neural network. an input data processing unit that obtains the result of
A data processing device comprising:
The learning unit inputs learning data to which the results of the data processing task have been assigned in advance to the multilayer neural network, and the noise data and the filter used in the previous convolution layer included in the multilayer neural network are configured to input the learning data to the multilayer neural network. A data processing device having orthogonality and learning so that the obtained result of the data processing task matches the result of the data processing task given in advance to the learning data. . The learning unit corresponds to each region to be convolved when convolving a filter group used in a previous convolution layer included in the multilayer neural network and a filter in the filter group used in the previous convolution layer with a predetermined stride. 2. The data processing device according to claim 1, wherein the filter and the noise data are learned so that the noise data region has orthogonality with the noise data region. The input data is an image,
The learning unit arranges one of a noise block group consisting of a predetermined reference noise block and a derived noise block obtained by shifting the reference noise block according to a predetermined shift pattern. The data processing device according to claim 2, wherein the data processing device learns to generate the noise data. 4. The data processing device according to claim 3, wherein the learning unit learns so that all noise blocks of the noise block group and each of the filters have orthogonality. The data processing device according to claim 4, wherein the learning unit obtains the derived noise block by shifting the noise block using the shift pattern according to the stride. The input data is an image,
The stride is an integer multiple of the size of the filter,
The learning unit learns so that a predetermined noise block and the filter have orthogonality,
3. The data processing device according to claim 2, wherein the noise data is obtained by arranging one or more noise blocks obtained through learning. The input data is an image,
The noise-added data is obtained by adding the noise data after dropping randomly determined pixels among the pixels of the input data,
The data processing according to any one of claims 1 to 6, wherein the learning section deletes randomly determined pixels among the pixels of the learning data and then inputs the data to the multilayer neural network for learning. Device. The input data is an image,
The noise-added data is obtained by adding random noise to the input data and then adding the noise data,
7. The data processing device according to claim 1, wherein the learning section adds random noise to the learning data and then inputs the data to the multilayer neural network for learning. a learning unit learns a filter used in each convolution layer in a multilayer neural network that performs a predetermined data processing task on input data, and noise data to be added to the input data;
The input data processing unit inputs noise-added data obtained by adding the noise data learned by the learning unit to the input data to the multilayer neural network, and outputs the noise added data to the multilayer neural network. determining a result of the data processing task based on the data processing method, the data processing method comprising:
The learning unit inputs learning data to which the results of the data processing task have been assigned in advance to the multilayer neural network, and the noise data and the filter used in the previous convolution layer included in the multilayer neural network are configured to input the learning data to the multilayer neural network. A data processing method having orthogonality and learning so that the obtained result of the data processing task matches the result of the data processing task assigned in advance to the learning data. .