JP6137916B2 - Signal processing apparatus, signal processing method, and signal processing system - Google Patents

Description

  The present invention relates to a signal processing technique for extracting a feature amount suitable for pattern identification from image data or the like.

  Image processing for detecting a specific pixel pattern appearing in an image or distinguishing it from another is known. The former is, for example, face detection (detects a face-like pattern from the image), and the latter is, for example, face authentication processing (identifies an individual from the detected face). For example, in the face authentication process, a face image is converted into feature amount data and registered in advance, and a given face image (a partial image of a face area) is converted into similar feature amount data and registered in advance. This is a process of determining similarity with the feature amount of the face image and identifying an individual according to the result.

  When the face authentication process is executed using a snapshot photographed by a digital camera or the like, the face image includes those photographed under various illumination conditions. In order to correctly identify even face images with different illumination conditions, it is desirable that the feature amount used for pattern identification is robust against fluctuations due to the illumination conditions of the pixel pattern. LBP (Local Binary Pattern) has been proposed as a feature quantity having such characteristics (Non-patent Document 2).

  In addition, a feature amount obtained by applying several tens of Gabor Wavelet filter processes to an input image and extracting LBP from each post-filter data has been proposed. For example, Non-Patent Documents 1 and 6 propose LGBP (Local Gabor Binary Pattern).

  According to Non-Patent Document 6, instead of using a feature obtained by simply extracting an LBP from an input image as a feature amount for face authentication, an LBP extracted from the result of applying a Gabor Wavelet filter process to the input image (LGBP) The face authentication accuracy is better when using.

  On the other hand, CNN (Convolutional Neural Networks) of Non-Patent Document 4 has been proposed as a method of learning a spatial filter suitable for an identification target and extracting a feature amount using the spatial filter obtained by learning. In the CNN, feature quantities are extracted by hierarchically performing spatial filter processing on an input image. For the learning of the spatial filter, an error back propagation method is generally used. The error back propagation method is a supervised learning method, and learning is performed using a set of learning data and correct data of CNN output for the learning data. That is, the coefficient of the spatial filter is updated so that the error between the CNN output for the learning data and the correct answer data becomes small.

  As described above, LGBP is widely used in face authentication and the like as an effective feature quantity for pattern identification. However, the Gabor Wavelet filter used in LGBP has a problem that the number of dimensions and the number of data are large and the processing load is large. Further, the Gabor Wavelet filter is not originally designed for identifying a specific pattern. Therefore, there is a possibility that there is a more appropriate spatial filter than the Gabor Wavelet filter for the purpose of specific pattern identification (for example, identification of a face image). Further, as described above, the CNN can design a spatial filter suitable for the identification target by learning. For this reason, it is conceivable to use a spatial filter designed based on learning of an actual identification target (for example, identification of a face image) instead of a Gabor Wavelet filter.

W. Zhang, S. Shan, W. Gao, X. Chen, and H. Zhang, “Local Gabor Binary Pattern Histogram Sequence (LGBPHS): A Novel Non-Statistical Model for Face Representation and Recognition”, Proc. IEEE International Conference on Computer Vision, pp. 768-791, 2005. T. Ojala, M. Pietikainen, and D. Harwood, “A Comparative Study of Texture Measures with Classification Based on Featured Distributions”, Pattern Recognition, Vol. 29, pp. 51-59, 1996. Ichiro Murase, Shunichi Kaneko, Satoru Igarashi, “Robust Image Matching by Incremental Sign Correlation”, IEICE Transactions D-II, Vol. J83-D-II, No. 5, pp. 1323-1331, 2000. Y. LeCun, K. Kavukvuoglu, and C. Farabet, “Convolutional Networks and Applications in Vision”, Proc. IEEE International Symposium on Circuits and Systems, pp. 253-256, 2010. S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification”, Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 539-546, 2005. Z. Lei, S. Liao, R. He, M. Pietikainen, SZ Li, “Gabor Volume Based Local Binary Pattern for Face Representation and Recognition”, Proc. IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1-6 , 2008.

  However, in order to apply the error back-propagation method, which is a CNN learning method, all processing elements in a series of image processing on an input image need to be continuous functions (differentiable). That is, since the step function that is a discontinuous function is included in the encoding such as LBP and the incremental code, it cannot be learned by the error back propagation method. As a result, it is not possible to optimize a CNN that can extract a feature amount suitable for pattern identification by learning.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique capable of extracting a feature amount suitable for pattern identification from image data or the like.

  In order to solve the above-described problems, the signal processing device of the present invention has the following configuration. That is, in the signal processing device, generation means for generating processing result data by executing spatial filter processing on input data, and predetermined encoding processing using a discontinuous function for the processing result data First encoding means for generating encoding processing result data and approximate encoding processing result data by executing approximate encoding processing approximating the discontinuous function with a continuous function for the processing result data A second encoding means for generating the updating means, an updating means for updating the weighting coefficient of the spatial filter processing based on the approximate encoding processing result data, and the updating of the weighting coefficient by the updating means Control means for controlling to provide processing result data to the second encoding means, and to provide the processing result data to the first encoding means in other cases. .

  ADVANTAGE OF THE INVENTION According to this invention, the technique which enables extraction of the feature-value suitable for pattern identification from image data etc. can be provided.

It is a figure explaining the outline | summary of the signal processing in 1st Embodiment. It is a figure explaining the encoding process in 1st Embodiment. It is a figure explaining the approximate encoding process in 1st Embodiment. It is a conceptual diagram of the CNN learning device in 1st Embodiment. It is a figure which shows the structure of the data processor in 1st Embodiment. It is a flowchart which shows operation | movement of each mode in the data processor in 1st Embodiment. It is a figure explaining the relationship between the coefficient of a tanh function, and a shape. It is a figure explaining the process which extracts LBP from an input pixel value. It is a figure explaining the process which extracts an increment code | symbol from an input pixel value. It is a figure explaining the example of storage to the memory of a network structure of CNN. It is a figure explaining the example of the storing to the memory of the filter coefficient of CNN. It is a figure which shows the network structure of CNN exemplarily. It is a figure which shows an example of the result image of face authentication. It is a figure which shows the structure of the data processor in 2nd Embodiment. It is a block diagram which shows the structure of the signal processing part in 2nd Embodiment. It is a figure which shows the structure of the data processing system by 3rd Embodiment. It is a figure which shows the structure of the client apparatus in 3rd Embodiment. It is a block diagram which shows the structure of the feature extraction part in 3rd Embodiment. It is a figure which shows the Gaussian function which approximates a pulse function and the said pulse function.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
As a first embodiment of the information processing apparatus according to the present invention, a data processing apparatus that extracts a feature quantity suitable for face authentication from a face image will be described below as an example. Here, the face authentication means a process for identifying an individual by comparing a feature amount extracted from an input face image with registered data created in advance. In the first embodiment, an example applied to feature extraction processing in face authentication will be described, but the present invention can also be applied to feature extraction processing in other pattern identification.

<1. Overview of signal processing>
FIG. 1 is a diagram for explaining an outline of signal processing in the first embodiment. Reference numeral 100 denotes an input image. Reference numeral 109 denotes CNN (Convolutional Neural Networks) processing, which performs CNN processing on the input image 100. Reference numeral 110 denotes an encoding process, which executes a predetermined encoding process on the CNN process result data. Details of CNN (Convolutional Neural Networks) processing 109 and encoding processing 110 (LBP and incremental code) will be described below.

<1.1. CNN (Convolutional Neural Networks) processing>
The CNN is a neural network that extracts a feature amount by propagating an input image in the forward direction and performing a convolution operation using a plurality of different spatial filters. Here, the two-dimensional data storing the convolution calculation result is referred to as “feature extraction plane”.

  Subsequently, the feature extraction plane is divided into non-overlapping local regions, and an average value calculation (integration process) for each local region is executed. By averaging, robustness against minute geometrical variations (average movement, rotation, etc.) of the identification target in the input image is improved. Here, the two-dimensional data generated by such processing is referred to as “integrated plane”. Then, a feature extraction surface is generated by performing convolution operation using a plurality of different spatial filters again on the integrated surface. In this way, the CNN extracts a desired feature amount by hierarchically repeating the feature extraction process and the integration process.

  Reference numerals 101a to 101c denote feature extraction planes in the first layer, which are two-dimensional data for storing convolution calculation results by a two-dimensional spatial filter. Reference numeral 105 denotes the relationship between the input and output of a two-dimensional convolution operation for calculating the feature extraction plane 101c from the input image 100. The following formula (1) is a calculation formula for the convolution calculation for generating the feature extraction surface 101 c from the input image 100.

i (x, y): input pixel value at coordinates (x, y) u (x, y): calculation result at coordinates (x, y) w (c, r): filter at coordinates (x + c, y + r) coefficient
width, height: filter size i (x, y) corresponds to a pixel value (luminance value) of the input image 100. The result of nonlinear processing of u (x, y) obtained here by a hyperbolic tangent (tanh) function or the like becomes the pixel value of the feature extraction surface 101c. Similarly, 101a and 101b are generated by convolution calculation on the input image 100. Here, the spatial filters used to generate 101a-c have different coefficients.

  Reference numerals 102a to 102c denote integration planes, which are two-dimensional data for storing the results of integration processing. Reference numeral 106 denotes a relationship between input and output of integration processing for calculating the integration surface 102c from the feature extraction surface 101c. Reference numeral 107 represents the relationship between the input and output of a two-dimensional convolution operation for calculating the feature extraction surface 103c from the integrated surface 102c. Reference numerals 103a to 103c denote feature extraction planes in the third hierarchy, which are results obtained by adding the values obtained by performing non-linear processing on the convolution calculation output results for all of the integration planes 102a to 102c in the previous hierarchy using a tanh function or the like. Accordingly, in the example shown in FIG. 1, nine different spatial filters are used to generate the feature extraction surfaces 103a-c from the integrated surfaces 102a-c.

  The parameter determined by learning is a coefficient of the spatial filter used for generating the feature extraction surface. In the example shown in FIG. 1, three spatial filters used to generate feature extraction surfaces 101a to 101c from the input image 100, and spaces used to generate feature extraction surfaces 103a to 103c from the integrated surfaces 102a to 102c. The coefficients of a total of 12 spatial filters of 9 filters are determined by learning.

<1.2. Encoding process>
The encoding process 110 is a code based on the magnitude relationship between the target pixel (target area) and the reference pixel (reference area) with respect to the CNN process result data (in the example illustrated in FIG. 1, the feature extraction surfaces 103a to 103c). This is a process to apply Such encoding processing includes LBP or incremental code.

<1.2.1. LBP>
FIG. 8 is a diagram for explaining processing for extracting an LBP from an input pixel value. LBP is based on the pixel value of the pixel of interest (x, y) and the pixel values of eight reference pixels (x + x _n , y + y _n ) surrounding the pixel of interest. This is a process of encoding.

ここで、
ｉ（ｘ，ｙ）：座標（ｘ，ｙ）での入力画素値
ＬＢＰ（ｘ，ｙ）：座標（ｘ，ｙ）でのＬＢＰ
（ｘ_ｎ，ｙ_ｎ）：参照画素の注目画素に対する相対位置
ｘ_ｎ＝｛−１，０，１｝，ｙ_ｎ＝｛−１，０，１｝，ｘ_ｎ ^２＋ｙ_ｎ ^２≠０
ただし、

here,
i (x, y): input pixel value at coordinates (x, y) LBP (x, y): LBP at coordinates (x, y)
(X _n , y _n ): relative position of the reference pixel with respect to the target pixel x _n = {− 1, 0, ₁ }, y _n = {− 1, 0, ₁ }, x _n ² + y _n ² ≠ 0
However,

である。

It is.

In the example shown in FIG. 8, (x _n , y _n ) surrounds the target pixel counterclockwise starting from the pixel adjacent to the left of the target pixel. In particular,
(X ₀ , y ₀ ) = (− 1, 0)
(X ₁ , y ₁ ) = (− 1, 1)
(X ₂ , y ₂ ) = (0, 1)
...
(X ₇ , y ₇ ) = (− 1, −1)
It is said.

  Equation (3) is a step function (step function), which is 1 when the reference pixel value is greater than or equal to the target pixel value, and 0 when the opposite is true. Since the LBP expresses only the magnitude relationship between the target pixel and the reference pixel, even if the pixel value fluctuates due to a change in the illumination condition, the LBP should be identified as long as the magnitude relationship between the brightness of the target position and the reference position does not change. There is a characteristic that can be.

<1.2.2. Increment sign>
FIG. 9 is a diagram for explaining processing for extracting an incremental code from an input pixel value. Even if the incremental code of Non-Patent Document 3 is applied instead of the above-described LBP, the same effect as that of LBP can be obtained. The incremental sign at the coordinates (x, y) is calculated by the following equation.

i (x, y): Input pixel value at coordinates (x, y) IS (x, y): Incremental sign at coordinates (x, y) (x ₀ , y ₀ ): Relative to reference pixel of reference pixel Position x ₀ = {-1, ₀ , 1}, y ₀ = {-1, ₀ , 1}, x ₀ ² + y ₀ ² ≠ 0
Comparing equation (2) and equation (4), it can be seen that the LBP with n = 0 corresponds to the incremental sign. In the example shown in FIG. 9, a pixel whose relative position with respect to the target pixel is (x ₀ , y ₀ ) = (0, −1) is used as a reference pixel.

<1.2.3. Example of encoding process>
FIG. 2 is a diagram for explaining the encoding process (first encoding process) in the first embodiment. In addition, the same number is provided about the same component as FIG. Here, in order to simplify the description, an example in the case of using an incremental code will be described. Even when LBP is used, it can be applied in the same manner as the incremental sign.

  The encoding process result data 104a to 104c are two-dimensional data for storing the results of applying the encoding process to the feature extraction planes 103a to 103c that are outputs of the CNN. Reference numeral 108 denotes a relationship between input and output in the encoding process for calculating the encoding process result data 104c from the feature extraction surface 103c.

  202a-c is a pixel comparison process, which compares the pixel values of the target pixels 203a-c on the feature extraction surfaces 103a-c with the pixel values of the reference pixels 204a-c. Here, the difference obtained by subtracting the pixel values of the target pixels 203a to 203c from the reference pixels 204a to 204c is calculated.

  Reference numerals 201a to 201c denote comparison processing result data, which are two-dimensional data for storing results obtained by processing the feature extraction surfaces 103a to 103c by the pixel comparison processes 202a to 202c. The comparison processing result data 201a to 201c are feature amounts having different directional characteristics according to the relative positions of the reference pixels 204a to 204c used for comparison. For example, when the comparison processing result data 201b is generated, the pixel adjacent to the target pixel 203b is set as the reference pixel 204b, and thus the generated comparison processing result data 201b has a directional characteristic corresponding to a change in the vertical pixel value. The feature amount has Here, in order to generate feature amounts having various directional characteristics, the relative positions of the reference pixels 204a to 204c are set to different directions (horizontal, vertical, and diagonal).

  The encoding process result data 104a to 104c are two-dimensional data for storing the results of processing the comparison process result data 201a to 201c by the step function process 205. The step function process 205 calculates the step function shown in Expression (3) with each pixel value of the comparison process result data 201a to 201c as an input. Expression (5) is a calculation expression for generating the encoding processing result data 104a to 104c from the feature extraction surfaces 103a to 103c.

u (x, y): Pixel value of the previous layer surface at coordinates (x, y) v (x, y): Calculation result at coordinates (x, y) (x ₀ , y ₀ ): Reference pixel In the example shown in FIG. 2, the relative positions (x ₀ , y ₀ ) of the reference pixels are (−1, 0), (0, −1), (1, − 1).

  The encoding process result data 104a to 104c express only the magnitude relationship between the pixel values of the target pixels 203a to 203c on the feature extraction surfaces 103a to 103c and the pixel values of the reference pixels 204a to 204c. Therefore, even when the pixel pattern of the feature extraction surfaces 103a to 103c changes due to a change in illumination conditions, the encoding processing result is obtained unless the magnitude relationship between the pixel values of the target pixels 203a to 203c and the pixel values of the reference pixels 204a to 204c is reversed. The pixel values of the data 104a to 104c do not change.

<2. Determination of filter coefficients by learning>
In the first embodiment, the filter coefficient of CNN is determined by learning so that the output result of the signal processing shown in FIG.

<2.1. Learning device>
FIG. 4 is a conceptual diagram of the CNN learner in the first embodiment. Here, a known Siasese learning device (Non-Patent Document 5) is used. The Siasese learning device is a learning device that performs learning based on a pair of input data and a label indicating whether the pair of input data is in the same class. Specifically, the CNN is learned so that the distance between CNN outputs is small for input data of the same class, and conversely, the distance between CNN outputs is large for input data of different classes.

  The learning database 406 is a database that stores learning data. Here, the learning data is data including a face image and a person ID corresponding to the face image. Here, the person ID is for identifying a person corresponding to the face image, and is represented by an integer value, for example. For example, the values of 0, 1, and 2 are set for the person ID in the order registered in the database. Further, character string data such as a name and a nickname may be associated with the person ID. Preferably, the face image is an image converted so that both eyes are aligned horizontally and have a predetermined size. Here, in order for the output result of the signal processing to be a robust feature quantity against various variations, it is desirable that the face image includes various variations with respect to the face orientation in the pan / tilt direction, facial expressions, illumination conditions, and the like.

  The image pair selection 407 selects a face image pair to be used for learning from the learning database 406. Here, it is assumed that a pair of face images is randomly selected from all the face images stored in the learning database 406 each time. Then, the selected face images 401a and 401b are input to the CNN processes 109a and 109b, respectively. Further, a value of 0 is set in the label 405 when the IDs of the selected face images are the same, and 1 when they are different. The label 405 is used when calculating the loss in the error (Loss) calculation 404.

  The CNN processes 109a and 109b execute a CNN process having the same network configuration as that of the CNN process 109 shown in FIG. The CNN processing 109a, b performs CNN processing on the face images 401a, b selected by the image pair selection 407 to generate CNN processing result data. Note that the CNN processes 109a and 109b share the same filter coefficient 408.

  The encoding processes 110a and 110b execute the same process as the encoding process 110 described with reference to FIG. The encoding processes 110a and 110b encode the CNN process result data generated by the CNN processes 109a and 109b to generate encoding process result data. Here, since the CNN processes 109a and 109b and the encoding processes 110a and 110b have the same configuration and the filter coefficients of the CNN processes 109a and 109b are the same, the encoding processes 110a and 110b are generated if the input images are the same. The encoding process result data to be the same.

  The distance calculation 403 calculates the distance between the two encoding process result data generated by the encoding processes 110a and 110b. Here, the L1 norm between the cases where the encoding process result data is a vector is used as the distance measure. For example, if the size of one surface of the encoding process result data is W × H and the number of surfaces of the encoding process result data is N, the dimension of the vector is W × H × N. Note that other distance measures such as Euclidean distance and cosine distance may be used. Expression (6) is a calculation expression for calculating the L1 norm between the encoding process result data generated by the encoding processes 110a and 110b.

w: Filter coefficient of CNN E (w): L1 norm between encoding process result data v _n (w): Encoding process result data generated from input image n (n: index in image pair)
w is a vector whose elements are CNN filter coefficients.
Note that both v and E are functions of w because their values change depending on the filter coefficient of CNN. v ₁ (w) and v ₂ (w) are encoding process result data generated from the face images 401a and 401b, respectively.

  In the loss calculation 404, the loss is calculated based on the L1 norm calculated by the distance calculation 403 and the label 405 generated by the image pair selection 407. Expression (7) is a calculation expression for calculating Loss from the L1 norm and the label 405.

Y: Label value (0: Person ID of image pair is the same, 1: Person ID of image pair is different)
L (w): Error (Loss)
Q: A constant set to the upper limit value of E (w).

  When the person IDs of the face images 401 a and 401 b are the same, Y = 0 is input as the label 405. In this case, L (w) which is Loss is a small value if the value of E (w) is small, and conversely a large value if the value of E (w) is large. This means that for the same person, the value of Loss decreases as the distance between the encoding process result data decreases.

  Further, when the person IDs of the face images 401a and 401b are different, Y = 1. In this case, L (w) which is Loss is a large value when the value of E (w) is small, and conversely, it is a small value when the value of E (w) is large. This means that for different persons, the value of Loss decreases as the distance between the encoding process result data increases.

  Through the processing described above, the value of Loss is calculated from the pair of face images. Subsequently, based on the calculated loss, the CNN filter coefficient 408 is updated by the error back propagation method. A procedure for updating the filter coefficient 408 by the error back propagation method will be described below.

<2.2. Learning by back propagation method>
In the error backpropagation method, parameters are updated using a gradient descent method in order to minimize the error function. Here, the error function is L (w), and the parameter is the filter coefficient w. Here, w needs to be initialized before learning starts. Therefore, here, w is initialized with a random number. Alternatively, a coefficient of a known spatial filter such as a Gabor Wavelet filter or a Sobel filter may be set and initialized. Further, additional re-learning may be performed by setting a filter coefficient obtained by previous learning. Expression (8) is an expression showing a method of updating the i-th element of w by the gradient descent method.

w _i : i-th element of w before update w _i ': i-th element of w after update ρ: update coefficient To update w _i by calculating equation (8), ∂L (w ) / it is necessary to obtain the ∂w _i. Here, since L (w) depends only on w _i through E (w), the chain law of partial differentiation is applied to transform ∂L (w) / ∂w _i as shown in equation (9). .

Here, ∂L (w) / ∂E (w) is obtained by partial differentiation of Equation (7) with respect to E (w). Further, ∂E (w) / ∂w _i, similar to that decompose ∂L (w) / ∂w _i, can be modified as shown in equation (10).

v _j : j-th element of v (w) Here, ∂E (w) / ∂v _j is obtained by partial differentiation of equation (6) with respect to v _j . Here, ∂E (w) / ∂v _j needs to be calculated for each of v1 _j included in v1 (w) and v2 _j included in v2 (w).

In Equation (10), the result of decomposition for each element v _{j of the} encoding processing result data v (w) is added. This is because E (w) depends on w _i through all elements v _j of v (w).

∂v _j / ∂w _i can be further modified as shown in equation (11).

∂ｖ_ｊ／∂ｕ_ｊは、式（５）を偏微分することにより得られる。さらに、∂ｕ_ｊ／∂ｗ_ｉは、式（４）をｗ_ｉについて偏微分することにより得られる。

∂v _j / ∂u _j is obtained by partial differentiation of equation (5). Furthermore, ∂u _j / ∂w _i is obtained by equation (4) is partially differentiated for _{w i.}

As described above, in the backpropagation, as indicated by broken line arrow in FIG. 4, ∂L (w) / ∂E (w), ∂E (w) / ∂v j, the partial differential ... such The calculation is sequentially performed in the opposite direction to the case of calculating Loss from the input face image. This gives the final ∂L (w) / ∂w _i. Then, the filter coefficient is updated according to Expression (8).

Here, when the back propagation from the distance calculating 403 the encoding process 110a, the b, ∂E (w) / ∂v j _are those corresponding to _v 1 _(w), the one corresponding to _v 2 (w) Divide and propagate back. The CNN filter coefficient 408 is updated first by back propagation for v ₁ (w) and then updated by back propagation for v ₂ (w).

<2.3. Approximate encoding process>
As described above, in order to learn the CNN filter coefficient 408 by the error back-propagation method, all operations need to be differentiable functions. However, in the step function process 205 in the encoding processes 110a and 110b, the step function shown in Expression (3) is used. Since the step function is a discontinuous function at t = 0, it cannot be differentiated in the vicinity thereof. Therefore, こ の v _j / ∂u _j in equation (11) cannot be calculated as it is.

  Therefore, in the first embodiment, the step function in the step function processing 205 is approximated by replacing it with a differentiable continuous function. Here, the tanh function is used as a continuous function that approximates the step function, but any function may be used as long as it is a continuous function that can sufficiently approximate the step function. For example, it can be modified to use a sigmoid function. Here, an encoding process in which a step function is approximated by a continuous function is referred to as an “approximate encoding process”.

  FIG. 3 is a diagram illustrating the approximate encoding process (second encoding process) in the first embodiment. This corresponds to the encoding process of FIG. 2 in which the step function is replaced with a tanh function. Note that the same constituent elements as those in FIG.

  The approximate encoding process result data 302a to 302c are two-dimensional data for storing the results of processing the comparison process result data 201a to 201c by the tanh function process 301. The tanh function processing 301 generates approximate coding processing result data 302a to 302c by calculating the tanh function with each pixel value of the comparison processing result data 201a to 201c as an input.

  Expression (12) is a calculation expression for generating the approximate encoding process result data 302a to 302c from the feature extraction planes 103a to 103c by the approximate encoding process shown in FIG.

u (x, y): pixel value of the previous layer surface at coordinates (x, y) v to (x, y): operation result at coordinates (x, y) k: coefficient for determining the slope of the tanh function Here, k is a coefficient that determines the slope of the tanh function.

  FIG. 7 is a diagram illustrating the relationship between the coefficient of the tanh function and the shape. As the value of k increases, the slope of the tanh function increases. By setting the value of k as k >> 1, the tanh function has characteristics close to those of the step function.

  If the value of k is too large, the result of differentiating the tanh function is a very large value near t = 0, and close to 0 otherwise. This means that in the learning process, the filter coefficient is greatly updated near t = 0, and the filter coefficient is hardly updated otherwise. As a result, there arises a problem that learning is difficult to converge. However, if the value of k is set small, the error between the tanh function and the step function increases. In addition, it turned out that a more suitable result is obtained by experiment from k = 0.5-4.0.

<3. Configuration of data processing apparatus>
FIG. 5 is a diagram illustrating a configuration of the data processing device according to the first embodiment. In the following, a data processing apparatus capable of performing both of the above-described learning of CNN filter coefficients and pattern identification processing based on the determined filter coefficients will be described.

  The data storage unit 501 is a part that holds image data, and is usually configured by a hard disk, a flexible disk, an optical storage medium (CD, DVD), a semiconductor storage medium (memory cards of various standards, USB memory), or the like. In addition to image data, the data storage unit 501 can store programs and other data. Alternatively, a part of the RAM 505 described later may be used as the data storage unit 501. Alternatively, a storage device of a destination device connected by the communication unit 502 described later may be virtually configured to be used via the communication unit 502.

  The display unit 507 is a device that displays an image before image processing, an image after image processing, or an image such as a GUI, and generally uses a CRT, a liquid crystal display, or the like. Alternatively, it may be a display device outside the device connected by a cable or the like.

  The input unit 506 is a device for inputting instructions and data from the user, and includes a keyboard and a pointing device. Note that examples of the pointing device include a mouse, a trackball, a trackpad, and a tablet. Alternatively, when the data processing device is applied to a device such as a known digital camera device or printer, the data processing device may be configured with buttons, a dial, or the like. Further, the keyboard may be configured by software (software keyboard), and may be configured to input characters by operating buttons, dials, or the pointing device mentioned above.

  Alternatively, the display unit 507 and the input unit 506 may be the same device as in a known touch screen device. In that case, the input by the touch screen is handled as the input of the input unit 506.

  Reference numeral 503 denotes a CPU that executes the above-described processes and controls the operation of the entire data processing apparatus. The ROM 504 and RAM 505 provide the CPU 503 with programs, data, work areas, and the like necessary for the processing. When a program necessary for processing to be described later is stored in the data storage unit 501 or stored in the ROM 504, the program is once read into the RAM 505 and executed. In FIG. 1, although there is only one CPU (CPU 503), a configuration in which a plurality of CPUs are provided may be employed.

  A communication unit 502 is an I / F for performing communication between devices. For example, a known wired LAN standard represented by the IEEE 802.3 series standard, a wired communication system such as USB (Universal Serial Bus), IEEE 1284, IEEE 1394, a telephone line, or the like may be used. Alternatively, it may be a wireless communication system such as infrared (IrDA), a known wireless LAN standard represented by the IEEE 802.11 series standard, Bluetooth (registered trademark), UWB (Ultra Wide Band), or the like.

  Although FIG. 5 shows a diagram in which the input unit 506, the data storage unit 501, and the display unit 507 are all included in one apparatus, they may be configured separately. In that case, each part is connected so that communication is possible mutually by a well-known communication system. There may be additional components other than those described above.

<4. Processing flow of data processing apparatus>
FIG. 6 is a flowchart showing the operation in each mode in the data processing apparatus according to the first embodiment. In the following, the CPU 503 executes various programs to execute the processing shown in the following flowchart.

  In step S601, the operation mode is determined. The operation mode is designated by the user through the input unit 506, for example. Here, it is assumed that there are three types of operation modes: (a) learning mode, (b) identification mode, and (c) registration data creation mode. Hereinafter, the processing flow of the data processing apparatus in each mode will be described.

<4.1. (A) Learning mode>
The learning mode is an operation mode for performing learning. It is assumed that the face image used for learning is created according to the following procedure prior to the learning process, and is stored in the data storage unit 501 in association with the corresponding person ID. First, the image data stored in the data storage unit 501 is read into the RAM 505. Next, the image data in the RAM 505 is converted into an 8-bit unsigned luminance image. Then, a face area is detected by a known face detection method, and the face image resized to a predetermined size is stored in the data storage unit 501. Preferably, facial organ positions such as eyes, nose, and mouth are detected, and image conversion is performed so that both eyes are aligned horizontally and have a predetermined size based on the detected organ positions. For detecting the organ position, a known Active Appearance Model, Active Shape Model, or the like can be used.

  Further, it is desirable that the face image includes various variations in the face direction in the pan / tilt direction, facial expressions, illumination conditions, and the like so that the feature amount is robust against various variations. The above processing may be performed by the CPU 503, or the result of executing similar processing by an external device may be stored in the data storage unit 501 via the communication unit 502.

  In step S602, a learning number counter p is initialized to p = 0. Here, the learning number counter p is used to determine completion of learning in step S603.

  In step S603, it is determined whether learning for the number of repetitions designated in advance has been completed. Here, the number of repetitions is set in advance as M (M is an integer of 1 or more). In step S603, it is determined whether p <M is satisfied. If not established, it is determined that the learning has been completed, and the processing according to the flowchart of FIG. 6 is terminated. On the other hand, when it is established, it is determined that the learning is not completed, and the process proceeds to steps S604 to S611.

  In step S604, the first face image used for learning is randomly selected from the data storage unit 501 and stored in the RAM 505. Further, the person ID associated with the selected face image is stored in the RAM 505.

  In step S605, CNN processing is performed on the face image selected in step S604 to generate CNN processing result data. Here, it is assumed that the network configuration and filter coefficient of CNN are stored in RAM 505. Of course, it may be stored in the data storage unit 501 or the ROM 504. In this case, the CNN processing is executed after the CNN network configuration and filter coefficients are once read into the RAM 505.

  Details of the network configuration of the CNN, the storage format of the filter coefficient in the RAM 505, and the operation of the CPU 503 when executing the CNN processing will be described in order.

  FIG. 10 is a diagram for explaining an example of storage in the memory of the network configuration of the CNN. Here, a three-layer CNN network configuration is shown, but the configuration is not limited to three layers, and is generally a configuration of an L layer (L is an integer of 1 or more). Here, the RAM 505 is a set of three types of data, that is, processing contents (“feature extraction” or “integration”) 1001 of each layer, the number of surfaces to be generated 1002, and an address 1003, arranged in order from the first layer. Hold on. In addition, at the position indicated by the address 1003, the two-dimensional arrays 1004a to 1004c are stored in which all the combinations of each surface of the previous layer and each surface of the current layer are ○ when there is connection and × when there is no connection. .

  FIG. 12 is a diagram visualizing the network configuration of the CNN shown in FIG. The method for storing the CNN network configuration in the RAM 505 is not limited to the example shown in FIG. Any format can be used as long as the CPU 503 can identify the processing in each layer, the number of planes generated in each layer, and the connection relationship between layers.

  FIG. 11 is a diagram for explaining an example of storing CNN filter coefficients in a memory. Here, the size and coefficients of the spatial filter are held in the RAM 505 as a one-dimensional array. Further, a one-dimensional array in which the head addresses of the filter coefficients are arranged in order is held in the RAM 505. Here, the RAM 505 stores a spatial filter necessary for generating all feature extraction planes.

  For example, in the example shown in FIG. 10, three spatial filters necessary for generating a first layer feature extraction surface from an input image and a third layer feature extraction surface from a second layer integration surface are generated. A total of 12 spatial filters, including the 9 spatial filters necessary for this, are held. Here, it is assumed that the RAM 505 holds the spatial filter necessary for generating the feature extraction plane of the first hierarchy and the spatial filter necessary for generating the feature extraction plane of the third hierarchy in this order.

  It is assumed that initial values are set in advance for the filter coefficients. The initial value may be a random number or may be a filter coefficient obtained by previous learning. Furthermore, it may be a coefficient of a known spatial filter such as a Gabor Wavelet filter or a Sobel filter. The method of storing CNN filter coefficients in the RAM 505 is not limited to the example shown in FIG. Any format may be used as long as the CPU 503 can identify the size and coefficient of each spatial filter.

  The CPU 503 executes processing for each surface of the previous layer in accordance with a two-dimensional array representing the connection between layers stored at the position indicated by the address 1003. Here, the number of connection relationships to be read in each hierarchy is determined by the number of faces in the previous hierarchy (1 if the previous hierarchy is an input image) and the number of faces in the current hierarchy. For example, the number of connection relationships read in the third hierarchy is 3 × 4 = 12, since the number of faces in the previous hierarchy is 3 and the number of faces in the current hierarchy is 4.

  The CPU 503 executes the processing indicated by the processing content 1001 for each surface of the previous layer. When the processing content 1001 is feature extraction, the spatial filter shown in FIG. 11 is read into the RAM 505 and the spatial filter processing is executed. When reading the kth filter coefficient, first, the width (height) and height (height) of the spatial filter stored at the position of address k and address k + 1 are read. Next, based on the values of width and height, a spatial filter coefficient is read in order from the position of the address k + 2, thereby creating a two-dimensional spatial filter having a size of width × height. On the other hand, when the processing content 1001 is “integration”, the above-described integration processing is executed.

  In step S606, the approximate encoding process result data is generated by executing the approximate encoding process using the tanh function shown in Expression (12) for the CNN process result data generated in step S605.

  In step S607, the second face image used for learning is selected from the data storage unit 501 and stored in the RAM 505. Here, it is assumed that the second face image is randomly selected from the ones excluding the first face image selected in step S604. Further, the person ID of the selected face image is stored in the RAM 505. Similar to the first face image, the CNN process and the approximate encoding process using the tanh function are executed in the order of steps S605 and S606 to generate approximate encoding process result data.

  In step S608, it is confirmed whether or not the two person IDs selected in steps S604 and S607 match. If the two person IDs match, a label “0” is generated, and if they do not match, a label “1” is generated.

  In step S609, Loss is calculated using the approximate encoding processing result data generated from the two face images and the label generated in step S608. First, an L1 norm is generated according to equation (6) using two approximate encoding processing result data. Next, Loss is calculated according to Equation (7) using the L1 norm and the label.

  In step S610, the CNN filter coefficient stored in the RAM 505 is updated by the error back propagation method described above using the loss calculated in step S609. In step S611, the learning number counter p is incremented, and the process returns to step S603.

<4.2. (B) Identification mode>
The identification mode is an operation mode for performing pattern identification using the signal processing result. The pattern identification result is used for face authentication processing, for example. Assume that a face image used for face authentication is created prior to face authentication processing according to the following procedure and stored in the RAM 505. First, the image data stored in the data storage unit 501 is stored in the RAM 505. Next, the image data in the RAM 505 is converted into an 8-bit unsigned luminance image. Then, a face area is detected by a known face detection method, and a face image resized to a predetermined size is stored in the RAM 505. At this time, the position and size of the detected face area in the original image are stored in the RAM 505 in association with the face image as information for displaying the result of face authentication. Alternatively, the result of executing the same processing by the external device may be stored in the RAM 505 via the communication unit 502.

  In step S612, preprocessing is performed on the read face image. Specifically, the face organ position is detected using a known Active Appearance Model, Active Shape Model, etc., so that both eyes are aligned horizontally and have a predetermined size based on the detected organ position. Convert image.

  In step S605, CNN processing result data is generated by performing CNN processing on the preprocessed face image, as in the case of the learning mode described above.

  In step S613, the incremental encoding process shown in Expression (5) is performed on the CNN process result data generated in step S605. A vector in which increment codes for all pixels of interest are arranged is used as a feature amount used for identification.

  In step S614, the dimension of the feature amount generated in step S613 is reduced. By reducing the dimension so that only information effective for identification is extracted from the feature amount, the amount of calculation in the subsequent processing can be reduced. The dimension reduction may be performed using a known conversion matrix using a known Principal Component Analysis, Locality Preserving Projection, or the like. Here, the transformation matrix is an array of base vectors that define a vector space after dimension reduction. Using the transformation matrix, the feature vector in which the feature amounts are arranged in a line is projected from the original space to the space defined by the base vector. The conversion matrix is stored in the ROM 504 or the data storage unit 501 as part of data or a program, and is read into the RAM 505 in advance. The CPU 503 executes dimension reduction processing while referring to it.

  In step S615, identification processing is executed using the feature amount after dimension reduction obtained in step S614. Here, the feature vector after dimension reduction is called a “projection vector”. In the identification process, the projection vector and registered data are collated.

  The registration data is data including, for example, a registration vector and a corresponding person ID. Preferably, character string data such as a name and a nickname is stored in association with the person ID. The registered data is stored in the data storage unit 501 and is read into the RAM 505 prior to face authentication processing. A method of creating registration data will be described later.

  In the identification process, the person ID of the input face image is determined based on the similarity between the projection vector and the registered vector and a threshold value specified in advance. Here, the similarity is described as the Euclidean distance between vectors in the feature space after dimension reduction. In this case, since the projection vector and the registered vector can be interpreted to be similar vectors as the distance is small, it can be said that the registered vector (the image on which the distance is small) is similar to the input face image.

  First, the distance between the projection vector and all registered vectors is calculated, and the registered vectors are sorted in ascending order of distance. Next, the distance (minimum distance) between the projection vector and the first registered vector after sorting is compared with a preset threshold value.

  When the minimum distance is equal to or smaller than the threshold, the person ID of the input face image is the person ID of the registered vector at the head after sorting. On the other hand, if the minimum distance is greater than the threshold, it is determined that no person in the input image is registered. In this case, for example, an ID value corresponding to a non-registered person determined in advance by the system is set as the person ID of the input face image.

  In step S616, the person ID obtained in S615 is stored in association with information (face area position / size) for displaying the result of face authentication stored in the RAM 505.

  The above processing is executed for each of all face images detected from the original image. When face authentication processing for all face images is completed, an identification result is output. As an example of the identification result output, it is conceivable that a face authentication result image is created based on the original image stored in the RAM 505, the position / size information of the face area, and the person ID of the face area, and displayed on the display unit 507. .

  FIG. 13 is a diagram illustrating an example of a face authentication result image. In the example of FIG. 13, each face area is displayed as a rectangular frame, and the person ID of the face area or the associated character string is displayed above the face area. As another output method, a method in which the position / size information of the face area, the person ID of the face area, and the like are associated with the original image and stored in the data storage unit 501 is also conceivable. Further, instead of storing the data in the data storage unit 501, the same information may be transmitted to an external device via the communication unit 502.

  In the above description, the results are output when the processing is completed for all the face images. However, the output processing may be executed every time processing for one face image is completed. Here, in step S612, image conversion is performed so that both eyes in the face image are horizontally aligned and have a predetermined size. In order to increase the identification accuracy, it is preferable to perform such image conversion. However, when it is necessary to improve speed or reduce resources, the image conversion process may be omitted.

<4.3. (C) Registration data creation mode>
The registration data creation mode is an operation mode for creating registration data to be a reference pattern used in face authentication using a signal processing result. Here, the registration data is data including a registration vector and a person ID corresponding to the registration vector.

  In step S617, a face image used to create registration data is selected. First, the image data stored in the data storage unit 501 is stored in the RAM 505. Next, a face area is detected from the image data in the RAM 505 by a known face detection method, and an image showing the detected face area with a rectangular frame is displayed on the display unit 507. The user selects a face area to be registered from the face areas through the input unit 506. The selected face area image is resized to a predetermined size and stored in the RAM 505. If there is no face area to be registered, an instruction to display the next image is input.

  A feature amount after dimension reduction is generated from the selected face image through processing similar to steps S612 to S614 described in the identification mode. This is stored in the RAM 505 as a registered vector.

  In step S618, the registered vector and the person ID are associated and stored in the data storage unit 501. A procedure for associating the registered vector with the person ID will be described below.

  First, the person ID already stored in the data storage unit 501 or character string data associated with the person ID is displayed on the display unit 507. Preferably, the face image is stored in the data storage unit 501 together with the registration data, and the face image is displayed together with the person ID or character string data.

  Next, the user designates a person ID or character string data that is considered to correspond to the face image selected in S617 through the input unit 506. Then, the specified person ID is stored in the data storage unit 501 in association with the registered vector. On the other hand, if there is no corresponding person ID or character string data, the fact is input via the input unit 506. In this case, a new person ID is associated with the registered vector and stored in the data storage unit 501.

  As described above, according to the first embodiment, learning is performed by approximating a discontinuous function (non-differentiable function) to a continuous function. Thereby, it is possible to perform learning by CNN for a processing system including encoding processing such as LBP and incremental code. As a result, it is possible to more suitably extract a feature amount used for pattern identification from image data or the like. In the first embodiment, the processing in the three modes (learning mode, identification mode, and registration data creation mode) has been described as being performed by one device, but each may be performed by a plurality of separate devices.

(Second Embodiment)
In the second embodiment, an example in which main processing is configured using a hardware circuit will be described. That is, the second embodiment is mainly different in that main processing is executed by a signal processing unit 1401 that is a hardware circuit such as an ASIC or a DSP.

<Configuration of data processing apparatus>
FIG. 14 is a diagram illustrating a configuration of a data processing device according to the second embodiment. In addition, the same number is attached | subjected about the same component as FIG. 5, and description is abbreviate | omitted here. In the following, only the parts different from the first embodiment will be described. As described above, in the second embodiment, main processing is executed by the signal processing unit 1401, and the CPU 503 is exclusively used for control of the signal processing unit 1401.

  First, the CPU 503 reads an operation mode input by the user through the input unit 506. Here, the operation mode is information that determines the operation of the signal processing unit 1401. Here, it is assumed that there are two types of operation modes, “identification mode” and “learning mode”.

  As in the first embodiment described above, the “identification mode” is an operation mode in which the feature value used for pattern identification is extracted from the input image by performing the signal processing shown in FIG. 1 on the input image. It is. On the other hand, the “learning mode” is an operation mode for learning the CNN filter coefficient in the signal processing shown in FIG.

  The CPU 503 stores data required by the signal processing unit 1401 in the RAM 505 according to the input operation mode. When “identification mode” is selected as the operation mode, the CPU 503 detects a face area from the image data stored in the data storage unit 501 by a known face detection method as in the first embodiment. Further, a face image in which both eyes are arranged horizontally and converted into a predetermined size is stored in the RAM 505. On the other hand, when “learning mode” is selected as the operation mode, it is associated with two selected face images and face images from among all the learning face images stored in the data storage unit 501. The person ID is stored in the RAM 505.

  The CPU 503 stores necessary data in the RAM 505 and then transmits information on the selected operation mode to the signal processing unit 1401. When a signal indicating that the processing is completed is received from the signal processing unit 1401, information on the next operation mode is read, and the above-described processing is repeatedly executed.

<Configuration of signal processing unit>
FIG. 15 is a block diagram illustrating a configuration of a signal processing unit in the second embodiment. The signal processing unit 1401 switches the processing path according to the designation of the operation mode received from the user. Thereby, feature extraction processing and learning are realized by the same circuit. Hereinafter, each block in the signal processing unit 1401 will be described in detail with reference to FIG.

  A memory 1508 stores four types of data including a face image, a CNN network configuration, a CNN filter coefficient, and approximate encoding processing result data after tanh encoding processing, and includes a known RAM, a register, and the like. Here, it is assumed that the face image is obtained by converting both eyes horizontally and image-converted to a predetermined size, as in the first embodiment. In addition, it is assumed that the CNN network configuration and the CNN filter coefficients are stored, for example, in the formats described in FIGS. 10 and 11 of the first embodiment, respectively.

  Reference numeral 1501 denotes a control unit that controls the operation of each block in the signal processing unit 1401. First, the control unit 1501 reads the network configuration and filter coefficients of the CNN used by the CNN processing unit 1502 from the RAM 505 and stores them in the memory 1508. And it waits until the designation | designated of operation mode is received from CPU503, The process demonstrated below is started by receiving the designation | designated of operation mode as a trigger.

Identification Mode The control unit 1501 first transmits a signal instructing the switch 1503 to input the output of the CNN processing unit 1502 to the incremental encoding processing unit 1504. Next, a face image to be subjected to feature extraction is read from the RAM 505 and stored in the memory 1508. Subsequently, the control unit 1501 transmits a signal instructing the CNN processing unit 1502 to execute CNN processing on the face image stored in the memory 1508.

  When encoding process result data is received from the incremental encoding processing unit 1504, the data is stored in the RAM 505. Here, the encoding process result data may be transmitted to an external data processing apparatus via the communication unit 502. Further, the control unit 1501 transmits a signal indicating that the processing is completed to the CPU 503 and waits until receiving the designation of the next operation mode.

  When the signal processing circuit is applied to face authentication, the CPU 503 is configured to execute dimension reduction processing and identification processing using the encoding processing result data stored in the RAM 505 as a feature amount. Of course, the dimension reduction process and the identification process may be combined to form a hardware circuit such as the signal processing unit 1401.

Learning Mode The control unit 1501 first transmits a signal instructing the switch 1503 to input the output of the CNN processing unit 1502 to the tanh encoding processing unit 1505. Next, the two learning face images stored in the RAM 505 are read and stored in the memory 1508. Further, the person ID associated with these face images is also read.

  Subsequently, the control unit 1501 transmits a signal instructing the CNN processing unit 1502 to execute CNN processing on the first face image among the two face images stored in the memory 1508. When a signal indicating that the approximate encoding process has been completed is received from the tanh encoding processing unit 1505, a signal instructing to execute the CNN process on the second face image is transmitted to the CNN processing unit 1502.

  When receiving a signal indicating that the approximate encoding process for the second face image has been completed from the tanh encoding processing unit 1505, the control unit 1501 compares the two person IDs received in advance. If the person IDs are the same, a label “0” is generated, and if the person IDs are different, a label “1” is generated and transmitted to the loss calculation unit 1506.

  When a signal indicating that the filter coefficient update has been completed is received from the filter coefficient update unit 1507, the updated filter coefficient is read from the memory 1508 and stored in the RAM 505. Then, a signal indicating that the process is completed is transmitted to the CPU 503 and waits until the next operation mode designation is received.

  Returning to the description of each part shown in FIG. In response to an instruction from the control unit 1501, the CNN processing unit 1502 reads the CNN network configuration and filter coefficients stored in the memory 1508. Then, referring to the face image stored in the memory 1508, the same CNN processing as in step S605 is executed. When the CNN process for one face image is completed, the process waits until the next instruction is received.

  The switch 1503 is a switch that distributes the CNN processing result data generated by the CNN processing unit 1502 to either the incremental encoding processing unit 1504 or the tanh encoding processing unit 1505 in accordance with an instruction from the control unit 1501.

  The incremental encoding processing unit 1504 performs the same incremental encoding processing as that in step S613 in the first embodiment on the CNN processing result data, and transmits the encoding processing result data to the control unit 1501.

  Here, as described above, the input image is a face image converted to a predetermined size. The network configuration of the CNN and the size of the spatial filter are determined in advance. Therefore, the size of the CNN processing result data generated by the CNN processing unit 1502 can be calculated in advance. Here, it is assumed that the incremental encoding processing unit 1504 is configured to hold a register or the like therein, and the control unit 1501 sets information related to the CNN output result data size in the register in advance. The incremental encoding processing unit 1504 transmits a signal indicating that the encoding process has been completed to the control unit 1501 when the encoding process for the image of the size of the CNN processing result data stored in the register or the like is completed.

  The tanh encoding processing unit 1505 performs the approximate encoding process similar to step S606 in the first embodiment on the CNN process result data, and stores the approximate encoding process result data in the memory 1508. Here, it is assumed that the tanh encoding processing unit 1505 holds a register and the like, and the control unit 1501 presets information related to the CNN output result data size therein. The tanh encoding processing unit 1505 transmits a signal indicating that the approximate encoding process has been completed to the control unit 1501 when the approximate encoding process for the image of the size of the CNN processing result data stored in the register or the like is completed.

  When the loss calculation unit 1506 receives the label information from the control unit 1501, the loss calculation unit 1506 calculates the loss similar to that in step S609 in the first embodiment, using the approximate encoding processing result data and the label information stored in the memory 1508. To do. Then, the result is transmitted to the filter coefficient update unit 1507.

  When receiving the loss from the loss calculation unit 1506, the filter coefficient update unit 1507 updates the filter coefficient of the CNN stored in the memory 1508, similarly to step S610 in the first embodiment. When the filter coefficient update unit 1507 completes the update of the filter coefficient, the filter coefficient update unit 1507 transmits a signal indicating the completion of the update to the control unit 1501. Note that although a configuration in which the memory 1508 is included in the signal processing circuit is shown here, an external RAM or the like connected to the signal processing circuit may be used.

  As described above, in the second embodiment, the example in which the main part of the signal processing shown in FIG. 1 is realized as a hardware circuit has been described. Generally, high-speed processing is achieved by configuring the main part of signal processing with hardware circuits. Note that it is not necessary for all of the signal processing unit 1401 to be hardware circuits, and some of the processing may be realized by software processing.

(Third embodiment)
In the third embodiment, a signal processing system in which learning processing and pattern identification processing are executed by separate devices will be described. FIG. 16 is a diagram showing a configuration of a data processing system according to the third embodiment. In the data processing system shown in FIG. 16, the server device 1601 executes learning processing, and the client device 1602 executes pattern identification processing. Note that the server device 1601 is preferably configured by a distributed processing system such as publicly known cloud computing. In FIG. 16, only one client device is shown, but a configuration may be adopted in which a plurality of client devices are connected to one server device.

<Configuration of each device in the system>
Server Device The server device 1601 may logically have the same configuration as that of the data processing device shown in FIG. 5, and executes the processing shown in FIG. However, in the third embodiment, since the server device 1601 executes only the learning process, the process in the identification mode or the registration data creation mode may be omitted. Note that the server device 1601 includes a CNN processing unit (second generation unit) for performing learning processing.

  When the server apparatus 1601 receives a learning process request signal from the client apparatus 1602 via its own communication unit 502, the server apparatus 1601 is configured to execute the learning process shown in steps S602 to S611 in the first embodiment.

  Here, the CNN filter coefficient may be included in the learning process request signal. In this case, the received filter coefficient is stored in the RAM 505, and the learning process is executed with the filter coefficient as an initial value. It is also possible to configure the person ID to be included in the request signal for the learning process. In this case, the server device 1601 and the client device 1602 share the person ID, and the server device 1601 is configured to execute a learning process that increases the identification accuracy for the face image of the received person ID. Furthermore, it is also possible to configure to include a face image in the request signal for the learning process. In this case, the server apparatus 1601 is configured to execute a learning process that increases the identification accuracy for the received face image.

  When the learning process is completed, the server device 1601 transmits the learned (updated) filter coefficient to the client device 1602 via the communication unit 502.

  Here, the server apparatus 1601 has been described as performing a learning process triggered by reception of a request signal from the client apparatus 1602, but the server apparatus 1601 may be configured to autonomously perform a learning process. In this case, for example, the learning process is configured to be triggered by a lapse of a certain time from the last learning process. Then, when the learning process ends, the server device 1601 may start communication with the client device 1602 and transmit the learning result filter coefficient. Alternatively, when the client device 1602 requests a filter coefficient, the server device 1601 may be configured to transmit the latest filter coefficient at that time.

Client Device FIG. 17 is a diagram illustrating a configuration of a client device according to the third embodiment. In addition, the same number is attached | subjected about the same component as FIG. 5, and description is abbreviate | omitted here. Reference numeral 1701 denotes a feature extraction unit, which executes processing similar to the processing in the identification mode described in the first and second embodiments. Note that the client device 1602 includes a CNN processing unit (first generation unit) for performing identification processing.

  FIG. 18 is a block diagram illustrating a configuration of a feature extraction unit according to the third embodiment. In contrast to the configuration of the signal processing unit 1401 illustrated in FIG. 15, the feature extraction unit 1701 has a configuration in which components necessary for processing in the learning mode are excluded. The client device 1602 performs feature extraction processing on the face image stored in the RAM 505 using the feature extraction unit 1701, and executes pattern identification processing by the CPU 503 using the generated encoding processing result data.

<System processing flow>
When updating the filter coefficient of the CNN, the client device 1602 transmits a learning process request signal to the server device 1601 via the communication unit 502. Alternatively, the CNN filter coefficient stored in the RAM 505 in the client device 1602 is transmitted. Alternatively, the face image or person ID included in the registration data is transmitted. In this case, transmission of a filter coefficient, a face image, or a person ID may be used as a learning process request signal.

  The client device 1602 may be configured to wait until a filter coefficient is received from the server device 1601 after transmitting a request signal for learning processing to the server device 1601. Alternatively, the pattern identification process can be continued using the filter coefficient of the CNN before update without waiting for reception of the filter coefficient. In this case, for example, the server device 1601 may be configured to start communication with the client device 1602 at the timing when the learning process is completed. Alternatively, the client device 1602 inquires of the server device 1601 whether or not the learning process has been completed by using an instruction from the user on the client device 1602 side or a timer interrupt as a trigger, and receives the filter coefficient if the learning process has been completed. You may comprise as follows.

  When the client device 1602 receives the updated filter coefficient from the server device 1601 via the communication unit 502, the client device 1602 stores the updated filter coefficient in the RAM 505.

  As described above, in the second embodiment, the learning process in the client device is omitted, and the learning process is executed in the server device, whereby the mounting load (equipment cost, etc.) in the client device can be reduced. Further, when a plurality of client devices 1602 are connected to the server device 1601, there is an advantage that the filter coefficient learned by the server device 1601 can be easily shared among a plurality of clients. That is, learning can be performed more efficiently and an appropriate filter can be determined more quickly. Furthermore, the same filter can be shared among a plurality of client apparatuses that execute pattern identification processing, and the same pattern identification result can be obtained between the plurality of client apparatuses.

(Modification)
In the above-described embodiment, the description is given assuming an example in which the processing target data is applied to two-dimensional image data (face image). However, the present invention can be applied to one-dimensional data such as an audio signal, and can be similarly applied to three-dimensional or more data.

  In the above-described embodiment, the example in which the step function is used as the discontinuous function processing for the comparison result between the target pixel and the reference pixel in the CNN processing result data has been described. However, the step function is not limited to the step function. It can also be used. In the step function, binarization (“0” or “1”) is performed according to the relative magnitude of the two values to be compared. On the other hand, the pulse function binarizes (“0” or “1”) according to the absolute value of the difference between the two values to be compared. FIG. 19 is a diagram illustrating a pulse function that is a discontinuous function and a Gaussian function that approximates the pulse function.

  Furthermore, in the above-described embodiment, the example in which the pixel values of one target pixel and one reference pixel in the CNN processing result data are compared has been described, but the present invention is not limited to the pixel value of one pixel. For example, instead of the pixel value of one pixel, an average value of pixel values in an area of (m × m) pixels may be used.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (11)

Generating means for generating processing result data by executing spatial filtering on the input data;
First encoding means for executing a predetermined encoding process using a discontinuous function on the processing result data to generate encoding process result data;
A second encoding means for executing an approximate encoding process in which the discontinuous function is approximated by a continuous function with respect to the processing result data, and generating approximate encoding process result data;
Updating means for updating a weighting factor of the spatial filter processing based on the approximate encoding processing result data;
Providing the processing result data to the second encoding means when updating the weighting coefficient by the updating means, and providing the processing result data to the first encoding means in other cases; Control means for controlling
A signal processing apparatus comprising:   The signal processing apparatus according to claim 1, wherein the other cases include at least one of a case of performing pattern identification and a case of registering a reference pattern used for the pattern identification.   The signal processing apparatus according to claim 1, further comprising pattern identification means for performing pattern identification based on the encoding processing result data.   The predetermined encoding process compares the value in the region of interest in the processing result data with the value in one or more reference regions at a predetermined relative position with respect to the region of interest, and compares the comparison processing result data. 4. The signal processing apparatus according to claim 1, wherein the signal processing apparatus is a process that generates and performs an operation using the discontinuous function on the comparison processing result data. 5.   5. The signal processing apparatus according to claim 1, wherein the spatial filter process is a CNN (Convolutional Neural Networks) process, and the weighting coefficient is a filter coefficient in the CNN process. 6.   The signal processing apparatus according to claim 5, wherein the updating unit updates a weighting coefficient of the spatial filter processing using an error back propagation method.   The signal processing apparatus according to claim 1, wherein the discontinuous function is a step function, and the continuous function is a tanh function or a sigmoid function.   The signal processing apparatus according to claim 1, wherein the discontinuous function is a pulse function, and the continuous function is a Gaussian function. A generation step of generating processing result data by executing spatial filtering on the input data;
A first encoding step of executing a predetermined encoding process using a discontinuous function on the processing result data to generate encoding process result data;
A second encoding step of performing an approximate encoding process in which the discontinuous function is approximated by a continuous function on the processing result data to generate approximate encoding process result data;
An update step of updating a weighting coefficient of the spatial filter processing based on the approximate encoding processing result data;
A control step for performing control so that the second encoding step is executed when the weighting factor is updated, and the first encoding step is executed in other cases;
A signal processing method characterized by comprising: The program for functioning a computer as each means of the signal processing apparatus of any one of Claims 1 thru | or 8 . A signal processing system including a client device and a server device,
The client device is
First generation means for generating processing result data by executing spatial filtering on the input data;
First encoding means for executing predetermined encoding processing using a discontinuous function on the processing result data generated by the first generation means to generate encoding processing result data;
Pattern identifying means for performing pattern identification based on the encoding processing result data;
Have
The server device
Second generation means for generating processing result data by executing the spatial filter processing on input data;
Second encoding means for executing approximate encoding processing for approximating the discontinuous function with a continuous function on the processing result data generated by the second generating means to generate approximate encoding processing result data; ,
Updating means for updating a weighting factor of the spatial filter processing of the second generation means based on the approximate encoding processing result data;
In addition,
The server device includes a transmission unit that transmits the weighting factor updated by the updating unit to the client device, and the client device uses the weighting factor received from the server device as a spatial filter of the first generation unit. A signal processing system comprising setting means for setting as a processing weight coefficient.

Images