JP2023080908A - Information processing apparatus and control method of information processing apparatus - Google Patents

Abstract

To provide an information processing apparatus which can reduce a calculation amount while having less influence on inference accuracy, in processing of identifying an object in a moving image.SOLUTION: The information processing apparatus includes: acquisition means which acquires continuous frames of a moving image; and identification means which identifies an object from each of the frames of the moving image by using a calculator including a plurality of hierarchies, in which a degree of influence being a product sum of outputs from individual nodes included in a first hierarchy to a prescribed node in a second hierarchy and weights respectively corresponding to the outputs is calculated and is inputted to the prescribed node. On the basis of a first degree of influence inputted to the prescribed node for a first frame and a partial calculation result of calculation of a second degree of influence inputted to the prescribed node for a second frame after the first frame, the identification means determines whether or not calculation of the second degree of influence is interrupted.SELECTED DRAWING: Figure 10

Description

The present invention relates to an information processing device and a control method for the information processing device.

In recent years, learning by deep learning, inference research, and application to the real world have progressed, and large-scale neural networks have come to be used. A multi-class classifier capable of identifying about 1000 types of objects has also been implemented.

For example, a multi-class classifier that can distinguish about 1000 kinds of objects calculates the probability that each object exists in an image. Limited. As the types of objects that can be identified increase, the ratio of the amount of calculation for objects that do not exist in the screen to the overall amount of calculation of the multi-class classifier increases, so the efficiency of identifying objects in the image deteriorates.

As a method for reducing the amount of calculation, for example, Patent Document 1 discloses that useful information and useless information are distinguished in the latter layer of a neural network, and the feature amount or weight of the portion judged to be useless is set to zero. Discloses how to replace.

JP 2019-200648 A

S. Haykin, "Neural Networks A Comprehensive Foundation 2nd Edition", Prentice Hall, pp. 156-255, July 1998

The technique described in Patent Literature 1 does not take correlation in the temporal direction into consideration, and therefore cannot be expected to have a sufficient effect of reducing the amount of calculation for temporally continuous images such as moving images. Further, even if the amount of computation is reduced on the post-stage side of the neural network, when using a multi-class classifier, the effect of suppressing the increase in the amount of computation associated with an increase in the types of objects cannot be expected.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an information processing apparatus capable of reducing the amount of calculation while suppressing the influence on inference accuracy in object identification processing in moving images.

The information processing device of the present invention includes:
acquisition means for acquiring successive frames of a moving image;
A calculator including a plurality of hierarchies, wherein the degree of impact is the product sum of the output from each node included in the first hierarchy to a predetermined node in the second hierarchy and the weight corresponding to each of the outputs and an identification means for identifying an object from each frame of the moving image using a calculator that inputs the calculated degree of influence to the predetermined node,
The identifying means includes a first influence input to the predetermined node in a first frame and a second influence input to the predetermined node in a second frame after the first frame. It is characterized in that it is determined whether or not to interrupt the calculation of the second influence based on a partial calculation result of the calculation of the influence.

Advantageous Effects of Invention According to the present invention, it is possible to reduce the amount of calculation while suppressing the influence on inference accuracy in object identification processing in a moving image.

1 is a block diagram illustrating the configuration of an information processing device; FIG. FIG. 4 is a diagram showing an example structure of a neural network for identifying subject classes; It is a figure explaining the feature detection process and feature integration process of CNN. 1 is a schematic diagram of a model structure of a neural network; FIG. 6 is a flowchart illustrating inference processing using an inference model; 10 is a flowchart illustrating an inference calculation for a first frame image; FIG. 4 is a diagram for explaining computing units used in nodes of an inference model; FIG. 4 is a diagram illustrating the configuration of a sum-of-products calculator; 7 is a graph showing an example of a subject class identification result; 10 is a flowchart illustrating inference calculations for frames other than the first frame; 10 is a graph showing an example of changes in the degree of influence in modified example 1;

<Embodiment>
Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In the present embodiment, the information processing apparatus reduces the amount of computation that is wasted when using a neural network model for multi-class classification.

FIG. 1 is a block diagram illustrating the configuration of an information processing apparatus 100. As illustrated in FIG. The information processing apparatus 100 includes a CPU 101 , a ROM 102 , a memory 103 , an image acquisition section 104 and an identification section 105 . Each component included in the information processing apparatus 100 is connected to a system bus 110 and can exchange data with each other via the system bus 110 .

CPU 101 controls image acquisition unit 104 and identification unit 105 by reading programs stored in ROM 102 into memory 103 and executing them.

The ROM 102 stores various programs for the CPU 101 to operate. Various programs for the operation of the CPU 101 are not limited to the ROM 102, and may be stored, for example, in a hard disk or the like.

The memory 103 is, for example, a RAM, and is used as a work memory for the CPU 101 to execute various programs. The CPU 101 can read a program stored in the ROM 102 to the memory 103 and execute it.

The image acquisition unit 104 acquires continuous frame images of a moving image for which an object (subject) is to be identified. The image acquiring unit 104 can acquire a moving image from, for example, an imaging unit (not shown) included in the information processing apparatus 100 . Also, the image acquisition unit 104 may acquire a moving image from an imaging device connected to the information processing device 100 .

The identification unit 105 identifies (detects) an object from the frame image acquired by the image acquisition unit 104 using an inference model (learned model) of a neural network. Based on the calculation result for the previous frame and a part of the calculation result for the current frame, the identification unit 105
Determines whether or not to interrupt the operation for the current frame. By not performing computations after being interrupted, the amount of computations for identifying objects is reduced.

A basic configuration of an inference model of a neural network will be described with reference to FIGS. 2 and 3, taking a CNN (Convolutional Neural Network) as an example. FIG. 2 shows the basic configuration of a CNN that identifies object classes from an input image that is two-dimensional image data.

A CNN contains multiple layers, each containing two layers called the feature detection layer (S-layer) and the feature integration layer (C-layer). In the example of FIG. 2, an input image input to the CNN is processed in order from the 1st layer to the Xth layer.

In the CNN, first, in the S layer, the features of the input image are detected based on the features detected in the preceding layer. Next, the features detected in the S layer are integrated in the C layer and input to the next layer as detection results in the current layer.

The S-layer includes multiple feature-detecting cell planes and detects different features for each feature-detecting cell plane. The C layer contains multiple feature integration cell planes and pools the detection results in the feature detection cell planes of the S layer. In the example of FIG. 2, the output layer (X-th layer), which is the final layer, is composed of the S layer without using the C layer. The feature detection cell plane and the feature integration cell plane are also collectively referred to as cell planes.

Details of the feature detection processing in the feature detection cell plane and the feature integration processing in the feature integration cell plane will be described with reference to FIG. In FIG. 3, rectangles indicate cell planes. The L-th order S-layer contains a plurality of feature detection cell planes, and the L-1-th order C-layer and L-th order C-layer each contain a plurality of feature integration cell planes.

The feature detection cell surface is composed of a plurality of feature detection neurons, and the feature detection neurons are connected to the layer C, which is the previous layer, in a predetermined structure. The feature-integrating cell surface is composed of a plurality of feature-integrating neurons, and the feature-integrating neurons are connected to the S layer of the same hierarchy with a predetermined structure.

In the m-th cell plane of the L-th layer S layer, the output value of the feature detection neuron at the position (ξ, ζ) is y _m ^LS (ξ, ζ), and in the m-th cell plane of the L-th layer C layer, the position The output value of the feature integration neuron of (ξ, ζ) is expressed as y _m ^LC (ξ, ζ). Assuming that the coupling coefficients of the neurons are w _m ^LS (n, u, v) and w _m ^LC (u, v), each output value can be expressed as follows.

f in Equation 1 is an activation function, which may be a sigmoid function such as a logistic function or a hyperbolic tangent function, and is realized by a tanh function, for example. u _m ^LS (ξ, ζ) is the internal state of the feature detection neuron at the position (ξ, ζ) in the m-th cell plane of the L-th layer S layer. The output value of the feature detection neuron shown in Equation 1 is the internal state of the feature detection neuron u _m ^LS
It is calculated by transforming (ξ, ζ) with the activation function f.

The output value of the feature integration neuron shown in Equation 2 is obtained by a simple linear sum of the coupling coefficient w _m ^LC (u, v) and the m-th output value of the L-th layer S layer without using an activation function. Calculated. When no activation function is used, the internal state u _m ^LC (ξ, ζ) of the feature integration neuron is equal to the output value y _m ^LC (ξ, ζ). Also, y _n ^L-1C (ξ+u, ζ+v) in Equation 1 and y _m ^LS (ξ+u, ζ+v) in Equation 2 are referred to as output values to which the feature integration neuron and the feature detection neuron are coupled, respectively.

ξ, ζ, u, v, and n in Equations 1 and 2 will be explained. Position (ξ, ζ) corresponds to position coordinates in the input image. If y _m ^LS (ξ, ζ) is higher than the output value of the feature detection neuron at other positions, the feature detected in the L-th layer, S-layer, m-th cell plane is the pixel position (ξ, ζ) of the input image. It means that there is a high possibility that there is a

n in Formula 1 means the L-1 layer C layer n-th cell surface, and is referred to as an integration target feature number. The internal state u _m ^LS (ξ, ζ) of the feature detection neuron is obtained by combining the coupling coefficient w _m ^LS (n, u, v) and the coupling It is calculated by a sum-of-products operation with the previous output value y _n ^L-1C (ξ+u, ζ+v).

(u, v) are the relative position coordinates of the coupling coefficient, and the sum-of-products operation is performed in a finite range (u, v) according to the size of the feature to be detected. The finite range (u,v) is called the receptive field. The size of the receptive field is represented by the number of horizontal pixels×the number of vertical pixels in the combined range, and is hereinafter referred to as the receptive field size.

In Equation 1, y _n ^L-1C (ξ+u, ζ+v) is the input image y ^in_image (ξ+u, ζ+v) or the input position map y ^in_posi_map (ξ+u,
ζ+v). The distribution of neurons and pixels is discrete, and the destination feature number is also discrete. Therefore, ξ, ζ, u, v, and n are discrete values rather than continuous variables. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which have a finite range.

w _m ^LS (n, u, v) in Equation 1 is the coupling coefficient distribution for detecting a given feature.
By adjusting the coupling coefficients to appropriate values, it is possible to detect a given feature. In constructing (learning) the CNN, various test patterns are presented, and the coupling coefficients are iteratively adjusted so that y _m ^LS (ξ, ζ) are appropriate output values.

w _m ^LC (u, v) in Equation 2 is expressed as Equation 3 using a two-dimensional Gaussian function.

(u, v) is a finite range, and the finite range (u, v) is called the receptive field, and the size of the receptive field range is called the receptive field size, similar to the description of the feature detection neuron. The receptive field size may be set to an appropriate value according to the size of the feature detected on the m-th cell plane of the L-th layer S layer. σ in Equation 3 is a feature size factor, which is set to an appropriate constant according to the receptive field size. Specifically, the feature size factor σ is preferably set so that the value of the coupling coefficient can be considered to be approximately 0 at the outermost side of the receptive field.

By performing the above-described calculations in each layer of the neural network, it becomes possible to identify the subject class in the final layer, S layer.

A specific learning method of the neural network will be explained. In this embodiment, the coupling coefficients are adjusted by supervised learning. In supervised learning, a ^test pattern is given and the output value of a neuron is actually obtained _. desired output value). The coupling coefficients can be modified, for example, by using the least squares method in the final feature detection layer and by using the error backpropagation method in the intermediate feature detection layers. For correction of the coupling coefficients by the least-squares method, the error backpropagation method, or the like, the known method disclosed in Non-Patent Document 1, for example, can be used.

When the neural network is trained in advance, a large number of specific patterns to be detected and patterns not to be detected are prepared as test patterns for learning. When the activation function is the tanh function and a specific pattern to be detected is presented, a teacher signal is given so that the output value becomes 1 to the neurons in the region where the specific pattern exists in the feature detection cell surface of the final layer. be done. Conversely, when a pattern that should not be detected is presented, a teacher signal is given to the neuron in the area where the pattern that should not be detected exists so that the output value becomes -1.

By the above method, a neural network for identifying object classes from input images is constructed. Calculations in the actual detection (identification) process are the coupling coefficients w _m ^LS (n
, u, v). If the output value of a neuron on the feature detection cell surface of the final layer is equal to or greater than a predetermined value, it is determined that the subject of the subject class to be identified exists in the position (region) corresponding to the neuron.

The model structure of the neural network will be described with reference to FIG. FIG. 4 is a schematic diagram of the model structure of a neural network. In the description of FIG. 4, a neuron is called a node, a connection between neurons is called an edge, and a connection coefficient representing the strength of the connection between neurons is called an edge weight. Nodes are represented by circles, and edges connecting nodes are represented by straight lines.

An inference model 400 (operator) shown in FIG. 4 has a model structure including an input layer 401 , a hidden layer 403 and an output layer 405 . The input layer 401 has N nodes, the hidden layer 403 has M nodes, and the output layer 405 has 10 nodes. be. Hidden layer 403 nodes perform operations with arbitrary activation functions. A group of edges connecting the input layer 401 and the hidden layer 403 is called a first layer, and a group of edges connecting the hidden layer 403 and the output layer 405 is called a second layer.

An inference process for identifying a subject class will be described with reference to FIGS. 5 to 9. FIG. FIG. 5 is a flowchart illustrating inference processing using the inference model 400 . In step S501, the CPU 101 determines whether the image (frame image) acquired by the image acquisition unit 104 is the first frame.

When the CPU 101 determines that the acquired image is the first frame, the process proceeds to step S503. When the CPU 101 determines that the acquired image is not the first frame, the process proceeds to step S505.

In the following description, the CPU 101 switches the processing depending on whether it is the first frame (first frame), but the present invention is not limited to this. The CPU 101 may determine whether or not the subject has changed based on, for example, the difference in pixel values between the previous frame (first frame) and the current frame (second frame). . Then, the CPU 101 may advance the process to step S503 if it determines that the subject has changed, and advance the process to step S505 if it determines that the subject has not changed. Since the reference value for determining whether or not to interrupt the calculation is updated according to the change of the subject, the CPU 101 can further reduce unnecessary calculation.

In step S<b>503 , the CPU 101 executes inference operation 1 . The processing flow of inference operation 1 will be described with reference to FIG. FIG. 6 is a flowchart illustrating inference operation 1 for the first frame image.

In step S<b>601 , the CPU 101 executes the inference operation 11 . The inference calculation 11 is a process of first calculating an input (influence) to each node of the hidden layer 403, then passing the calculated influence through an activation function, and obtaining an output value from the node. be.

First, the CPU 101 computes inputs to each node included in the hidden layer 403 of the inference model 400 of FIG. Input a _j ⁽¹⁾ (j=1, 2, 3, . . . , M) to each node of hidden layer 403 will be described with reference to FIGS. j indicates the number of each node included in the hidden layer 403 . In the example of FIG. 4, hidden layer 403 has M nodes. "(1)" of the input a _j ⁽¹⁾ to each node of the hidden layer 403 indicates that a _j ⁽¹⁾ is the operation result in the first layer. The input to node 421 is denoted a ₁ ⁽¹⁾ .

FIG. 7 is a diagram for explaining computing units used in the nodes of the inference model 400. As shown in FIG. FIG. 7 shows an example of computation for nodes included in the hidden layer 403 . The CPU 101 first uses the sum-of-products calculator 701 to calculate inputs a _j ⁽¹⁾ (j=1, 2, 3, . . . , M) for each node. When calculating the input a _j ⁽¹⁾ to the j-th node of the hidden layer 403, the input to the sum-of-products calculator 701 is the output value x _i ⁽¹⁾ (i = 1, 2, 3, ..., N) and edge weights w _j,i ⁽¹⁾ (i = 1, 2, 3, ..., N). The product-sum calculator 701 calculates the product-sum of x _i ^{(1) and} w _j,i ⁽¹⁾ as a _j (1). For example, at node 421 (j=1), product-sum calculator 701 calculates the product-sum of x _i ⁽¹⁾ and w _1,i ⁽¹⁾ as a ₁ ⁽¹⁾ .

FIG. 8 is a diagram illustrating the configuration of the sum-of-products calculator 701 shown in FIG. CPU 101 repeats the integration of x _i ⁽¹⁾ and w _j,i ⁽¹⁾ and the addition of the integration results for each node of hidden layer 403, and inputs a _j ⁽¹⁾ (j=1 , 2, 3, . . . , M). The input a _j ⁽¹⁾ to the node obtained by the calculation in the sum-of-products calculator 701 is stored in the memory 103 as the degree of influence.

Next, CPU 101 obtains output value x _j ⁽²⁾ by passing input a _j ⁽¹⁾ to each node of hidden layer 403 through activation function calculator 702 . For example, the output value from node 421 is x ₁ ⁽²⁾ obtained by passing a ₁ ⁽¹⁾ through activation function calculator 702 . The activation function calculator 702 uses a sigmoid function, a tanh function, or the like as an activation function. CPU 101 passes input a _j ⁽¹⁾ to each node included in hidden layer 403 through activation function calculator 702 to generate x _j ⁽²⁾ (j=1, 2, 3, . . . , M) as get.

Although FIG. 4 shows an example with one hidden layer 403, there may be a plurality of hidden layers. If there are multiple hidden layers, the CPU 101 executes the inference operation 11 for each hidden layer. This provides an input to each node included in multiple hidden layers.

In step S603 of FIG. 6, the CPU 101 executes the inference operation 12. FIG. The inference calculation 12 is a process of first calculating the input (influence) to each node of the output layer 405, then passing the calculated influence through the activation function, and acquiring the output value from the node. be.

First, the CPU 101 inputs a _k ⁽²⁾ (k=1, 2, 3, . . . , 10) to each node included in the output layer 405 of the inference model 400 of FIG. sum-of-products calculator 7
01 is used for calculation.

When calculating the input a _k ⁽²⁾ to the k-th node of the output layer 405, the input to the sum-of-products calculator 701 is the output value x _j ⁽²⁾ (j = 1, 2, 3, ..., M) and edge weights w _k,j ⁽²⁾ (j = 1, 2, 3, ..., M). The product-sum calculator 701 calculates the product-sum of x _j ⁽²⁾ and w _k,j ⁽²⁾ as a _k ⁽ 2). For example, at the node 423, the product-sum calculator 701 calculates the product-sum of x _j ⁽²⁾ and w _1,j ⁽²⁾ as a ₁ ⁽²⁾ . The input a _k ⁽²⁾ to the node acquired by the calculation in the sum-of-products calculator 701 is stored in the memory 103 as the degree of influence.

CPU 101 passes input a _k ⁽²⁾ to each node to activation function calculator 702 . In the output layer 405, the activation function calculator 702 is, for example, the Softmax function. The Softmax function is represented by Equation 4 below. e in Equation 4 is Napier's number.

FIG. 9 is a graph showing an example of object class identification results. The output y _k from each node in the output layer 405 is shown as a bar graph. The output y _k from the k-th node (hereinafter referred to as node k) of the output layer 405 is the subject existence probability of the subject class corresponding to node k. In FIG. 9, the output _y6 from node 6 of the output layer 405 is higher than the outputs from other nodes, and the probability that an object of the object class corresponding to node 6 exists is Higher than the probability that the subject exists.

In FIG. 9, output _y6 from node 6 is high, followed by output _y2 from node 2, output _y7 from node 7, and outputs from other nodes are lower than _y2 and _y7 . Become. Therefore, the object existence probability of the corresponding object class is highest at node 6, and decreases in the order of node 2, node 7, and other nodes.

In step S505 of FIG. 5, the CPU 101 executes the inference operation 2, which is the process when it is determined that the image acquired by the image acquiring unit 104 is not the first frame. The processing flow of the inference operation 2 will be described with reference to FIG. FIG. 10 is a flowchart illustrating an inference operation 2 other than the first frame.

In step S<b>1001 , the CPU 101 executes an inference operation 11 . Since the inference operation 11 in step S1001 is the same as the inference operation 11 in step S601 of FIG. 6, description thereof is omitted.

In step S<b>1003 , the CPU 101 executes the inference operation 21 . The inference operation 21 is a process of partially calculating the input a _k ⁽²⁾ (k=1, 2, 3, . . . , 10) to the output layer 405 (second hierarchy). a _k ⁽²⁾ is the output value x _j ⁽²⁾ of M nodes in the hidden layer 403 (first layer) and the corresponding w _k,j ⁽²⁾ , as shown in Equation 5 below. It is obtained by sum-of-products operation. In the inference operation 21, the operation based on the expression 5 is divided into two as shown in the expression 6, and the CPU 101 executes the operation of the first term of the expression 6.

ただし、ｍ＝Ｍ／２（Ｍが偶数の場合）、（Ｍ－１）／２（Ｍが奇数の場合）とする。推論演算２１は、式６の第１項を演算する処理である。式６の第１項は、隠れ層４０３の一部のノードについての積和である。例えば、ノード４２３への入力ａ₁ ⁽²⁾についての推論演算２１は、隠れ層４０３のＭ個のノードのうち、ｍ個の出力値ｘ_j ⁽²⁾と対応するｗ_1,j ⁽²⁾との積和の演算処理である。

However, m=M/2 (when M is an even number) and (M-1)/2 (when M is an odd number). The inference calculation 21 is a process of calculating the first term of Equation 6. The first term in Equation 6 is the sum of products for some nodes in hidden layer 403 . For example, the inference operation 21 for the input a ₁ ⁽²⁾ to the node 423 generates w _{1,j (} ^{2 )} It is a calculation process of sum of products with .

In steps S1005 and S1007, the CPU 101 determines whether or not to execute the calculation of the second term of Expression 6, and suspends the calculation of the degree of influence without executing the calculation that does not affect the identification result of the subject. By doing so, the amount of calculation is suppressed.

In step S1005, the CPU 101 determines whether the difference between the partial calculation result (hereinafter also referred to as the partial influence) of the calculation of the influence (second influence) in the current frame and the reference value is within a predetermined range. It is determined whether or not. The reference value is a value corresponding to the sum of products (the first term in Equation 6) for some nodes of the hidden layer 403 in the first frame (first frame). A value obtained by multiplying the stored a _k ⁽²⁾ (first degree of influence) by 0.5 can be used.

The degree of influence is a _k ⁽²⁾ calculated by Equation 6, and the partial degree of influence is the calculation result of the first term of Equation 6. In this case, the CPU 101 determines whether the difference between 0.5 times a _k ⁽²⁾ (reference value) and the calculation result (partial influence degree) of the first term of Equation 6 is within a predetermined range. .

When temporally continuous frames such as a moving image are input to the image acquisition unit 104, the correlation between the continuous frames is generally high. For this reason, there is a possibility that the subject detected in the temporally consecutive previous frame will be detected in the current frame, and the subject that was not detected in the consecutive previous frame will not be detected in the current frame.

If the difference between the reference value and the calculation result of the first term of Equation 6 is within a predetermined range, the CPU 101 can determine that the same subject features are captured in the first frame and the current frame. The predetermined range is set, for example, based on the degree of influence in the first frame (a _k ⁽²⁾ stored in memory 103). Specifically, the predetermined range may be 3% of the degree of influence or 5% of the reference value. By increasing the predetermined range, the number of cases where the calculation of the second term of Equation 6 is not performed increases, so the amount of calculation is further suppressed.

The value of m that divides the first term and the second term of Equation 6 is about half of M, but is not limited to this, and may be about 1/α (0<1/α<1) of M. . The reference value is 0.5 times a _j ⁽²⁾ in the first frame when m is about half of M, but when m is about 1/α of M, a _j ⁽²⁾ may be approximately 1/α.

If the CPU 101 determines in step S1005 that the difference between the reference value and the partial influence degree is within the predetermined range, the process advances to step S1007. If the CPU 101 determines that the difference between the reference value and the partial influence degree is not within the predetermined range, the process advances to step S1009. If the difference between the reference value and the partial influence degree is not within the predetermined range, the CPU 101 determines that there is no correlation between the first frame and the current frame, and a _k ⁽²⁾ is calculated.

In step S1007, the CPU 101 determines whether or not the partial influence is equal to or less than the threshold. The partial influence degree is, for example, the calculation result of the first term of Expression 6. Since the degree of influence increases when the features of the subject are captured well, the CPU 101 also executes the calculation of the second term of Equation 6 when the partial degree of influence is greater than the threshold.

On the other hand, since the degree of influence is small when the features of the subject are not detected, the CPU 101 does not perform the calculation of the second term of Equation 6 when the partial degree of influence is equal to or less than the threshold. When the calculation is interrupted in this way, a _k ⁽²⁾ may be set to the value of a _k ⁽²⁾ in the first frame, which is the previous frame. A predetermined value such as the value of the first term or zero may be set. When the calculation is interrupted, the CPU 101 inputs the value set in a _k ⁽²⁾ to the node k of the output layer 405 as the degree of influence.

When the calculation is interrupted and the value of the first term of Equation 6 or the predetermined value is set to a _k ⁽²⁾ , a _k ⁽²⁾ stored in memory 103 is updated in inference calculation 1 of the first frame. Instead, it may be used as the degree of influence for determining the reference value in the next frame as well. In this case, a _k ⁽²⁾ stored in memory 103 is not updated until inference operation 1 (step S503) is next executed.

A threshold to be compared with the partial influence is set as follows, for example. Weight w k _{,j (} ^{2) (} j=1, 2, 3, . and the outputs x _j ⁽²⁾ etc. from the nodes are normalized to the range 0-1. The threshold can be set, for example, to a value equal to or greater than 10% of M, which is the maximum value of a _k ⁽²⁾ .

The percentage of M to be the threshold value may be set according to the ratio of the number of terms of the sum of products included in the first term of Equation 6 to the number of terms M of a _k ^(2). . Also, what percentage of M the threshold is set to may be set based on the accuracy and speed of identifying the object class. Also, the threshold value is not limited to the maximum value M when the weights and the like are normalized, and may be set based on the maximum value that a _k ⁽²⁾ can take.

If the CPU 101 determines in step S1007 that the degree of partial influence is equal to or less than the threshold, the process advances to step S1011. If the CPU 101 determines that the partial influence level is greater than the threshold, the process advances to step S1009. If the partial influence degree is greater than the threshold, the CPU 101 determines that the output y _k (existence probability of the corresponding subject) is high, and the second
Compute a _k ⁽²⁾ including terms.

In step S<b>1009 , the CPU 101 executes the inference operation 22 . The inference operation 22 performs the operation processing of the second term of Equation 6 to operate the input a _k ⁽²⁾ to the node of the output layer 405, passes the operation result through the activation function, and obtains the output from the node This is the process of acquiring the output value. The CPU 101 passes a _k ⁽²⁾ obtained by executing the arithmetic processing of the second term of Equation 6 to the activation function as in the inference operation 12 of step S603.

The CPU 101 may store the obtained a _k ⁽²⁾ in the memory 103 as the degree of influence for determining the reference value in the next frame. Further, the CPU 101 does not store a _k ⁽²⁾ in the inference calculation 22 in step S1009, and does not store the influence a _k (2) stored ⁱⁿ the memory 103 until inference calculation 1 (step S503) is next executed. ²⁾ may be used to determine the reference value.

In step S1011, the CPU 101 determines whether or not the calculation of the input a _k ⁽²⁾ to each node of the output layer 405 has been completed. For example, if the calculation of input a _k ⁽²⁾ to each node has been executed, the CPU 101 can determine that the calculation has ended. When the CPU 101 determines that the computation has ended, the processing of the inference computation 2 shown in FIG. 10 ends. If the CPU 101 determines that the calculation has not ended, the process returns to step S1003.

Note that in step S1005 of the inference calculation 2 in FIG. 10, the reference value is set based on a _k ⁽²⁾ (influence) stored in the memory 103 in the first frame. can't The reference value may be set based on a _k ⁽²⁾ stored in memory 103 in the frame immediately preceding the current frame.

Further, in the present embodiment, a _k ⁽²⁾ of the first frame is stored as the degree of influence, but the CPU 101 executes inference calculation 1 (step S503) for each predetermined number of frames in the same manner as the first frame. good too. When the inference calculation 1 is executed, the CPU 101 sets a _k ⁽²⁾ calculated in the inference calculation 12 (step S603) as a new influence level, and updates the influence level stored in the memory 103 .

Also, in equations 5 and 6, an example of interrupting part of the computation of the input a _k ⁽²⁾ to each node of the output layer 405 (computation in the second layer in FIG. 4) to reduce the amount of computation is given. As explained above, it is possible to reduce the amount of calculation in the first layer as well. In the calculation of the input a _j ⁽¹⁾ to each node of the hidden layer 403 (calculation in the first layer in FIG. 4), the information processing device converts a _j ⁽¹⁾ into the first term and the first term Divided into two items, the same processing as the processing from step S1003 to step S1009 in FIG. 10 is executed. The information processing apparatus 100 can reduce the amount of computation by not performing the computation of the second term of a _j ⁽¹⁾ at some nodes.

Further, when there are a plurality of hidden layers, similar to the calculation of the input to each node of the output layer 405, part of the calculation of the input to each node of the hidden layer next to the hidden layer 403 is interrupted, and the calculation The amount may be reduced.

According to the above embodiment, the information processing apparatus 100 calculates the degree of influence based on the degree of influence in the frame prior to the current frame and a partial calculation result of the degree of influence in the current frame. is interrupted. Thereby, the information processing apparatus 100 can reduce the amount of calculation while suppressing the influence on the inference accuracy. The information processing apparatus 100 can shorten the inference calculation time by early interrupting the calculation of the node whose influence degree is equal to or less than the threshold.

(Modification 1)
In inference calculation 12 in step S603 of FIG. 6, a _k ⁽²⁾ is stored as the degree of influence. In Modified Example 1, the degree of influence, like the degree of influence b _k ⁽²⁾ shown in Equation 7, is obtained by combining the output x _j ⁽²⁾ from each node of the hidden layer 403 and the weight w _k,j ⁽²⁾ of the edge. It may be configured to be monotonically increasing as an unsigned value.

In this case, the inference model 400 uses unsigned values for x _j ⁽²⁾ and w _k,j ⁽²⁾ even during the training phase. If the degree of influence is configured as shown in Equation 7, the value of the degree of influence b _k ⁽²⁾ monotonically increases as the calculation progresses as shown in FIG. 11, so the value does not decrease after exceeding the threshold. In the example of FIG. 11, the CPU 101 can determine that the threshold has been exceeded before executing m product-sum operations, which is approximately half of the M product-sum operations. Therefore, the CPU 101 can more quickly determine whether or not a partial calculation result (partial influence degree) of the influence degree calculation is equal to or greater than the threshold value in the processing of step S1007 in FIG.

(Modification 2)
The input a _k ⁽²⁾ to each node of the output layer 405, for example, the node 423, is the input a _j ⁽¹⁾ ( degree of influence). Among the nodes of the hidden layer 403, the CPU 101 preferentially performs a sum-of-products operation on a node having a greater degree of influence in the first frame than other nodes. The time until the threshold value is exceeded is shortened, and the CPU 101 can quickly determine whether or not to perform the inference operation 22 .

For example, in inference calculation 21 in step S1003 of FIG. 10, if the intermediate calculation result of a _k ⁽²⁾ exceeds the threshold, the CPU 101 determines that the degree of influence increases, proceeds to step S1009, and executes inference calculation 22. do. The threshold here can be determined in the same manner as the threshold in step S1007.

CPU 101 computes a _k ⁽²⁾ by giving priority to a node having a greater degree of influence than other nodes among the nodes of hidden layer 403 in the previous frame instead of the determinations in steps S1005 and S1007. . As a result, the CPU 101 can more quickly determine whether or not the threshold has been exceeded.

Note that the nodes in the hidden layer 403 do not have to be sorted in strict order of influence, and a node with a greater influence may be prioritized over other nodes. For example, the CPU 101 may classify the nodes into four groups according to the degree of influence, and perform calculations in order from the node in the group having the greater degree of influence than the other groups. In this case, it is not necessary to consider the degree of influence within the group. Grouping according to the degree of influence can be, for example, four groups of the top 100% to 75%, 75% to 50%, 50% to 25%, and 25% to 0% of the degree of influence. The number of groups is not limited to four, and may be determined according to the processing load due to grouping or rearrangement.

In addition, the CPU 101 preferentially performs the sum-of-products operation on a node having a higher degree of influence a _j ⁽¹⁾ among the nodes in the hidden layer 403, but the degree of influence is not limited to a _j ⁽¹⁾ . In Modified Example 2, the degree of influence for determining the order of operations for the input a _k ⁽²⁾ to the nodes of the output layer 405 is the output value x _j ⁽²⁾ in the first frame and the edge weight w _{k ,} It may be the product of _j ⁽²⁾ . In this case, the CPU 101 stores in the memory 103 the product of the output value x _j ⁽²⁾ from each node of the hidden layer 403 and the corresponding weight w _k,j ⁽²⁾ in the first frame. The CPU 101 uses the product of the output value and the weight in the first frame as the degree of influence, and gives _priority to a node whose degree of influence is greater than that of other nodes ^{. 2)} Execute the operation.

(Modification 3)
When the entire frame image is filtered to extract the feature amount, the information processing apparatus 100 may thin out part of the frame image and input it to the inference model 400 . For example, after applying a filter to lines 0-2 of the frame image, the CPU 101 thins out the scanning with a filter (lines 1-3, etc.) with lines 1 to 5 as the leading rows, and scans lines 6-8. may be filtered. When the CPU 101 thins out and applies the filter to the end of the frame image, the CPU 101 may return to the line 1-3 and apply the filter. It should be noted that the frame image is not limited to being thinned out in units of lines in the vertical direction, and may be thinned out in the horizontal direction or both vertically and horizontally.

By thinning out a portion of the frame image and scanning the filter, the CPU 101 can roughly grasp the characteristics of the entire image. The CPU 101 interrupts the calculation at an early stage based on the degree of influence in the previous frame and a partial calculation result of the calculation of the degree of influence in the current frame, even with an image in which the features are roughly captured, and reduces the amount of calculation. can be reduced.

When an image signal is handled as an input, the input x _i ⁽¹⁾ may be an unsigned 8-bit RGB signal or YUV signal with an offset. A YUV signal can express color information by a combination of a luminance signal (Y), a difference (U) between the luminance signal and the blue component, and a difference (V) between the luminance signal and the red component.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist. Moreover, it is also possible to appropriately combine the plurality of features described in the embodiments.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100: information processing device, 101: CPU, 104: image acquisition unit, 105: identification unit, 400: inference model

Claims (12)

acquisition means for acquiring successive frames of a moving image;
A calculator including a plurality of hierarchies, wherein the degree of impact is the product sum of the output from each node included in the first hierarchy to a predetermined node in the second hierarchy and the weight corresponding to each of the outputs and an identification means for identifying an object from each frame of the moving image using a calculator that inputs the calculated degree of influence to the predetermined node,
The identifying means includes a first influence input to the predetermined node in a first frame and a second influence input to the predetermined node in a second frame after the first frame. determining whether or not to interrupt the calculation of the second degree of influence based on a partial calculation result of the calculation of the degree of influence. The identification means is a value corresponding to the sum of products for some nodes of the first hierarchy in the second frame and the sum of products for the some nodes in the first frame. When the difference from the reference value is within a predetermined range and the sum of products for some nodes of the first hierarchy in the second frame is equal to or less than a threshold, the 2. The information processing apparatus according to claim 1, wherein the computation of the second degree of influence is interrupted. 3. The information processing apparatus according to claim 2, wherein said predetermined range is set based on said first degree of influence. The threshold is set based on the maximum possible value of the product sum of the output from each node included in the first hierarchy to a predetermined node in the second hierarchy and the weight corresponding to each of the outputs. 4. The information processing apparatus according to claim 2, wherein: 4. The identifying means inputs a calculation result or a predetermined value at the time when the calculation of the second degree of influence is interrupted to the predetermined node as the second degree of influence. The information processing device according to any one of . The identifying means prioritizes a node having a greater degree of influence than other nodes among the nodes of the first hierarchy in the first frame, and determines the second influence in the second frame. 6. The information processing apparatus according to claim 1, wherein the sum of products of degrees is calculated. In the first frame, the identification means classifies each node of the first hierarchy into a plurality of groups based on the degree of influence, and from the node of the group having the degree of influence greater than that of other groups, 7. The information processing apparatus according to claim 6, wherein the sum of products of said second degree of influence in said second frame is calculated in order. 8. Information according to any one of claims 1 to 7, characterized in that said identifying means uses an unsigned value as a weight used in calculating said first degree of influence and said second degree of influence. processing equipment. 9. The information processing apparatus according to claim 1, wherein said identifying means thins out a part of frame images of said moving image and inputs it to said calculator. 10. The information processing apparatus according to any one of claims 1 to 9, wherein said identifying means updates said first degree of influence every predetermined number of frames of said moving image. an acquisition step of acquiring successive frames of the video image;
A calculator including a plurality of hierarchies, wherein the degree of impact is the product sum of the output from each node included in the first hierarchy to a predetermined node in the second hierarchy and the weight corresponding to each of the outputs and an identification step of identifying an object from each frame of the moving image using a calculator that inputs the calculated degree of influence to the predetermined node;
have
In the identifying step, a first influence input to the predetermined node in a first frame and a second influence input to the predetermined node in a second frame after the first frame. and determining whether or not to interrupt the calculation of the second degree of influence based on a partial calculation result of the calculation of the degree of influence. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 10.

Images