JP6943105B2 - Information processing systems, information processing devices, and programs - Google Patents

Description

The present invention relates to an information processing system, an information processing device, and a program.

Currently, surveillance cameras are being installed in various places, and the number of surveillance cameras is on the rise. In addition, surveillance cameras connected to networks are becoming widespread, and various applications using images (still images or moving images) obtained from surveillance cameras via networks are conceivable.

For example, Patent Document 1 below proposes a system that utilizes an image obtained by imaging with a surveillance camera for crime prevention, etc. by transmitting it to a server via a network and performing necessary processing on the server. There is. Further, Patent Document 1 below describes that real-time performance is maintained by sharing processing between a surveillance camera and a plurality of servers.

Japanese Patent No. 361220

Sachin Sudhakar Farfade, 2 outsiders, "Multi-view Face Detection Using Deep Convolutional Neural Networks", In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval (ICMR), June 2015, p.643-650 Florian Schroff, 2 outsiders, "FaceNet: A Unified Embedding for Face Recognition and Clustering", In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June 2015

However, in the technique described in Patent Document 1, since the process of extracting a person candidate image from the captured image is performed by the surveillance camera, if the surveillance camera does not have the performance to perform the process in real time, Real-time performance could not be maintained. In such a system for sharing processing, a mechanism capable of sharing processing in finer units has been desired.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is a new and improved information processing system and information processing capable of sharing processing in finer units. To provide equipment and programs.

In order to solve the above problem, according to a certain viewpoint of the present invention, it is an information processing system that performs neural network processing using a neural network composed of at least n layers when n is an integer of 2 or more. When k is an integer of 1 or more and n-1 or less, the input data is used as an input, and the neural network processing using the first layer to the kth layer of the neural network is performed to perform the k-th layer output value. A first neural network processing unit that outputs A first transmitting unit that transmits to, a first receiving unit that receives the communication k-layer output value from the first communication network, and the communication k-layer received by the first receiving unit. A second conversion unit that converts an output value into the k-th layer output value, and a second that performs the neural network processing using at least the k + 1 layer of the neural network with the k-layer output value as an input. When q is an integer of 1 or more and n or less with the neural network processing unit, the qth layer output value output by the qth layer of the neural network in the neural network processing using the first input data as an input. Based on the above, an information processing system is provided that includes a processing sharing determination unit that determines the value of k in the neural network processing that uses a second input data different from the first input data as an input.

Pre Symbol processing sharing determination unit sets a value of the k indicating the processing sharing boundary between said second neural network processing unit and the first neural network processing unit may determine.

The first input data and the second input data are sensing data acquired by sensing, and the information processing system acquires the second input data based on the qth layer output value. It may further have a resolution determination unit that determines the resolution related to the sensing.

The processing sharing determination unit has lower processing performance than the first neural network processing unit and the second neural network processing unit when the resolution determination unit determines a higher value as the resolution. The value of k may be determined so that the processing load on the side becomes smaller.

The first input data and the second input data are image data, the neural network is a neural network for recognizing an object included in the image data, and the qth layer output value is the qth layer output value. It may include information about the detection result of the object.

When n is an integer of 3 or more and m is an integer of k + 1 or more and n-1 or less, the second neural network processing unit uses the k + 1 layer to the mth layer of the neural network. The neural network processing that has been performed is performed to output the m-th layer output value, and the information processing system communicates with a third conversion unit that converts the m-layer output value into a communication m-layer output value. A second transmitting unit that transmits the m-layer output value for communication to the second communication network, a second receiving unit that receives the m-layer output value for communication from the second communication network, and the second receiving unit. A fourth conversion unit that converts the communication mth layer output value received by the reception unit into the mth layer output value, and at least the m + 1th layer of the neural network with the mth layer output value as input. A third neural network processing unit that performs the neural network processing using the above may be further provided.

Further, in order to solve the above problem, according to another viewpoint of the present invention, when n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, a neural network composed of at least n layers. A first neural network processing unit that performs neural network processing using the first layer to the kth layer of the network and outputs the kth layer output value, and the kth layer output value for communication kth layer When q is an integer of 1 or more and n or less, the first conversion unit that converts the output value, the first transmission unit that transmits the k-layer output value for communication to the first communication network, and the first In the neural network processing using the input data of the above, the second input data different from the first input data is input based on the qth layer output value output by the qth layer of the neural network. Provided is an information processing apparatus including a processing sharing determination unit for determining the value of k in the neural network processing.

Further, in order to solve the above problem, according to another viewpoint of the present invention, when n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, a neural computer composed of at least n layers. A function of performing neural network processing using the first layer to the kth layer of the network and outputting the kth layer output value, and a function of converting the kth layer output value into a communication kth layer output value. And the function of transmitting the k-layer output value for communication to the first communication network, and the neural network processing in which the first input data is input when q is an integer of 1 or more and n or less. A function of determining the value of k in the neural network processing in which a second input data different from the first input data is input based on the qth layer output value output by the qth layer of the network. When a program for implementing the computer is provided with.

Further, in order to solve the above problem, according to another viewpoint of the present invention, when n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, a neural network composed of at least n layers. The k-th layer output value for communication obtained by converting the k-layer output value output by performing neural network processing using the first layer to the k-th layer of the network is obtained from the first communication network. A first receiving unit to receive, a second conversion unit that converts the communication k-layer output value received by the first receiving unit into the k-layer output value, and the k-layer output value. as input, wherein the second neural network processing unit to perform the neural network processing that uses at least a k + 1 So Uchi neural network, have a, when the q and an integer from 1 to n, a first input In the neural network process using data as an input, the second input data different from the first input data is input based on the qth layer output value output by the qth layer of the neural network. An information processing system is provided in which the value of k in the neural network processing is determined.

Further, in order to solve the above problem, according to another viewpoint of the present invention, when n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, a neural network composed of at least n layers. The k-th layer output value for communication obtained by converting the k-layer output value output by performing neural network processing using the first layer to the k-th layer of the network is obtained from the first communication network. Among the neural networks, the function of receiving, the function of converting the k-layer output value for communication received by the first receiving unit into the k-layer output value, and the k-layer output value as input. A program for realizing a function of performing the neural network processing using at least the first k + 1 layer on a computer, and when q is an integer of 1 or more and n or less, the first input data is used as an input. The k in the neural network processing in which the second input data different from the first input data is input based on the qth layer output value output by the qth layer of the neural network in the neural network processing. A program is provided that determines the value of.

As described above, according to the present invention, it is possible to divide the processing into finer units.

It is explanatory drawing for demonstrating the schematic structure of the monitoring system 900 common to each embodiment of this invention. It is explanatory drawing for demonstrating the outline of the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the monitoring system 900-1 which concerns on the same embodiment. It is a sequence diagram which shows the processing flow of the monitoring system 900-1 which concerns on the same embodiment. It is explanatory drawing for demonstrating the outline of the 2nd Embodiment of this invention. It is a block diagram which shows the structural example of the monitoring system 900-2 which concerns on the same embodiment. It is a sequence diagram which shows the processing flow of the monitoring system 900-2 which concerns on the same embodiment. It is explanatory drawing for demonstrating the outline of the 3rd Embodiment of this invention. It is a block diagram which shows the structural example of the monitoring system 900-3 which concerns on the same embodiment. It is a table which shows an example of the determination of the integer k, the integer m, and the resolution (frame rate and resolution) by the determination unit 253. It is a sequence diagram which shows the processing flow of the monitoring system 900-3 which concerns on the same embodiment. It is explanatory drawing which shows the hardware configuration example.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of the plurality of components having substantially the same functional configuration, only the same reference numerals are given.

<< 1. Overview >>
<1-1. Background ＞
Currently, surveillance cameras are installed all over the city, and the number of surveillance cameras is on the rise. In addition, most of the surveillance cameras are connected to a network, and various applications using images (still images or moving images) obtained from the surveillance cameras via the network can be considered.

For example, Patent Document 1 proposes a technique of transmitting an captured image obtained by imaging of a surveillance camera to a server via a network and performing necessary processing on the server to utilize it for crime prevention or the like. There is. Further, Patent Document 1 also describes that real-time performance is maintained by sharing processing between a surveillance camera and a plurality of servers.

However, the technique described in Patent Document 1 has the ability for the surveillance camera to perform the processing in real time because the surveillance camera performs the process of detecting the person from the captured image and extracting the candidate image of the person. Without it, real-time performance cannot be maintained. In order to maintain real-time performance, for example, it is possible to reduce the frame rate (time resolution) or the resolution (spatial resolution), but in such a case, the accuracy of person detection may decrease. Therefore, a mechanism that can divide the processing into smaller units has been desired.

Further, in the technique of Patent Document 1, an image of a person can be transmitted from a surveillance camera to a server. Therefore, if the communication content between the surveillance camera and the server is stolen, by visually checking the image, it is possible to know when and where the person was and what the person was doing. There was a risk that the information would be grasped and the privacy of the person concerned would be infringed.

<1-2. Basic configuration>
The background of the embodiment of the present invention has been described above. The inventor of the present invention has come to create an embodiment of the present invention with the above-mentioned circumstances as a point of view.

Hereinafter, the basic configuration of the monitoring system 900 common to each embodiment of the present invention will be described with reference to FIG. FIG. 1 is an explanatory diagram for explaining a schematic configuration of a monitoring system 900 common to each embodiment of the present invention. As shown in FIG. 1, the surveillance system 900 includes a surveillance camera 1, an intermediate server 2, a recognition server 3, a communication network 5A, and a communication network 5B.

As shown in FIG. 1, the surveillance camera 1 and the intermediate server 2 are connected via the communication network 5A, and the intermediate server 2 and the recognition server 3 are connected via the communication network 5B.

The communication network 5A and the communication network 5B are wired or wireless transmission lines of information transmitted from a device or system connected to the communication network 5A and the communication network 5B, respectively. For example, the communication network 5 (communication network 5A and communication network 5B) includes public network such as the Internet, telephone network, satellite communication network, various LANs (Local Area Network) including Ethernet (registered trademark), and WAN. (Wide Area Network) and the like may be included. Further, the communication network 5 may include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).

The surveillance system 900 performs neural network processing using a neural network in order to recognize a person's face (an example of an object) included in the image data obtained by the imaging of the surveillance camera 1. The neural network for realizing face recognition can be generated by a method such as the above-mentioned Non-Patent Document 1 or Non-Patent Document 2. Further, the neural network processing using the neural network can be specified by the neural network parameters.

For example, the neural network process performed by the monitoring system 900 may include a face detection process for finding a person's face from image data and a face recognition process for collating who the detected face is. However, in the neural network processing performed by the monitoring system 900, the face detection processing and the face recognition processing are not always clearly distinguished.

A neural network is a mathematical model that aims to express some characteristics of brain function by computer simulation. For example, a neural network is composed of a large number of layers, and can perform a process of extracting a feature amount of input data, a process of identifying the extracted feature amount, and the like by a large number of layers.

The monitoring system 900 according to each embodiment of the present invention divides the processing into layers of the neural network, and divides the divided processing into at least two devices of the surveillance camera 1, the intermediate server 2, and the recognition server 3. With such a configuration, it is possible to divide the processing in small units, and it is easy to maintain real-time performance even when imaging is performed at a higher resolution and a higher frame rate.

Further, in the second embodiment and the third embodiment described later, the image data is not transmitted between the devices, but the output value which is the processing result of each stage shared by each device is transmitted. Even if the communication content is stolen, privacy is unlikely to be infringed.

The basic configuration of the monitoring system 900 common to each embodiment of the present invention has been described above. Hereinafter, each embodiment of the present invention that realizes the above-mentioned effects will be described in detail in order.

<< 2. Detailed description of each embodiment >>
<2-1. First Embodiment>
(Overview)
First, an outline of the monitoring system 900 according to the first embodiment of the present invention will be described. In the following, the surveillance system 900 according to the first embodiment of the present invention will be referred to as a surveillance system 900-1, and the surveillance camera 1, the intermediate server 2, and the recognition server 3 of the surveillance system 900-1 will be monitored, respectively. It is referred to as a camera 1-1, an intermediate server 2-1 and a recognition server 3-1.

FIG. 2 is an explanatory diagram for explaining an outline of the first embodiment of the present invention. FIG. 2 shows the neural network NN1 used by the monitoring system 900-1 according to the present embodiment. As shown in FIG. 2, the neural network NN1 is a neural network composed of n layers. In this embodiment, n is an integer of at least 2 or more.

Further, the neural network NN1 shown in FIG. 2 is a neural network in which the output value of each layer is input to the next (right) layer of the layer. The output value of each layer is not limited to the scalar value, and may be, for example, a vector value. Hereinafter, as shown in FIG. 2, the layers constituting the neural network NN1 according to the present embodiment are arranged in order from the left, the first layer L ₁ , the second layer L ₂ , the third layer L ₃ , ..., The nth layer. It is called _{L n.}

The monitoring system 900-1 according to the present embodiment shares the neural network processing using the neural network NN1 between the intermediate server 2-1 and the recognition server 3-1. In the example shown in FIG. 2, the intermediate server 2-1 is _{in charge of the neural network processing of the first layer L 1} to the k layer L _k , and the recognition server 3-1 is in charge of the first k + 1 layer L _{k + 1} to the nth layer L _n. Responsible for neural network processing. Here, k is an integer of 1 or more and n-1 or less, and the intermediate server 2-1 and the recognition server 3-1 (more accurately, the intermediate server 2-1 and the recognition server 3-1 are described later. The processing sharing boundary (of each processing unit) is shown.

In the present embodiment, the integer k indicating the processing sharing boundary between the intermediate server 2-1 and the recognition server 3-1 may be a preset value. For example, the integer k is set by the user so as to satisfy the required processing time in consideration of the processing performance of the intermediate server 2-1 and the recognition server 3-1 and the frame rate and resolution of the surveillance camera 1-1. May be good. In the present embodiment, in the neural network NN1, the face detection process and the face recognition process may or may not be clearly distinguished. Even when the face detection process and the face recognition process are clearly distinguished in the neural network NN1, the integer k indicating the processing sharing boundary between the intermediate server 2-1 and the recognition server 3-1 is the face detection process. It can be set independently of the boundary of face recognition processing. Furthermore, the integer k indicating the processing sharing boundary between the intermediate server 2-1 and the recognition server 3-1 may be set regardless of what processing each layer of the neural network NN1 is for. .. With such a configuration, it is possible to divide the processing into the intermediate server 2-1 and the recognition server 3-1 in finer units.

(Configuration example)
The outline of the monitoring system 900-1 according to the present embodiment has been described above. Subsequently, a configuration example of the monitoring system 900-1 according to the present embodiment will be described in more detail. FIG. 3 is a block diagram showing a configuration example of the monitoring system 900-1 according to the present embodiment.

As shown in FIG. 3, the surveillance camera 1-1 includes an imaging unit 111 and a communication interface unit 120.

The imaging unit 111 is a camera module that acquires image data (an example of sensing data) by imaging (an example of sensing). For example, the image pickup unit 111 converts light into an electric signal by taking an image of the surrounding real space using an image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and obtains image data. To generate. The image capturing unit 111 provides image data to the communication interface unit 120.

The communication interface unit 120 mediates communication between the surveillance camera 1-1 and other devices. The communication interface unit 120 supports an arbitrary wireless communication protocol or a wired communication protocol, and establishes a communication connection with another device via the communication network 5A or directly. As shown in FIG. 3, the communication interface unit 120 includes a conversion unit 122 and a communication unit 124.

The conversion unit 122 converts the data into data (communication data) in a format that can be transmitted by the communication unit 124. For example, the conversion unit 122 converts the image data provided by the imaging unit 111 into communication image data and provides the image data to the communication unit 124.

The communication unit 124 transmits communication data to another device via the communication network 5A or directly, or receives communication data from the other device. For example, the communication unit 124 transmits the communication image data provided by the conversion unit 122 to the communication network 5A. In the present embodiment, since the image data for communication is transmitted from the surveillance camera 1-1 to the intermediate server 2-1 via the communication network 5A, it is desirable that the communication network 5A is a communication network having high security. ..

As shown in FIG. 3, the intermediate server 2-1 is an information processing device including a communication interface unit 220, a processing unit 231 and a storage unit 240.

The communication interface unit 220 mediates communication between the intermediate server 2-1 and other devices. The communication interface unit 220 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with or directly from the communication network 5A or the communication network 5B. As shown in FIG. 3, the communication interface unit 220 includes a conversion unit 222 and a communication unit 224.

The conversion unit 222 converts (reversely transforms) the communication data received by the communication unit 224 into data for handling by the processing unit 231 and the storage unit 240, and provides the data to the processing unit 231 and the storage unit 240. For example, the conversion unit 222 converts the communication image data received from the communication network 5A by the communication unit 224 into image data (input data in the present embodiment) and provides it to the processing unit 231.

Further, the conversion unit 222 converts the data into data (communication data) in a format that can be transmitted by the communication unit 224. For example, the conversion unit 222 functions as the first conversion unit in the present embodiment, and converts the k-th layer output value of the neural network NN1 output from the processing unit 231 described later into the k-th layer output value for communication. It is provided to the communication unit 224.

The communication unit 224 transmits communication data to another device, or receives communication data from the other device, via the communication network 5A, via the communication network 5B, or directly. For example, the communication unit 224 functions as the first transmission unit in the present embodiment, and transfers the k-layer output value for communication provided by the conversion unit 222 to the communication network 5B (the first communication network in the present embodiment). Send. Further, the communication unit 224 receives the communication image data transmitted by the surveillance camera 1-1 from the communication network 5A.

The processing unit 231 performs neural network processing. The neural network processing performed by the processing unit 231 can be specified by, for example, the neural network parameters stored in the storage unit 240. Incidentally, the neural network parameters stored in the storage unit 240 may be a parameter corresponding to the entire neural network NN1, corresponding parameter from the first layer L ₁ to the k-th layer L _k of the neural network NN1 It may be. An integer k indicating a processing sharing boundary of the processing unit 331 of the processing unit 231 and the recognition server 3-1 described later may be stored in the storage unit 240.

Processing unit 231 functions as a first neural network processing unit in the present embodiment, use of the input image data, from the first layer L ₁ of the neural network NN1 shown in FIG. 2 to the k-th layer L _k Performs the neural network processing that was used. The processing unit 231 functions as a first neural network processing unit in the present embodiment, and outputs the k-th layer output value is an output value of the k-th layer L _k to a communication interface unit 220.

The storage unit 240 stores programs and data used for the operation of the intermediate server 2-1. Further, the storage unit 240 stores the neural network parameters related to the neural network NN1.

As shown in FIG. 3, the recognition server 3-1 is an information processing device including a communication interface unit 320, a processing unit 331, and a storage unit 340.

The communication interface unit 320 mediates communication between the recognition server 3-1 and other devices. The communication interface unit 320 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with another device via the communication network 5B or directly. As shown in FIG. 3, the communication interface unit 320 includes a conversion unit 322 and a communication unit 324.

The conversion unit 322 converts (reversely transforms) the communication data received by the communication unit 324 into data for handling by the processing unit 331 and the storage unit 340, and provides the data to the processing unit 331 and the storage unit 340. For example, the conversion unit 322 functions as a second conversion unit in the present embodiment, and the communication unit 324 receives the communication k-layer output value received from the communication network 5B (the first communication network in the present embodiment). It is converted into a k-layer output value and provided to the processing unit 331.

The communication unit 324 transmits the communication data to another device via the communication network 5B or directly, or receives the communication data from the other device. For example, the communication unit 324 functions as the first reception unit in the present embodiment, and receives the communication k-th layer output value transmitted by the intermediate server 2-1 from the communication network 5B.

The processing unit 331 performs neural network processing. The neural network processing performed by the processing unit 331 can be specified by, for example, the neural network parameters stored in the storage unit 340. The neural network parameter stored in the storage unit 340 may be a parameter corresponding to the entire neural network NN1, or a parameter corresponding to the k + 1st layer L _{k + 1} to the nth layer L _{n of the neural network NN1.} It may be. An integer k indicating a processing sharing boundary between the processing unit 231 and the processing unit 331 included in the intermediate server 2-1 may be stored in the storage unit 340.

The processing unit 331 functions as a second neural network processing unit in the present embodiment, receives the k-th layer output value as an input, and has k + 1 layer L _{k + 1} to n-th layer L _{n of the neural network NN1 shown in FIG.} Perform neural network processing using up to. Processing unit 331 is an output value of the n-th layer L _n outputs the n-th layer output values may be stored for example in the storage unit 340. Alternatively, the nth layer output value output by the processing unit 331 may be displayed on a display unit (not shown), or is converted into communication data by the conversion unit 322 and then transmitted to another device by the communication unit 324. May be done. As described above, the neural network NN1 according to the present embodiment is a neural network for face recognition, and the nth layer output value includes, for example, who's face in the image captured by the surveillance camera 1-1. Can include information about whether or not it is possible.

The storage unit 340 stores programs and data used for the operation of the recognition server 3-1. Further, the storage unit 340 stores the neural network parameters related to the neural network NN1.

(Operation example)
The configuration example of the monitoring system 900-1 according to the first embodiment of the present invention has been described above. Subsequently, an operation example of the present embodiment will be described with reference to FIG. FIG. 4 is a sequence diagram showing a processing flow of the monitoring system 900-1 according to the present embodiment.

As shown in FIG. 4, first, the surveillance camera 1-1 acquires image data by imaging the imaging unit 111 (S102). Subsequently, the conversion unit 122 of the surveillance camera 1-1 converts the image data into communication image data (S106). Further, the communication unit 124 of the surveillance camera 1-1 transmits the communication image data to the communication network 5A, and the communication unit 224 of the intermediate server 2-1 receives the communication image data from the communication network 5A (S108). ..

Subsequently, the conversion unit 222 of the intermediate server 2-1 converts the communication image data received in step S108 into image data (S110). Further, the processing unit 231 of the intermediate server 2-1, and inputs the image data, the k-th layer output by performing a neural network processing using the first layer L ₁ of the neural network NN1 to the k-th layer L _k The value is output (S112).

Subsequently, the conversion unit 222 of the intermediate server 2-1 converts the k-th layer output value into the communication k-th layer output value (S114). Further, the communication unit 224 of the intermediate server 2-1 transmits the k-layer output value for communication to the communication network 5B, and the communication unit 324 of the recognition server 3-1 transmits the k-layer output value for communication to the communication network 5B. Received from (S116).

Subsequently, the conversion unit 322 of the recognition server 3-1 converts the communication k-th layer output value received in step S116 into the k-th layer output value (S118). Further, the processing unit 331 of the recognition server 3-1 takes the kth layer output value as an input and _{performs neural network processing using the k + 1 layer L k + 1} to the nth layer L _{n of} the neural network NN1 to perform the first neural network processing. The n-layer output value is output (S120).

(effect)
The first embodiment of the present invention has been described above. According to the present embodiment, the neural network processing using the image data acquired by the imaging of the surveillance camera 1-1 as the input is shared between the intermediate server 2-1 and the recognition server 3-1. Further, as described above, the integer k indicating the processing sharing boundary between the intermediate server 2-1 and the recognition server 3-1 can be set independently of the processing related to each layer, so that the processing can be set in a finer unit. It is possible to share the work between 2-1 and the recognition server 3-1. Further, communication related to image data is not performed between the intermediate server 2-1 and the recognition server 3-1 and the k-layer output value for communication is communicated. With such a configuration, even if the communication content between the intermediate server 2-1 and the recognition server 3-1 via the communication network 5B is stolen, privacy is unlikely to be infringed.

<2-2. Second embodiment>
(Overview)
In the first embodiment, the example in which the neural network processing is shared by the two devices of the intermediate server 2-1 and the recognition server 3-1 has been described, but the neural network processing may be shared by three or more devices. It is possible. Hereinafter, an example in which the neural network processing is shared by the three devices of the surveillance camera 1, the intermediate server 2, and the recognition server 3 will be described as a second embodiment of the present invention. In the following, the surveillance system 900 according to the second embodiment of the present invention will be referred to as a surveillance system 900-2, and the surveillance camera 1, the intermediate server 2, and the recognition server 3 of the surveillance system 900-2 will be monitored, respectively. It is called a camera 1-2, an intermediate server 2-2, and a recognition server 3-2.

FIG. 5 is an explanatory diagram for explaining the outline of the second embodiment of the present invention. FIG. 5 shows the neural network NN2 used by the monitoring system 900-2 according to the present embodiment. As shown in FIG. 5, the neural network NN2 is a neural network composed of n layers. In this embodiment, n is an integer of at least 3 or more.

Further, the neural network NN2 shown in FIG. 5 is a neural network in which the output value of each layer is input to the next (right) layer of the layer. The output value of each layer is not limited to the scalar value, and may be, for example, a vector value. Hereinafter, as shown in FIG. 5, each layer constituting the neural network NN2 according to the present embodiment is arranged in order from the left, the first layer L ₁ , the second layer L ₂ , the third layer L ₃ , ..., The nth layer. It is called _{L n.}

In the monitoring system 900-2 according to the present embodiment, the neural network processing using the neural network NN2 is shared by the surveillance camera 1-2, the intermediate server 2-2, and the recognition server 3-2. In the example shown in FIG. 5, the monitoring camera 1-2 is responsible for neural network processing of the first layer _{L 1} ~ k-th layer _{L k.} Further, the intermediate server 2-2 is in charge of the neural network processing of the _{first k + 1 layer L k + 1} to the mth layer L _m. Further, the recognition server 3-2 is in charge of the neural network processing of _{the m + 1 layer L m + 1} to the nth layer L _n. Here, k is an integer of 1 or more and m-1 or less, and the surveillance camera 1-2 and the intermediate server 2-2 (more accurately, the surveillance camera 1-2 and the intermediate server 2-2 are described later). The processing sharing boundary (of each processing unit) is shown. Further, m is an integer of k + 1 or more and n-1 or less, and the intermediate server 2-2 and the recognition server 3-2 (more accurately, the intermediate server 2-2 and the recognition server 3-2, respectively, as described later). It shows the processing sharing boundary (of the processing unit).

In the present embodiment, the integer k indicating the processing sharing boundary between the surveillance camera 1-2 and the intermediate server 2-2 and the integer m indicating the processing sharing boundary between the intermediate server 2-2 and the recognition server 3-2 are preset. It may be a value. For example, the integer k and the integer m are required in consideration of the processing performance of the surveillance camera 1-2, the intermediate server 2-2, and the recognition server 3-2, the frame rate and the resolution of the surveillance camera 1-2, and the like. It may be set by the user to satisfy the processing time. In the present embodiment, in the neural network NN2, the face detection process and the face recognition process may or may not be clearly distinguished. Even when the face detection process and the face recognition process are clearly distinguished in the neural network NN2, the integers k and m are set independently of the boundary between the face detection process and the face recognition process. obtain. Furthermore, the integer k and the integer m may be set independently of what processing each layer of the neural network NN2 is for. With such a configuration, it is possible to divide the processing into the surveillance camera 1-2, the intermediate server 2-2, and the recognition server 3-3 in finer units.

(Configuration example)
The outline of the monitoring system 900-2 according to the present embodiment has been described above. Subsequently, a configuration example of the monitoring system 900-2 according to the present embodiment will be described in more detail. FIG. 6 is a block diagram showing a configuration example of the monitoring system 900-2 according to the present embodiment. Since the monitoring system 900-2 according to the present embodiment has the same configuration as the monitoring system 900-1 according to the first embodiment in part, the description will be omitted as appropriate.

As shown in FIG. 6, the surveillance camera 1-2 is an information processing device including an imaging unit 112, a processing unit 132, a storage unit 140, and a communication interface unit 170.

The imaging unit 112 is a camera module that acquires image data (an example of sensing data) by imaging (an example of sensing) in the same manner as the imaging unit 111 described with reference to FIG. However, the imaging unit 112 according to the present embodiment is different from the imaging unit 111 described with reference to FIG. 3 in that image data (input data in the present embodiment) is provided to the processing unit 132.

The processing unit 132 performs neural network processing. The neural network processing performed by the processing unit 132 can be specified by, for example, the neural network parameters stored in the storage unit 140. Incidentally, the neural network parameters stored in the storage unit 140 may be a parameter corresponding to the entire neural network NN2, corresponding parameter from the first layer L ₁ to the k-th layer L _k of the neural network NN2 It may be. An integer k indicating a processing sharing boundary of the processing unit 232 of the processing unit 132 and the intermediate server 2-2 described later may be stored in the storage unit 140.

Processing unit 132 functions as a first neural network processing unit in the present embodiment, an input image data provided from the imaging unit 112, first from the first layer L ₁ of the neural network NN2, shown in FIG. 5 performing a neural network processing using up to k layer L _k. The processing unit 132 functions as a first neural network processing unit in the present embodiment, and outputs the k-th layer output value is an output value of the k-th layer L _k to a communication interface unit 170.

The storage unit 140 stores programs and data used for the operation of the surveillance cameras 1-2. Further, the storage unit 140 stores the neural network parameters related to the neural network NN2.

The communication interface unit 170 mediates communication with other devices by the surveillance cameras 1-2, similarly to the communication interface unit 120 described with reference to FIG. The communication interface unit 170 supports an arbitrary wireless communication protocol or a wired communication protocol, and establishes a communication connection with another device via the communication network 5A or directly. As shown in FIG. 6, the communication interface unit 170 includes a conversion unit 172 and a communication unit 174.

The conversion unit 172 converts the data into data (communication data) in a format that can be transmitted by the communication unit 174. For example, the conversion unit 172 functions as the first conversion unit in the present embodiment, converts the k-th layer output value of the neural network NN2 output from the processing unit 132 into the k-th layer output value for communication, and is the communication unit. Provide to 174.

The communication unit 174 transmits communication data to another device via the communication network 5A or directly, or receives communication data from the other device. For example, the communication unit 174 transmits the communication k-layer output value provided by the conversion unit 172 to the communication network 5A (the first communication network in the present embodiment).

The communication unit 174 transmits communication data to another device via the communication network 5A or directly, or receives communication data from the other device. For example, the communication unit 174 functions as the first transmission unit in the present embodiment, and transfers the k-layer output value for communication provided by the conversion unit 172 to the communication network 5A (the first communication network in the present embodiment). Send.

As shown in FIG. 6, the intermediate server 2-2 is an information processing device including a processing unit 232, a storage unit 240, and a communication interface unit 270. Since the function of the storage unit 240 shown in FIG. 6 is the same as the function of the storage unit 240 described with reference to FIG. 3, the description thereof will be omitted.

The processing unit 232 performs neural network processing. The neural network processing performed by the processing unit 232 can be specified by, for example, the neural network parameters stored in the storage unit 240. The neural network parameter stored in the storage unit 240 in the present embodiment may be a parameter corresponding to the entire neural network NN2, or from the k + 1 layer L _{k + 1} to the mth layer L _{m of the neural network NN2.} It may be a parameter corresponding to. Further, an integer k indicating the processing sharing boundary between the processing unit 132 and the processing unit 232 of the surveillance camera 1-2, and an integer m indicating the processing sharing boundary of the processing unit 232 and the processing unit 332 of the recognition server 3-1 described later. May be stored in the storage unit 240.

The processing unit 232 functions as the second neural network processing unit in the present embodiment, receives the output value of the k-th layer as an input, and has the k + 1 layer L _{k + 1} to the m-th layer L _{m of the neural network NN2 shown in FIG.} Perform neural network processing using up to. The processing unit 232 functions as a second neural network processing unit in this embodiment, outputs an m-th layer output value is an output value of the m-th layer L _m to the communication interface unit 270.

The communication interface unit 270 mediates communication between the intermediate server 2-2 and other devices, similarly to the communication interface unit 220 described with reference to FIG. The communication interface unit 270 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with or directly from the communication network 5A or the communication network 5B. The communication interface unit 270 includes a conversion unit 272 and a communication unit 274 as shown in FIG.

The conversion unit 272 converts (reversely converts) the communication data received by the communication unit 274 into data for handling by the processing unit 232 and the storage unit 240, and provides the data to the processing unit 232 and the storage unit 240. For example, the conversion unit 272 functions as a second conversion unit in the present embodiment, converts the communication k-layer output value received from the communication network 5A by the communication unit 274 into the k-layer output value, and processes the processing unit 232. To provide to.

Further, the conversion unit 272 converts the data into data (communication data) in a format that can be transmitted by the communication unit 274. For example, the conversion unit 272 functions as a third conversion unit in the present embodiment, converts the mth layer output value of the neural network NN2 output from the processing unit 232 into the communication mth layer output value, and communicates unit. Provide to 274.

The communication unit 274 transmits communication data to another device, or receives communication data from the other device, via the communication network 5A, via the communication network 5B, or directly. For example, the communication unit 274 functions as the first reception unit in the present embodiment, and receives the communication k-layer output value transmitted by the surveillance camera 1-1 from the communication network 5A. Further, the communication unit 274 functions as a second transmission unit in the present embodiment, and transfers the communication m-layer output value provided by the conversion unit 272 to the communication network 5B (second communication network in the present embodiment). Send.

As shown in FIG. 6, the recognition server 3-2 is an information processing device including a processing unit 332, a storage unit 340, and a communication interface unit 370. Since the function of the storage unit 340 shown in FIG. 6 is the same as the function of the storage unit 340 described with reference to FIG. 3, the description thereof will be omitted.

The processing unit 332 performs neural network processing. The neural network processing performed by the processing unit 332 can be specified by, for example, the neural network parameters stored in the storage unit 340. The neural network parameter stored in the storage unit 340 in the present embodiment may be a parameter corresponding to the entire neural network NN2, or from the m + 1 layer L _{m + 1} to the nth layer L _{n of the neural network NN2.} It may be the corresponding parameter. Further, the integer m indicating the processing sharing boundary between the processing unit 232 and the processing unit 332 of the intermediate server 2-2 may be stored in the storage unit 340.

The processing unit 332 functions as a third neural network processing unit in the present embodiment, receives the output value of the mth layer as an input, and has m + 1 layer L _{m + 1} to the nth layer L _{n of the neural network NN2 shown in FIG.} Perform neural network processing using up to. The n layer output value is an output value of the n-th layer L _n which is output by the processing unit 332 may be stored in the storage unit 340, may be displayed on the display unit (not shown), conversion unit After being converted into communication data by 372, it may be transmitted to another device by the communication unit 374.

The communication interface unit 370 mediates communication between the recognition server 3-2 and other devices. The communication interface unit 370 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with another device via the communication network 5B or directly. As shown in FIG. 6, the communication interface unit 370 includes a conversion unit 372 and a communication unit 374.

The conversion unit 372 converts (reversely converts) the communication data received by the communication unit 374 into data for handling by the processing unit 332 and the storage unit 340, and provides the data to the processing unit 332 and the storage unit 340. For example, the conversion unit 372 functions as a fourth conversion unit in the present embodiment, and the communication unit 374 receives the communication m-layer output value received from the communication network 5B (second communication network in the present embodiment). It is converted into an m-layer output value and provided to the processing unit 332.

The communication unit 374 transmits the communication data to another device via the communication network 5B or directly, or receives the communication data from the other device. For example, the communication unit 374 functions as a second reception unit in the present embodiment, and receives the communication m-layer output value transmitted by the intermediate server 2-2 from the communication network 5B.

(Operation example)
The configuration example of the monitoring system 900-2 according to the second embodiment of the present invention has been described above. Subsequently, an operation example of the present embodiment will be described with reference to FIG. 7. FIG. 7 is a sequence diagram showing a processing flow of the monitoring system 900-2 according to the present embodiment.

As shown in FIG. 7, first, the surveillance camera 1-2 acquires image data by imaging the imaging unit 112 (S202). Subsequently, the processing unit 132 of the monitoring camera 1-2, and inputs the image data, the k-th layer by performing a neural network processing using the first layer L ₁ to the k-th layer L _k of the neural network NN2 The output value is output (S204).

Subsequently, the conversion unit 172 of the surveillance camera 1-2 converts the k-th layer output value into the communication k-th layer output value (S206). Further, the communication unit 174 of the surveillance camera 1-2 transmits the k-layer output value for communication to the communication network 5A, and the communication unit 274 of the intermediate server 2-2 transmits the k-layer output value for communication to the communication network 5A. Received from (S208).

Subsequently, the conversion unit 272 of the intermediate server 2-2 converts the communication k-th layer output value received in step S208 into the k-th layer output value (S210). Further, the processing unit 232 of the intermediate server 2-2 takes the k-th layer output value as an input and _{performs neural network processing using the k + 1 layer L k + 1} to the m-th layer L _m of the neural network NN2. The m-layer output value is output (S212).

Subsequently, the conversion unit 272 of the intermediate server 2-2 converts the m-th layer output value into the communication m-layer output value (S214). Further, the communication unit 274 of the intermediate server 2-2 transmits the communication m-layer output value to the communication network 5B, and the communication unit 374 of the recognition server 3-2 transmits the communication m-layer output value to the communication network 5B. Received from (S216).

Subsequently, the conversion unit 372 of the recognition server 3-2 converts the communication m-layer output value received in step S216 into the m-layer output value (S218). Further, the processing unit 332 of the recognition server 3-2 receives as input the m-th layer output value, the performing a neural network processing using the (m + 1) -th layer L _{m + 1} of the neural network NN2 to the n layer L _n The n-layer output value is output (S220).

(effect)
The second embodiment of the present invention has been described above. According to the present embodiment, the neural network processing using the image data acquired by the imaging of the surveillance camera 1-2 as the input is performed by the surveillance camera 1-2, the intermediate server 2-2, and the recognition server 3-2. It is shared by two devices. Further, as described above, the integer k indicating the processing sharing boundary between the surveillance camera 1-2 and the intermediate server 2-2 and the integer m indicating the processing sharing boundary between the intermediate server 2-2 and the recognition server 3-2 are each layer. Since it can be set independently of the processing related to, it is possible to divide the processing in finer units. Further, communication related to image data is not performed between the devices, and the k-th layer output value for communication or the m-th layer output value for communication is communicated. Depending on the configuration, the communication content between the surveillance camera 1-2 and the intermediate server 2-2 via the communication network 5A, or between the intermediate server 2-2 and the recognition server 3-2 via the communication network 5B. Even if the contents of the communication are stolen, privacy is unlikely to be infringed.

In the second embodiment, the example in which the neural network processing is shared by three devices has been described, but the present invention is not limited to this example, and the neural network processing is shared by four or more devices. You may.

<2-3. Third Embodiment>
(Overview)
In the first embodiment and the second embodiment described above, an example in which an integer k and an integer m indicating a processing sharing boundary between devices are preset has been described, but the processing sharing boundary is dynamically determined. Is also possible. Hereinafter, an example in which the processing sharing boundary is dynamically determined will be described as a third embodiment of the present invention. In the following, the surveillance system 900 according to the third embodiment of the present invention will be referred to as a surveillance system 900-3, and the surveillance camera 1, the intermediate server 2, and the recognition server 3 of the surveillance system 900-3 will be monitored, respectively. It is referred to as a camera 1-3, an intermediate server 2-3, and a recognition server 3-3.

FIG. 8 is an explanatory diagram for explaining the outline of the third embodiment of the present invention. FIG. 8 shows the neural network NN3 used by the monitoring system 900-3 according to the present embodiment. As shown in FIG. 8, the neural network NN3 is a neural network composed of n layers. In this embodiment, n is an integer of at least 3 or more.

Further, the neural network NN3 shown in FIG. 8 is a neural network in which the output value of each layer is input to the next (right) layer of the layer. The output value of each layer is not limited to the scalar value, and may be, for example, a vector value. Hereinafter, as shown in FIG. 8, the layers constituting the neural network NN3 according to the present embodiment are arranged in order from the left, the first layer L ₁ , the second layer L ₂ , the third layer L ₃ , ..., The nth layer. It is called _{L n.}

In the present embodiment, each layer included in the neural network NN3 is clearly distinguished from the face detection process and the face recognition process as shown in FIG. As shown in FIG. 8, the neural network NN3 is the first layer _{L 1} ~ q-th layer _{L q} corresponds to the face detection process, the q + 1 layer _{L q + 1} ~ n-th layer _{L n} corresponds to the face authentication process .. In other words, the q layer output value is an output value of the q layer L _q includes information about the face detection result. Here, q is an integer of 1 or more and n-1 or less.

Similar to the monitoring system 900-2 according to the second embodiment, the monitoring system 900-3 according to the present embodiment performs neural network processing using the neural network NN3 with the surveillance camera 1-3 and the intermediate server 2-3. , And the recognition server 3-3. In the example shown in FIG. 8, the monitoring cameras 1-3 responsible for neural network processing of the first layer _{L 1} ~ k-th layer _{L k.} Further, the intermediate server 2-3 is in charge of the neural network processing of the _{first k + 1 layer L k + 1} to the mth layer L _m. Further, the recognition server 3-3 is in charge of the neural network processing of _{the m + 1 layer L m + 1} to the nth layer L _n. Here, k is an integer of 1 or more and q-1 or less, and the surveillance cameras 1-3 and the intermediate server 2-3 (more accurately, the surveillance cameras 1-3 and the intermediate server 2-3 are described later). The processing sharing boundary (of each processing unit) is shown. Further, m is an integer of q + 1 or more and n-1 or less, and the intermediate server 2-3 and the recognition server 3-3 (more accurately, the intermediate server 2-3 and the recognition server 3-3, respectively, as described later) are used. It shows the processing sharing boundary (of the processing unit).

In the present embodiment, the integer k indicating the processing sharing boundary between the surveillance camera 1-3 and the intermediate server 2-3 and the integer m indicating the processing sharing boundary between the intermediate server 2-3 and the recognition server 3-3 change dynamically. Can be. The details of the method for determining the integer k and the integer m will be described later, but as described above, k is determined in the range of 1 or more and q-1 or less, and m is determined in the range of q + 1 or more and n-1 or less. According to such a constitution, the intermediate server 2-3 will always be possible to perform a neural network processing using the second q layer L _q. Therefore, the intermediate server 2-3 can obtain the qth layer output value including the information regarding the result of the face detection process using the image data (first input data) of the current frame as an input. Then, the intermediate server 2-3 inputs the resolution related to the imaging (sensing) of the image data (second input data) of the next frame and the image data of the next frame based on the qth layer output value. Determine the processing sharing boundary of the neural network processing. With such a configuration, it is possible to perform more accurate face recognition while sharing the processing so as to satisfy the required processing time, for example.

(Configuration example)
The outline of the monitoring system 900-3 according to the present embodiment has been described above. Subsequently, a configuration example of the monitoring system 900-3 according to the present embodiment will be described in more detail. FIG. 9 is a block diagram showing a configuration example of the monitoring system 900-3 according to the present embodiment. Since the monitoring system 900-3 according to the present embodiment has the same configuration as the monitoring system 900-1 according to the first embodiment and the monitoring system 900-2 according to the second embodiment in part. The explanation will be given while omitting as appropriate.

As shown in FIG. 9, the surveillance cameras 1-3 are information processing devices including an imaging unit 113, a processing unit 133, a storage unit 140, a processing control unit 163, and a communication interface unit 180. Since the function of the storage unit 140 shown in FIG. 9 is the same as the function of the storage unit 140 described with reference to FIG. 6, the description thereof will be omitted.

The imaging unit 113 is a camera module that acquires image data (an example of sensing data) by imaging (an example of sensing) in the same manner as the imaging unit 112 described with reference to FIG. However, the imaging unit 113 according to the present embodiment will be described with reference to FIG. 6 in that it performs imaging at a frame rate (time resolution) and a resolution (spatial resolution) in accordance with the control of the processing control unit 163 described later. It is different from the imaging unit 112.

The processing unit 133 performs neural network processing. The neural network processing performed by the processing unit 133 can be specified by, for example, the neural network parameters stored in the storage unit 140. Incidentally, the neural network parameters stored in the storage unit 140 may be a parameter corresponding to the entire neural network NN3, corresponding parameter from the first layer L ₁ to the q layer L _q of the neural network NN3 It may be.

Processor 133 functions as a first neural network processing unit in the present embodiment, an input image data provided from the imaging unit 113, first from the first layer L ₁ of the neural network NN3 shown in FIG. 8 performing a neural network processing using up to k layer L _k. The processing unit 133 functions as a first neural network processing unit in the present embodiment, and outputs the k-th layer output value is an output value of the k-th layer L _k to a communication interface unit 180.

As described above, the integer k indicating the processing sharing boundary of the processing unit 233 of the processing unit 133 and the intermediate server 2-3 described later can change dynamically. For example, processor 133 performs a neural network processing using the first layer L ₁ in accordance with the control of the processing control unit 163 to be described later up to the k layer L _k.

The processing control unit 163 controls the imaging unit 113 and the processing unit 133 based on the processing control information provided from the communication interface unit 180, which will be described later. The processing control information provided from the communication interface unit 180 to the processing control unit 163 includes, for example, frame rate (time resolution) and resolution (spatial resolution) information related to imaging (an example of sensing) of the imaging unit 113. You may be. Then, the processing control unit 163 may change the settings of the frame rate (time resolution) and the resolution (spatial resolution) of the imaging unit 113 based on the processing control information. Further, the processing control information provided from the communication interface unit 180 to the processing control unit 163 includes information (value) of an integer k indicating a processing sharing boundary of the processing unit 233 possessed by the processing unit 133 and the intermediate server 2-3 described later. May be included.

The communication interface unit 180 mediates communication between the surveillance cameras 1-3 and other devices. The communication interface unit 180 supports an arbitrary wireless communication protocol or a wired communication protocol, and establishes a communication connection with another device via the communication network 5A or directly. As shown in FIG. 9, the communication interface unit 180 includes a conversion unit 182 and a communication unit 184.

The conversion unit 182 converts the data into data (communication data) in a format that can be transmitted by the communication unit 184. For example, the conversion unit 178 functions as the first conversion unit in the present embodiment, converts the k-layer output value of the neural network NN3 output from the processing unit 133 into the k-layer output value for communication, and converts the k-layer output value for communication into the communication unit. Provide to 184.

Further, the conversion unit 182 converts (reversely transforms) the communication data received by the communication unit 184 into data for handling by the processing unit 133, the storage unit 140, and the processing control unit 163, and the processing unit 133 and the storage unit 140, It is provided to the processing control unit 163. For example, the conversion unit 182 converts the communication processing control information received from the communication network 5A by the communication unit 184 into the processing control information and provides it to the processing control unit 163.

The communication unit 184 transmits communication data to another device via the communication network 5A or directly, or receives communication data from the other device. For example, the communication unit 184 transmits the communication layer k layer output value provided by the conversion unit 182 to the communication network 5A (the first communication network in the present embodiment). The communication unit 184 receives the communication processing control information transmitted by the intermediate servers 2-3 from the communication network 5A.

As shown in FIG. 9, the intermediate server 2-3 is an information processing device including a processing unit 233, a storage unit 240, a determination unit 253, a processing control unit 263, and a communication interface unit 280. Since the function of the storage unit 240 shown in FIG. 9 is the same as the function of the storage unit 240 described with reference to FIG. 3, the description thereof will be omitted.

The processing unit 233 performs neural network processing. The neural network processing performed by the processing unit 233 can be specified by, for example, the neural network parameters stored in the storage unit 240. The neural network parameter stored in the storage unit 240 in the present embodiment may be a parameter corresponding to the entire neural network NN3.

The processing unit 233 functions as a second neural network processing unit in the present embodiment, receives the output value of the k-th layer as an input, and has the k + 1 layer L _{k + 1} to the m-th layer L _{m of the neural network NN3 shown in FIG.} Perform neural network processing using up to. The processing unit 233 functions as a second neural network processing unit in this embodiment, outputs an m-th layer output value is an output value of the m-th layer L _m to the communication interface unit 280.

As described above, the integer k indicating the processing sharing boundary of the processing unit 133 and the processing unit 233 of the surveillance camera 1-3, and the processing sharing boundary of the processing unit 332 of the processing unit 232 and the recognition server 3-1 described later are set. The indicated integer m can change dynamically. For example, the processing unit 233 performs neural network processing using the _{first k + 1 layer L k + 1} to the mth layer L _{m under} the control of the processing control unit 263 described later.

The determination unit 253 functions as a processing sharing determination unit, and sets a value of an integer k indicating the processing sharing boundary between the processing unit 133 and the processing unit 233 and a value of an integer m indicating the processing sharing boundary between the processing unit 232 and the processing unit 332. decide. For example, the determination unit 253 uses the image data of the next and subsequent frames (second input data) based on the qth layer output value in the neural network processing in which the image data of the current frame (first input data) is input. ) As an input, the value of the integer k and the value of the integer m in the neural network processing may be determined.

Further, the determination unit 253 functions as a resolution determination unit, and determines the resolution (frame rate and resolution) related to the imaging of the imaging unit 113 included in the surveillance cameras 1-3. For example, the determination unit 253 uses the image data of the next and subsequent frames (second input data) based on the qth layer output value in the neural network processing in which the image data of the current frame (first input data) is input. ) May be determined.

Here, as described above, the qth layer output value includes information indicating the result of face detection, and for example, the determination unit 253 determines whether or not the face is detected based on the qth layer output value. Is possible. Therefore, the determination unit 253 can determine the value of the integer k, the value of the integer m, and the above resolution based on the face detection result.

FIG. 10 is a table showing an example of determination of the integer k, the integer m, and the resolution (frame rate and resolution) by the determination unit 253. In FIG. 10, p, which is the value of k in the initial setting, is an integer of 1 or more and q or less, and r, which is the value of m in the initial setting, is an integer of q or more and n-1 or less. good.

As shown in FIG. 10, the determination unit 253 sets the resolution as a resolution determination unit so that the resolution (frame rate and resolution) is high so that the face recognition process is performed with higher accuracy at the time of face detection. You may. However, in such a case, since the overall processing load is also high, there is a possibility that the processing cannot be completed in the required processing time unless the processing division is changed. Therefore, when determining a higher value as the resolution, the determination unit 253 sets the value of k so that the processing load of the processing unit 133 and the processing unit 233, which has the lower processing performance, becomes smaller. You may decide.

For example, in the present embodiment, it is assumed that the processing unit 233 of the intermediate server 2-3 has higher processing performance than the processing unit 133 of the surveillance camera 1-3. Therefore, in the example shown in FIG. 10, the value of k is determined to be smaller than the default value of p at the time of face detection in which the frame rate and the resolution are determined to be high values. Similarly, the processing unit 333 of the recognition server 3-3 may have higher processing performance than the processing unit 233 of the intermediate server 2-3. Therefore, in the example shown in FIG. 10, the value of m is determined to be smaller than the initial setting r at the time of face detection in which the frame rate and the resolution are determined to be high values.

Further, as shown in FIG. 10, in order to reduce the processing load when the face is not detected, the determination unit 253 may set the resolution as the resolution determination unit so that the resolution (frame rate and resolution) is low. good. In such a case, the overall processing load is also low, so by changing the processing division and having the processing unit 133 share the processing as much as possible within the range that satisfies the required processing time, the processing unit 233 and the processing unit in the subsequent stage are used. The processing load of 333 can be further reduced. Therefore, in the example shown in FIG. 10, when the face is not detected and the frame rate and the resolution are determined to be low values, the value of k is determined to be equal to or more than the initial setting p and smaller than q. Further, in the example shown in FIG. 10, when the face is not detected and the frame rate and the resolution are determined to be low values, the value of m is determined to be equal to or more than the initial setting r and smaller than n.

The method of determining the value of the integer k, the value of the integer m, and the resolution by the determination unit 253 has been described above with reference to FIG. 10, but FIG. 10 shows only one example and is not limited to such an example. For example, the determination unit 253 may determine the value of the integer k, the value of the integer m, and the resolution in more steps according to the number of detected faces.

The determination unit 253 generates processing control information including the determined integer k value, integer m value, and resolution information, and provides the processing control information to the processing control unit 263 and the communication interface unit 280.

The explanation will be continued by returning to FIG. The processing control unit 263 controls the processing unit 233 based on the value of the integer k and the value of the integer m included in the processing control information provided by the determination unit 253. The processing control information provided from the determination unit 253 to the processing control unit 263 may include information (value) of an integer k and an integer m.

The communication interface unit 280 mediates communication with other devices by the intermediate server 2-3, similarly to the communication interface unit 220 described with reference to FIG. The communication interface unit 280 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with or directly from the communication network 5A or the communication network 5B. The communication interface unit 280 includes a conversion unit 282 and a communication unit 284 as shown in FIG.

The conversion unit 282 converts (reversely converts) the communication data received by the communication unit 284 into data for handling by the processing unit 233 and the storage unit 240, and provides the data to the processing unit 233 and the storage unit 240. For example, the conversion unit 282 functions as a second conversion unit in the present embodiment, converts the communication k-layer output value received from the communication network 5A by the communication unit 284 into the k-layer output value, and processes the processing unit 233. To provide to.

Further, the conversion unit 282 converts the data into data (communication data) in a format that can be transmitted by the communication unit 284. For example, the conversion unit 282 functions as a third conversion unit in the present embodiment, converts the mth layer output value of the neural network NN3 output from the processing unit 233 into the communication mth layer output value, and communicates unit. Provide to 284. Further, the conversion unit 282 according to the present embodiment converts the processing control information provided by the determination unit 253 into communication processing control information and provides it to the communication unit 284.

The communication unit 284 transmits communication data to another device, or receives communication data from another device, via the communication network 5A, via the communication network 5B, or directly. For example, the communication unit 284 functions as the first reception unit in the present embodiment, and receives the communication k-layer output value transmitted by the surveillance camera 1-1 from the communication network 5A. Further, the communication unit 284 functions as a second transmission unit in the present embodiment, and transfers the communication m-layer output value provided by the conversion unit 282 to the communication network 5B (second communication network in the present embodiment). Send. Further, the communication unit 284 according to the present embodiment transmits the communication processing control information provided by the conversion unit 282 to the communication network 5A and the communication network 5B.

As shown in FIG. 6, the recognition server 3-3 is an information processing device including a processing unit 333, a storage unit 340, and a communication interface unit 380. Since the function of the storage unit 340 shown in FIG. 9 is the same as the function of the storage unit 340 described with reference to FIG. 3, the description thereof will be omitted.

The processing unit 333 performs neural network processing. The neural network processing performed by the processing unit 333 can be specified by, for example, the neural network parameters stored in the storage unit 340. Incidentally, the neural network parameters storage unit 340 stores in this embodiment may be a parameter corresponding to the entire neural network NN3, from the q + 1 layer L _{q + 1} of the neural network NN3 to the n layer L _n It may be the corresponding parameter.

The processing unit 333 functions as a third neural network processing unit in the present embodiment, receives the output value of the mth layer as an input, and has m + 1 layer L _{m + 1} to nth layer L _{n of the neural network NN3 shown in FIG.} Perform neural network processing using up to. The n layer output value is an output value of the n-th layer L _n which is output by the processing unit 333 may be stored in the storage unit 340, may be displayed on the display unit (not shown), conversion unit After being converted into communication data by 382, it may be transmitted to another device by the communication unit 384.

As described above, the integer m indicating the processing sharing boundary between the processing unit 232 and the processing unit 332 of the intermediate server 2-3 can change dynamically. For example, the processing unit 333 performs neural network processing using _{the m + 1 layer L m + 1} to the nth layer L _n according to the control of the processing control unit 363 described later.

The processing control unit 363 controls the processing unit 333 based on the processing control information provided from the communication interface unit 380 described later. The processing control information provided from the communication interface unit 380 to the processing control unit 363 may include information (value) of an integer m indicating the processing unit 233 and the processing sharing boundary of the processing unit 333.

The communication interface unit 380 mediates communication between the recognition server 3-3 and other devices. The communication interface unit 380 supports any wireless communication protocol or wired communication protocol, and establishes a communication connection with another device via the communication network 5B or directly. The communication interface unit 380 includes a conversion unit 382 and a communication unit 384 as shown in FIG.

The conversion unit 382 converts (reversely converts) the communication data received by the communication unit 384 into data for handling by the processing unit 333 and the storage unit 340, and provides the data to the processing unit 333 and the storage unit 340. For example, the conversion unit 382 functions as a fourth conversion unit in the present embodiment, and the communication unit 384 uses the communication m-layer output value received from the communication network 5B (second communication network in the present embodiment) as the first. It is converted into an m-layer output value and provided to the processing unit 333. Further, the conversion unit 382 converts the communication processing control information received from the communication network 5B by the communication unit 384 into the processing control information and provides it to the processing control unit 363.

The communication unit 384 transmits communication data to another device via the communication network 5B or directly, or receives communication data from the other device. For example, the communication unit 384 functions as a second reception unit in the present embodiment, and receives the communication m-layer output value transmitted by the intermediate server 2-3 from the communication network 5B. Further, the communication unit 384 receives the communication processing control information transmitted by the intermediate servers 2-3 from the communication network 5B.

(Operation example)
The configuration example of the monitoring system 900-3 according to the third embodiment of the present invention has been described above. Subsequently, an operation example of the present embodiment will be described with reference to FIG. FIG. 11 is a sequence diagram showing a processing flow of the monitoring system 900-3 according to the present embodiment.

As shown in FIG. 11, first, the surveillance cameras 1-3 acquire image data by imaging the imaging unit 113 (S302). Subsequently, the processing unit 133 of the monitoring camera 1-3, and inputs the image data, the k-th layer by performing a neural network processing using the first layer L ₁ to the k-th layer L _k of the neural network NN3 The output value is output (S304).

Subsequently, the conversion unit 182 of the surveillance cameras 1-3 converts the k-th layer output value into the communication k-th layer output value (S206). Further, the communication unit 184 of the surveillance camera 1-3 transmits the k-layer output value for communication to the communication network 5A, and the communication unit 284 of the intermediate server 2-3 transmits the k-layer output value for communication to the communication network 5A. Received from (S308).

Subsequently, the conversion unit 282 of the intermediate server 2-3 converts the communication k-th layer output value received in step S308 into the k-th layer output value (S310). Further, the processing unit 233 of the intermediate server 2-3 uses the k-th layer output value as an input and performs neural network processing using _{the k + 1 layer L k + 1} to the q-th layer L _{q of the neural network NN3.} The q-layer output value is output (S314).

Subsequently, the determination unit 253 of the intermediate server 2-3 determines the integer k value, the integer m value, and the resolution based on the qth layer output value indicating the face detection result, and determines the integer k value and the integer m. Process control information including the value of and the resolution information is generated (S314). Further, the conversion unit 282 of the intermediate server 2-3 converts the processing control information into the communication processing control information (S316). Then, the communication unit 274 of the intermediate server 2-2 transmits the communication processing control information to the communication network 5A, and the communication unit 184 of the surveillance camera 1-3 receives the communication processing control information from the communication network 5A ( S318).

Subsequently, the conversion unit 182 of the surveillance cameras 1-3 converts the communication processing control information into the processing control information (S320). Further, the processing control unit 163 of the surveillance cameras 1-3 changes the resolution setting related to the image acquisition (imaging) of the imaging unit 113 based on the processing control information (S322).

Subsequently, the processing unit 233 of the intermediate server 2-3 _{performs neural network processing using the q + 1 layer L q + 1} to the m layer L _{m of} the neural network NN3, and outputs the m layer output value (S324). ).

Further, the conversion unit 282 of the intermediate server 2-3 converts the m-th layer output value into the communication m-layer output value (S326). Further, the communication unit 284 of the intermediate server 2-3 transmits the communication processing control information and the communication mth layer output value to the communication network 5B, and the communication unit 384 of the recognition server 3-3 transmits the communication processing control information. And the m-layer output value for communication are received from the communication network 5B (S328).

Subsequently, the conversion unit 382 of the recognition server 3-3 converts the communication processing control information and the communication mth layer output value received in step S328 into the processing control information and the communication mth layer output value, respectively (S330). .. Further, the processing unit 333 of the recognition server 3-3 receives as input the m-th layer output value, the performing a neural network processing using the (m + 1) -th layer L _{m + 1} of the neural network NN3 to the n layer L _n The n-layer output value is output (S332).

The above-mentioned processes of steps S302 to S332 may be repeated as appropriate or as necessary. Then, the imaging of the next frame is performed with the resolution set in step S322, and the neural network processing using the image data of the next frame as an input is shared based on the new processing sharing boundary determined in step S314. It is controlled to be done.

(effect)
The third embodiment of the present invention has been described above. According to the present embodiment, the neural network processing using the image data acquired by the imaging of the surveillance cameras 1-3 as the input is performed by the surveillance cameras 1-3, the intermediate server 2-3, and the recognition server 3-3. It is shared by two devices. Further, as described above, the integer k indicating the processing sharing boundary between the surveillance camera 1-3 and the intermediate server 2-3 and the integer m indicating the processing sharing boundary between the intermediate server 2-3 and the recognition server 3-3 are dynamic. Therefore, it is possible to divide the processing more flexibly. Further, as described above, by controlling the resolution related to the imaging by the surveillance cameras 1-3, it is possible to perform face recognition with higher accuracy.

<< 3. Modification example >>
The first embodiment, the second embodiment, and the third embodiment of the present invention have been described above. Hereinafter, some modifications of each of the above embodiments will be described. In addition, each modification described below may be applied to each embodiment individually, or may be applied to each embodiment in combination. Further, each modification may be applied in place of the configuration described in each embodiment, or may be additionally applied to the configuration described in each embodiment.

<3-1. Modification 1>
In the above embodiment, an example in which the surveillance system 900 includes one surveillance camera 1, one intermediate server 2, and one recognition server 3 has been described, but the present technology is not limited to such an example. The surveillance system 900 may include a plurality of each device. For example, a plurality of intermediate servers 2 may correspond to one recognition server 3, or a plurality of surveillance cameras may correspond to one intermediate server 2. 1 may correspond. In the case of such a configuration, in the third embodiment described above, it is possible to reduce the processing load of the intermediate server 2 and the recognition server 3 when the face is not detected, so that the processing resources can be efficiently used.

<3-2. Modification 2>
In the above embodiment, a neural network process using a neural network in which the output value of each layer is input to the layer having the next number (order) of the layer is described as an example. Not limited. For example, the present technology is also applicable to neural network processing using a neural network in which the output of a plurality of layers is input to one layer or the output of one layer is input to a plurality of layers. In such a case, the number may be set so that the number of the layer on the side where the output value is input (receiver) is larger than the number of the layer on the side where the output value is output.

<3-3. Modification 3>
In the above embodiment, an example in which neural network parameters are stored in advance in the storage unit 140, the storage unit 240, and the storage unit 340 has been described, but the present technology is not limited to such an example. For example, the neural network parameter may be stored only in the storage unit of one device, and the other device may receive the neural network parameter from the one device.

<3-4. Modification 4>
In the third embodiment, an example in which the integer k and the integer m indicating the processing sharing boundary are determined based on the output value of the qth layer indicating the face detection result has been described, but the present technology is not limited to such an example. .. For example, the determination unit 253 may determine the processing sharing boundary in place of or in addition to the qth layer output value, based on the communication status, the current resource status of each processing unit, and the like.

<< 4. Hardware configuration example >>
Each embodiment of the present invention has been described above. Information processing such as the above-mentioned neural network processing, sharing determination processing, and resolution determination processing is realized by the cooperation between the software and the hardware of the surveillance camera 1, the intermediate server 2, and the recognition server 3. Hereinafter, the hardware configuration of the information processing device 1000 will be described as an example of the hardware configuration of the monitoring camera 1, the intermediate server 2, and the recognition server 3, which are the information processing devices according to the embodiment of the present invention.

FIG. 12 is an explanatory diagram showing a hardware configuration of the information processing apparatus 1000 according to the embodiment of the present invention. As shown in FIG. 12, the information processing apparatus 1000 includes a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, an input device 1004, and an output device 1005. , A storage device 1006 and a communication device 1007.

The CPU 1001 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing device 1000 according to various programs. Further, the CPU 1001 may be a microprocessor. The ROM 1002 stores programs, calculation parameters, and the like used by the CPU 1001. The RAM 1003 temporarily stores a program used in the execution of the CPU 1001 and parameters that are appropriately changed in the execution. These are connected to each other by a host bus composed of a CPU bus or the like. Mainly, by collaboration between CPU 1001, ROM 1002 and RAM 1003 and software, for example, processing units 132, 133, 231, 232, 233, 331, 332, 333, determination unit 253, processing control units 163, 263, 363, etc. The function is realized.

The input device 1004 includes input means for the user to input information such as a mouse, keyboard, touch panel, buttons, microphone, switch, and lever, and an input control circuit that generates an input signal based on the input by the user and outputs the input signal to the CPU 1001. It is composed of such as. By operating the input device 1004, the user of the information processing device 1000 can input various data to the information processing device 1000 and instruct the processing operation.

The output device 1005 includes, for example, a liquid crystal display (LCD) device, an OLED device, and a display device such as a lamp. Further, the output device 1005 includes an audio output device such as a speaker and headphones. For example, the display device displays an captured image, a generated image, or the like. On the other hand, the voice output device converts voice data and the like into voice and outputs the data.

The storage device 1006 is a device for storing data. The storage device 1006 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting the data recorded on the storage medium, and the like. The storage device 1006 stores a program executed by the CPU 1001 and various data. The storage device 1006 corresponds to the storage units 140, 240, and 340.

The communication device 1007 is a communication interface composed of, for example, a communication device for connecting to a communication network. Further, the communication device 1007 may include a wireless LAN (Local Area Network) compatible communication device, an LTE (Long Term Evolution) compatible communication device, a wire communication device that performs wired communication, or a Bluetooth (registered trademark) communication device. The communication device 1007 corresponds to the communication interface units 120, 170, 180, 220, 270, 280, 320, 370, 380.

<< 5. Conclusion >>
As described above, according to the embodiment of the present invention, it is possible to divide the processing into finer units. Therefore, it becomes easier to satisfy the required processing time, and for example, it becomes possible to more easily maintain the real-time performance of the processing.

Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various modifications or modifications within the scope of the technical ideas described in the claims. , These are also naturally understood to belong to the technical scope of the present invention.

For example, in the above embodiment, the example in which the monitoring system 900 has the communication network 5A and the communication network 5B has been described, but the present invention is not limited to such an example. For example, the communication network 5A and the communication network 5B may be the same communication network.

Further, in the sequence diagram in the above embodiment, a series of processes was started from the surveillance camera 1, but the recognition server 3 accesses the surveillance camera 1 to start the process, and the series of processes is started. May be good.

Further, each step in the above embodiment does not necessarily have to be processed in chronological order in the order described as a sequence diagram. For example, each step in the processing of the above embodiment may be processed in an order different from the order described as the sequence diagram, or may be processed in parallel.

Further, according to the above embodiment, a computer program for exerting the same functions as the above-mentioned configurations of the surveillance camera 1, the intermediate server 2, and the recognition server 3 on the hardware such as the CPU 1001, the ROM 1002, and the RAM 1003 is also provided. It is possible. Also provided is a recording medium on which the computer program is recorded.

1 Surveillance camera 2 Intermediate server 3 Recognition server 5 Communication network 111 Imaging unit 120 Communication interface unit 122 Conversion unit 124 Communication unit 132 Processing unit 140 Storage unit 163 Processing control unit 220 Communication interface unit 222 Conversion unit 224 Communication unit 231 Processing unit 240 Storage unit Unit 253 Decision unit 263 Processing control unit 320 Communication interface unit 322 Conversion unit 324 Communication unit 331 Processing unit 340 Storage unit 363 Processing control unit 900 Monitoring system

Claims (10)

An information processing system that performs neural network processing using a neural network composed of at least n layers, where n is an integer of 2 or more.
When k is an integer of 1 or more and n-1 or less, the input data is used as an input, and the neural network processing using the first layer to the kth layer of the neural network is performed to obtain the kth layer output value. The first neural network processing unit to output and
A first conversion unit that converts the k-th layer output value into a communication k-th layer output value, and
A first transmitter that transmits the communication k-layer output value to the first communication network, and
A first receiving unit that receives the k-layer output value for communication from the first communication network, and
A second conversion unit that converts the communication k-layer output value received by the first reception unit into the k-layer output value, and
A second neural network processing unit that uses the k-th layer output value as an input and performs the neural network processing using at least the k + 1th layer of the neural network.
When q is an integer of 1 or more and n or less, the first layer is based on the output value of the qth layer output by the qth layer of the neural network in the neural network process using the first input data as an input. A process sharing determination unit that determines the value of k in the neural network process that uses a second input data different from the input data of
Information processing system. Before Symbol processing sharing determining unit,
It said first and neural network processing unit that determine the value of the k indicating the processing sharing boundaries with the second neural network processing unit, the information processing system according to claim 1. The first input data and the second input data are sensing data acquired by sensing, and are
The information according to claim 1 or 2 , wherein the information processing system further includes a resolution determining unit that determines the resolution related to the sensing for acquiring the second input data based on the qth layer output value. Processing system. The processing sharing determination unit has lower processing performance than the first neural network processing unit and the second neural network processing unit when the resolution determination unit determines a higher value as the resolution. The information processing system according to claim 3 , wherein the value of k is determined so that the processing load on the side becomes smaller. The first input data and the second input data are image data, and are
The neural network is a neural network for recognizing an object included in the image data.
The information processing system according to any one of claims 1 to 4 , wherein the q-th layer output value includes information on a detection result of the object. The n is an integer of 3 or more.
When m is an integer of k + 1 or more and n-1 or less,
The second neural network processing unit performs the neural network processing using the k + 1 layer to the mth layer of the neural network, outputs the mth layer output value, and outputs the mth layer output value.
The information processing system
A third conversion unit that converts the m-layer output value into a communication m-layer output value, and
A second transmitter that transmits the communication m-layer output value to the second communication network, and
A second receiving unit that receives the communication m-layer output value from the second communication network, and
A fourth conversion unit that converts the communication m-layer output value received by the second receiving unit into the m-layer output value, and
A third neural network processing unit that uses the mth layer output value as an input and performs the neural network processing using at least the m + 1th layer of the neural network.
The information processing system according to any one of claims 1 to 5, further comprising. When n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, neural network processing using the first layer to the kth layer of the neural network composed of at least n layers is performed. , The first neural network processing unit that outputs the k-th layer output value, and
A first conversion unit that converts the k-th layer output value into a communication k-th layer output value, and
A first transmitter that transmits the communication k-layer output value to the first communication network, and
When q is an integer of 1 or more and n or less, the first layer is based on the output value of the qth layer output by the qth layer of the neural network in the neural network process using the first input data as an input. A process sharing determination unit that determines the value of k in the neural network process that uses a second input data different from the input data of
Information processing device. When n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, neural network processing using the first layer to the kth layer of the neural network composed of at least n layers is performed. , The function to output the k-th layer output value, and
A function to convert the k-th layer output value into a communication k-th layer output value,
A function of transmitting the k-layer output value for communication to the first communication network, and
When q is an integer of 1 or more and n or less, the first layer is based on the output value of the qth layer output by the qth layer of the neural network in the neural network process using the first input data as an input. A function of determining the value of k in the neural network process using a second input data different from the input data of
A program to realize the above on a computer. When n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, neural network processing using the first layer to the kth layer of the neural network composed of at least n layers is performed. A first receiving unit that receives the k-layer output value for communication obtained by converting the output k-layer output value from the first communication network, and
A second conversion unit that converts the communication k-layer output value received by the first reception unit into the k-layer output value, and
A second neural network processing unit that uses the k-th layer output value as an input and performs the neural network processing using at least the k + 1th layer of the neural network.
Have a,
When q is an integer of 1 or more and n or less, the first layer is based on the output value of the qth layer output by the qth layer of the neural network in the neural network process using the first input data as an input. An information processing system in which the value of k in the neural network processing in which a second input data different from the input data of the above is input is determined. When n is an integer of 2 or more and k is an integer of 1 or more and n-1 or less, neural network processing using the first layer to the kth layer of the neural network composed of at least n layers is performed. A function to receive the k-layer output value for communication obtained by converting the output k-layer output value from the first communication network, and
A function of converting the k-th layer output value for communication into the k-th layer output value,
A function of using the k-th layer output value as an input and performing the neural network processing using at least the k + 1th layer of the neural network, and
A program for realizing the to the computer,
When q is an integer of 1 or more and n or less, the first layer is based on the output value of the qth layer output by the qth layer of the neural network in the neural network process using the first input data as an input. A program in which the value of k in the neural network process using a second input data different from the input data of the above is determined.

Images