JP2020109551A - Information processing device and information processing system - Google Patents

Abstract

To provide an information processing device and an information processing system which can increase processing efficiency of a plurality of processors.SOLUTION: An information processing device is disclosed in the present application. In one aspect, the information processing device comprises: a receiving unit which receives resource information about processing state from each of N processors where N is an integer greater than or equal to two; a calculation unit which fits the received resource information of the N processors to a computational expression to calculate a processing time for a processing request by an application for each of the N processors; a selection unit which selects one processor from the N processors on the basis of the processing time of the N processors; and a transmission unit which transmits the processing request to the one processor.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device and an information processing system.

In an information processing apparatus in which a plurality of processors are communicably connected, parallel distributed control may be performed in which processing is distributed to the plurality of processors.

JP, 2017-37533, A

However, if parallel distributed control is fixedly performed in the information processing apparatus, the processing efficiency of the plurality of processors may be reduced.

The disclosed technology has been made in view of the above, and an object thereof is to provide an information processing apparatus and an information processing system that can improve the processing efficiency of a plurality of processors.

In one aspect, the information processing apparatus disclosed in the present application, when N is an integer of 2 or more, a receiving unit that receives resource information regarding a processing status from each of N processors, and the received N pieces of resource information. A calculation unit that applies the resource information of the processor to a calculation formula to calculate the processing time for the processing request by the application for each of the N processors, and the N calculation units based on the processing times of the N processors. It has a selection part which selects one processor from a processor, and a transmission part which transmits the processing request to the one processor.

In one aspect, an information processing system disclosed in the present application includes the above information processing device and a plurality of processors, each of which is communicably connected to the information processing device.

According to one aspect of the information processing device disclosed in the present application, the processing efficiency of the plurality of processors can be improved.

FIG. 1 is a diagram showing a hardware configuration of an information processing system according to the embodiment. FIG. 2 is a diagram illustrating a functional configuration of the information processing device according to the embodiment. FIG. 3 is a diagram illustrating a functional configuration of the inference processing device according to the embodiment. FIG. 4 is a diagram showing a data structure of the discrimination parameter information in the embodiment. FIG. 5 is a diagram showing an update process (in the case of coprocessor A and process J) of the discrimination parameter information in the embodiment. FIG. 6 is a diagram showing an update process (in the case of coprocessor A and process K) of the discrimination parameter information in the embodiment. FIG. 7 is a diagram showing an update process (in the case of coprocessor B and process J) of the discrimination parameter information in the embodiment. FIG. 8 is a diagram showing an operation using a model file of the inference processing device in the embodiment. FIG. 9 is a diagram showing a conversion process from the common format to the format of the input layer of the model file in the embodiment. FIG. 10 is a diagram illustrating conversion processing from the format of the output layer of the model file to the common format in the embodiment. FIG. 11 is a diagram showing an operation using another model file of the inference processing device in the embodiment. FIG. 12 is a diagram showing a conversion process from the common format to the format of the input layer of another model file in the embodiment. FIG. 13 is a diagram showing a conversion process from another model file output layer format to a common format in the embodiment. FIG. 14 is a diagram illustrating a functional configuration of the inference processing device according to the modified example of the embodiment. FIG. 15 is a diagram illustrating a hardware configuration of an information processing system according to another modification of the embodiment. FIG. 16 is a diagram showing a software configuration of an information processing system in another modification of the embodiment. FIG. 17 is a diagram illustrating a hardware configuration of a PCIe bridge controller according to another modification of the embodiment. FIG. 18 is a diagram illustrating a PCIe layer configuration according to another modification of the embodiment. FIG. 19 is a diagram showing how another processor looks from the coprocessor G in another modification of the embodiment. FIG. 20 is a diagram showing how another processor looks from the coprocessor D in another modification of the embodiment. FIG. 21 is a diagram illustrating a data transfer process between processors via a PCIe bridge controller according to another modification of the embodiment.

Hereinafter, an embodiment of an information processing system disclosed in the present application will be described in detail with reference to the drawings. The disclosed technology is not limited to this embodiment. Further, in the embodiments, configurations having the same function are denoted by the same reference numerals, and overlapping description will be omitted.

(Embodiment)
The information processing system according to the embodiment is a parallel distributed system having a main processor and a plurality of coprocessors, and parallel distributed control in which processing is distributed to the plurality of coprocessors is performed on the main processor side. In the information processing system, if the parallel distributed control is fixedly performed on the main processor side, the processing efficiency of the plurality of coprocessors may decrease.

For example, in consideration of the difference in the arithmetic processing capacities of a plurality of coprocessors, control that allocates a processing with a large load to a coprocessor with a high processing capacity and allocates a processing with a small load to a coprocessor with a low processing capacity can be considered. .. In this control, when coprocessors that are high-speed for a specific process are mixed in the parallel distributed system, it may be appropriate (for example, the optimum coprocessor depending on the current availability of resources in the coprocessors). ) It may not be parallel distributed control.

When each coprocessor (each inference processing device) has a learned model generated by machine learning in advance, each inference processing device may perform various inference processes using the learned model. At this time, if the same learned model is used for various inference processes in the inference processing device, the inference process may not be performed efficiently.

Therefore, in the present embodiment, in the information processing system, the resource information collected from the plurality of coprocessors is applied to the calculation formulas corresponding to the number of coprocessors to calculate the processing time, and the obtained coprocessors are processed. We aim to improve the efficiency of parallel distributed control in real time by selecting a coprocessor to allocate inference processing based on processing time and sending an inference request.

Further, the inference processing identifier for identifying the inference processing is included in the inference request so that each coprocessor (each inference processing device) corresponds to the inference processing identifier included in the received inference request among the plurality of model files. The inference processing in each coprocessor (each inference processing device) is made efficient by specifying the model file to be read and executing the inference processing.

Specifically, in the information processing system, a plurality of calculation formulas associated with combinations of processing contents and coprocessors are prepared. Each calculation formula includes a plurality of parameters (a plurality of discrimination parameters) corresponding to a plurality of coprocessors, and is configured to apply (substitute) the resource information of the plurality of coprocessors to obtain the processing time. The information processing system applies the resource information collected from the plurality of coprocessors to the calculation formulas corresponding to the number of coprocessors corresponding to the inference request by the application among the prepared formulas to reduce the processing time. calculate. The information processing system selects a coprocessor to which processing is to be allocated and transmits an inference request based on the processing times of the plurality of coprocessors obtained by calculation. For example, the information processing system can select the coprocessor with the shortest processing time and send the inference request. As a result, since the processing content can be considered as a performance difference in addition to the processing capacity, the parallel distributed control can be made highly accurate, and the current availability of resources can be considered, so that the parallel distributed control can be made efficient in real time. ..

Further, the information processing system transmits an inference request including the inference processing identifier and the input data to the selected coprocessor (each inference processing device). The plurality of coprocessors (the plurality of inference processing devices) each have a plurality of model files corresponding to different inference processes. Each of the plurality of model files corresponds to a learned inference model in which machine learning has been performed so as to be suitable for different inference processes. The coprocessor (inference processing device) that has received the inference request identifies the model file corresponding to the inference processing identifier included in the inference request, reads the identified model file, and executes the inference processing. As a result, compared to the case where the same learned model is used for various inference processes, a model file that has been made efficient by machine learning can be used for each inference process, so that each coprocessor (each inference process) can be used. The reasoning process in the device can be made efficient.

Specifically, the information processing system 1 can be configured as shown in FIGS. 1 and 2. FIG. 1 is a diagram showing a hardware configuration of the information processing system 1. FIG. 2 is a diagram showing a functional configuration of the information processing device 100.

When N is an arbitrary integer of 2 or more, the information processing system 1 includes an information processing device 100, a relay device 200, and N inference processing devices 300-1 to 300-N.

The information processing apparatus 100 has, as a hardware configuration, a motherboard 101, a main processor 102, a display (Display) 103, a USB (Universal Serial Bus) interface 104, and an Ethernet (Ethernet) interface 105 as illustrated in FIG. 1. , A DIMM (Dual Inline Memory Module) 106, an SSD (Solid State Drive) 107, an HDD (Hard Disk Drive) 108, and a TPM (Trusted Platform Module) 109.

The mother board 101 shown in FIG. 1 is a board on which components having the main function of the information processing apparatus 100 are mounted. The main processor 102 is a processor having a main function of the information processing apparatus 100, and can adopt an electronic circuit such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The display 103 functions as a display unit that displays various kinds of information. A USB device can be connected to the USB interface 104, and communication between the USB device and the main processor 102 can be performed. An Ethernet cable can be connected to the Ethernet interface 105, and communication between an external device and the main processor 102 via the Ethernet cable can be mediated. The DIMM 107 is a volatile storage medium such as a RAM (Random Access Memory) capable of temporarily storing various information. The SSD 107 and the HDD 108 are non-volatile storage media that can store various types of information even after power is turned off. The TPM 109 is a module that realizes the security function of the system.

In addition, as shown in FIG. 2, the information processing apparatus 100 has, as a functional configuration, upper applications 110-1 to 110-n (n is an integer of 2 or more), an API (Application Programming Interface) 120, and an AI cluster. It has a management unit 130. Each functional configuration shown in FIG. 2 can be functionally configured in the main processor 102.

The relay device 200 illustrated in FIG. 1 includes a bridge board 201 and a bridge controller 202. The bridge board 201 is a board on which the bridge controller 202 is mounted. The bridge controller 202 bridges a plurality of inference processing devices 300 to the information processing device 100, and mediates (relays) communication between the information processing device 100 and the plurality of inference processing devices 300.

The plurality of inference processing devices 300-1 to 300-N are connected to the relay device 200 in parallel with each other. Each of the inference processing devices 300-1 to 300-N has conversion boards (conv. boards) 301-1 to 301-N and coprocessors 302-1 to 302-N. The conversion board 301 is also called an accelerator board, and is a board on which hardware to be additionally used to increase the processing capability of the information processing system 1 is mounted.

The coprocessor 302 is a processor suitable for parallel arithmetic processing such as AI (Artificial Intelligence) inference processing and image processing, and may be an accelerator such as a GPU (Graphics Processing Unit) or a dedicated chip. The coprocessor 302 may also be a combination of CPU and GPU.

The AI inference process is an inference process using artificial intelligence (AI) and includes an inference process using an inference model using a neural network having a multilayer structure (hierarchical neural network). Each processor 302 can perform a machine learning on an inference model using a hierarchical neural network to generate a learned inference model and can utilize the learned inference model.

If the same learned model is used for various inference processes, the model file 304 as the learned inference model may not perform the inference process efficiently.

For example, in inference processing by machine learning, a system that combines a plurality of inference processing that extracts a plurality of focused data from large data and further digs the plurality of focused data may be required. As an example, it can be considered to realize it by the following two methods (a) and (b).
(A) A high-performance inference processing device executes a plurality of processes.
(B) In a plurality of inference processing devices, the inference process performed by each device is fixed and executed.

For example, in the case of a system in which a person is detected from a camera image and the number of persons by gender is counted, the following processes (1) and (2) are performed.
(1) A person extraction inference process is executed from a camera image.
(2) As a result of (1), the following inference processes (A) and (A) are executed for each extracted person.
(A) Execute the gender inference process.
(A) Count up the number of men and women according to the judgment result of (a).

When this processing is applied to the above-mentioned patterns (a) and (b), the following (a′) and (b′) are obtained.
(A') In the high-performance inference processing apparatus, all the person extraction inference processing and the male/female determination inference processing are developed into an executable state, and the processing of (1) and (2) is executed.
(B') When it is executed by a plurality of inference processing devices, one person extraction inference process is executed, and a plurality of extracted male and female judgment inference processes are executed by a plurality of people.

Inference processing to be executed is fixed regardless of whether it is executed by a high-performance inference processing device or a plurality of inference processing devices, and the performance of the inference processing device cannot be utilized efficiently. there is a possibility.

In consideration of this point, the contents of the inference processing are classified and categorized to some extent, and in each of the inference processing devices 300-1 to 300-N, a plurality of model files 304 corresponding to different inference processing are prepared and the inference to be executed Allow the process to be specified by the inference request identifier.

That is, when M is an arbitrary integer of 2 or more, each inference processing device 300-1 to 300-N has a host OS (Operating System) 307, a driver 306, middleware 305, and M units, as shown in FIG. Model files 304-1 to 304-M and an inference application 303. FIG. 3 is a diagram showing a functional configuration of the inference processing device 300.

The M model files 304-1 to 304-M correspond to different inference processes. The middleware 305 can execute each of the M model files 304-1 to 304-M. The inference application 303 receives an inference request including an inference processing identifier, input data, and a parameter from the outside, and specifies a model file corresponding to the inference processing identifier among the M model files 304-1 to 304-M. The middleware 305 reads the model file 304 specified by the inference application 303 and executes inference processing.

For example, in the initial state, the model files 304-1 to 304-M including the inference model using the hierarchical neural network and the middleware 305 that should execute the inference process are the information processing device 100 shown in FIG. Stored in the SSD 107 and/or the HDD 108.

The hierarchical neural network has a hierarchical structure and may have a plurality of intermediate layers between an input layer and an output layer. The plurality of intermediate layers include, for example, a convolutional layer, an activation function layer, a pooling layer, a fully connected layer, and a softmax layer. In the convolutional layer, the convolutional operation (convolution processing) is performed on the neuron data input by the input layer, and the features of the input neuron data are extracted. In the activation function layer, the features extracted in the convolutional layer are emphasized. In the pooling layer, the input neuron data is thinned out. In the fully connected layer, the extracted features are combined to generate a variable indicating the feature. The softmax layer transforms the variables generated in the fully connected layer into probabilities. The neuron data obtained as a result of the calculation by the softmax layer is output to the output layer, and the output layer performs a predetermined process (for example, image identification). The number and location of each layer may change at any time depending on the required architecture. That is, the hierarchical structure of the neural network and the configuration of each layer can be predetermined by the designer according to the object to be identified.

That is, the model files 304-1 to 304-M as the inference model include definition information and weight information for the inference model (hierarchical neural network). The definition information is data that stores information about the neural network. For example, the definition information stores information indicating the configuration of the neural network such as the hierarchical structure of the neural network, the configuration of units in each layer, and the connection relationship of the units. When recognizing an image, the definition information stores, for example, information indicating the configuration of a convolutional neural network determined by a designer or the like. The weight information is data in which weight values such as weight values used in the calculation of each layer of the neural network are stored. The value of the weight stored in the weight information is set to a predetermined initial value in the initial state, and is updated according to learning.

When the information processing system 1 is started, the model files 304-1 to 304-M and the middleware 305 are read from the SSD 107 and/or the HDD 108 by the main processor 102, and a predetermined inference processing device 300 (via the bridge controller 202). It is loaded into a given coprocessor 302). The predetermined inference processing device 300 (predetermined coprocessor 302) performs machine learning of the model files 304-1 to 304-M by the middleware 305. The middleware 305 performs machine learning suitable for the corresponding inference processing on each of the model files 304-1 to 304-M corresponding to different inference processing.

This machine learning can be machine learning using a multilayered neural network, and is also called deep learning. Deep learning is advancing in many fields as the number of layers of neural networks is increasing. For example, deep learning exhibits recognition accuracy as high as humans in image/sound recognition.

In deep learning, the features of the identification target are automatically learned in the neural network by performing supervised learning on the identification target. In deep learning, a target to be identified is identified using a neural network that has learned features. The predetermined inference processing device 300 (predetermined coprocessor 302) learns different features of the identification target for the plurality of model files 304-1 to 304-M so that the models are suitable for different inference processing. You may let me.

For example, in the case of person detection in an image as an inference process, in deep learning, a large amount of an image of an entire person is used as a learning image for supervised learning, so that the features of the entire person in the image are analyzed by a neural network. Learn automatically. Alternatively, exemplifying face detection in an image as the inference processing, in deep learning, a large number of images of human faces are used as learning images to perform supervised learning to determine the features of the human faces in the images. Automatically learn neural networks.

In the error back-propagation method generally used in supervised learning, learning data is forward-propagated to a neural network for recognition, and a recognition result is compared with a correct answer to obtain an error. In the error back-propagation method, the error between the recognition result and the correct answer is propagated to the neural network in the opposite direction to that at the time of recognition, and the weight of each layer of the neural network is changed to approach the optimal solution.

There are many neurons (nerve cells) in the brain. Each neuron receives a signal from another neuron and passes a signal to another neuron. The brain performs various information processing according to the flow of this signal. The neural network is a model that realizes such characteristics of brain functions on a computer. A neural network connects units that imitate brain neurons hierarchically. Units are also called nodes. Each unit receives data from another unit, applies a weight to the data, and passes the data to the other unit. The neural network can identify (recognize) various identification targets by changing the weight of the unit by learning and changing the data to be passed.

In deep learning, a learned inference model capable of inference processing such as identifying an identification target reflected in an image can be generated by using a neural network that has learned features in this way.

When the learned inference model is generated, the predetermined coprocessor 302 uses the model file 304 as the learned inference model and the middleware 305 used for machine learning via the bridge controller 202 and the main processor 102 to drive the SSD 107 and/or the HDD 108. To be stored in.

For example, the model file 304 and the middleware 305 are read from the SSD 107 and/or the HDD 108 by the main processor 102 at or before the time when the learned inference model is to be used, and each coprocessor 302-1 is read via the bridge controller 202. ˜302-N. The middleware 305 allows each coprocessor 302 to execute a predetermined AI inference process by using the model file 304 as a learned inference model (see FIG. 3 ).

In the AI inference process, the same process can be repeatedly executed. For example, in the person detection inference process, the detection of a person in an image may be repeatedly performed. When a process such as the AI inference process that repeatedly executes the same process is executed by the coprocessor 302, it is possible to estimate an approximate processing time from the resource status of the plurality of coprocessors 302-1 to 302-N. If it is possible to utilize it for parallel distributed control, more efficient distributed control becomes possible on the information processing apparatus 100 side. That is, even when the processing performances of the plurality of coprocessors 302-1 to 302-N are different from each other, based on the resource status information of the processors of the plurality of coprocessors 302-1 to 302-N for each processing content. The processing time of the processing content can be calculated for each coprocessor 302-1 to 302-N. Then, based on the calculated processing times of the plurality of coprocessors 302-1 to 302-N, the coprocessor 302 appropriate for the processing content (task) (for example, returning the fastest response) can be selected. , The tasks can be distributed more efficiently, and the processing performance of the entire system can be improved.

For example, the API 120 shown in FIG. 2 functions as an interface for the higher-level applications 110-1 to 110-n to perform interprocess communication with the AI cluster management unit 130, and defines, for example, the object exchange format as a common format. .. In other words, the API 120 absorbs the difference in format between the upper applications 110-1 to 110-n and enables transmission and reception of information in the common format with the AI cluster management unit 130. In response to the inference instruction from the higher-level application 110, the API 120 can generate the inference request in the common format as the inference request including the inference processing identifier, the input data, and the parameter (see FIG. 9A).

The inference instruction from the application to the API 120 can be implemented as, for example, a function estimate(AIName, InData, Param, Func).

"AIName" specified as an argument of the function estimate corresponds to the inference processing identifier in the inference request, and information for identifying the AI inference processing (model file) to be executed can be specified by string type data. .. For example, “AIName” may be information (for example, the name of the model file) identifying the model files 304-1 to 304-k (see FIG. 3) (FIG. 9A, FIG. 12A). reference).

"InData" specified as an argument of the function estimate corresponds to the input data in the inference request, and the input data to be processed by the AI inference process can be specified by multidimensional array (numpy) type data. “InData” is, for example, image data, and can be a multidimensional array that returns the gradation value (color depth) as an element when the pixel position (horizontal position, vertical position) is given as an argument (FIG. 9A, FIG. 12A).

“Param” specified as an argument of the function estimate corresponds to the parameter in the inference request, and the condition for narrowing down in the AI inference process can be specified by the data of the string type (FIG. 9(a), FIG. 12A).

As a return value, the function estimate can return an acceptance number as integer (int) type data.

The upper applications 110-1 to 110-n generate a predetermined inference instruction and execute the API 120 in response to an instruction from the user or a request from the system side. The API 120 uses a common format (see FIG. 9A) for an inference request including an inference processing identifier, input data, and a parameter in response to a request instruction specified as a function estimate (AIName, InData, Param, Func), for example. It is generated and transmitted to the AI cluster management unit 130. That is, every time the upper application 110 executes the API 120, an inference request corresponding to the inference instruction of the application is transmitted to the AI cluster management unit 130 in a common format as an inference request by the upper application 110. After that, the API 120 returns the reception number to the higher-level application 110 as a return value of the function estimate (AIName, InData, Param, Func), for example.

The AI cluster management unit 130 illustrated in FIG. 2 manages AI inference processing by the plurality of inference processing devices 300-1 to 300-N. The AI cluster management unit 130 monitors which AI inference process (which model file) is being executed by which inference processing device 300 (which coprocessor 302), and which AI inference process (which model file) is inferred. It controls which processing unit 300 (which coprocessor 302) should be executed. The AI cluster management unit 130 can perform parallel distributed control in which the AI inference processing is distributed to the plurality of coprocessors 302-1 to 302-N.

That is, when the AI cluster management unit 130 receives an inference request from the higher-level application 110, the resource information collected from the plurality of coprocessors 302-1 to 302-N corresponding to the number of coprocessors 302 ( In the case of FIG. 2, the processing time is calculated by applying the calculation formula of 7 pieces. The AI cluster management unit 130 allocates the AI inference processing indicated by the inference processing identifier included in the inference request based on the processing times of the plurality of coprocessors 302-1 to 302-N obtained by the calculation. Select. The AI cluster management unit 130 may select, as the coprocessor to execute the AI inference process, the coprocessor 302 that has the shortest processing time obtained by the calculation among the plurality of coprocessors 302-1 to 302-N.

The AI cluster management unit 130 includes a parallel distribution control unit 131, a transmission unit 132, a reception unit 133, a processor monitoring unit 134, discrimination parameter information 135, a measurement unit 136, and an updating unit 137. The processor monitoring unit 134 has a resource information collecting unit 1341.

The parallel distributed control unit 131 can receive an inference request by an application from the API 120 and perform parallel distributed control in which the AI inference processing is distributed to the plurality of coprocessors 302-1 to 302-N according to the inference request. The parallel distribution control unit 131 has a transmission queue 1311 and a transmission destination determination unit 1312.

The transmission queue 1311 queues the inference request supplied from the API 120. The transmission queue 1311 is a queue buffer having a FIFO (First In First Out) structure, and each inference request is dequeued in the queued order.

The transmission destination determination unit 1312 has a calculation unit 1312a and a selection unit 1312b. When the calculation unit 1312a receives the dequeued inference request (inference request in the common format) from the transmission queue 1311, the calculation unit 1312a refers to the discrimination parameter information 135 and performs a calculation corresponding to the inference processing identifier included in the dequeued inference request. Identify the expression.

The discrimination parameter information 135 includes N×M calculation formulas associated with combinations of N different coprocessors and different M processing contents. Each calculation formula included in the discrimination parameter information 135 includes N discrimination parameters corresponding to N coprocessors. Each of the N discriminant parameters includes a conversion coefficient for converting the value of the resource information into the processing time, and a contribution rate indicating how much the resource information of the corresponding coprocessor affects the processing time (predicted value). May be included. The calculation unit 1312a can specify N calculation formulas corresponding to the inference processing identifier among the N×M calculation formulas.

For example, the discrimination parameter information 135 has a data structure as shown in FIG. FIG. 4 is a diagram showing a data structure of the discrimination parameter information 135. The discrimination parameter information 135 has a processor identification information column 1351, a process identification information column 1352, and a calculation formula column 1353. Information for identifying a processor is recorded in the processor identification information column 1351, and for example, processor names such as coprocessor A, coprocessor B,..., Coprocessor G can be recorded. Coprocessor A, coprocessor B,..., Coprocessor G are names of coprocessor 302-1, coprocessor 302-2,..., Coprocessor 302-N, for example. In the process identification information column 1352, information identifying the process content (that is, information corresponding to the inference process identifier) is recorded, and for example, process names such as process J, process K,... Can be recorded. .. Process J, process K,... Are names of the model files 304-1, 304-2,. The process J may correspond to, for example, an inference process for detecting a person in an image. The process K may correspond to, for example, an inference process for detecting a face area of a person in an image. The calculation formula column 1353 stores a calculation formula used for calculating the processing time. By referring to the discrimination parameter information 135 shown in FIG. 4, the calculation formula corresponding to each combination of the coprocessor and the processing content can be specified.

For example, the calculation formula corresponding to the combination of the coprocessor A and the process J (person detection process) can be specified as the following formula 1.
_{t AJ = k AJ1 x 1 +} k AJ2 x 2 + ··· + k AJN x N ··· Equation 1

In Expression 1, t _AJ indicates a processing time (predicted value) calculated when the processing J is executed by the coprocessor A. k _AJ1 , k _AJ2 ,..., k _AJN are discrimination parameters corresponding to the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N, respectively. Contribution indicating how much the conversion coefficient to be converted into the processing time and the resource information of the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N influence the processing time (predicted value). And rates. _{x 1, x 2, ···,} x N are each coprocessor 302-1, coprocessor 302-2, ..., to the value of the resource information of the coprocessor 302-N is applied (assignment) Indicates a variable.

Alternatively, for example, the calculation formula corresponding to the combination of the coprocessor A and the process K (face detection process) can be specified as the following formula 2.
t _AK =k _AK1 x ₁ +k _AK2 x ₂ +...+k _AKN x _N ...Equation 2

In Expression 2, t _AK indicates a processing time (predicted value) calculated when the processing K is executed by the coprocessor A. k _AK1 , k _AK2 ,..., K _AKN are determination parameters corresponding to the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N, respectively. Contribution indicating how much the conversion coefficient to be converted into the processing time and the resource information of the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N influence the processing time (predicted value). And rates. _{x 1, x 2, ···,} x N are each coprocessor 302-1, coprocessor 302-2, ..., to the value of the resource information of the coprocessor 302-N is applied (assignment) Indicates a variable.

Alternatively, for example, the calculation formula corresponding to the combination of the coprocessor B (coprocessor 302-2) and the process J (person detection process) can be specified as the following formula 3.
_{t BJ = k BJ1 x 1 +} k BJ2 x 2 + ··· + k BJN x N ··· Equation 3

In Expression 3, t _BJ indicates the processing time (predicted value) calculated when the processing J is executed by the coprocessor B. k _BJ1 , k _BJ2 ,..., k _BJN are determination parameters corresponding to the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N, respectively. Contribution indicating how much the conversion coefficient to be converted into the processing time and the resource information of the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N influence the processing time (predicted value). And rates. _{x 1, x 2, ···,} x N are each coprocessor 302-1, coprocessor 302-2, ..., to the value of the resource information of the coprocessor 302-N is applied (assignment) Indicates a variable.

When the calculation unit 1312a illustrated in FIG. 2 specifies the calculation formula corresponding to the inference processing identifier, the calculation unit 1312a waits until the resource information of each coprocessor 302-1 to 302-N is supplied.

Each of the coprocessors 302-1 to 302-N receives it at a predetermined cycle (for example, every few minutes to several hours) and/or from the processor monitoring unit 134 via the parallel distributed control unit 131 and the transmission unit 132. In response to the request, the own resource information is transmitted to the AI cluster management unit 130. The resource information includes, for example, the processor usage rate [%].

Upon receiving the resource information regarding the processing status from each coprocessor 302-1 to 302-N, the receiving unit 133 supplies the resource information of each coprocessor 302-1 to 302-N to the resource information collecting unit 1341. Upon receiving the resource information of each coprocessor 302-1 to 302-N, the resource information collection unit 1341 supplies the resource information of each coprocessor 302-1 to 302-N to the calculation unit 1312a.

In response to this, the calculation unit 1312a applies the value of the resource information of each coprocessor 302-1 to 302-N to the calculation formula to calculate the processing time of each coprocessor 302-1 to 302-N.

When the number of coprocessors is N and the number of processing contents is M, the calculation unit 1312a applies (substitutes) the resource information values of the N processors to each of the N calculation formulas. The processing time (predicted value) of N processors is obtained as the calculation result.

The calculation unit 1312a supplies the processing time of each of the coprocessors 302-1 to 302-N obtained by the calculation to the selection unit 1312b.

The selection unit 1312b selects a plurality of coprocessors 302-1 to 302-N as a coprocessor to execute the inference processing indicated by the inference processing identifier based on the processing time of each of the coprocessors 302-1 to 302-N. Select one coprocessor 302. The selection unit 1312b may select one coprocessor 302 corresponding to the shortest processing time (predicted value) from the plurality of coprocessors 302-1 to 302-N. The selection unit 1312b supplies the transmission unit 132 with the transmission destination information designating one selected coprocessor 302 as the transmission destination and the inference request in the common format.

The transmission unit 132 transmits an inference request (inference request for AI inference processing) in the common format to the coprocessor 302 (inference processing device 300) designated by the destination information. Thereby, the parallel distributed control in which the AI inference processing is distributed to the plurality of coprocessors 302-1 to 302-N can be efficiently performed.

Here, the processing performance of the plurality of coprocessors 302-1 to 302-N may change due to version upgrade of the firmware and/or replacement of hardware. Considering that point, the determination parameter information 135 can be updated at a predetermined update timing.

For example, when the discriminant parameter information 135 is updated when the selected coprocessor 302 executes the AI inference process, the measurement unit 136 causes the coprocessor 302 to transmit the inference request and then the coprocessor 302. Until the processing completion notification is received as the processing time for the value of the resource information of each coprocessor 302-1 to 302-N collected by the resource information collection unit 1341 and the measured processing time Can be accumulated. The update unit 137 acquires the measurement result of the accumulated processing time of each coprocessor from the measurement unit 136 at a predetermined update timing (for example, the update timing of every several weeks), and executes the accumulated processing of each coprocessor. Multiple regression analysis is performed using time, and the calculation formula of each coprocessor is newly obtained. At this time, the discrimination parameter in the calculation formula of each coprocessor is updated. The updating unit 137 accesses the discrimination parameter information 135 and replaces the calculation formula of each coprocessor with the newly calculated calculation formula by overwriting. As a result, the updating unit 137 updates the discrimination parameter information 135.

For example, consider a case in which the coprocessor A is selected based on the processing time (predicted value) obtained by Expression 1 and the processing J (person detection processing) is executed. By this execution, the measurement unit 136 causes the resource information x ₁ =90[%], x ₂ =45[%],..., X _N =20[ shown in the first line of FIG. %] of the processing time t _AJm =0.4 [s] is measured. If the calculation formula of Formula 1 corresponding to the combination of the coprocessor A and the process J is a formula obtained by multiple regression analysis for the past measurement results after the second line in FIG. There is a possibility that the calculation formula may be changed by adding the measurement result in the first line and performing the multiple regression analysis. Therefore, the measurement unit 136 supplies the measurement result obtained by adding the measurement result of the first line to the past measurement result of the second and subsequent lines in FIG. The updating unit 137 performs the multiple regression analysis using the measured value t _AJm of the processing time as the objective variable and the resource information x _{1 to} x _N as the explanatory variables, and newly obtains Formula 4 below.
_{t AJ = k AJ1 'x 1} + k AJ2' x 2 + ··· + k AJN 'x N ··· equation (4)

In Equation _{4, k AJ1 ', k AJ2} ', ···, k AJN ' are each coprocessor 302-1, coprocessor 302-2, ..., the updated corresponding to the coprocessor 302-N This is a discrimination parameter. Comparing Expression 1 and Expression 4, the discrimination parameters corresponding to the coprocessor 302-1, the coprocessor 302-2,..., Coprocessor 302-N in the calculation formula are updated, and the other parts are the same. It turns out that The updating unit 137 can hold the newly calculated calculation formula of Expression 4.

At a predetermined update timing, the update unit 137 accesses the discrimination parameter information 135, and as shown in FIG. 5B, a newly calculated formula corresponding to the combination of the coprocessor A and the process J is obtained. Replace the calculation formula of 4 by overwriting. As a result, the updating unit 137 updates the discrimination parameter information 135. Note that FIG. 5 is a diagram showing an update process (in the case of coprocessor A and process J) of the discrimination parameter information 135.

Alternatively, for example, consider a case where the coprocessor A is selected based on the processing time (predicted value) obtained by Expression 2 and the processing K (face detection processing) is executed. By this execution, the measurement unit 136 causes the resource information x ₁ =90[%], x ₂ =45[%],..., X _N =20[ shown in the first line of FIG. %] of the processing time t _AKm =0.3 [s]. If the calculation formula of the formula 2 corresponding to the combination of the coprocessor A and the process K is the formula obtained by the multiple regression analysis on the past data on and after the second line in FIG. 6A, it is newly obtained. The calculation formula may be changed by adding the measurement results of the first line and performing multiple regression analysis. Therefore, the measurement unit 136 supplies the measurement result obtained by adding the data of the first line to the past data of the second and subsequent lines in FIG. 6A to the updating unit 137. The updating unit 137 performs the multiple regression analysis using the measured value t _AJm of the processing time as the objective variable and the resource information x _{1 to} x _N as the explanatory variables, and newly obtains Equation 5 below.
t _AK =k _{AK1'x 1} +k _{AK2'x 2} +...+k _{AKN'x N} ...Equation 5

In Equation 5, k _AK1 ′, k _AK2 ′,..., K _AKN ′ are the updated values corresponding to the coprocessor 302-1, coprocessor 302-2,..., Coprocessor 302-N, respectively. This is a discrimination parameter. Comparing Expression 2 and Expression 5, the discrimination parameters corresponding to the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N in the calculation formula are updated, and other parts are the same. It turns out that The updating unit 137 can hold the newly obtained calculation formula of Expression 5.

At a predetermined update timing, the update unit 137 accesses the determination parameter information 135, and as shown in FIG. 6B, a newly calculated calculation formula corresponding to the combination of the coprocessor A and the process K is obtained. Replace the calculation formula of 5 by overwriting. As a result, the updating unit 137 updates the discrimination parameter information 135. 6 is a diagram showing an update process (in the case of coprocessor A and process K) of the discrimination parameter information 135.

Alternatively, for example, consider a case where the coprocessor B is selected based on the processing time (predicted value) obtained by Expression 3 and the processing J (person detection processing) is executed. By this execution, the measurement unit 136 causes the resource information x ₁ =90[%], x ₂ =45[%],..., X _N =20[ shown in the first line of FIG. 7A. %] of the processing time t _BJm =0.3 [s]. If the calculation formula of the formula 3 corresponding to the combination of the coprocessor B and the process J is the formula obtained by the multiple regression analysis on the past data on and after the second line in FIG. 7A, it is newly obtained. The calculation formula may be changed by adding the measurement results of the first line and performing multiple regression analysis. Therefore, the measurement unit 136 supplies the measurement result obtained by adding the data of the first line to the past data of the second and subsequent lines in FIG. 7A to the updating unit 137. The updating unit 137 performs multiple regression analysis using the measurement value t _BJm of the processing time as an objective variable and the resource information x _{1 to} x _N as explanatory variables, and newly obtains the following Equation 6.
t _BJ =k _{BJ1'x 1} +k _{BJ2'x 2} +...+k _{BJN'x N} ...Equation 6

In Expression 6, k _BJ1 ′, k _BJ2 ′,..., K _BJN ′ are the _{updated values} corresponding to the coprocessor 302-1, the coprocessor 302-2,..., The coprocessor 302-N, respectively. This is a discrimination parameter. Comparing Expression 3 and Expression 6, the discrimination parameters corresponding to the coprocessor 302-1, the coprocessor 302-2,..., Coprocessor 302-N in the calculation expression are updated, and the other portions are the same. It turns out that The updating unit 137 may hold the newly calculated calculation formula of Expression 6.

At a predetermined update timing, the update unit 137 accesses the determination parameter information 135, and as shown in FIG. 7B, a newly calculated formula corresponding to the combination of the coprocessor B and the process J is obtained. Replace the calculation formula of 6 by overwriting. As a result, the updating unit 137 updates the discrimination parameter information 135. 7. FIG. 7 is a diagram showing an update process (in the case of coprocessor B and process J) of the discrimination parameter information 135.

As a result, the parallel distributed control unit 131 can improve the accuracy of the parallel distributed control in response to changes in the processing performance of the coprocessors 302-1 to 302-N.

Next, the operation of the inference processing device 300 using the model file 304 will be described with reference to FIG. FIG. 8 is a diagram showing an operation using the model file 304 of the inference processing device 300.

In the inference processing apparatus 300 shown in FIG. 8, the inference application 303 is used by the inference application 303 among the plurality of middleware and the plurality of model files stored in the SSD 107 and/or the HDD 108 of the information processing apparatus 100 by the virtual environment technology. The middleware 305 and the model files 304-1 to 304-M are loaded. The inference application 303 initializes the middleware 305 and the model files 304-1 to 304-M when the inference processing device 300 is activated.

When the inference processing device 300 is selected as the transmission destination by the AI cluster management unit 130, the inference application 303 receives the inference request from the upper application 110 via the API 120 and the AI cluster management unit 130. The inference request includes an inference process identifier, input data (image, voice, text, etc.) that is a target of the inference process, and parameters for setting execution conditions of the inference process. The inference request is converted to the common format by the API 120, and the inference application 303 receives the inference request in the common format. The inference application 303 identifies the model file 304 corresponding to the inference processing identifier included in the inference request, and converts the input data in the common format included in the inference request into the format of the identified model file 304.

Here, the formats of the M model files 304-1 to 304-M may be different from each other. That is, the inference application 303 absorbs the difference in format between the model files 304-1 to 304-M and shares a common format with the higher-level applications 110-1 to 110-n (that is, with the API 120). Enables to send and receive information.

For example, when the inference processing identifier corresponds to the model file 304-1, as shown in FIG. 8, the inference application 303 determines that the model file to be executed is the model file 304 based on the inference processing identifier included in the inference request. -1 is specified and the middleware 305 is notified. Further, the inference application 303 converts the input data in the common format into the input data in the format of the model file 304-1 and supplies it to the middleware 305. The middleware 305 inputs the format-converted input data to the input layer of the model file 304-1 and causes the model file 304-1 to execute inference processing. When output data (output data in the format of the model file 304-1) as the execution result of the inference process is supplied from the model file 304-1 to the middleware 305, the middleware 305 infers the output data of the model file 304-1. Supply to the application 303. The inference application 303 converts the output data in the format of the model file 304-1 into the output data in the common format. The inference application 303 transmits the output data after the format conversion to the upper application 110 via the AI cluster management unit 130 and the API 120.

The format of the inference request received by the inference application 303 from the higher-level application 110 side is converted into a common format by the API 120, for example, the format shown in FIG. 9A.

The common format shown in FIG. 9A is common for object determination of images regardless of the type of model file. In the inference request, ai_name [character string] may be designated as the inference processing identifier. Image width: width [integer], image height: height [integer], color depth parameter: color_depth [integer], reliability threshold: confidence_threshold [floating point number] ], the maximum number of returns: [integer] can be specified. Image_data [byte array] (byte array of “width×height×3×color_depth” (3=three layers of RGB)) can be designated as data (image) to be inferred.

When the model file 304-1 is a numeral recognition model, the numeral recognition model (model file 304-1) determines whether or not the image contains numerals 0-9. The input image is a 28×28 monochrome image only, and the entire image is determined.

In this case, the inference application 303 converts the input data in the common format shown in FIG. 9A into input data that has been resized to 28×28 and converted into monochrome, and then the model file 304-1 shown in FIG. Use as input data of format.

The format of the model file 304-1 shown in FIG. 9B is the format of the input data of the character recognition model (model file 304-1), and is not specified (maximum as a parameter for setting the execution condition of the inference process). There is no specification of the number or threshold). A 28×28×1 float type (floating point number type) 32-bit array can be designated as the data (image) to be inferred.

For example, when the reliability threshold value is set to 0.5 as a parameter and a color image of 100×100×3×8 bits is received as input data in the common format, the inference application 303 determines that the inference application 303 is 28×28×32 bits. Can be converted into a monochrome image of the input data of the model file 304-1.

Note that FIG. 9 is a diagram showing a conversion process from the common format to the format of the input layer of the model file 304-1.

The output data as the execution result of the inference process in the model file 304-1 is output data in the format of the model file 304-1 as shown in FIG.

The format of the model file 304-1 shown in FIG. 10A is the format of the output data of the character recognition model (model file 304-1). As the execution result of the inference process, 10 float type (floating point numbers) 32 bit array of type (indicates the probability that each element is a number of 0-9. For example, the probability of being the 0th element... "0" to the probability of the 9th element... "9". ) Can be specified.

In this case, the inference application 303 selects an element that is equal to or larger than the threshold value out of 10 arrays for the output data in the format of the model file 304-1 shown in FIG. The type and reliability of the object are returned as the probability of that number, and the position designates the entire image, which is output data in the common format shown in FIGS. 10B and 10C.

The common format shown in FIG. 10B is a format of output data to the higher-level application 110 side and is common for object determination of images regardless of the type of model file. In the output data in the common format, [character string] is specified as the inference processing identifier, and the success or failure of the inference processing is: result_code [integer] (0: normal, other than 0: abnormal) as the execution result of the inference processing. Number of results: number_result [integer] is specified. Also, in the number_result [integer] (or the number of number_result), object_class [integer] (a code indicating the kind of the object, different for each model) can be specified as the kind of the object. Left: left [integer], top: top [integer], width: width [integer], height: height [integer], reliability: confidence [floating point number] can be specified as the position of the object on the image. .. Note that FIG. 10 is a diagram showing a conversion process from the format of the output layer of the model file 304-1 to the common format.

For example, array 0.0, 0.0, 0.0, 0.3, 0.7, 0.0, 0.8, 0.6, 0.0, 0.0
When is output as output data from the model file 304-1, the inference application 303 outputs the output data to the success or failure of the inference process: 0 * success The number of results of the inference process: 3 * 3 out of 10 is 0.5 or more Numerical value {
[
Object type: 6 * 6th element (starting from 0)
Position of object on image: 0, 0, 100, 100 * Overall image reliability: 0.8
],
[
Object type: 4 * 4th element (starting from 0)
Position of the object on the image: 0, 0, 100, 100 * Overall image reliability: 0.7 * The fourth numerical value is the reliability],
[
Object type: 7 * 7th element (starting from 0)
Position of the object on the image: 0, 0, 100, 100 * Overall image reliability: 0.6 * 7th numerical value is reliability]
}
To output data in a common format.

Alternatively, for example, when the inference processing identifier corresponds to the model file 304-2, the inference application 303 determines that the model file to be executed is the model from the inference processing identifier included in the inference request, as shown in FIG. The file is identified as the file 304-2 and the middleware 305 is notified. Further, the inference application 303 converts the input data in the common format into the input data in the format of the model file 304-2 and supplies it to the middleware 305. The middleware 305 inputs the format-converted input data to the input layer of the model file 304-2, and causes the model file 304-2 to execute inference processing. When the output data (output data in the format of the model file 304-2) as the execution result of the inference process is supplied from the model file 304-2 to the middleware 305, the middleware 305 infers the output data of the model file 304-2. Supply to the application 303. The inference application 303 converts the output data in the format of the model file 304-2 into output data in the common format. The inference application 303 transmits the output data after the format conversion to the upper application 110 via the AI cluster management unit 130 and the API 120.

The format of the inference request received by the inference application 303 from the higher-level application 110 side is converted into a common format by the API 120, for example, the format shown in FIG. The common format shown in FIG. 12A is the same as the common format shown in FIG. 9A.

When the model file 304-2 is an object recognition model, the object recognition model (model file 304-2) determines whether or not the image includes an identifiable object (person, car, cat, etc.). The input image is an image of an arbitrary size, and when the entire image is determined and an object is detected, the type (number), position, and reliability of the object are output.

In this case, the inference application 303 converts the input data (input image data) of the common format shown in FIG. 12A into input data in which one dimension (depth) is added, and the model file shown in FIG. The input data is in the format 304-2.

The format of the model file 304-2 shown in FIG. 12B is the format of the input data of the object recognition model (model file 304-2), and it is desired to determine the maximum number as a parameter for setting the execution condition of the inference process. Object type, reliability threshold, data (image) to be subjected to inference processing: height×width×depth×3 unit32 array (depth is normally not required for 3D calculation) obtain.

For example, when the reliability threshold value is set to 0.5 and the maximum number is set to 5 as a parameter, and a color image of 100×100×3×8 bits is received as input data of the common format, the inference application 303 Is converted to an image of 100×100×1×3×32 bits and used as input data of the model file 304-2, and the result of the inference process is narrowed down to 0.5 as the reliability threshold and 5 as the maximum number. Can be used as a parameter for

Note that FIG. 12 is a diagram showing a conversion process from the common format to the format of the input layer of the model file 304-2.

The output data as the execution result of the inference process in the model file 304-2 is output data in the format of the model file 304-2 as shown in FIG.

The format of the model file 304-2 shown in FIG. 13A is the format of the output data of the object recognition model (model file 304-2), and the number of judged objects is [integer] as the execution result of the inference process. The type of object: maximum number of arrays, the position of objects: maximum number of arrays (positions are returned at a ratio of 1 in the vertical and horizontal directions), object reliability: maximum number of arrays can be specified.

In this case, the inference application 303 has a certain reliability with respect to the output data in the format of the model file 304-2 shown in FIG. 13A, from the array of the type, the position, the reliability, and the determined number of objects. Only the elements having the values or more are output, and the output data is in the common format shown in FIGS. 13B and 13C.

The common format shown in FIGS. 13B and 13C is the same as the common format shown in FIGS. 10B and 10C.

For example,
Number of judged objects 2
Array of object types [1, 4, 0, 0, 0] (maximum number)
Array of object positions [[0.2, 0.15, 0.3, 0.2], [0.7, 0.8, 0.2, 0.1], 0, 0, 0] (maximum (For the number of pieces)
Array of object reliability [0.8, 0.3, 0, 0, 0] (maximum number)
Is output as output data from the model file 304-1, the inference application 303 outputs the output data to the success or failure of the inference process: 0 *Successful number of results of the inference process: 1 *Only one of 5 has a reliability of 0 .5 or more {
[
Object type: 1 *Code indicating the object type (defined for each model)
Position of object on image: 20, 15, 30, 20 * Convert position from ratio to coordinate value Reliability: 0.8
]
}
To output data in a common format.

As described above, in the embodiment, in the information processing system 1, the resource information collected from the N coprocessors 302-1 to 302-N is applied to the calculation formula to calculate the processing time of each of the N coprocessors. Based on the processing times of the N coprocessors that are calculated, the coprocessor to which the inference processing is allocated is selected and the inference request is transmitted. For example, the information processing system 1 can select the coprocessor 302 having the shortest processing time and transmit the inference request. As a result, the current availability of resources can be taken into consideration, and parallel distributed control can be made efficient in real time.

Further, in the embodiment, in the information processing device 100, the measurement unit 136 measures the processing time by the processor 302 that performs the inference process in response to the inference request. The updating unit 137 performs multiple regression analysis based on the received resource information of the N coprocessors 302-1 to 302-N and the measured processing time, and updates the calculation formula. As a result, it is possible to deal with a change in the processing performance of the N coprocessors 302-1 to 302-N, and it is possible to improve the accuracy of parallel distributed control.

Further, in the embodiment, in the information processing apparatus 100, the calculation formula includes N determination parameters corresponding to the N coprocessors coprocessors 302-1 to 302-N, and the updating unit 137 uses N in the calculation formula. Update the discriminant parameters. As a result, the calculation formula can be updated in response to the change in the processing performance of the N coprocessors 302-1 to 302-N.

Further, in the embodiment, in the information processing device 100, the inference request includes the inference processing identifier that identifies the content of the inference processing. The calculation 1312a unit associates the resource information of the N coprocessors 302-1 to 302-N with a combination of different N coprocessors 302-1 to 302-N and different M inference processing contents. It is applied to N calculation formulas corresponding to the inference processing content identified by the inference processing identifier among the N×M calculation expressions. As a result, when coprocessors that are fast for specific processing are mixed in the parallel distributed system, the processing content can be taken into consideration as a performance difference in addition to the processing capacity, so parallel distributed control can be performed at a high level. The accuracy can be improved.

The information processing apparatus 100 according to the embodiment also generates an inference request including an inference processing identifier that identifies the content of the inference processing and transmits the inference request to the inference processing apparatus 300. As a result, the inference processing device 300 identifies the model file by the inference processing identifier, suppresses the development of unnecessary model files, develops only the model files necessary for the inference processing in the memory, and executes the inference processing. be able to. Therefore, the inference processing apparatus 300 may be provided with hardware including a memory or the like having such a performance that one model file can be expanded and inference processing can be executed. As a result, the information processing system 1 can efficiently execute the inference processing even in the inference processing device 300 having low hardware performance and meet the request of the higher-level application.

Further, in the embodiment, in the information processing apparatus 100, the N coprocessors coprocessors 302-1 to 302-N to be controlled by the parallel distributed control perform the inference processing according to the inference request. As a result, the coprocessor 302 executes a process for repeatedly executing the same process, so that the approximate processing time can be easily estimated from the resource statuses of the plurality of coprocessors 302-1 to 302-N.

In the embodiment, in the inference processing device 300, the M model files 304-1 to 304-M correspond to different inference processes. The middleware 305 can execute each of the M model files 304-1 to 304-M. The inference application 303 receives an inference request including an inference processing identifier and input data from the information processing apparatus 100, and specifies a model file corresponding to the inference processing identifier from the M model files 304-1 to 304-M. .. The middleware 305 reads the identified model file and executes inference processing. As a result, compared to the case where the same learned model is used for various inference processes, the model file 304 that has been made efficient by machine learning can be used for each inference process. The processing can be made efficient.

Further, in the embodiment, in the inference processing device 300, the inference application 303 receives the inference request including the inference processing identifier and the input data in the common format from the information processing device 100. The inference application 303 converts the input data in the common format into input data in a format that can be executed by the model file 304 corresponding to the inference processing identifier among the M model files 304-1 to 304-M. The middleware 305 executes the specified model file 304 by using the input data of that format. This allows the model file 304 to execute the inference process using the input data received from the information processing apparatus 100 side.

Further, in the embodiment, in the inference processing device 300, the inference application 303 converts the output data in the format for the model file 304 obtained by executing the specified model file 304 into the output data in the common format for information processing. Send to the device 100. As a result, the execution result of the inference process of the model file 304 can be provided to the information processing device 100 side.

Further, in the embodiment, the inference processing apparatus 300 receives an inference request including an inference processing identifier corresponding to another model file 304 and input data in a common format from the information processing apparatus 100. The inference application 303 converts the input data in the common format into input data in a format that can be executed by another model file 304 corresponding to the inference processing identifier among the M model files 304-1 to 304-M. The middleware 305 executes the specified other model file 304 by using the input data of the format. As a result, the inference process can be executed by the other model file 304 using the input data received from the information processing device 100 side.

Further, in the embodiment, in the inference processing device 300, the inference application 303 converts the output data in the format for the other model file 304 obtained by executing the other identified model file 304 into the output data in the common format. Then, the information is transmitted to the information processing apparatus 100. As a result, the execution result of the inference process of the other model file 304 can be provided to the information processing apparatus 100 side.

Each inference processing device 300-1 to 300-N may include a plurality of middlewares 305i-1 and 305i-2, as shown in FIG. FIG. 14 is a diagram illustrating a functional configuration of the inference processing device 300 according to the modified example of the embodiment.

In this case, each of the plurality of middlewares 305i-1 and 305i-2 corresponds to one or more model files in the M model files 304-1 to 304-M. For example, in the case of FIG. 14, the middleware 305i-1 corresponds to one model file 304-1, and the middleware 305i-2 corresponds to (M-1) model files 304-2 to 304-M. It corresponds.

The inference application 303 identifies a set corresponding to the inference processing identifier among a plurality of sets each including the middleware 305i and the model file 304 corresponding to the middleware. The middleware 305i included in the specified set executes the model file 304 included in the specified set.

For example, in the case of FIG. 14, as M sets, (middleware 305i-1, model file 304-1), (middleware 305i-2, model file 304-2), (middleware 305i-2, model file 304-3) , (Middleware 305i-2, model file 304-M) exist.

When the inference processing identifier corresponds to the model file 304-1, the inference application 303 identifies (middleware 305i-1, model file 304-1) as a set corresponding to the inference processing identifier. In this case, the middleware 305i-1 executes the model file 304-1.

When the inference processing identifier corresponds to the model file 304-2, the inference application 303 identifies (middleware 305i-2, model file 304-2) as a set corresponding to the inference processing identifier. In this case, the middleware 305i-2 executes the model file 304-2.

As described above, even when each of the inference processing devices 300-1 to 300-N has a plurality of middlewares 305i-1 and 305i-2, it is possible to use the model file 304 that is made efficient by machine learning for each inference process. Therefore, the inference processing in the inference processing device 300 can be made efficient.

Alternatively, the bridge controller 202 may be a PCIe bridge controller 3 compatible with a PCIe (Peripheral Component Interconnect express) standard as shown in FIGS. FIG. 15 is a diagram illustrating a hardware configuration of an information processing system according to another modification of the embodiment. FIG. 16 is a diagram showing a software configuration of an information processing system in another modification of the embodiment. FIG. 17 is a diagram illustrating a hardware configuration of a PCIe bridge controller according to another modification of the embodiment. FIG. 18 is a diagram illustrating a PCIe layer configuration according to another modification of the embodiment. FIG. 19 is a diagram showing how another processor looks from the coprocessor G in another modification of the embodiment. FIG. 20 is a diagram showing how another processor looks from the coprocessor D in another modification of the embodiment. FIG. 21 is a diagram illustrating a data transfer process between processors via a PCIe bridge controller according to another modification of the embodiment.

The information processing system 1 illustrated in FIG. 15 includes a PCIe bridge controller 3 and a plurality of (eight in the example illustrated in FIG. 4) platforms 2-1 to 2-8. The platforms 2-1 to 2-8 are connected to the PCIe bridge controller 3, respectively.

In addition, hereinafter, as the code indicating the platform, reference numerals 2-1 to 2-8 are used when it is necessary to specify one of the plurality of platforms, but reference numeral 2 is used when referring to an arbitrary platform. The platform 2 may be called the PC platform 2.

The platform 2-1 includes a main processor 21-1. The main processor 21-1 corresponds to the main processor 102 of the embodiment. The platforms 2-2 to 2-8 include coprocessors 21-2 to 21-8, respectively. The coprocessors 21-2 to 21-8 correspond to the coprocessors 302-1 to 302-N of the embodiment, respectively.

The main processor 21-1 and the coprocessors 21-2 to 21-8 may be provided by different manufacturers (vendors). For example, the main processor 21-1, the coprocessor 21-2, the coprocessor 21-3, the coprocessor 21-4, the coprocessor 21-5, the coprocessor 21-6, the coprocessor 21-7, and the coprocessor 21-8 are , A company, B company, C company, D company, E company, F company, G company, and H company, respectively.

Further, hereinafter, the coprocessor 21-2, the coprocessor 21-3, the coprocessor 21-4, the coprocessor 21-5, the coprocessor 21-6, the coprocessor 21-7, and the coprocessor 21-8 are respectively referred to as coprocessors. Sometimes referred to as A, B, C, D, E, F, G. Also, different platforms may be connected to the EPs mounted on the PCIe bridge controller. Further, two or more EPs may be connected to one platform, and the platform side may use the RCs to communicate with the PCIe bridge controller.

In addition, hereinafter, as the reference numeral indicating the processor, reference numerals 21-1 to 21-8 or reference numerals A to G are used when it is necessary to specify one of the plurality of processors, but when referring to any processor, 21 is used.

The platforms 2-1 to 2-8 are computer environments that perform arithmetic processing such as AI inference processing and image processing, and include a processor 21, a storage 23 and a memory (physical memory) 22 shown in FIG.

In the platform 2, various functions are realized by the processor 21 executing programs stored in the memory 22 and the storage 23.

The storage 23 is a storage device such as a hard disk drive (HDD), an SSD (solid state drive), and a storage class memory (storage class memory: SCM), and stores various data.

The memory 22 is a storage memory including a ROM (Read Only Memory) and a RAM (Random Access Memory). Various software programs and data for these programs are written in the ROM of the memory 22. The software program on the memory 22 is appropriately read and executed by the processor 21. The RAM of the memory 22 is used as a primary storage memory or a working memory.

The processor 21 (main processor 21-1 or coprocessors 21-2 to 21-8) controls the entire platform 2. The processor 21 may be a multiprocessor. The processor 21 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processing FPGA), a DSP (Digital Signal Progressive, Incremental Integrated Circuit, ASIC). It may be any one of the Field Programmable Gate Array). Further, the processor 21 may be a combination of two or more types of elements among CPU, GPU, MPU, DSP, ASIC, PLD, and FPGA. For example, the coprocessors 21-2 to 21-8 may be a combination of CPU and GPU.

In the information processing system 1 illustrated in FIG. 16, the platform 2-1 uses Windows as an OS, and the store management program is executed on this OS. The store management program corresponds to the upper application 110 of the embodiment. Each of the platforms 2-2 and 2-3 uses Linux (registered trademark) as an OS, and the distributed processing programs (distributed processing A and B) are executed on this OS. The distributed processing programs (distributed processing A and B) correspond to the inference processing application 303 of the embodiment.

Each platform 2 is provided with a bridge driver 20, and the platform 2 communicates with the PCIe bridge controller 3 and another platform 2 via this bridge driver 20. The communication method by the bridge driver 20 will be described later.

Each platform 2 includes a processor 21 and a memory (physical memory) 22, and each function is realized by the processor 21 executing an OS, various programs, drivers, etc. stored in the memory 22.

The processors 21 (main processor 21-1 or coprocessors 21-2 to 21-8) included in each platform 2 may be provided by different vendors. In the example illustrated in FIG. 11, a platform having a plurality of RCs (for example, an x86 processor manufactured by Intel Corporation) may be used for at least a part of the platforms 2 (for example, the platform 2-1).

Further, each platform 2 is configured to be independently operable so as not to affect other driver configurations.

In the platform 2, a part of the storage area of the memory 22 is used as a communication buffer 221 in which data transferred between the platforms 2 (between the processors 21) is temporarily stored, as described later with reference to FIG.

The PCIe bridge controller 3 realizes communication of data and the like between the plurality of platforms 2-1 to 2-8.

The PCIe bridge controller 3 shown in FIG. 17 is, for example, a relay device having an EP of 8 channels in one chip. As shown in FIG. 17, the PCIe bridge controller 3 includes a CPU 31, a memory 32, an interconnect bus 33, and a plurality of (eight in the example shown in FIG. 17) slots 34-1 to 34-8.

Devices configured to satisfy the PCIe standard are connected to the slots 34-1 to 34-8. For example, in the information processing system 1, the platform 2 is connected to each of the slots 34-1 to 34-8.

In addition, hereinafter, as the code indicating the slot, the codes 34-1 to 34-8 are used when it is necessary to specify one of the plurality of slots, but the code 34 is used to indicate an arbitrary slot.

Note that one processor 2 may be connected to one slot 34 like the platforms 2-2 to 2-8 in FIG. 15, and a plurality of (such as the platform 2-1 in FIG. One platform 2 may be connected to the slots 34 (two in the example of FIG. 4), and various modifications can be implemented.

By assigning a plurality of slots 34 to one platform 2 like the platform 2-1 in FIG. 15, it is possible to allow the platform 2-7 to perform communication using a wide communication band.

Each slot 34 is connected to the interconnect 33 via an internal bus. A CPU 31 and a memory 32 are connected to the interconnect 33. As a result, each slot 34, the CPU 31 and the memory 32 are communicably connected to each other via the interconnect 33.

The memory 32 is a storage memory (physical memory) including a ROM and a RAM, for example. In the ROM of the memory 32, a software program relating to data communication control and data for this program are written. The software program on the memory 32 is appropriately read and executed by the CPU 31. The RAM of the memory 32 is used as a primary storage memory or a working memory.

Further, each platform 2 is provided with a memory area 35 (see FIG. 21) corresponding to each slot, and a plurality of storage areas divided by the number of slots are set in the memory area 35. Is associated with one of the slots 305. That is, the memory area 35 of the memory 22 is provided with a storage area corresponding to each of the slots #0 to #7.

The PCIe bridge controller 3 uses the storage area of the memory area 35 associated with each slot to transfer data between the platforms 2 as described later.

The CPU 31 controls the PCIe bridge controller 3 as a whole. The CPU 31 may be a multiprocessor. Any one of MPU, DSP, ASIC, PLD and FPGA may be used instead of the CPU 31. Further, the CPU 31 may be a combination of two or more types of elements among a CPU, MPU, DSP, ASIC, PLD, and FPGA.

Then, the CPU 31 executes the software program stored in the memory 32 to realize the data transfer between the platforms 2 (between the processors 21) in the PCIe bridge controller 3.

The PCIe bridge controller 3 uses PCIe in order to speed up data transfer between the platforms 2, and as shown in FIG. 15, each processor provided in each platform 2 operates as an RC, and between the EPs operating as devices. Achieve data transfer.

Specifically, in the information processing system 1, the processor of each platform 2 operates as a PCIe RC as a data transfer interface. Further, the PCIe bridge controller 3, that is, the slot 34 to which each platform 2 is connected is operated as an EP for each platform 2 (processor 21).

As a method of connecting the PCIe bridge controller 3 to the processor 21 as an EP, various known methods can be used.

For example, the PCIe bridge controller 3 connects to the processor 21 as an EP by notifying the processor 21 of a signal indicating that it functions as an EP when connecting to the platform 2.

The PCIe bridge controller 3 tunnels data by EP to EP (End Point to End Point) and transfers the data to a plurality of RCs. Communication between processors is logically connected when a PCIe transaction occurs, and when data transfer is not concentrated on one processor, data can be transferred between the processors in parallel.

FIG. 18 shows an example in which communication is performed between the coprocessor A of the platform 2-2 and the coprocessor B of the platform 2-3.

In the source platform 2-2, the data generated in the RC coprocessor A is sequentially transferred through the software, the transaction layer, the data link layer, and the physical layer (PHY), and the PCIe bridge controller is provided in the physical layer. 3 physical layer.

In the PCIe bridge controller 3, the physical layer, the data link layer, the transaction layer, and the software are sequentially transferred, and the data is transferred to the EP corresponding to the RC of the destination platform 2 by tunneling.

That is, in the PCIe bridge controller 3, data is transferred from one RC (coprocessor 21-2) to another RC (coprocessor 21-3) by tunneling the data between EPs.

In the destination platform 2-3, the data transferred from the PCIe bridge controller 3 is sequentially transferred through the physical layer (PHY), the data link layer, the transaction layer, and the software, and the data of the destination platform 2-3 is transferred. Transferred to coprocessor B.

In the information processing system 1, the communication between the processors 21 (between the platforms 2) is logically connected when a PCIe transaction occurs.

When data transfer from a plurality of other processors 21 is not concentrated on a specific processor 21 connected to one of the eight slots of the PCIe bridge controller 3, a different arbitrary plurality of sets of the respective processors 21 are provided. Data may be transferred in parallel between them.

For example, when the coprocessor A of the platform 2-2 and the coprocessor B of the platform 2-3 each try to communicate with the main processor of the platform 2-1, the PCIe bridge controller 3 determines that the coprocessor A , The processing of the coprocessor B is processed serially.

However, when communication is performed between different processors such as main processor-coprocessor A, coprocessor B-coprocessor C, coprocessor D-coprocessor E, and communication is not concentrated on a specific processor, PCIe is used. The bridge controller 3 processes communication between the processors 21 in parallel.

FIG. 19 is a diagram illustrating how the processor 21-8 (processor G) looks in the other processor 21 in the information processing system 1 as an example of the embodiment, and FIG. 20 illustrates the processor 21-5 (processor D) from the processor 21-8 (processor D). It is a figure which illustrates the appearance of the other processor 21 of FIG.

Even when communication is performed between the processors 21, only the PCIe bridge controller 3 can be seen from the OS (for example, the device manager of Windows) on each processor 21, and the other processor 21 of the connection destination is directly managed. No need. That is, the device driver of the PCIe bridge controller 3 may manage the processor 21 connected to the end of the PCIe bridge controller 3.

Therefore, it is not necessary to prepare a device driver for operating the processor 21 of each of the transmission source and the reception destination, and communication between the processors 21 can be performed only by performing communication processing to the PCIe bridge controller 3 by the driver of the PCIe bridge controller 3. Can be done.

A data transfer method between the processors 21 via the PCIe bridge controller 3 in the information processing system 1 as an example of the embodiment configured as described above will be described with reference to FIG.

In the example shown in FIG. 21, a case where data is transferred from the platform 2-1 connected to the slot #0 to the platform 2-5 connected to the slot #4 will be described.

The platform 2-1 of the data transmission source stores the data transmitted by software (hereinafter sometimes referred to as transmission data) from the storage 23 or the like provided in the platform 2-1 to the memory area 35 of the platform 2-1. (See P1). The memory area 35 may be a part of the communication buffer 221. The memory area 35 is an area provided in each of the plot forms 2 in the memory 22 and the like with the same size. The memory area 35 is divided according to the number of slots. The divided storage areas of the memory area 35 are associated with any of the slots. For example, the storage area indicated by Slot#0 in the memory area 35 is associated with the platform 2-1 connected to Slot#0, and the storage area indicated by Slot#4 in the memory area 35 is changed to Slot#0. It is associated with the connected platform 2-5. The platform 2-1 stores the transmission data in the area (here, Slot#4) assigned to the destination slot in the memory area 35.

The bridge driver 20 acquires or generates, based on the storage area of the memory area 35 of the platform 2, slot information indicating a destination slot and address information indicating an address in a divided area in the destination memory area 35. (See symbol P2).

In the transmission source EP, the bridge driver 20 passes transfer data including slot information, address information, and transmission data to the relay device 3 (reference P3). As a result, the PCIe bridge controller 3 transfers the transfer data to the destination platform 2-4 by connecting the transmission source slot and the transmission destination slot by EPtoEP based on the slot information (see reference numeral P4). Based on the slot information and the address information, the bridge driver 20 of the transmission destination transmits the transmission data (or the transfer data to the area of the address indicated by the address information in the storage area corresponding to the slot #4 of the memory area 35 of the platform 2 of the transmission destination). Data) is stored (see reference numeral P5).

In the destination platform 2, for example, the program reads the transmission data stored in the memory area 35 and moves it to the memory (local memory) 22 or the storage 23 (see symbols P6 and P7).

As described above, the data (transfer data) is transferred from the transfer source platform 2-1 to the transfer destination platform 2-5.

As described above, in the information processing system 1, the PCIe bridge controller 3 mediates data transfer between the EPs in the PCIe bridge controller 3. As a result, data transfer can be realized between the plurality of RCs (processors 21) connected to the PCIe bridge controller 3.

That is, each processor 21 is independently operated as RC of PCIe, and in the PCIe bridge controller 3, a device connected to each processor 21 is connected as EP, and data transfer between E is performed. Thereby, the problem caused by the device driver can be avoided, and the high speed data transfer can be operated as one system.

Further, since it is possible to transfer data between different processors 21 as long as it has a data communication function conforming to the PCIe standard, a processor to be used without having to worry about the presence or absence of a device driver and a support OS. It is possible to expand the 21 options.

Since each processor 21 is connected via the PCIe bridge controller 3 serving as an EP, it is not necessary to add a device driver for the RC at the EP end. Therefore, it is not necessary to develop a device driver, and a defect caused by adding a device driver does not occur.

In the information processing system 1, a general processor such as an ARM processor or an FPGA is required to operate as RC, and thus can be easily added as the processor 21 of the information processing system 1.

Since the PCIe bridge controller 3 is connected (communicated) by PCIe, high-speed transfer that cannot be realized by Ethernet can be realized. It is also possible to perform transmission/reception of high-definition video of 4K, 8K, etc. between processors and parallel calculation of large-scale big data.

Further, since a dedicated processor specialized for each function such as image processing and data search can be connected, it is possible to add functions and improve performance at low cost.

Furthermore, in the information processing system 1, it is not necessary to perform system virtualization or the like, and the system performance does not deteriorate due to system virtualization. Therefore, the information processing system 1 can also be applied to a system intended for high-load computation such as AI inference and image processing.

The disclosed technique is not limited to the above-described embodiment, and various modifications can be implemented without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selected or omitted as necessary, or may be appropriately combined.

For example, in the configuration shown in FIG. 17, the PCIe bridge controller 3 has eight slots 34-1 to 34-8, but it is not limited to this and various modifications can be implemented. That is, the PCIe bridge controller 3 may have seven or less slots or nine or more slots 34.

In the above-described embodiment, PCIe is taken as an example of the I/O interface of each unit, but the I/O interface is not limited to PCIe. For example, the I/O interface of each unit may be any technology capable of transferring data between the device (peripheral controller) and the processor by the data transfer bus. The data transfer bus may be a general-purpose bus that can transfer data at high speed in a local environment (for example, one system or one device) provided in one housing or the like. The I/O interface may be either a parallel interface or a serial interface.

The I/O interface may have a structure capable of point-to-point connection and serial transfer of data on a packet basis. The I/O interface may have a plurality of lanes in the case of serial transfer. The layer structure of the I/O interface may include a transaction layer that generates and decodes a packet, a data link layer that performs error detection and the like, and a physical layer that converts serial and parallel. Further, the I/O interface is the highest level of the hierarchy and has a root complex having one or more ports, end points which are I/O devices, switches for increasing ports, bridges for converting protocols, etc. May be included. The I/O interface may multiplex data to be transmitted and a clock signal by a multiplexer and transmit the multiplexed data. In this case, the receiving side may separate the data and clock signals with a demultiplexer.

1 Information Processing System 100 Information Processing Device 132 Transmission Unit 133 Reception Unit 136 Measurement Unit 137 Update Unit 300 Inference Processing Device 302 Coprocessor 303 Inference Application 304 Model File 305, 305i Middleware 1312a Calculation Unit 1312b Selection Unit

In one aspect, the information processing apparatus disclosed in the present application, when N is an integer of 2 or more, a receiving unit that receives resource information regarding a processing status from each of N processors, and the received N pieces of resource information. From the resource information of the processor, the conversion parameter for converting the value of the resource information of the own processor into the processing time is multiplied by a variable indicating the value of the resource information of the own processor and the value of the resource information of another processor. (N-1) products obtained by multiplying the contribution rate parameter indicating the degree of influence on the processing time in the own processor by the variable indicating the value of the resource information of the other processor for each of the (N-1) other processors. Based on the processing time of each of the N processors, the calculation unit calculating the processing time of the processing request by the application for each of the N processors. It has a selection part which selects one processor from this processor, and a transmission part which transmits the processing request to the one processor.

Claims (6)

When N is an integer of 2 or more, a receiving unit that receives resource information regarding the processing status from each of the N processors,
A calculation unit that applies the received resource information of the N processors to a calculation formula to calculate a processing time for a processing request by an application for each of the N processors;
A selection unit that selects one processor from the N processors based on the processing time of the N processors;
A transmitter for transmitting the processing request to the one processor;
Information processing device equipped with. A measuring unit that measures a processing time by the one processor for the processing request;
An update unit that performs multiple regression analysis based on the received resource information of the N processors and the measured processing time, and updates the calculation formula,
The information processing apparatus according to claim 1, further comprising: The calculation formula includes N parameters corresponding to the N processors,
The information processing apparatus according to claim 2, wherein the updating unit updates the N parameters in the calculation formula. The processing request includes a processing identifier for identifying processing content,
The calculation unit identifies the resource information of the N processors by the process identifier of N×M calculation formulas associated with a combination of different N processors and different M process contents. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is applied to N of the calculation formulas corresponding to the processed content. The information processing apparatus according to any one of claims 1 to 4, wherein each of the N processors is an inference processing apparatus that performs inference processing according to the processing request. An information processing apparatus according to any one of claims 1 to 5,
When N is an integer of 2 or more, N processors, each of which is communicably connected to the information processing device,
Information processing system equipped with.

Images