KR20200023239A - Electronic device and operating method for processing a neural network model by using a plurality of processors - Google Patents

Abstract

Provided is a method for processing a neural network model by using a plurality of processors in an electronic device. The method for processing a neural network model by using a plurality of processors in an electronic device comprises: allocating a plurality of layers included in a neural network model to at least one slice; allocating the at least one slice to a plurality of processors based on a processing time of each of the plurality of processors for each of the at least one slice; and processing a neural network model by using the plurality of processors based on the allocation result, wherein the processing time includes a switching time for receiving data required to process a current slice by a current processor from a previous processor processing a previous slice.

Description

Electronic device and operating method for processing a neural network model by using a multiple of processors}

The present disclosure relates to an electronic device that processes a neural network model using a plurality of processors, and a method of operating the same.

The electronic device may perform face recognition, voice recognition, image processing, and the like, using a neural network model-based deep learning technology, and provide the result to the user.

The electronic device may process the neural network model using a plurality of processors, for example, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a digital signal processor (DSP), and the like. The electronic device may process the neural network model using the plurality of processors by assigning a plurality of layers constituting the neural network model to the plurality of processors.

However, processing speed and accuracy may vary depending on the performance of each processor and the characteristics of each layer. Accordingly, there is a demand for a method of allocating a layer of a neural network model to be processed by each processor based on the processing capability of the plurality of processors and the characteristics of the layers.

SUMMARY An object of the present disclosure is to solve the above-described problem, and to provide an electronic device for processing a neural network model using a plurality of processors and a method of operating the same.

In addition, the present invention provides a computer-readable recording medium having recorded thereon a program for executing the method on a computer. The technical problem to be solved is not limited to the above technical problems, and other technical problems may exist.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure, the step of assigning a plurality of layers included in the neural network model to at least one slice; Assigning the at least one slice to the plurality of processors based on a processing time of each of the plurality of processors for each of the at least one slice; And processing the neural network model by using the plurality of processors, based on the assigned result, and processing the neural network model by using the plurality of processors. Includes the switching time taken for the current processor to receive data needed to process the current slice from the previous processor processing the previous slice.

In addition, a second aspect of the present disclosure includes a memory for storing a neural network model; Allocating a plurality of layers included in the neural network model to at least one slice, and assigning the at least one slice to the plurality of processors based on a processing time of each of the plurality of processors for each of the at least one slice, At least one processor configured to process the neural network model using the plurality of processors based on the assigned result; And an output unit configured to output a result of processing the neural network model, wherein the processing time includes a switching time taken by the current processor to receive data necessary for processing the current slice from a previous processor processing the previous slice. An electronic device can be provided.

In addition, the third aspect of the present disclosure may provide a recording medium storing a program for performing the method of the first aspect or the second aspect.

According to an embodiment of the present disclosure, the neural network model may be processed more quickly and accurately using a plurality of processors.

1 is a diagram illustrating an example of processing a neural network model in an electronic device according to an embodiment of the present disclosure.
2 is a diagram illustrating an example in which a layer of a neural network model is allocated to at least one slice, according to an exemplary embodiment.
3 is a block diagram illustrating an internal configuration of an electronic device according to an embodiment of the present disclosure.
4 is a block diagram illustrating an internal configuration of an electronic device according to an embodiment of the present disclosure.
5 is a flowchart illustrating a method of processing a neural network model using a plurality of processors according to an exemplary embodiment.
6 is a flowchart illustrating a method of allocating a slice to a plurality of processors according to an exemplary embodiment.
7 is a diagram illustrating an example of allocating a processor to a slice, according to an exemplary embodiment.
8 is a diagram illustrating an example of allocating a memory in a layer according to an embodiment.
9 is a flowchart illustrating a method of allocating memory for input / output data of a layer, according to an exemplary embodiment.
10 is a diagram illustrating an example of identifying a blob in a layer according to an embodiment.
11 illustrates an example of allocating memory to a blob of a neural network model including a blob in a layer according to an embodiment.
12 is a diagram illustrating an example in which a neural network model is processed by a plurality of processors according to an exemplary embodiment.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this means that it may further include other components, except to exclude other components unless otherwise stated.

Functions related to artificial intelligence according to the present disclosure are operated through a processor and a memory. The processor may consist of one or a plurality of processors. In this case, the one or more processors may be a general purpose processor such as a CPU, an AP, a digital signal processor (DSP), a graphics dedicated processor such as a GPU, a vision processing unit (VPU), or an artificial intelligence dedicated processor such as an NPU. One or more processors control to process the input data according to a predefined operating rule or artificial intelligence model stored in the memory. Alternatively, when one or a plurality of processors is an AI dedicated processor, the AI dedicated processor may be designed with a hardware structure specialized for processing a specific AI model.

The predefined action rule or artificial intelligence model is characterized by being made through learning. In this case, it is made through learning that a basic AI model is trained using a plurality of learning data by a learning algorithm, thereby creating a predefined action rule or AI model set to perform a desired characteristic (or purpose). It means load. Such learning may be made in the device itself in which the artificial intelligence according to the present disclosure is performed, or may be made through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples.

The artificial intelligence model may consist of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operation through an operation between a calculation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by learning results of the AI model. For example, the plurality of weights may be updated to reduce or minimize a loss value or a cost value acquired in the AI model during the learning process. Artificial neural networks may include deep neural networks (DNNs), for example, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), Deep Q-Networks, and the like, but are not limited to the above examples.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an example of processing a neural network model 110 in an electronic device 1000 according to an embodiment of the present disclosure.

Referring to FIG. 1, the electronic apparatus 1000 may process the neural network model 110 by using a processor 1300 including a plurality of processors 1310, 1320, and 1330.

According to an embodiment of the present disclosure, the electronic apparatus 1000 may process the neural network model 110 by using a plurality of processors through a compilation step and a task execution step. For example, the electronic apparatus 1000 may include a compile step of allocating a task to be performed by a plurality of processors, and a task processing step of performing a task using a plurality of processors according to the allocation result in the compilation step. Can be done.

Compiling may be performed by at least one processor among the plurality of processors.

According to an embodiment, the compiling step may include an operation of allocating a memory for storing data used to perform a task, as well as an operation of allocating a task to be performed by a plurality of processors.

The electronic device 1000 according to an embodiment may be implemented in various forms. For example, the electronic device 1000 described in the present specification may include a digital camera, a smart phone, a laptop computer, a tablet PC, an electronic book terminal, a digital broadcasting terminal, and personal digital assistants (PDAs). There may be, but is not limited to, a Portable Multimedia Player (PMP), navigation, MP3 player, and the like. The electronic device 1000 described herein may be a wearable device that can be worn by a user. Wearable devices may be accessory devices (eg, watches, rings, cuffs, ankle bands, necklaces, glasses, contact lenses), head-mounted-devices (HMDs), textile or apparel-integrated devices (e.g., Electronic clothing), a body-attachable device (eg, a skin pad), or a living implantable device (eg, an implantable circuit), but is not limited thereto. Hereinafter, for convenience of description, the case where the electronic device 1000 is a smartphone will be described as an example.

The electronic apparatus 1000 according to an embodiment may perform various operations using the neural network model 110. For example, the neural network model 110 may include an artificial intelligence model such as a deep neural network (DNN), a recurrent neural network (RNN), a convolutional neural network (CNN), and the like. The neural network model 110 according to an exemplary embodiment may include various types of artificial intelligence models.

According to an embodiment, the electronic apparatus 1000 may perform various operations such as recognizing data or generating new data using the at least one neural network model 110, and provide the result to the user. have.

The neural network model 110 according to an embodiment may include a plurality of layers. Each layer included in the neural network model 110 may include at least one function for processing data. For example, the neural network model 110 including the CNN model may be configured by a combination of various types of layers such as a convolution layer, a max pooling layer, and a platen layer.

For example, in the convolution layer, an operation of extracting feature information about input data may be performed. In the max pooling layer, an operation of extracting main data from input data may be performed. In the platen layer, an operation of converting a value of input data into a one-dimensional value may be performed. In addition to the above-described example, the neural network model 110 may include various types of layers.

The electronic device 1000 according to an embodiment may include a plurality of processors 1310, 1320, and 1330. For example, the electronic apparatus 1000 may include various types of processors such as a CPU, a GPU, an NPU, and a DSP. Each processor may have different performance and features. For example, CPUs are slower than other processors, but feature high accuracy and energy efficiency.

Table 1 below shows the relative characteristics of each of the processors of the CPU, GPU, NPU, and DSP.

CPU GPU DSP NPU speed Slow speed Very fast Very fast accuracy Toughness Toughness Sensitive Sensitive Energy efficiency good Bad Very good Very good Ease of performing new tasks facility Somewhat difficult Very difficult impossible Code inspection facility Somewhat difficult Very difficult impossible

The characteristics of each processor described in Table 1 are merely examples, and may vary according to various factors such as a characteristic of a task processed in each processor or a current state of the processor.

According to an embodiment of the present disclosure, one processor among the processors 1310, 1320, and 1330 of the electronic apparatus 1000 may be allocated to each layer included in the neural network model 110 in the compilation step. The electronic apparatus 1000 allocates a processor to each layer based on the performance of each of the processors 1310, 1320, and 1330 as shown in Table 1, and the characteristics of each layer included in the neural network model 110. Can be.

According to an embodiment of the present disclosure, the speed and accuracy of the processor for processing the layer may be different according to the characteristics of the layer to be processed. The electronic apparatus 1000 according to an embodiment may allocate a processor to be processed by each layer of the plurality of processors by determining whether a processor is suitable for processing each layer of the neural network model 110. have.

According to an embodiment, when a processor to which a layer is to be processed is allocated, the electronic apparatus 1000 may process the neural network model 110 using a plurality of processors according to the allocation result.

2 is a diagram illustrating an example in which a layer of the neural network model 110 is allocated to at least one slice, according to an exemplary embodiment.

The nodes shown in FIG. 2 represent some layers that make up the neural network model 110. According to an embodiment of the present disclosure, the neural network model 110 may be processed in the electronic apparatus 1000 by transmitting the result of the operation performed in each layer in the direction of the arrow to the next layer.

According to an embodiment of the present disclosure, a plurality of layers of the neural network model 110 may be allocated to at least one slice, and each layer may be allocated to a processor in slice units. According to an embodiment, the electronic apparatus 1000 may allocate a layer to a processor in a slice unit including at least one layer instead of a layer unit, thereby reducing the amount of computation generated by the allocation operation.

According to an embodiment of the present disclosure, the electronic apparatus 1000 may allocate the plurality of layers to slices by determining at least one layer of the plurality of layers constituting the neural network model 110 as a slice point. According to an embodiment, each layer may be allocated to a different slice based on the slice point.

For example, the electronic apparatus 1000 may sequentially determine whether to determine each layer from layers "conv2d_9" to "activation_11" as a slice point. In addition, it may be determined whether the slice point is sequentially from "conv2d_7" to "activation_8" after "activation_11". It may be determined whether "average_pooling2d_1" to "activation_6" are the same as the slice points in order.

According to an embodiment of the present disclosure, the electronic apparatus 1000 according to an embodiment may include whether each layer is a point where a plurality of layers are branched, is a point at which a plurality of layers are combined, whether a task that is processed by the same processor is included, and high accuracy The slice point may be determined based on at least one of whether or not includes the required operation. According to an embodiment of the present disclosure, as in the above-described various criteria, the current layer may be determined as a slice point according to whether the current layer may be switched to a processor different from the processor of the previous layer. In addition to the above-described examples, the slice point may be determined according to various criteria.

For example, since "conv2d_9" is one of a plurality of layers branched from "max_pooling2d_2", it may be determined as a slice point as it corresponds to a branching point of the plurality of layers. Thus, "conv2d_9" may belong to a new slice 1 210 that is different from the slice previously determined.

The layers of "activation_9" to "actionvation_11" may be assigned to the slice 210 determined above by not being determined as the slice point according to the above-described criteria for determining the slice point.

 Since "conv2d_7" is one of a plurality of layers branched from "max_pooling2d_2" like "conv2d_9", it is determined as a slice point and may belong to a new slice 220. Layers of "conv2d_7" to "actionvation_8" may be assigned to slice 2 220 determined above, as not determined as the slice point.

Since "average_pooling2d_1" is one of a plurality of layers branched from "max_pooling2d_2" like "conv2d_9", it is determined as a slice point and may belong to a new slice 3 (230). The layer of "conv2d_12" may be assigned to slice 3 230 determined above, as it is not determined as the slice point.

"activation_12" may be determined as a slice point as it does not include a task that can be processed in the same processor as the layers of slice 3 230, or includes a task that requires high accuracy, and thus a new slice 4 240 may be determined. Can belong.

For example, when no processor capable of processing "activation_12" and "conv2d_12", which are layers included in slice 3230, exists in the processor capable of processing "activation_12", it is allocated to slice 3230. "Activation_12" cannot belong to slice 3 230 because the processor cannot process "activation_12". According to an embodiment, as processors are allocated in units of slices, layers belonging to the same slice may be processed by the same processor.

As another example, even when "activation_12" includes a task requiring high accuracy, "activation_12" may be determined as a slice point so that a processor suitable for "activation_12" may be allocated without being affected by another layer.

Since "conv2d_6" is one of a plurality of layers branched from "max_pooling2d_2" like "conv2d_9", it is determined as a slice point and may be allocated to a new slice 5 (250). The layer of "activation_6" may be assigned to slice 5 250 determined above, as the slice point is not determined.

According to an embodiment, one processor may be allocated to each slice to which at least one layer is allocated. According to an embodiment, the electronic apparatus 1000 may determine a processor to process each slice, based on a processing time of each processor for each slice.

The processing time according to an embodiment of the present invention is a switching time required for a current processor to process a task of at least one layer included in a current slice, and data required for a current processor to process a current slice from another processor. May include time. For example, the data required to process the current slice received from another processor may include data input to a layer included in the current slice. Also, another processor may be a previous processor that processes a previous slice and outputs data input to the current slice.

According to an embodiment, the switching time may indicate a time taken for data of a processing result of the previous slice to be transferred to the current processor when the previous processor processing the previous slice and the current processor processing the current slice are different.

For example, when the first processor 1310 is allocated to the slice 3 230, the switching time of the second processor 1320 with respect to the slice 4 240 is that the output data of “conv2d_12” is determined by the first processor ( The time transferred from the 1310 to the second processor 1320 may be represented. In addition, the switching time of the third processor 1330 with respect to the slice 4 240 may indicate a time when the output data of “conv2d_12” is transferred from the first processor 1310 to the third processor 1330.

According to an exemplary embodiment, the switching time may include not only a time at which data is transferred from the first processor 1310 to the second processor 1320, but also a time at which data transferred according to the data format of the second processor 1320 is converted. It may further include.

In addition, the switching time according to an embodiment may be increased as the size of the transmitted data increases.

In addition, the switching time according to an embodiment may be determined as 0 when data between the processors is not transferred when the previous processor processing the previous slice and the current processor processing the current slice are the same. For example, the switching time of the first processor 1310 for the slice 4 240 may be determined to be 0 by assigning the first processor 1310 to the slice 3 230.

Accordingly, according to an embodiment, not only the time required for processing a layer operation of a slice by each processor, but also based on a switching time for transferring data between processors so that a plurality of slices may be processed through a plurality of processors. The processor to process each slice may be determined.

3 is a block diagram illustrating an internal configuration of an electronic apparatus 1000 according to an exemplary embodiment.

4 is a block diagram illustrating an internal configuration of an electronic device 1000 according to an exemplary embodiment.

Referring to FIG. 3, the electronic apparatus 1000 may include a processor 1300, a memory 1700, and an output unit 1200. However, not all components illustrated in FIG. 3 are essential components of the electronic apparatus 1000. The electronic device 1000 may be implemented by more components than those illustrated in FIG. 3, or the electronic device 1000 may be implemented by fewer components than those illustrated in FIG. 3.

For example, as shown in FIG. 4, the electronic device 1000 may include a user input unit 1100 in addition to the processor 1300, the memory 1700, and the output unit 1200. ), The sensing unit 1400, the communication unit 1500, and the A / V input unit 1600 may be further included.

The user input unit 1100 means a means for a user to input data for controlling the electronic apparatus 1000. For example, the user input unit 1100 includes a key pad, a dome switch, a touch pad (contact capacitive type, pressure resistive layer type, infrared sensing type, surface ultrasonic conduction type, and integral type). Tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, and the like, but are not limited thereto.

According to an embodiment of the present disclosure, the user input unit 1100 may receive a user input required to perform an operation by the neural network model 110. According to an embodiment of the present disclosure, the electronic apparatus 1000 may process the neural network model 110 using a plurality of processors and output a processing result according to a user input received from the user input unit 1100.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210, an audio output unit 1220, and a vibration motor 1230. have.

The display unit 1210 displays and outputs information processed by the electronic apparatus 1000.

Meanwhile, when the display unit 1210 and the touch pad form a layer structure to form a touch screen, the display unit 1210 may be used as an input device in addition to the output device. The display unit 1210 may include a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, and a three-dimensional display. 3D display, an electrophoretic display. The electronic apparatus 1000 may include two or more display units 1210 according to the implementation form of the electronic apparatus 1000.

The sound output unit 1220 outputs audio data received from the communication unit 1500 or stored in the memory 1700.

The vibration motor 1230 may output a vibration signal. In addition, the vibration motor 1230 may output a vibration signal when a touch is input to the touch screen.

According to an embodiment, a result of processing the neural network model 110 by the processor 1300 may be output in various forms through the display unit 1210, the sound output unit 1220, and the vibration motor 1230. .

The processor 1300 typically controls the overall operation of the electronic apparatus 1000. For example, the processor 1300 executes programs stored in the memory 1700 to thereby execute a user input unit 1100, an output unit 1200, a sensing unit 1400, a communication unit 1500, and an A / V input unit 1600. ) Can be controlled overall. The electronic device 1000 may include a plurality of processors 1300.

The processor 1300 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The command may be provided from the memory 1700 to the processor 1300 or may be received through the communication unit 1500 and provided to the processor 1300. For example, the processor 1300 may be configured to execute an instruction according to a program code stored in a recording device such as a memory.

The processor 1300 according to an embodiment allocates a plurality of layers included in the neural network model 110 to at least one slice, and allocates a processor to process each slice among the plurality of processors included in the processor 1300. can do. In addition, based on the assigned result, the processor 1300 may process the neural network model 110.

According to one embodiment, a processor for each slice may be allocated based on the processing time of the processor for the slice. The processing time according to an embodiment may include a switching time required for transferring data between processors.

According to an embodiment of the present disclosure, the processor 1300 may identify at least one layer included in at least one slice allocated to the first processor among the plurality of processors. In addition, the processor 1300 may identify at least one blob each representing at least one data among data input to the identified layer, output data, and data temporarily stored in the identified layer, and the data of the identified blob Allocate memory for storing

According to an embodiment, the memory for the current blob may be allocated by determining whether the use period of the previous blob is terminated before the data of the current blob is generated according to the processing order of each layer. Further, the size of the allocated memory may be determined based on the largest data size among data sizes of at least one blob to which the same memory is allocated.

The sensing unit 1400 may detect a state of the electronic device 1000 or a state around the electronic device 1000 and transmit the detected information to the processor 1300.

The sensing unit 1400 may include a geomagnetic sensor 1410, an acceleration sensor 1420, a temperature / humidity sensor 1430, an infrared sensor 1440, a gyroscope sensor 1450, a position sensor (Eg, GPS) 1460, barometric pressure sensor 1470, proximity sensor 1480, and RGB sensor (illuminance sensor) 1490, but are not limited thereto.

According to an embodiment, the information detected by the sensing unit 1400 may be used as input information of the neural network model 110 or may be used to update the neural network model 110. In addition to the above-described example, the information detected by the sensing unit 1400 may be used according to various methods for processing the neural network model 110.

The communicator 1500 may include one or more components that allow the electronic device 1000 to communicate with the server 2000 or an external device (not shown). For example, the communicator 1500 may include a short range communicator 1510, a mobile communicator 1520, and a broadcast receiver 1530.

The short-range wireless communication unit 1510 includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared ray ( IrDA, an infrared data association (WIRD) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (ultra wideband) communication unit, Ant + communication unit and the like, but may not be limited thereto.

The mobile communication unit 1520 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call call signal, or a text / multimedia message.

The broadcast receiving unit 1530 receives a broadcast signal and / or broadcast related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an embodiment of the present disclosure, the electronic device 1000 may not include the broadcast receiver 1530.

According to an embodiment of the present disclosure, the communication unit 1500 may transmit a result of processing the neural network model 110 to an external device. Alternatively, the communication unit 1500 may receive information necessary for processing the neural network model 110 from an external device.

The A / V input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 and a microphone 1620. The camera 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a photographing mode. The image captured by the image sensor may be processed by the processor 1300 or a separate image processor (not shown). The microphone 1620 receives an external sound signal and processes the external sound signal into electrical voice data.

According to an embodiment, the image data or the audio data obtained by the A / V input unit 1600 may be used as input information of the neural network model 110 or used to update the neural network model 110. In addition to the above-described examples, the image data or the audio data may be used according to various methods for processing the neural network model 110.

The memory 1700 may store a program for processing and controlling the processor 1300, and may store data input to or output from the electronic device 1000.

The memory 1700 according to an exemplary embodiment may store information about the neural network model 110 and information about performance of a plurality of processors. For example, the information about the performance of a plurality of processors may include information about a processor's processing speed for a layer, information about a layer that can be processed by each processor, and information about time to switch data between different processors. have. In addition to the above-described examples, the information about the performance of the plurality of processors may include various types of information required for allocating slices to the processors.

The memory 1700 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM (RAM, Random Access Memory) Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic Disk It may include at least one type of storage medium of the optical disk.

Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions. For example, the programs stored in the memory 1700 may be classified into a UI module 1710, a touch screen module 1720, a notification module 1730, and the like. .

The UI module 1710 may provide a specialized UI, GUI, and the like that interoperate with the electronic device 1000 for each application. The touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the processor 1300. The touch screen module 1720 according to some embodiments may recognize and analyze a touch code. The touch screen module 1720 may be configured as separate hardware including a controller.

Various sensors may be provided inside or near the touch screen to detect a touch or proximity touch of the touch screen. An example of a sensor for sensing a touch of a touch screen is a tactile sensor. The tactile sensor refers to a sensor that senses the contact of a specific object to the extent that a person feels or more. The tactile sensor may sense various information such as the roughness of the contact surface, the rigidity of the contact object, the temperature of the contact point, and the like.

The user's touch gesture may include tap, touch and hold, double tap, drag, pan, flick, drag and drop, and swipe.

The notification module 1730 may generate a signal for notifying occurrence of an event of the electronic device 1000.

5 is a flowchart illustrating a method of processing a neural network model using a plurality of processors according to an exemplary embodiment.

Referring to FIG. 5, in operation 510, the electronic device 1000 may allocate a plurality of layers constituting the neural network model 110 to at least one slice. The electronic apparatus 1000 according to an embodiment may allocate a plurality of layers to at least one slice by determining whether each layer is a slice point.

For example, if the first layer is determined to be a slice point, a new first slice may be generated, and the first layer may be assigned to the first slice instead of the previous slice. The previous slice may indicate a slice in which at least one layer is already allocated before determining whether the first layer is a slice point. On the other hand, if the first layer is not determined as the slice point, the first layer may be allocated to the previous slice.

According to an embodiment, after the first layer is assigned to the first slice, if the second layer is determined as a slice point, a second slice may be newly created and the second layer may be allocated to the newly created second slice. . On the other hand, if the second layer is not determined slice point, the second layer may be assigned to the first slice.

In operation 520, the electronic apparatus 1000 may allocate a plurality of layers configuring the neural network model 110 to at least one slice, and then assign each slice to a plurality of processors. According to an embodiment, the plurality of processors may process the neural network model 110 by processing at least one layer included in a slice allocated to each processor.

According to an embodiment of the present disclosure, the electronic apparatus 1000 may allocate each slice to the plurality of processors based on the processing time required for each processor to process the slice. The processing time according to an embodiment may further include not only the time required for each processor to process the slice, but also the switching time required to receive data from another processor to process the slice.

In operation 530, the electronic apparatus 1000 may process the neural network model 110 using a plurality of processors based on the allocation result of operation 520.

According to an embodiment, as processors to process each slice are allocated to each other, a plurality of slices arranged in parallel in the neural network model 110 may be simultaneously processed by a plurality of processors. Therefore, according to an exemplary embodiment, each layer of the neural network model 110 is processed faster than when each layer of the neural network model 110 is aligned in a line according to a topological sorting method, and processed sequentially. Can be.

6 is a flowchart illustrating a method of allocating a slice to a plurality of processors according to an exemplary embodiment.

Referring to FIG. 6, in operation 610, the electronic apparatus 1000 may obtain information about a slice s and a processor p.

In an embodiment, the slice s and the processor p may represent one slice s and one processor p of at least one slice and a plurality of processors. According to an embodiment of the present disclosure, in operation 610, a processor that may process layers included in the slice s among the plurality of processors included in the electronic device 100 may be determined as the processor p.

For example, the information about the slice s may include information about the operation of at least one layer included in the slice s. In addition, information about processor p may include information about a task that may be performed by processor p, information about the time that the task is processed by processor p, information about the switching time it takes to transfer data between different processors, and so on. It may include information about the performance of p. In addition to the above-described examples, the information about the slice s and the processor p may include various information that may be used to predict the processing time and the accuracy of the processing result of the processor p processing the slice s. .

In operation 620, when the slice s is processed by the processor p based on the information obtained in operation 610, the electronic apparatus 1000 may predict the accuracy of the processing result of the slice s and determine whether the predicted accuracy is less than or equal to the reference value. It can be determined.

According to an embodiment, when the predicted accuracy is less than or equal to the reference value, in step 650, it may be determined whether there is a processor other than the processor p. The other processor described above may be a processor capable of processing slice s among a plurality of processors of the electronic apparatus 1000 that have not yet been determined to allocate slice s.

According to an embodiment, in operation 660, the electronic apparatus 1000 may identify another processor capable of processing the slice s. The electronic apparatus 1000 may repeatedly perform the operations of steps 610 to 640 with respect to the identified other processor.

In operation 630, the electronic apparatus 1000 may estimate a time required for processing a task of the slice s by the processor p based on the information obtained in operation 610. For example, the electronic apparatus 1000 may determine the processing time of the processor p for each task type, and based on the determined processing time, determine the processing time of the processor p for the work of at least one layer included in the slice s. Can be obtained. In addition to the above-described examples, the processing time of the slice s by the processor p can be estimated in various ways.

In operation 640, the electronic apparatus 1000 may estimate a switching time taken for the input data of the slice s to be transferred to the processor p from another processor, based on the information obtained in operation 610.

For example, the input data of slice s may be delivered to processor p by a processor that processes the previous slice from which the input data is output. The previous slice according to an embodiment may indicate a slice for outputting data to be input to the slice s. In addition, the switching time may represent a time taken for the input data of the slice s to be transferred to the processor p in the processor processing the previous slice. On the other hand, when the processor processing the previous slice is the same as the processor p, the switching time may be determined to be zero.

According to an embodiment, the switching time may be estimated based on a time taken for data to be transferred between different processors of the electronic apparatus 1000.

In operation 650, the electronic apparatus 1000 may determine another slice of the processors of the electronic apparatus 1000, for which slice s, at least one of accuracy, processing time, and switch time is not determined according to operations 620 to 640. It can be determined whether there exists. If there is another processor, the electronic apparatus 1000 may repeat the operations of steps 610 to 640 with respect to the other processor identified in step 660.

In addition, with respect to slice s, if there is no other processor for which at least one of accuracy, processing time, and switch time is not determined according to steps 620 through 640, the electronic apparatus 1000 may select slice s in step 670. The processor may be allocated to one processor of the plurality of processors.

According to an embodiment of the present disclosure, a processor to process slice s may be determined based on a processing time and a switch time of each processor for slice s. For example, the processor having the smallest sum of processing time and switch time may be determined as a processor to process slice s.

According to an embodiment of the present disclosure, a processor having an accuracy equal to or less than a reference value is not determined as a processor to process slice s of step 670 and may be excluded. Therefore, even if the processing time or the switch time is short, the processor with low accuracy for the processing result may not be allocated to the processor to process the slice s.

According to an embodiment of the present disclosure, after the processor for the slice s is allocated, the operations of steps 610 to 670 may be repeatedly performed for the other slices to which the processor is not allocated in addition to the slice s.

7 is a diagram illustrating an example of allocating a processor to a slice, according to an exemplary embodiment.

Referring to FIG. 7, 710 illustrates an example of a graph of allocating a processor to a slice according to a processing time and a switching time. In the graph of 710, the node represents the time taken to process the slice's work, and the arrow represents the time taken to switch data from the previous node to the current node.

According to an embodiment of the present disclosure, unlike a method in which processors are sequentially allocated for each slice as in the method illustrated in FIG. 6, a processor to determine a slice may be determined according to a path having the smallest sum of processing times at each node. Can be. The processing time according to an embodiment may include a switching time as well as a time required for a processor to process a slice at each node.

For example, slices arranged in series in the neural network model 110 among at least one slice may include a plurality of nodes 711, 721, 722, 723, 731, 732, 733, representing a combination of different slices and processors. 741 and 742 may be allocated to the plurality of processors based on a path having the smallest sum of processing times among the plurality of paths generated by connecting the slices according to the arrangement order of the slices.

The input data 711 input to the slice 1 may be processed by a CPU which is a basic processor. According to the direction of the arrow, the input data 711 may be switched to one of the processors, such as CPU, GPU and NPU. As the input data 711 is switched to any one processor, the input data 711 may be input to a layer included in the slice 1 and processed at one of the nodes 721, 722, and 723.

According to one embodiment, the NPU is excluded from the processor allocation operation for slice 2 as the accuracy of the result of processing slice 2 by the NPU is below a reference value, or because slice 2 includes a layer that cannot be processed by the NPU. Can be.

According to an embodiment, as shown in the graph 710, the allocation method of the CPU, the GPU, and the NPU for the slices 1 to 3 may be determined as one of a total of 36 paths. Processors for slices 1 to 3 may be allocated according to a path having a minimum sum of processing time and switching time of each node among the 36 paths.

750 is a graph illustrating an example in which a processor is allocated to a slice according to an embodiment. A processor may be allocated to each slice 751, 752, 753, 754, 755, 756 including at least one layer.

GPU 16 and GPU 32, described in 750, represent 16-bit and 32-bit GPUs, respectively, and may be processed with different processors in one embodiment.

According to one embodiment, the slices 752 and 754 and 755 may be processed simultaneously in parallel as they are allocated by different processors. Therefore, according to an embodiment, some layers arranged in parallel in the neural network model 110 may be processed simultaneously in parallel by a plurality of processors, thereby further improving processing speed.

According to one embodiment, slices of 755 and 756 assigned the same processor may be processed by the NPU after each layer is serially aligned so that they may be processed simultaneously by the NPU. For example, each layer may be sorted according to a topological sort method and then processed simultaneously by the NPU.

According to an embodiment of the present disclosure, at least one slice may be allocated to the plurality of processors according to a combination of the method of 710 of FIG. 7 and the method of FIG. 6.

In the neural network model 110, slices arranged in series, for example, slices of 751, 752, and 753, are represented by a plurality of nodes representing a combination of different slices and processors, such as the method illustrated in FIG. 7. A processor may be allocated based on a path having the smallest sum of processing times among a plurality of paths generated by concatenating according to an arrangement order of slices.

In addition, the slices arranged in parallel in the neural network model 110, for example, slices of 754, 755, 756, may be processed according to the method shown in FIG. 6, the processing time, the switching time, and the accuracy of the processor for the slice. Based on this, a processor may be allocated.

8 is a diagram illustrating an example of allocating a memory in a layer according to an embodiment.

According to an embodiment of the present invention, a convolutional layer among the layers may include a 1 × 1 convolution 812, a DC (Depthwise Convolution, 813 to 815), and a 1 × 1 convolution 816 from channel 1 as shown in FIG. It may include a task performed repeatedly to N. Operation for channel 1 may be performed in step 810, and operation for channel 2 may be performed in step 820 after step 810 is performed. Channel N may likewise be performed in step 830 after the steps preceding channel N are performed.

According to an embodiment, the channel may correspond to each of a plurality of data having the same size in the input data input to the layer. For example, in the first channel, first data of the plurality of data may be processed, and in the second channel, second data may be processed. Therefore, when a plurality of image data having the same size is input to the layer, the plurality of input data may be sequentially processed for each channel by the number of the plurality of input data.

According to one embodiment, in the convolution layer, each task may be processed sequentially. For example, after 1 × 1 convolution 812 is performed on input data of all channels, DCs 813 to 815 may be performed based on a result of 1 × 1 convolution 812 being performed. In addition, the 1 × 1 convolution 816 may be performed based on the result of performing the DCs 813 to 815.

According to an embodiment, instead of being sequentially processed for each of the above-described tasks, the tasks may be sequentially performed from channel 1 to channel N. FIG. For example, if there are 24 56x56 input data, there are 24 channels, and operations from channel 1 to channel 24 may be sequentially performed for each channel.

According to an embodiment of the present disclosure, when operations are sequentially performed for each channel, a 294KB size memory for storing 24 56x56 size data 811, 821, and 831 input to the convolutional layer may be allocated. For example, when the input data is image data, a memory for storing a plurality of image data of the same size may be allocated.

According to an embodiment, when a job is sequentially processed for each channel, instead of allocating a memory for storing data for all channels, only a memory for storing data of one channel may be allocated. Therefore, when a job is sequentially processed for each channel, a memory space allocated for data storage may be reduced, as compared with when the job is sequentially processed for each job.

In operation 810, the 1 × 1 convolution 812, the DC 813, 814, 815, and the 1 × 1 convolution 816, which are operations of the channel 1, may be sequentially performed. The 1x1 convolution 812 performed in step 810 may be performed on the first data of 24 data of size 56x56 of 811 and the first data of 24 data of size of 1x24 of 812.

According to one embodiment, for the 1x1 convolution 812 processing, instead of all 24 data of 1x24 size of 812 being allocated to memory, only the first data of 24 data of size 1x24 of 812, which is used only in step 810, is used. This memory can be allocated. Twenty four 1x24 size data used for the 1x1 convolution 812 processing may be values stored in advance in the electronic apparatus 1000.

In step 820, without additional memory allocation, second data of size 822 of 1 × 24 may be stored in the same space as the memory of the data of 812 allocated in step 810.

Thus, according to one embodiment, 1x1 convolutions 812, 822, and 832 are performed on each of the 24 channels, and instead of allocating a memory having a size of 14KB in which all 24 data of 1x24 size can be stored, Only memory of size 0.09KB can be allocated in which one piece of data of size 1x24 can be stored. According to an embodiment, the allocated memory space may be reused for 1 × 1 convolution processing in the current channel for 1 × 1 convolution processing in the previous channel. Thus, the memory space required to process the layer in each channel can be reduced.

Memory allocation for data processed at DCs 813, 814, 815 may also be allocated for data processed at 813-815, rather than all data. 813 is data obtained by performing the 1 × 1 convolution 812 of 812 and may include 56 × 56 data. Also, 814 is kernel data of 3x3 size used for DC. Kernel data of 3x3 size may be a value previously stored in the electronic apparatus 1000 for DC processing. 815 is data obtained as a result of the DC processing, and may include 56 × 56 data.

According to one embodiment, for processing DC 813, 814, 815 in channel 1, 813 56x56 size data (12.25KB), 814 3x3 size kernel data (0.03KB), 815 56x56 size data Memory of size 24.53KB may be allocated for storing (12.25KB). In step 820 of the channel 2, the data of 823, 824, and 825 may be stored in the same space as the memory of the data of 813, 814, and 815 allocated in the previous step 810 without additional memory allocation.

Allocations for data processed in the 1 × 1 convolution 816 may likewise be allocated memory for data processed at 816, rather than all data. At 816, 144 data of 1 × 1 size may be used in the 1 × 1 convolution 816. Therefore, a memory of size 0.09 KB, in which 144 pieces of data of size 1 × 1 may be stored, may be allocated. In operation 820, the data of 826 may be stored in the same space as the memory of the data of 816 allocated in operation 810, without additional memory allocation.

As a result of the 1 × 1 convolution 816 being performed, at 817, 24 data of 56 × 56 size may be obtained. According to an embodiment of the present disclosure, a 294KB size memory capable of storing 24 pieces of 56x56 size data acquired at 817 may be allocated. In step 820, as in 817, 24 data having a size of 56 × 56 may be obtained as the 1 × 1 convolution 826 is performed at 827. Then, even in channels 3 to N, 24 data of 56x56 size for each channel can be obtained according to the result of 1x1 convolution, DC, and 1x1 convolution.

According to an embodiment, by concatenating 24 pieces of 56x56 data output from each channel, the final result for the convolutional layer may be obtained.

Thus, a total of 612 KB of memory space, which is the sum of the at least one data used for processing 810 on channel 1, may be allocated. According to an embodiment of the present disclosure, in channels 2 to N 820 and 830, operations of each channel are sequentially performed, so that 612 KB of memory space allocated in channel 1 may be reused without additional memory allocation. Therefore, all the tasks of the channel 1 through the channel N (810, 820, 830) of the convolutional layer are processed, but only 612 KB of memory space can be allocated and used.

According to an embodiment, the size of the memory space used to process the layer is not limited to the above-described example, and various sizes of memory space may be allocated and used according to the size of the processed data.

In addition, FIG. 8 illustrates an example in which input data is processed by a convolution layer, but according to an embodiment, a plurality of input data may be input to various types of layers as well as the convolution layer. . According to an embodiment of the present disclosure, as in the case where the memory space is allocated to the convolutional layer described above, the memory space for another type of layer may be as large as necessary to process a job in the first channel among the plurality of channels. Memory may be allocated for job processing on the plurality of channels of the layer. In the other channels except the first channel, the memory allocated for the operation of the first channel can be reused without additional memory allocation.

9 is a flowchart illustrating a method of allocating memory for input / output data of a layer, according to an exemplary embodiment.

Memory according to an embodiment may be allocated to blobs corresponding to input / output data of a layer, respectively. The neural network model 110 according to an embodiment may include a layer on which a task is performed and a blob corresponding to each data input and output to the layer.

According to an embodiment of the present disclosure, a blob for intermediate data generated due to an internal function of the layer may be identified as well as input / output data of the layer. Accordingly, the electronic apparatus 1000 may allocate the memory in consideration of the blob for the intermediate data.

In one embodiment, memory allocation refers to an operation of pre-allocating a space of a memory where data is to be stored in a compilation step. In the memory allocation according to an embodiment, after the blobs to be allocated to the same memory space are determined, the size of each memory space may be determined based on the data size of the blobs.

Referring to FIG. 9, in operation 910, the electronic apparatus 1000 may identify a layer allocated to each processor. According to an embodiment of the present disclosure, the electronic apparatus 1000 identifies at least one layer included in at least one slice allocated to a first processor among a plurality of processors according to a method of allocating slices to a plurality of processors. can do. The electronic apparatus 1000 according to an embodiment may perform memory allocation according to an embodiment of each processor.

In operation 920, the electronic apparatus 1000 may determine an execution order of at least one layer allocated to one processor. According to an embodiment of the present disclosure, the electronic apparatus 1000 may determine a processing order of at least one layer according to a phase alignment method. The present invention is not limited to the above-described example, and the processing order for one layer can be determined according to various methods.

In operation 930, the electronic apparatus 1000 may identify an internal function included in each layer, and in operation 940, the blob including data temporarily stored in the layer including the internal function may be generated by the identified internal function. Can be identified. According to an embodiment, the blob input and output to each layer may be a blob already identified according to the information about the structure of the neural network model 110.

For example, if a layer includes two or more internal functions, there may be a blob for intermediate data that is temporarily stored between the two internal functions. In addition to the above-described examples, the blob may be identified in operation 940 except for blobs for data input / output for a layer among blobs for data input or output through an internal function of the layer, according to an exemplary embodiment.

According to an embodiment of the present disclosure, for a blob temporarily stored due to an internal function of a layer, a memory in which data is to be stored while performing a task, such as input / output data of the layer, must be allocated. Accordingly, the electronic apparatus 1000 may perform memory allocation in consideration of a blob stored temporarily in the layer as well as a blob for the input / output data of the layer.

In operation 950, the electronic apparatus 1000 may allocate a blob for input / output data for each layer and at least one of the blobs identified for the internal function of each layer identified in operation 940. have.

According to an embodiment of the present disclosure, the memory for the current blob may be allocated based on whether the use period of the previous blob is terminated before the data of the current blob is finished according to the processing order of each layer. For example, if the use interval of the previous blob ends before the data of the current blob is generated, the same memory as the memory allocated to the blob may be allocated to the current blob.

The use section according to an embodiment may be determined based on a section in which blob data is used by at least one layer. For example, the usage section may indicate a section in which the data of the blob is read or written by performing the operation of each layer from the memory space (ex. Buffer) in which the data is stored.

In addition, the interval in which the data of the blob is used by the at least one layer for determining the use interval may be determined according to the lifetime of the blob. For example, the lifetime of the blob may be a preset value or may be determined based on the execution interval of a function of the layer on which the blob is processed. In addition to the above-described examples, the use interval of the blob may be determined according to various methods.

For example, when the operation by the first layer is finished, the electronic apparatus 1000 may no longer use the temporarily stored data by the input data of the first layer and the internal function of the first layer. Therefore, the temporary storage data and the output data of the second layer operated after the first layer may use a memory space in which the temporary data stored by the input data of the first layer and the internal function of the first layer are stored.

The electronic apparatus 1000 according to an embodiment may determine the size of the memory as the largest data size among data sizes of at least one blob to which the same memory is allocated.

For example, for blob 1 indicating input data of the first layer and blob 2 indicating temporary stored data of the first layer, output data of blob 3 and second layer indicating temporary storage data of the second layer, respectively It may be determined that the same memory as the blob 4 indicating is allocated. For example, a first memory may be allocated to blobs 1 and 3 and a second memory may be allocated to blobs 2 and 4.

When the data size of blob 1 is 1 kb and the data size of blob 3 is 2 kb, the first memory may be allocated to a size of 2 kb, which is the largest data size among the blobs allocated to the first memory. In addition, when the data size of the blob 2 is 4 kb and the data size of the blob 4 is 5 kb, the second memory may be allocated to a size of 5 kb, which is the largest data size among the blobs allocated to the second memory.

10 is a diagram illustrating an example of identifying a blob in a layer according to an embodiment.

Referring to FIG. 10, internal functions “im2col” 1011 and “sgemm” 1013 may be identified with respect to a “conv2d_65” 1010 layer among a plurality of layers included in the neural network model 110. Also, the internal function blob b ₂ (1012) is a temporary storage between (1011, 1013) can be identified.

Blob b ₁ according to one embodiment 1020, the blob and b ₃ (1030) is a blob that represents the input data and output data for each "conv2d_65" (1010) layer.

Therefore, according to one embodiment, for the "conv2d_65" (1010) layers, blob b ₁ (1020), blob b ₂ (1012), and blob b ₃ (1030) are identified, the memory for the identified blobs Allocation can be performed.

According to one embodiment, the memory allocated for each blob may be allocated based on whether the usage interval of the previous blob is terminated before the data of the current blob is generated.

For example, the blob b ₁ (1020) is, and may be more longer be used starting when the "sgemm" (1013) is processed after the processing of the "im2col" (1011) termination. B blob using blocks ₁ 1020, it may be ended before the data of the blob b ₃ (1030) is generated. Therefore, according to one embodiment, the blob b ₁ (1020) and the blob b ₃ (1030), may be assigned the same memory. For example, "sgemm" when 1013 is processed, the data in the blob b ₁ after 1020 the previously stored blobs b ₁ 1020 is deleted in the same memory space allocated, blob b ₃ (1030), the Can be stored.

In addition, the blob is b ₂ (1012), there is a blob b ₁ (1020) Memory The use of the blob that previously present in the interval assigned to the blob ending before generation of the blob b ₂ (1012) may be re-allocated.

11 illustrates an example of allocating memory to a blob of the neural network model 110 including a blob in a layer according to an embodiment.

According to an embodiment, blobs 1101, 1103, 1104, 1105, 1106, 1108, 1109, 1115, and 1111 representing input / output data of a plurality of layers may be identified. In addition, blobs 1102 and 1107 representing data that may be temporarily stored due to the internal function of each layer may be identified.

In FIG. 11, hatched blocks represent layers on which operations can be performed, and blocks that are not hatched represent blobs representing data.

According to an embodiment of the present disclosure, the data blob 1101, the col_buffer1 blob 1102, and the conv1 blob 1103 may be allocated to different memories as the use intervals of the blobs overlap each other. For example, data blob 1101, col_buffer1 blob 1102 and conv1 blob 1103 may be allocated to b1, b2, and b3 memories, respectively.

In addition, the data blob 1101 and the col_buffer1 blob 1102 may be no longer used after the relu1 layer processed after the conv1 layer after being used in the conv1 layer. Accordingly, the use intervals of the data blob 1101 and the col_buffer1 blob 1102 may be terminated before the relu1 blob 1104 is generated. The relu1 blob 1104 may be allocated to a b1 or b2 memory, which is the same memory as the data blob 1101 or the col_buffer1 blob 1102. In an embodiment, it is assumed that relu1 blob 1104 is allocated to b1 memory.

In addition, the norm1 blob 1105 may be allocated to a memory in which the blob in which the use interval ends is stored before the norm1 blob 1105 is generated. For example, norm1 blob 1105 may be allocated to b2 or b3 memory. In an embodiment, assume that norm1 blob 1105 is allocated to b2 memory.

In addition, before the pool1 blob 1106 is generated, the pool1 blob 1106 may be allocated to a b1 or b2 memory, which is a memory in which the blob in which the use interval ends is stored. In an embodiment, it is assumed that the pool1 blob 1106 is allocated to the b1 memory.

In addition, the col_buffer2 blob 1107 may be allocated to a b2 or b3 memory, which is a memory in which the blob in which the use interval ends is stored before the col_buffer2 blob 1107 is generated. In an embodiment, assume that col_buffer2 blob 1107 is allocated to b2 memory.

In addition, the conv2 blob 1108 may be allocated to the b3 memory, which is a memory in which the blob in which the use interval ends is stored, before the conv2 blob 1108 is generated.

In addition, before the relu2 blob 1109 is generated, the relu2 blob 1109 may be allocated to a b1 or b2 memory, which is a memory in which the blob in which the use interval ends is stored. In an embodiment, it is assumed that relu2 blob 1109 is allocated to b1 memory.

In addition, the norm2 blob 1115 may be allocated to a b2 or b3 memory, which is a memory in which the blob in which the use period ends is stored before the norm2 blob 1115 is generated. In an embodiment, assume that norm2 blob 1110 is allocated to b2 memory.

In addition, before the pool2 blob 1111 is generated, the pool2 blob 1111 may be allocated to a b1 or b2 memory, which is a memory in which the blob in which the use interval ends is stored. In an embodiment, it is assumed that the pool2 blob 1110 is allocated to the b1 memory.

According to one embodiment, after assigning each blob to b1, b2 or b3 memory according to the above-described method, based on the largest value of the data size of the blobs allocated to each memory, b1, b2 and b3 memory The size of can be determined.

According to an embodiment, the data size of each blob may have a value as shown in Table 2 below.

Blob names type Data size (MB) Allocated memory data I / O data 0.59 b1 col_buffer1 Internal data 4.19 b2 conv1 I / O data 1.11 b3 relu1 I / O data 1.11 b1 norm1 I / O data 1.11 b2 pool1 I / O data 0.27 b1 col_buffer2 Internal data 6.67 b2 conv2 I / O data 0.71 b3 relu2 I / O data 0.71 b1 norm2 I / O data 0.71 b2 pool2 I / O data 0.17 b1

According to an embodiment, 1.11 MB, which is the largest value among the data blobs 1101, relu1 blobs 1104, pool1 blobs 1106, relu2 blobs 1109, and pool2 blobs 1111, allocated to the b1 memory, is selected. Based on the size of the b1 memory may be determined.

Further, the size of the b2 memory may be determined based on 6.67 MB, which is the largest value among the data sizes of the col_buffer1 blob 1102, the norm1 blob 1105, the col_buffer2 blob 1107, and the norm2 blob 1110 allocated to the b2 memory. .

In addition, the size of the b3 memory may be determined based on 1.11 MB, which is the largest value among the data sizes of the colv1 blob 1103 and the conv2 blob 1108 allocated to the b3 memory.

Therefore, according to an embodiment, a memory space of 8.89 MB in which b1, b2, and b3 memory sizes are combined may be used to store the blobs illustrated in FIG. 11.

If the memory is allocated only for the blob for the input / output data without the blob for the internal data, the memory allocation space may increase because the memory for storing the blob for the internal data must be reallocated.

However, according to an embodiment, as the memory is allocated in consideration of the blobs for the input / output data as well as the blobs for the internal data generated by the internal functions of the layer, a smaller memory space may be allocated.

12 is a diagram illustrating an example in which a neural network model is processed by a plurality of processors according to an exemplary embodiment.

According to the method of allocating layers to a plurality of processors according to an embodiment, the layers 1 to 4 1201, 1202, 1203, and 1204 may be processed by a CPU, a GPU, a GPU, and a CPU, respectively.

The blobs for the input / output data and the blobs for the internal data of the layer 1 1201 and the layer 4 1204 processed by the CPU may be allocated memory according to the memory allocation method according to an embodiment. For example, at least one of the memories allocated to a blob of layer 1 1201 (eg, a blob containing input / output data or internal data of layer 1) may be reassigned to the blob of layer 4 1204. Can be.

According to an embodiment, when blobs for a plurality of internal data exist in the same layer, the same memory may be allocated between the blobs.

According to an embodiment, blobs 1 to 4 (1205, 1206, 1207, and 1208) may exist in layer 1 1201 for different internal data generated by different internal functions in layer 1 1201. . For example, the output data by the internal functions 1 to 4 arranged in series may be represented by blobs 1 to 4 (1205, 1206, 1207, and 1208), respectively. Also, the data in blob 4 1208 can be used as input data for internal function 5.

According to one embodiment, after the operation of the internal function 1 of the blob 1 1205 is finished, the operation of the internal function 3 of the blob 3 1207 may be performed, so that the use interval of the blob 1 1205 is set to the blob 3 1207 may end before it is generated. Thus, memory allocated to blob 1 1205 may be reallocated to blob 3 1207.

Also, after the operation of the internal function 2 of the blob 2 1206 is finished, the operation of the internal function 4 of the blob 4 1208 may be performed, so that the use interval of the blob 2 1206 is determined by the blob 4 1208. It can be terminated before it is created. Thus, memory allocated to blob 2 1206 may be reallocated to blob 4 1208.

According to an embodiment of the present disclosure, the neural network model may be processed more quickly and accurately using a plurality of processors.

One embodiment may also be implemented in the form of a recording medium containing instructions executable by a computer, such as a program module executed by the computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, or program modules, and includes any information delivery media.

In addition, in this specification, “unit” may be a hardware component such as a processor or a circuit, and / or a software component executed by a hardware component such as a processor.

The above description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims (15)

In the method for processing a neural network model using a plurality of processors in an electronic device,
Allocating a plurality of layers included in the neural network model to at least one slice;
Assigning the at least one slice to the plurality of processors based on a processing time of each of the plurality of processors for each of the at least one slice; And
Based on the assigned result, processing the neural network model using the plurality of processors,
Wherein the processing time comprises a switching time taken for the current processor to receive data necessary for processing the current slice from the previous processor processing the previous slice. The method of claim 1,
By determining at least one layer of the plurality of layers as a slice point, the plurality of layers are assigned to the at least one slice,
The slice point may include whether each of the plurality of layers is a branching point of a plurality of layers, whether a plurality of layers are combined, whether a plurality of layers are combined, a task that can be processed by the same processor, and high accuracy is required. The method is determined based on at least one of whether to include the task. The method of claim 1, wherein the at least one slice is
A plurality of nodes representing a combination of different slices and processors are allocated to the plurality of processors based on a path having the smallest sum of processing times among the plurality of paths generated by connecting the at least one slice in an arrangement order. , Way. The method of claim 1,
The plurality of input data input to any one of the plurality of layers is sequentially processed for each channel by the number of the plurality of input data in the layer,
For a plurality of channels of the layer, as much memory as is needed to process a job in a first channel of the plurality of channels is allocated. The method of claim 1,
Identifying at least one layer included in at least one slice assigned to a first processor of the plurality of processors;
Identifying at least one blob each representing at least one of data input to the identified layer, output data, and data temporarily stored within the identified layer; And
Allocating a memory for storing data of the identified blob. 6. The method of claim 5, wherein allocating the memory
Determining a processing order of the identified layers; And
Based on the determined processing order, allocating memory for the current blob by determining whether the usage interval of the previous blob ends before data for the current blob is generated. 6. The method of claim 5, wherein allocating the memory
Determining the size of the memory based on the largest data size of the data sizes of at least one blob to which the same memory is allocated. An electronic device for processing neural network models,
A memory for storing the neural network model;
Allocating a plurality of layers included in the neural network model to at least one slice, and assigning the at least one slice to the plurality of processors based on a processing time of each of the plurality of processors for each of the at least one slice, At least one processor for processing the neural network model based on the assigned result; And
The neural network model includes an output unit for outputting a processed result.
Wherein the processing time comprises a switching time taken for the current processor to receive data necessary for processing the current slice from a previous processor processing the previous slice. The method of claim 8, wherein the plurality of layers are assigned to the at least one slice by determining at least one layer of the plurality of layers as a slice point.
The slice point may include whether each of the plurality of layers is a branching point of a plurality of layers, whether a plurality of layers are combined, whether a plurality of layers are combined, a task that can be processed by the same processor, and high accuracy is required. The electronic device is determined based on at least one of including a task. The method of claim 8, wherein the at least one slice,
A plurality of nodes representing a combination of different slices and processors are allocated to the plurality of processors based on a path having the smallest sum of processing times among the plurality of paths generated by connecting the at least one slice in an arrangement order. , Electronic device. The method of claim 8,
The plurality of input data input to any one of the plurality of layers is sequentially processed for each channel by the number of the plurality of input data in the layer,
The memory of the plurality of channels of the layer is allocated an amount of memory necessary for processing a job in a first channel of the plurality of channels. 10. The system of claim 8, wherein the at least one processor is
Identify at least one layer included in at least one slice allocated to a first processor of the plurality of processors,
Identifying at least one blob respectively representing at least one of data input to the identified layer, output data, and data temporarily stored inside the identified layer,
Allocating a memory for storing data of the identified blob. 13. The system of claim 12, wherein the at least one processor is
Allocating memory for the current blob by determining a processing order of the identified layers and determining, based on the determined processing order, whether the usage interval of the previous blob is terminated before data of the current blob is generated, Electronic devices. 13. The system of claim 12, wherein the at least one processor is
And determine the size of the memory based on the largest data size of the data sizes of at least one blob to which the same memory is allocated. A computer-readable recording medium having recorded thereon a program for implementing the method of any one of claims 1 to 7.

Images