KR102161690B1 - Electric apparatus and method for control thereof - Google Patents

Abstract

An electronic device is disclosed. The electronic device includes a memory in which a weight matrix trained by a convolution neural network (CNN) is stored, a processor that converts matrices representing color components constituting each pixel of the image into a single integrated matrix when an image is received, An image processor that performs filtering on an image by using the integrated matrix converted by the processor, and a display that outputs a filtering result of the image processor under control of the processor.

Description

Electronic device and its control method {ELECTRIC APPARATUS AND METHOD FOR CONTROL THEREOF}

The present invention is an artificial intelligence (AI) system that simulates functions such as cognition and judgment of the human brain using machine learning algorithms such as deep learning, and electronic devices and control methods that provide an image processing model among its applications. In more detail, the present invention relates to an electronic device using a CNN (Convolution Neural Network) trained image processing model and a control method thereof.

The artificial intelligence (AI) system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, the machine learns, judges, and becomes smarter. As artificial intelligence systems are used, their recognition rate improves and users' tastes can be understood more accurately, and existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

Artificial intelligence technology consists of machine learning (deep learning) and component technologies using machine learning.

Machine learning is an algorithm technology that classifies/learns the features of input data by itself, and element technology is a technology that simulates functions such as cognition and judgment of the human brain using machine learning algorithms such as deep learning. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology that recognizes and applies/processes human language/text, and includes natural language processing, machine translation, dialogue systems, question and answer, and speech recognition/synthesis. Visual understanding is a technology that recognizes and processes objects like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, and image improvement. Inference prediction is a technique that logically infers and predicts information by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data creation/classification), knowledge management (data utilization), and the like. Motion control is a technology that controls autonomous driving of a vehicle and movement of a robot, and includes movement control (navigation, collision, driving), operation control (behavior control), and the like.

For example, the artificial intelligence system can learn various images and be applied to the generation of an image processing model according to the learning result. Also, when a new image is received through a process similar to learning based on the generated image model, the image may be filtered and classified and provided.

However, as the dimension of the image processing model increases, the amount of data required for processing increases exponentially, and there is a problem that a delay time occurs due to data transmission and reception between the processor and the image processor.

The present invention is in accordance with the above-described necessity, and an object of the present invention is to provide an electronic device and control methods thereof that provide filtering results without performance degradation using a weight matrix and a bias matrix trained by a convolution neural network (CNN). Is in.

In order to achieve the above object, the electronic device according to an embodiment of the present invention configures each pixel of the image when a CNN (Convolution Neural Network) trained weight matrix is stored and an image is received. A processor that converts matrices representing color components to one unified matrix, an image processor that performs filtering on the image using the unified matrix converted by the processor, and a filtering result of the image processor under control of the processor And a display that outputs, wherein the image processor obtains a convolution matrix by performing matrix multiplication of the unified matrix and the weight matrix, and filters characteristics of each pixel region of the image using the convolution matrix, and the The weight matrix is a matrix obtained by integrating a plurality of CNN-trained matrices in an interleaved form, and the integrated matrix is a matrix having the weight matrix and the number of rows and columns that can be calculated.

In addition, the image processor may include a plurality of cores for performing matrix multiplication between each row constituting the unified matrix and a corresponding column of the weight matrix in parallel.

Here, the memory further includes a bias matrix, and the image processor may add the bias matrix to the convolution matrix and the convolution matrix.

Also, the memory further includes a first bias matrix and a second bias matrix, and the image processor may selectively add one of the first and second bias matrices to the convolution matrix.

Here, the image processor, by applying an activation function to a convolution matrix in which one of the first and second bias matrices is summed, obtains a plurality of output layers, and is fully-connected (FC) based on the plurality of output layers. ) A layer is obtained, and characteristics of each region can be identified according to the class score of each region included in the FC layer.

Here, the image processor may obtain the FC layer by summing values of positions corresponding to each other in the plurality of output layers.

Also, the image processor may obtain the FC layer by comparing values of positions corresponding to each other in the plurality of output layers and collecting values having the largest values.

Also, the processor may control the image processor to perform the filtering by collectively transmitting parameter values and a command used for the filtering to the image processor together with the integrated matrix.

In addition, the processor may control the image processor to repeatedly perform the filtering a plurality of times.

Meanwhile, in a method for controlling an electronic device according to an embodiment of the present disclosure, when an image is received, converting matrices representing color components constituting each pixel of the image into one unified matrix, a convolution neural network (CNN) ) Filtering the image using the trained weight matrix and the integrated matrix, and outputting the filtering result, wherein the filtering comprises: the integrated matrix And obtaining a convolutional matrix by performing matrix multiplication on the weighting matrix and filtering characteristics of each pixel region of the image using the convolutional matrix, wherein the weighting matrix is a plurality of CNN-trained matrices Is an interleaved matrix, and the integrated matrix is a matrix having the weight matrix and the number of rows and columns that can be calculated.

In addition, it may include a plurality of cores for performing matrix multiplication between each row constituting the unified matrix and a corresponding column of the weight matrix in parallel.

Here, performing the filtering may include adding a bias matrix to the convolution matrix.

Further, the performing of the filtering includes the step of selectively summing one of the first and second bias matrices to the convolution matrix, wherein the first bias matrix is a matrix composed of 1, and the second bias matrix May be a matrix composed of zeros.

Here, the performing of the filtering may include obtaining a plurality of output layers by applying an activation function to a convolution matrix in which one of the first and second bias matrices is summed, and FC based on the plurality of output layers. It may include obtaining a (fully-connected) layer and identifying characteristics of each region according to the class score of each region included in the FC layer.

In addition, in the obtaining of the FC layer, the FC layer may be obtained by summing values of positions corresponding to each other in the plurality of output layers.

Further, in the obtaining of the FC layer, the FC layer may be obtained by comparing values of positions corresponding to each other in the plurality of output layers and collecting values having the largest values.

Further, it includes the step of collectively transmitting parameter values and commands used for the filtering to an image processor together with the integrated matrix, and the filtering may be performed by the image processor.

In addition, the step of performing the filtering may be performed by repeating the filtering a plurality of times.

Meanwhile, in a non-transitory computer-readable medium storing a computer command so that the electronic device performs an operation when executed by a processor of the electronic device according to an embodiment of the present disclosure, the operation is, when an image is received, Converting matrices representing color components constituting each pixel of the image into one unified matrix, filtering the image using a CNN (Convolution Neural Network) trained weight matrix and the unified matrix And outputting the filtering result, wherein the filtering comprises: obtaining a convolutional matrix by performing matrix multiplication of the unified matrix and the weighting matrix, and using the convolutional matrix Filtering the characteristics of each pixel region of the image, wherein the weight matrix is a matrix obtained by integrating a plurality of CNN-trained matrices in an interleaved form, and the integrated matrix is calculated with the weight matrix It may be a matrix having this possible number of rows and columns.

According to the various embodiments of the present invention as described above, the electronic device provides while minimizing the delay time due to data transmission/reception between the processor and the image processor using a weight matrix and a bias matrix trained by a convolution neural network (CNN). I can.

1 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure.
3 is a diagram for describing matrix multiplication according to an embodiment of the present disclosure.
4 is a diagram for describing an activation function according to an embodiment of the present disclosure.
5 is a flowchart illustrating an operation method for obtaining a filtering result according to an embodiment of the present disclosure.
6 is a diagram illustrating a command for obtaining a filtering result according to an embodiment of the present disclosure.
7 is a diagram illustrating a command for obtaining a filtering result according to an embodiment of the present disclosure.
8 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention.

The terms used in the present specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure have selected general terms that are currently widely used as possible while taking functions of the present disclosure into consideration, but this may vary according to the intention or precedent of a technician working in the field, the emergence of new technologies, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, not the name of a simple term.

Since the embodiments of the present disclosure may apply various transformations and may have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiment, it is to be understood to include all conversions, equivalents, or substitutes included in the disclosed spirit and technical scope. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by terms. The terms are only used for the purpose of distinguishing one component from another component.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "comprise" are intended to designate the existence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other It is to be understood that the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance the possibility of being excluded.

In the present disclosure, a "module" or "unit" performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module except for the "module" or "unit" that needs to be implemented with specific hardware and implemented as at least one processor (not shown). Can be.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the exemplary embodiments described herein. In addition, in the drawings, parts not related to the description are omitted in order to clearly describe the present disclosure, and similar reference numerals are attached to similar parts throughout the specification.

1 is a block diagram illustrating an electronic device 100 according to an embodiment of the present disclosure.

As illustrated in FIG. 1, the electronic device 100 includes a memory 110, a processor 120, an image processor 130, and a display 140.

The electronic device 100 according to an embodiment of the present disclosure may be a device capable of artificial intelligence learning. For example, a user terminal device, a display device, a set-top box, a tablet PC (tablet personal computer), a smart phone (smart phone), an e-book reader, a desktop PC (desktop PC) ), a laptop PC, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, and the like. Alternatively, the electronic device 100 may refer to a system in which a cloud computing environment is built. However, the present disclosure is not limited thereto, and the electronic device 100 may be any device capable of artificial intelligence learning.

The memory 110 may store a weight matrix trained by a convolution neural network (CNN). Here, CNN is a multilayer neural network with a special connection structure designed for voice processing and image processing. In particular, the CNN can variously filter images through preprocessing on pixels and recognize the characteristics of the image.

The memory 110 may store an image processing model including a matrix trained by CNN as basic data including a plurality of images. The matrix learned from the basic data may be variously called a filter, a kernel, a weight matrix, etc., but hereinafter, it will be collectively referred to as a weight matrix for convenience of description.

The memory 110 according to an embodiment of the present disclosure may further include a bias matrix. Here, the bias matrix means a matrix composed of constants. The electronic device 100 according to an embodiment may obtain a filtering result using a bias matrix in a process of filtering an image. Here, the bias matrix is a configuration for preventing overfitting of an image processing model including a CNN-trained weight matrix. Here, overfitting means that when the electronic device 100 obtains the characteristics of the image by applying a weight matrix to the image included in the basic data, the characteristics of the acquired image and the actual characteristics of the image coincide, whereas the other image ( For example, the characteristics of an image obtained by applying a weighting matrix to an image received in real time) means that a case is frequently issued that does not match the actual characteristics of the image.

The processor 120 overall controls the operation of the electronic device 100.

According to an embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON), but is not limited thereto, and the central processing unit ( central processing unit (CPU)), microcontroller unit (MCU), micro processing unit (MPU), controller, application processor (AP), or communication processor (CP), ARM processor In addition, the processor 140 may include one or more of, or may be defined by a corresponding term. In addition, the processor 140 may be implemented as a system on chip (SoC) or a large scale integration (LSI) with a built-in processing algorithm, or an FPGA ( Field Programmable gate array) can also be implemented.

When an image is received, the processor 120 may convert matrices representing color components constituting each pixel of the image into a single integrated matrix. Here, the image refers to various types of images such as images and contents that the electronic device 100 receives in real time. The color component may mean a pixel value corresponding to each pixel of an image. For example, when each pixel of the received image is expressed through 8 bits, one pixel has one of a total of 256 color components ranging from 0 to 255. Accordingly, the processor 120 may convert the received image into a matrix using each pixel value.

If the received image is multi-channel data, the processor 120 according to an embodiment of the present disclosure may obtain a matrix for each channel and convert the obtained plurality of matrices into one integrated matrix. As an example, the received image is 3-channel data of R, G, and B, and the processor 120 may obtain a matrix corresponding to each of R, G, and B. A detailed description of this will be given in FIG. 3.

The processor 120 according to an embodiment of the present disclosure may control the image processor 130 to perform filtering on a received image. In addition, the display 140 may be controlled to output the filtering result.

The image processor 130 may be a special purpose processor for processing graphic data. In particular, the image processor 130 may perform filtering on an image using the converted integrated matrix.

The image processor 130 is a GPU (Graphic Processing Unit) and may be implemented as a component different from the processor 120, but is not limited thereto. Various embodiments of the present disclosure may be performed by the processor 120. As an example, it goes without saying that the graphic processing unit of the processor 120 may perform filtering on a received image according to various embodiments of the present disclosure.

The image processor 130 generates a screen including various objects using an operation unit (not shown) and a rendering unit (not shown). Based on the received control command, the operation unit calculates attribute values such as coordinate values, shape, size, color, etc. in which each object is to be displayed according to the layout of the screen. The rendering unit generates screens of various layouts including objects based on the attribute values calculated by the calculation unit. The screen generated by the rendering unit is displayed in the display area of the display 140.

Also, the image processor 130 is a component that processes image data. The image processor 130 may perform various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on image data.

In particular, the image processor 130 may obtain a convolution matrix by performing convolution on the unified matrix and the weight matrix. Here, convolution means that the weight matrix performs matrix multiplication while shifting the matrix for the image for each region, and calculates a result value according to the matrix multiplication. For example, if the matrix for the received image is 7 × 7 and the weight matrix is 3 × 3, the weight matrix performs matrix multiplication while shifting for each area in the matrix for the image, and obtains a 3 × 3 output matrix. I can. Hereinafter, the output matrix obtained according to convolution will be collectively referred to as a convolution matrix.

The size of the matrix is an example and is not limited thereto. For example, the matrix may have various sizes according to the received image. In addition, it goes without saying that the size may be variously adjusted as padding and stride are performed in the process of performing convolution. Here, padding means filling a specific pixel value by a preset size on all sides of the received image. When padding is performed, the matrix for the image and the convolution matrix may have the same size. Stride refers to the shift interval of the weight matrix. For example, if starride (stride=3), the image processor 130 may perform convolution while shifting by three spaces in the matrix.

However, in order to perform convolution, the image processor 130 according to an embodiment of the present disclosure may not shift the weight matrix and the image matrix for each area. The transformed integrated matrix may be a matrix having rows and columns capable of multiplying the weight matrix and the matrix. The weight matrix stored in the memory 110 may be an integrated matrix in an interleaved form. Here, the interleaved matrix may be a matrix having rows and columns capable of multiplying the transformed integrated matrix and the matrix.

For example, if the received image is a 2D 3x3 matrix, the processor 110 may convert it into a 4x8 integrated matrix. The image processor 130 may obtain a convolution matrix by multiplying a 4×8 integrated matrix with a weight matrix stored in the memory 110 and a matrix. Here, the weight matrix may be an interleaved form capable of performing matrix multiplication with a 4×8 integrated matrix, that is, an 8×2 matrix. The image processor 130 may obtain a 4×2 convolution matrix by performing matrix multiplication between the unified matrix and the weight matrix. A detailed description of this will be given in FIG. 3.

The image processor 130 according to an embodiment of the present disclosure may include a plurality of cores. The image processor 130 may perform matrix multiplication between each row constituting the unified matrix and a corresponding column of the weight matrix in parallel using a plurality of cores. Convolution between the unified matrix and the weight matrix may mean performing matrix multiplication between the unified matrix and the weight matrix, or applying a filter or kernel to an image.

For example, it may be assumed that the image processor 130 performs matrix multiplication between a 4x8 integrated matrix and an 8x2 weight matrix. A matrix multiplication between one row of the unified matrix and one column of the weight matrix may be performed by a first core, and a matrix multiplication between one row of the unified matrix and two columns of the weight matrix may be performed by a second core. In this way, a total of eight matrix multiplications may be performed in parallel by each of the first to eighth cores. Accordingly, the time required to obtain the convolution matrix can be reduced. As an example, NVIDIA®'s GPU may contain 3,584 CUDA® cores, and 3,584 arithmetic operations can be performed in parallel.

The image processor 130 according to an embodiment of the present disclosure may add a bias matrix to a convolution matrix. As an example, the bias matrix may be a matrix having a preset value of 1×1. The image processor 130 may add a preset value included in the bias matrix to all elements included in the convolution matrix.

As another example, the image processor 130 may selectively apply any one of the first and second bias matrices to the convolution matrix. As an example, the first bias matrix may be a 1×1 matrix composed of 1s, and the second bias may be a 1×1 matrix composed of 0s. The image processor 130 may selectively add one of the first and second bias matrices to the convolution matrix based on various purposes.

The image processor 140 according to an embodiment of the present disclosure may apply an activation function to a convolution matrix in which a bias matrix is added. Here, various types of functions such as an Activation Function Identity Function, Logistic Sigmoid Function, Hyperbolic Tangent (tanh) Function, ReLU Function, and Leaky ReLU Function may be applied. For example, the image processor 130 may apply the RELU function max(0,x) to all elements.

The image processor 140 may obtain a plurality of output layers by applying an activation function, and obtain a fully-connected (FC) layer based on the output layer. Here, the FC layer is an output layer, and characteristics of each region can be identified according to the class score of each region included in the FC layer. As an example, the class score means an amount activated after an activation function is applied, and the image processor 130 may identify a category corresponding to each activation amount. Here, the identified category may mean a characteristic of each area. For example, according to the class score of each region included in the FC layer, objects (eg, a car, a ship, a cat, etc.) included in each region may be identified.

The image processor 130 according to an embodiment of the present disclosure may obtain an FC layer by summing values of positions corresponding to each other in a plurality of output layers.

As another example, the image processor 130 may obtain the FC layer by comparing values of positions corresponding to each other in a plurality of output layers and collecting values having the largest values.

The display 140 is a display of various types, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a liquid crystal on silicon (LCoS), or digital light processing (DLP). Can be implemented. However, the present invention is not limited thereto, and of course, it may be implemented as various types of displays capable of displaying images.

In particular, the display 140 may output the filtering result of the image processor 130 under the control of the processor 120.

2 is a block diagram illustrating a detailed configuration of an electronic device 100 according to an embodiment of the present disclosure.

According to FIG. 2, the electronic device 100 includes a memory 110, a processor 120, an image processor 130, a display 140, a communication unit 150, a user interface unit 160, and an audio processing unit 170. Include. Among the components shown in FIG. 2, detailed descriptions of parts that overlap with the components shown in FIG. 1 will be omitted.

The processor 120 overall controls the operation of the electronic device 100 by using various programs stored in the memory 110.

Specifically, the processor 120 includes the RAM 121, the ROM 122, the main CPU 123, the graphic processing unit 124, the first to n interfaces 125-1 to 125-n, and the bus 126. Include.

The RAM 121, the ROM 122, the main CPU 123, the graphic processing unit 124, the first to n interfaces 125-1 to 125-n, and the like may be connected to each other through the bus 126.

The first to nth interfaces 125-1 to 125-n are connected to the various components described above. One of the interfaces may be a network interface that is connected to an external device through a network.

The main CPU 123 accesses the memory 110 and performs booting using the O/S stored in the memory 110. Then, various operations are performed using various programs stored in the memory 110.

The ROM 122 stores an instruction set for booting the system, and the like. When the turn-on command is input and power is supplied, the main CPU 123 copies the O/S stored in the memory 110 to the RAM 121 according to the instruction stored in the ROM 122, and executes the O/S. Boot. When booting is completed, the main CPU 123 copies various application programs stored in the memory 110 to the RAM 121 and executes the application programs copied to the RAM 121 to perform various operations.

The processor 120 according to an embodiment of the present disclosure may collectively transmit parameter values and a command used for the image processor 130 to perform filtering to the image processor 130 together with an integrated matrix. Accordingly, a delay time due to data transmission/reception between the processor 120 and the image processor 130 may be minimized. A detailed description of this will be given in FIGS. 5 to 7.

The processor 120 according to an embodiment of the present disclosure may control the image processor to repeatedly perform filtering a plurality of times.

Meanwhile, the operation of the processor 120 described above may be performed by a program stored in the memory 110.

The memory 110 stores various data such as an O/S (Operating System) software module for driving the electronic device 100, an image processing model including a weight matrix and a bias matrix, a CNN learning module, and the like.

The memory 110 may be implemented as an internal memory such as ROM or RAM included in the processor 120, or may be implemented as a memory separate from the processor 120. In this case, the memory 110 may be implemented in the form of a memory embedded in the electronic device 100 according to the purpose of data storage, or may be implemented in a form of a memory that is detachable to the electronic device 100. For example, in the case of data for driving the electronic device 100, it is stored in a memory embedded in the electronic device 100, and in the case of data for an extended function of the electronic device 100, the electronic device 100 is detachable. It can be stored in memory whenever possible. Meanwhile, in the case of a memory embedded in the electronic device 100, a memory that is implemented in a form such as a nonvolatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD), and can be attached to the electronic device 100 In the case of, it may be implemented in a form such as a memory card (eg, micro SD card, USB memory, etc.), an external memory (eg, USB memory) connectable to a USB port.

The communication unit 150 is a component that communicates with various types of external devices according to various types of communication methods. The communication unit 150 includes a Wi-Fi chip 151, a Bluetooth chip 152, a wireless communication chip 153, an NFC chip 154, and the like. The processor 120 communicates with various external devices using the communication unit 150.

The WiFi chip 151 and the Bluetooth chip 152 perform communication in a WiFi method and a Bluetooth method, respectively. In the case of using the Wi-Fi chip 151 or the Bluetooth chip 152, various types of connection information such as SSID and session key may be first transmitted and received, and then various types of information may be transmitted and received after a communication connection using the same. The wireless communication chip 153 refers to a chip that performs communication according to various communication standards such as IEEE, zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evoloution (LTE). The NFC chip 154 refers to a chip operating in a Near Field Communication (NFC) method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz, and 2.45 GHz.

The processor 120 may receive an image processing model including an image, a weight matrix, a bias matrix, and an activation function from an external device through the communication unit 150.

The user interface unit 160 receives various user interactions. Here, the user interface unit 160 may be implemented in various forms according to an implementation example of the electronic device 100. For example, the user interface unit 160 may be a button provided in the electronic device 100, a microphone receiving a user's voice, a camera detecting a user's motion, or the like. Alternatively, when the electronic device 100 is implemented as a touch-based electronic device, the user interface unit 160 may be implemented in the form of a touch screen forming a layer structure with a touch pad. In this case, the user interface unit 160 can be used as the display 140 described above.

The audio processing unit 160 is a component that processes audio data. The audio processing unit 160 may perform various processing such as decoding, amplification, noise filtering, and the like for audio data.

3 is a diagram for describing matrix multiplication according to an embodiment of the present disclosure.

Referring to FIG. 3, if the received image is multidimensional, a plurality of matrices 10 for an image representing color components constituting each pixel of the image may be obtained. For example, if the received image is two-dimensional, two matrices for the image may be obtained. The processor 120 may convert a plurality of matrices 10 for an image into one integrated matrix 10-1.

The image processor 130 may convert the weight matrix 20 into an interleaved matrix 20-1 in order to perform matrix multiplication between the integrated matrix 10-1 and the weight matrix 20. However, the electronic device 100 according to an embodiment of the present invention may pre-store the weight matrix 20 as an interleaved matrix 20-1. Accordingly, the image processor 130 does not need to convert the weight matrix 20 to the interleaved matrix 20-1, and performs matrix multiplication using the previously stored interleaved matrix 20-1. I can.

For example, the processor 120 may convert a 3×3×2 matrix 10 into a 4×8×1 integrated matrix 10-1. The interleaved matrix 20-1 has a size of 8×4×1. The image processor 130 can perform matrix multiplication by using the previously stored interleaved matrix 20-1 without changing the size of the weight matrix 20 from 2 × 2 × 2 to 8 × 2 × 1. have.

Meanwhile, convolution between the matrix 10 and the weight matrix 20 for an image requires a shift of the weight matrix 20 and has a disadvantage in that the computation process is complicated. The computation time and resources required to obtain the output layer 30 are high.

The matrix multiplication between the integrated matrix 10-1 and the interleaved matrix 20-1 is performed only by matrix multiplication of the rows of the integrated matrix 10-1 and the columns of the interleaved matrix 20-1, so the convolution matrix The computation time and resources required to obtain (30-1) are small.

The image processor 130 according to an embodiment of the present disclosure may include a plurality of cores. Each of the plurality of cores may perform a matrix multiplication of the rows of the integrated mattress 10-1 and the columns of the interleaved matrix 20-1 in parallel.

The image processor 130 according to an embodiment of the present disclosure may obtain the output layer 30 based on the convolution matrix 30-1.

Hereinafter, an embodiment in which a bias matrix and an activation function are applied to the convolution matrix 30-1 will be described.

4 is a diagram for describing an activation function according to an embodiment of the present disclosure.

A bias matrix may be added to the convolution matrix 30-1. As an example, the bias matrix may be a matrix having a preset value of 1×1. The image processor 130 may add a preset value included in the bias matrix to all elements included in the convolution matrix 30-1.

The image processor 130 may apply an activation function to the convolution matrix 30-1. By applying the activation function to the convolution matrix (or feature map, feature map) obtained through matrix multiplication (or convolution, filter, kernel), it is possible to identify whether or not a specific object is included in the image. As an example, the ReLu function can be applied in the CNN model. As shown in FIG. 4, the RELU function can process max(0,x) and negative numbers as 0. However, it is not limited thereto, and of course, a Sigmoid (or tanh) activation function may be applied.

5 is a flowchart illustrating an operation method for obtaining a filtering result according to an embodiment of the present disclosure.

Referring to FIG. 5, when an image is received (S510), matrices representing color components constituting each pixel of the image may be converted into one integrated matrix (S520). For example, the processor 120 may convert the matrix 10 for an image into an integrated matrix 10-1 using the im2col command.

Subsequently, the image processor 130 obtains the convolution matrix 30-1 by performing matrix multiplication on the integrated matrix 10-1 and the interleaved weight matrix 20-1, and the convolution matrix 30-1 A bias matrix may be added to) and an activation function may be applied (S530).

The processor 120 according to an embodiment of the present disclosure collectively transmits parameter values and commands used in step S530 to the image processor 130 together with the integrated matrix 10-1, so that the image processor 130 performs step S530. You can control the image processor to perform Accordingly, a delay time according to data transmission/reception between the processor 120 and the image processor 130 may be minimized.

The electronic device 100 may output the filtering result of the image processor 130 obtained in step S530 (S540).

6 and 7 are diagrams for explaining a command for obtaining a filtering result according to an embodiment of the present disclosure.

According to FIG. 6, in step S510, the processor 130 may convert a plurality of matrices 10 for an image into one integrated matrix 10-1 according to a = im2con(input) command.

5 and 6, under the control of the processor 100 in the transpose step, the image processor 130 performs aTrans = trans(a) for the integrated matrix, and b = interleave (weights) in the interleave step. Thus, the weight matrix 20 can be converted into an interleaved matrix 20-1.

Subsequently, under the control of the processor 120, the image processor 130 may perform convolution according to c=gemm(aTrans, b, 1, 0) in the convolution step, and obtain the convolution matrix 30-1. . Subsequently, under the control of the processor 120, the image processor 130 acquires a bias matrix in a form that can be summed with the convolution matrix 30-1 according to biasMatrix = gemm (bias, constants, 1, 0) in the Bias GEMM step. can do.

Subsequently, under the control of the processor 120, the image processor 130 may apply the ReLu function to the convolution matrix in which the bias matrix is summed.

In this way, each step is required to transmit commands, parameters, etc. of the processor 120 to the image processor 130. A delay time occurs according to transmission and reception of commands, parameters, data, and the like between the processor 120 and the image processor 130.

Hereinafter, a method of performing filtering by the image processor 130 according to an embodiment of the present disclosure will be described.

Referring to FIG. 7, in step S510, the processor 130 may convert a plurality of matrices 10 for an image into one integrated matrix 10-1 according to a = im2con(input) command.

5 and 7, in step S530, under the control of the processor 100, the image processor 130 performs output = cbr(a, interleaved weights, bias) on the integrated matrix to obtain the output layer 30. I can.

The weight matrix 20 and the bias matrix may be pre-stored in a form capable of multiplying and summating an integrated matrix and matrix, respectively. Accordingly, the image processor 130 may not be required to transmit commands and parameters of the processor 120 for performing the Transpose, Interleave, and Bias GEMM steps of FIG. 5.

The processor 120 collectively transmits parameter values and instructions used for filtering (e.g., output = cbr(a, interleaved weights, bias)) to the image processor together with the integrated matrix, and the image processor 130 Filtering may be performed without transmitting an additional command and parameter of 120 to obtain a filtering result (eg, an output layer). As commands, parameters, and data are transmitted and received between the processor 120 and the image processor 130, the delay time may be reduced.

8 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention.

According to the control method of the electronic device illustrated in FIG. 8, when an image is received, matrices representing color components constituting each pixel of the image are converted into a single integrated matrix (S810).

Subsequently, a convolution matrix is obtained by performing matrix multiplication on the unified matrix and a weight matrix trained by a convolution neural network (CNN) (S820).

Subsequently, the characteristics of each pixel area of the image are filtered using the convolution matrix (S830).

Subsequently, the filtering result is output (S840).

Here, the electronic device may include a plurality of cores that perform matrix multiplication between each row constituting the integrated matrix and a corresponding column of the weight matrix in parallel.

In addition, step S830 of performing filtering may include adding the bias matrix to the convolution matrix.

In addition, the step S830 of performing filtering includes the step of selectively summing one of the first and second bias matrices to the convolution matrix, wherein the first bias matrix is a matrix composed of 1, and the second bias matrix is 0. It may be a composed matrix.

Here, in step S830 of performing filtering, applying an activation function to a convolution matrix in which one of the first and second bias matrices is summed to obtain a plurality of output layers, FC (fully) based on the plurality of output layers -connected) may include obtaining a layer and identifying characteristics of each region according to the class score of each region included in the FC layer.

Here, in the step of obtaining the FC layer, the FC layer may be obtained by summing values of positions corresponding to each other in the plurality of output layers.

In addition, in the obtaining of the FC layer, values of positions corresponding to each other in the plurality of output layers are compared, values having the largest values are collected, and the FC layer may be obtained.

Further, it includes the step of collectively transmitting parameter values and instructions used for filtering to the image processor together with the integrated matrix, and filtering may be performed by the image processor.

Further, in step S830 of performing filtering, filtering may be repeatedly performed a plurality of times.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. According to software implementation, embodiments such as procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing a processing operation according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When a computer instruction stored in such a non-transitory computer-readable medium is executed by a processor, a specific device may cause the processing operation according to the above-described various embodiments to be performed.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as registers, caches, and memory. Specific examples of non-transitory computer-readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is generally in the technical field belonging to the disclosure without departing from the gist of the disclosure claimed in the claims Various modifications are possible by those skilled in the art of course, and these modifications should not be individually understood from the technical idea or perspective of the present disclosure.

100: electronic device 110: memory
120: processor 130: image processor
140: display

Claims (19)

A memory storing a weight matrix trained by a convolution neural network (CNN);
A processor for converting matrices representing color components constituting each pixel of the image into one unified matrix when an image is received;
An image processor that performs filtering on the image by using the integrated matrix converted by the processor; And
Including; a display for outputting the filtering result of the image processor under the control of the processor,
The image processor,
A convolutional matrix is obtained by performing matrix multiplication of the unified matrix and the weighting matrix, and characteristics of each pixel region of the image are filtered using the convolutional matrix,
The weight matrix is a matrix in which a plurality of CNN-trained matrices are integrated in an interleaved form,
The integrated matrix is a matrix having the weight matrix and the number of rows and columns that can be calculated,
The image processor,
It includes a plurality of cores,
Each of the plurality of cores,
The electronic device, wherein matrix multiplication between each row constituting the unified matrix and a corresponding column of the weight matrix is performed in parallel. delete The method of claim 1,
The memory further includes a bias matrix,
The image processor summing the bias matrix to the convolution matrix to the convolution matrix. The method of claim 1,
The memory further includes a first bias matrix and a second bias matrix,
The image processor to selectively add one of the first and second bias matrices to the convolution matrix. The method of claim 4,
The image processor,
Applying an activation function to a convolution matrix in which one of the first and second bias matrices is summed, to obtain a plurality of output layers,
Obtaining a fully-connected (FC) layer based on the plurality of output layers,
The electronic device that identifies characteristics of each region according to the class score of each region included in the FC layer. The method of claim 5,
The image processor,
The electronic device that obtains the FC layer by summing values of positions corresponding to each other in the plurality of output layers. The method of claim 5,
The image processor,
The electronic device comprising: comparing values of positions corresponding to each other in the plurality of output layers, collecting values having the largest values, and obtaining the FC layer. The method of claim 1,
The processor,
And controlling the image processor to perform the filtering by collectively transmitting parameter values and commands used for the filtering to the image processor together with the integrated matrix. The method of claim 1,
The processor,
Controlling the image processor to repeatedly perform the filtering a plurality of times. In the control method of an electronic device for processing an image,
When an image is received, converting matrices representing color components constituting each pixel of the image into one unified matrix;
Filtering the image using a convolution neural network (CNN) trained weight matrix and the unified matrix; And
Including; outputting the filtering result;
The step of performing the filtering,
Obtaining a convolution matrix by performing matrix multiplication on the unified matrix and the weight matrix;
Filtering the characteristics of each pixel area of the image using the convolution matrix; and
The weight matrix is a matrix in which a plurality of CNN-trained matrices are integrated in an interleaved form,
The integrated matrix is a matrix having the weight matrix and the number of rows and columns that can be calculated,
Obtaining the convolution matrix,
A control method for obtaining the convolution matrix by performing a matrix multiplication between each row constituting the unified matrix and a corresponding column of the weight matrix in parallel using a plurality of cores included in the electronic device. delete The method of claim 10,
The step of performing the filtering,
Summing a bias matrix to the convolution matrix. The method of claim 10,
The step of performing the filtering,
Optionally summing one of the first and second bias matrices to the convolutional matrix,
The first bias matrix is a matrix composed of 1,
Wherein the second bias matrix is a matrix composed of zeros. The method of claim 13,
The step of performing the filtering,
Obtaining a plurality of output layers by applying an activation function to a convolution matrix in which one of the first and second bias matrices is summed;
Obtaining a fully-connected (FC) layer based on the plurality of output layers; And
Identifying characteristics of each region according to the class score of each region included in the FC layer; including, control method The method of claim 14,
The step of obtaining the FC layer,
The control method, wherein the FC layer is obtained by summing values of positions corresponding to each other in the plurality of output layers. The method of claim 14,
The step of obtaining the FC layer,
Comparing values of positions corresponding to each other in the plurality of output layers, collecting values having the largest values, and obtaining the FC layer. The method of claim 10,
Transmitting parameter values and commands used for the filtering together with the integrated matrix to an image processor; Including,
The filtering is performed by the image processor. The method of claim 10,
The step of performing the filtering,
A control method for repeatedly performing the filtering a plurality of times. A non-transitory computer-readable medium storing computer instructions to cause the electronic device to perform an operation when executed by a processor of the electronic device, the operation comprising:
When an image is received, converting matrices representing color components constituting each pixel of the image into one unified matrix;
Filtering the image using a convolution neural network (CNN) trained weight matrix and the unified matrix; And
Including; outputting the filtering result;
The step of performing the filtering,
Obtaining a convolution matrix by performing matrix multiplication on the unified matrix and the weight matrix;
Filtering the characteristics of each pixel area of the image using the convolution matrix; and
The weight matrix is a matrix in which a plurality of CNN-trained matrices are integrated in an interleaved form,
The integrated matrix is a matrix having the weight matrix and the number of rows and columns that can be calculated,
Obtaining the convolution matrix,
A non-transitory computer-readable medium for obtaining the convolution matrix by performing a matrix multiplication between each row constituting the integrated matrix and a corresponding column of the weight matrix using a plurality of cores included in the electronic device .

Images