KR20190100487A - Electric apparatus and method for control thereof - Google Patents

Electric apparatus and method for control thereof Download PDF

Info

Publication number
KR20190100487A
KR20190100487A KR1020180013954A KR20180013954A KR20190100487A KR 20190100487 A KR20190100487 A KR 20190100487A KR 1020180013954 A KR1020180013954 A KR 1020180013954A KR 20180013954 A KR20180013954 A KR 20180013954A KR 20190100487 A KR20190100487 A KR 20190100487A
Authority
KR
South Korea
Prior art keywords
matrix
image
filtering
processor
plurality
Prior art date
Application number
KR1020180013954A
Other languages
Korean (ko)
Inventor
임태규
송의현
한승훈
문보석
Original Assignee
삼성전자주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 삼성전자주식회사 filed Critical 삼성전자주식회사
Priority to KR1020180013954A priority Critical patent/KR20190100487A/en
Publication of KR20190100487A publication Critical patent/KR20190100487A/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4046Scaling the whole image or part thereof using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/20Image enhancement or restoration by the use of local operators
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20024Filtering details
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]

Abstract

An electronic device is disclosed. The electronic device includes a memory in which a convolutional neural network (CNN) trained weight matrix is stored, a processor converting matrices representing color components constituting each pixel of the image into one unified matrix when an image is received; The image processor performs filtering on the image using the integrated matrix converted by the processor, and the display outputs the filtering result of the image processor according to the control of the processor.

Description

ELECTRIC APPARATUS AND METHOD FOR CONTROL THEREOF}

The present invention utilizes a machine learning algorithm such as deep learning, an artificial intelligence (AI) system that simulates the functions of the human brain, such as cognition and judgment, and an electronic device and control method for providing an image processing model among the applications. In more detail, the present invention relates to an electronic device using a CNN trained image processing model and a control method thereof.

Artificial Intelligence (AI) system is a computer system that implements human-level intelligence. Unlike conventional rule-based smart systems, the machine learns and judges itself and becomes smart. As the AI system is used, the recognition rate is improved and the user's taste can be understood more accurately. The existing Rule-based smart system is gradually replaced by the deep learning-based AI system.

AI technology is composed of elementary technologies that utilize machine learning (deep learning) and machine learning.

Machine learning is an algorithm technology that classifies / learns characteristics of input data by itself, and element technology is a technology that simulates the functions of human brain cognition and judgment by using machine learning algorithms such as deep learning. It consists of technical areas such as understanding, reasoning / prediction, knowledge representation, and motion control.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying / processing human language / characters and includes natural language processing, machine translation, dialogue system, question and answer, speech recognition / synthesis, and the like. Visual understanding is a technology that recognizes and processes objects as human vision, and includes object recognition, object tracking, image retrieval, person recognition, scene understanding, spatial understanding, and image enhancement. Inference prediction is a technique of judging information, logically inferring, and predicting information, and includes knowledge / probability based inference, optimization prediction, preference based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation / classification) and knowledge management (data utilization). Motion control is a technology for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), operation control (action control), and the like.

For example, the artificial intelligence system can be applied to learning various images and generating an image processing model according to the learning result. In addition, when a new image is received through a process similar to learning based on the generated image model, the image may be filtered, classified, and provided.

However, as the dimension of the image processing model increases, the amount of data required for processing increases exponentially, and there is a problem that a delay time occurs due to data transmission and reception between the processor and the image processor.

SUMMARY OF THE INVENTION An object of the present invention is to provide an electronic device and a method of controlling the same, which provide a filtering result without deterioration by using a convolutional neural network (CNN) trained weight matrix and a bias matrix. Is in.

According to an embodiment of the present disclosure, an electronic device includes a memory in which a convolutional neural network (CNN) trained weight matrix is stored, and when an image is received, configures each pixel of the image. A processor for converting the matrices representing the color components into one unified matrix, an image processor performing filtering on the image using the unified matrix converted by the processor, and a filtering result of the image processor according to control of the processor Wherein the image processor is configured to perform a matrix multiplication on the integration matrix and the weight matrix to obtain a convolution matrix, filter the characteristics of each pixel region of the image using the convolution matrix, Weighting matrix Is a matrix in which a plurality of CNN-learned matrices are integrated in an interleaved form, and the unified matrix is a matrix having a number of rows and columns operable with the weight matrix.

In addition, the image processor may include a plurality of cores each performing in parallel a matrix product between each row constituting the unified matrix and a corresponding column of the weight matrix.

The memory may further include a bias matrix, and the image processor may add the bias matrix to the convolution matrix.

The memory may further include a first bias matrix and a second bias matrix, wherein the image processor may selectively sum one of the first and second bias matrices into the convolution matrix.

Here, the image processor is configured to apply an activation function to a convolution matrix summed with one of the first and second bias matrices, to obtain a plurality of output layers, and to fully-connected based on the plurality of output layers. ) Layer, and the characteristics of each region can be identified according to the class score of each region included in the FC layer.

Here, the image processor may acquire the FC layer by summing values of positions corresponding to each other in the plurality of output layers.

In addition, the image processor may obtain the FC layer by comparing values of positions corresponding to each other in the plurality of output layers and collecting values having the largest value.

The processor may control the image processor to collectively transmit parameter values and instructions used for the filtering together with the integration matrix to the image processor to perform the filtering.

The processor may control the image processor to repeatedly perform the filtering a plurality of times.

Meanwhile, in an electronic device control method according to an embodiment of the present disclosure, when an image is received, converting matrices representing color components constituting each pixel of the image into one unified matrix, CNN (Convolution Neural Network) Performing filtering on the image using the trained weight matrix and the integration matrix and outputting the filtering result, wherein the performing of the filtering comprises: the integration matrix; And performing a matrix multiplication on the weight matrix to obtain a convolution matrix, and filtering characteristics of respective pixel regions of the image using the convolution matrix, wherein the weight matrix is a plurality of CNN-learned matrices. Is an interleaved matrix The integrated matrix, the matrix having the weighting matrix and the calculation of the total number of rows and columns.

In addition, the plurality of cores may perform a matrix multiplication between each row constituting the unified matrix and a corresponding column of the weight matrix.

Here, the performing of the filtering may include adding a bias matrix to the convolution matrix.

In addition, performing the filtering may include selectively adding one of a first and a second bias matrix to the convolution matrix, wherein the first bias matrix is a matrix consisting of one and the second bias matrix. May be a matrix composed of zeros.

The performing of the filtering may include applying an activation function to a convolution matrix obtained by adding one of the first and second bias matrices, to obtain a plurality of output layers, and based on the plurality of output layers. The method may include obtaining a (fully-connected) layer and identifying characteristics of each region according to a class score of each region included in the FC layer.

In the obtaining of the FC layer, the FC layer may be obtained by summing values of positions corresponding to each other in the plurality of output layers.

In the obtaining of the FC layer, the FC layer may be obtained by comparing values of positions corresponding to each other in the plurality of output layers and collecting values having the largest value.

The method may further include collectively transmitting parameter values and instructions used for the filtering together with the integration matrix to the image processor, wherein the filtering may be performed by the image processor.

In addition, the filtering may be performed by repeating the filtering a plurality of times.

Meanwhile, in a non-transitory computer readable medium storing computer instructions for performing an operation when the electronic device is executed by a processor of an electronic device according to an embodiment of the present disclosure, the operation may include: Converting matrices representing the color components constituting each pixel of the image into one unified matrix, a convolutional neural network (CNN) trained weight matrix, and filtering the image using the unified matrix And outputting the filtering result, and the performing of the filtering may include performing a matrix product of the integration matrix and the weight matrix to obtain a convolution matrix, using the convolution matrix. Filter the characteristics of each pixel region of the image And a ringing step, wherein the weight matrix is a matrix obtained by integrating a plurality of CNN-learned matrices in an interleaved form, and wherein the integration matrix has a number of rows and columns that can be calculated with the weight matrix. It can be a matrix.

According to various embodiments of the present disclosure, an electronic device may provide a minimum delay time due to data transmission and reception between a processor and an image processor by using a weight matrix and a bias matrix trained by a convolutional neural network (CNN). Can be.

1 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure.
3 is a diagram for describing a matrix product according to an exemplary embodiment.
4 is a diagram for describing an activation function, according to an exemplary embodiment.
5 is a flowchart illustrating an operating method for obtaining a filtering result according to an exemplary embodiment.
6 is a diagram for describing a command for obtaining a filtering result according to an exemplary embodiment.
7 is a diagram for describing a command for obtaining a filtering result according to an exemplary embodiment.
8 is a flowchart illustrating a control method of an electronic device according to an embodiment of the present disclosure.

Terms used herein will be briefly described, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure selected general terms widely used as far as possible in consideration of functions in the present disclosure, but may vary according to the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in certain cases, there is also a term arbitrarily selected by the applicant, in which case the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meanings of the terms and the contents throughout the present disclosure, rather than simply the names of the terms.

Embodiments of the present disclosure may be variously modified and have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the scope to the specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the scope of the disclosed spirit and technology. In describing the embodiments, when it is determined that the detailed description of the related known technology may obscure the gist, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "consist" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, and one or more other It is to be understood that the present invention does not exclude the possibility of the presence or the addition of features, numbers, steps, operations, components, parts, or a combination thereof.

In the present disclosure, a "module" or "unit" performs at least one function or operation, and may be implemented in hardware or software or in a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" may be integrated into at least one module except for "modules" or "units", which need to be implemented with specific hardware, and are implemented with at least one processor (not shown). Can be.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted for simplicity of explanation, and like reference numerals designate like parts throughout the specification.

1 is a block diagram illustrating an electronic device 100 according to an embodiment of the present disclosure.

As shown in FIG. 1, the electronic device 100 includes a memory 110, a processor 120, an image processor 130, and a display 140.

The electronic device 100 according to an embodiment of the present disclosure may be a device capable of artificial intelligence learning. For example, a user terminal device, a display device, a set-top box, a tablet personal computer, a smart phone, an e-book reader, a desktop PC ), A laptop PC, a workstation, a server, a personal digital assistant, a portable multimedia player, a MP3 player, and the like. Alternatively, the electronic device 100 may mean a system itself in which a cloud computing environment is built. However, the present invention is not limited thereto, and the electronic device 100 may be any device as long as the device is capable of artificial intelligence learning.

The memory 110 may store a weight matrix trained by a convolutional neural network (CNN). Here, CNN is a multilayer neural network with a special connection structure designed for speech processing, image processing, and the like. In particular, the CNN may filter the image in various ways and recognize the characteristics of the image by preprocessing the pixel.

The memory 110 may store an image processing model including a CNN trained matrix as basic data including a plurality of images. The matrix learned from the basic data may be called variously, such as a filter, a kernel, a weight matrix, etc. Hereinafter, for convenience of description, the matrix is collectively referred to as a weight matrix.

The memory 110 according to an embodiment of the present disclosure may further include a bias matrix. Here, the bias matrix means a matrix composed of constants. According to an embodiment, the electronic device 100 may obtain a filtering result by using a bias matrix in the filtering process for the image. Here, the bias matrix is configured to prevent overfitting of the image processing model including the CNN trained weight matrix. Here, when the electronic device 100 obtains the characteristics of the image by applying the weight matrix to the image included in the basic data, the characteristics of the acquired image and the actual characteristics of the image match, whereas other images ( For example, an image obtained by applying a weighting matrix to an image received in real time means that a case in which the characteristic of the image does not match the actual characteristic of the image is frequently issued.

The processor 120 controls the overall operation of the electronic device 100.

According to an embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON), but is not limited thereto. central processing unit (CPU), microcontroller unit (MCU), micro processing unit (MPU), controller, application processor (AP), or communication processor (CP), ARM processor The processor 140 may be implemented by a System on Chip (SoC), a large scale integration (LSI), or an FPGA (Integrated Processing Algorithm). Field Programmable Gate Array) may be implemented.

When the image is received, the processor 120 may convert matrices representing the color components constituting each pixel of the image into one unified matrix. Here, the image may refer to various types of images, such as an image and content that the electronic device 100 receives in real time. The color component may mean a pixel value corresponding to each pixel of the image. For example, when each pixel of the received image is represented through 8 bits, one pixel has one of a total of 256 color components of 0 to 255. Accordingly, the processor 120 may convert the received image into a matrix using each pixel value.

If the received image is data of multiple channels, the processor 120 may obtain a matrix for each channel and convert the obtained plurality of matrices into one unified matrix. For example, the received image is three channel data of R, G, and B, and the processor 120 may obtain a matrix corresponding to each of R, G, and B. A detailed description thereof will be provided with reference to FIG. 3.

The processor 120 according to an embodiment of the present disclosure may control the image processor 130 to perform filtering on the received image. In addition, the display 140 may be controlled to output the filtering result.

The image processor 130 may be a special purpose processor for the processing of graphic data. In particular, the image processor 130 may perform filtering on the image using the transformed integrated matrix.

The image processor 130 is a GPU (Graphic Processing Unit), but may be implemented as a component that is distinct from the processor 120, but is not limited thereto. Various embodiments of the present disclosure may be performed by the processor 120. For example, the graphic processor of the processor 120 may perform filtering on the received image according to various embodiments of the present disclosure.

The image processor 130 generates a screen including various objects by using a calculator (not shown) and a renderer (not shown). The calculating unit calculates attribute values such as coordinate values, shapes, sizes, colors, and the like in which objects are displayed according to the layout of the screen based on the received control command. The renderer generates screens of various layouts including objects based on the attribute values calculated by the calculator. The screen generated by the renderer is displayed in the display area of the display 140.

In addition, the image processor 130 is a component that performs processing on image data. The image processor 130 may perform various image processing such as decoding, scaling, noise filtering, frame rate conversion, resolution conversion, and the like on the image data.

In particular, the image processor 130 may obtain a convolution matrix by performing convolution on the integration matrix and the weight matrix. Here, the convolution means that the weight matrix performs matrix multiplication while shifting the matrix for the image by region, and calculates a result value according to the matrix multiplication. As an example, if the matrix for the received image is 7 × 7 and the weight matrix is 3 × 3, the weight matrix performs matrix multiplication by shifting by region within the matrix for the image and obtains an output matrix of 3 × 3. Can be. Hereinafter, the output matrix obtained according to the convolution will be collectively referred to as a convolution matrix.

The size of the matrix is one embodiment, but is not limited thereto. For example, the matrix may have various sizes depending on the received image. In addition, the padding and the stride may be performed in the process of performing the convolution so that the size may be variously adjusted. Here, padding means filling a specific pixel value by a predetermined size in all directions of the received image. If padding is performed, the matrix and the convolution matrix for the image may have the same size. Stride means the shift interval of the weight matrix. For example, if starride (stride = 3), the image processor 130 may perform convolution while shifting by three spaces in the matrix.

However, the image processor 130 according to an embodiment of the present disclosure may not shift the weight matrix and the matrix for the image for each region in order to perform convolution. The transformed unified matrix can be a matrix having rows and columns that can be matrix multiplied by the weight matrix. The weight matrix stored in the memory 110 may be an integrated matrix in an interleaved form. Here, the interleaved matrix may be a matrix having rows and columns capable of matrix multiplication with the transformed integrated matrix.

For example, if the received image is a two-dimensional 3 × 3 matrix, the processor 110 may convert the integrated image into a 4 × 8 matrix. The image processor 130 may obtain a convolution matrix by performing matrix multiplication with a matrix of 4 × 8 and a weight matrix stored in the memory 110. Here, the weight matrix may be an interleaved form capable of performing matrix multiplication with a 4 × 8 integrated matrix, that is, an 8 × 2 matrix. The image processor 130 may obtain a 4 × 2 convolution matrix by performing matrix multiplication between the integration matrix and the weight matrix. A detailed description thereof will be provided with reference to FIG. 3.

The image processor 130 according to an embodiment of the present disclosure may include a plurality of cores. The image processor 130 may perform a matrix product between each row constituting the unified matrix and the corresponding column of the weight matrix in parallel using a plurality of cores. The convolution between the integration matrix and the weight matrix may mean performing matrix multiplication between the integration matrix and the weight matrix, or applying a filter or kernel to the image.

As an example, it may be assumed that the image processor 130 performs matrix multiplication between a 4 × 8 integrated matrix and an 8 × 2 weighted matrix. Matrix multiplication between one row of the integration matrix and one column of the weight matrix may be performed by the first core, and matrix multiplication between two rows of the integration matrix and two columns of the weight matrix may be performed by the second core. As such, a total of eight matrix products may be performed in parallel by each of the first to eighth cores. Accordingly, the time required to obtain the convolution matrix can be reduced. For example, NVIDIA® GPUs can include 3,584 CUDA® cores, which can be arithmetic performed in parallel by 3,584 operations.

The image processor 130 according to an embodiment of the present disclosure may add the bias matrix to the convolution matrix. For example, the bias matrix may be a matrix having a predetermined value of 1 × 1. The image processor 130 may add all of the elements included in the convolution matrix to a predetermined value included in the bias matrix.

As another example, the image processor 130 may selectively apply any one of the first and second bias matrices to the convolution matrix. For example, the first bias matrix may be a 1 × 1 matrix composed of 1s, and the second bias matrix may be a 1 × 1 matrix composed of 0s. The image processor 130 may selectively sum any one of the first and second bias matrices into the convolution matrix based on various purposes.

The image processor 140 according to an embodiment of the present disclosure may apply an activation function to the convolution matrix in which the bias matrix is summed. Here, various types of functions such as an activation function, an identity function, a logistic sigmoid function, a hyperbolic tangent (tanh) function, a relu function, and a leaky relu function may be applied. As an example, the image processor 130 may apply the RELU function max (0, x) to all elements.

The image processor 140 may apply an activation function to obtain a plurality of output layers and to obtain a fully-connected (FC) layer based on the output layer. Here, the FC layer is an output layer and may identify characteristics of each region according to class scores of respective regions included in the FC layer. For example, the class score may mean an amount activated after the activation function is applied, and the image processor 130 may identify a category corresponding to each activation amount. Here, the identified category may mean a characteristic of each region. For example, objects (eg, cars, boats, cats, etc.) included in each area may be identified according to class scores of each area included in the FC layer.

The image processor 130 according to an embodiment of the present disclosure may acquire the FC layer by summing values of positions corresponding to each other in the plurality of output layers.

As another example, the image processor 130 may obtain values of the FC layer by comparing values of positions corresponding to each other in the plurality of output layers and collecting the values having the largest values.

The display 140 may be a variety of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a liquid crystal on silicon (LCoS), or a digital light processing (DLP). Can be implemented. However, the present invention is not limited thereto and may be implemented as various types of displays capable of displaying images.

In particular, the display 140 may output the filtering result of the image processor 130 under the control of the processor 120.

2 is a block diagram illustrating a detailed configuration of an electronic device 100 according to an embodiment of the present disclosure.

According to FIG. 2, the electronic device 100 may include a memory 110, a processor 120, an image processor 130, a display 140, a communication unit 150, a user interface unit 160, and an audio processing unit 170. Include. A detailed description of parts overlapping with those shown in FIG. 1 among the elements shown in FIG. 2 will be omitted.

The processor 120 controls the overall operation of the electronic device 100 using various programs stored in the memory 110.

In detail, the processor 120 may access the RAM 121, the ROM 122, the main CPU 123, the graphics processor 124, the first to n interfaces 125-1 to 125-n, and the bus 126. Include.

The RAM 121, the ROM 122, the main CPU 123, the graphics processor 124, and the first to n-interfaces 125-1 to 125-n may be connected to each other through the bus 126.

The first to n interfaces 125-1 to 125-n are connected to the various components described above. One of the interfaces may be a network interface connected to an external device via a network.

The main CPU 123 accesses the memory 110 and performs booting using an operating system stored in the memory 110. In addition, various operations are performed using various programs stored in the memory 110.

The ROM 122 stores a command set for system booting. When the turn-on command is input and the power is supplied, the main CPU 123 copies the O / S stored in the memory 110 to the RAM 121 according to the command stored in the ROM 122, and executes the O / S. Boot up. When the booting is completed, the main CPU 123 copies various application programs stored in the memory 110 to the RAM 121 and executes the application programs copied to the RAM 121 to perform various operations.

The processor 120 according to an exemplary embodiment may collectively transmit parameter values and instructions used by the image processor 130 to perform filtering together with the integration matrix to the image processor 130. Accordingly, delay time due to data transmission and reception between the processor 120 and the image processor 130 may be minimized. Detailed description thereof will be made with reference to FIGS. 5 to 7.

The processor 120 according to an embodiment of the present disclosure may control the image processor to repeatedly perform filtering a plurality of times.

Meanwhile, the above-described operation of the processor 120 may be performed by a program stored in the memory 110.

The memory 110 stores various data such as an operating system (O / S) software module for driving the electronic device 100, an image processing model including a weight matrix and a bias matrix, a CNN learning module, and the like.

The memory 110 may be implemented as an internal memory such as a ROM or a RAM included in the processor 120 or may be implemented as a memory separate from the processor 120. In this case, the memory 110 may be implemented in the form of a memory embedded in the electronic device 100 according to a data storage purpose, or may be implemented in the form of a memory detachable to the electronic device 100. For example, data for driving the electronic device 100 is stored in a memory embedded in the electronic device 100, and data for expansion function of the electronic device 100 is detachable from the electronic device 100. Possible memory can be stored. Meanwhile, the memory embedded in the electronic device 100 may be implemented in the form of a nonvolatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD), and may be attached or detached to the electronic device 100. For example, the memory card may be implemented in the form of a memory card (eg, micro SD card, USB memory, etc.), an external memory (eg, USB memory) connectable to a USB port.

The communication unit 150 is a component that performs communication with various types of external devices according to various types of communication methods. The communicator 150 includes a Wi-Fi chip 151, a Bluetooth chip 152, a wireless communication chip 153, an NFC chip 154, and the like. The processor 120 communicates with various external devices using the communication unit 150.

The Wi-Fi chip 151 and the Bluetooth chip 152 communicate with each other by WiFi and Bluetooth. In the case of using the Wi-Fi chip 151 or the Bluetooth chip 152, various connection information such as SSID and session key may be transmitted and received first, and then various communication information may be transmitted and received using the same. The wireless communication chip 153 refers to a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evoloution (LTE), and the like. The NFC chip 154 refers to a chip operating in a near field communication (NFC) method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860-960 MHz, 2.45 GHz, and the like.

The processor 120 may receive an image processing model including an image, a weight matrix, a bias matrix, and an activation function from the external device through the communication unit 150.

The user interface unit 160 receives various user interactions. Here, the user interface 160 may be implemented in various forms according to the implementation example of the electronic device 100. For example, the user interface 160 may be a button provided in the electronic device 100, a microphone for receiving a user voice, a camera for detecting a user motion, or the like. Alternatively, when the electronic device 100 is implemented as a touch-based electronic device, the user interface 160 may be implemented in the form of a touch screen that forms a mutual layer structure with the touch pad. In this case, the user interface 160 may be used as the display 140 described above.

The audio processor 160 is a component that performs processing on audio data. The audio processor 160 may perform various processing such as decoding, amplification, noise filtering, and the like on the audio data.

3 is a diagram for describing a matrix product according to an exemplary embodiment.

According to FIG. 3, when the received image is multidimensional, a plurality of matrices 10 for the image representing the color components constituting each pixel of the image may be obtained. For example, if the received image is two-dimensional, two matrixes of the image may be obtained. The processor 120 may convert the plurality of matrices 10 for the image into one unified matrix 10-1.

The image processor 130 may convert the weight matrix 20 into an interleaved matrix 20-1 to perform matrix multiplication between the integration matrix 10-1 and the weight matrix 20. However, the electronic device 100 according to an embodiment of the present disclosure may prestore the weight matrix 20 as the interleaved matrix 20-1. Accordingly, the image processor 130 may perform matrix multiplication using the pre-stored interleaved matrix 20-1 without having to convert the weight matrix 20 into the interleaved matrix 20-1. Can be.

For example, the processor 120 may convert a 3 × 3 × 2 matrix 10 into a 4 × 8 × 1 unified matrix 10-1. The interleaved matrix 20-1 has a size of 8x4x1. The image processor 130 may perform matrix multiplication using the pre-stored interleaved matrix 20-1 without changing the size of the weight matrix 20 from 2 × 2 × 2 to 8 × 2 × 1. have.

On the other hand, the convolution between the matrix 10 and the weight matrix 20 for the image requires a shift of the weight matrix 20 and has a disadvantage in that the calculation process is complicated. The computation time and resources required to obtain the output layer 30 are high.

 The matrix multiplication between the unified matrix 10-1 and the interleaved matrix 20-1 performs only the matrix multiplication of the rows of the unified matrix 10-1 with the columns of the interleaved matrix 20-1. Less computation time and resources are required to obtain (30-1).

The image processor 130 according to an embodiment of the present disclosure may include a plurality of cores. Each of the plurality of cores may perform a matrix product of the rows of the integrated mattress 10-1 and the columns of the interleaved matrix 20-1 in parallel.

The image processor 130 according to an embodiment of the present disclosure may acquire the output layer 30 based on the convolution matrix 30-1.

Hereinafter, an embodiment of applying the bias matrix and the activation function to the convolution matrix 30-1 will be described.

4 is a diagram for describing an activation function, according to an exemplary embodiment.

The bias matrix may be added to the convolution matrix 30-1. For example, the bias matrix may be a matrix having a predetermined value of 1 × 1. The image processor 130 may add all the elements included in the convolution matrix 30-1 to a predetermined value included in the bias matrix.

The image processor 130 may apply an activation function to the convolution matrix 30-1. By applying an activation function to a convolution matrix (or a feature map) obtained through matrix multiplication (or convolution, filter, kernel), it is possible to identify whether an image includes a specific object. For example, the ReLu function may be applied to the CNN model. As shown in FIG. 4, the RELU function may treat max (0, x) and all negative numbers as zero. However, the present invention is not limited thereto, and the sigmoid (or tanh) activation function may be applied.

5 is a flowchart illustrating an operating method for obtaining a filtering result according to an exemplary embodiment.

Referring to FIG. 5, when an image is received (S510), matrices representing color components constituting each pixel of the image may be converted into one unified matrix (S520). For example, the processor 120 may convert the matrix 10 for the image into the unified matrix 10-1 using the im2col command.

Subsequently, the image processor 130 performs matrix multiplication on the integrated matrix 10-1 and the interleaved weight matrix 20-1 to obtain the convolution matrix 30-1, and then the convolution matrix 30-1. ), The bias matrix may be added, and an activation function may be applied (S530).

The processor 120 according to an embodiment of the present disclosure transmits the parameter values and the instructions used in the step S530 together with the integrated matrix 10-1 to the image processor 130 so that the image processor 130 may perform the step S530. The image processor may be controlled to perform the operation. Accordingly, delay time due to data transmission and reception between the processor 120 and the image processor 130 may be minimized.

The electronic device 100 may output a filtering result of the image processor 130 obtained in operation S530 (S540).

6 and 7 are diagrams for describing an instruction for obtaining a filtering result according to an exemplary embodiment.

According to FIG. 6, in operation S510, the processor 130 may convert the plurality of matrices 10 for an image into one unified matrix 10-1 according to a = im2con (input) command.

5 and 6, under the control of the processor 100 in the transpose step, the image processor 130 performs aTrans = trans (a) on the integrated matrix, and b = interleave (weights) in the interleave step. The weight matrix 20 can be converted into an interleaved matrix 20-1.

Subsequently, under the control of the processor 120, the image processor 130 may perform convolution according to c = gemm (aTrans, b, 1, 0) in the convolution step, and obtain the convolution matrix 30-1. . Subsequently, under the control of the processor 120, the image processor 130 obtains a bias matrix that can be summed with the convolution matrix 30-1 according to biasMatrix = gemm (bias, constants, 1, 0) in the Bias GEMM step. can do.

Subsequently, under the control of the processor 120, the image processor 130 may apply the ReLu function to the convolution matrix obtained by adding the bias matrix.

As such, each step requires transmission of instructions, parameters, and the like of the processor 120 to the image processor 130. Delay time occurs according to the transmission and reception of instructions, parameters, data, and the like between the processor 120 and the image processor 130.

Hereinafter, a filtering method of the image processor 130 according to an exemplary embodiment will be described.

According to FIG. 7, in operation S510, the processor 130 may convert the plurality of matrices 10 for the image into one unified matrix 10-1 according to a = im2con (input) command.

5 and 7, according to the control of the processor 100 in operation S530, the image processor 130 performs output = cbr (a, interleaved Weights, bias) on the integrated matrix to obtain the output layer 30. Can be.

The weight matrix 20 and the bias matrix may be pre-stored in the form of an integral matrix, a matrix product and a summation, respectively. Accordingly, the image processor 130 may not require command and parameter transmission of the processor 120 to perform the Transpose, Interleave, and Bias GEMM steps of FIG. 5.

The processor 120 collectively sends the parameter values and instructions (eg, output = cbr (a, interleaved Weights, bias)) used for filtering together with the integrated matrix to the image processor, and the image processor 130 Filtering may be performed without additional command and parameter transmission of 120 to obtain a filtering result (eg, an output layer). The delay time may be reduced by minimizing transmission and reception of instructions, parameters, data, and the like between the processor 120 and the image processor 130.

8 is a flowchart illustrating a control method of an electronic device according to an embodiment of the present disclosure.

According to the control method of the electronic device illustrated in FIG. 8, when an image is received, matrices representing color components constituting each pixel of the image are converted into one unified matrix (S810).

Subsequently, a convolution matrix is obtained by performing matrix multiplication on the integration matrix and the CNN trained weight matrix (S820).

Subsequently, the characteristics of the respective pixel regions of the image are filtered using the convolution matrix (S830).

Next, the filtering result is output (S840).

In this case, the electronic device may include a plurality of cores each performing a matrix product between each row constituting the unified matrix and a corresponding column of the weight matrix in parallel.

In addition, step S830 of performing filtering may include adding a bias matrix to a convolution matrix.

In addition, the step S830 of performing filtering may include selectively adding one of the first and second bias matrices to the convolution matrix, wherein the first bias matrix is a matrix composed of ones and the second bias matrix is zero. It may be a constructed matrix.

In operation S830, filtering may be performed by applying an activation function to a convolution matrix obtained by adding one of the first and second bias matrices to obtain a plurality of output layers. -connecting) obtaining the layer and identifying the characteristics of each region according to the class score of each region included in the FC layer.

Here, in the obtaining of the FC layer, the FC layer may be obtained by summing values of positions corresponding to each other in the plurality of output layers.

In the obtaining of the FC layer, the FC layer may be obtained by comparing values of positions corresponding to each other in the plurality of output layers and collecting the values having the largest values.

It also includes the step of sending the parameter values and instructions used for filtering together with the integrated matrix to the image processor, wherein the filtering may be performed by the image processor.

In addition, in operation S830 of performing filtering, the filtering may be repeated a plurality of times.

Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device by using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing a processing operation according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in the non-transitory computer readable medium may cause the specific device to perform the processing operations according to the above-described various embodiments when executed by the processor.

A non-transitory computer readable medium refers to a medium that stores data semi-permanently and is readable by a device, not a medium storing data for a short time such as a register, a cache, a memory, and the like. Specific examples of non-transitory computer readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

While the above has been illustrated and described with respect to preferred embodiments of the present disclosure, the present disclosure is not limited to the above-described specific embodiments, and is normally made in the art without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present disclosure.

100: electronic device 110: memory
120: processor 130: image processor
140: display

Claims (19)

  1. A memory in which a convolutional neural network (CNN) trained weight matrix is stored;
    A processor that, when received, converts matrices representing the color components of each pixel of the image into one unified matrix;
    An image processor to perform filtering on the image using the integrated matrix converted by the processor; And
    And a display configured to output a filtering result of the image processor according to the control of the processor.
    The image processor,
    Perform a matrix multiplication on the unified matrix and the weight matrix to obtain a convolution matrix, filter the characteristics of each pixel region of the image using the convolution matrix,
    The weight matrix is a matrix in which a plurality of CNN-learned matrices are integrated in an interleaved form,
    And the integration matrix is a matrix having a number of rows and columns operable with the weight matrix.
  2. The method of claim 1,
    The image processor,
    And a plurality of cores each performing in parallel a matrix product between each row constituting the unified matrix and the corresponding column of the weight matrix.
  3. The method of claim 1,
    The memory further comprises a bias matrix,
    And the image processor adds the bias matrix to the convolution matrix to the convolution matrix.
  4. The method of claim 1,
    The memory further comprises a first bias matrix and a second bias matrix,
    And the image processor selectively adds one of the first and second bias matrices to the convolution matrix.
  5. The method of claim 4, wherein
    The image processor,
    Apply an activation function to the convolution matrix summed with one of the first and second bias matrices to obtain a plurality of output layers,
    Obtain a fully-connected (FC) layer based on the plurality of output layers,
    And identify characteristics of each region according to a class score of each region included in the FC layer.
  6. The method of claim 5,
    The image processor,
    And adding the values of positions corresponding to each other in the plurality of output layers to obtain the FC layer.
  7. The method of claim 5,
    The image processor,
    And comparing the values of positions corresponding to each other in the plurality of output layers and collecting the values having the largest values to obtain the FC layer.
  8. The method of claim 1,
    The processor,
    And control the image processor to collectively transmit parameter values and instructions used for the filtering together with the integration matrix to the image processor to perform the filtering.
  9. The method of claim 1,
    The processor,
    And control the image processor to repeatedly perform the filtering a plurality of times.
  10. In the control method of an electronic device for processing an image,
    When the image is received, converting the matrices representing the color components constituting each pixel of the image into one unified matrix;
    Performing filtering on the image using a convolutional neural network (CNN) trained weight matrix and the integration matrix; And
    Outputting the filtering result;
    Performing the filtering,
    Performing a matrix multiplication on the integration matrix and the weight matrix to obtain a convolution matrix;
    Filtering the characteristic of each pixel region of the image using the convolution matrix;
    The weight matrix is a matrix in which a plurality of CNN-learned matrices are integrated in an interleaved form,
    And wherein the integration matrix is a matrix having the number of rows and columns operable with the weight matrix.
  11. The method of claim 10,
    And a plurality of cores each performing in parallel a matrix product between each row constituting the unified matrix and the corresponding column of the weight matrix.
  12. The method of claim 10,
    Performing the filtering,
    Summing a bias matrix to the convolution matrix.
  13. The method of claim 10,
    Performing the filtering,
    Optionally summing one of the first and second bias matrices to the convolution matrix;
    The first bias matrix is a matrix consisting of 1,
    And the second bias matrix is a matrix consisting of zeros.
  14. The method of claim 13,
    Performing the filtering,
    Obtaining a plurality of output layers by applying an activation function to a convolution matrix summed with one of the first and second bias matrices;
    Obtaining a fully-connected (FC) layer based on the plurality of output layers; And
    And identifying characteristics of respective areas according to class scores of respective areas included in the FC layer.
  15. The method of claim 14,
    Acquiring the FC layer,
    And adding the values of positions corresponding to each other in the plurality of output layers to obtain the FC layer.
  16. The method of claim 14,
    Acquiring the FC layer,
    And comparing the values of positions corresponding to each other in the plurality of output layers and collecting the values having the largest values to obtain the FC layer.
  17. The method of claim 10,
    Batch transmitting the parameter values and instructions used for the filtering together with the integration matrix to an image processor;
    And the filtering is performed by the image processor.
  18. The method of claim 10,
    Performing the filtering,
    And repeat the filtering a plurality of times.
  19. A non-transitory computer readable medium for storing computer instructions for performing an operation when executed by a processor of an electronic device, the operation comprising:
    When the image is received, converting the matrices representing the color components constituting each pixel of the image into one unified matrix;
    Performing filtering on the image using a convolutional neural network (CNN) trained weight matrix and the integration matrix; And
    Outputting the filtering result;
    Performing the filtering,
    Performing a matrix multiplication on the integration matrix and the weight matrix to obtain a convolution matrix;
    Filtering the characteristic of each pixel region of the image using the convolution matrix;
    The weight matrix is a matrix in which a plurality of CNN-learned matrices are integrated in an interleaved form,
    And wherein the unified matrix is a matrix having a number of rows and columns operable with the weight matrix.
KR1020180013954A 2018-02-05 2018-02-05 Electric apparatus and method for control thereof KR20190100487A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020180013954A KR20190100487A (en) 2018-02-05 2018-02-05 Electric apparatus and method for control thereof

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020180013954A KR20190100487A (en) 2018-02-05 2018-02-05 Electric apparatus and method for control thereof
PCT/KR2019/001510 WO2019151841A1 (en) 2018-02-05 2019-02-07 Electronic device and method for controlling same

Publications (1)

Publication Number Publication Date
KR20190100487A true KR20190100487A (en) 2019-08-29

Family

ID=67478873

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020180013954A KR20190100487A (en) 2018-02-05 2018-02-05 Electric apparatus and method for control thereof

Country Status (2)

Country Link
KR (1) KR20190100487A (en)
WO (1) WO2019151841A1 (en)

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5777367B2 (en) * 2011-03-29 2015-09-09 キヤノン株式会社 Pattern identification device, pattern identification method and program
CN106156807A (en) * 2015-04-02 2016-11-23 华中科技大学 The training method of convolutional neural networks model and device
CN108701236A (en) * 2016-01-29 2018-10-23 快图有限公司 Convolutional neural networks

Also Published As

Publication number Publication date
WO2019151841A1 (en) 2019-08-08

Similar Documents

Publication Publication Date Title
Gulli et al. Deep learning with Keras
US9536293B2 (en) Image assessment using deep convolutional neural networks
US20120207394A1 (en) Multi-stage image pattern recognizer
CN102737278B (en) The projection of joint non-linear stochastic, limited Boltzmann machine and based on batch can parallel optimization come using deep convex network
Olsson et al. From unknown sensors and actuators to actions grounded in sensorimotor perceptions
US9852326B2 (en) Facial expression capture for character animation
Liu et al. Deepfood: Deep learning-based food image recognition for computer-aided dietary assessment
US9928410B2 (en) Method and apparatus for recognizing object, and method and apparatus for training recognizer
WO2016155564A1 (en) Training method and apparatus for convolutional neutral network model
Kheradpisheh et al. Deep networks can resemble human feed-forward vision in invariant object recognition
US20160034788A1 (en) Learning image categorization using related attributes
Liu et al. A deep spatial contextual long-term recurrent convolutional network for saliency detection
US10452979B2 (en) Convolution neural network training apparatus and method thereof
US10417525B2 (en) Object recognition with reduced neural network weight precision
US10354362B2 (en) Methods and software for detecting objects in images using a multiscale fast region-based convolutional neural network
US9734567B2 (en) Label-free non-reference image quality assessment via deep neural network
US10346726B2 (en) Image recognition method and apparatus, image verification method and apparatus, learning method and apparatus to recognize image, and learning method and apparatus to verify image
CN106649542A (en) Systems and methods for visual question answering
US20160239706A1 (en) Convolution matrix multiply with callback for deep tiling for deep convolutional neural networks
US9904875B2 (en) Processing images using deep neural networks
Harley An interactive node-link visualization of convolutional neural networks
CN105493032A (en) Cross-device user interface selection
DE202016107446U1 (en) Rotation of data for calculations in neural networks
JP2016512632A (en) System and method for assigning voice and gesture command areas
CN108351984A (en) The depth convolutional neural networks of hardware-efficient

Legal Events

Date Code Title Description
A201 Request for examination