WO2023058859A1 - 디스플레이 장치 및 그 동작방법 - Google Patents
디스플레이 장치 및 그 동작방법 Download PDFInfo
- Publication number
- WO2023058859A1 WO2023058859A1 PCT/KR2022/010800 KR2022010800W WO2023058859A1 WO 2023058859 A1 WO2023058859 A1 WO 2023058859A1 KR 2022010800 W KR2022010800 W KR 2022010800W WO 2023058859 A1 WO2023058859 A1 WO 2023058859A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- model information
- layer
- layers
- pixels
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 54
- 238000012545 processing Methods 0.000 claims abstract description 142
- 238000013528 artificial neural network Methods 0.000 claims abstract description 47
- 238000012549 training Methods 0.000 description 35
- 238000010586 diagram Methods 0.000 description 23
- 238000001514 detection method Methods 0.000 description 20
- 238000004891 communication Methods 0.000 description 19
- 230000006870 function Effects 0.000 description 18
- 230000008569 process Effects 0.000 description 14
- 238000004590 computer program Methods 0.000 description 10
- 241001465754 Metazoa Species 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 8
- 238000011017 operating method Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 210000002569 neuron Anatomy 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000013139 quantization Methods 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- APTZNLHMIGJTEW-UHFFFAOYSA-N pyraflufen-ethyl Chemical compound C1=C(Cl)C(OCC(=O)OCC)=CC(C=2C(=C(OC(F)F)N(C)N=2)Cl)=C1F APTZNLHMIGJTEW-UHFFFAOYSA-N 0.000 description 1
- 238000000060 site-specific infrared dichroism spectroscopy Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Definitions
- Various embodiments relate to a display device and an operating method thereof. More specifically, it relates to a display device that performs image processing using a neural network and an operating method thereof.
- the learning-based upscaling method refers to transforming and enlarging a low-quality, low-resolution image into a high-quality, high-resolution image using a large-capacity, high-complexity network in which parameter values are determined through learning.
- the network to be used can be designed in various structures, and can be selected according to the system by adjusting the depth of the network (the number of layers) and the number of operation parameters of the network (the size of the filter kernel).
- the learning-based upscaling method shows superior image restoration performance compared to existing signal processing or interpolation-based upscaling methods (eg, Bicubic interpolation, Bilinear interpolation, and Lanczos interpolation).
- interpolation-based upscaling methods eg, Bicubic interpolation, Bilinear interpolation, and Lanczos interpolation.
- the structure of a single input and a single output shows limitations in specialized reconstruction for each object having various characteristics in the image.
- the types of objects included in the image may include people, texts, graphics, artifacts (buildings, sculptures, cars, etc.), and natural objects (animals, plants). There are distinguishing features.
- the disclosed embodiments may provide a display device capable of changing a parameter for each layer of a neural network based on characteristics of each object included in an image and performing image processing using the changed parameter, and an operating method thereof. .
- a display device that performs image processing using a neural network including a plurality of layers may include a display, a memory storing one or more instructions, and executing the one or more instructions stored in the memory, 1 Based on object characteristics corresponding to each of the pixels included in the image, a plurality of model information corresponding to each of the pixels is obtained, and the model information corresponding to each of the pixels input to the neural network is obtained. and identifying a plurality of model information corresponding to each of the plurality of layers based on information about when each of the pixels is processed in the neural network, and based on the plurality of model information, the plurality of models. Updating parameters of the layers, and processing the first image through the plurality of layers to which the updated parameters are applied in each of the plurality of layers, thereby obtaining a second image, and displaying the second image. You can control the display.
- the information on the viewpoint may be selected from among information on data input time when each of the pixels is input to each of the plurality of layers and information on data processing time when each of the pixels is processed in each of the plurality of layers. may contain at least one.
- At least one of the data input time and the data processing time may have a constant cycle.
- the information on the viewpoint may be determined based on the location of each of the pixels in the first image.
- the plurality of layers may be connected in series, and the processor may sequentially input pixel values of pixels included in the first image to the plurality of layers.
- the plurality of layers include a first layer and a second layer next to the first layer, the pixels of the first image include a first pixel and a second pixel, and the processor, the one or more instructions By executing, based on a first pixel value of the first pixel being input to the first layer, a parameter of the first layer is updated to a parameter included in first model information corresponding to the first pixel; Based on the first pixel value of the first pixel being input to the second layer and the second pixel value of the second pixel being input to the first layer, the parameter of the first layer is set to the second layer.
- a parameter included in the second model information corresponding to a pixel may be updated, and a parameter of the second layer may be updated as a parameter included in the first model information.
- the plurality of layers further includes a third layer next to the second layer,
- the pixels of the first image further include a third pixel, and the processor, by executing the one or more instructions, transfers the first pixel value of the first pixel to the third layer to the second pixel.
- a third model corresponding to the third pixel based on the input of the second pixel value to the second layer and the input of the third pixel value of the third pixel to the first layer. It is possible to update parameters included in information, update parameters of the second layer to parameters included in the second model information, and update parameters of the third layer to parameters included in the first model information. .
- the processor may, by executing the one or more instructions, detect object regions included in the first image, and pixels included in the first image based on the plurality of model information corresponding to the object regions. It is possible to obtain the plurality of model information corresponding to each.
- the processor may obtain the plurality of pieces of model information respectively corresponding to the pixels based on a weighted sum of model information corresponding to the object regions by executing the one or more instructions. .
- the processor may, by executing the one or more instructions, based on a distance between a first pixel included in the first image and a center of each of the object regions, corresponding to each of the object regions. Weights of the plurality of model information may be determined, and model information corresponding to the first pixel may be obtained based on the plurality of model information respectively corresponding to the object regions and the determined weights.
- a method of operating a display device that performs image processing using a neural network including a plurality of layers includes receiving a first image, and pixels corresponding to pixels included in the first image. Obtaining a plurality of pieces of model information respectively corresponding to the pixels based on object characteristics, processing the model information and each of the pixels respectively corresponding to the pixels input to the neural network in the neural network.
- the information on the viewpoint may be selected from among information on data input time when each of the pixels is input to each of the plurality of layers and information on data processing time when each of the pixels is processed in each of the plurality of layers. may contain at least one.
- At least one of the data input time and the data processing time may have a constant cycle.
- the information on the viewpoint may be determined based on the location of each of the pixels in the first image.
- the plurality of layers may be connected in series, and a pixel value of each of pixels included in the first image may be sequentially input to the plurality of layers and sequentially output.
- the plurality of layers include a first layer and a second layer next to the first layer, the pixels of the first image include a first pixel and a second pixel, and parameters of each of the plurality of layers
- the updating may include updating a parameter of the first layer to a parameter included in first model information corresponding to the first pixel, based on a first pixel value of the first pixel being input to the first layer. and parameters of the first layer based on the first pixel value of the first pixel being input to the second layer and the second pixel value of the second pixel being input to the first layer. It may include updating a parameter included in second model information corresponding to the second pixel, and updating a parameter of the second layer to a parameter included in the first model information.
- the plurality of layers further include a third layer subsequent to the second layer, the pixels of the first image further include a third pixel, and the updating of parameters of each of the plurality of layers comprises: The first pixel value of the first pixel is input to the third layer, the second pixel value of the second pixel is input to the second layer, and the third pixel value of the third pixel is input to the first layer Based on this, updating the parameter of the first layer to a parameter included in the third model information corresponding to the third pixel, updating the parameter of the second layer to a parameter included in the second model information, The method may include updating parameters of the third layer to parameters included in the first model information.
- the operating method may further include detecting object regions included in the first image, and obtaining model information corresponding to each of the pixels includes the plurality of model information corresponding to the object regions. Based on the information, obtaining the plurality of model information respectively corresponding to the pixels included in the first image.
- the obtaining of model information corresponding to each of the pixels may include obtaining the plurality of model information corresponding to each of the pixels based on a weighted sum of model information corresponding to the object regions.
- Obtaining model information corresponding to each of the pixels may include the plurality of models corresponding to the object regions, based on a distance between a first pixel included in the first image and a center of each of the object regions. determining weights of model information of , and acquiring model information corresponding to the first pixel based on the plurality of model information respectively corresponding to the object regions and the determined weights.
- a non-transitory computer-readable recording medium includes the steps of receiving a first image, based on object characteristics corresponding to each of the pixels included in the first image, and corresponding to each of the pixels. Acquiring a plurality of model information, the plurality of layers based on the model information corresponding to each of the pixels input to the neural network and information about a point in time when each of the pixels is processed in the neural network; Identifying a plurality of model information corresponding to each of the plurality of model information, updating parameters of the plurality of layers based on the plurality of model information, in each of the plurality of layers, the updated parameters are applied
- a program containing instructions to be executed may be stored.
- a display device obtains object characteristics for each region and pixel based on objects included in an image, and performs image processing using different model information according to the object characteristics for each region and pixel. can do. Accordingly, accuracy or performance of image processing may be improved.
- a display device may update parameters for each layer without updating all parameters included in an image processing network with the same model information, thereby preventing delay in image processing.
- FIG. 1 is a diagram illustrating a display device according to an exemplary embodiment.
- FIG. 2 is a diagram illustrating the structure of an image processing network according to an exemplary embodiment.
- FIG. 3 is a flowchart illustrating a method of operating a display device according to an exemplary embodiment.
- FIG. 4 is a diagram for explaining a method of acquiring model information corresponding to a first image by a display device according to an exemplary embodiment.
- FIG. 5 is a diagram illustrating model information for each pixel and information about a point in time when a pixel is processed in an image processing network, according to an exemplary embodiment.
- 6 and 7 are diagrams for explaining a method of updating a parameter of an image processing network based on parameter update information in a display device according to an exemplary embodiment.
- FIG. 8 is a diagram for explaining a method of acquiring a plurality of model information according to an exemplary embodiment.
- FIG. 9 is a diagram for explaining a method of obtaining model information corresponding to a first image according to an exemplary embodiment.
- FIG. 10 is a diagram for explaining a method of obtaining model information corresponding to a first image according to an exemplary embodiment.
- FIG. 11 is a block diagram illustrating a configuration of a display device according to an exemplary embodiment.
- FIG. 12 is a block diagram illustrating a configuration of a display device according to another exemplary embodiment.
- the expression “at least one of a, b, or c” means “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b” and c”, or variations thereof.
- the term "user” means a person who controls a system, function, or operation, and may include a developer, administrator, or installer.
- 'image' or 'picture' may indicate a still image, a motion picture composed of a plurality of continuous still images (or frames), or a video.
- model information may refer to parameters of the neural network of the model, such as inputs to neurons of the neural network, weights applied to neurons to be multiplied, bias applied to each neuron, and the like.
- FIG. 1 is a diagram illustrating a display device according to an exemplary embodiment.
- a display device 100 may be an electronic device that receives an image and performs image processing on the received image.
- the image processing may include upscaling, quality processing, and the like, but is not limited thereto.
- the display device 100 includes TVs, mobile phones, tablet PCs, digital cameras, camcorders, laptop computers, desktops, e-book readers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), It can be implemented in various forms such as navigation, MP3 players, wearable devices, and the like.
- PDAs personal digital assistants
- PMPs portable multimedia players
- the display device 100 may include a display and display an image on which image processing has been performed.
- the display device 100 may be a fixed electronic device disposed at a fixed location or a mobile electronic device having a portable form, and may be a digital broadcasting receiver capable of receiving digital broadcasting.
- the embodiments may be easily implemented in an image processing device having a large display such as a TV, but is not limited thereto.
- the display device 100 may perform upscaling or quality improvement processing of the image 10 received or input from an external device or external server.
- the display device 100 may display an upscaling or improved image quality.
- the display device 100 inputs the input image 10 to the image processing network 30 and performs an operation on each of a plurality of layers included in the image processing network 30, thereby upscaling or improving image quality.
- An output image 20 may be acquired.
- the image processing network 30 may include a structure in which the first to nth layers 31, 32, 33, ..., 39 are connected in series. Accordingly, data input to the first layer 31 is processed through calculation with parameters of the first layer 31 , and the processed result is input to the second layer 32 . In addition, data input to the second layer 32 may be processed through calculation with parameters of the second layer 32 , and the processed result may be input to the third layer 33 . In this way, the data input to the first layer 31 is sequentially processed in each of the first to nth layers 31, 32, 33, ..., 39, so that the final data in the nth layer 39 is obtained. can be output.
- the input image 10 may be input to the image processing network 30 in units of regions. For example, it may be input to the image processing network 30 in units of a predetermined area having each of the pixels included in the input image 10 as a center pixel.
- an operation of inputting the input image 10 to the image processing network 30 may mean an operation of inputting the input image 10 to the first layer 31 of the image processing network 30 . .
- the size of the region of the input image 10 may be determined based on the size of the parameter (filter kernel) of the first layer 31 .
- the size of the parameter of the first layer 31 is 3 x 3
- the input to the first layer 31 The size of the region may be 3 x 3.
- the input image 10 may include one or more objects.
- the input image 10 has characteristics that are distinguished for each object region according to the type of object included in the input image 10 , and a model optimized for image processing varies for each object region.
- the display apparatus 100 acquires object characteristics for each region and each pixel based on the objects included in the input image 10, and obtains object characteristics for each region and each pixel.
- Image processing may be performed using model information.
- the display device 100 may perform image processing using different model information for each region and each pixel included in the input image 10, and the image processing will be described in detail with reference to FIG. 2. do.
- FIG. 2 is a diagram illustrating the structure of an image processing network according to an exemplary embodiment.
- the image processing network 30 may include a structure in which a plurality of layers are connected in series.
- the plurality of layers may include first through n-th layers 31, 32, 33, ..., 39.
- the first image 210 may be input to the image processing network 30 in units of regions.
- the first area 211 , the second area 212 , and the third area 213 of the first image 210 may be sequentially input to the image processing network 30 .
- the first region 211 or the center pixel of the first region 211 may have a first object characteristic (eg, “face” characteristic)
- the second region 212 or the second region 212 ) may have a second object property (eg, a “text” property).
- the third area 213 or a central pixel of the third area 213 may have a third object property (eg, “background” property).
- the display apparatus 100 performs image processing on the first to third areas 211, 212, and 213 using different model information according to object characteristics instead of the same model information. , it is possible to improve the accuracy or performance of image processing.
- the display apparatus 100 obtains model information corresponding to the object characteristics of the input region, and according to the model information, the image processing network ( 30) may update parameters of a plurality of layers included in.
- the display device 100 simultaneously updates parameters for all of the plurality of layers included in the image processing network 30, the input region corresponds to the first to nth layers 31, 32, 33, . .., 39), the processing of the next area is delayed until it is sequentially processed.
- the image processing network 30 processes the first area 211
- the image processing network 30 uses first model information corresponding to a first object characteristic of the first area 211. All parameters of the first to nth layers 31, 32, 33, ..., 39 of may be simultaneously updated. While the image processing network 30 processes the first area 211, the image processing network 30 processes the first area ( The second area 212 cannot be processed until the image processing of 211) is completed.
- a delay occurs in image processing of the currently input area until image processing of the previously input area is completed.
- the delay time increases.
- the display device 100 does not update all parameters included in the image processing network 30 to the same model information based on model information corresponding to a region input to the image processing network 30. Parameters can be updated for each layer.
- the display apparatus 100 may obtain model information of a region input to a corresponding layer in each of the layers, and update parameters for each layer based on the obtained model information.
- the display apparatus 100 is configured to generate an image processing network 30 based on model information for each region (by pixel) of an input image and information on a viewpoint processed by the image processing network 30 for each region (by pixel). Parameters of each of a plurality of layers included in may be updated.
- regions of the image input to the image processing network 30 are processed and output in the order input to the image processing network 30 .
- regions of the first image 210 are input to the first layer 31, and the first to nth layers 31, 32, 33, ..., 39 are sequentially pipelined. , and output from the n-th layer 39.
- the region processed in the K-th layer at time T is processed in the K+1-th layer at time T+1, based on the model information of the K-th layer at time T (previous time), at time T+1 ( Parameters of the K+1th layer of the present time) may be updated.
- the first region 211, the second region 212, and the third region 213 of the first image are sequentially input to the image processing network 30,
- the first region 211 is processed by the first layer 31, at time T+1, the first region 211 is processed by the second layer 32, which is a layer following the first layer 31.
- the second region 212 is processed in the first layer 31 .
- the first region 211 is processed by the third layer 33, which is the next layer after the second layer 32, and the second region 212 is processed by the second layer 32.
- the third region 213 is processed in the first layer 31 .
- the parameters of the first layer 31 at time T may be updated based on the first model information corresponding to the first region 211 .
- the parameters of the first layer 211 are updated based on the second model information corresponding to the second area 212, and the parameters of the second layer 212 are based on the first model information. so it can be updated.
- the parameters of the first layer 31 are updated based on the third model information corresponding to the third region 213, and the parameters of the second layer 212 are updated based on the second model information. Therefore, the parameters of the third layer 213 may be updated based on the first model information.
- the display device 100 may perform an operation based on the parameter updated in each of the layers, and input the result values calculated in each of the layers to the next layer.
- the display device 100 may obtain a second image 220 by processing the first image 210 by repeating the parameter update operation and the calculation operation in each of the layers at predetermined intervals.
- FIG. 3 is a flowchart illustrating a method of operating a display device according to an exemplary embodiment.
- the display apparatus 100 may receive a first image (S310).
- the display device 100 may receive or receive a first image from an external device or an external server.
- the display apparatus 100 may obtain model information corresponding to the received first image (S320).
- the display apparatus 100 may extract object characteristics corresponding to each of the pixels included in the first image, and obtain model information corresponding to each of the pixels based on the object characteristics. For example, the display apparatus 100 may detect one or more objects included in the first image and determine one or more object regions based on the detected objects. In this case, the display apparatus 100 may use a method of expressing the detected object area included in the first image as a bounding box (rectangle) and a segmentation method of expressing the area in units of pixels. However, it is not limited thereto.
- the display apparatus 100 may determine model information corresponding to object characteristics of the object region as model information corresponding to pixels included in the object region. For example, the display apparatus 100 may allocate model information corresponding to face characteristics to pixels included in the "face" region detected in the first image. Also, the display device 100 may allocate model information corresponding to text characteristics to pixels included in the “text” area.
- the display apparatus 100 may obtain model information corresponding to each of the pixels included in the first image in the same manner as above.
- model information is obtained for each pixel
- the present invention is not limited thereto, and model information may be obtained for each region of the first image including one or more pixels.
- the display apparatus 100 may obtain model information of each of the pixels included in the first image based on a weighted sum of model information corresponding to object regions detected in the first image. For example, the display apparatus 100 may determine a weight for model information corresponding to the object region based on a distance between a first pixel included in the first image and the center of the object region detected in the first image. there is. The display apparatus 100 may obtain model information of the first pixel based on the determined weight. This will be described later in detail with reference to FIGS. 9 and 10 .
- the display device 100 may obtain model information corresponding to each layer for each layer based on model information corresponding to each pixel and information about when each pixel is processed in an image processing network. Yes (S330).
- the display apparatus 100 may obtain model information corresponding to a layer for each layer based on pixel information included in the first image, model information corresponding to the pixel, and parameter update information to which viewpoint information corresponding to the pixel is mapped. there is.
- view information corresponding to a pixel may indicate a time when a region having the corresponding pixel as a center pixel is input to the first layer of the image processing network or a region input to the first layer is processed in the first layer. .
- the display device 100 may obtain model information corresponding to the first layer based on the model information of each pixel and the point in time when a region having each of the pixels as a center pixel is input to the first layer of the image processing network. there is.
- the display device 100 may obtain model information corresponding to the remaining layers of the image processing network, based on model information used to set parameters of the previous layer for the remaining layers except for the first layer at a previous point in time.
- the display device 100 may update a parameter of each layer based on model information acquired for each layer (S340).
- the display apparatus 100 updates parameters of the first layer to parameters included in first model information corresponding to the first region based on the input of the first region of the first image to the first layer, and
- the area is input to the second layer, which is a layer following the first layer, and based on the fact that the second area is input to the first layer, parameters of the first layer are changed to parameters included in the second model information corresponding to the second area.
- the parameters of the second layer may be updated to parameters included in the first model information.
- parameters of the first layer are set to the third layer based on inputs of the first area to the third layer, which is a layer following the second layer, the second area to the second layer, and the third area to the first layer.
- the display device 100 may obtain a second image by performing an operation in each of a plurality of layers based on the updated parameter (S350).
- the display device 100 may perform an operation in each of a plurality of layers based on the updated parameter and input the result of the operation to the next layer.
- the display device 100 may acquire the second image by repeating the parameter update operation and the calculation operation in each of the plurality of layers at predetermined intervals.
- the display device 100 may display the acquired second image (S360).
- FIG. 4 is a diagram for explaining a method of acquiring model information corresponding to a first image by a display device according to an exemplary embodiment.
- the display device 100 may include an object detection module.
- An object detection module may include appropriate logic, circuitry, interfaces, and/or codes operated to detect an object in the first image 410 .
- the object detection module identifies objects included in an image using the object detection network 420 and obtains type, size, and location information of the identified objects.
- the object detection network 420 may be a neural network that receives an image and detects at least one object included in the input image.
- the object detection network 420 detects one or more objects from the first image 410 using one or more neural networks, and includes object classes corresponding to the one or more detected objects and locations of the objects.
- Object information can be output.
- object detection network 420 generally includes three steps: selecting object candidate regions, extracting features from each candidate region, and classifying types of object candidate regions by applying a classifier to the extracted features. can do.
- localization performance can be improved through post-processing such as bounding box regression.
- the object detection network 420 may be a Deep Neural Network (DNN) having a plurality of internal layers that perform operations, and the internal layers are convolutional layers that perform convolution operations. It may be a Convolution Neural Network (CNN) composed of, but is not limited thereto.
- DNN Deep Neural Network
- CNN Convolution Neural Network
- an object detection network 420 may include a region proposal module 421 , a CNN 422 , and a classifier module 423 .
- the region suggestion module 421 may extract a candidate region from the first image 410 .
- the number of candidate regions may be limited to a preset number, but is not limited thereto.
- the CNN 422 may extract feature information from the region generated by the region suggestion module 421 .
- the classifier module 423 may perform classification by receiving feature information extracted from the CNN 422 as an input.
- the neural network In order for the neural network to accurately output result data corresponding to input data, the neural network must be trained according to a purpose.
- 'training' means inputting various data into the neural network, analyzing the input data, classifying the input data, and/or extracting features necessary for generating result data from the input data. It can mean training the neural network so that the neural network can discover or learn how to do it by itself.
- the neural network may train training data (eg, a plurality of different images) to optimize and set weight values inside the neural network. And, by self-learning the input data through a neural network having optimized weight values, a desired result is output.
- a weight value included in the object detection network 420 allows the object detection network 420 to detect at least one object included in an image input to the object detection network 420 through training.
- the object detection network 420 may be trained to detect various types of object information, such as face (person), text, artifact, and natural object (background), from the image.
- the object detection network 420 on which training is completed may receive an image, detect at least one object included in the image, and output the detected result.
- the object detection network 420 may detect various types of object areas included in the first image 410, such as a face (person), text, artifact, and natural object (background).
- the image 430 output from the object detection network 420 may include information about the object detected in the input first image 410 .
- the information on the object may include information on the class of the detected object and a bounding box 435 indicating the location of the detected object.
- objects detected in the first image 410 input in various formats may be displayed on the output image 430 .
- the detected object area is illustrated as having a quadrangular bounding box shape, but is not limited thereto.
- the object detection module may segment the first image in pixel units and detect an object region based on the segmented region. In this case, object regions of various shapes may be detected.
- the display apparatus 100 may obtain model information corresponding to the first image 410 based on object characteristics of detected object regions. For example, a model A corresponding to a face characteristic may be assigned to pixels included in the first object area 441 and the second object area 442 . In addition, a model B corresponding to text characteristics may be assigned to pixels included in the third object area 443, and a model B corresponding to background characteristics may be assigned to pixels included in the remaining area, the fourth object area 444. C can be assigned.
- FIG. 5 is a diagram illustrating model information for each pixel and information about a point in time when a pixel is processed in an image processing network, according to an exemplary embodiment.
- the display apparatus 100 may obtain model information 520 corresponding to each of the pixels included in the first image 510 . Since the method for obtaining the model information 520 has been described in detail with reference to FIG. 4 , a detailed description thereof will be omitted.
- the display device 100 obtains information (view information) regarding a point in time when each of the pixels included in the first image 510 is processed in the image processing network 30 according to an exemplary embodiment. can do.
- the viewpoint information may include information about a viewpoint when a region having each of the pixels included in the first image as a center pixel is input to the image processing network 30 .
- the time point at which the first region having the first pixel P1 as the center pixel is input to the image processing network 30 is T1
- the second pixel P2 is the center pixel.
- a time point at which the second region to be input to the image processing network 30 may be T2.
- the viewpoint information according to an embodiment is information on a viewpoint when a region having each of the pixels included in the first image as a center pixel is input to the first layer, or each of the pixels included in the first image is a center pixel.
- An area referred to as may include information about a point in time at which an operation with a parameter is performed in the first layer. However, it is not limited thereto.
- viewpoint information may be determined based on the order in which each region included in the first image 510 is input to the image processing network 30, the input cycle, and the cycle processed in each layer.
- the viewpoint information is information about the viewpoints when the regions are input to the image processing network 30, the first region and the second region are input to the image processing network 30 in order, and the first and second regions are inputted to the image processing network 30.
- view information T2 of the second area may be determined as T1 (view information of the first area) + P0.
- viewpoint information is information on the viewpoints when the regions are processed in the first layer 31, and is input to the image processing network in the order of the first region and the second region.
- viewpoint information T2 of the second region may be determined as T1 (view information of the first region) + P1.
- the display apparatus 100 may obtain model information 520 corresponding to each pixel and parameter update information 530 mapped to corresponding viewpoint information.
- Parameter update information 530 may appear in various forms.
- 6 and 7 are diagrams for explaining a method of updating a parameter of an image processing network based on parameter update information in a display device according to an exemplary embodiment.
- the display device 100 may include a parameter update unit 610 and a model information storage unit (eg, memory) 620 .
- the parameter update unit 610 includes appropriate logic, circuits, interfaces, and /or may contain code.
- the parameter updater 610 may determine model information corresponding to a layer for each layer based on model information for each pixel included in the parameter update information.
- the parameter update unit 610 may obtain the determined model information from the model information storage unit 620 and update the parameters of the layer.
- the model information storage unit 620 may store a plurality of model information received from an external device or an external server.
- the plurality of model information may be parameter information of an image processing network in which an image processing network is trained using training image sets having different object characteristics, and training has been completed. A method of acquiring a plurality of pieces of model information will be described later in detail with reference to FIG. 8 .
- a clock signal for inputting a pixel value of a region or pixel to the image processing network 30 according to an embodiment, and a clock signal applied to the image processing network 30 and a clock signal applied to the parameter update unit 610 are synchronized.
- the parameter update unit 610 may receive parameter update information and update parameters of a plurality of layers based on the parameter update information. This will be described in detail with reference to FIG. 7 .
- the parameter update information may be information in which pixel information, model information corresponding to the pixel, and viewpoint information corresponding to the pixel are mapped.
- the viewpoint information corresponding to the pixel is the time when the region having the corresponding pixel as the center pixel is input to the first layer 31 or the region input to the first layer 31 is processed in the first layer 31.
- Viewpoint information may indicate data input time or data processing time of each layer. However, it is not limited thereto.
- the parameter update unit 610 generates the first layer 31 based on the model information of each of the pixels included in the parameter update information and the point in time when a region having each of the pixels as a center pixel is input to the first layer 31.
- the parameters of 1 layer 31 can be updated.
- the parameter update information includes information that the model information corresponding to the pixel P1 is the model A and that the region centered on the pixel P1 is input as the first layer at time T1
- the parameter update As shown in FIG. 7 , the unit 610 may set the parameter of the first layer 31 as the first parameter included in the model A based on the time point T1.
- parameter update information includes information indicating that the model information corresponding to the second pixel P2 is model A and that the region centered on the second pixel P2 is input to the first layer 31 at time T2
- parameter update The unit 610 may maintain the first parameter as the first parameter without updating the parameter of the first layer 31 .
- parameter update information includes information indicating that the model information corresponding to the third pixel P3 is model B and that the area centered on the third pixel P3 is input to the first layer 31 at time point T3, parameter update The unit 610 may update the parameter of the first layer ( ) to a second parameter included in the model B based on time T3.
- the parameter updater 610 may update the parameters of the layers other than the first layer 31 based on model information used to set the parameters of the previous layer at a previous point in time.
- the parameter update unit 610 uses the parameters of the second layer 32 at time T2 and parameters of the first layer 31, which is the previous layer, at time T1, which is a previous time T2, included in the model A. It can be set as a parameter.
- the parameter of the second layer 32 at time T4 may be updated to the fourth parameter included in the model B used for parameter setting of the first layer 31 at the previous time T3.
- the parameter update unit 610 may update parameters of the third layer 31 to the n-th layer 39 in the same manner as that of the second layer 32 .
- calculation of the updated parameter and the input area may be performed in each of the layers.
- the model information (eg, model information A) applied to the previous layer (eg, the second layer 32) at the previous time point (eg, T2) is the next time point (eg, T2).
- T3 may be applied to the next layer (eg, the third layer 33).
- a set of model information eg, model information CBBBAA
- it may be successively assigned to layers (eg, the first layer 31 , the second layer 32 , and the third layer 33 ).
- the rightmost model (eg model A) is assigned to the first layer 31 at time T1
- the first two models in the series of model information (Model information AA) is assigned to the first layer 31 and the second layer 32 respectively at time T2
- the first three models in a series of model information (model information AAB) are assigned to the first layer 31 at time T3.
- the second layer 32, and the third layer 33 respectively.
- Mapping information between viewpoints, layers 31-33, and a set of model information is determined and stored in memory before the layers 31-33 are updated, and is delayed according to data input time or data processing time. Without it, the corresponding model information can be applied to the layers 31-33.
- Views according to an embodiment may have a preset cycle, in which the cycle is a cycle of inputting regions (pixels) to the image processing network 30, a cycle of inputting and outputting regions to each layer, It may be determined based on the time required for calculations performed in each of the layers, etc., but is not limited thereto.
- the point of view information included in the parameter update information is the point in time when each region is input to the first layer 31, but even when each region is processed in the first layer 31, Parameters for a plurality of layers may be updated in the same manner.
- the image processing performance is improved while the image processing is performed. delay can be avoided.
- FIG. 8 is a diagram for explaining a method of acquiring a plurality of model information according to an exemplary embodiment.
- a plurality of pieces of model information may be determined by an external device, and in this case, the external device may be a separate device different from the display device 100 according to an embodiment.
- the external device may determine parameters included in the image processing network 810 by training the image processing network 810 based on a training data set.
- the image processing network 810 of FIG. 8 may have the same structure as the image processing network 30 described in FIGS. 1 to 7 .
- the image processing network 810 may be a deep neural network (DNN) including a plurality of layers, and in order for the neural network to accurately output result data corresponding to input data, the neural network is trained ( training) should be done.
- 'training' is inputting various data into the neural network, analyzing the input data, classifying the input data, and/or extracting features necessary for generating result data from the input data. It can mean training the neural network so that the neural network can discover or learn how to do it by itself.
- the neural network may learn training data and optimize and set internal parameters (weights, coefficients) of the neural network.
- a neural network set with optimized parameters can output a desired result by self-learning input data.
- the image processing network 810 may be a neural network that receives an image, performs image processing such as upscaling or quality processing, and outputs an image-processed image.
- the external device may obtain a plurality of pieces of model information by training the image processing network 810 based on a plurality of training data sets.
- a plurality of training data sets may be generated based on a plurality of training images having different object characteristics.
- the first training data set may include low-resolution face images and high-resolution face images.
- the second training data set may include low-resolution text images and high-resolution text images.
- the third training data set may include low-resolution animal images and high-resolution animal images, and the fourth training data set may include low-resolution background images and high-resolution background images.
- the external device may determine first model information (parameter information of model A) by training the image processing network 810 based on the first training data set.
- the external device inputs the low-resolution face image included in the first training data set to the image processing network 810, and outputs the image in a direction that minimizes the difference between the output image (output data) and the high-resolution face image.
- first model information (parameter information of model A) may be determined.
- the trained image processing network 810 may be an upscaling model (model A) optimized for a face image.
- the external device may determine second model information (parameter information of model B) by training the image processing network 810 based on the second training data set. Accordingly, based on the second training data set, the trained image processing network 810 may be an upscaling model (model B) optimized for text images.
- model B upscaling model
- the external device may determine third model information (parameter information of model C) by training the image processing network 810 based on the third training data set. Accordingly, based on the third training data set, the trained image processing network 810 may be an upscaling model (model C) optimized for animal images.
- model C upscaling model
- the external device may determine fourth model information (parameter information of model D) by training the image processing network 810 based on the fourth training data set. Accordingly, based on the fourth training data set, the trained image processing network 810 may be an upscaling model (model D) optimized for the background image.
- model D upscaling model
- FIG. 9 is a diagram for explaining a method of obtaining model information corresponding to a first image according to an exemplary embodiment.
- the display apparatus 100 may detect one or more objects included in the first image 910 and determine an object area. For example, the display apparatus 100 may divide the first image 910 into a first object area 921 where a human face is detected and a second object area 922 where a human face is not detected. In this case, the display apparatus 100 according to an exemplary embodiment assigns a first model (model A) corresponding to facial characteristics to pixels included in the first object area 921 and assigns a first model (model A) to the second object area 922. A second model (model B) corresponding to the background characteristics may be assigned to included pixels.
- the display device 100 updates parameters included in the image processing network based on model information corresponding to each of the pixels included in the first image 910, and updates the updated parameters. Based on the parameter, image processing may be performed on a region centered on each of the pixels.
- model information is transferred from the first model information to the second model information or , the second model information is changed to the first model information.
- discontinuity may occur at the boundary between the first object region and the second object region of the second image output from the image processing network due to a sudden change in parameters.
- the display device 100 may change the pixels included in the first image 910 so that model information between the first object area 921 and the second object area 922 is gradually changed.
- Model information corresponding to each may be obtained based on a weighted sum of first model information corresponding to the first object region 921 and second model information corresponding to the second object region 922 .
- model information corresponding to each of the pixels included in the first image is expressed by Equation 1 below.
- Model information first weight X first model information + (1 - first weight) X second model information
- Equation 1 can be expressed as Equation 2 as follows.
- Model information second model information + first weight X (first model information - second model information)
- Equation 2 if the difference between the first model information and the second model information is defined as a delta model, Equation 2 can be simply expressed as Equation 3 below.
- Model Information Second Model Information + First Weight X Delta Model
- the delta model of Equation 3 can be used to obtain model information without additional calculation by calculating the difference between the first model information and the second model information in advance.
- a method for obtaining model information without using a delta model requires a multiplication operation twice as many as the number of parameters included in the model, and when the number of parameters is 1 million, 2 million multiplication operation is required.
- the multiplication operation is reduced by half compared to the method in Equation 1. Accordingly, when implementing a network, additional power consumption can be minimized.
- the delta model represents a difference between models, and the statistical distribution of parameters included in the delta model is mainly concentrated in a small range. Therefore, the delta model is advantageous for quantization or compression.
- a partial region of the image may be expressed as a background region and one object region. Therefore, a method for acquiring model information using a delta model according to an embodiment is 3 It is also applicable to the case of including more than one object area.
- the display apparatus 100 determines a first weight based on a distance between a first pixel 930 included in the first image 910 and the center 940 of the first object area 921. can decide In this case, when the first object area 921 has a bounding box shape, the center 940 of the first object area 921 may be calculated through Equation 4 below.
- Center coordinate of object area (upper left coordinate of object area + lower right coordinate of object area)/2
- the first weight has a value of 0 or more and 1 or less, and is in inverse proportion to a distance between the first pixel 930 and the center 940 of the first object area 921 . For example, as the distance from the center 940 of the first object area 921 increases, the first weight value decreases, and as the distance decreases, the first weight value increases.
- model information corresponding to each of the pixels included in the first image 910 is represented by a weighted sum of first model information (model information A) and second model information (model information B)
- a parameter according to an embodiment
- the update unit 610 may calculate a weighted sum of the first model information and the second model information based on the first weight, and update the parameter of the layer based on the calculated model information.
- FIG. 10 is a diagram for explaining a method of obtaining model information corresponding to a first image according to an exemplary embodiment.
- a first image 1010 may be divided into three or more object regions.
- the first image 1010 may be divided into first to fourth object regions 1021 , 1022 , 1023 , and 1024 .
- the first object area 1021 is an area where a face is detected and may include face characteristics.
- the second object area 1022 is an area where text is detected and may include text characteristics,
- the third object area 1023 is an area where animals are detected and may include animal characteristics, and the fourth object area 1024 is an area where animals are detected.
- the display device 100 includes model information corresponding to each of the pixels included in the first image 1010, first model information (A) corresponding to face characteristics, and second model information (A) corresponding to text characteristics. It may be obtained based on a weighted sum of model information (B), third model information (C) corresponding to animal characteristics, and fourth model information (D) corresponding to background characteristics.
- the model information for the first pixel 1030 included in the first image 1010 is a value obtained by applying a first weight to the first model information (A), and a value obtained by applying a second weight to the second model information (B). It can be obtained by summing a value obtained by applying a weighted value, a value obtained by applying a third weighted value to the third model information (C), and a value obtained by applying a fourth weighted value to the fourth model information (D).
- the first weight is a value inversely proportional to the distance between the first pixel 1030 and the center 1040 of the first object area 1021
- the second weight is between the first pixel 1030 and the second object area ( 1022) is a value inversely proportional to the distance between the center 1050
- the third weight is a value inversely proportional to the distance between the first pixel 1030 and the center 1060 of the third object area 1023.
- the first to third weights may be normalized values
- the fourth weight may be determined as a value obtained by subtracting the first to third weights from 1.
- the display device 100 uses only weights determined according to the distance between the pixel and the center of the object area to calculate model information corresponding to the pixel, but is not limited thereto, and the object area is not limited thereto.
- the weight may be determined by further considering the width of the object region, the ratio of the object region in the first image, the brightness of the object region, and the like, as well as the distance to the first image.
- FIG. 11 is a block diagram illustrating a configuration of a display device according to an exemplary embodiment.
- a display device 100 may include an image receiver 110, a processor 120, a memory 130, and a display 140.
- the image receiving unit 110 may include a communication interface, an input/output interface, and the like.
- the communication interface may transmit/receive data or signals with an external device or server.
- the communication interface may include a transceiver, a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, and the like.
- each communication module may be implemented in the form of at least one hardware chip.
- the Wi-Fi module and the Bluetooth module perform communication using the Wi-Fi method and the Bluetooth method, respectively.
- various types of connection information such as an SSID and a session key are first transmitted and received, and various types of information can be transmitted and received after communication is established using the same.
- the wireless communication module includes zigbee, 3 rd Generation (3G), 3 rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE Advanced (LTE-A), 4 th Generation (4G), and 5G (5 th Generation) may include at least one communication chip that performs communication according to various wireless communication standards.
- the input/output interface receives video (eg, motion picture, etc.), audio (eg, voice, music, etc.), and additional information (eg, EPG, etc.) from the outside of the display device 100.
- Input and output interfaces include HDMI (High-Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port , D-subminiature (D-SUB), digital visual interface (DVI), component jack, and PC port.
- the image receiving unit 110 may receive one or more images.
- the processor 120 controls overall operation of the display device 100 and signal flow between durable components of the display device 100 and processes data.
- the processor 120 may include a single core, a dual core, a triple core, a quad core, and multiple cores thereof. Also, the processor 120 may include a plurality of processors. For example, the processor 120 may be implemented as a main processor and a sub processor operating in a sleep mode.
- the processor 120 may include at least one of a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), and a Video Processing Unit (VPU). Alternatively, according to embodiments, it may be implemented in the form of a system on chip (SoC) in which at least one of a CPU, a GPU, and a VPU is integrated.
- SoC system on chip
- the memory 130 may store various data, programs, or applications for driving and controlling the display device 100 .
- a program stored in memory 130 may include one or more instructions.
- a program (one or more instructions) or application stored in memory 130 may be executed by processor 120 .
- the processor 120 may include at least one of the object detection module described in FIG. 4 and the parameter update unit described in FIG. 6 .
- the processor 120 may extract object characteristics corresponding to each of the pixels included in the received first image, and obtain model information corresponding to each of the pixels based on the object characteristics. For example, the processor 120 may detect one or more objects included in the first image and determine one or more object regions based on the detected objects. In this case, the processor 120 may use a method of expressing the detected object area included in the first image as a bounding box (rectangle) and a segmentation method of expressing the area in units of pixels. However, it is not limited thereto.
- the processor 120 may determine model information corresponding to object characteristics of the object region as model information corresponding to pixels included in the object region. For example, the processor 120 may allocate model information corresponding to face characteristics to pixels included in the "face" region detected in the first image. In addition, model information corresponding to text characteristics may be assigned to pixels included in the “text” area. The processor 120 may obtain model information corresponding to each of the pixels included in the first image in the same manner as above. Also, the processor 120 may obtain model information for each region included in the first image.
- the processor 120 may obtain model information of each of pixels included in the first image based on a weighted sum of model information corresponding to object regions detected in the first image. For example, the processor 120 determines a weight for model information corresponding to the object region based on a distance between a first pixel included in the first image and a center of the object region detected in the first image; Based on the determined weight, model information of the first pixel may be obtained.
- the processor 120 may receive parameter update information in which pixel information of pixels included in the first image, model information corresponding to the pixels, and viewpoint information corresponding to the pixels are mapped.
- view information corresponding to a pixel may be a time when a region having the corresponding pixel as a center pixel is input to the first layer of the image processing network or a time when the region input to the first layer is processed in the first layer.
- the processor 120 obtains model information corresponding to the first layer, based on the model information of each pixel and the time point when the region having each of the pixels as the center pixel is input to the first layer 31 of the image processing network. can do.
- the processor 120 may obtain model information corresponding to the remaining layers of the image processing network, based on model information used to set parameters of the previous layer, for the remaining layers except for the first layer. there is.
- the processor 120 may update a parameter of each layer based on model information obtained for each layer. For example, the processor 120 updates a parameter of the first layer to a parameter included in first model information corresponding to the first region based on the input of the first region of the first image to the first layer. , Based on the fact that the first area is input to the second layer, which is a layer following the first layer, and the second area is input to the first layer, parameters of the first layer are converted to second model information corresponding to the second area. Parameters included in the model information may be updated, and parameters of the second layer may be updated as parameters included in the first model information.
- the processor 120 converts the first region to a third layer that is a layer following the second layer, the second region to the second layer, and the third region to the first layer. Updates the parameters of the parameters included in the third model information corresponding to the third region, updates the parameters of the second layer to parameters included in the second model information, and updates the parameters of the third layer to the first model information. It can be updated with included parameters.
- the processor 120 may perform an operation in each of a plurality of layers based on the updated parameter, and input the result of the operation to the next layer.
- the processor 120 may obtain the second image by repeating the parameter update operation and the calculation operation in each of the plurality of layers at predetermined intervals.
- the processor 120 may control the second image to be displayed on the display 140 .
- the display 140 converts an image signal, a data signal, an OSD signal, a control signal, and the like processed by the processor 120 to generate a driving signal.
- the display 140 may be implemented as a PDP, LCD, OLED, flexible display, or the like, and may also be implemented as a 3D display. Also, the display 140 may be configured as a touch screen and used as an input device in addition to an output device.
- the display 140 may display the second image on which upscaling or quality processing has been performed.
- FIG. 12 is a block diagram illustrating a configuration of a display device according to another exemplary embodiment.
- the display device 1200 of FIG. 12 may be an embodiment of the display device 100 described with reference to FIGS. 1 to 11 .
- a display device 1200 includes a tuner unit 1240, a processor 1210, a display unit 1220, a communication unit 1250, a sensing unit 1230, and an input/output unit. 1270, a video processing unit 1280, an audio processing unit 1285, an audio output unit 1260, a memory 1290, and a power supply unit 1295.
- the communication unit 1250 of FIG. 12 is a component corresponding to the communication interface included in the image receiver 110 of FIG. 11, and the input/output unit 1270 of FIG. 12 is an input/output included in the image receiver 110 of FIG. It is a configuration corresponding to the interface, and the processor 1210 of FIG. 12 is the processor 120 of FIG. 11, the memory 1290 of FIG. 12 is the memory 130 of FIG. 11, and the display unit 1220 of FIG. It is a configuration corresponding to the display 140 in FIG. 11 . Therefore, the same contents as those described above will be omitted.
- the tuner unit 1240 selects broadcast signals received by wire or wirelessly through amplification, mixing, resonance, etc., among many radio wave components to be received by the display device 100. Only the frequency of the channel can be selected by tuning.
- the broadcast signal includes audio, video, and additional information (eg, Electronic Program Guide (EPG)).
- EPG Electronic Program Guide
- the tuner unit 1240 may receive broadcast signals from various sources such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting.
- the tuner unit 1240 may receive a broadcasting signal from a source such as analog broadcasting or digital broadcasting.
- the sensing unit 1230 detects a user's voice, a user's video, or a user's interaction, and may include a microphone 1231, a camera unit 1232, and a light receiving unit 1233.
- the microphone 1231 receives the user's utterance.
- the microphone 1231 may convert the received voice into an electrical signal and output it to the processor 1210 .
- the user's voice may include, for example, a voice corresponding to a menu or function of the display apparatus 1200 .
- the camera unit 1232 may receive an image (eg, continuous frames) corresponding to a user's motion including a gesture within the camera recognition range.
- the processor 1210 may select a menu displayed on the display device 1200 or perform a control corresponding to the motion recognition result by using the received motion recognition result.
- the light receiving unit 1233 receives an optical signal (including a control signal) received from an external control device through a light window (not shown) of a bezel of the display unit 1220 .
- the light receiving unit 1233 may receive an optical signal corresponding to a user input (eg, touch, pressure, touch gesture, voice, or motion) from the control device.
- a control signal may be extracted from the received optical signal under control of the processor 1210 .
- the processor 1210 controls overall operation of the display device 1200 and signal flow between internal components of the display device 1200 and processes data.
- the processor 1210 may execute an operation system (OS) and various applications stored in the memory 1290 when there is a user's input or when a pre-set stored condition is satisfied.
- OS operation system
- the processor 1210 stores signals or data input from the outside of the display device 1200, or RAM used as a storage area corresponding to various tasks performed in the display device 1200, the display device 1200 It may include a ROM and a processor in which a control program for control of is stored.
- the video processing unit 1280 processes video data received by the display device 1200 .
- the video processing unit 1280 may perform various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on video data.
- the audio processing unit 1285 processes audio data.
- the audio processing unit 1285 may perform various processes such as decoding or amplifying audio data and filtering noise. Meanwhile, the audio processing unit 1285 may include a plurality of audio processing modules to process audio corresponding to a plurality of contents.
- the audio output unit 1260 outputs audio included in the broadcast signal received through the tuner unit 1240 under the control of the processor 1210 .
- the audio output unit 1260 may output audio (eg, voice, sound) input through the communication unit 1250 or the input/output unit 1270 .
- the audio output unit 1260 may output audio stored in the memory 1290 under the control of the processor 1210 .
- the audio output unit 1260 may include at least one of a speaker, a headphone output terminal, or a Sony/Philips Digital Interface (S/PDIF) output terminal.
- S/PDIF Sony/Philips Digital Interface
- the power supply 1295 supplies power input from an external power source to components inside the display device 1200 under the control of the processor 1210 . Also, the power supply unit 1295 may supply power output from one or more batteries located inside the display device 1200 to internal components under the control of the processor 1210 .
- the memory 1290 may store various data, programs, or applications for driving and controlling the display device 1200 under the control of the processor 1210 .
- the memory 1290 includes a broadcast reception module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical reception module, a display control module, an audio control module, an external input control module, a power control module, It may include a power control module of an external device connected wirelessly (eg, Bluetooth), a voice database (DB), or a motion database (DB).
- the modules and database of the memory 1290 include a broadcast reception control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, a light reception control function, and a display control in the display device 1200. It may be implemented in the form of software to perform a function, audio control function, external input control function, power control function, or power control function of an external device connected wirelessly (eg, Bluetooth).
- the processor 1210 may perform each function using these software stored in the memory 1290.
- FIGS. 11 and 12 are block diagrams for one embodiment.
- Each component of the block diagram may be integrated, added, or omitted according to specifications of the display device 100 or 1200 that is actually implemented. That is, if necessary, two or more components may be combined into one component, or one component may be subdivided into two or more components.
- the functions performed in each block are for explaining the embodiments, and the specific operation or device does not limit the scope of the present invention.
- a method of operating a display device may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium.
- the computer readable medium may include program instructions, data files, data structures, etc. alone or in combination.
- Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software.
- Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks.
- - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like.
- Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.
- the operating method of the display device according to the disclosed embodiments may be included in a computer program product and provided.
- Computer program products may be traded between sellers and buyers as commodities.
- a computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored.
- a computer program product may include a product in the form of a S/W program (eg, a downloadable app) that is distributed electronically through a manufacturer of an electronic device or an electronic marketplace (eg, Google Play Store, App Store). there is.
- a part of the S/W program may be stored in a storage medium or temporarily generated.
- the storage medium may be a storage medium of a manufacturer's server, an electronic market server, or a relay server temporarily storing SW programs.
- a computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of a server and a client device.
- the computer program product may include a storage medium of the third device.
- the computer program product may include a S/W program itself transmitted from the server to the client device or the third device or from the third device to the client device.
- one of the server, the client device and the third device may execute the computer program product to perform the method according to the disclosed embodiments.
- two or more of the server, the client device, and the third device may execute the computer program product to implement the method according to the disclosed embodiments in a distributed manner.
- a server may execute a computer program product stored in the server to control a client device communicatively connected to the server to perform a method according to the disclosed embodiments.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims (15)
- 복수의 레이어들을 포함하는 뉴럴 네트워크를 이용하여, 영상 처리를 수행하는 디스플레이 장치에 있어서,디스플레이;하나 이상의 인스트럭션들을 저장하는 메모리; 및상기 메모리에 저장된 상기 하나 이상의 인스트럭션들을 실행함으로써,제1 영상에 포함되는 픽셀들 각각에 대응하는 객체 특성에 기초하여, 상기 픽셀들에 각각 대응하는 복수의 모델 정보들을 획득하고,상기 뉴럴 네트워크에 입력되는 상기 픽셀들에 각각 대응하는 상기 모델 정보들과 상기 픽셀들 각각이 상기 뉴럴 네트워크에서 처리되는 시점에 대한 정보에 기초하여, 상기 복수의 레이어들에 각각 대응하는 복수의 모델 정보들을 식별하고,상기 복수의 모델 정보들에 기초하여, 상기 복수의 레이어들의 파라미터들을 업데이트하고,상기 복수의 레이어들 각각에서, 상기 업데이트된 파라미터들이 적용된 상기 복수의 레이어들을 통해 상기 제1 영상을 처리함으로써, 제2 영상을 획득하고,상기 제2 영상을 디스플레이하도록 상기 디스플레이를 제어하는, 디스플레이 장치.
- 제1항에 있어서,상기 시점에 대한 정보는,상기 픽셀들 각각이 상기 복수의 레이어들 각각에 입력되는 데이터 입력 시간에 대한 정보, 및 상기 픽셀들 각각이 상기 복수의 레이어들 각각에서 처리되는 데이터 처리 시간에 대한 정보 중 적어도 하나를 포함하는, 디스플레이 장치.
- 제2항에 있어서,상기 데이터 입력 시간 및 상기 데이터 처리 시간 중 적어도 하나는 일정한 주기를 가지는, 디스플레이 장치.
- 제1항에 있어서,상기 시점에 대한 정보는, 상기 제1 영상에서 상기 픽셀들 각각의 위치에 기초하여 결정되는, 디스플레이 장치.
- 제1항에 있어서,상기 복수의 레이어들은 직렬로 연결되고,상기 프로세서는,상기 제1 영상에 포함되는 픽셀들 각각의 픽셀 값을 상기 복수의 레이어들에 순차적으로 입력하는, 디스플레이 장치.
- 제1항에 있어서,상기 복수의 레이어들은 제1 레이어 및 상기 제1 레이어 다음의 제2 레이어를 포함하고, 상기 제1 영상의 상기 픽셀들은 제1 픽셀 및 제2 픽셀을 포함하고,상기 프로세서는, 상기 하나 이상의 인스트럭션들을 실행함으로써,상기 제1 픽셀의 제1 픽셀 값이 상기 제1 레이어에 입력되는 것에 기초하여, 상기 제1 레이어의 파라미터를 상기 제1 픽셀에 대응하는 제1 모델 정보에 포함되는 파라미터로 업데이트하고,상기 제1 픽셀의 상기 제1 픽셀 값이 상기 제2 레이어로 입력되고, 상기 제2 픽셀의 제2 픽셀 값이 상기 제1 레이어로 입력되는 것에 기초하여, 상기 제1 레이어의 파라미터를 상기 제2 픽셀에 대응하는 제2 모델 정보에 포함되는 파라미터로 업데이트하고, 상기 제2 레이어의 파라미터를 상기 제1 모델 정보에 포함되는 파라미터로 업데이트하는, 디스플레이 장치.
- 제6항에 있어서,상기 복수의 레이어들은 상기 제2 레이어 다음의 제3 레이어를 더 포함하고,상기 제1 영상의 상기 픽셀들은 제3 픽셀을 더 포함하며,상기 프로세서는, 상기 하나 이상의 인스트럭션들을 실행함으로써,상기 제1 픽셀의 상기 제1 픽셀 값이 상기 제3 레이어로, 상기 제2 픽셀의 상기 제2 픽셀 값이 상기 제2 레이어로, 상기 제3 픽셀의 제3 픽셀 값이 상기 제1 레이어로 입력되는 것에 기초하여,상기 제1 레이어의 파라미터를 상기 제3 픽셀에 대응하는 제3 모델 정보에 포함되는 파라미터로 업데이트하고,상기 제2 레이어의 파라미터를 상기 제2 모델 정보에 포함되는 파라미터로 업데이트하고,상기 제3 레이어의 파라미터를 상기 제1 모델 정보에 포함되는 파라미터로 업데이트하는, 디스플레이 장치.
- 제1항에 있어서,상기 프로세서는, 상기 하나 이상의 인스트럭션들을 실행함으로써, 상기 제1 영상에 포함되는 객체 영역들을 검출하고,상기 객체 영역들에 대응하는 상기 복수의 모델 정보들에 기초하여, 상기 제1 영상에 포함되는 픽셀들에 각각 대응하는 상기 복수의 모델 정보들을 획득하는, 디스플레이 장치.
- 제8항에 있어서,상기 프로세서는, 상기 하나 이상의 인스트럭션들을 실행함으로써,상기 객체 영역들에 대응하는 모델 정보들의 가중 합에 기초하여, 상기 픽셀들에 각각 대응하는 상기 복수의 모델 정보들을 획득하는, 디스플레이 장치.
- 제9항에 있어서,상기 프로세서는, 상기 하나 이상의 인스트럭션들을 실행함으로써,상기 제1 영상에 포함되는 제1 픽셀과 상기 객체 영역들 각각의 중심 과의 거리에 기초하여, 상기 객체 영역들에 각각 대응하는 상기 복수의 모델 정보들의 가중치들을 결정하고,상기 객체 영역들에 각각 대응하는 상기 복수의 모델 정보들과 상기 결정된 가중치들에 기초하여, 상기 제1 픽셀에 대응하는 모델 정보를 획득하는, 디스플레이 장치.
- 복수의 레이어들을 포함하는 뉴럴 네트워크를 이용하여, 영상 처리를 수행하는 디스플레이 장치의 동작 방법에 있어서,제1 영상을 수신하는 단계;상기 제1 영상에 포함되는 픽셀들 각각에 대응하는 객체 특성에 기초하여, 상기 픽셀들에 각각 대응하는 복수의 모델 정보들을 획득하는 단계;상기 뉴럴 네트워크에 입력되는 상기 픽셀들에 각각 대응하는 상기 모델 정보들과 상기 픽셀들 각각이 상기 뉴럴 네트워크에서 처리되는 시점에 대한 정보에 기초하여, 상기 복수의 레이어들에 각각 대응하는 복수의 모델 정보들을 식별하는 단계;상기 복수의 모델 정보들에 기초하여, 상기 복수의 레이어들의 파라미터들을 업데이트하는 단계;상기 복수의 레이어들 각각에서, 상기 업데이트된 파라미터들이 적용된 상기 복수의 레이어들을 통해 상기 제1 영상을 처리함으로써, 제2 영상을 생성하는 단계; 및상기 제2 영상을 디스플레이하는 단계를 포함하는, 디스플레이 장치의 동작 방법.
- 제11항에 있어서,상기 시점에 대한 정보는,상기 픽셀들 각각이 상기 복수의 레이어들 각각에 입력되는 데이터 입력 시간에 대한 정보, 및 상기 픽셀들 각각이 상기 복수의 레이어들 각각에서 처리되는 데이터 처리 시간에 대한 정보 중 적어도 하나를 포함하는, 디스플레이 장치의 동작방법.
- 제12항에 있어서,상기 데이터 입력 시간 및 상기 데이터 처리 시간 중 적어도 하나는 일정한 주기를 가지는, 디스플레이 장치의 동작방법.
- 제11항에 있어서,상기 시점에 대한 정보는, 상기 제1 영상에서 상기 픽셀들 각각의 위치에 기초하여 결정되는, 디스플레이 장치의 동작방법.
- 제1 영상을 수신하는 단계;상기 제1 영상에 포함되는 픽셀들 각각에 대응하는 객체 특성에 기초하여, 상기 픽셀들에 각각 대응하는 복수의 모델 정보들을 획득하는 단계;상기 뉴럴 네트워크에 입력되는 상기 픽셀들에 각각 대응하는 상기 모델 정보들과 상기 픽셀들 각각이 상기 뉴럴 네트워크에서 처리되는 시점에 대한 정보에 기초하여, 상기 복수의 레이어들에 각각 대응하는 복수의 모델 정보들을 식별하는 단계;상기 복수의 모델 정보들에 기초하여, 상기 복수의 레이어들의 파라미터들을 업데이트하는 단계;상기 복수의 레이어들 각각에서, 상기 업데이트된 파라미터들이 적용된 상기 복수의 레이어들을 통해 상기 제1 영상을 처리함으로써, 제2 영상을 생성하는 단계; 및상기 제2 영상을 디스플레이하는 단계를 포함하는 복수의 레이어들을 포함하는 뉴럴 네트워크를 통해 이미지 처리를 수행하는 방법을 실행하도록 하는 인스트럭션들을 포함하는 프로그램이 저장된 비 일시적인 컴퓨터로 읽을 수 있는 기록매체.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP22878682.8A EP4318378A4 (en) | 2021-10-07 | 2022-07-22 | DISPLAY DEVICE AND OPERATING METHOD THEREOF |
CN202280044885.8A CN117561537A (zh) | 2021-10-07 | 2022-07-22 | 显示设备及其操作方法 |
US17/887,236 US20230114954A1 (en) | 2021-10-07 | 2022-08-12 | Display device and operating method of the same |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020210133450A KR20230050127A (ko) | 2021-10-07 | 2021-10-07 | 디스플레이 장치 및 그 동작방법 |
KR10-2021-0133450 | 2021-10-07 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/887,236 Continuation US20230114954A1 (en) | 2021-10-07 | 2022-08-12 | Display device and operating method of the same |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023058859A1 true WO2023058859A1 (ko) | 2023-04-13 |
Family
ID=85803521
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/KR2022/010800 WO2023058859A1 (ko) | 2021-10-07 | 2022-07-22 | 디스플레이 장치 및 그 동작방법 |
Country Status (2)
Country | Link |
---|---|
KR (1) | KR20230050127A (ko) |
WO (1) | WO2023058859A1 (ko) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20180071947A (ko) * | 2016-12-20 | 2018-06-28 | 서울대학교산학협력단 | 영상 처리 장치 및 방법 |
KR20180126220A (ko) * | 2017-05-17 | 2018-11-27 | 삼성전자주식회사 | 객체를 식별하는 방법 및 디바이스 |
KR20200047307A (ko) * | 2018-10-26 | 2020-05-07 | 주식회사 스트라드비젼 | 유용한 학습 데이터를 취사 선별하기 위한 cnn 기반 학습 방법 및 학습 장치 그리고 이를 이용한 테스트 방법 및 테스트 장치 |
US20200311491A1 (en) * | 2014-08-29 | 2020-10-01 | Google Llc | Processing images using deep neural networks |
-
2021
- 2021-10-07 KR KR1020210133450A patent/KR20230050127A/ko unknown
-
2022
- 2022-07-22 WO PCT/KR2022/010800 patent/WO2023058859A1/ko active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200311491A1 (en) * | 2014-08-29 | 2020-10-01 | Google Llc | Processing images using deep neural networks |
KR20180071947A (ko) * | 2016-12-20 | 2018-06-28 | 서울대학교산학협력단 | 영상 처리 장치 및 방법 |
KR20180126220A (ko) * | 2017-05-17 | 2018-11-27 | 삼성전자주식회사 | 객체를 식별하는 방법 및 디바이스 |
KR20200047307A (ko) * | 2018-10-26 | 2020-05-07 | 주식회사 스트라드비젼 | 유용한 학습 데이터를 취사 선별하기 위한 cnn 기반 학습 방법 및 학습 장치 그리고 이를 이용한 테스트 방법 및 테스트 장치 |
Non-Patent Citations (1)
Title |
---|
KIM SEON KUK, CHIL WOO LEE: "Semantic Image Segmentation Combining Image-level and Pixel-level Classification", JOURNAL OF KOREA MULTIMEDIA SOCIETY, vol. 21, no. 12, 1 December 2018 (2018-12-01), pages 1425 - 1430, XP093055582 * |
Also Published As
Publication number | Publication date |
---|---|
KR20230050127A (ko) | 2023-04-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020101143A1 (en) | Image processing apparatus and method of operating the same | |
WO2021251689A1 (en) | Electronic device and controlling method of electronic device | |
WO2019143227A1 (en) | Electronic device providing text-related image and method for operating the same | |
WO2020235860A1 (en) | Image processing apparatus and image processing method thereof | |
WO2020197018A1 (en) | Image processing apparatus and image processing method thereof | |
WO2021261836A1 (en) | Image detection apparatus and operation method thereof | |
WO2021029505A1 (en) | Electronic apparatus and control method thereof | |
WO2020027519A1 (ko) | 영상 처리 장치 및 그 동작방법 | |
WO2020105871A1 (ko) | 전자 장치 및 그 제어 방법 | |
WO2019182269A1 (ko) | 전자 장치, 전자 장치의 이미지 처리 방법 및 컴퓨터 판독 가능 매체 | |
WO2020180105A1 (en) | Electronic device and control method thereof | |
WO2020067759A1 (en) | Display apparatus control method and display apparatus using the same | |
WO2019231068A1 (en) | Electronic device and control method thereof | |
WO2020091519A1 (en) | Electronic apparatus and controlling method thereof | |
WO2020204277A1 (en) | Image processing apparatus and image processing method thereof | |
WO2021251614A1 (en) | Image processing apparatus and method of operating the same | |
WO2020050508A1 (en) | Image display apparatus and operation method of the same | |
EP3738305A1 (en) | Electronic device and control method thereof | |
WO2020231243A1 (en) | Electronic device and method of controlling thereof | |
WO2021075758A1 (en) | Electronic apparatus and controlling method thereof | |
WO2021251632A1 (en) | Display device for generating multimedia content, and operation method of the display device | |
WO2023085865A1 (ko) | 디스플레이 장치 및 그 동작방법 | |
WO2023058859A1 (ko) | 디스플레이 장치 및 그 동작방법 | |
WO2023090819A1 (ko) | 영상에서 인지적 노이즈를 제거하는 영상 처리 방법 및 장치 | |
WO2019088592A1 (ko) | 전자 장치 및 이의 제어방법 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22878682 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2022878682 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2022878682 Country of ref document: EP Effective date: 20231102 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202317078598 Country of ref document: IN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202280044885.8 Country of ref document: CN |
|
NENP | Non-entry into the national phase |
Ref country code: DE |