DE102021105217A1 - Image-guided adjustment for super-resolution operations - Google Patents

Abstract

An image processing circuit (200) performs super-resolution, hereinafter also referred to as SR, operations. The image processing circuit (200) has a memory (120) in order to store several parameter sets of several artificial intelligence, hereinafter also referred to as AI, models. The image processing circuit (200) further comprises an image guidance module (110), a parameter decision module (130) and an SR machine (140). The image guide module (110) operates to capture a representative feature in an image sequence comprising a current frame and previous frames within a time window. The parameter decision module (130) operates to adjust parameters of one or more AI models based on a measurement of the representative feature. The SR engine (140) operates to process the current frame using the one or more AI models with the adjusted parameters, thereby generating a high resolution image for display.

Description

Technical area

Embodiments of the invention relate to an edge device having image processing capabilities for performing super-resolution operations.

background

Modern image display devices typically add value to images before they are displayed. Some devices can perform super resolution with image enhancement. Super resolution refers to the function of upscaling a low resolution (LR) image to a higher resolution image referred to as a super resolution (SR) image; e.g. from an input image of (720 x 480) pixels to an output image of (3840 x 2160) pixels. Super-resolution operations can cause degraded image quality in terms of the degree of blurring, noise, distortion, color condition, sharpness, contrast, and so on.

A typical edge device, such as a television set or a smartphone, has limited computing power due to strict requirements for energy consumption and thermal performance. Image enhancement operations in an edge device are thus typically based on algorithms and parameters that are preconfigured by the device manufacturer. There is limited flexibility for adapting the pre-configuration once the device is in use by a consumer. The restricted flexibility can adversely affect the quality of the output image if input images have diverse contents and qualities. Thus, there is a need to improve image enhancement operations to minimize the impact on output image quality.

summary

In one embodiment, image processing circuitry for performing super-resolution (SR) operations is provided. The image processing circuit has a memory for storing a plurality of parameter sets of a plurality of artificial intelligence (AI) models; an image guide module for capturing a representative feature in an image sequence comprising a current frame and previous frames within a time window; a parameter decision module for adjusting parameters of one or more Al models based on a measurement of the representative feature; and an SR engine for processing the current frame using the one or more AI models with the adjusted parameters, thereby generating a high resolution image for display.

In another embodiment, a method for performing SR operations is provided. The method comprises: detecting a representative feature in an image sequence having a current frame and previous frames within a time window; Adjusting parameters of one or more AI models based on a measurement of the representative feature; and processing the current frame using the one or more AI models with the adjusted parameters to thereby generate a high resolution image for display.

Other aspects and features will become apparent to those of ordinary skill in the art after reviewing the following description of certain embodiments in conjunction with the accompanying figures.

Figure list

• 1 ist ein Blockdiagramm, das anpassbare Super-Auflösungs- (SR-) Operationen gemäß einer Ausführungsform darstellt.
• 2 ist ein Blockdiagramm, das eine Bildverarbeitungsschaltung darstellt, die SR-Operationen ausführt, gemäß einer Ausführungsform.
• 3A stellt ein Parameterverbindungsmodul gemäß einer Ausführungsform dar.
• 3B stellt ein Parameterverbindungsmodul gemäß einer anderen Ausführungsform dar.
• 3C stellt ein Parameterverbindungsmodul gemäß noch einer anderen Ausführungsform dar.
• 4 zeigt ein Beispiel von SR-Operationen, die durch eine SR-Maschine ausgeführt werden, gemäß einer Ausführungsform.
• 5 zeigt ein Beispiel von SR-Operationen, die durch eine SR-Maschine ausgeführt werden, gemäß einer anderen Ausführungsform.
• 6 ist ein Ablaufdiagramm, das ein Verfahren für SR-Operationen gemäß einer Ausführungsform darstellt.
• 7 stellt ein Beispiel einer Vorrichtung gemäß einer Ausführungsform dar.

The present invention is illustrated, by way of example and not by way of limitation, in the figures of the accompanying drawings, in which like reference characters indicate like elements. It should be noted that different references to “an” embodiment in this disclosure do not necessarily refer to the same embodiment, and such references mean at least one. Further, it is believed that when a particular feature, structure, or characteristic is described in connection with an embodiment, someone skilled in the art associates such feature, structure, or characteristic with other embodiments, whether or not expressly is described or not.

• 1 Figure 13 is a block diagram illustrating customizable super-resolution (SR) operations according to one embodiment.
• 2 Figure 13 is a block diagram illustrating image processing circuitry that performs SR operations, according to one embodiment.
• 3A illustrates a parameter link module according to an embodiment.
• 3B illustrates a parameter link module according to another embodiment.
• 3C Figure 3 illustrates a parameter link module according to yet another embodiment.
• 4th Figure 10 shows an example of SR operations performed by an SR machine, according to an embodiment.
• 5 FIG. 10 shows an example of SR operations performed by an SR machine in accordance with another embodiment.
• 6th Figure 13 is a flow diagram illustrating a method for SR operations according to one embodiment.
• 7th illustrates an example of a device according to an embodiment.

Detailed description

In the following description, numerous specific details are set forth. It is understood, however, that embodiments of the invention can be practiced without these specific details. In other instances, well known circuits, structures, and techniques have not been shown in detail in order not to obscure an understanding of this description. However, it will be recognized by those skilled in the art that the invention can be practiced without such specific details. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

Embodiments of the invention provide an image processing circuit which can automatically adapt its operations to input image content and quality. The image processing circuit comprises a super-resolution (SR) engine which operates on low-resolution (LR) input frames using AI models with dynamically adjustable parameters and outputs SR frames with a required high resolution. The SR machine can remove artifacts in the input images. The parameters of the AI models are trained in advance and stored in the memory or registers of the image processing circuit. The pre-training can be carried out with a training set comprising LR images and corresponding high resolution (HR) images. The parameters can be adjusted during a runtime based on a representative feature that is captured in the input images. The adaptability of these parameters enables the SR machine to record various input image contents and qualities and to generate output images with increased quality.

The terms “LR” and “HR” are relative to each other; that is, an LR image has fewer pixels than an HR image for the same display size (e.g., N square inches). For example, an LR image can have (720 x 480) pixels and an HR image can have (3840 x 2160) pixels for the same display size. It is understood that an LR image and an HR image can have any number of pixels as long as the LR image has fewer pixels for the same display size than the HR image. The resolution of an SR image is higher than the resolution of an LR image, and can be the same or lower than that of an HR image. In the following description, the terms “picture” and “frame” are used interchangeably.

In general, the pre-trained parameters are obtained from an offline training process that uses a limited number and types of training images. Thus, the previously trained parameters generally do not cover input images of diverse content and quality. The number of previously trained parameters can be limited due to the limited storage capacity in the image processing circuit. A conventional SR circuit, which relies only on the pre-trained parameters, may not be able to handle different types of input images with different levels of image quality. As a result, a conventional SR circuit may be unable to remove artifacts present in the input images.

In accordance with embodiments described herein, the SR engine uses one or more AI models in performing the SR operations. Each AI model is characterized or defined by a set of AI operations and pre-trained parameters associated with the AI operations. In one embodiment, two or more pre-trained parameter sets can be weighted and combined to form an adapted parameter set for use by the SR machine. The weighting of the parameters can be based on a measurement of the representative feature that is captured in an input image sequence. In one embodiment, the input image sequence can have a predetermined or configurable number of frames in a video. The input image sequence may include a current frame and previous frames in a time window, such as frame (t-n) to frame (t), where t is a unit of time and n is a positive integer. If the input image content and / or the input image quality changes, the parameters of the AI model can be adapted accordingly without the AI models having to be retrained. For an edge device with limited processing resources and storage capacities (e.g. a smart television set, a smartphone, a loT device, etc.), the methodology described here provides great flexibility during a runtime with minimal additional calculation effort.

1 FIG. 13 is a block diagram illustrating customizable super-resolution (SR) operations according to FIG represents one embodiment. An AI training module 180 trains AI models with a collection of training sets in a training database 181 . Each training set contains LR images and their corresponding HR images. The training database 181 has training sets of several (eg N) different training set types. A training set type can indicate a scene type, a deterioration type, a color state, a compression type, a resolution or some other indicator of an image content or a quality. In addition, a training set type can also indicate a level, an index, a rating or a measurement of a scene type, a deterioration type, a color state, a compression type, a resolution or another indicator of an image content or a quality. The output of the AI training module 180 is a collection of parameter sets; eg N parameter sets, where N can be any positive integer greater than one. Each parameter set contains parameters of an AI model for SR operations. Examples of the AI models include, but are not limited to, convolutional neural networks (CNNs), machine learning networks, deep learning networks, etc. Each parameter set can be labeled with the type of training set for which it is trained; For example, a parameter set P1 can be characterized as comprising a computer-generated (CG) scene if it is generated from a corresponding training set which has LR and HR images of a CG scene.

In one embodiment, the training is carried out in an offline training phase. In one embodiment, the offline training can be performed by a server, cloud computing system, computer, or other computing platform. The N parameter sets can be stored in a memory 120 of an edge device that executes SR operations, saved or downloaded into it. The memory 120 may include one or more of volatile and / or non-volatile random access memory (RAM) devices, read only memory (ROM) devices, or a combination of the foregoing.

Examples of an edge device, also referred to as a device, may include: a television, a smartphone, a computing device, a network-connected device, a gaming device, an entertainment device, an Internet-of-Things (loT) device, or any device, which is suitable for processing and displaying images and / or videos.

An on-line processing phase begins when the device receives a sequence of input frames through an input port. The input frames are LR images. The sequence of input frames includes a current frame (e.g. frame (t)) and past frames in a time slot; e.g. frame (t-n) to frame (t), where t is a unit of time and n is a configurable positive integer. The time window can be a continuously sliding time window so that the parameters can be adjusted for each input frame. Alternatively, the time window can be applied to an input stream every n frames, periodically with a configurable period, or based on a sensed event.

The on-line processing phase is carried out in the device. An image guidance module captures and / or measures during the online processing phase 110 a representative feature of the content and / or the quality in the input frame sequence within the time window. In one embodiment, the image guidance module 110 calculate a linear combination or a statistical combination of the input frames to obtain information on the frames including picture contents, picture quality, and so on. Changes in image content, an image quality and other image indices, types, ratings, and levels can thus be recorded and measured in the time window. The image guidance module 110 then performs a detection and / or measurement of a representative feature on the input frames or a combination of the input frames. In one embodiment, the image guidance module performs 110 statistical operations and / or AI operations to acquire and / or measure a representative characteristic. Examples of AI operations include, but are not limited to, convolutional neural network (CNN) operations, machine learning operations, deep learning operations, etc.

In one embodiment, the representative feature generated by the image guidance module 110 is detected and / or measured, comprise at least a classification of image content, such as a scene type, and a display of an image quality, such as an image quality index. Other representative features may include a predefined scene type (e.g., a CG image or a real image, a street scene or a sky scene, etc.), an image quality index, a deterioration type, a deterioration degree, a color condition, an image resolution, an interference signal level, an interference signal type, a Video compression parameters, amount of coding artifacts, color saturation, sharpness, contrast, etc. include, but are not limited to.

The captured representative feature is provided as an input by a parameter decision module 130 receive. In one embodiment, the parameter decision module identifies 130 one or more parameter sets in the memory 120 based on the representative characteristic. For example, the representative feature may be an image quality measurement which is an indication of an image quality such as an image quality index, a deterioration type, a deterioration degree, a color condition, an image resolution, an interference signal level, an interference signal type, a video compression parameter, an amount of coding artifacts, a color saturation, a Sharpness, a contrast, etc. is. When the representative feature is an image quality index q, the parameter decision module selects 130 from memory 120 a parameter set which has been trained in advance with images of an image quality index which is equal to or close to q (ie lies within a predefined threshold value of q). When the memory 120 has two sets of parameters which have been trained in advance with images of image quality indices q1 and q2, where q1 <q <q2, the parameter decision module 130 output a weighted combination of these two parameter sets as the adjusted parameters. If the representative feature comprises an image quality measurement and a classification of image content (e.g. an image quality index q for a CG scene), a pre-trained parameter set P1 can be identified as having a CG scene and another pre-trained parameter set P2 can be identified as an image quality index q having to be identified. Then the parameter decision module generates 130 adjusted parameters as output by adjusting the parameters from the sets P1 and P2 based on a measurement (eg an image quality index q in a CG scene) of the representative feature. The adjusted parameters can be used as the new parameters of one or more AI models for SR operations. In some embodiments, a weighted combination of more than two sets of parameters can be calculated and used as the new parameters for SR operations.

During the on-line processing phase, an SR machine receives 140 in the device the adjusted parameters and a current frame (eg frame (t)) of a low resolution. The SR machine 140 performs SR operations on the current frames using one or more AI models and the adjusted parameters determined by the parameter decision module 130 to be generated. In some embodiments, the SR machine can 140 Eliminate artifacts in a current frame using one or more AI models and the parameters adjusted based on a scene type and an image quality index measured from the input image sequence. In some embodiments, the SR machine can 140 Eliminate artifacts in a current frame using one or more AI models and the parameters adjusted or selected based on the captured or measured representative feature in the input image sequence.

An example of the SR operations performed by the SR machine 140 are performed are CNN operations. Other machine learning operations and deep learning operations can also be performed by the SR machine 140 are executed. The SR machine 140 generates a high resolution output frame. The output frame can be sent to a display through an output port.

2 Fig. 3 is a block diagram showing an image processing circuit 200 that performs SR operations, according to one embodiment. The image processing circuit 200 instructs the image guidance module 110 , the parameter decision module 130 who have favourited SR machine 140 and the memory 120 which saves the parameter sets, as in the example of 1 described. In some embodiments, the image processing circuit 200 an AI processor, graphics processing unit (GPU), application specific integrated circuit (ASIC), or other general purpose or special purpose processing circuit. In one embodiment, the image guidance modules are 110 and the SR module 140 Part of an AI processor operable to perform CNN calculations to capture the representative feature and process a current frame. In one embodiment, the image processing circuit 200 be implemented as a system-on-a-chip (SoC). In some embodiments, the image processing circuit 200 implemented in more than one chip in the same electronic device.

In the embodiment of 2 instructs the parameter decision module 130 further a parameter weight generator 210 and a parameter link module 220 on. The parameter weight generator 210 receives a representative feature identified by the image guidance module 110 is detected and / or measured. In one example, the image guidance module 110 output a representative feature with a measurement; eg an interference signal level σ, a degree of blurring y and a JPEG compression level α, where σ, γ and α are numerical values. In another example, the image guidance module 110 output a representative feature with a scene classification evaluation of a first scene type and a second scene type. In yet another example, the image guidance module 110 output a representative feature with an image state measurement of a brightness, a saturation and a contrast. The parameter weight generator 210 generates weights for the representative characteristic based on the measurement.

The parameter decision module 130 further comprises a parameter connection module 220 which applies the weights to the parameter sets associated with the representative characteristic. In one embodiment, the parameter link module calculates 220 a weighted sum using the weights. For example, the parameter weight generator 210 generate weightings w ₁ , w ₂ , w ₃ , ... w _N for N different parameter sets, and the parameter connection module 220 can calculate a weighted sum of the N parameter sets in order to generate a new parameter set Pnew. For example, the parameter weight generator 210 generate a weighting w ₁ for an interference signal level σ and a weighting w ₂ for a degree of blurring y. The parameter connection module 220 can calculate a weighted sum Pnew = (w ₁ × Pi + w ₂ × Pj), where Pi is the parameter set for the interference signal level and Pj is the parameter set for the blurriness. The new parameter set Pnew is used by the SR machine for one or more AI models for SR operations on a current frame.

3A represents a parameter connection module 310 according to one embodiment. The parameter link module 310 can be an example of the parameter link module 220 in 2 be. In this example, the parameter connection module 310 has N multipliers in order to multiply the weightings w ₁ , w ₂ , w ₃ , ... w _N by their corresponding parameter sets. Some of the weights can be zero and the total sum of the weights is equal to one. The parameter connection module 310 further has an adder 315 to calculate a weighted sum of the parameter sets as an output.

3B represents a parameter connection module 320 according to another embodiment. The parameter connection module 320 can be an example of the parameter link module 220 in 2 be. In this example, the parameter connection module 320 has N multipliers in order to multiply the weightings w ₁ , w ₂ , w ₃ , ... w _N by their corresponding parameter sets. Some of the weights can be zero and the total sum of the weights is equal to one. The parameter connection module 320 further has a multiplexer 325 to select a weighted parameter set as output. In one embodiment, only one of the weights (e.g., w _i ) is equal to one and all others are zero. Thus the multiplexer chooses 325 the parameter set that _{corresponds to the weighting w i} as output. The selected parameter set is used by the SR machine 140 used to perform Al-based SR operations.

3C represents a parameter connection module 330 according to yet another embodiment. The parameter link module 330 can be an example of the parameter link module 220 in 2 be. In this example the parameter connection module connects 330 the weightings w ₁ , w ₂ , w ₃ , ... w _{N in} pairs with their corresponding parameter sets. Some of the weights can be zero and the total sum of the weights is equal to one. The parameter connection module 330 can send the (non-zero weighted parameter set) pairs to the SR machine for further processing 140 output. This embodiment enables the SR machine 140 to perform weighted AI operations associated with 5 are to be described.

Depending on the content and the quality of the input image sequence, the parameter connection module 220 in 2 one of the in the 3A , 3B and 3C Implement the connection operations shown. Thus, in one embodiment, the parameter connection module 220 a combination of those in the 3A , 3B and 3C circuits shown to implement various selection and connection operations.

4th shows an example of SR operations performed by the SR machine 140 are performed, according to one embodiment. In this example, the SR machine has 140 a CNN accelerator 410 to perform CNN operations on input frames.

In one embodiment, the CNN accelerator 410 Hardware components specialized in accelerating neural network operations through convolution operations, fully connected operations, activation, pooling, normalization, element-wise mathematical calculations, etc. In some embodiments, the CNN accelerator 410 multiple computation units and memories (e.g. static random access memory (SRAM)), each computation unit further including multiplier and adder circuits for performing mathematical operations, such as multiply-and-accumulate (MAC) operations, to speed up the convolution, the Activation, pooling, normalization, and other neural network operations. The CNN accelerator 410 can perform fixed and floating point neural network operations. The CNN accelerator 410 can perform the SR operations to generate high definition output frames with improved image quality.

The CNN accelerator 410 performs convolution operations by computing a convolution an input feature assignment with a core. For example, an input characteristic mapping 430 from a previous layer of CNN operations with a core 420 can be folded to an output characteristic map 440 to be spent on the next shift. The core 420 has a set of parameters; e.g. a set of 3x3 parameters. In one embodiment, the parameters are in the core 420 the new parameters (also referred to as the adjusted parameters) generated by the parameter decision module 130 in 1 , or more precisely from the parameter link module 220 in 2 to be generated. In the example of 4th calculates the parameter connection module 220 a weighted sum of several parameter sets, as in the embodiment of FIG 3A each of these parameter sets defining a different core. In this example is the core 420 (θ _f ) = w ₁ θ ₁ + w ₂ θ ₂ + ... + w _N θ _N. The core 420 is from the parameter link module 220 to the SR machine 140 passed to perform CNN operations.

5 shows an example of SR operations performed by the SR machine 140 are performed, according to another embodiment. In this example, the SR machine is calculating 140 a weighted sum of AI operations (AI_OP1, AI_OP2, ...), such as CNN operations or other deep learning or machine learning operations. In one embodiment, the SR machine receives 140 the (non-zero weighted parameter set) pairs from the parameter link module 330 in 3C . Each AI-OP is executed using one of the parameter sets. The output of each AI_OP is multiplied by a corresponding weighting. The weighted sum can be sent to the next layer of AI processing.

6th Figure 6 is a flow diagram illustrating a method 600 for performing SR operations, according to an embodiment. For example, the method 600 can be performed by the image processing circuitry 200 in 2 , one in 7th Device to be described and / or one of the in connection with the 1 , 3A , 3B and 3C described embodiments are carried out. It is understood that the embodiments are for illustrative purposes only; other devices with image processing capabilities may perform method 600.

The method 600 begins at step 610 when a device detects a representative feature in an image sequence comprising a current frame and previous frames within a time window. The apparatus adjusts parameters of one or more AI models based on a measurement of the representative feature in step 620. The apparatus, at step 630, processes the current frame using the one or more AI models with the adjusted parameters to thereby generate a high resolution image for display.

To adjust the parameters, the device can generate weights for two or more pre-trained parameter sets of the AI models based on the measurement of the representative feature, and the device applies the weights to the two or more pre-trained parameter sets to thereby the adjusted parameters to generate. In some embodiments, the apparatus can calculate a weighted sum of the weights and the two or more pre-trained parameter sets to generate the adjusted parameters. In some embodiments, the device may compute a weighted sum of cores of a CNN to obtain a new core to process the current frame.

7th provides an example of a device 700 according to one embodiment. The device 700 can the image processing circuit 200 from 2 which performs the above image enhancement operations including the SR operations. An example of the device 700 is a television that receives low resolution video (e.g. 720 x 480 pixels) and performs SR operations to upgrade the video to high definition (e.g. 3840 × 2160 pixels) for display on the television screen. Alternatively, the device 700 a smartphone, computing device, network connected device, gaming device, entertainment device, Internet of Things (loT) device, or device capable of sharing images and / or videos process and display.

The device 700 has processing hardware 710 on. In one embodiment, the processing hardware 710 comprise one or more processors, such as one or more of: a central processing unit (CPU), a GPU, a digital processing unit (DSP), an AI processor, a multimedia processor, other general purpose and / or special purpose processing circuitry. In one embodiment, the processing hardware 710 a hardware accelerator such as a CNN accelerator. In one embodiment, the processing hardware 710 the image guidance module 110 , the parameter decision module 130 and the SR machine 140 in the aforementioned embodiments.

The device 700 further comprises storage and retention hardware 720 on that with the processing hardware 710 connected is. The storage and retention hardware 720 may include storage devices such as dynamic random access memory (DRAM), SRAM, flash memory, and other non-perishable machine-readable storage media; e.g. volatile or non-volatile storage devices. The storage and retention hardware 720 may further include storage devices, for example any type of fixed or magnetic storage device. In one embodiment, the storage and retention hardware 720 previously trained parameter sets 725 save as in connection with 1 and 2 described. In some embodiments, the storage and retention hardware 720 Instructions store which, when passed through the processing hardware 710 run, cause the processing hardware 710 the aforementioned operations for generating high resolution images, such as method 600 of FIG 6th , executes.

The device 700 can also be a display panel 730 to display information such as images, videos, messages, web pages, games, texts, and other types of text, image, and video data. The device 700 can also be audio hardware 740 , such as a microphone and a loudspeaker, for receiving and generating sounds.

In some embodiments, the device can 700 also a network interface 750 have to connect to a wired and / or wireless network for sending and / or receiving voice, digital data and / or media signals. It is understood that the embodiment of FIG 7th is simplified for illustration purposes. Additional hardware components can be included.

The operations of the flowchart of 6th are with reference to the exemplary embodiments of FIG 2 and 7th has been described. However, it should be understood that the operations of the flowchart of FIG 6th by embodiments of the invention other than the embodiments of FIG 2 and 7th can be carried out, and the embodiments of 2 and 7th Perform operations different from those discussed with respect to the flowchart. Although the 6th shows a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlay certain operations, etc.) .

Various functional components or blocks have been described here. As will be appreciated by those skilled in the art, the functional blocks are preferably implemented by circuits (either dedicated circuits or general purpose circuits that operate under the control of one or more processors and encoded instructions) that typically include transistors that operate in such a manner are designed to control the operation of the circuit in accordance with the functions and operations described herein. For example, the image guidance module 110 , the parameter decision module 130 and the SR machine 140 from 1 be functional blocks implemented by either dedicated circuits or general-purpose circuits.

Although the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the described embodiments and can be practiced with modifications and changes within the spirit and scope of the appended claims. The description is therefore to be regarded as illustrative rather than restrictive.

In summary, an image processing circuit performs 200 Super-resolution (SR) operations off. The image processing circuit 200 assigns a memory 120 to save multiple parameter sets of multiple artificial intelligence (AI) models. The image processing circuit 200 further has an image guidance module 110 , a parameter decision module 130 and an SR machine 140 on. The image guidance module 110 operates to capture a representative feature in an image sequence comprising a current frame and previous frames within a time window. The parameter decision module 130 operates to adjust parameters of one or more AI models based on a measurement of the representative feature. The SR machine 140 operates to process the current frame using the one or more AI models with the adjusted parameters to thereby generate a high resolution image for display.

List of reference symbols

110

Image guidance module

120

Storage

130

Parameter decision module

140

AI-SR module, SR machine

180

AI training module

181

Training database

200

Image processing circuit

210

Parameter weight generator

220

Parameter connection module

310

Parameter connection module

315

Adder

320

Parameter connection module

325

multiplexer

330

Parameter connection module

410

CNN accelerator

420

core

430

Input characteristic assignment

440

Output characteristic assignment

700

contraption

710

Processing hardware

720

Storage and retention hardware

725

Parameter sets

730

Display panel

740

Audio hardware

750

Network interface

Claims (20)

Image processing circuit (200), which is ready to perform super-resolution, hereinafter also referred to as SR, operations, comprising: a memory (120) in order to store a plurality of parameter sets of a plurality of artificial intelligence, hereinafter also referred to as AI, models; an image guide module (110) to capture a representative feature in an image sequence comprising a current frame and past frames within a time window; a parameter decision module (130) to adjust parameters of one or more Al models based on a measurement of the representative feature; and an SR engine (140) to process the current frame using the one or more AI models with the adjusted parameters, thereby generating a high resolution image for display. Image processing circuit (200) according to Claim 1 wherein the parameter decision module (130) is further operable to: generate weights for two or more pre-trained parameter sets of the AI models based on the measurement of the representative feature; and applying the weights to the two or more pre-trained parameter sets to thereby generate the adjusted parameters. Image processing circuit (200) according to Claim 2 wherein the parameter decision module (130) further comprises an adder (315) operable to calculate a weighted sum from the weights and the two or more pre-trained parameter sets to generate the adjusted parameters. Image processing circuit (200) according to one of the Claims 1 until 3 , wherein the parameter decision module (130) further comprises an adder (315) which is operational for calculating a weighted sum of cores of a convolutional neural network, hereinafter also referred to as CNN, in order to obtain a new core for the SR machine, to process the current frame. Image processing circuit (200) according to one of the Claims 1 until 4th , wherein the representative feature has an image quality measurement value q, and the two or more pre-trained parameter sets are pre-trained with images of image quality measurement values q1 and q2, where q1 <q <q2. Image processing circuit (200) according to one of the Claims 1 until 5 wherein the representative feature comprises at least one image quality measurement value and a classification of image content. Image processing circuit (200) according to one of the Claims 1 until 6th wherein the parameter decision module (130) further comprises a multiplexer (325) operable to: select one of a plurality of pre-trained parameter sets as a new parameter set based on the measurement of the representative feature; and outputting the selected set of parameters to the SR machine (140) for processing the current frame. Image processing circuit (200) according to one of the Claims 1 until 7th , further comprising a CNN accelerator (410) operable to perform CNN calculations in processing the current frame. Image processing circuit (200) according to one of the Claims 1 until 8th wherein the representative feature comprises one or more of: an image quality index, a scene type, one Type of deterioration, a degree of deterioration, a color condition, an image resolution, a noise level, a noise type, a video compression parameter, an amount of coding artifacts, a color saturation, a sharpness and a contrast. Image processing circuit (200) according to one of the Claims 1 until 9 wherein the SR engine (140) is further operable to: remove artifacts in the current frame using the one or more AI models and the parameters adjusted based on a scene type and an image quality index derived from the image sequence be measured. A method for performing super-resolution, hereinafter also referred to as SR, operations, comprising: Detecting a representative feature in an image sequence comprising a current frame and previous frames within a time window; Adaptation of parameters of one or more artificial intelligence, hereinafter also referred to as AI, models based on a measurement of the representative feature; and Processing the current frame using the one or more AI models with the adjusted parameters, thereby generating a high resolution image for display. Procedure according to Claim 11 wherein adjusting the parameters further comprises: generating weights for two or more pre-trained parameter sets of the AI models based on the measurement of the representative feature; and applying the weights to the two or more pre-trained parameter sets to thereby generate the adjusted parameters. Procedure according to Claim 12 wherein applying the weights further comprises: computing a weighted sum of the weights and the two or more pre-trained parameter sets to generate the adjusted parameters. Procedure according to Claim 12 or 13th wherein applying the weights further comprises: calculating a weighted sum of kernels of a convolutional neural network, hereinafter also referred to as CNN, in order to obtain a new kernel for processing the current frame. Method according to one of the Claims 11 until 14th , wherein the representative feature has an image quality measurement value q, and the two or more pre-trained parameter sets are pre-trained with images of image quality measurement values q1 and q2, where q1 <q <q2. Method according to one of the Claims 11 until 15th wherein the representative feature comprises at least one image quality measurement value and a classification of image content. Method according to one of the Claim 11 until 16 wherein adjusting the parameters further comprises: selecting a parameter set from a plurality of pre-trained parameter sets based on the measurement of the representative feature; and processing the current frame using the one or more AI models with the selected parameter set. Method according to one of the Claims 11 until 17th wherein processing the current frame further comprises: performing CNN calculations with the adjusted parameters to process the current frame. Method according to one of the Claims 11 until 18th , wherein the representative feature comprises one or more of: an image quality index, a scene type, a type of deterioration, a degree of deterioration, a color state, an image resolution, a noise level, a noise type, a video compression parameter, an amount of coding artifacts, a color saturation, a sharpness and a Contrast. Method according to one of the Claims 11 until 19th wherein processing the current frame further comprises: eliminating artifacts in the current frame using the one or more AI models and the parameters adjusted based on a scene type and an image quality index measured by the image sequence.

Images