WO2020220346A1

WO2020220346A1 - Image processing method and apparatus

Info

Publication number: WO2020220346A1
Application number: PCT/CN2019/085370
Authority: WO
Inventors: 徐晨剑
Original assignee: 华为技术有限公司
Priority date: 2019-04-30
Filing date: 2019-04-30
Publication date: 2020-11-05
Also published as: CN113728354A

Abstract

An image processing method and apparatus. For a received image, the resolution of the received image is adjusted according to state control information of an image processing system; under different application scenarios, the state control information of the image processing system is different. Therefore, the mode of adjusting the resolution according to state control information provided by the embodiments of the present application can satisfy requirements of resolution under service scenarios.

Description

Image processing method and device

Technical field

This application relates to the field of video technology, and in particular to an image processing method and device.

Background technique

As people's demand for high-quality, high-definition images and videos on terminal devices gradually increases, 2K to 8K high-resolution images, videos and display screens have emerged. Limited by online transmission bandwidth, power consumption, performance, device storage space, etc., the resolution of the content played by users is lower than the device screen resolution. Using super-resolution technology, images and videos can be generated from low-resolution images or videos that are transmitted or stored on the network to high-resolution images or videos.

At present, terminal devices generally adapt to specific super-resolution algorithms to output specific resolutions, and terminal devices may be applied to different business scenarios. Therefore, terminal devices that only output specific resolutions may not meet the requirements of multiple business scenarios.

Summary of the invention

The present application provides an image processing method and device to solve the problem of not being able to meet the requirements of multiple business scenarios.

In a first aspect, the present application provides an image processing method including the following steps: acquiring display state information; wherein the display state information is used to characterize the display state of the display device; and the second resolution is determined according to the display state information. To meet the requirements of the display state of the display device, the second resolution is the target resolution determined for the image displayed by the display device; the received source image with the first resolution is converted into the target image with the second resolution.

Exemplarily, the method may be executed by an image processing device. The display device may be a part of the image processing device, or two devices that are separately deployed, and the two devices are coupled.

Using the above method, the target resolution is determined according to the requirements of the display state of the display device, so that the resolution of the converted image can be adapted to different application scenarios. The solutions provided in the embodiments of this application can be applied to different application scenarios, for example, the resolutions of display devices in different application scenarios are different, and the method provided in this application is adaptive based on the display state of the display device in the current application scenario. Adjust the resolution of the input image accordingly.

In a possible design, converting the received source image with the first resolution into the target image with the second resolution includes: determining according to the first resolution and the second resolution of the received source image Target magnification; performing a resolution conversion operation on the source image with the first resolution according to the target magnification to obtain a target image with a second resolution, and the second resolution is obtained by enlarging the target magnification of the first resolution. The above design provides a simple and easy way to convert image resolution.

In a possible design, performing a resolution conversion operation on the source image with the first resolution according to the target magnification includes: selecting the first super-resolution algorithm in the super-resolution algorithm set according to the target magnification, and adopting the first super-resolution algorithm. The super-resolution algorithm converts the source image with the first resolution into the target image with the second resolution. The first super-resolution algorithm is the magnification in the super-resolution algorithm whose concentration is less than or equal to the target magnification. The largest super-resolution algorithm; among them, the super-resolution algorithm sets a variety of super-resolution algorithms including the first super-resolution algorithm, one super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to magnification different.

In the above design, a variety of super-resolution algorithms are integrated inside the image processing device, and the super-resolution algorithm can be adaptively selected according to the actual needs in the application scenario, so that the image processing device can be applied to different scenarios. Since the existing terminal adapts to a specific super-resolution algorithm to output a specific resolution, if it is changed to another hardware platform, the resolution algorithm needs to be reconfigured to meet the requirements of the hardware platform. The solution provided by the embodiment of this application can reduce Integration workload of different hardware platforms.

In a possible design, converting the received source image with the first resolution into the target image with the second resolution includes: performing M resolutions on the source image with the first resolution according to the target magnification The conversion operation obtains a target image with a second resolution, where M is an integer greater than 0; among them, in the M resolution conversion operations, the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the target magnification; The image obtained by the i resolution conversion operation is the input of the i+1 resolution conversion operation, and i+1 is an integer less than or M and greater than 1.

In the above design, one or more super-resolution algorithms are integrated in the image processing device, and the super-resolution algorithm and the number of executions can be adaptively selected according to the actual needs in the application scenario, so that the image processing device can be applied to different Scene.

In a possible design, the display status information includes at least one of the resolution of the display device, the interface type of the display device, or the size of the display device.

In a possible design, determining the second resolution according to the display status information includes: determining the resolution of the display device as the second resolution; or, combining the highest resolution supported by the interface type of the display device with the display device The minimum of the resolutions of the display device is determined to be the second resolution; or, the minimum of the highest resolution supported by the size of the display device and the resolution of the display device is determined to be the second resolution; or The minimum value among the highest resolution supported by the size, the highest resolution supported by the interface type of the display device, and the resolution of the display device is determined as the second resolution. The above design exemplarily provides several simple and easy ways to determine the target resolution.

In a possible design, it further includes: acquiring stream status information, which is used to characterize the stream status of the video stream to which the source image belongs; and determining the second resolution according to the display status information, and the second resolution meets the requirements of the display device The requirements for the display status include: determining the second resolution according to the stream status information and the display status information, and the second resolution meets the requirements of the stream status of the image stream and the requirements of the display status of the display device.

In the above design, on the basis of using the display status information as the target resolution to be determined, the determination conditions for determining the target resolution are increased or decreased, which not only improves the accuracy of the determined target resolution, but also further increases the applicable scenarios.

In a possible design, the stream state information includes the frame rate of the video stream and/or parameters used to represent the format of the image stream.

Exemplarily, when the second resolution is determined according to the stream status information and the display status information, the maximum resolution supported by each parameter included in the display status information and the maximum resolution supported by each parameter included in the stream status information may be combined The minimum value in is used as the second resolution.

In a possible design, the method further includes: in the process of performing a resolution conversion operation on the source image with the first resolution according to the target magnification, acquiring resource operating status information; and determining that the resource operating status information meets a preset condition , Continue to perform the resolution conversion operation on the source image with the first resolution according to the target magnification. The resource running status information indicates the running status of the resource on the image processing device for performing the resolution conversion operation.

In a possible design, it further includes: when it is determined that the resource operating state information does not meet the preset condition, the target magnification is reduced to obtain the second target magnification, and the source image with the first resolution is determined according to the reduced second target magnification. The resolution conversion operation is performed to obtain a second target image with a third resolution, and the third resolution is obtained by enlarging the second target magnification of the first resolution.

In the above design, the resolution is further adjusted based on the resource operation of the image processing device in the process of performing the resolution conversion operation, so as to improve the accuracy of the determined target resolution and improve the adaptability to the current application environment.

In a possible design, the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.

Based on the same inventive concept as the method of the first aspect, in the second aspect, an embodiment of the present application provides an image processing device that can implement the method provided in the first aspect and any one of the possible designs of the first aspect. Features. The function can be realized by hardware, or by hardware executing corresponding software, or by a combination of software and hardware. The hardware or software includes one or more modules corresponding to the above-mentioned functions.

An image processing device provided by an embodiment of the present application may include a state monitoring module and a super-resolution conversion module; wherein the state monitoring module is used to obtain display state information; wherein the display state information is used to characterize the display state of the display device The super-resolution conversion module is used to determine the second resolution according to the display status information, the second resolution meets the requirements of the display status of the display device, and the second resolution is the target resolution determined for the image displayed by the display device; The received source image with the first resolution is converted into the target image with the second resolution.

In a possible design, the super-resolution conversion module is specifically used to: according to the received source image of the received source image when converting the received source image with the first resolution into the target image with the second resolution The first resolution and the second resolution determine the target magnification; according to the target magnification, the resolution conversion operation is performed on the source image with the first resolution to obtain the target image with the second resolution, and the second resolution is the first resolution Magnified target magnification is obtained.

In a possible design, the super-resolution conversion module, when performing a resolution conversion operation on the source image with the first resolution according to the target magnification, is specifically used to: select the second in the super-resolution algorithm set according to the target magnification. A super-resolution algorithm, and the first super-resolution algorithm is used to convert the source image with the first resolution into the target image with the second resolution. The first super-resolution algorithm is the super-resolution algorithm. The concentration ratio is less than or equal to The super-resolution algorithm with the largest magnification among the super-resolution algorithms of the target magnification; among them, the super-resolution algorithm concentrates on multiple super-resolution algorithms including the first super-resolution algorithm, and one super-resolution algorithm corresponds to one magnification , Different super-resolution algorithms correspond to different magnifications.

In a possible design, the super-resolution conversion module, when converting the received source image with the first resolution into the target image with the second resolution, is specifically used to: The source image of the resolution performs M resolution conversion operations to obtain the target image with the second resolution. N is an integer greater than 0, and M is an integer greater than 0. Among them, in the M resolution conversion operations, M resolutions The magnification product of the super-resolution algorithm used in the rate conversion is equal to the target magnification; the image obtained by the i-th resolution conversion operation is the input of the i+1-th resolution conversion operation, i+1 is less than or M and greater than 1 Integer.

In a possible design, when determining the second resolution according to the display status information, the super-resolution conversion module is specifically used to: determine the resolution of the display device as the second resolution; or, set the interface of the display device The minimum of the highest resolution supported by the type and the resolution of the display device is determined as the second resolution; alternatively, the minimum of the highest resolution supported by the size of the display device and the resolution of the display device is determined as the second resolution Two resolutions; or, the minimum of the highest resolution supported by the size of the display device, the highest resolution supported by the interface type of the display device, and the resolution of the display device is determined as the second resolution.

In a possible design, the status monitoring module is also used to obtain stream status information, which is used to characterize the stream status of the video stream to which the source image belongs; the super-resolution conversion module determines the second Resolution, when the second resolution meets the requirements of the display status of the display device, it is specifically used to: determine the second resolution according to the stream status information and the display status information, and the second resolution meets the requirements of the stream status of the image stream and the display Requirements for the display status of the equipment.

In a possible design, the status monitoring module is also used to obtain resource operating status information during the process of the super-resolution conversion module performing resolution conversion operations on the source image with the first resolution according to the target magnification. The status information indicates the operating status of the resource on the image processing device used to perform the resolution conversion operation; when it is determined that the resource operating status information meets the preset condition, continue to perform the resolution conversion operation on the source image with the first resolution according to the target magnification .

In a possible design, the super-resolution conversion module is also used to reduce the target magnification to obtain the second target magnification when it is determined that the resource operating state information does not meet the preset condition, and the second target magnification is obtained according to the reduced second target magnification. A source image of one resolution performs a resolution conversion operation to obtain a second target image with a third resolution, and the third resolution is obtained by enlarging the first resolution by the second target magnification.

In a third aspect, an embodiment of the present application provides an image processing method, including: a set-top box receives a source image, the source image has a first resolution; the set-top box acquires status information; wherein, the status information includes The display status information of the display performance of the device, the set-top box and the display device transmit image data through a transmission interface; the set-top box determines the first target of magnification for the source image according to the status information and the first resolution Magnification; the set-top box selects at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution, wherein the second The resolution of the target image is adapted to the display performance of the display device.

In the embodiment of this application, the set-top box determines the target magnification corresponding to the super-resolution algorithm according to the acquired state information. The state information includes the display state information of the display device. For displays of various display specifications, the set-top box can determine the matching magnification. The magnification is adapted to the display performance of the display after the magnification. Therefore, the set-top box can independently adapt to display devices of various display specifications. The display specifications of the display device are not limited, and manual settings are not required, which greatly improves the user experience .

In a possible design, the state information further includes: stream state information used to characterize the video stream to which the source image belongs.

In a possible design, the status information further includes: resource running status information, and the resource running status information indicates the running status of the resource on the set-top box.

In a possible design, the set-top box determines the first target magnification according to the state information and the first resolution, including: the set-top box constructs the first state according to the state information and the first resolution Vector, the first state vector includes a parameter value for indicating the state information and a state quantity of the first resolution; the set-top box determines, according to the first state mapping rule, what is mapped by the first state vector The first target magnification.

In a possible design, the set-top box selects at least one super-resolution algorithm according to a first target magnification, and amplifies the source image at the first target magnification to obtain a target image with a second resolution, including: The set-top box selects the first super-resolution algorithm in the super-resolution algorithm set according to the first target magnification, and uses the first super-resolution algorithm to convert the source image into the target image; wherein, the first super-resolution algorithm The resolution algorithm is the super-resolution algorithm with the largest magnification among the super-resolution algorithms whose magnification is less than or equal to the first target magnification; the super-resolution algorithm set includes the first super-resolution algorithm There are a variety of super-resolution algorithms, one super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to different magnifications.

In a possible design, the set-top box selects at least one super-resolution algorithm according to a first target magnification, and amplifies the source image at the first target magnification to obtain a target image with a second resolution, including: The set-top box performs M resolution conversion operations on the source image with the first resolution according to the first target magnification to obtain the target image with the second resolution, where M is an integer greater than 0; where In the M resolution conversion operations, the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the first target magnification; the image obtained by the i-th resolution conversion operation is the i+1th The input of the resolution conversion operation, i+1 is an integer less than or M and greater than 1.

In a possible design, the display state information includes at least one of the resolution of the display device, the transmission interface type of the display device, or the size of the display device.

In a possible design, determining the first target magnification for the source image according to the status information and the first resolution includes: determining the second resolution according to the status information, and according to The first resolution and the second resolution determine the first target magnification.

When determining the second resolution according to the status information, the minimum value among the maximum resolutions supported under each parameter included in the status information may be used as the second resolution.

Illustratively, several examples are as follows:

The second resolution is the resolution of the display device; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device; or,

The second resolution is the minimum of the highest resolution supported by the size of the display device and the resolution of the display device; or,

The second resolution is the smallest value among the highest resolution supported by the size of the display device, the highest resolution supported by the interface type of the display device, and the resolution of the display device; or,

The second resolution is the highest resolution supported by the interface type of the display device under the frame rate of the video stream; or,

The second resolution is the minimum of the highest resolution supported by the interface type of the display device and the resolution of the display device at the frame rate of the video stream; or,

The second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the maximum resolution of the display device when the frame rate of the video stream is used. The minimum value in resolution; or,

The second resolution is the highest resolution supported by the interface type of the display device when the format of the image stream is adopted; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device when the format of the image stream is adopted; or,

The second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the resolution of the display device when the format of the image stream is adopted. The smallest value in; or,

The second resolution is the highest resolution supported by the interface type of the display device when the frame rate of the video stream and the format of the image stream are adopted.

In a possible design, the method further includes: the set-top box uses the at least one resolution conversion algorithm to perform an operation of enlarging the first target magnification on the source image, acquiring the resource running status Information; when the set-top box determines that the resource operating state information meets a preset condition, it continues to perform a magnification operation of the first target magnification on the source image according to the first target magnification.

In one possible design, it also includes:

When it is determined that the resource operating state information does not meet the preset condition, the first target magnification is reduced to obtain a second target magnification, and the source image is enlarged according to the reduced second target magnification to obtain a A three-resolution second target image, where the third resolution is obtained by enlarging the second target magnification of the first resolution.

In a fourth aspect, an embodiment of the present application provides an image processing device, which is applied to a set-top box, and includes:

A transmission interface for receiving a source image, the source image having a first resolution;

A status monitoring module, configured to obtain status information, the status information including display status information used to characterize the display performance of the display device, and the set-top box and the display device transmit image data through the transmission interface;

Wherein, the display state information is used to characterize the display state of the display device;

The super-resolution conversion module is configured to receive a source image and determine a first target magnification for the source image according to the state information and the first resolution; the first target magnification selects at least one super-resolution The rate conversion algorithm enlarges the source image at the first target magnification to obtain a target image with a second resolution, wherein the target image with the second resolution is compatible with the display performance of the display device.

Exemplarily, the transmission interface may include a transmission interface and a reception interface. The transmission interface is used for transmitting image data to the display device, and the receiving interface is used for receiving the source image.

Exemplarily, the function of the transmission interface may be implemented by the sending unit and the receiving unit. The sending unit is used for sending image data to the display device, and the receiving unit is used for receiving the source image.

In a possible design, the status information further includes: resource running status information, and the resource running status information indicates the running status of the resource on the set-top box. The resources can be hardware resources or software resources.

In a possible design, the super-resolution conversion module is specifically used for:

Construct a first state vector according to the state information and the first resolution, where the first state vector includes a parameter value indicating the state information and a state quantity of the first resolution; The state mapping rule determines the first target magnification to which the first state vector is mapped.

In a possible design, the super-resolution conversion module selects at least one super-resolution algorithm according to the first target magnification, and enlarges the source image to the first target magnification to obtain a second resolution When the target image is used, it is specifically used to: select the first super-resolution algorithm in the super-resolution algorithm set according to the first target magnification, and use the first super-resolution algorithm to convert the source image into the target image; wherein , The first super-resolution algorithm is a super-resolution algorithm with the largest magnification among super-resolution algorithms with a set magnification of less than or equal to the first target magnification; the set of super-resolution algorithms includes the first There are multiple super-resolution algorithms including one super-resolution algorithm. One super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to different magnifications.

In a possible design, the super-resolution conversion module selects at least one super-resolution algorithm according to the first target magnification, and enlarges the source image at the first target magnification to obtain a target image with a second resolution When, it is specifically used to: perform M resolution conversion operations on the source image with the first resolution according to the first target magnification to obtain the target image with the second resolution, and M is greater than 0 Wherein, in the M resolution conversion operations, the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the first target magnification; the image obtained by the i-th resolution conversion operation is the first The input of i+1 resolution conversion operations, i+1 is an integer less than or M and greater than 1.

In a possible design, the super-resolution conversion module is specifically configured to determine the first target magnification for the source image according to the state information and the first resolution. The second resolution, and the first target magnification is determined according to the first resolution and the second resolution; wherein the second resolution is the resolution of the display device; or,

In a possible design, the super-resolution conversion module is further configured to obtain the resource running status during the process of performing the operation of zooming in the first target magnification on the source image by using the at least one resolution conversion algorithm Information; when it is determined that the resource operating state information meets a preset condition, continue to perform an operation of magnifying the first target magnification on the source image according to the first target magnification.

In a possible design, the super-resolution conversion module is also used to reduce the first target magnification to obtain the second target magnification when it is determined that the resource operating state information does not meet the preset condition, and according to the reduced all The second target magnification performs an enlargement operation on the source image to obtain a second target image with a third resolution, where the third resolution is obtained by enlarging the second target magnification by the first resolution.

In a fifth aspect, an embodiment of the present application provides an image processing device, which includes a processor that invokes instructions in the memory to implement any of the method embodiments and method embodiments of the first or third aspect described above. The functions involved in a possible design.

Exemplarily, the image processing apparatus may further include a transmission interface through which the processor receives or transmits image data. The transmission interface may include a transmission interface and a reception interface, the transmission interface is used to transmit image data to the display device, and the reception interface is used to receive the source image.

In the sixth aspect, the embodiments of the present application provide a computer program product, including a computer program. When the computer program is executed on a computer or processor, it will enable the computer or processor to implement the first aspect or the third method described above. The functions involved in any possible design of the embodiment and method embodiment.

In the seventh aspect, the embodiments of the present application provide a computer-readable storage medium for storing programs and instructions. When these programs and instructions are invoked and executed in a computer, the computer can execute the above-mentioned first or third aspects. The functions involved in the method embodiment and any possible design of the method embodiment are described.

In an eighth aspect, the embodiments of the present application provide a chip system, which includes a processor and may also include a memory, for implementing the functions involved in the above method. The chip system can be composed of chips, or can include chips and other discrete devices.

Description of the drawings

FIG. 1A is a schematic structural diagram of an image processing system 100 provided by an embodiment of this application;

1B is a schematic structural diagram of another image processing system 100 provided by an embodiment of this application;

2 is a schematic structural diagram of an image processing device 102 provided by an embodiment of the application;

FIG. 3 is a flowchart of an image processing method provided by an embodiment of the application;

4 is a schematic diagram of a controlled super-resolution algorithm provided by an embodiment of the application;

FIG. 5 is a schematic structural diagram of another image processing apparatus 102 provided by an embodiment of this application;

FIG. 6 is a schematic diagram of an image processing method provided by an embodiment of the application;

FIG. 7 is a schematic diagram of status monitoring information provided by an embodiment of this application;

FIG. 8A is a schematic diagram of another image processing apparatus 102 provided by an embodiment of the application;

FIG. 8B is a schematic diagram of another image processing apparatus 102 provided by an embodiment of the application;

FIG. 8C is a schematic diagram of another image processing apparatus 102 provided by an embodiment of the application.

Detailed ways

Part of the terms involved in this application will be described in detail below.

The term "at least one" referred to in this application refers to one, or more than one, that includes one, two, three and more; "multiple" refers to two, or more than two, that includes two, Three and more. In addition, it should be understood that in the description of this application, words such as “first” and “second” are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating Or imply the order. "And/or" describes the association relationship of the associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the associated objects are in an "or" relationship. "The following at least one item (a)" or similar expressions refers to any combination of these items, including any combination of a single item (a) or plural items (a). For example, at least one item (a) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple .

Resolution. Resolution is a parameter used to measure the amount of data in an image. It is usually expressed as ppi (Pixel per inch). The 320*180 of the video refers to the effective pixels in the horizontal and vertical directions. The window is small. The ppi value is higher and it looks clearer.

Super-resolution technology refers to the use of one or more low-resolution images to obtain a high-resolution image by means of hardware or software, thereby increasing the resolution of the original input image. Super-resolution algorithms are divided into generalized super-resolution algorithms and narrowly-defined super-resolution algorithms. The generalized super-resolution is an image resolution magnification algorithm, and the narrow one generally refers to an image quality enhancement method using machine learning methods. The difference between the super-resolution algorithm and the traditional image magnification algorithm is that the traditional image magnification algorithm is manually designed, which is a fixed method and has a lower computational consumption; the super-resolution algorithm is based on deep learning or machine learning, and requires a training process and complex processing. , The calculation consumes a lot, and the performance requirements of the processor are higher.

Super-resolution algorithms include many types, such as BiCubic, and super-resolution algorithms based on deep learning, for example, Super-Resolution Convolutional Neural Network (SRCNN), Another example is the super-resolution algorithm (hisilicon super-resolution, HiSR) of the mobile terminal, and the super-resolution algorithm based on dictionary training. The embodiments of this application do not specifically limit the super-resolution algorithm, and all existing super-resolution algorithms are applicable to this application.

Magnification refers to the multiple by which the resolution is enlarged. For example, the input resolution name is 1080P, the foreign letter P means progressive scan, and the output resolution name is 2K, which generally refers to a display device or content with a horizontal resolution of about 2000 pixels, and the resolution ratio is 2. Enlarge the input resolution by 2 times to get the output resolution. Exemplarily, in different application scenarios, different resolution names may correspond to different resolutions. In a certain application scenario, a resolution name may correspond to only one resolution. However, in different application scenarios, the same resolution name can correspond to different resolutions. For example, the resolution name is 2K, the common resolution corresponding to 2K is 2560×1440, and there are derivative resolutions, such as 2048×1536, 1998×1080, 2048×858, etc.

It should be understood that the magnification factor involved in the embodiments of the present application is not absolutely equal to the ratio of the output resolution to the input resolution, but is approximately equal to the ratio of the output resolution to the input resolution. For example, if the ratio of output resolution to input resolution is equal to 3.5, the resolution magnification is equal to the largest integer less than 3.5, which is 3 times.

The embodiments of the application provide an image processing method and device. For the received image, the resolution of the received image is adjusted according to the state control information of the image processing system, and the state control information of the image processing system in different application scenarios Different, therefore, based on the method of adjusting the resolution through the state control information provided by the embodiment of the present application, the resolution requirement in the business scenario can be met.

Hereinafter, the embodiments of the present application will be described in detail with reference to the drawings.

As shown in FIG. 1A, the image processing system 100 provided by the embodiment of the present application may include a signal source 101 and the image processing apparatus 102 provided by the embodiment of the present application. The signal source 101 is the source of processing content of the image processing device 102, which may be a picture or a video frame. The image source can be the network, mobile storage media, camera, camera equipment, etc. The image processing device 102 is configured to process the image from the signal source 101 according to the image processing method provided in the embodiment of the present application.

In an example, the image processing device 102 as shown in FIG. 1A may have a display function, and the image processing system 100 provided by the embodiment of the present application may also display images that have undergone image processing. The image is output to a display device. In this case, the image processing device 102 may be a display device such as a television or a display with image processing functions.

In another example, in the structure of another image processing system 100 as shown in FIG. 1B, the system 100 further includes a display device 103, where the display device 103 may be a device with a display function, such as a television, The display may also be a display screen, a projector, etc. The display device 103 is used for receiving the image transmitted by the image processing device 102 and displaying the received image. The image processing apparatus 102 here may be a playback device, such as a set-top box.

As an example, the image processing device 102 provided by the embodiment of the present application may have a structure as shown in FIG. 2. It can be seen that the image processing device 102 may include a processing unit 201, and the processing unit 201 may be used to implement the The steps involved in the image processing method, the specific image processing method provided in the embodiment of the present application will be described in detail later, and the description will not be repeated here.

The processing unit 201 may include one or more of the following: a central processing unit (CPU), or other general-purpose processors, image signal processors (image signal processors, ISP), microprocessors, and digital signal processing Digital signal processor (DSP), application specific integrated circuit (ASIC), graphics processing unit (GPU), neural-network processing unit (NPU), field programmable gate Array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. Optionally, the CPU may be a single-CPU processor or a multi-CPU processor; optionally, the CPU may be a processor group composed of multiple processors, between multiple processors Coupled to each other through one or more buses.

Exemplarily, the image processing device 102 may further include a storage unit 202, which may be used to store computer program instructions, including an operating system (Operation System, OS), various user applications, and program codes for executing the solutions of the present application. The memory can also be used to store video data, image data, etc.; the CPU can be used to execute the computer program code stored in the memory to implement the methods in the embodiments of the present application. The storage unit 202 can be coupled with the processing unit 201 to support the processing unit 201 to call computer programs and instructions in the storage unit 202 to implement the steps involved in the image processing method provided in the embodiments of the present application. In addition, the storage unit 202 can also use For storing data. In the various embodiments of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices. For example, it can be coupled through various interfaces, transmission lines or buses. Optionally, the memory may be a non-power-down volatile memory, such as an embedded multi-media card (EMMC), universal flash storage (UFS), or read-only memory, ROM), or other types of static storage devices that can store static information and instructions, or volatile memory (volatile memory) such as random access memory (RAM) or can store information and Other types of dynamic storage devices for instructions can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical discs Storage, optical disc storage (including compact discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store program codes in the form of instructions or data structures and can Any other computer-readable storage medium accessed by the computer, but not limited to this.

Exemplarily, the image processing device 102 may further include a sending unit 203 and/or a receiving unit 204, where the sending unit 203 may be used to output processed images, and the receiving unit 204 may receive images from the signal source 101. Exemplarily, the sending unit 203 and/or the receiving unit 204 may be an image output interface, such as a high definition multimedia interface (HDMI), a video graphics array (VGA), and a digital video interface (digital video interface). video interface, DVI), display interface (DisplayPort, DP).

Exemplarily, the image processing apparatus 102 may further include a display unit 205, such as a display screen or a display panel, for displaying the image processed by the processing unit 201.

Exemplarily, the image processing device 102 may further include a video decoding unit 206 for decoding an image from the signal source 101, so that the processing unit 201 processes the image decoded by the video decoding unit 206.

Optionally, the image processing device 102 may also include a dedicated video or graphics processor, a microprocessor, and a microcontroller MCU, etc. For example, the dedicated video/graphics processor may be a dedicated ISP.

The following describes an image processing method provided by an embodiment of the present application with reference to FIG. 3. The method may be executed by an image processing device, for example, by the image processing device 102. The method includes the following steps:

S301: The image processing apparatus receives a source image, where the source image has a first resolution.

S302: The image processing apparatus obtains status information. Wherein, the state information includes display state information used to characterize the display performance of the display device, and the display device and the image processing device are coupled through a transmission interface to transmit image data. Exemplarily, the display device may be located inside the image processing device or outside the image processing device.

Exemplarily, the image processing apparatus may be applied to a set-top box, for example, it may be a processor or a processing chip disposed inside the set-top box. In this case, the display device may be a television.

It should be noted that the order of receiving the source image and obtaining the display state information is not specifically limited in this application. For example, the display state information may be obtained before the receiving source image, or the display state information may be obtained after the source image is received, or after the source image is received. The operation of acquiring the display status information is performed during the process of the source image, which is not specifically limited in this application.

S303: The image processing apparatus determines a first target magnification for the source image according to the state information and the first resolution. The target image obtained by enlarging the source image at the first target magnification can be adapted to the display performance of the display device. The resolution of the target image obtained by enlarging the first target magnification is the second resolution.

Exemplarily, when the image processing device determines the first target magnification for the source image according to the status information and the first resolution, it may first determine the second resolution of the target image according to the status information, and then determine the second resolution of the target image according to the first resolution and the first resolution. The second resolution determines the first target magnification.

S304: Select at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution. The target image of the second resolution satisfies the requirement of the state information. Wherein, the target image with the second resolution is adapted to the display performance of the display device.

Exemplarily, the information in the state may include: display state information, flow state information, or resource operation state information.

Exemplarily, the display status information may include one or more of the resolution of the display device, the interface type of the display device, or the size of the display device.

Exemplarily, the stream state information includes the frame rate of the video stream and/or parameters used to represent the format of the image stream.

Exemplarily, the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.

In an example, the display status information includes the resolution of the display device, and in performing S302, the image processing device may determine the resolution of the display device as the second resolution.

In another example, the display status information includes the interface type of the display device, and in performing S302, the image processing device may determine the highest resolution supported by the interface type of the display device as the second resolution.

The interface types of the display device may include multiple types, such as VGA, DVI, HDMI, or DP.

For example, the resolution name of the highest resolution supported by VGA is 1080P. Illustratively, the resolution name of 1080P is 1920×1080. When the interface type of the display device is VGA, 1920×1080 can be configured as the second resolution. In the following description, the resolution that meets a certain resolution name, such as 1920×1080, meets 1080P, is called 1080P resolution.

For another example, DVI generally supports the resolution name of the highest resolution 1080P, and when the interface type of the display device is VGA, 1920×1080 can be configured as the second resolution.

For another example, HDMI includes multiple transmission protocol versions. The highest resolution supported by HDMI 1.0, HDMI 1.1, and HDMI 1.2 are all 1600×1200, and the highest resolution supported by HDMI 1.3 is 2048×1536. The name of the corresponding resolution is 2K, the highest resolution supported by HDMI 1.4 is 4096×2160, the corresponding resolution name can be 4K, the highest resolution supported by HDMI 2.0 is 4096×2160, and the corresponding resolution name can be 4K.

It should be understood that the same resolution name can correspond to multiple resolutions, such as 2K resolution, which generally refers to a display device or content with a horizontal resolution of about 2000 pixels. The common resolution corresponding to 2K is 2560×1440, and there are derivative resolutions, such as 2048×1536, 1998×1080, 2048×858, etc. For another example, 4K, the common resolution is 4096×2160, and there are derivative resolutions, such as 4096×3112, 3656×2664, and 3840×2160.

When configuring, you can determine the resolution corresponding to a certain resolution name according to the actual situation. Illustratively, the display device can generally be configured with a specific resolution. Of course, in a scenario where a display device is applied, when a resolution name corresponds to only one resolution, the display device may also be configured with only the resolution name, or the corresponding relationship between the resolution name and the resolution.

Exemplarily, when determining the version of the HDMI transmission protocol, the external display device hardware, display device system software, and HDMI communication line must meet the standards of a certain transmission protocol version at the same time, then the transmission protocol version is determined to be effective. In addition, it should be understood that the HDMI standard is backward compatible. When the transmission protocol version corresponding to a certain link is inconsistent with the transmission protocol version corresponding to other links, the lowest transmission protocol version is taken as the HDMI transmission protocol version this time.

For another example, DP generally supports the highest resolution resolution name is 4K.

In yet another example, when the display status information includes the size of the display device, in performing S302, the image processing device may determine the highest resolution supported by the size of the display device as the second resolution. Therefore, the output image of the second resolution can meet the size requirement of the display device in the state vector. At this time, the second resolution image is adapted to the display device.

In another example, when the display status information includes the resolution of the display device and the interface type of the display device, then in execution S302, the image processing device may compare the highest resolution supported by the interface type of the display device with the interface type of the display device. The minimum value among the resolutions is determined as the second resolution. Therefore, the output image of the second resolution can meet the requirements of the size of the display device and the resolution of the display device in the state vector. At this time, the second resolution image is adapted to the display device.

For example, if the interface type of the display device is HDMI 2.0, the highest resolution supported is 4096×2160, and the resolution of the display device is 2048×1080, it can be determined that the second resolution is 2048×1080.

In another example, when the display status information includes the size of the display device and the resolution of the display device, then in execution S302, the image processing device compares the highest resolution supported by the size of the display device with the resolution of the display device. The minimum value of is determined as the second resolution.

In another example, when the display status information includes the size of the display device, the interface type of the display device, and the resolution of the display device, then in execution S302, the image processing device can set the highest resolution supported by the size of the display device The minimum value among the highest resolution supported by the interface type of the display device and the resolution of the display device is determined as the second resolution.

When determining the target resolution (second resolution) based on the display status information of the display device (such as the resolution, size, or interface type of the display device), select the resolution that the display device can support as the target resolution, such as When the resolution is higher, the output image with higher resolution is output, which improves the user experience. When the resolution of the display device is lower, the output resolution is limited, and the performance requirements and power consumption of super-resolution algorithm execution are reduced. For another example, if the resolution of the display device is high and the size and interface type are sufficient to support the resolution, then the output image with the higher resolution will be output. When the resolution of the display device is low, but the size and interface type are not enough to support the resolution. At the resolution, the output resolution is limited, and the performance requirements and power consumption of the super-resolution algorithm are reduced.

S303: The set-top box selects at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution.

It should be understood that the embodiments of the present application may also be suitable for reducing the resolution of an image, that is, the second resolution may be lower than or equal to the first resolution. For example, the embodiment of the present application may be configured with a super-resolution conversion algorithm with a magnification ratio of less than 1, equal to 1, and greater than 1. In the subsequent description, the first resolution is enlarged as an example, that is, the resolution of the input image is enlarged as an example.

In a possible implementation manner, in the embodiment of the present application, when determining the target magnification according to the state information, the input state function f _i (x) may be determined by mapping, for example, for the parameter x _i included in the state information , F _i (x) can include mapping and/or normalization to obtain the state quantity; construct the input state vector X=[x ₁ ,x ₂ ,...x _n ] ^T ; the state mapping rule can be through the state mapping function F(X) Indicates that it is used to output the over-divided control amount, such as target magnification. Illustratively, F(X)=AX+B. In the embodiments of this application, the target magnification is taken as an example.

Exemplarily, when the image processing apparatus determines the first target magnification according to the state information and the first resolution, it may first construct a state vector based on the state information and the first resolution, and the state vector includes parameter values respectively used to indicate that the state information includes And the state quantity of the first resolution; and then the first target magnification mapped by the state vector is determined according to the state mapping rule. The status information includes the resolution, size, and interface type of the display device. The state vector can include 4 state quantities. For example, the state vector is expressed as X=[x ₁ , x ₂ , x ₃ , x ₄ ] ^T , x ₁ , x ₂ , x ₃ , x ₄ can be used to indicate the first resolution, the resolution and size of the display device, respectively And the type of interface. Use the state vector as the input of F(X) to get the target resolution.

In a possible implementation manner, after the image processing apparatus determines the second resolution, when S303 is performed, after the image processing apparatus determines the target magnification, the controlled super-resolution conversion method provided in this embodiment of the application will have the first resolution. The source image of the resolution is enlarged at the first target magnification to obtain the target image of the second resolution. The controlled super-resolution conversion method is a super-resolution conversion method with a controllable output resolution.

In the embodiment of this application, the target magnification is used as the basis for selecting the super-resolution algorithm and the number of resolution conversions. For example, the super-resolution algorithm of the target magnification is selected to perform a resolution conversion. For example, a super-resolution algorithm can be selected. The resolution algorithm performs multiple resolution conversions, and the number of executions can be determined according to the target magnification. Another example is to select a super-resolution combination with a combined magnification that reaches the target magnification among the multiple super-resolution algorithm combinations configured.

The controlled super-resolution conversion method is exemplarily described as follows.

The first method uses a super-resolution algorithm set, which includes super-resolution algorithms with multiple magnifications. In other words, the super-resolution algorithm includes multiple super-resolution algorithms, one super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to different magnifications. For example, referring to Fig. 4, the super-resolution algorithm set includes ×2 super-resolution algorithm, ×3 super-resolution algorithm, and ×4 super-resolution algorithm. Optionally, the magnification may also be a non-integer, such as ×2.25 super-resolution algorithm, ×3.5 super-resolution algorithm, and ×4.25 super-resolution algorithm.

In one manner, when the image processing apparatus converts the received source image with the first resolution into the target image with the second resolution, it can be implemented in the following manner:

The image processing device selects the first super-resolution algorithm in the super-resolution algorithm set according to the target magnification, and uses the first super-resolution algorithm to convert the source image with the first resolution into the target image with the second resolution. The super-resolution algorithm is the super-resolution algorithm with the largest magnification among the super-resolution algorithms whose magnification is less than or equal to the target magnification; among them, the super-resolution algorithm focuses on multiple types including the first super-resolution algorithm Super-resolution algorithm, a super-resolution algorithm corresponds to one magnification, and different kinds of super-resolution algorithms correspond to different magnifications.

Exemplarily, when the magnification corresponding to the super-resolution algorithm included in the super-resolution algorithm set includes a non-integer, the image processing device may compare the second resolution with the first resolution when determining the target magnification according to the first resolution and the second resolution. The ratio of a resolution is determined as the target magnification. When selecting the first super-resolution algorithm in the super-resolution algorithm set, select the super-resolution algorithm with the largest magnification among the super-resolution algorithms less than or equal to the target magnification.

The magnifications corresponding to the super-resolution algorithms included in the super-resolution algorithm set may also be integers. When the image processing device determines the target magnification according to the first resolution and the second resolution, the ratio of the second resolution to the first resolution may be rounded down to determine the target magnification. For example, if the ratio of the second resolution to the first resolution is 3.5, the rounded down value is 3, and the target magnification is 3. When the first super-resolution algorithm is selected in the super-resolution algorithm set, the first super-resolution algorithm corresponding to the target magnification can be selected. Of course, when the ratio of the second resolution to the first resolution is an integer, in this case, the ratio is the target magnification.

The second method is a multi-level super-resolution conversion method. This mode is a controllable conversion mode at the termination time. The magnification can be the same or different each time.

In the second way, when the image processing device converts the source image with the first resolution into the target image with the second resolution, it can be implemented in the following ways:

Performing M resolution conversion operations on the source image with the first resolution according to the target magnification to obtain the target image with the second resolution, where M is an integer greater than 0;

Among them, when M is greater than 1, in M resolution conversion operations, the magnification product of the super-resolution algorithm used in M resolution conversion is equal to the target magnification; the image obtained by the i-th resolution conversion operation is the i+1 The input of the sub-resolution conversion operation, i+1 is an integer less than or equal to M and greater than 1.

In the M resolution conversion operation, the super-resolution algorithm used may be different or the same.

In the first optional case, the same super-resolution algorithm is executed cyclically, that is, when the super-resolution algorithm is executed multiple times, the magnification is the same each time. In the first case, only the image A super-resolution algorithm is configured in the processing device, that is, N=1, and the super-resolution corresponds to one magnification, and the complexity is low. For example, each time the magnification is ×2, execute M times, M is an integer greater than 0, and when M takes a different value, the multi-level super-resolution conversion method can support the target magnification of ×2, ×4, ×8, ... ,As shown in Figure 4. Every time after the resolution is converted through the super-resolution algorithm, it can be judged whether the target condition is reached, if it is reached, it will be terminated, and if it is not reached, the resolution conversion will continue. The target condition may be a target resolution or a target magnification or a target number of times.

In the first case, taking the target times as an example, as shown in FIG. 4, the ratio of the target magnification to the magnification of the super-resolution algorithm is the target times. The image processing device uses a super-resolution algorithm to perform resolution conversion on the source image with the first resolution as the input image, and counts the number of resolution conversion operations, and determines whether the counting result reaches the target number of times, if the counting result is less than the target The resolution conversion is performed using the super-resolution algorithm as the input image until the count result reaches the target number of times. When the count result reaches the target number of times, the image obtained by the resolution conversion is The target image.

In the first case, take the target resolution as an example. For example, the first resolution is 1280×720, the second resolution is 2560×1440, and the magnification of the first super-resolution algorithm adopted is ×2, then After the first resolution image is converted using the first super-resolution algorithm, the resolution obtained does not meet the second resolution, and then the first super-resolution algorithm is used to perform a resolution conversion to obtain the resolution It is 2560×1440, which meets the second resolution, and stops execution. The image obtained by the second resolution conversion is the target image with the second resolution.

In the second optional case, multiple super-resolution algorithms are configured in the image processing device, each super-resolution algorithm corresponds to a magnification, and different super-resolution algorithms have different magnifications. When the super-resolution algorithm is executed multiple times, the magnification may be different or the same.

In one example, the target magnification can be used as a condition, and the combined super-resolution algorithm can be selected from multiple super-resolution algorithms. The combined super-resolution algorithm can include one-rate super-resolution algorithm or multiple-rate super-resolution Rate algorithm. The magnification of the combined super-resolution algorithm is equal to the target magnification. The magnification of the combined super-resolution algorithm is the product of the magnifications of each super-resolution algorithm included in the combination.

For example, the multiple super-resolution algorithms include ×2 super-resolution algorithms and ×3 super-resolution algorithms. For example, when the target magnification is ×4, the combined super-resolution algorithm includes two ×2 super-resolution algorithms, and the two ×2 super-resolution algorithms are the same. For another example, the target magnification is ×6, and the combined super-resolution algorithm includes ×2 super-resolution algorithm and ×3 super-resolution algorithm. For another example, if the target magnification is ×9, the combined super-resolution algorithm includes 3 ×3 super-resolution algorithms, and these 3 ×3 super-resolution algorithms are the same. As an example, when a super-resolution algorithm is selected from a variety of super-resolution algorithms, conditions such as the difference between the magnification of the super-resolution algorithm and the target magnification can be used as the cost to pass the cost function, and then the optimal combination can be searched.

As another example, a combination strategy table corresponding to different magnifications may be configured in advance, for example, see Table 1.

Table 1

组合倍率Combined magnification	策略Strategy
×2×2	×2超分辨率算法×2 super resolution algorithm
×3×3	×3超分辨率算法×3 super resolution algorithm
×4×4	×2超分辨率算法，2次×2 super resolution algorithm, 2 times
×5×5	×5超分辨率算法×5 super resolution algorithm
×6×6	×2超分辨率算法，×3超分辨率算法×2 super-resolution algorithm, ×3 super-resolution algorithm

Based on this, in a possible implementation manner, when determining the second resolution (target resolution), it can be determined not only based on the display state of the display device, but also based on the stream state of the video stream, that is, in the state information It can also include stream status information of the video stream. The image processing device can also acquire stream status information while acquiring display status information. The stream status information is used to characterize the stream status of the video stream to which the source image with the first resolution belongs; for example, the image processing apparatus may first determine the second resolution of the target image according to the stream status information and the display status information, and then The target magnification is determined according to the first resolution and the second resolution. The target image of the second resolution is adapted to the streaming state of the video stream and the display performance of the display device.

Exemplarily, the stream state information includes the frame rate of the video stream and/or parameters used to indicate the format of the video stream. The resolution supported by the frame rate of different video streams may be different. Different video stream formats may support different resolutions.

The interface types of different display devices may support different resolutions at different frame rates. For example, HDMI: HDMI 1.3 supports 1080P/120FPS (frames per second), HDMI 1.4 supports 4K/30FPS or 2K/60FPS, HDMI 2.0 supports 4K/60FPS.

In an example, the display status information includes the interface type of the display device, and the stream status information includes the frame rate of the video stream. When the image processing device determines the second resolution according to the stream status information and the display status information, it may determine the highest resolution supported by the interface type of the display device at the frame rate of the video stream as the second resolution.

For example, when the interface of the display device is HDMI 1.4, for a 30FPS stream, the second resolution is determined to be 4K resolution, for a 60FPS stream, the second resolution is determined to be 2K resolution, and for a 120FPS stream, the second resolution is determined It is 1080P resolution. For another example, when the interface of the display device is HDMI2.0, for 60FPS (or less than 60FPS) stream, the second resolution is determined to be 4K resolution, and for 120FPS stream, the second resolution is determined to be 2K resolution; another example is display When the interface of the device is DP, for a 60FPS (or less than 60FPS) code stream, the second resolution is determined to be 4K resolution, and for a 120FPS code stream, the second resolution is determined to be 2K resolution.

When the code stream frame rate or code stream resolution can match the available resources of the image processing device, a certain multiple of super-resolution algorithm is implemented to increase the output resolution and improve user experience; when the code stream frame rate is too high, image processing The resources may not be enough to support the requirements of the super-resolution algorithm. The target resolution can be reduced, the target magnification can be reduced, the demand for image processing resources can be reduced, and the occurrence of jams and other adverse effects can be prevented.

In another example, the display status information includes the interface type of the display device and the resolution of the display device, and the stream status information includes the frame rate of the video stream. When the image processing device determines the second resolution according to the stream status information and the display status information, it can distinguish the highest resolution supported by the interface type of the display device at the frame rate of the video stream from the smallest resolution of the display device The rate is determined as the second resolution.

For example, if the resolution of the display device is 4K resolution, the first resolution is 720P resolution, the interface of the display device is HDMI 1.4, and the frame rate of the video stream is 60FPS, it can be determined that the second resolution is 2K resolution.

In general, the higher the magnification of the super-resolution algorithm, the more complex the calculation, the more memory/processing resources are required to run the super-resolution algorithm. Based on this, pre-set memory/processing resource occupancy alert determination conditions (as preset conditions) may be required, for example, the memory occupancy rate is not more than 80%, the processor occupancy is not more than 80%, or the weighted average of the two is not more than 75 %. Based on this, the embodiment of the present application can determine whether to reduce the target magnification according to the occupancy of storage resources and/or processing resources of the image processing apparatus.

It should be understood that when determining the first target magnification, the state information used may include one or more parameters in the display state information and the stream state information. The maximum resolution supported by different parameters is different, so that the state information is When multiple parameters are included, when determining the second resolution, the smallest resolution among the resolutions supported by the multiple parameters is taken as the second resolution. For example, the status information includes two parameters, which are the resolution of the display device and the transmission interface type, and the second resolution is the minimum value of the resolution of the display device and the resolution supported by the transmission interface type. For another example, the status information includes the type and frame rate of the transmission interface. At different frame rates, the type of transmission interface supports different resolutions, so the resolution supported by the transmission interface at this frame rate can be used as the second resolution. .

In a possible implementation manner, when the state information further includes flow state information, the constructed state vector also includes a state quantity used to indicate the parameters included in the flow state information. The status information includes the resolution, size, interface type, frame rate, and image stream format of the display device. The state vector can include 6 state quantities. For example, the state vector is expressed as X=[x ₁ ,x ₂ ,x ₃ ,x ₄ ,x ₅ ,x ₆ ] ^T , x ₁ ,x ₂ ,x ₃ ,x ₄ ,x ₅ ,x ₆ can be used to indicate The first resolution, the resolution, size, interface type, frame rate, and image stream format of the display device. The first state vector is used as the input of F(X) to obtain the first target resolution. It should be noted that the order of the state variables corresponding to the parameters in the state vector is pre-configured according to the needs of F(X).

Exemplarily, the image processing device acquires the resource operating status information of the image processing device during the process of performing the resolution conversion operation on the source image with the first resolution according to the target magnification; the resource operating status information is used to indicate the image processing device The running state of the resource; when the image processing device determines that the resource running state information meets the preset condition, it continues to perform the resolution conversion operation on the source image with the first resolution according to the target magnification. Exemplarily, when determining that the resource operating state information does not meet the preset condition, the image processing apparatus reduces the target magnification to obtain the second target magnification, and performs a resolution conversion operation on the source image with the first resolution according to the second target magnification, Obtain a second target image with a third resolution at the target magnification after the resolution is enlarged and reduced, and the third resolution is the resolution obtained by enlarging the second target magnification by the first resolution.

It should be understood that the resources on the image processing apparatus may include hardware resources and/or software resources. In the embodiments of the present application, hardware resources, such as processor resources and memory resources, are taken as examples.

Exemplarily, when the controlled super-resolution conversion mode is the first mode, after the second resolution is determined according to the display status of the display device and/or the stream status of the video stream, the second resolution is determined according to the first resolution and the second resolution. After the resolution has determined the target magnification, the super-resolution algorithm corresponding to the target magnification can be tested for a preset time. When it is determined that the resource operating status information does not meet the preset conditions, the target magnification is reduced, and the reduction is selected in the super-resolution algorithm set The third super-resolution algorithm corresponding to the target magnification of, uses the third super-resolution algorithm to convert the source image with the first resolution into the second target image with the third resolution. For example, before the reduction, the target magnification is ×4, and the target magnification after the reduction may be ×3.

Exemplarily, when the controlled super-resolution conversion method is the second method, after the target magnification is determined according to the first resolution and the second resolution, taking the first case as an example, the same super-resolution can be executed repeatedly The algorithm is executed for the preset time, and when the resource running status information does not meet the preset conditions, the number of cycles to execute the same super-resolution algorithm can be reduced, that is, the target magnification is reduced, and the same super-resolution cycle is executed according to the reduced number of times The rate algorithm converts the source image with the first resolution into the second target image with the third resolution. Taking the second case as an example, the combined super-resolution algorithm can be executed for a preset time, and when the resource operating status information is monitored that does not meet the preset conditions, the target magnification is reduced, and the combined super-resolution algorithm is re-determined according to the reduced target magnification , And use the newly determined combined super-resolution algorithm to perform resolution conversion operations.

Image processing device resource operating status information, selecting processor occupancy rate and memory occupancy rate and other system resource operating status information. For example, processor occupancy rate=[0-100%], memory occupancy rate=[0-100%]. When the processor occupancy rate or memory occupancy rate is too high, it is not enough to support the device resource requirements of the super-resolution algorithm. The output resolution can be reduced to reduce the demand for device resources and prevent adverse effects such as jams; Or when the memory resources meet the demand, the output resolution can be increased to enhance the user experience.

Based on the same inventive concept, an embodiment of the present application provides an image processing device, which has the function of implementing the image processing method provided by any of the foregoing method embodiments. This function can be realized by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions.

An image processing device 102 provided by an embodiment of the present application may have a structure as shown in FIG. 2, wherein the processing unit 201 may be configured to execute steps S301, S302, and S303 shown in the method-side embodiment of the present application.

In an example, the structure of another image processing device 102 provided in an embodiment of the present application is shown in FIG. 5. The image processing device may include a state monitoring module 501 and a super-resolution conversion module 502. The state monitoring module 501 may be used to perform the steps described in the method S301 of the embodiment of the present application, and the controlled super-resolution module 502 may be used to perform the steps described in the method S302 of the embodiment of the present application.

With the above structure, the status monitoring module 501 in the image processing device 102 can be used to monitor status control information. The status control information (may be referred to as status information for short) can include the display status information of the display device. If the image to be processed belongs to the frame image in the video stream, the status control information can also include the stream status information and/or resource operation of the video stream. status information. The controlled super-resolution module 502 converts the source image with the first resolution into the target image with the second resolution according to the state control information.

Exemplarily, the controlled super-resolution module 502 may include a state control module 502A and a resolution conversion module 502B. The state control module 502A is used to determine the target magnification according to the state control information. Specifically, the state control module 502A determines the second resolution according to the state control information, and determines the target magnification according to the second resolution. The resolution conversion module 502B is configured to perform a resolution conversion operation on the source image with the first resolution according to the target magnification to obtain the target image with the second resolution with the resolution magnification target magnification.

Exemplarily, the image processing device 102 may further include a preprocessing module 503, which is used to perform image preprocessing on the image from the signal source, and input the image preprocessed into the controlled super-resolution module 502 . The main purpose of image preprocessing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of related information and simplify data to the greatest extent. For example, image preprocessing can include operations such as smoothing and filtering.

Exemplarily, the image processing device 102 may further include a post-processing module 504 configured to perform post-processing operations on the image output by the controlled super-resolution module 502, such as noise reduction and image enhancement. The post-processing module 504 inputs the image after the post-processing operation to the display device.

As an example, the image pre-processing operation and post-processing operation in the embodiment of the present application can also be performed by one module, that is, the pre-processing module 503 and the post-processing module 504 can be combined into one module.

Exemplarily, the functions of the state monitoring module 501, the super-resolution conversion module 502, the pre-processing module 503, and the post-processing module 504 may be implemented by the processing unit 201. Specifically, the state monitoring module 501, the super-resolution conversion module 502, the pre-processing module 503, and the post-processing module 504 may correspond to the processing unit 201.

For example, referring to FIG. 6 for an exemplary description of the interaction flow diagram of each module included in the image processing apparatus 102. After the image of the signal source is input to the image processing device 102, it is first processed by the preprocessing module 503. After the basic image preprocessing is completed, it is input to the resolution conversion module 502B of the controlled super-resolution module 502; the state control module 502A receives The state monitoring module 501 inputs the state control vector (hereinafter referred to as the state vector), and completes the conversion mapping of the state vector to the super-resolution algorithm (referred to as the super-resolution algorithm) control quantity (may be referred to as the super-resolution control quantity); resolution conversion module 502B When performing the resolution conversion operation processed by the pre-processing module 503, the super-division control amount generated by the state control module 502A is controlled; the post-processing module 504 performs related image post-processing on the high-resolution image output by the super-resolution algorithm operating. Exemplarily, the over-resolution control amount may be the target magnification. In FIG. 6, the first case where the controlled super-resolution module 502 adopts the second method is taken as an example for illustration.

Exemplarily, for ease of understanding, the image processing method provided in the embodiment of the present application is described by the following vector function relationship. The status monitoring module 501 needs to monitor status control information according to specific business scenarios, and construct a vector x _i corresponding to the status control information, i∈{1,2,...n}, where n is the total number of status information; for example, for status information x _i , determine the input state function f _i (x), f _i (x) may include mapping and/or normalization; construct the input state control vector X=[x ₁ , x ₂ ,...x _n ] ^T ; construct the state The mapping function F(X) is used to output the over-division control amount, for example, the target magnification. Illustratively, F(X)=AX+B.

Specifically, referring to FIG. 7 shows a selection scheme of the state control information acquired by the state monitoring module 501, the resource operation state information (such as processor occupancy rate, memory occupancy rate, etc.) and input source ( (I.e. video stream) stream status information (such as bit stream frame rate, bit stream resolution, bit stream format, etc.) and display status information of the display device (such as display device resolution, display device size, display device interface type Etc.) as the basis for super-resolution algorithm regulation.

The super-resolution module 502 is controlled based on the state vector X and the input image (image after preprocessing) control, when performing the resolution conversion operation on the input image, the input is set low resolution image P _in, P _out is the output high-resolution image, X is input from the control state vector, F (X) is a state mapping function, S (P, F) as a function of the controlled super-resolution conversion corresponding manner, the input image P _in low resolution and a state control amount X, the output is a high-resolution image P _out , then there is

P _out =S(P _in ,F(X)).

Specifically, determining which super-resolution algorithm or several super-resolution algorithms, or how many super-resolution conversion operations are performed, can be determined according to the target magnification output by F(X). For the specific method of determining the target magnification, refer to the foregoing description of the controlled super-resolution conversion method, which is not repeated here. The state mapping operation where F(X) is the state mapping function is executed by the state control module 502A, and the execution operation corresponding to the S(P, F) function is executed by the resolution conversion module 502B.

For example, the controlled super-resolution conversion method adopts the first method, that is, the super-resolution algorithm set. The resolution conversion module 502B can directly select which super-resolution algorithm to execute according to the output of the state control module 502A. For another example, the controlled super-resolution conversion method adopts the second method, that is, the multi-level super-resolution conversion method. Taking the first case as an example, the resolution conversion module 502B can directly determine to perform the resolution based on the output of the state control module 502A. The number of conversion operations. Taking the second case as an example, the resolution conversion module 502B can directly select the combined super-resolution algorithm based on the output of the state control module 502A, use the target magnification as the termination condition, and update F( X), judge whether the converted magnification reaches the target magnification. E.g:

Illustratively, take the state control information shown in FIG. 7 as an example, and take the application to a TV set-top box to display each frame image of a video stream as an example.

A1: Get the resolution Fps and frame rate Rs of the input code stream;

A2. Obtain the interface category T of the display device (TV) output by the TV set-top box and the resolution Rd of the display device, and determine the upper limit Ro of the output resolution;

Exemplarily, when the display device interface is VGA, the maximum output resolution Rif is determined to be 1080P; when the display device interface is DVI, the maximum output resolution Rif is determined to be 1080P; when the display device interface is HDMI1.3, the maximum output resolution is determined Resolution Rif is 1080P; for HDMI1.4, for 30FPS stream, determine the maximum output resolution Rif is 4K, for 60FPS stream, determine the maximum output resolution Rif is 2K, for 120FPS stream, determine the maximum output resolution Rif is 1080P; when it is HDMI2.0, for 60FPS (or less than 60FPS) stream, determine the maximum output resolution Rif is 4K, for 120FPS stream, determine the maximum output resolution Rif is 2K; when the interface of the display device is DP For a code stream of 60FPS (or less than 60FPS), the maximum output resolution Rif is determined to be 4K, and for a code stream of 120FPS, the maximum output resolution Rif is determined to be 2K.

Combine the resolution Rd of the display device and the maximum output resolution Rif, and take the smaller value as the upper limit Ro of the output resolution.

A3, according to the resolution Rs of the input code stream and the upper limit Ro of the output resolution, deduce the required target magnification Sr;

Sraw=Ro/Rs;

Exemplarily, if Sraw is a non-integer, a positive integer smaller than Sraw and closest to Sraw can be selected as Sr. If Sraw=3.5, the controlled super-resolution algorithm supports ×2, ×3, ×4, then Sr is equal to less than 3.5 and the magnification closest to 3.5 is ×3.

A4, further determining whether the target magnification needs to be updated based on the memory and/or processor occupation resource alert determination condition.

Step 1. Based on A3 to obtain the target magnification Sr, the image processing device 102 may try to run the super-resolution algorithm for a preset duration (for example, 1 s) without outputting to the display device.

Step 2. Detect the memory/processor resource occupancy rate of the super-resolution algorithm implementing the target magnification Sr. For example, setting the memory and/or processor occupancy warning conditions are: 1) the memory occupancy rate is less than or equal to 80%, 2) the processor occupancy rate is less than or equal to 80%, 3) the weighted average of the two is not more than 75%, such as a* Lmem+b*Lpcs<=75%, where Lmem represents memory usage, Lpcs represents processor usage, and a and b are weighting coefficients.

Step 3. After monitoring the implementation of the target magnification Sr over-dividing algorithm, determine whether to trigger the memory/processor occupation alert.

For example, the size of the memory occupancy rate is used as the condition to trigger the occupancy alert. If the condition 1) is not met, the occupancy alert is triggered. For another example, when the processor occupancy rate is used as the condition for triggering the occupation alert, if the condition 2) is not met, the occupation alert is triggered. Another example is to use the processor occupancy rate and memory size as the conditions for triggering the occupancy alert. One way is to trigger the occupancy alert if conditions 1) and 2) are not met; the other way is if condition 3 is not met ), the occupation alert is triggered.

Step 4. If the memory/processor occupancy alert is not triggered, the current input target Sr is used as the output Sout; if the memory/processor occupancy alert is triggered, the target multiplier is reduced by one level to obtain Sr'. As the input of step 1, execute step 1, test run the super-resolution algorithm of Sr' for 1 second, query the memory/processor resources required to implement the super-resolution algorithm under Sr', and query whether the memory occupation alert will be triggered. Based on step 4, the final output magnification Sout is obtained.

A5, Sout run using super resolution algorithm, using an input video image P _in S (P _in, Sout) output high-resolution video image Pout.

For example, see Table 2 to illustrate the output resolution of the output image corresponding to a different state control information.

Table 2

Exemplarily, during the video playback process, the memory/processor occupancy resources may be sampled once at a certain interval to determine whether to trigger the memory/processor occupancy alert. If it is triggered, similar to the step 4, reduce the magnification by one level, and update Sout to perform the resolution conversion operation using the updated Sout.

It should be understood that the state monitoring module 501 and the controlled super-resolution module 502 in the image processing device 102 shown in FIG. 5 can be composed of a central processing unit, a general-purpose processor, a digital signal processor, an application specific integrated circuit, and field programmable A gate array or other programmable logic devices, transistor logic devices, hardware components or any combination thereof can be implemented, which can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor may also be a combination that implements computing functions, for example, a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. In addition, the storage module that the image processing apparatus 102 may include may be a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory.

Exemplarily, referring to FIG. 8A, another possible structure of the image processing apparatus 102 in the embodiment of the present application includes a main processor 801, a memory 802, and a video processor 803.

In an example, the main processor 801 may be used to support the image processing device 102 to implement monitoring state control information and perform resolution conversion operations. For example, the main processor 801 can be used to monitor status control information, such as the method shown in step S301 that can be executed by the main processor 801, and the main processor 801 can also be used to perform target resolution determination according to the status control information. Adjust the resolution of the input image, for example, the method shown in steps S302 and S303 that can be executed by the main processor 801. The main processor 801 can also perform functions other than the image processing related functions provided by the embodiments of the present application, such as receiving image signals from a signal source, decoding the image signals, and sending the decoded image signals to the video processor 803 . The video processor 803 may be used to support the image processing device 102 to implement related functions of video signal processing. For example, the video processor 803 may be used to perform image preprocessing and postprocessing operations. The main processor 801 may be used to implement related functions of the state monitoring module 501 and the controlled super-resolution module 502 in FIG. 5. The video processor 803 may be used to implement the related functions of the preprocessing module 503 and the postprocessing module 504 in FIG. 5, as shown in 8B. The memory 802 is used to support the main processor 801 and the video processor 803 to call computer programs and instructions in the memory 802 to implement the steps involved in the video processing method provided by the embodiments of the present application. In addition, the memory 802 is also used to store data, such as Used to store super-resolution algorithms and related configuration parameters. The memory 802 may include both volatile and non-volatile memory, such as memory and hard disk. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electronic Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM) ) And direct memory bus random access memory (direct rambus RAM, DRRAM).

In another example, the main processor 801 may be used to support the image processing device 102 to implement monitoring state control information, and the resolution conversion operation is performed by the video processor 803. For example, the main processor 801 can be used to monitor state control information, such as the method shown in step S301 that the main processor 801 can execute. The main controller 801 can also perform functions other than the image processing related functions provided by the embodiments of the present application, such as receiving image signals from a signal source, decoding the image signals, and sending the decoded image signals to the video processor 803 . The video processor 803 can be used to support the image processing device 102 to implement related functions of video signal processing. For example, the video processor 803 can be used to determine the target resolution according to the state control information, and adjust the resolution of the input image based on the target resolution, such as video The processor 803 may execute the methods shown in steps S302 and S303. The video processor 803 can also be used to perform image pre-processing and post-processing operations. The main processor 801 may be used to implement related functions of the state monitoring module 501 in FIG. 5. The video processor 803 can be used to implement the related functions of the controlled super-resolution module 502, the pre-processing module 503, and the post-processing module 504 in FIG. 5, as shown in 8C. In an example, acquiring the processing resource occupancy rate when the image processing apparatus 102 runs the super-resolution algorithm refers to the processor occupancy rate of the video processor 803.

It should be understood that the image processing device 102 shown in FIG. 8A and FIG. 8B only exemplarily embodies the structure required by the image processing device 102 to execute the above-mentioned image processing method involved in the embodiment of the application, and the embodiment of the application does not exclude the image The processing device 102 also has other structures. For example, the image processing device 102 may also include a display device for displaying high-resolution images output by the video processor 1203; for another example, the image processing device 102 may also include necessary interfaces to implement Image input and output of processed images.

In addition, it should be understood that all steps performed by the image processing apparatus 102 shown in FIGS. 8A and 8B may be completed by the main processor 801. At this time, the image processing apparatus 102 may only include the main processor 801 and the memory 802.

In specific implementation, the main processor 801 and the video processor 803 may be central processing units, general-purpose processors, digital signal processors, application specific integrated circuits, field programmable gate arrays or other programmable logic devices, transistor logic devices, hardware The components or any combination thereof can implement or execute various exemplary logical blocks, modules, and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor may also be a combination that implements computing functions, for example, a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. In addition, in a possible implementation manner, all the functions of the video processor 803 can also be implemented through software using the main processor 801.

Exemplarily, the image processing apparatus 102 provided in the embodiments of the present application can be applied to smart devices such as set-top boxes, televisions, mobile phones, and other display devices with resolution conversion processing functions, and image processing devices to support the above devices to realize the application. The image processing method provided by the embodiment.

Exemplarily, the storage resources on which the function of the controlled super-resolution module 502 depends can be distinguished from the storage resources on which the functions of other modules depend. Programs and instructions for implementing the functions of the controlled super-resolution module 502 may be configured in a dedicated memory, such as dedicated memory resources and dedicated storage resources. The super-resolution algorithm and configuration parameters are stored in a dedicated memory resource, and the operation of the super-resolution algorithm depends on a dedicated memory resource. In the example, when acquiring the memory occupancy rate when the image processing device runs the super-resolution algorithm, what is acquired is the occupancy rate of dedicated memory resources.

Based on the same inventive concept, the embodiments of the present application provide a computer program product, including a computer program. When the computer program is executed on a computer, the computer will enable the computer to implement any of the above-mentioned image processing method embodiments. Function.

Based on the same inventive concept, the embodiments of the present application provide a computer program, which, when executed on a computer, will enable the computer to implement the functions involved in any of the foregoing image processing method embodiments.

Based on the same inventive concept, the embodiments of the present application provide a computer-readable storage medium for storing programs and instructions. When these programs and instructions are invoked and executed in a computer, the computer can execute any of the above-mentioned image processing method embodiments. The functions involved in.

Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

This application is described with reference to flowcharts and/or block diagrams of methods, equipment (systems), and computer program products according to the embodiments of this application. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. The instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.

Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the scope of the embodiments of the present application. In this way, if these modifications and variations of the embodiments of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

An image processing method, characterized by comprising:

The set-top box receives a source image, the source image having a first resolution;

The set-top box acquires state information; wherein the state information includes display state information used to characterize the display performance of a display device, and the set-top box and the display device transmit image data through a transmission interface;

Determining, by the set-top box, a first target magnification for the source image according to the status information and the first resolution;

The set-top box selects at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution, where the second resolution is The target image is adapted to the display performance of the display device.
The method according to claim 1, wherein the state information further includes stream state information used to characterize the video stream to which the source image belongs.
The method according to claim 1 or 2, wherein the status information further includes resource operating status information, and the resource operating status information indicates the operating status of the resources on the set-top box.
The method according to any one of claims 1 to 3, wherein the set-top box determining the first target magnification according to the status information and the first resolution comprises:

The set-top box constructs a state vector according to the state information and the first resolution, and the state vector includes a parameter value for indicating the state information and a state quantity of the first resolution;

The set-top box determines the first target magnification mapped by the state vector according to the state mapping rule.
The method according to any one of claims 1 to 4, wherein the set-top box selects at least one super-resolution algorithm according to a first target magnification, and enlarges the source image to the first target magnification to obtain a Two-resolution target image, including:

The set-top box selects a first super-resolution algorithm in a super-resolution algorithm set according to the first target magnification, and uses the first super-resolution algorithm to convert the source image into the target image;

Wherein, the first super-resolution algorithm is a super-resolution algorithm with the largest magnification among super-resolution algorithms whose intensive magnification is less than or equal to the first target magnification; and the super-resolution algorithm includes the There are multiple super-resolution algorithms including the first super-resolution algorithm. One super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to different magnifications.
The method according to any one of claims 1 to 4, wherein the set-top box selects at least one super-resolution algorithm according to a first target magnification, and enlarges the source image to the first target magnification to obtain a Two-resolution target image, including:

The set-top box performs M resolution conversion operations on the source image with the first resolution according to the first target magnification to obtain the target image with the second resolution, where M is an integer greater than 0;

Wherein, in the M resolution conversion operations, the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the first target magnification; the image obtained by the i-th resolution conversion operation is the i+1th The input of the sub-resolution conversion operation, i+1 is an integer less than or M and greater than 1.
The method according to any one of claims 1-6, wherein the display status information includes the resolution of the display device, the transmission interface type of the display device, or the size of the display device. At least one.
8. The method according to claims 2-7, wherein the stream status information includes a frame rate of the video stream and/or a parameter used to represent the format of the image stream.
The method according to any one of claims 3-8, wherein the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.
The method according to claim 8, wherein determining a first target magnification for the source image according to the state information and the first resolution comprises:

Determining the second resolution according to the state information, and determining the first target magnification according to the first resolution and the second resolution;

Wherein, the second resolution is the resolution of the display device; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device; or,

The second resolution is the minimum of the highest resolution supported by the size of the display device and the resolution of the display device; or,

The second resolution is the smallest value among the highest resolution supported by the size of the display device, the highest resolution supported by the interface type of the display device, and the resolution of the display device; or,

The second resolution is the highest resolution supported by the interface type of the display device under the frame rate of the video stream; or,

The second resolution is the minimum of the highest resolution supported by the interface type of the display device and the resolution of the display device at the frame rate of the video stream; or,

The second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the maximum resolution of the display device when the frame rate of the video stream is used. The minimum value in resolution; or,

The second resolution is the highest resolution supported by the interface type of the display device when the format of the image stream is adopted; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device when the format of the image stream is adopted; or,

The second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the resolution of the display device when the format of the image stream is adopted. The smallest value in; or,

The second resolution is the highest resolution supported by the interface type of the display device when the frame rate of the video stream and the format of the image stream are used; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device when the frame rate of the video stream and the format of the image stream are used .
The method according to any one of claims 3-9, wherein the method further comprises:

Acquiring, by the set-top box, the resource operating state information in a process of performing an operation of enlarging the first target magnification on the source image by using the at least one resolution conversion algorithm;

When determining that the resource operating state information meets a preset condition, the set-top box continues to perform an operation of zooming in on the source image at the first target magnification according to the first target magnification.
The method of claim 11, further comprising:

When it is determined that the resource operating state information does not meet the preset condition, the first target magnification is reduced to obtain a second target magnification, and the source image is enlarged according to the reduced second target magnification to obtain a A three-resolution second target image, where the third resolution is obtained by enlarging the second target magnification of the first resolution.
An image processing device, characterized in that it is applied to a set-top box, and includes:

A transmission interface for receiving a source image, the source image having a first resolution;

A status monitoring module, configured to obtain status information, the status information including display status information used to characterize the display performance of the display device, and the set-top box and the display device transmit image data through the transmission interface;

A super-resolution conversion module, configured to determine a first target magnification for the source image according to the state information and the first resolution;

The super-resolution conversion module is further configured to select at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution, wherein , The target image with the second resolution is adapted to the display performance of the display device.
The apparatus according to claim 13, wherein the state information further comprises: stream state information used to characterize the video stream to which the source image belongs.
The device according to claim 13 or 14, wherein the status information further comprises: resource running status information, and the resource running status information indicates the running status of the resource on the set-top box.
The device according to any one of claims 13 to 15, wherein the super-resolution conversion module is specifically configured to:

Construct a first state vector according to the state information and the first resolution, where the first state vector includes a parameter value indicating the state information and a state quantity of the first resolution; The state mapping rule determines the first target magnification to which the first state vector is mapped.
The device according to any one of claims 13-16, wherein the super-resolution conversion module is specifically configured to:

Selecting a first super-resolution algorithm in a super-resolution algorithm set according to the first target magnification, and using the first super-resolution algorithm to convert the source image into the target image;

Wherein, the first super-resolution algorithm is a super-resolution algorithm with the largest magnification among super-resolution algorithms whose intensive magnification is less than or equal to the first target magnification; and the super-resolution algorithm includes the There are multiple super-resolution algorithms including the first super-resolution algorithm. One super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to different magnifications.
The device according to any one of claims 13-16, wherein the super-resolution conversion module is specifically configured to:

Performing M resolution conversion operations on the source image with the first resolution according to the first target magnification to obtain the target image with the second resolution, where M is an integer greater than 0;

Wherein, in the M resolution conversion operations, the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the first target magnification; the image obtained by the i-th resolution conversion operation is the i+1th The input of the sub-resolution conversion operation, i+1 is an integer less than or M and greater than 1.
The method according to any one of claims 13-18, wherein the display status information includes the resolution of the display device, the transmission interface type of the display device, or the size of the display device. At least one.
The device according to any one of claims 14-19, wherein the stream state information comprises a frame rate of the video stream and/or a parameter used to represent the format of the image stream.
The device according to any one of claims 15-20, wherein the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.
The device according to claim 20, wherein the super-resolution conversion module is specifically configured to determine the second resolution according to the state information, and according to the first resolution and the second resolution. The resolution determines the first target magnification;

Wherein, the second resolution is the resolution of the display device; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device; or,

The second resolution is the minimum of the highest resolution supported by the size of the display device and the resolution of the display device; or,

The second resolution is the smallest value among the highest resolution supported by the size of the display device, the highest resolution supported by the interface type of the display device, and the resolution of the display device; or,

The second resolution is the highest resolution supported by the interface type of the display device under the frame rate of the video stream; or,

The second resolution is the minimum of the highest resolution supported by the interface type of the display device and the resolution of the display device at the frame rate of the video stream; or,

The second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the maximum resolution of the display device when the frame rate of the video stream is used. The minimum value in resolution; or,

The second resolution is the highest resolution supported by the interface type of the display device when the format of the image stream is adopted; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device when the format of the image stream is adopted; or,

The second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the resolution of the display device when the format of the image stream is adopted. The smallest value in; or,

The second resolution is the highest resolution supported by the interface type of the display device when the frame rate of the video stream and the format of the image stream are used; or,

The second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device when the frame rate of the video stream and the format of the image stream are used .
The device according to any one of claims 15-22, wherein the super-resolution conversion module is further configured to:

In the process of performing the operation of magnifying the first target magnification on the source image by using the at least one resolution conversion algorithm, acquiring the resource operating state information;

When it is determined that the resource operating state information satisfies a preset condition, continue to perform an operation of enlarging the first target magnification on the source image according to the first target magnification.
The device of claim 23, wherein the super-resolution conversion module is further configured to:

When it is determined that the resource operating state information does not meet a preset condition, reduce the first target magnification to obtain a second target magnification;

Perform an enlargement operation on the source image according to the reduced second target magnification to obtain a second target image with a third resolution, where the third resolution is the first resolution to enlarge the second target The magnification is obtained.
An image processing device, characterized in that the device comprises: a processor and a transmission interface;

The processor receives or sends image data through the transmission interface;

The processor is configured to call software instructions in the memory to execute the method according to any one of claims 1 to 12.
A computer-readable storage medium, characterized in that instructions are stored in the computer-readable storage medium, which when run on a computer or processor, cause the computer or processor to execute any of claims 1 to 12 The method described in one item.
A computer program product containing instructions, wherein when the computer program product runs on a computer or a processor, the computer or the processor is caused to execute the method according to any one of claims 1 to 12.