WO2020220346A1 - Procédé et appareil de traitement d'image - Google Patents

Procédé et appareil de traitement d'image Download PDF

Info

Publication number
WO2020220346A1
WO2020220346A1 PCT/CN2019/085370 CN2019085370W WO2020220346A1 WO 2020220346 A1 WO2020220346 A1 WO 2020220346A1 CN 2019085370 W CN2019085370 W CN 2019085370W WO 2020220346 A1 WO2020220346 A1 WO 2020220346A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
display device
super
image
target
Prior art date
Application number
PCT/CN2019/085370
Other languages
English (en)
Chinese (zh)
Inventor
徐晨剑
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN201980095791.1A priority Critical patent/CN113728354A/zh
Priority to PCT/CN2019/085370 priority patent/WO2020220346A1/fr
Publication of WO2020220346A1 publication Critical patent/WO2020220346A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof

Definitions

  • This application relates to the field of video technology, and in particular to an image processing method and device.
  • images and videos can be generated from low-resolution images or videos that are transmitted or stored on the network to high-resolution images or videos.
  • terminal devices generally adapt to specific super-resolution algorithms to output specific resolutions, and terminal devices may be applied to different business scenarios. Therefore, terminal devices that only output specific resolutions may not meet the requirements of multiple business scenarios.
  • the present application provides an image processing method and device to solve the problem of not being able to meet the requirements of multiple business scenarios.
  • the present application provides an image processing method including the following steps: acquiring display state information; wherein the display state information is used to characterize the display state of the display device; and the second resolution is determined according to the display state information.
  • the second resolution is the target resolution determined for the image displayed by the display device; the received source image with the first resolution is converted into the target image with the second resolution.
  • the method may be executed by an image processing device.
  • the display device may be a part of the image processing device, or two devices that are separately deployed, and the two devices are coupled.
  • the target resolution is determined according to the requirements of the display state of the display device, so that the resolution of the converted image can be adapted to different application scenarios.
  • the solutions provided in the embodiments of this application can be applied to different application scenarios, for example, the resolutions of display devices in different application scenarios are different, and the method provided in this application is adaptive based on the display state of the display device in the current application scenario. Adjust the resolution of the input image accordingly.
  • converting the received source image with the first resolution into the target image with the second resolution includes: determining according to the first resolution and the second resolution of the received source image Target magnification; performing a resolution conversion operation on the source image with the first resolution according to the target magnification to obtain a target image with a second resolution, and the second resolution is obtained by enlarging the target magnification of the first resolution.
  • performing a resolution conversion operation on the source image with the first resolution according to the target magnification includes: selecting the first super-resolution algorithm in the super-resolution algorithm set according to the target magnification, and adopting the first super-resolution algorithm.
  • the super-resolution algorithm converts the source image with the first resolution into the target image with the second resolution.
  • the first super-resolution algorithm is the magnification in the super-resolution algorithm whose concentration is less than or equal to the target magnification.
  • the largest super-resolution algorithm; among them, the super-resolution algorithm sets a variety of super-resolution algorithms including the first super-resolution algorithm, one super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to magnification different.
  • the super-resolution algorithm can be adaptively selected according to the actual needs in the application scenario, so that the image processing device can be applied to different scenarios. Since the existing terminal adapts to a specific super-resolution algorithm to output a specific resolution, if it is changed to another hardware platform, the resolution algorithm needs to be reconfigured to meet the requirements of the hardware platform.
  • the solution provided by the embodiment of this application can reduce Integration workload of different hardware platforms.
  • converting the received source image with the first resolution into the target image with the second resolution includes: performing M resolutions on the source image with the first resolution according to the target magnification
  • the conversion operation obtains a target image with a second resolution, where M is an integer greater than 0; among them, in the M resolution conversion operations, the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the target magnification;
  • the image obtained by the i resolution conversion operation is the input of the i+1 resolution conversion operation, and i+1 is an integer less than or M and greater than 1.
  • one or more super-resolution algorithms are integrated in the image processing device, and the super-resolution algorithm and the number of executions can be adaptively selected according to the actual needs in the application scenario, so that the image processing device can be applied to different Scene.
  • the display status information includes at least one of the resolution of the display device, the interface type of the display device, or the size of the display device.
  • determining the second resolution according to the display status information includes: determining the resolution of the display device as the second resolution; or, combining the highest resolution supported by the interface type of the display device with the display device
  • the minimum of the resolutions of the display device is determined to be the second resolution; or, the minimum of the highest resolution supported by the size of the display device and the resolution of the display device is determined to be the second resolution; or
  • the minimum value among the highest resolution supported by the size, the highest resolution supported by the interface type of the display device, and the resolution of the display device is determined as the second resolution.
  • it further includes: acquiring stream status information, which is used to characterize the stream status of the video stream to which the source image belongs; and determining the second resolution according to the display status information, and the second resolution meets the requirements of the display device
  • the requirements for the display status include: determining the second resolution according to the stream status information and the display status information, and the second resolution meets the requirements of the stream status of the image stream and the requirements of the display status of the display device.
  • the determination conditions for determining the target resolution are increased or decreased, which not only improves the accuracy of the determined target resolution, but also further increases the applicable scenarios.
  • the stream state information includes the frame rate of the video stream and/or parameters used to represent the format of the image stream.
  • the maximum resolution supported by each parameter included in the display status information and the maximum resolution supported by each parameter included in the stream status information may be combined
  • the minimum value in is used as the second resolution.
  • the method further includes: in the process of performing a resolution conversion operation on the source image with the first resolution according to the target magnification, acquiring resource operating status information; and determining that the resource operating status information meets a preset condition , Continue to perform the resolution conversion operation on the source image with the first resolution according to the target magnification.
  • the resource running status information indicates the running status of the resource on the image processing device for performing the resolution conversion operation.
  • it further includes: when it is determined that the resource operating state information does not meet the preset condition, the target magnification is reduced to obtain the second target magnification, and the source image with the first resolution is determined according to the reduced second target magnification.
  • the resolution conversion operation is performed to obtain a second target image with a third resolution, and the third resolution is obtained by enlarging the second target magnification of the first resolution.
  • the resolution is further adjusted based on the resource operation of the image processing device in the process of performing the resolution conversion operation, so as to improve the accuracy of the determined target resolution and improve the adaptability to the current application environment.
  • the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.
  • an embodiment of the present application provides an image processing device that can implement the method provided in the first aspect and any one of the possible designs of the first aspect.
  • the function can be realized by hardware, or by hardware executing corresponding software, or by a combination of software and hardware.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • An image processing device may include a state monitoring module and a super-resolution conversion module; wherein the state monitoring module is used to obtain display state information; wherein the display state information is used to characterize the display state of the display device
  • the super-resolution conversion module is used to determine the second resolution according to the display status information, the second resolution meets the requirements of the display status of the display device, and the second resolution is the target resolution determined for the image displayed by the display device;
  • the received source image with the first resolution is converted into the target image with the second resolution.
  • the super-resolution conversion module is specifically used to: according to the received source image of the received source image when converting the received source image with the first resolution into the target image with the second resolution
  • the first resolution and the second resolution determine the target magnification; according to the target magnification, the resolution conversion operation is performed on the source image with the first resolution to obtain the target image with the second resolution, and the second resolution is the first resolution Magnified target magnification is obtained.
  • the super-resolution conversion module when performing a resolution conversion operation on the source image with the first resolution according to the target magnification, is specifically used to: select the second in the super-resolution algorithm set according to the target magnification.
  • a super-resolution algorithm, and the first super-resolution algorithm is used to convert the source image with the first resolution into the target image with the second resolution.
  • the first super-resolution algorithm is the super-resolution algorithm.
  • the concentration ratio is less than or equal to The super-resolution algorithm with the largest magnification among the super-resolution algorithms of the target magnification; among them, the super-resolution algorithm concentrates on multiple super-resolution algorithms including the first super-resolution algorithm, and one super-resolution algorithm corresponds to one magnification , Different super-resolution algorithms correspond to different magnifications.
  • the super-resolution conversion module when converting the received source image with the first resolution into the target image with the second resolution, is specifically used to:
  • the source image of the resolution performs M resolution conversion operations to obtain the target image with the second resolution.
  • N is an integer greater than
  • M is an integer greater than 0.
  • M resolutions The magnification product of the super-resolution algorithm used in the rate conversion is equal to the target magnification; the image obtained by the i-th resolution conversion operation is the input of the i+1-th resolution conversion operation, i+1 is less than or M and greater than 1 Integer.
  • the display status information includes at least one of the resolution of the display device, the interface type of the display device, or the size of the display device.
  • the super-resolution conversion module when determining the second resolution according to the display status information, is specifically used to: determine the resolution of the display device as the second resolution; or, set the interface of the display device The minimum of the highest resolution supported by the type and the resolution of the display device is determined as the second resolution; alternatively, the minimum of the highest resolution supported by the size of the display device and the resolution of the display device is determined as the second resolution Two resolutions; or, the minimum of the highest resolution supported by the size of the display device, the highest resolution supported by the interface type of the display device, and the resolution of the display device is determined as the second resolution.
  • the status monitoring module is also used to obtain stream status information, which is used to characterize the stream status of the video stream to which the source image belongs;
  • the super-resolution conversion module determines the second Resolution, when the second resolution meets the requirements of the display status of the display device, it is specifically used to: determine the second resolution according to the stream status information and the display status information, and the second resolution meets the requirements of the stream status of the image stream and the display Requirements for the display status of the equipment.
  • the stream state information includes the frame rate of the video stream and/or parameters used to represent the format of the image stream.
  • the status monitoring module is also used to obtain resource operating status information during the process of the super-resolution conversion module performing resolution conversion operations on the source image with the first resolution according to the target magnification.
  • the status information indicates the operating status of the resource on the image processing device used to perform the resolution conversion operation; when it is determined that the resource operating status information meets the preset condition, continue to perform the resolution conversion operation on the source image with the first resolution according to the target magnification .
  • the super-resolution conversion module is also used to reduce the target magnification to obtain the second target magnification when it is determined that the resource operating state information does not meet the preset condition, and the second target magnification is obtained according to the reduced second target magnification.
  • a source image of one resolution performs a resolution conversion operation to obtain a second target image with a third resolution, and the third resolution is obtained by enlarging the first resolution by the second target magnification.
  • the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.
  • an embodiment of the present application provides an image processing method, including: a set-top box receives a source image, the source image has a first resolution; the set-top box acquires status information; wherein, the status information includes The display status information of the display performance of the device, the set-top box and the display device transmit image data through a transmission interface; the set-top box determines the first target of magnification for the source image according to the status information and the first resolution Magnification; the set-top box selects at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution, wherein the second The resolution of the target image is adapted to the display performance of the display device.
  • the set-top box determines the target magnification corresponding to the super-resolution algorithm according to the acquired state information.
  • the state information includes the display state information of the display device.
  • the set-top box can determine the matching magnification.
  • the magnification is adapted to the display performance of the display after the magnification. Therefore, the set-top box can independently adapt to display devices of various display specifications.
  • the display specifications of the display device are not limited, and manual settings are not required, which greatly improves the user experience .
  • the state information further includes: stream state information used to characterize the video stream to which the source image belongs.
  • the status information further includes: resource running status information, and the resource running status information indicates the running status of the resource on the set-top box.
  • the set-top box determines the first target magnification according to the state information and the first resolution, including: the set-top box constructs the first state according to the state information and the first resolution Vector, the first state vector includes a parameter value for indicating the state information and a state quantity of the first resolution; the set-top box determines, according to the first state mapping rule, what is mapped by the first state vector The first target magnification.
  • the set-top box selects at least one super-resolution algorithm according to a first target magnification, and amplifies the source image at the first target magnification to obtain a target image with a second resolution, including: The set-top box selects the first super-resolution algorithm in the super-resolution algorithm set according to the first target magnification, and uses the first super-resolution algorithm to convert the source image into the target image; wherein, the first super-resolution algorithm
  • the resolution algorithm is the super-resolution algorithm with the largest magnification among the super-resolution algorithms whose magnification is less than or equal to the first target magnification; the super-resolution algorithm set includes the first super-resolution algorithm
  • the super-resolution algorithm set includes the first super-resolution algorithm
  • the set-top box selects at least one super-resolution algorithm according to a first target magnification, and amplifies the source image at the first target magnification to obtain a target image with a second resolution, including:
  • the set-top box performs M resolution conversion operations on the source image with the first resolution according to the first target magnification to obtain the target image with the second resolution, where M is an integer greater than 0; where In the M resolution conversion operations, the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the first target magnification; the image obtained by the i-th resolution conversion operation is the i+1th The input of the resolution conversion operation, i+1 is an integer less than or M and greater than 1.
  • the display state information includes at least one of the resolution of the display device, the transmission interface type of the display device, or the size of the display device.
  • the stream state information includes the frame rate of the video stream and/or parameters used to represent the format of the image stream.
  • the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.
  • determining the first target magnification for the source image according to the status information and the first resolution includes: determining the second resolution according to the status information, and according to The first resolution and the second resolution determine the first target magnification.
  • the minimum value among the maximum resolutions supported under each parameter included in the status information may be used as the second resolution.
  • the second resolution is the resolution of the display device.
  • the second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device; or,
  • the second resolution is the minimum of the highest resolution supported by the size of the display device and the resolution of the display device; or,
  • the second resolution is the smallest value among the highest resolution supported by the size of the display device, the highest resolution supported by the interface type of the display device, and the resolution of the display device; or,
  • the second resolution is the highest resolution supported by the interface type of the display device under the frame rate of the video stream.
  • the second resolution is the minimum of the highest resolution supported by the interface type of the display device and the resolution of the display device at the frame rate of the video stream; or,
  • the second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the maximum resolution of the display device when the frame rate of the video stream is used.
  • the minimum value in resolution or,
  • the second resolution is the highest resolution supported by the interface type of the display device when the format of the image stream is adopted; or,
  • the second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device when the format of the image stream is adopted; or,
  • the second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the resolution of the display device when the format of the image stream is adopted.
  • the second resolution is the highest resolution supported by the interface type of the display device when the frame rate of the video stream and the format of the image stream are adopted.
  • the method further includes: the set-top box uses the at least one resolution conversion algorithm to perform an operation of enlarging the first target magnification on the source image, acquiring the resource running status Information; when the set-top box determines that the resource operating state information meets a preset condition, it continues to perform a magnification operation of the first target magnification on the source image according to the first target magnification.
  • it also includes:
  • the first target magnification is reduced to obtain a second target magnification, and the source image is enlarged according to the reduced second target magnification to obtain a A three-resolution second target image, where the third resolution is obtained by enlarging the second target magnification of the first resolution.
  • an image processing device which is applied to a set-top box, and includes:
  • a transmission interface for receiving a source image, the source image having a first resolution
  • a status monitoring module configured to obtain status information, the status information including display status information used to characterize the display performance of the display device, and the set-top box and the display device transmit image data through the transmission interface;
  • the display state information is used to characterize the display state of the display device
  • the super-resolution conversion module is configured to receive a source image and determine a first target magnification for the source image according to the state information and the first resolution; the first target magnification selects at least one super-resolution
  • the rate conversion algorithm enlarges the source image at the first target magnification to obtain a target image with a second resolution, wherein the target image with the second resolution is compatible with the display performance of the display device.
  • the transmission interface may include a transmission interface and a reception interface.
  • the transmission interface is used for transmitting image data to the display device, and the receiving interface is used for receiving the source image.
  • the function of the transmission interface may be implemented by the sending unit and the receiving unit.
  • the sending unit is used for sending image data to the display device
  • the receiving unit is used for receiving the source image.
  • the state information further includes: stream state information used to characterize the video stream to which the source image belongs.
  • the status information further includes: resource running status information, and the resource running status information indicates the running status of the resource on the set-top box.
  • the resources can be hardware resources or software resources.
  • the super-resolution conversion module is specifically used for:
  • the state mapping rule determines the first target magnification to which the first state vector is mapped.
  • the super-resolution conversion module selects at least one super-resolution algorithm according to the first target magnification, and enlarges the source image to the first target magnification to obtain a second resolution
  • the target image is specifically used to: select the first super-resolution algorithm in the super-resolution algorithm set according to the first target magnification, and use the first super-resolution algorithm to convert the source image into the target image; wherein ,
  • the first super-resolution algorithm is a super-resolution algorithm with the largest magnification among super-resolution algorithms with a set magnification of less than or equal to the first target magnification; the set of super-resolution algorithms includes the first There are multiple super-resolution algorithms including one super-resolution algorithm.
  • One super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to different magnifications.
  • the super-resolution conversion module selects at least one super-resolution algorithm according to the first target magnification, and enlarges the source image at the first target magnification to obtain a target image with a second resolution
  • M is greater than 0
  • the magnification product of the super-resolution algorithm used in the M resolution conversion is equal to the first target magnification
  • the image obtained by the i-th resolution conversion operation is the first
  • the input of i+1 resolution conversion operations, i+1 is an integer less than or M and greater than 1.
  • the display state information includes at least one of the resolution of the display device, the transmission interface type of the display device, or the size of the display device.
  • the stream state information includes the frame rate of the video stream and/or parameters used to represent the format of the image stream.
  • the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.
  • the super-resolution conversion module is specifically configured to determine the first target magnification for the source image according to the state information and the first resolution.
  • the second resolution, and the first target magnification is determined according to the first resolution and the second resolution; wherein the second resolution is the resolution of the display device; or,
  • the second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device; or,
  • the second resolution is the minimum of the highest resolution supported by the size of the display device and the resolution of the display device; or,
  • the second resolution is the smallest value among the highest resolution supported by the size of the display device, the highest resolution supported by the interface type of the display device, and the resolution of the display device; or,
  • the second resolution is the highest resolution supported by the interface type of the display device under the frame rate of the video stream.
  • the second resolution is the minimum of the highest resolution supported by the interface type of the display device and the resolution of the display device at the frame rate of the video stream; or,
  • the second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the maximum resolution of the display device when the frame rate of the video stream is used.
  • the minimum value in resolution or,
  • the second resolution is the highest resolution supported by the interface type of the display device when the format of the image stream is adopted; or,
  • the second resolution is the minimum value of the highest resolution supported by the interface type of the display device and the resolution of the display device when the format of the image stream is adopted; or,
  • the second resolution is the highest resolution supported by the interface type of the display device, the highest resolution supported by the size of the display device, and the resolution of the display device when the format of the image stream is adopted.
  • the second resolution is the highest resolution supported by the interface type of the display device when the frame rate of the video stream and the format of the image stream are adopted.
  • the super-resolution conversion module is further configured to obtain the resource running status during the process of performing the operation of zooming in the first target magnification on the source image by using the at least one resolution conversion algorithm Information; when it is determined that the resource operating state information meets a preset condition, continue to perform an operation of magnifying the first target magnification on the source image according to the first target magnification.
  • the super-resolution conversion module is also used to reduce the first target magnification to obtain the second target magnification when it is determined that the resource operating state information does not meet the preset condition, and according to the reduced all
  • the second target magnification performs an enlargement operation on the source image to obtain a second target image with a third resolution, where the third resolution is obtained by enlarging the second target magnification by the first resolution.
  • an embodiment of the present application provides an image processing device, which includes a processor that invokes instructions in the memory to implement any of the method embodiments and method embodiments of the first or third aspect described above.
  • the functions involved in a possible design are included in a possible design.
  • the image processing apparatus may further include a transmission interface through which the processor receives or transmits image data.
  • the transmission interface may include a transmission interface and a reception interface, the transmission interface is used to transmit image data to the display device, and the reception interface is used to receive the source image.
  • the embodiments of the present application provide a computer program product, including a computer program.
  • the computer program When the computer program is executed on a computer or processor, it will enable the computer or processor to implement the first aspect or the third method described above.
  • the embodiments of the present application provide a computer-readable storage medium for storing programs and instructions.
  • the computer can execute the above-mentioned first or third aspects. The functions involved in the method embodiment and any possible design of the method embodiment are described.
  • the embodiments of the present application provide a chip system, which includes a processor and may also include a memory, for implementing the functions involved in the above method.
  • the chip system can be composed of chips, or can include chips and other discrete devices.
  • FIG. 1A is a schematic structural diagram of an image processing system 100 provided by an embodiment of this application.
  • FIG. 1B is a schematic structural diagram of another image processing system 100 provided by an embodiment of this application.
  • FIG. 2 is a schematic structural diagram of an image processing device 102 provided by an embodiment of the application.
  • FIG. 3 is a flowchart of an image processing method provided by an embodiment of the application.
  • FIG. 4 is a schematic diagram of a controlled super-resolution algorithm provided by an embodiment of the application.
  • FIG. 5 is a schematic structural diagram of another image processing apparatus 102 provided by an embodiment of this application.
  • FIG. 6 is a schematic diagram of an image processing method provided by an embodiment of the application.
  • FIG. 7 is a schematic diagram of status monitoring information provided by an embodiment of this application.
  • FIG. 8A is a schematic diagram of another image processing apparatus 102 provided by an embodiment of the application.
  • FIG. 8B is a schematic diagram of another image processing apparatus 102 provided by an embodiment of the application.
  • FIG. 8C is a schematic diagram of another image processing apparatus 102 provided by an embodiment of the application.
  • At least one refers to one, or more than one, that includes one, two, three and more; “multiple” refers to two, or more than two, that includes two, Three and more.
  • words such as “first” and “second” are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating Or imply the order.
  • “And/or” describes the association relationship of the associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, and B exists alone, where A, B can be singular or plural.
  • At least one item (a) or similar expressions refers to any combination of these items, including any combination of a single item (a) or plural items (a).
  • at least one item (a) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple .
  • Resolution is a parameter used to measure the amount of data in an image. It is usually expressed as ppi (Pixel per inch).
  • ppi Pixel per inch
  • the 320*180 of the video refers to the effective pixels in the horizontal and vertical directions. The window is small. The ppi value is higher and it looks clearer.
  • Super-resolution technology refers to the use of one or more low-resolution images to obtain a high-resolution image by means of hardware or software, thereby increasing the resolution of the original input image.
  • Super-resolution algorithms are divided into generalized super-resolution algorithms and narrowly-defined super-resolution algorithms.
  • the generalized super-resolution is an image resolution magnification algorithm, and the narrow one generally refers to an image quality enhancement method using machine learning methods.
  • the difference between the super-resolution algorithm and the traditional image magnification algorithm is that the traditional image magnification algorithm is manually designed, which is a fixed method and has a lower computational consumption; the super-resolution algorithm is based on deep learning or machine learning, and requires a training process and complex processing. , The calculation consumes a lot, and the performance requirements of the processor are higher.
  • Super-resolution algorithms include many types, such as BiCubic, and super-resolution algorithms based on deep learning, for example, Super-Resolution Convolutional Neural Network (SRCNN), Another example is the super-resolution algorithm (hisilicon super-resolution, HiSR) of the mobile terminal, and the super-resolution algorithm based on dictionary training.
  • SRCNN Super-Resolution Convolutional Neural Network
  • HiSR super-resolution Convolutional Neural Network
  • the embodiments of this application do not specifically limit the super-resolution algorithm, and all existing super-resolution algorithms are applicable to this application.
  • Magnification refers to the multiple by which the resolution is enlarged.
  • the input resolution name is 1080P
  • the foreign letter P means progressive scan
  • the output resolution name is 2K, which generally refers to a display device or content with a horizontal resolution of about 2000 pixels
  • the resolution ratio is 2. Enlarge the input resolution by 2 times to get the output resolution.
  • different resolution names may correspond to different resolutions.
  • a resolution name may correspond to only one resolution.
  • the same resolution name can correspond to different resolutions.
  • the resolution name is 2K
  • the common resolution corresponding to 2K is 2560 ⁇ 1440
  • magnification factor involved in the embodiments of the present application is not absolutely equal to the ratio of the output resolution to the input resolution, but is approximately equal to the ratio of the output resolution to the input resolution. For example, if the ratio of output resolution to input resolution is equal to 3.5, the resolution magnification is equal to the largest integer less than 3.5, which is 3 times.
  • the embodiments of the application provide an image processing method and device.
  • the resolution of the received image is adjusted according to the state control information of the image processing system, and the state control information of the image processing system in different application scenarios Different, therefore, based on the method of adjusting the resolution through the state control information provided by the embodiment of the present application, the resolution requirement in the business scenario can be met.
  • the image processing system 100 provided by the embodiment of the present application may include a signal source 101 and the image processing apparatus 102 provided by the embodiment of the present application.
  • the signal source 101 is the source of processing content of the image processing device 102, which may be a picture or a video frame.
  • the image source can be the network, mobile storage media, camera, camera equipment, etc.
  • the image processing device 102 is configured to process the image from the signal source 101 according to the image processing method provided in the embodiment of the present application.
  • the image processing device 102 as shown in FIG. 1A may have a display function, and the image processing system 100 provided by the embodiment of the present application may also display images that have undergone image processing.
  • the image is output to a display device.
  • the image processing device 102 may be a display device such as a television or a display with image processing functions.
  • the system 100 further includes a display device 103, where the display device 103 may be a device with a display function, such as a television, The display may also be a display screen, a projector, etc.
  • the display device 103 is used for receiving the image transmitted by the image processing device 102 and displaying the received image.
  • the image processing apparatus 102 here may be a playback device, such as a set-top box.
  • the image processing device 102 provided by the embodiment of the present application may have a structure as shown in FIG. 2. It can be seen that the image processing device 102 may include a processing unit 201, and the processing unit 201 may be used to implement the The steps involved in the image processing method, the specific image processing method provided in the embodiment of the present application will be described in detail later, and the description will not be repeated here.
  • the processing unit 201 may include one or more of the following: a central processing unit (CPU), or other general-purpose processors, image signal processors (image signal processors, ISP), microprocessors, and digital signal processing Digital signal processor (DSP), application specific integrated circuit (ASIC), graphics processing unit (GPU), neural-network processing unit (NPU), field programmable gate Array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the CPU may be a single-CPU processor or a multi-CPU processor; optionally, the CPU may be a processor group composed of multiple processors, between multiple processors Coupled to each other through one or more buses.
  • the image processing device 102 may further include a storage unit 202, which may be used to store computer program instructions, including an operating system (Operation System, OS), various user applications, and program codes for executing the solutions of the present application.
  • the memory can also be used to store video data, image data, etc.; the CPU can be used to execute the computer program code stored in the memory to implement the methods in the embodiments of the present application.
  • the storage unit 202 can be coupled with the processing unit 201 to support the processing unit 201 to call computer programs and instructions in the storage unit 202 to implement the steps involved in the image processing method provided in the embodiments of the present application.
  • the storage unit 202 can also use For storing data.
  • coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices.
  • the memory can be coupled through various interfaces, transmission lines or buses.
  • the memory may be a non-power-down volatile memory, such as an embedded multi-media card (EMMC), universal flash storage (UFS), or read-only memory, ROM), or other types of static storage devices that can store static information and instructions, or volatile memory (volatile memory) such as random access memory (RAM) or can store information and
  • volatile memory volatile memory (volatile memory) such as random access memory (RAM) or can store information and
  • RAM random access memory
  • Other types of dynamic storage devices for instructions can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical discs Storage, optical disc storage (including compact discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store program codes in the form
  • the image processing device 102 may further include a sending unit 203 and/or a receiving unit 204, where the sending unit 203 may be used to output processed images, and the receiving unit 204 may receive images from the signal source 101.
  • the sending unit 203 and/or the receiving unit 204 may be an image output interface, such as a high definition multimedia interface (HDMI), a video graphics array (VGA), and a digital video interface (digital video interface).
  • the image processing apparatus 102 may further include a display unit 205, such as a display screen or a display panel, for displaying the image processed by the processing unit 201.
  • a display unit 205 such as a display screen or a display panel, for displaying the image processed by the processing unit 201.
  • the image processing device 102 may further include a video decoding unit 206 for decoding an image from the signal source 101, so that the processing unit 201 processes the image decoded by the video decoding unit 206.
  • a video decoding unit 206 for decoding an image from the signal source 101, so that the processing unit 201 processes the image decoded by the video decoding unit 206.
  • the image processing device 102 may also include a dedicated video or graphics processor, a microprocessor, and a microcontroller MCU, etc.
  • the dedicated video/graphics processor may be a dedicated ISP.
  • the following describes an image processing method provided by an embodiment of the present application with reference to FIG. 3.
  • the method may be executed by an image processing device, for example, by the image processing device 102.
  • the method includes the following steps:
  • the image processing apparatus receives a source image, where the source image has a first resolution.
  • the image processing apparatus obtains status information.
  • the state information includes display state information used to characterize the display performance of the display device, and the display device and the image processing device are coupled through a transmission interface to transmit image data.
  • the display device may be located inside the image processing device or outside the image processing device.
  • the image processing apparatus may be applied to a set-top box, for example, it may be a processor or a processing chip disposed inside the set-top box.
  • the display device may be a television.
  • the order of receiving the source image and obtaining the display state information is not specifically limited in this application.
  • the display state information may be obtained before the receiving source image, or the display state information may be obtained after the source image is received, or after the source image is received.
  • the operation of acquiring the display status information is performed during the process of the source image, which is not specifically limited in this application.
  • the image processing apparatus determines a first target magnification for the source image according to the state information and the first resolution.
  • the target image obtained by enlarging the source image at the first target magnification can be adapted to the display performance of the display device.
  • the resolution of the target image obtained by enlarging the first target magnification is the second resolution.
  • the image processing device determines the first target magnification for the source image according to the status information and the first resolution, it may first determine the second resolution of the target image according to the status information, and then determine the second resolution of the target image according to the first resolution and the first resolution.
  • the second resolution determines the first target magnification.
  • S304 Select at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution.
  • the target image of the second resolution satisfies the requirement of the state information.
  • the target image with the second resolution is adapted to the display performance of the display device.
  • the information in the state may include: display state information, flow state information, or resource operation state information.
  • the display status information may include one or more of the resolution of the display device, the interface type of the display device, or the size of the display device.
  • the stream state information includes the frame rate of the video stream and/or parameters used to represent the format of the image stream.
  • the resource operating status information includes processing resource occupancy rate and/or storage resource occupancy rate.
  • the display status information includes the resolution of the display device, and in performing S302, the image processing device may determine the resolution of the display device as the second resolution.
  • the display status information includes the interface type of the display device, and in performing S302, the image processing device may determine the highest resolution supported by the interface type of the display device as the second resolution.
  • the interface types of the display device may include multiple types, such as VGA, DVI, HDMI, or DP.
  • the resolution name of the highest resolution supported by VGA is 1080P.
  • the resolution name of 1080P is 1920 ⁇ 1080.
  • 1920 ⁇ 1080 can be configured as the second resolution.
  • the resolution that meets a certain resolution name, such as 1920 ⁇ 1080, meets 1080P is called 1080P resolution.
  • DVI generally supports the resolution name of the highest resolution 1080P, and when the interface type of the display device is VGA, 1920 ⁇ 1080 can be configured as the second resolution.
  • HDMI includes multiple transmission protocol versions.
  • the highest resolution supported by HDMI 1.0, HDMI 1.1, and HDMI 1.2 are all 1600 ⁇ 1200, and the highest resolution supported by HDMI 1.3 is 2048 ⁇ 1536.
  • the name of the corresponding resolution is 2K
  • the highest resolution supported by HDMI 1.4 is 4096 ⁇ 2160
  • the corresponding resolution name can be 4K
  • the highest resolution supported by HDMI 2.0 is 4096 ⁇ 2160
  • the corresponding resolution name can be 4K.
  • the same resolution name can correspond to multiple resolutions, such as 2K resolution, which generally refers to a display device or content with a horizontal resolution of about 2000 pixels.
  • the common resolution corresponding to 2K is 2560 ⁇ 1440, and there are derivative resolutions, such as 2048 ⁇ 1536, 1998 ⁇ 1080, 2048 ⁇ 858, etc.
  • 4K the common resolution is 4096 ⁇ 2160, and there are derivative resolutions, such as 4096 ⁇ 3112, 3656 ⁇ 2664, and 3840 ⁇ 2160.
  • the display device can generally be configured with a specific resolution.
  • the display device may also be configured with only the resolution name, or the corresponding relationship between the resolution name and the resolution.
  • the transmission protocol version is determined to be effective.
  • the HDMI standard is backward compatible.
  • DP generally supports the highest resolution resolution name is 4K.
  • the image processing device may determine the highest resolution supported by the size of the display device as the second resolution. Therefore, the output image of the second resolution can meet the size requirement of the display device in the state vector. At this time, the second resolution image is adapted to the display device.
  • the image processing device may compare the highest resolution supported by the interface type of the display device with the interface type of the display device.
  • the minimum value among the resolutions is determined as the second resolution. Therefore, the output image of the second resolution can meet the requirements of the size of the display device and the resolution of the display device in the state vector. At this time, the second resolution image is adapted to the display device.
  • the interface type of the display device is HDMI 2.0
  • the highest resolution supported is 4096 ⁇ 2160
  • the resolution of the display device is 2048 ⁇ 1080
  • the image processing device compares the highest resolution supported by the size of the display device with the resolution of the display device. The minimum value of is determined as the second resolution.
  • the image processing device can set the highest resolution supported by the size of the display device The minimum value among the highest resolution supported by the interface type of the display device and the resolution of the display device is determined as the second resolution.
  • the target resolution (second resolution) based on the display status information of the display device (such as the resolution, size, or interface type of the display device)
  • select the resolution that the display device can support as the target resolution such as When the resolution is higher, the output image with higher resolution is output, which improves the user experience.
  • the resolution of the display device is lower, the output resolution is limited, and the performance requirements and power consumption of super-resolution algorithm execution are reduced.
  • the resolution of the display device is high and the size and interface type are sufficient to support the resolution, then the output image with the higher resolution will be output.
  • the resolution of the display device is low, but the size and interface type are not enough to support the resolution. At the resolution, the output resolution is limited, and the performance requirements and power consumption of the super-resolution algorithm are reduced.
  • the set-top box selects at least one super-resolution conversion algorithm according to the first target magnification to enlarge the source image at the first target magnification to obtain a target image with a second resolution.
  • the embodiments of the present application may also be suitable for reducing the resolution of an image, that is, the second resolution may be lower than or equal to the first resolution.
  • the embodiment of the present application may be configured with a super-resolution conversion algorithm with a magnification ratio of less than 1, equal to 1, and greater than 1.
  • the first resolution is enlarged as an example, that is, the resolution of the input image is enlarged as an example.
  • F(X) AX+B.
  • the target magnification is taken as an example.
  • the image processing apparatus when it determines the first target magnification according to the state information and the first resolution, it may first construct a state vector based on the state information and the first resolution, and the state vector includes parameter values respectively used to indicate that the state information includes And the state quantity of the first resolution; and then the first target magnification mapped by the state vector is determined according to the state mapping rule.
  • the status information includes the resolution, size, and interface type of the display device.
  • the state vector can include 4 state quantities.
  • the controlled super-resolution conversion method provided in this embodiment of the application will have the first resolution.
  • the source image of the resolution is enlarged at the first target magnification to obtain the target image of the second resolution.
  • the controlled super-resolution conversion method is a super-resolution conversion method with a controllable output resolution.
  • the target magnification is used as the basis for selecting the super-resolution algorithm and the number of resolution conversions.
  • the super-resolution algorithm of the target magnification is selected to perform a resolution conversion.
  • a super-resolution algorithm can be selected.
  • the resolution algorithm performs multiple resolution conversions, and the number of executions can be determined according to the target magnification.
  • Another example is to select a super-resolution combination with a combined magnification that reaches the target magnification among the multiple super-resolution algorithm combinations configured.
  • the controlled super-resolution conversion method is exemplarily described as follows.
  • the first method uses a super-resolution algorithm set, which includes super-resolution algorithms with multiple magnifications.
  • the super-resolution algorithm includes multiple super-resolution algorithms, one super-resolution algorithm corresponds to one magnification, and different super-resolution algorithms correspond to different magnifications.
  • the super-resolution algorithm set includes ⁇ 2 super-resolution algorithm, ⁇ 3 super-resolution algorithm, and ⁇ 4 super-resolution algorithm.
  • the magnification may also be a non-integer, such as ⁇ 2.25 super-resolution algorithm, ⁇ 3.5 super-resolution algorithm, and ⁇ 4.25 super-resolution algorithm.
  • the image processing apparatus converts the received source image with the first resolution into the target image with the second resolution, it can be implemented in the following manner:
  • the image processing device selects the first super-resolution algorithm in the super-resolution algorithm set according to the target magnification, and uses the first super-resolution algorithm to convert the source image with the first resolution into the target image with the second resolution.
  • the super-resolution algorithm is the super-resolution algorithm with the largest magnification among the super-resolution algorithms whose magnification is less than or equal to the target magnification; among them, the super-resolution algorithm focuses on multiple types including the first super-resolution algorithm Super-resolution algorithm, a super-resolution algorithm corresponds to one magnification, and different kinds of super-resolution algorithms correspond to different magnifications.
  • the image processing device may compare the second resolution with the first resolution when determining the target magnification according to the first resolution and the second resolution.
  • the ratio of a resolution is determined as the target magnification.
  • the magnifications corresponding to the super-resolution algorithms included in the super-resolution algorithm set may also be integers.
  • the ratio of the second resolution to the first resolution may be rounded down to determine the target magnification. For example, if the ratio of the second resolution to the first resolution is 3.5, the rounded down value is 3, and the target magnification is 3.
  • the first super-resolution algorithm is selected in the super-resolution algorithm set, the first super-resolution algorithm corresponding to the target magnification can be selected.
  • the ratio of the second resolution to the first resolution is an integer, in this case, the ratio is the target magnification.
  • the second method is a multi-level super-resolution conversion method.
  • This mode is a controllable conversion mode at the termination time.
  • the magnification can be the same or different each time.
  • the image processing device converts the source image with the first resolution into the target image with the second resolution, it can be implemented in the following ways:
  • the magnification product of the super-resolution algorithm used in M resolution conversion is equal to the target magnification; the image obtained by the i-th resolution conversion operation is the i+1
  • the input of the sub-resolution conversion operation, i+1 is an integer less than or equal to M and greater than 1.
  • the super-resolution algorithm used may be different or the same.
  • the same super-resolution algorithm is executed cyclically, that is, when the super-resolution algorithm is executed multiple times, the magnification is the same each time.
  • N the image A super-resolution algorithm
  • M the multi-level super-resolution conversion method can support the target magnification of ⁇ 2, ⁇ 4, ⁇ 8, ... ,As shown in Figure 4.
  • the target condition may be a target resolution or a target magnification or a target number of times.
  • the ratio of the target magnification to the magnification of the super-resolution algorithm is the target times.
  • the image processing device uses a super-resolution algorithm to perform resolution conversion on the source image with the first resolution as the input image, and counts the number of resolution conversion operations, and determines whether the counting result reaches the target number of times, if the counting result is less than the target.
  • the resolution conversion is performed using the super-resolution algorithm as the input image until the count result reaches the target number of times. When the count result reaches the target number of times, the image obtained by the resolution conversion is The target image.
  • the target resolution is 1280 ⁇ 720
  • the second resolution is 2560 ⁇ 1440
  • the magnification of the first super-resolution algorithm adopted is ⁇ 2
  • the first resolution image is converted using the first super-resolution algorithm
  • the resolution obtained does not meet the second resolution
  • the first super-resolution algorithm is used to perform a resolution conversion to obtain the resolution It is 2560 ⁇ 1440, which meets the second resolution, and stops execution.
  • the image obtained by the second resolution conversion is the target image with the second resolution.
  • each super-resolution algorithm corresponds to a magnification
  • different super-resolution algorithms have different magnifications.
  • the magnification may be different or the same.
  • the target magnification can be used as a condition, and the combined super-resolution algorithm can be selected from multiple super-resolution algorithms.
  • the combined super-resolution algorithm can include one-rate super-resolution algorithm or multiple-rate super-resolution Rate algorithm.
  • the magnification of the combined super-resolution algorithm is equal to the target magnification.
  • the magnification of the combined super-resolution algorithm is the product of the magnifications of each super-resolution algorithm included in the combination.
  • the multiple super-resolution algorithms include ⁇ 2 super-resolution algorithms and ⁇ 3 super-resolution algorithms.
  • the combined super-resolution algorithm includes two ⁇ 2 super-resolution algorithms, and the two ⁇ 2 super-resolution algorithms are the same.
  • the target magnification is ⁇ 6
  • the combined super-resolution algorithm includes ⁇ 2 super-resolution algorithm and ⁇ 3 super-resolution algorithm.
  • the target magnification is ⁇ 9
  • the combined super-resolution algorithm includes 3 ⁇ 3 super-resolution algorithms, and these 3 ⁇ 3 super-resolution algorithms are the same.
  • conditions such as the difference between the magnification of the super-resolution algorithm and the target magnification can be used as the cost to pass the cost function, and then the optimal combination can be searched.
  • a combination strategy table corresponding to different magnifications may be configured in advance, for example, see Table 1.
  • the second resolution when determining the second resolution (target resolution), it can be determined not only based on the display state of the display device, but also based on the stream state of the video stream, that is, in the state information It can also include stream status information of the video stream.
  • the image processing device can also acquire stream status information while acquiring display status information.
  • the stream status information is used to characterize the stream status of the video stream to which the source image with the first resolution belongs; for example, the image processing apparatus may first determine the second resolution of the target image according to the stream status information and the display status information, and then The target magnification is determined according to the first resolution and the second resolution.
  • the target image of the second resolution is adapted to the streaming state of the video stream and the display performance of the display device.
  • the stream state information includes the frame rate of the video stream and/or parameters used to indicate the format of the video stream.
  • the resolution supported by the frame rate of different video streams may be different. Different video stream formats may support different resolutions.
  • HDMI HDMI 1.3 supports 1080P/120FPS (frames per second)
  • HDMI 1.4 supports 4K/30FPS or 2K/60FPS
  • HDMI 2.0 supports 4K/60FPS.
  • the display status information includes the interface type of the display device
  • the stream status information includes the frame rate of the video stream.
  • the image processing device determines the second resolution according to the stream status information and the display status information, it may determine the highest resolution supported by the interface type of the display device at the frame rate of the video stream as the second resolution.
  • the second resolution is determined to be 4K resolution
  • the second resolution is determined to be 2K resolution
  • the second resolution is determined It is 1080P resolution.
  • the second resolution is determined to be 4K resolution, and for 120FPS stream, the second resolution is determined to be 2K resolution;
  • another example is display
  • the interface of the device is DP
  • the second resolution is determined to be 4K resolution
  • the second resolution is determined to be 2K resolution.
  • the code stream frame rate or code stream resolution can match the available resources of the image processing device, a certain multiple of super-resolution algorithm is implemented to increase the output resolution and improve user experience; when the code stream frame rate is too high, image processing The resources may not be enough to support the requirements of the super-resolution algorithm.
  • the target resolution can be reduced, the target magnification can be reduced, the demand for image processing resources can be reduced, and the occurrence of jams and other adverse effects can be prevented.
  • the display status information includes the interface type of the display device and the resolution of the display device
  • the stream status information includes the frame rate of the video stream.
  • the resolution of the display device is 4K resolution
  • the first resolution is 720P resolution
  • the interface of the display device is HDMI 1.4
  • the frame rate of the video stream is 60FPS
  • pre-set memory/processing resource occupancy alert determination conditions may be required, for example, the memory occupancy rate is not more than 80%, the processor occupancy is not more than 80%, or the weighted average of the two is not more than 75 %. Based on this, the embodiment of the present application can determine whether to reduce the target magnification according to the occupancy of storage resources and/or processing resources of the image processing apparatus.
  • the state information used may include one or more parameters in the display state information and the stream state information.
  • the maximum resolution supported by different parameters is different, so that the state information is
  • the second resolution the smallest resolution among the resolutions supported by the multiple parameters is taken as the second resolution.
  • the status information includes two parameters, which are the resolution of the display device and the transmission interface type, and the second resolution is the minimum value of the resolution of the display device and the resolution supported by the transmission interface type.
  • the status information includes the type and frame rate of the transmission interface. At different frame rates, the type of transmission interface supports different resolutions, so the resolution supported by the transmission interface at this frame rate can be used as the second resolution. .
  • the constructed state vector when the state information further includes flow state information, also includes a state quantity used to indicate the parameters included in the flow state information.
  • the status information includes the resolution, size, interface type, frame rate, and image stream format of the display device.
  • the state vector can include 6 state quantities.
  • the first state vector is used as the input of F(X) to obtain the first target resolution. It should be noted that the order of the state variables corresponding to the parameters in the state vector is pre-configured according to the needs of F(X).
  • the image processing device acquires the resource operating status information of the image processing device during the process of performing the resolution conversion operation on the source image with the first resolution according to the target magnification; the resource operating status information is used to indicate the image processing device The running state of the resource; when the image processing device determines that the resource running state information meets the preset condition, it continues to perform the resolution conversion operation on the source image with the first resolution according to the target magnification.
  • the image processing apparatus when determining that the resource operating state information does not meet the preset condition, reduces the target magnification to obtain the second target magnification, and performs a resolution conversion operation on the source image with the first resolution according to the second target magnification, Obtain a second target image with a third resolution at the target magnification after the resolution is enlarged and reduced, and the third resolution is the resolution obtained by enlarging the second target magnification by the first resolution.
  • the resources on the image processing apparatus may include hardware resources and/or software resources.
  • hardware resources such as processor resources and memory resources, are taken as examples.
  • the controlled super-resolution conversion mode is the first mode
  • the second resolution is determined according to the display status of the display device and/or the stream status of the video stream
  • the second resolution is determined according to the first resolution and the second resolution.
  • the super-resolution algorithm corresponding to the target magnification can be tested for a preset time.
  • the target magnification is reduced, and the reduction is selected in the super-resolution algorithm set
  • the third super-resolution algorithm corresponding to the target magnification of uses the third super-resolution algorithm to convert the source image with the first resolution into the second target image with the third resolution. For example, before the reduction, the target magnification is ⁇ 4, and the target magnification after the reduction may be ⁇ 3.
  • the controlled super-resolution conversion method is the second method
  • the algorithm is executed for the preset time, and when the resource running status information does not meet the preset conditions, the number of cycles to execute the same super-resolution algorithm can be reduced, that is, the target magnification is reduced, and the same super-resolution cycle is executed according to the reduced number of times
  • the rate algorithm converts the source image with the first resolution into the second target image with the third resolution.
  • the combined super-resolution algorithm can be executed for a preset time, and when the resource operating status information is monitored that does not meet the preset conditions, the target magnification is reduced, and the combined super-resolution algorithm is re-determined according to the reduced target magnification , And use the newly determined combined super-resolution algorithm to perform resolution conversion operations.
  • the output resolution can be reduced to reduce the demand for device resources and prevent adverse effects such as jams; Or when the memory resources meet the demand, the output resolution can be increased to enhance the user experience.
  • an embodiment of the present application provides an image processing device, which has the function of implementing the image processing method provided by any of the foregoing method embodiments.
  • This function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • An image processing device 102 provided by an embodiment of the present application may have a structure as shown in FIG. 2, wherein the processing unit 201 may be configured to execute steps S301, S302, and S303 shown in the method-side embodiment of the present application.
  • the image processing device may include a state monitoring module 501 and a super-resolution conversion module 502.
  • the state monitoring module 501 may be used to perform the steps described in the method S301 of the embodiment of the present application
  • the controlled super-resolution module 502 may be used to perform the steps described in the method S302 of the embodiment of the present application.
  • the status monitoring module 501 in the image processing device 102 can be used to monitor status control information.
  • the status control information (may be referred to as status information for short) can include the display status information of the display device. If the image to be processed belongs to the frame image in the video stream, the status control information can also include the stream status information and/or resource operation of the video stream. status information.
  • the controlled super-resolution module 502 converts the source image with the first resolution into the target image with the second resolution according to the state control information.
  • the controlled super-resolution module 502 may include a state control module 502A and a resolution conversion module 502B.
  • the state control module 502A is used to determine the target magnification according to the state control information.
  • the state control module 502A determines the second resolution according to the state control information, and determines the target magnification according to the second resolution.
  • the resolution conversion module 502B is configured to perform a resolution conversion operation on the source image with the first resolution according to the target magnification to obtain the target image with the second resolution with the resolution magnification target magnification.
  • the image processing device 102 may further include a preprocessing module 503, which is used to perform image preprocessing on the image from the signal source, and input the image preprocessed into the controlled super-resolution module 502 .
  • a preprocessing module 503 which is used to perform image preprocessing on the image from the signal source, and input the image preprocessed into the controlled super-resolution module 502 .
  • the main purpose of image preprocessing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of related information and simplify data to the greatest extent.
  • image preprocessing can include operations such as smoothing and filtering.
  • the image processing device 102 may further include a post-processing module 504 configured to perform post-processing operations on the image output by the controlled super-resolution module 502, such as noise reduction and image enhancement.
  • the post-processing module 504 inputs the image after the post-processing operation to the display device.
  • the image pre-processing operation and post-processing operation in the embodiment of the present application can also be performed by one module, that is, the pre-processing module 503 and the post-processing module 504 can be combined into one module.
  • the functions of the state monitoring module 501, the super-resolution conversion module 502, the pre-processing module 503, and the post-processing module 504 may be implemented by the processing unit 201.
  • the state monitoring module 501, the super-resolution conversion module 502, the pre-processing module 503, and the post-processing module 504 may correspond to the processing unit 201.
  • the image of the signal source is input to the image processing device 102, it is first processed by the preprocessing module 503.
  • the state control module 502A receives The state monitoring module 501 inputs the state control vector (hereinafter referred to as the state vector), and completes the conversion mapping of the state vector to the super-resolution algorithm (referred to as the super-resolution algorithm) control quantity (may be referred to as the super-resolution control quantity); resolution conversion module 502B
  • the super-division control amount generated by the state control module 502A is controlled; the post-processing module 504 performs related image post-processing on the high-resolution image output by the super-resolution algorithm operating.
  • the over-resolution control amount hereinafter referred to as the state vector
  • FIG. 7 shows a selection scheme of the state control information acquired by the state monitoring module 501, the resource operation state information (such as processor occupancy rate, memory occupancy rate, etc.) and input source (I.e. video stream) stream status information (such as bit stream frame rate, bit stream resolution, bit stream format, etc.) and display status information of the display device (such as display device resolution, display device size, display device interface type Etc.) as the basis for super-resolution algorithm regulation.
  • the resource operation state information such as processor occupancy rate, memory occupancy rate, etc.
  • input source I.e. video stream
  • stream status information such as bit stream frame rate, bit stream resolution, bit stream format, etc.
  • display status information of the display device such as display device resolution, display device size, display device interface type Etc.
  • the super-resolution module 502 is controlled based on the state vector X and the input image (image after preprocessing) control, when performing the resolution conversion operation on the input image, the input is set low resolution image P in, P out is the output high-resolution image, X is input from the control state vector, F (X) is a state mapping function, S (P, F) as a function of the controlled super-resolution conversion corresponding manner, the input image P in low resolution and a state control amount X, the output is a high-resolution image P out , then there is
  • determining which super-resolution algorithm or several super-resolution algorithms, or how many super-resolution conversion operations are performed can be determined according to the target magnification output by F(X).
  • F(X) the state mapping function
  • S(P, F) the execution operation corresponding to the S(P, F) function
  • the controlled super-resolution conversion method adopts the first method, that is, the super-resolution algorithm set.
  • the resolution conversion module 502B can directly select which super-resolution algorithm to execute according to the output of the state control module 502A.
  • the controlled super-resolution conversion method adopts the second method, that is, the multi-level super-resolution conversion method.
  • the resolution conversion module 502B can directly determine to perform the resolution based on the output of the state control module 502A. The number of conversion operations.
  • the resolution conversion module 502B can directly select the combined super-resolution algorithm based on the output of the state control module 502A, use the target magnification as the termination condition, and update F( X), judge whether the converted magnification reaches the target magnification.
  • A1 Get the resolution Fps and frame rate Rs of the input code stream
  • the maximum output resolution Rif is determined to be 1080P; when the display device interface is DVI, the maximum output resolution Rif is determined to be 1080P; when the display device interface is HDMI1.3, the maximum output resolution is determined Resolution Rif is 1080P; for HDMI1.4, for 30FPS stream, determine the maximum output resolution Rif is 4K, for 60FPS stream, determine the maximum output resolution Rif is 2K, for 120FPS stream, determine the maximum output resolution Rif is 1080P; when it is HDMI2.0, for 60FPS (or less than 60FPS) stream, determine the maximum output resolution Rif is 4K, for 120FPS stream, determine the maximum output resolution Rif is 2K; when the interface of the display device is DP For a code stream of 60FPS (or less than 60FPS), the maximum output resolution Rif is determined to be 4K, and for a code stream of 120FPS, the maximum output resolution Rif is determined to be 2K.
  • Sraw is a non-integer
  • A4 further determining whether the target magnification needs to be updated based on the memory and/or processor occupation resource alert determination condition.
  • Step 1 Based on A3 to obtain the target magnification Sr, the image processing device 102 may try to run the super-resolution algorithm for a preset duration (for example, 1 s) without outputting to the display device.
  • a preset duration for example, 1 s
  • Step 2 Detect the memory/processor resource occupancy rate of the super-resolution algorithm implementing the target magnification Sr.
  • Step 3 After monitoring the implementation of the target magnification Sr over-dividing algorithm, determine whether to trigger the memory/processor occupation alert.
  • the size of the memory occupancy rate is used as the condition to trigger the occupancy alert. If the condition 1) is not met, the occupancy alert is triggered. For another example, when the processor occupancy rate is used as the condition for triggering the occupation alert, if the condition 2) is not met, the occupation alert is triggered. Another example is to use the processor occupancy rate and memory size as the conditions for triggering the occupancy alert. One way is to trigger the occupancy alert if conditions 1) and 2) are not met; the other way is if condition 3 is not met ), the occupation alert is triggered.
  • Step 4 If the memory/processor occupancy alert is not triggered, the current input target Sr is used as the output Sout; if the memory/processor occupancy alert is triggered, the target multiplier is reduced by one level to obtain Sr'.
  • execute step 1 test run the super-resolution algorithm of Sr' for 1 second, query the memory/processor resources required to implement the super-resolution algorithm under Sr', and query whether the memory occupation alert will be triggered. Based on step 4, the final output magnification Sout is obtained.
  • A5 Sout run using super resolution algorithm, using an input video image P in S (P in, Sout) output high-resolution video image Pout.
  • the memory/processor occupancy resources may be sampled once at a certain interval to determine whether to trigger the memory/processor occupancy alert. If it is triggered, similar to the step 4, reduce the magnification by one level, and update Sout to perform the resolution conversion operation using the updated Sout.
  • the state monitoring module 501 and the controlled super-resolution module 502 in the image processing device 102 shown in FIG. 5 can be composed of a central processing unit, a general-purpose processor, a digital signal processor, an application specific integrated circuit, and field programmable A gate array or other programmable logic devices, transistor logic devices, hardware components or any combination thereof can be implemented, which can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the embodiments of the present application.
  • the processor may also be a combination that implements computing functions, for example, a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on.
  • the storage module that the image processing apparatus 102 may include may be a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory.
  • FIG. 8A another possible structure of the image processing apparatus 102 in the embodiment of the present application includes a main processor 801, a memory 802, and a video processor 803.
  • the main processor 801 may be used to support the image processing device 102 to implement monitoring state control information and perform resolution conversion operations.
  • the main processor 801 can be used to monitor status control information, such as the method shown in step S301 that can be executed by the main processor 801, and the main processor 801 can also be used to perform target resolution determination according to the status control information. Adjust the resolution of the input image, for example, the method shown in steps S302 and S303 that can be executed by the main processor 801.
  • the main processor 801 can also perform functions other than the image processing related functions provided by the embodiments of the present application, such as receiving image signals from a signal source, decoding the image signals, and sending the decoded image signals to the video processor 803 .
  • the video processor 803 may be used to support the image processing device 102 to implement related functions of video signal processing.
  • the video processor 803 may be used to perform image preprocessing and postprocessing operations.
  • the main processor 801 may be used to implement related functions of the state monitoring module 501 and the controlled super-resolution module 502 in FIG. 5.
  • the video processor 803 may be used to implement the related functions of the preprocessing module 503 and the postprocessing module 504 in FIG. 5, as shown in 8B.
  • the memory 802 is used to support the main processor 801 and the video processor 803 to call computer programs and instructions in the memory 802 to implement the steps involved in the video processing method provided by the embodiments of the present application.
  • the memory 802 is also used to store data, such as Used to store super-resolution algorithms and related configuration parameters.
  • the memory 802 may include both volatile and non-volatile memory, such as memory and hard disk.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electronic Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • the volatile memory may be random access memory (RAM), which is used as an external cache.
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • DRAM synchronous dynamic random access memory
  • SDRAM double data rate synchronous dynamic random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory
  • direct rambus RAM direct rambus RAM
  • the main processor 801 may be used to support the image processing device 102 to implement monitoring state control information, and the resolution conversion operation is performed by the video processor 803.
  • the main processor 801 can be used to monitor state control information, such as the method shown in step S301 that the main processor 801 can execute.
  • the main controller 801 can also perform functions other than the image processing related functions provided by the embodiments of the present application, such as receiving image signals from a signal source, decoding the image signals, and sending the decoded image signals to the video processor 803 .
  • the video processor 803 can be used to support the image processing device 102 to implement related functions of video signal processing.
  • the video processor 803 can be used to determine the target resolution according to the state control information, and adjust the resolution of the input image based on the target resolution, such as video
  • the processor 803 may execute the methods shown in steps S302 and S303.
  • the video processor 803 can also be used to perform image pre-processing and post-processing operations.
  • the main processor 801 may be used to implement related functions of the state monitoring module 501 in FIG. 5.
  • the video processor 803 can be used to implement the related functions of the controlled super-resolution module 502, the pre-processing module 503, and the post-processing module 504 in FIG. 5, as shown in 8C.
  • acquiring the processing resource occupancy rate when the image processing apparatus 102 runs the super-resolution algorithm refers to the processor occupancy rate of the video processor 803.
  • the image processing device 102 shown in FIG. 8A and FIG. 8B only exemplarily embodies the structure required by the image processing device 102 to execute the above-mentioned image processing method involved in the embodiment of the application, and the embodiment of the application does not exclude the image
  • the processing device 102 also has other structures.
  • the image processing device 102 may also include a display device for displaying high-resolution images output by the video processor 1203; for another example, the image processing device 102 may also include necessary interfaces to implement Image input and output of processed images.
  • the main processor 801 and the video processor 803 may be central processing units, general-purpose processors, digital signal processors, application specific integrated circuits, field programmable gate arrays or other programmable logic devices, transistor logic devices, hardware
  • the components or any combination thereof can implement or execute various exemplary logical blocks, modules, and circuits described in conjunction with the disclosure of the embodiments of the present application.
  • the processor may also be a combination that implements computing functions, for example, a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on.
  • all the functions of the video processor 803 can also be implemented through software using the main processor 801.
  • the image processing apparatus 102 provided in the embodiments of the present application can be applied to smart devices such as set-top boxes, televisions, mobile phones, and other display devices with resolution conversion processing functions, and image processing devices to support the above devices to realize the application.
  • the image processing method provided by the embodiment provided by the embodiment.
  • the storage resources on which the function of the controlled super-resolution module 502 depends can be distinguished from the storage resources on which the functions of other modules depend.
  • Programs and instructions for implementing the functions of the controlled super-resolution module 502 may be configured in a dedicated memory, such as dedicated memory resources and dedicated storage resources.
  • the super-resolution algorithm and configuration parameters are stored in a dedicated memory resource, and the operation of the super-resolution algorithm depends on a dedicated memory resource.
  • the occupancy rate of dedicated memory resources when acquiring the memory occupancy rate when the image processing device runs the super-resolution algorithm, what is acquired is the occupancy rate of dedicated memory resources.
  • the embodiments of the present application provide a computer program product, including a computer program.
  • the computer program When the computer program is executed on a computer, the computer will enable the computer to implement any of the above-mentioned image processing method embodiments. Function.
  • the embodiments of the present application provide a computer program, which, when executed on a computer, will enable the computer to implement the functions involved in any of the foregoing image processing method embodiments.
  • the embodiments of the present application provide a computer-readable storage medium for storing programs and instructions.
  • the computer can execute any of the above-mentioned image processing method embodiments. The functions involved in.
  • the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • a computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

La présente invention concerne un procédé et un appareil de traitement d'image. Pour une image reçue, la résolution de l'image reçue est ajustée en fonction d'informations de commande d'état d'un système de traitement d'image ; dans différents scénarios d'application, les informations de commande d'état du système de traitement d'image sont différentes. Par conséquent, le mode d'ajustement de la résolution en fonction d'informations de commande d'état fourni par les modes de réalisation de la présente invention permet de satisfaire les exigences de résolution dans des scénarios de service.
PCT/CN2019/085370 2019-04-30 2019-04-30 Procédé et appareil de traitement d'image WO2020220346A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201980095791.1A CN113728354A (zh) 2019-04-30 2019-04-30 一种图像处理方法及装置
PCT/CN2019/085370 WO2020220346A1 (fr) 2019-04-30 2019-04-30 Procédé et appareil de traitement d'image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/085370 WO2020220346A1 (fr) 2019-04-30 2019-04-30 Procédé et appareil de traitement d'image

Publications (1)

Publication Number Publication Date
WO2020220346A1 true WO2020220346A1 (fr) 2020-11-05

Family

ID=73028752

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/085370 WO2020220346A1 (fr) 2019-04-30 2019-04-30 Procédé et appareil de traitement d'image

Country Status (2)

Country Link
CN (1) CN113728354A (fr)
WO (1) WO2020220346A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112492233A (zh) * 2020-11-27 2021-03-12 西安万像电子科技有限公司 图像传输系统收发两端分辨率的适配方法及装置
CN114025106A (zh) * 2021-12-16 2022-02-08 海宁奕斯伟集成电路设计有限公司 智能处理器、图像智能处理系统、方法及存储介质
CN116055655A (zh) * 2022-05-30 2023-05-02 荣耀终端有限公司 一种画幅尺寸的推荐方法及电子设备
CN116709016A (zh) * 2022-02-24 2023-09-05 荣耀终端有限公司 倍率切换方法和倍率切换装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101227554A (zh) * 2008-02-04 2008-07-23 中兴通讯股份有限公司 一种实现机顶盒在电视上自适应显示的方法及系统
US20100026695A1 (en) * 2008-08-04 2010-02-04 Kabushiki Kaisha Toshiba Image Processing Apparatus and Image Processing Method
US20100026685A1 (en) * 2008-08-04 2010-02-04 Kabushiki Kaisha Toshiba Image Processing Apparatus
CN103985085A (zh) * 2014-05-26 2014-08-13 三星电子（中国）研发中心 图像超分辨率放大的方法和装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101227554A (zh) * 2008-02-04 2008-07-23 中兴通讯股份有限公司 一种实现机顶盒在电视上自适应显示的方法及系统
US20100026695A1 (en) * 2008-08-04 2010-02-04 Kabushiki Kaisha Toshiba Image Processing Apparatus and Image Processing Method
US20100026685A1 (en) * 2008-08-04 2010-02-04 Kabushiki Kaisha Toshiba Image Processing Apparatus
CN103985085A (zh) * 2014-05-26 2014-08-13 三星电子（中国）研发中心 图像超分辨率放大的方法和装置

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112492233A (zh) * 2020-11-27 2021-03-12 西安万像电子科技有限公司 图像传输系统收发两端分辨率的适配方法及装置
CN114025106A (zh) * 2021-12-16 2022-02-08 海宁奕斯伟集成电路设计有限公司 智能处理器、图像智能处理系统、方法及存储介质
CN116709016A (zh) * 2022-02-24 2023-09-05 荣耀终端有限公司 倍率切换方法和倍率切换装置
CN116055655A (zh) * 2022-05-30 2023-05-02 荣耀终端有限公司 一种画幅尺寸的推荐方法及电子设备
CN116055655B (zh) * 2022-05-30 2023-09-26 荣耀终端有限公司 一种画幅尺寸的推荐方法及电子设备

Also Published As

Publication number Publication date
CN113728354A (zh) 2021-11-30

Similar Documents

Publication Publication Date Title
WO2020220346A1 (fr) Procédé et appareil de traitement d'image
JP6978542B2 (ja) 電子装置及びその制御方法
US7945120B2 (en) Apparatus for enhancing resolution using edge detection
US20200267396A1 (en) Human visual system adaptive video coding
WO2023000621A1 (fr) Procédé et appareil d'affichage à écrans multiples
EP4036841A1 (fr) Procédé et appareil de traitement d'image
WO2020259123A1 (fr) Procédé et dispositif de réglage de qualité d'image ainsi que support d'informations lisible
EP3828812B1 (fr) Appareil électronique et son procédé de commande
CN109587561B (zh) 视频处理方法、装置、电子设备及存储介质
US11893712B2 (en) Electronic device and control method thereof
US20150235621A1 (en) Image Processing Method and Device
WO2019095485A1 (fr) Procédé et appareil de traitement de signal vidéo et support d'informations lisible
CN111405362B (zh) 视频输出方法、装置、视频设备及计算机可读存储介质
WO2022261849A1 (fr) Procédé et système de sélection automatique d'algorithme de traitement d'image dépendant du contenu
US11436442B2 (en) Electronic apparatus and control method thereof
KR20210108027A (ko) 전자 장치 및 그 제어 방법
EP2475172A2 (fr) Appareil de traitement vidéo capable de contrôler dynamiquement une qualité d'image traitée d'après l'utilisation de bande passante de mémoire d'exécution
WO2019087984A1 (fr) Dispositif de traitement d'image, dispositif d'affichage, procédé de traitement d'image, programme de commande et support d'enregistrement
US11962859B2 (en) System and method for implementation of region of interest based streaming
US11392521B2 (en) Heterogeneous computing system and heterogeneous computing method using the system
US7982752B2 (en) Multi-format image display apparatus and method
CN112911186B (zh) 图像存储方法及装置、电子设备、存储介质
TWI597688B (zh) 圖層處理方法及裝置
CN112312045A (zh) 视频墙桌面图像分割处理方法、装置及存储介质、系统
US10846829B1 (en) Image sharpening with edge direction based undershoot and overshoot reduction

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19927128

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19927128

Country of ref document: EP

Kind code of ref document: A1