CN116847088B

CN116847088B - Image processing method, processing apparatus, and storage medium

Info

Publication number: CN116847088B
Application number: CN202311071925.0A
Authority: CN
Inventors: 刘雨田; 霍永凯
Original assignee: Shenzhen Transsion Holdings Co Ltd
Current assignee: Shenzhen Transsion Holdings Co Ltd
Priority date: 2023-08-24
Filing date: 2023-08-24
Publication date: 2024-04-05
Anticipated expiration: 2043-08-24
Also published as: CN117979010A; CN116847088A

Abstract

The application provides an image processing method, processing equipment and a storage medium, wherein the image processing method comprises the following steps: and determining or obtaining a prediction result of the first component and/or the second component of the pixel to be predicted according to the reference pixel of the first component. Through the technical scheme, the residual error value of the residual error block corresponding to the prediction block can be reduced, so that the compression efficiency in the image encoding and decoding process is improved.

Description

Image processing method, processing apparatus, and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, processing device, and storage medium.

Background

In the development of image codec technology, improvements made by various codec standards are being made in an effort to improve the codec effect of images from different aspects. And predicting the color components of an image during image encoding and decoding is also a hot problem in current research.

In the process of designing and implementing the present application, the inventors found that at least the following problems exist: in some implementations, the prediction of the color component (for example, the luminance component or the chrominance component) is performed on the current pixel to be predicted according to the pixel outside the block to be predicted where the pixel is located, so that the residual value of the finally obtained prediction block and the residual block corresponding to the prediction block is larger, thereby reducing the compression efficiency in the image encoding and decoding process.

The foregoing description is provided for general background information and does not necessarily constitute prior art.

Disclosure of Invention

Aiming at the technical problems, the application provides an image processing method, processing equipment and storage medium, which can reduce the residual value of a residual block corresponding to a predicted block, thereby improving the compression efficiency in the image encoding and decoding process.

The application provides an image processing method, which can be applied to processing equipment (such as an intelligent terminal or a server), and comprises the following steps:

and determining or obtaining a prediction result of the first component and/or the second component of the pixel to be predicted according to the reference pixel of the first component.

Optionally, the determining manner of the reference pixel includes at least one of the following:

a first mode: determining a reference pixel according to adjacent pixels of the pixel to be predicted or adjacent pixels of the co-located pixel of the pixel to be predicted;

the second mode is as follows: determining a reference pixel according to the vector information;

third mode: reference pixels are determined from the encoded image blocks.

Optionally, the first mode includes: the reference pixel is determined from neighboring pixels located above and/or to the left of the pixel to be predicted or a co-located pixel of the pixel to be predicted.

Optionally, the second mode includes: and determining the reference pixel according to the vector information and the position information of the pixel to be predicted.

Optionally, the second mode includes: and determining the reference pixel according to the vector information and the position information of the image block where the pixel to be predicted is located.

Optionally, the third mode includes: and determining a pixel corresponding to the pixel position to be predicted in the encoded image block as a reference pixel.

Optionally, the method further comprises at least one of:

the neighboring pixels include neighboring reconstructed pixels and/or neighboring predicted pixels;

the encoded image block is an image block determined according to a rate distortion optimization or image matching algorithm.

Optionally, the pixel to be predicted is determined or obtained according to the first parameter and the reference pixel.

Optionally, the method further comprises at least one of:

the first parameter is determined in a manner corresponding to at least one prediction mode;

the first parameter is determined or derived from a first reference pixel of the first component and/or the second component.

Optionally, the index or flag corresponding to the at least one prediction mode is located in the prediction mode list.

Optionally, the first reference pixel is located in at least one of the following areas:

adjacent areas of an image block where pixels to be predicted are located or adjacent areas of a homonymous block of the image block;

adjacent areas of reference image blocks corresponding to the image blocks where the pixels to be predicted are located or adjacent areas of co-located blocks of the reference image blocks corresponding to the image blocks where the pixels to be predicted are located;

A reference image block determined by the vector information;

the pixels to be predicted are located in non-adjacent areas of the image block or non-adjacent areas of the co-located blocks of the image block.

Optionally, the method further comprises at least one of:

the first component is a luminance component or a chrominance component;

the second component is a luminance component or a chrominance component;

the first component and the second component are different pixel components;

the reference pixels include at least one first pixel;

the pixels to be predicted comprise at least one second pixel or a plurality of third pixels;

the vector information includes a block vector or a motion vector.

Optionally, the method further comprises at least one of:

the combination of the at least one first pixel and the at least one second pixel is rectangular;

the plurality of third pixels are adjacent to each other;

the combination of the plurality of third pixels is rectangular;

the prediction results of the first component or the second component of each of the plurality of third pixels are the same.

The present application also provides a processing apparatus comprising: the image processing device comprises a memory and a processor, wherein the memory stores an image processing program, and the image processing program realizes the steps of the image processing method when being executed by the processor.

The present application also provides a storage medium storing a computer program which, when executed by a processor, implements the steps of any of the image processing methods described above.

As described above, the image processing method of the present application is applicable to a processing apparatus, and by acquiring or determining a reference pixel of a first component, a first component and/or a second component prediction result of a pixel to be predicted is determined or obtained from the reference pixel. Through the technical scheme, the residual error value of the residual error block corresponding to the prediction block can be effectively reduced in the image encoding and decoding process, so that the compression efficiency in the image encoding and decoding process is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.

Fig. 1 is a schematic hardware structure diagram of an intelligent terminal implementing various embodiments of the present application;

fig. 2 is a schematic diagram of a communication network system according to an embodiment of the present application;

FIG. 3 is a schematic diagram of the distribution of input pixels involved in convolving a cross-component model CCCM;

FIG. 4 is a schematic diagram of the distribution of input pixels involved in intra chroma prediction GL-CCCM based on a convolutional cross-component model of gradient and position;

FIG. 5A is a schematic diagram of an image encoder involved in an image processing method provided in an embodiment of the present application;

fig. 5B is a schematic diagram of an image decoder related to an image processing method provided in an embodiment of the present application;

fig. 6 is a flowchart of an image processing method according to the first embodiment;

fig. 7a to 7c are each a schematic diagram showing a combination between a pixel to be predicted and a reference pixel involved in an image processing method according to a third embodiment;

fig. 8 is a schematic diagram showing the combination of pixels to be predicted involved in an image processing method according to the third embodiment;

fig. 9a to 9f are schematic diagrams each showing a positional relationship between a combination of pixels to be predicted and a reference pixel according to an image processing method according to a third embodiment;

fig. 10 is a schematic diagram showing a case where an image processing method according to the third embodiment involves pixels that cannot be used as references;

Fig. 11a to 11c are schematic diagrams each showing a position condition of a pixel to be predicted and a reference pixel related to an image processing method according to a third embodiment;

fig. 12 is a schematic diagram of a block vector BV and a reference block involved in an image processing method according to the third embodiment;

fig. 13 is a schematic diagram of a reference pixel in a determination reference block involved in an image processing method according to a third embodiment;

fig. 14a and 14b are each a reference pixel schematic diagram related to an image processing method according to the third embodiment;

fig. 15 is a schematic diagram of a determination reference pixel involved in an image processing method according to the third embodiment;

fig. 16 is a schematic diagram of determination of a reference block by a motion vector MV, which is involved in an image processing method shown in accordance with the third embodiment;

fig. 17 is a schematic diagram of a reference pixel in a determination reference block involved in an image processing method according to the third embodiment;

fig. 18a to 18d are schematic diagrams each of a first coefficient determination reference pixel and a second coefficient determination reference pixel related to the image processing method according to the fourth embodiment;

fig. 19 is a schematic diagram of a first coefficient determination reference pixel and a second coefficient determination reference pixel to be determined involved in an image processing method according to the third embodiment;

Fig. 20 is a schematic diagram of a first coefficient determination reference pixel and a second coefficient determination reference pixel to be determined involved in an image processing method according to the third embodiment;

fig. 21 is a schematic diagram of a first coefficient determination reference pixel and a second coefficient determination reference pixel to be determined in accordance with the image processing method shown in the third embodiment.

The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings. Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the element defined by the phrase "comprising one … …" does not exclude the presence of additional identical elements in a process, method, article, or apparatus that comprises the element and/or the elements, features, or elements having the same name in different embodiments of the present application may have the same meaning or may have different meanings, a particular meaning of which is to be determined by its interpretation in this particular embodiment or by further combining the context of this particular embodiment.

It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope herein. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context. Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and/or "including" specify the presence of stated features, steps, operations, elements, components, items, categories, and/or groups, but do not preclude the presence, presence or addition of one or more other features, steps, operations, elements, components, items, categories, and/or groups. The terms "or," "and/or," "including at least one of," and the like, as used herein, may be construed as inclusive, or meaning any one or any combination. For example, "including at least one of: A. b, C "means" any one of the following: a, A is as follows; b, a step of preparing a composite material; c, performing operation; a and B; a and C; b and C; a and B and C ", again as examples," A, B or C "or" A, B and/or C "means" any of the following: a, A is as follows; b, a step of preparing a composite material; c, performing operation; a and B; a and C; b and C; a and B and C). An exception to this definition will occur only when a combination of elements, functions, steps or operations are in some way inherently mutually exclusive.

It should be understood that, although the steps in the flowcharts in the embodiments of the present application are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the figures may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily occurring in sequence, but may be performed alternately or alternately with at least some of the sub-steps or stages of other steps or steps.

The words "if", as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrase "if determined" or "if detected (stated condition or event)" may be interpreted as "when determined" or "in response to determination" or "when detected (stated condition or event)" or "in response to detection (stated condition or event), depending on the context.

It should be noted that, in this document, step numbers such as S10 are used for the purpose of describing corresponding contents more clearly and briefly, and not limiting the sequence in nature, and those skilled in the art may perform other steps first and then perform S10 when implementing the present invention, which are all within the scope of protection of the present application.

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

In the following description, suffixes such as "module", "component", or "unit" for representing elements are used only for facilitating the description of the present application, and are not of specific significance per se. Thus, "module," "component," or "unit" may be used in combination.

The processing device mentioned in the present application may be a server or an intelligent terminal, and the intelligent terminal may be implemented in various forms. For example, the smart terminals described in the present application may include smart terminals such as cell phones, tablet computers, notebook computers, palm computers, personal digital assistants (Personal Digital Assistant, PDA), portable media players (Portable Media Player, PMP), navigation devices, wearable devices, smart bracelets, pedometers, and stationary terminals such as digital TVs, desktop computers, and the like.

The following description will be given taking a mobile terminal as an example, and those skilled in the art will understand that the configuration according to the embodiment of the present application can be applied to a fixed type terminal in addition to elements particularly used for a moving purpose.

Referring to fig. 1, which is a schematic hardware structure of a mobile terminal implementing various embodiments of the present application, the mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an a/V (audio/video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111. Those skilled in the art will appreciate that the mobile terminal structure shown in fig. 1 is not limiting of the mobile terminal and that the mobile terminal may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

The following describes the components of the mobile terminal in detail with reference to fig. 1:

the radio frequency unit 101 may be used for receiving and transmitting signals during the information receiving or communication process, specifically, after receiving downlink information of the base station, processing the downlink information by the processor 110; and, the uplink data is transmitted to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. And/or the radio frequency unit 101 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol including, but not limited to, GSM (Global System of Mobile communication, global system for mobile communications), GPRS (General Packet Radio Service ), CDMA2000 (Code Division Multiple Access, 2000, CDMA 2000), WCDMA (Wideband Code Division Multiple Access ), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access, time Division synchronous code Division multiple access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution, frequency Division duplex long term evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution, time Division duplex long term evolution), 5G, 6G, and the like.

WiFi belongs to a short-distance wireless transmission technology, and a mobile terminal can help a user to send and receive e-mails, browse web pages, access streaming media and the like through the WiFi module 102, so that wireless broadband Internet access is provided for the user. Although fig. 1 shows a WiFi module 102, it is understood that it does not belong to the necessary constitution of a mobile terminal, and can be omitted entirely as required within a range that does not change the essence of the invention.

The audio output unit 103 may convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output as sound when the mobile terminal 100 is in a call signal reception mode, a talk mode, a recording mode, a voice recognition mode, a broadcast reception mode, or the like. Also, the audio output unit 103 may also provide audio output (e.g., a call signal reception sound, a message reception sound, etc.) related to a specific function performed by the mobile terminal 100. The audio output unit 103 may include a speaker, a buzzer, and the like.

The a/V input unit 104 is used to receive an audio or video signal. The a/V input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, the graphics processor 1041 processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The processed image frames may be displayed on the display unit 106. The image frames processed by the graphics processor 1041 may be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) via the microphone 1042 in a phone call mode, a recording mode, a voice recognition mode, and the like, and can process such sound into audio data. The processed audio (voice) data may be converted into a format output that can be transmitted to the mobile communication base station via the radio frequency unit 101 in the case of a telephone call mode. The microphone 1042 may implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated in the course of receiving and transmitting the audio signal.

The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor, optionally, the ambient light sensor may adjust the brightness of the display panel 1061 according to the brightness of ambient light, and the proximity sensor may turn off the display panel 1061 and/or the backlight when the mobile terminal 100 moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the acceleration in all directions (generally three axes), and can detect the gravity and direction when stationary, and can be used for applications of recognizing the gesture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration recognition related functions (such as pedometer and knocking), and the like; as for other sensors such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that may also be configured in the mobile phone, the detailed description thereof will be omitted.

The display unit 106 is used to display information input by a user or information provided to the user. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD), an Organic Light-Emitting Diode (OLED), or the like.

The user input unit 107 may be used to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the mobile terminal. Alternatively, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also referred to as a touch screen, may collect touch operations thereon or thereabout by a user (e.g., operations of the user on the touch panel 1071 or thereabout by using any suitable object or accessory such as a finger, a stylus, etc.) and drive the corresponding connection device according to a predetermined program. The touch panel 1071 may include two parts of a touch detection device and a touch controller. Optionally, the touch detection device detects the touch azimuth of the user, detects a signal brought by touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends the touch point coordinates to the processor 110, and can receive and execute commands sent from the processor 110. And/or the touch panel 1071 may be implemented in various types of resistive, capacitive, infrared, surface acoustic wave, and the like. The user input unit 107 may include other input devices 1072 in addition to the touch panel 1071. Alternatively, other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, mouse, joystick, etc., as specifically not limited herein.

Alternatively, the touch panel 1071 may overlay the display panel 1061, and when the touch panel 1071 detects a touch operation thereon or thereabout, the touch panel 1071 is transferred to the processor 110 to determine the type of touch event, and the processor 110 then provides a corresponding visual output on the display panel 1061 according to the type of touch event. Although in fig. 1, the touch panel 1071 and the display panel 1061 are two independent components for implementing the input and output functions of the mobile terminal, in some embodiments, the touch panel 1071 may be integrated with the display panel 1061 to implement the input and output functions of the mobile terminal, which is not limited herein.

The interface unit 108 serves as an interface through which at least one external device can be connected with the mobile terminal 100. For example, the external devices may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and an external device.

Memory 109 may be used to store software programs as well as various data. The memory 109 may mainly include a storage program area and a storage data area, and alternatively, the storage program area may store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. And/or memory 109 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

The processor 110 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by running or executing software programs and/or modules stored in the memory 109 and calling data stored in the memory 109, thereby performing overall monitoring of the mobile terminal. Processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor, the application processor optionally handling mainly an operating system, a user interface, an application program, etc., the modem processor handling mainly wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 110.

The mobile terminal 100 may further include a power source 111 (e.g., a battery) for supplying power to the respective components, and preferably, the power source 111 may be logically connected to the processor 110 through a power management system, so as to perform functions of managing charging, discharging, and power consumption management through the power management system.

Although not shown in fig. 1, the mobile terminal 100 may further include a bluetooth module or the like, which is not described herein.

In order to facilitate understanding of the embodiments of the present application, a communication network system on which the mobile terminal of the present application is based will be described below.

Referring to fig. 2, fig. 2 is a schematic diagram of a communication network system provided in the embodiment of the present application, where the communication network system is an LTE system of a general mobile communication technology, and the LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network ) 202, an epc (Evolved Packet Core, evolved packet core) 203, and an IP service 204 of an operator that are sequentially connected in communication.

Alternatively, the UE201 may be the terminal 100 described above, which is not described here again.

The E-UTRAN202 includes eNodeB2021 and other eNodeB2022, etc. Alternatively, the eNodeB2021 may connect with other enodebs 2022 over a backhaul (e.g., X2 interface), the eNodeB2021 is connected to the EPC203, and the eNodeB2021 may provide access for the UE201 to the EPC 203.

EPC203 may include MME (Mobility Management Entity ) 2031, HSS (Home Subscriber Server, home subscriber server) 2032, other MMEs 2033, SGW (Serving Gate Way) 2034, PGW (PDN Gate Way) 2035 and PCRF (Policy and Charging Rules Function, policy and tariff function entity) 2036, and the like. Optionally, MME2031 is a control node that handles signaling between UE201 and EPC203, providing bearer and connection management. HSS2032 is used to provide registers to manage functions such as home location registers (not shown) and to hold user specific information about service characteristics, data rates, etc. All user data may be sent through SGW2034 and PGW2035 may provide IP address allocation and other functions for UE201, PCRF2036 is a policy and charging control policy decision point for traffic data flows and IP bearer resources, which selects and provides available policy and charging control decisions for a policy and charging enforcement function (not shown).

IP services 204 may include the internet, intranets, IMS (IP Multimedia Subsystem ), or other IP services, etc.

Although the LTE system is described above as an example, it should be understood by those skilled in the art that the present application is not limited to LTE systems, but may be applied to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, 5G, and future new network systems (e.g., 6G), etc.

Based on the above-mentioned mobile terminal hardware structure and communication network system, various embodiments of the present application are presented.

The application provides an image processing method, which can determine or obtain a prediction result of a first component and/or a second component of a pixel to be predicted according to a reference pixel of the first component. In this way, the first component and/or the second component prediction result of the pixel to be predicted is determined or obtained according to the reference pixel by acquiring or determining the reference pixel of the first component. The residual error value of the residual error block corresponding to the predicted block can be effectively reduced in the image encoding and decoding process, so that the compression efficiency in the image encoding and decoding process is improved.

For ease of understanding, the following description will first explain terms of art that may be relevant to the present application.

Prediction mode (one)

In the process of encoding or decoding an image, prediction of an image block is an indispensable step. For example, the encoder predicts the image block to obtain a predicted block, constructs a residual block with smaller energy, and reduces transmission bits; the decoder obtains a residual block through entropy decoding, and decodes the residual block and a predicted block obtained through prediction in the decoder to obtain a decoded image block to achieve decoding of an image. The prediction of the image block by the encoder or the decoder may be implemented by some preset prediction modes, and the prediction modes may include modes of inter prediction and modes of intra prediction.

(II) convolutional Cross component model CCCM

The convolutional cross-component model CCCM uses filters to predict the chroma of the current image block from the already reconstructed luma pixels. Optionally, the filter used by CCCM consists of 5 taps of luminance pixels that exhibit a spatial distribution of plus sign shapes, a nonlinear term, and a bias term. Optionally, the 5 taps of the filter present a plus sign shaped spatial distribution of luminance pixels as input pixels, including center (C) luminance pixels, and up-samples (also referred to as north samples, N), down-samples (also referred to as south samples, S), left-samples (also referred to as west samples, W) and right-samples (also referred to as east samples, E) of center (C) luminance pixels. These input pixels are shown in particular in fig. 3.

The nonlinear term P is represented as the square of the center (C) luminance sample and scales to the range of values of the pixel values of the video image content: p= (c×c+midval) > > bit depth, optionally for a 10-bit depth video image, p= (c×c+512) > >10.

The offset term B represents a scalar offset between the input and output of the filter, optionally set to an intermediate chroma value. For a 10 bit-depth video image, the intermediate chroma value is 512.

The output of the filter is calculated as the convolution between the filter coefficients ci and the input values and clipped to the range of valid chroma samples:

predChromaVal = c0*C + c1*N + c2*S + c3*E + c4*W + c5*P + c6*B

(III) intra chroma prediction GL-CCCM based on a convolutional cross-component model of gradient and position

The gradient and position based convolutional cross-component model GL-CCCM uses gradient and position information instead of 4 spatially adjacent samples in the CCCM filter. Optionally, the GL-CCCM filter used for prediction is:

predChrmaVal=c0*C+c1*Gy+c2*Gx+c3*Y+c4*X+c5*P+c6*B

wherein Gy and Gx are vertical gradient and horizontal gradient respectively, and in combination with the spatial sampling for GL-CCCM shown in fig. 4, the calculation formulas of Gy and Gx are as follows:

Gy=（2N+NW+NE）-（2S+SW+SE）；

Gx=（2W+NW+SW）-（2E+NE+SE）。

optionally, the Y and X parameters are the vertical and horizontal positions of the center (C) luminance sample.

Alternatively, the Y and X parameters may be calculated with respect to the upper left corner coordinates of the image block.

In this application, "adjacent" means spatially adjacent. That is, the a object has no other object in the middle of the b object. For example, adjacent samples/pixels means that two samples/pixels are spatially contiguous; adjacent regions refer to two regions that are spatially contiguous, etc. For example, samples N, E, W, S, NW, NE, SW, SE surrounding sample C in fig. 4 are all adjacent samples to sample C.

Alternatively, referring to fig. 5A and 5B, fig. 5A is a schematic diagram of an image encoder related to an image processing method, and fig. 5B is a schematic diagram of an image decoder related to an image processing method. As shown in fig. 5A, an encoder at the encoding end generally divides an input video image into at least one image block according to frames, each image block may be subtracted from a prediction block obtained by prediction in a prediction mode to obtain a residual block, and a series of processes are performed on the residual block and related parameters of the prediction mode to obtain an encoded bitstream. Then, at the decoding end, as shown in fig. 5B, after receiving the bitstream, the decoder can obtain the prediction mode parameters by parsing the bitstream. Furthermore, an inverse transform unit and an inverse quantization unit of the decoder perform inverse transform and inverse quantization processing on the transform coefficients to obtain residual blocks. Optionally, the decoding unit of the decoder parses and decodes the encoded bitstream to obtain the prediction parameters and the associated side information. Next, a prediction processing unit of the decoder performs prediction processing using the prediction parameters, thereby determining a prediction block corresponding to the residual block. In this way, the decoder can obtain a reconstructed block by adding the obtained residual block and the corresponding prediction block. Optionally, the loop filtering unit of the decoder performs loop filtering processing on the reconstructed block to reduce distortion and improve video quality. Thus, the reconstructed blocks subjected to the loop filtering process are further combined into a decoded image which is stored in a decoded image buffer or output as a decoded video signal.

Optionally, the image processing method provided by the application can be applied to a scene for performing chroma prediction and/or luminance prediction on an image block in the video image encoding process. For example, scenes for intra prediction in image coding and scenes for inter prediction in image coding. Optionally, the image processing method provided by the application can also be used for performing chroma prediction and/or brightness prediction on the image block to be decoded in the video decoding process. For example, an intra-predicted scene during image decoding and an inter-predicted scene during image decoding.

First embodiment

In this embodiment, the execution body of the image processing method may be the above-mentioned processing device, or a cluster formed by the above-mentioned plurality of processing devices, and the processing device may be an intelligent terminal (such as the above-mentioned mobile terminal 100) or a server. Here, the image processing method will be described with a processing apparatus as an execution subject in the first embodiment of the image processing method.

As shown in fig. 6, fig. 6 is a flow chart of an image processing method according to the first embodiment, in the present embodiment, the image processing method includes the steps of:

S10: and determining or obtaining a prediction result of the first component and/or the second component of the pixel to be predicted according to the reference pixel of the first component.

Optionally, when the processing device performs chroma prediction and/or luminance prediction on a pixel to be predicted in the current image block to be processed by adopting an inter-frame prediction mode and/or an intra-frame prediction mode in the process of encoding or decoding the image, firstly acquiring or determining a reference pixel of at least one first component, and then predicting the pixel to be predicted of the first component and/or the second component according to a pixel value of the reference pixel, so as to determine or obtain a prediction result of the pixel to be predicted of the first component and/or the second component.

Optionally, when the processing device performs chroma prediction and/or luminance prediction on a pixel to be predicted in the current image block to be processed using an inter-frame prediction mode and/or an intra-frame prediction mode in the process of encoding or decoding the image, first obtains or determines at least one reference pixel of a first component and at least one reference pixel of a second component, and then predicts the pixel to be predicted of the first component or the second component according to a pixel value of the first component reference pixel and a pixel value of the second component, thereby determining or obtaining a prediction result of the pixel to be predicted of the first component or the second component.

Alternatively, in the present application, an image block may be an image region in an input video image (i.e., video frame) that is being encoded or decoded so that chroma prediction and/or luma prediction are required. The image block may also be an image region to be encoded or a reconstructed or decoded image region. Alternatively, an image block currently undergoing encoding or decoding processing may also be simply referred to as a current block, a current unit, or a current processing block. Alternatively, the reference pixel may be referred to as a reference sample, and the pixel to be predicted may also be referred to as a pixel to be predicted. Under the h.265/high efficiency video coding (High Efficiency Video Coding, HEVC) standard, both the reference pixel and the pixel to be predicted may be one pixel or sample in one coding tree Unit (Coding Tree Units, CTU) or Coding Unit (CU) in the input video image. The definition or kind of the image block, the reference pixel and the pixel to be predicted is not particularly limited in the present application.

Optionally, the first component is a luminance component or a chrominance component. Alternatively, the processing device may acquire or determine a reference pixel of at least one known luminance component when performing chroma prediction and/or luminance prediction for a pixel to be predicted in the image block, so as to perform prediction for the pixel to be predicted according to a pixel value of the reference pixel. Alternatively, the processing device may also predict the pixel to be predicted based on pixel values of reference pixels of at least one known chrominance component.

Optionally, the second component is a luminance component or a chrominance component. Alternatively, the processing device may predict the luminance component of the pixel to be predicted from a reference pixel of a known luminance component or a known chrominance component to obtain a luminance component prediction result. Alternatively, the processing device may predict the chrominance component of the pixel to be predicted from a reference pixel of a known luminance component or a known chrominance component to obtain a chrominance component prediction result.

Alternatively, in case the first component is a luminance component or a chrominance component and/or the second component is a luminance component or a chrominance component, the processing device may make any one of the following predictions by reference pixels of the known luminance component:

1. predicting a luminance component of a pixel to be predicted;

2. predicting the chrominance component of the pixel to be predicted;

3. and simultaneously predicting a luminance component and a chrominance component of the pixel to be predicted.

Optionally, the processing device may also make any of the following predictions by knowing the reference pixels of the chrominance components:

1. predicting a chrominance component of a pixel to be predicted;

2. predicting the brightness component of the pixel to be predicted;

Optionally, the first component and the second component are different pixel components. Alternatively, the processing device may predict the chrominance component of the pixel to be predicted from a reference pixel of a known luminance component to obtain a chrominance component prediction result. Alternatively, the processing device may predict the luminance component of the pixel to be predicted from a reference pixel of a known chrominance component to obtain a luminance component prediction result.

Optionally, in this embodiment and the other various embodiments set forth above and below, the processing device acts as an encoder to segment the video image after receiving the video image from the video source to obtain at least one image block. And the processing device may determine that the image block is the image block currently to be processed when performing chroma prediction and/or luma prediction for the current image block using an intra-prediction mode (particularly a cross-component intra-prediction mode). Then, the processing device takes the pixel needing to predict the chrominance component and/or the luminance component in the image block as a pixel to be predicted, and then predicts the chrominance component and/or the luminance component of the pixel to be predicted by directly utilizing the pixel value (chrominance value or luminance value) of at least one reference pixel which is adjacent to and/or in the image block and has a known chrominance component or luminance component, thereby obtaining the chrominance component prediction result and/or the luminance component prediction result of the pixel to be predicted.

Alternatively, the processing device as an encoder may use, for example, a rate distortion optimization algorithm to determine the intra prediction mode that the current image block ultimately adopts. Optionally, the processing device may calculate a rate-distortion cost corresponding to each prediction mode to determine a minimum rate-distortion cost from the rate-distortion costs corresponding to the multiple prediction modes, where the prediction mode corresponding to the minimum rate-distortion cost is the prediction mode that is finally adopted by the current image block. That is, assuming that the prediction mode that can be used is chroma intra prediction modes 0 to N (optionally including inter-component intra prediction modes) regarding the prediction processing of the image block currently to be chroma predicted, the processing apparatus determines the mode i as the intra prediction mode that is ultimately used for chroma prediction of the image block currently to be processed when calculating the prediction mode corresponding to the minimum rate distortion cost for chroma prediction as the mode i therein. Alternatively, i=0.

Optionally, after determining or deriving the chrominance component and/or the luminance component of the pixel to be predicted in the image block in a pixel-by-pixel manner according to the above procedure, the processing device as an encoder may further subtract the predicted value of the corresponding pixel in the predicted image block (i.e. the image block chrominance information of the image block) to obtain the residual value of the pixel and the residual block corresponding to the image block from the pixel value of the pixel in the current image block. Then, the residual block is transformed and quantized, and then encoded by an entropy encoder to finally form an encoded bitstream. Optionally, the encoded bitstream may further include a prediction parameter (entropy encoded and packed into the encoded bitstream) and related auxiliary information (side information) corresponding to the intra-prediction mode determined by the processing device through the above process. Alternatively, if the processing device adopts a cross-component intra prediction mode, the prediction parameters include at least indication information about a prediction operation performed using the cross-component intra prediction mode.

Alternatively, the transformed quantized residual block may be added to a corresponding predicted image block (also referred to as a prediction block) obtained using a prediction mode to obtain a reconstructed image block (also referred to as a reconstruction block). After obtaining the reconstructed image block, the processing device further performs loop filtering processing on the reconstructed image block to reduce distortion.

Alternatively, the processing device may receive the encoded bitstream transferred by the processing device as the encoder when the processing device acts as the decoder, and the decoding unit of the decoder may parse and decode the bitstream to obtain the prediction parameters after the processing device receives the bitstream encoded by the encoder as the decoder. Alternatively, an inverse transform unit and an inverse quantization unit of the decoder inverse-transform and inverse-quantize the transform coefficients to obtain the residual block. Then, the prediction processing unit of the decoder may use the residual block as a current processing block that needs to be decoded, and perform prediction processing using the prediction parameters, thereby determining a prediction block corresponding to the residual block.

Alternatively, when the same intra prediction mode as used by the encoder is adopted, the prediction parameters obtained by parsing the bitstream may be used by the decoder to obtain or determine a prediction mode that needs to be adopted for chroma prediction and/or luma prediction of the image block to be currently processed (e.g., when the prediction parameters indicate that the corresponding prediction mode is a cross-component intra prediction mode, the decoder uses the cross-component intra prediction mode as the intra prediction mode for chroma prediction and/or luma prediction of the decoded residual block). The decoder thus uses the intra prediction mode directly to predict the chrominance component and/or the luminance component of the pixel to be predicted using the pixel value (chrominance value or luminance value) of at least one reference pixel adjacent to and/or in the image block for which the chrominance component or luminance component is known, thereby obtaining a chrominance component prediction result and/or a luminance component prediction result of the pixel to be predicted.

Optionally, the processing device as a decoder determines or derives a chrominance component prediction result and/or a luminance component prediction result of at least one pixel to be predicted in the image block in a pixel-by-pixel manner (also referred to as a sample-by-sample manner). Optionally, after determining or deriving the chrominance information and/or luminance information of the image block, the processing device may further add the parsed residual block and the predicted value of the corresponding pixel in the predicted image block (the image block chrominance information and/or luminance information of the image block) to obtain the reconstructed block. Finally, the processing device also performs loop filtering processing on the reconstructed block through a loop filtering unit to reduce distortion and improve video quality. And the reconstructed block subjected to the loop filtering process is further combined into a decoded image which is stored in a decoded image buffer or output as a decoded video signal.

Alternatively, in this embodiment and the other various embodiments set forth above and below, the prediction mode used by the processing device as an encoder or decoder to predict the current image block (e.g., chroma block) may be a convolutional cross-component intra prediction model CCCM as shown in equation (1):

predchromval=c0×c+c1×n+c2×s+c3×e+c4×w+c5×p+c6×b formula (1)

Alternatively, taking chroma prediction as an example, predchamval is a chroma prediction value of a pixel to be predicted in an image block to be processed, C0-C6 are filter coefficients, C is a luminance value of a parity luminance pixel of the pixel to be predicted, N is a luminance value of a luminance pixel above/adjacent to north of the parity luminance pixel, S is a luminance value of a luminance pixel below/adjacent to south of the parity luminance pixel, E is a luminance value of a luminance pixel right/adjacent to east of the parity luminance pixel, W is a luminance value of a luminance pixel left/adjacent to west of the parity luminance pixel, and P is a nonlinear term. B is an offset term representing the scalar offset between input and output (B is set to the median chromaticity for video of 10 bit depth, i.e., 512), p= (c×c+midval) > > bitDepth (midVal is the median chromaticity of a chroma pixel, bitDepth is the bit depth of video content). The positional relationship of N, S, E, W and C is shown in FIG. 3.

Alternatively, in order to perform chroma intra prediction processing using the CCCM model shown in the above formula (1), the processing device needs to determine the filter coefficients C0 to C6 and the luminance values of C, N, S, E, W, P, B in the formula (1), and then obtains the chroma prediction result of the chroma sampling predChromaVal to be predicted based on the formula (1). Optionally, the manner in which the processing device determines the filter coefficients in formula (1) may be: at least one reference region is acquired or determined from within an image frame in which an image block currently requiring chroma prediction is located, and filter coefficients are determined from sample values of luminance/chrominance pixels in the at least one reference region. Optionally, at least one luminance reference region is obtained or determined when performing the chroma prediction. Optionally, at least one chroma reference region is obtained or determined when performing chroma prediction.

In this embodiment, the image processing method predicts the chrominance component and/or the luminance component of the pixel to be predicted by the processing apparatus using the pixel value of the reference pixel of at least one known chrominance component or luminance component when predicting the chrominance component and/or the luminance component of the image block. That is, the technical scheme fully considers the correlation between the pixel to be predicted and the reference pixel of the known chroma component or the known luma component, so that the residual value of the residual block corresponding to the predicted block can be effectively reduced in the image encoding and decoding process, and the compression efficiency in the image encoding and decoding process is improved.

Second embodiment

In the present embodiment, the image processing method will be described with the processing apparatus as the execution subject. On the basis of any one of the above embodiments, the determining manner of the reference pixel may include at least one of:

a first mode: and determining the reference pixel according to the adjacent pixels of the pixel to be predicted or the adjacent pixels of the co-located pixel of the pixel to be predicted.

Alternatively, when the processing device predicts a pixel to be predicted for a chrominance component and/or luminance component in the image block currently to be processed based on a reference pixel for a known chrominance component or luminance component, the processing device may first obtain or determine the reference pixel for the known chrominance component or luminance component based on neighboring pixels of the pixel to be predicted.

Alternatively, when predicting a pixel to be predicted for a chrominance component and/or luminance component in a current image block to be processed from a reference pixel for a known chrominance component or luminance component, the processing device may first obtain or determine the reference pixel for the known chrominance component or luminance component from neighboring pixels of a co-located pixel of the pixel to be predicted. The co-located pixels may be pixels that are co-located with the pixel to be predicted but belong to a different color component type. For example, if the pixel to be predicted is a chrominance component pixel, the co-located pixel of the pixel to be predicted is a luminance component pixel having the same position as the chrominance component pixel.

The second mode is as follows: a reference pixel is determined from the vector information.

Alternatively, when the processing device predicts a pixel to be predicted of a chrominance component and/or luminance component in the image block currently to be processed based on a reference pixel of a known chrominance component or luminance component, the processing device may first acquire or determine the reference pixel of the known chrominance component or luminance component based on a vector information (e.g., a block vector) within the same frame image in which the pixel to be predicted is located. Optionally, the processing device may also obtain or determine the reference pixel of the known chrominance component or luminance component based on a vector information (e.g., a motion vector) within a reference image (also referred to as a reference frame) of the image in which the pixel to be predicted is located. Alternatively, the reference image may be a previous frame image or a subsequent frame image of the image in which the pixel to be predicted is located.

Third mode: reference pixels are determined from the encoded image blocks.

Alternatively, when predicting the chrominance component and/or the luminance component of a pixel to be predicted in a current image block to be processed from reference pixels of a known chrominance component or luminance component, the processing device may first acquire or determine at least one encoded image block or encoded/decoded image region from within the same frame image in which the pixel to be predicted is located, and then acquire or determine the reference pixel of the known chrominance component or luminance component from the encoded image block or encoded/decoded image region. The encoded image block or encoded/decoded image region is not adjacent/neighboring the image block currently to be processed. Optionally, the processing device may further acquire or determine at least one encoded image block or encoded/decoded image area from within a previous frame image or a subsequent frame image of the image in which the pixel to be predicted is located, and then acquire or determine the reference pixel from the encoded image block or encoded/decoded image area. Alternatively, the encoded/decoded image region may be an image region determined by a motion estimation process at the encoding end. Alternatively, the encoded/decoded image region may be an image region determined by a motion vector/block vector at the decoding end.

Alternatively, the processing device may combine the first and second modes described above to determine reference pixels of a known luminance component or chrominance component. Optionally, the reference pixels a 0-an are determined by a first approach and the reference pixels b 0-bm are determined by a second approach. m and n are integers. Optionally, the processing device determines the reference pixels a 0-an from neighboring pixels of the pixel to be predicted and/or determines the reference pixels b 0-bm from vector information. The vector information includes a motion vector and/or a block vector. And then, taking the reference pixels a 0-an and the reference pixels b 0-bm as finally adopted reference pixels to further determine the prediction result of the first component and/or the second component of the pixel to be predicted.

Alternatively, the processing device may also combine only the first and third modes described above to determine reference pixels of a known luminance component or chrominance component. Optionally, the reference pixels a '0 to a' n are determined by the first method and the reference pixels b '0 to b'm are determined by the third method. m and n are integers. Optionally, the processing device determines the reference pixels a '0 to a' n by adjacent pixels of the pixels to be predicted, and determines the reference pixels b '0 to b'm by image blocks or image areas in which the image blocks to be currently processed are not adjacent. And then, taking the reference pixels a '0-a' n and the reference pixels b '0-b'm as finally adopted reference pixels to further determine the prediction result of the first component and/or the second component of the pixel to be predicted. Alternatively, the adjacent pixels may be pixels in the same image block as the pixel to be predicted (for example, pixels subjected to prediction processing), or pixels in an image block adjacent to the same image block (image block in which the pixel to be predicted is located) (for example, pixels subjected to reconstruction processing).

Optionally, the processing device may also combine the first, second and third modes described above to determine reference pixels of a known luminance component or chrominance component. Optionally, the reference pixel a″ 0~a ″ n is determined by the first means; determining a reference pixel b ''0~b '' n by a second means; and determining the reference pixel c '0~c' l by the third means. l, m and n are integers. Next, the reference pixels a '0 to a' n, the reference pixels b '0 to b'm, and the reference pixel c″ 0~c ″ l are used as final reference pixels to further determine the prediction result of the first component and/or the second component of the pixel to be predicted.

Alternatively, the values of at least two variables l, m, n in the above embodiments may be the same, or the values of all three variables may be the same.

Alternatively, in the present application, the processing device may also determine the reference pixel of the known luminance component or the chrominance component by combining only the above-described second mode and third mode. Alternatively, the processing device may determine at least one encoded image block from the vector information and then obtain or determine pixels of at least one known luminance component or chrominance component from the encoded image block as reference pixels. Optionally, the processing device determines the reference pixel a″ 0~a ″ n by a second way; and determining the reference pixel b ''0~b '' n by a second way. m and n are integers. And then, taking the reference pixels a '0-a' n and the reference pixels b '0-b'm as finally adopted reference pixels to further determine the prediction result of the first component and/or the second component of the pixel to be predicted.

Optionally, the image processing method may further include at least one of:

the first mode includes: the reference pixel is determined from neighboring pixels located above and/or to the left of the pixel to be predicted or a co-located pixel of the pixel to be predicted.

Alternatively, when the processing device acquires or determines a reference pixel of a known chrominance component or luminance component from the neighboring pixels of the pixel to be predicted, the reference pixel may be determined from among neighboring pixels located above and/or to the left of the pixel to be predicted. Alternatively, the reference pixel may be determined from among neighboring pixels (e.g., upper, lower, left, right) located around the pixel to be predicted. Alternatively, in the step of determining or obtaining the prediction result of the first component of the pixel to be predicted (i.e., in the same-component prediction process) from the reference pixel of the first component, the pixel value of the adjacent pixel located below or to the right of the pixel to be predicted in the image where the pixel to be predicted is located is unknown because the adjacent pixel below or to the right of the pixel to be predicted has not been subjected to the reconstruction process or the prediction process due to the process order of the raster scan when the pixel to be predicted is subjected to the prediction process. Therefore, in the case of predicting a pixel to be predicted of the same pixel component as the reference using the reference pixel, if the pixel value of the pixel to be predicted is determined or obtained by the lower pixel and/or the right pixel of the pixel to be predicted, it is necessary to determine an encoded/decoded image block in the reference image or the current image (i.e., the image from which the pixel to be predicted is derived) by a block vector or a motion vector, and determine at least one pixel in the encoded/decoded image block (or image area), and perform prediction processing with the pixel value of the at least one pixel as the fill value of the lower pixel and/or the right pixel of the pixel to be predicted.

Optionally, the neighboring pixels of the pixel to be predicted are located in the same image block as the pixel to be predicted. Optionally, the neighboring pixels of the pixel to be predicted are located in the encoded image block or the reconstructed image block neighboring the image block where the pixel to be predicted is located to be processed.

Alternatively, the processing device may determine the reference pixel of the known chrominance component or luminance component from among neighboring pixels located above and/or to the left of the co-located pixel of the pixel to be predicted when the reference pixel is acquired or determined from neighboring pixels of the co-located pixel of the pixel to be predicted. Alternatively, the reference pixel may be determined from among neighboring pixels (e.g., upper, lower, left, right) located around the co-located pixel of the pixel to be predicted. Optionally, in the step of determining or obtaining the prediction result of the second component of the pixel to be predicted from the reference pixel of the first component (i.e., in the cross-component prediction process), when the pixel to be predicted is subjected to the prediction process, the pixel value of the neighboring pixel located below or to the right of the co-located pixel of the pixel to be predicted in the image where the pixel to be predicted is located is unknown because the neighboring pixel below or to the right of the co-located pixel of the pixel to be predicted has not been subjected to the reconstruction process or the prediction process due to the process order of the raster scan. Therefore, in the case of predicting a pixel to be predicted of a different pixel component from a reference pixel by using the reference pixel, if a pixel value of the pixel to be predicted is determined or obtained by using a pixel below and/or a pixel to the right of the co-located pixel of the pixel to be predicted, it is necessary to determine an encoded/decoded image block in the reference image or the current image (i.e., the image from which the pixel to be predicted is derived) by a block vector or a motion vector, and determine at least one pixel in the encoded/decoded image block (or image area), and perform prediction processing by using the pixel value of the at least one pixel as a fill value of the pixel below and/or the pixel to the right of the co-located pixel of the pixel to be predicted. The pixel component of the determined pixel is different from the pixel component of the pixel to be predicted.

The second mode includes: and determining the reference pixel according to the vector information and the position information of the pixel to be predicted.

Alternatively, when the processing device obtains or determines the reference pixel of the known chrominance component or luminance component according to a vector information, the processing device may obtain or determine the reference pixel of the known chrominance component or luminance component according to the vector information and the position of the pixel to be predicted, in the current image in which the pixel to be predicted is located, the previous frame image of the current image, or the next frame image of the current image. Alternatively, the vector information is a block vector or a motion vector. For example, in the decoder, if the position of the pixel to be predicted in the current image is (x 0, y 0), the block vector is (xbv, ybv), the position of the reference pixel is (x0+ xbv, y0+ xbv). For another example, in the decoder, if the position of the pixel to be predicted in the current image is (x 0, y 0), the motion vector is (xmv, ymv), the position of the reference pixel in the previous frame image of the current image or the next frame image of the current image is (x0+ xmv, y0+ xmv).

At least one of the reference pixel position and the pixel value of the reference pixel is determined, i.e. the reference pixel is determined in the second way.

The second mode includes: and determining the reference pixel according to the vector information and the position information of the image block where the pixel to be predicted is located.

Alternatively, when the processing device obtains or determines the reference pixel of the known chrominance component or luminance component according to a vector information, the processing device may also determine the reference pixel according to the vector information and the position information of the image block where the pixel to be predicted is located.

Alternatively, the reference pixel of the known chrominance component or luminance component may be acquired or determined within the current image in which the pixel to be predicted is located, the previous frame image of the current image, or the next frame image of the current image, based on the vector information and the position information of the image block in which the pixel to be predicted is located.

For example, the reference pixel of the known chrominance component or luminance component may be acquired or determined from the block vector and the location information of the image block in which the pixel to be predicted is located within the current image in which the pixel to be predicted is located. Optionally, the position information of the pixel to be predicted is determined according to the position information of the block to be predicted, and the reference pixel is determined according to the block vector and the position information of the pixel to be predicted. Optionally, if the position of the image block in which the image block in the current image is located is (x 1, y 1), and the pixel to be predicted is the pixel of the 1 st row and the 2 nd column in the block to be predicted, the position of the pixel to be predicted is (x1+1, y1+2). Alternatively, if the block vector is (xbv 1, ybv 1), the position of the reference pixel is (x1+ xbv +1, y1+ xbv +2). After the location of the reference pixel is determined, the pixel value of the reference pixel may be further determined.

For another example, the reference pixel of the known chrominance component or luminance component may be acquired or determined within a previous frame image of the current image or a subsequent frame image of the current image in which the pixel to be predicted is located based on the motion vector and the position information of the image block in which the pixel to be predicted is located. Optionally, the position information of the pixel to be predicted is determined according to the position information of the block to be predicted, and the reference pixel is determined according to the motion vector and the position information of the pixel to be predicted. Optionally, if the position of the image block in which the image block in the current image is located is (x '1, y' 1), and the pixel to be predicted is the pixel of the 1 st row and the 2 nd column in the block to be predicted, the position of the pixel to be predicted is (x '1+1, y' 1+2). Alternatively, if the motion vector is (xmv, ymv 1), the position of the reference pixel is (x '1+xmv+1, y' 1+xmv+2). After the location of the reference pixel is determined, the pixel value of the reference pixel may be further determined.

The third mode includes: and determining a pixel corresponding to the pixel position to be predicted in the encoded image block as a reference pixel.

Alternatively, when the processing device acquires or determines the reference pixel from the encoded image block, at least one pixel corresponding to the pixel to be predicted in the encoded image block may be determined according to the position information of the pixel to be predicted, and the pixel of the chrominance component or the luminance component is known, so that the pixel is taken as the reference pixel.

Optionally, the position information indicates a position of the pixel to be predicted in the image block where it is located. For example, if the pixel to be predicted is the pixel of the 1 st row and the 2 nd column in the block to be predicted, the relative position information of the pixel to be predicted may be represented by coordinates (1, 2). An embodiment of the determination of the encoded image blocks is to set an area of a predetermined size in the encoded area of the current image. The encoded image block is then searched or determined in the region of predicted size. For example, the predetermined size region may be located at the left side or above an image block where a pixel to be predicted is located, the prediction size is larger than the size of the image block where the image block to be predicted is located, and the size of the encoded image block is larger than or equal to the size of the image block where the pixel to be predicted is located.

Optionally, the corresponding reference pixel is determined according to the position of the encoded block. For example, if the position coordinates of the encoded block are (x _cb ，y _cb ) The location information of the reference pixel in the encoded image block may be (x) _cb +1，y _cb +2）。

Alternatively, the method of searching or determining the encoded image block may be to determine a candidate image block from at least one candidate image block as the encoded image block according to a rate distortion optimization algorithm. Alternatively, at least one candidate image block may be determined from at least one candidate image block according to a rate distortion optimization algorithm, and an image block obtained by fusion processing of the at least one candidate image block may be used as an encoded image block (hereinafter referred to as a target image block, a reference image block). The order in which the at least one candidate image block is determined from the at least one candidate image block may be a raster scan order or a predefined search order.

The fusion process may be a process of generating one target image block from at least one image block to be fusion-processed. For example, the fusion process may be to obtain an image block with a pixel value being an average value of each pixel by obtaining an average pixel value of each corresponding position pixel in at least one image block to be fused, and take the image block as a target image block. For another example, the fusion processing may be to obtain a filtered image block by performing filtering processing on at least one image block to be fused, and take the image block as the target image block.

Optionally, the method of searching or determining the encoded image block may be that, in the encoded area, at least one candidate image block having a size equal to the size of the image block where the pixel to be predicted is located is determined, and according to an image matching algorithm, the matching degree/difference degree between the at least one candidate image block and the image block where the pixel to be predicted is located is determined, and a candidate image block is determined from the at least one candidate image block as the encoded image block. The degree of matching/degree of difference may be determined using algorithms such as the sum of absolute differences between coefficients (Sum of Absolute Differences, SAD), the absolute difference between coefficients after frequency domain transformation (Sum of Absolute Transformed Differences, SATD), etc.

Alternatively, the encoded image block may be determined by vector information (e.g., a block vector and/or a motion vector), and a pixel corresponding to the pixel position to be predicted may be determined as a reference pixel from the determined encoded image block. For example, if the position of the image block where the image block located in the current image is located is (x 1, y 1), and if the block vector is (xbv 1, ybv 1), the position of the encoded image block is (x1+ xbv1, y1+ ybv 1). For a detailed description of the related embodiments of determining the encoded image blocks by means of vector information, see below with respect to what is shown in fig. 12. For another example, if the position of the image block in which the image block in the current image is located is (x 1, y 1), and if the motion vector is (xmv 1, ybm 1), the position of the encoded image block in the reference image is (x1+ xmv1, y1+ ymv 1). The reference image may be a previous frame image of the current image or a subsequent frame image of the current image.

Optionally, the image processing method may further include at least one of:

Alternatively, the above-mentioned adjacent pixels located above and/or to the left of the pixel to be predicted may be only reconstructed pixels that are located adjacent to the pixel to be predicted among the image blocks that have been subjected to the reconstruction processing by the processing apparatus. For details, reference may be further made to the relevant paragraphs correspondingly described in fig. 11 a. The reconstructed pixels may be the reconstructed pixels of the luminance component or the reconstructed pixels of the chrominance component. Optionally, the predicted pixel and the residual pixel are added to obtain a reconstructed pixel. The above process is a reconstruction process. The reconstruction process may be performed in both the encoder and the decoder. Alternatively, the above-mentioned adjacent pixel may be only a predicted pixel that is adjacent to the pixel to be predicted and has been predicted from the current image block to be processed where the pixel to be predicted is located. That is, both the neighboring prediction pixel and the pixel to be predicted are in the current image block to be processed. The predicted pixel may be a predicted pixel of the predicted luminance component or a predicted pixel of the predicted chrominance component. For details, reference may be further made to the relevant paragraphs correspondingly described in fig. 11 b. Optionally, the above-mentioned neighboring pixels may also be the above-mentioned reconstructed pixels and predicted pixels at the same time. Optionally, the reconstructed pixel located adjacent to the pixel to be predicted is located outside the image block where the pixel to be predicted is located, and the predicted pixel located adjacent to the pixel to be predicted is located inside the image block where the pixel to be predicted is located. For example, the adjacent pixel located above the pixel to be predicted is a predicted pixel in which a luminance component or a chrominance component has been predicted in the image block to be processed in which the pixel to be predicted is currently located, and the adjacent pixel located on the left side of the pixel to be predicted is a reconstructed pixel located adjacent to the pixel to be predicted among the image blocks (i.e., reconstructed blocks) to which the reconstruction process has been performed. For details, reference may be further made to the relevant paragraphs correspondingly described in fig. 11 c. For another example, the adjacent pixel located on the left side of the pixel to be predicted is a predicted pixel in which a luminance component or a chrominance component has been predicted in the image block to be processed in which the pixel to be predicted is currently located, and the adjacent pixel located above the pixel to be predicted is a reconstructed pixel located adjacent to the pixel to be predicted among the image blocks (i.e., reconstructed blocks) to which the reconstruction process has been performed.

Alternatively, the processing device may obtain or determine at least one encoded image block from the same frame of image in which the pixel to be predicted is located (or from a previous frame of image or a subsequent frame of image in which the pixel to be predicted is located) according to a rate distortion optimization algorithm. Alternatively, a candidate image block may be determined as an encoded image block from the at least one candidate image block according to a rate distortion optimization algorithm. Alternatively, at least one candidate image block may be determined from at least one candidate image block according to a rate distortion optimization algorithm, and an image block obtained by subjecting the at least one candidate image block to a fusion process may be used as an encoded image block (hereinafter referred to as a target image block). For example, the processing device may sequentially calculate, within the same frame image (i.e., the current image) or the reference image in which the pixel to be predicted is located, a rate-distortion cost of each of at least one candidate image block having the same size in the encoding region as the current image block to be processed in which the pixel to be predicted is located, so as to take one candidate image block having the smallest rate-distortion cost as a target image block for acquiring or determining the reference pixel of the known chrominance component or luminance component. The above-described target image block is also referred to as a reference image block hereinafter. For another example, at least one candidate image block with the smallest rate-distortion cost may be obtained in the above example, and the target image block obtained by performing fusion processing on the at least one candidate image block may be further used to obtain or determine a reference pixel of a known chrominance component or luminance component.

Optionally, the processing device may further determine at least one encoded image block from the encoded at least one image block according to an image matching algorithm. Optionally, in the same frame of image where the pixel to be predicted is located (or may be a previous frame of image or a subsequent frame of image of the image where the pixel to be predicted is located), the processing device may sequentially calculate, by using an image matching algorithm, a matching degree between at least one candidate image block which has the same size and is coded and is located in the coded region and a current image block to be processed where the pixel to be predicted is located, and thus, take one candidate image block with the highest matching degree as a target image block, where the candidate image block is used for acquiring or determining a reference pixel of a known chrominance component or a luminance component.

Alternatively, the above image matching algorithm used by the processing device may be a SAD algorithm or a SATD algorithm. The SAD algorithm or the SATD algorithm may be a gray scale based template matching algorithm.

The block to be predicted is the image block where the pixel to be predicted is located. Optionally, the image processing method uses pixels related to the periphery of the pixel to be predicted as reference pixels to a large extent, and is suitable for a scene with high degree of correlation between the pixel to be predicted and the surrounding pixels.

In this embodiment, the image processing method performs prediction by using the pixel in the block to be predicted as the reference pixel and/or the pixel outside the block to be predicted by the processing apparatus, so that the prediction accuracy can be effectively improved when the same color component as the chroma component of the pixel predicts the pixel to be predicted.

Third embodiment

optionally, the reference pixel includes at least one first pixel.

Alternatively, the processing device may acquire or determine a first pixel of a known luminance component or chrominance component among the adjacent pixels of the pixel to be predicted, thereby predicting the pixel to be predicted of the luminance component and/or the chrominance component using the pixel value of the one first pixel. Optionally, the processing device may further obtain or determine at least one first pixel of which the luminance component or the chrominance component is known at the same time among neighboring pixels of the pixel to be predicted, so as to predict the luminance component and/or the chrominance component of the pixel to be predicted using the respective pixel value of the at least one first pixel.

Optionally, the pixel to be predicted includes at least one second pixel or a plurality of third pixels.

Alternatively, in one prediction process, the processing device may predict only the luminance component and/or the chrominance component of a single second pixel based on the pixel value of the reference pixel of the at least one known luminance component or chrominance component. Alternatively, in one prediction process, the processing device may also predict the luminance component and/or the chrominance component of each of the plurality of third pixels at the same time based on the pixel value of the reference pixel of the above at least one known luminance component or chrominance component. The types of the second pixel and the third pixel described above may be the same or different. Alternatively, in one prediction process, only the above-described second pixels, or only the above-described plurality of third pixels may be predicted.

Optionally, the image processing method may further include at least one of:

the plurality of third pixels are adjacent to each other;

the combination of the plurality of third pixels is rectangular;

Alternatively, as shown in fig. 7a, 7b and 7c, when the processing apparatus predicts only the second pixel of one individual luminance component based on the pixel value of the first pixel of the above-mentioned at least one luminance component, the combination of the at least one first pixel and the one individual second pixel based on the adjacent positional relationship may be rectangular. The processing device may predict only the second pixel of the one individual chrominance component based on the pixel value of the first pixel of the at least one chrominance component, and the combination of the at least one first pixel and the one individual second pixel based on the adjacent positional relationship may be rectangular.

Optionally, when the processing device predicts the luminance component of each of the plurality of third pixels simultaneously according to the pixel value of the first pixel of the at least one luminance component, each of the plurality of third pixels is adjacent to any one of the other third pixels in position, and the combination formed by the plurality of third pixels based on the adjacent positional relationship may also be rectangular. Optionally, when the processing device predicts the chrominance components of each of the plurality of third pixels simultaneously according to the pixel value of the first pixel of the at least one chrominance component, each of the plurality of third pixels is adjacent to any other third pixel in position, and the combination formed by the plurality of third pixels based on the adjacent positional relationship may also be rectangular.

Alternatively, when predicting the plurality of third pixels according to the pixel values of the first pixels, the processing apparatus may predict the plurality of third pixels using the same prediction model or one prediction model, and the prediction results of the plurality of pixels are the same. Alternatively, the plurality of third pixels may be predicted using equation (1) shown below. For example, if the number of the first pixels is plural, the plural first pixels may be used as the inputs of the formula, i.e., luma1 to LumaN. According to formula (1), a predicted value PreLuma is determined and used as a predicted value of the plurality of third pixels. Optionally, the prediction results of the plurality of third pixels are all the same. For another example, if the number of first pixels is one, the plurality of first pixels may be used as the input Luma1 of the formula. At this time, the formula (1) has only one input. According to formula (1), a predicted value PreLuma is determined and used as a predicted value of the plurality of third pixels. Similarly, the prediction results of the plurality of third pixels are the same.

In the above embodiment, the plurality of third pixels are located in the pixel combination to be predicted. Alternatively, as shown in fig. 8, in the case where the current image block to be processed (also referred to as a block to be predicted) is in a 16×16 format, the processing device may divide the block to be predicted into 4 combinations: a pixel combination to be predicted A, a pixel combination to be predicted B, a pixel combination to be predicted C and a pixel combination to be predicted D. Thus, each of the combinations of pixels to be predicted includes 4 pixels to be predicted (in this case, each pixel to be predicted is referred to as a third pixel). Alternatively, the processing device may use the same reference pixels for prediction for each pixel combination to be predicted.

Alternatively, in the case where the current image block to be processed (also referred to as a block to be predicted) is in 4*4 format, if the processing apparatus still divides the block to be predicted into 4 combinations. Each pixel to be predicted combination will then each comprise only 1 pixel to be predicted (each pixel to be predicted is now called the second pixel).

Alternatively, as shown in fig. 9a to 9f, taking a first pixel with a reference pixel as a plurality of known luminance components, which is also referred to as a reference luminance pixel or a reference luminance pixel, and the pixel to be predicted is a pixel to be predicted combination formed by a plurality of third pixels as an example, the positional relationship between the plurality of reference luminance pixels and the pixel to be predicted combination includes: the plurality of reference luminance pixels are one or several rows (columns) of luminance pixels above, to the left, and/or above-right of the combination of pixels to be predicted. Optionally, in different prediction modes, the range of the horizontal coordinates of the plurality of first pixels may or may not exceed the right boundary of the horizontal coordinates of the combination of pixels to be predicted; the range of the vertical coordinates of the plurality of first pixels may or may not exceed the lower boundary of the vertical coordinates of the combination of pixels to be predicted. For example, as shown in fig. 9a, the range of the horizontal coordinates of the plurality of first pixels does not exceed the right boundary of the horizontal coordinates of the pixel combination to be predicted; as shown in fig. 9b, the range of the horizontal coordinates of the plurality of first pixels exceeds the right boundary of the horizontal coordinates of the pixel combination to be predicted. Alternatively, in different prediction modes, the first pixels above the combination of pixels to be predicted may be one or more rows; and/or the first pixel to the left of the pixel combination to be predicted may be one or more columns. For example, as shown in fig. 9c, the first pixel above the combination of pixels to be predicted is a plurality of rows; as shown in fig. 9d, the first pixel to the left of the pixel combination to be predicted may be one or more columns. Alternatively, the plurality of reference luminance pixels are the rows of luminance pixels above the pixel combination to be predicted as shown in fig. 9c, and the plurality of reference luminance pixels are the columns of luminance pixels to the left of the pixel combination to be predicted as shown in fig. 9 d.

Alternatively, as shown in fig. 10, since the processing apparatus encodes and decodes the images in raster scan order, one or more pixels to be predicted, which are adjacent in position, constitute a combination of pixels to be predicted, and a luminance pixel at the lower left (labeled "in the figure:"sample of) is not predictable as a reference luminance pixel (i.e., the first pixel)Luminance components and/or chrominance components of the pixel combination to be predicted. />

Alternatively, when the processing apparatus determines the reference pixel among the adjacent pixels of the pixel to be predicted, the processing apparatus may determine, in the reconstruction region, a reconstructed pixel adjacent to the pixel to be predicted in position as the reference pixel according to the position of the pixel to be predicted in the current image block to be processed (also referred to as a block to be predicted), and/or determine, in the block to be predicted, a predicted pixel adjacent to the pixel to be predicted in position and having obtained or determined the predicted value as the reference pixel. Alternatively, as shown in fig. 11a, taking an example that the processing device determines a reference pixel (also referred to as a reference luminance pixel) of a known luminance component, if a pixel to be predicted (illustrated as a pixel sample to be predicted) is located on an upper left boundary of a block to be predicted, or a rectangle formed by combining pixels to be predicted is located above and left of the block to be predicted, the processing device may take a reconstructed pixel located above or left of the pixel to be predicted and in the image reconstruction region as the reference pixel. Alternatively, as shown in fig. 11b, if a pixel to be predicted (illustrated as a pixel sample to be predicted) is located at the left side edge of the block to be predicted, the processing device may simultaneously take the reconstructed pixel that is located at the left side of the pixel to be predicted and in the reconstructed region, and/or the predicted pixel that is located above the pixel to be predicted and also in the block to be predicted, as the reference pixel. Alternatively, when the pixel to be predicted is located at the left boundary of the block to be predicted, the processing apparatus may further take only the reconstructed pixel located in the image reconstruction region to the left of the pixel to be predicted as the reference pixel. Optionally, when the pixel to be predicted is located at the upper boundary of the block to be predicted, the processing device may simultaneously take the reconstructed pixel above the pixel to be predicted and located in the reconstructed region, and/or, the predicted pixel to the left of the pixel to be predicted and located in the block to be predicted, as the reference pixel. Alternatively, the processing device may take only reconstructed pixels above the pixels to be predicted and located in the reconstruction region as reference pixels when the pixels to be predicted are located at a boundary above the block to be predicted. Alternatively, as shown in fig. 11c, if a pixel to be predicted (illustrated as a pixel sample to be predicted) is located inside the block to be predicted, the processing device takes only the predicted pixel located in the block to be predicted and adjacent to the pixel to be predicted as the reference pixel.

Optionally, the above reconstructed region is a region formed by reconstructed pixels in the process of encoding and decoding the image by the processing device. Alternatively, the processing device may select the reconstruction pixels above or to the left of the block to be predicted to form the reconstruction region, in which case the shape of the reconstruction region may be an inverted "L" shape. Alternatively, the processing device may select only the reconstruction pixels above the block to be predicted to form the reconstruction region, or select only the reconstruction pixels to the left of the block to be predicted to form the reconstruction region, where the shape of the reconstruction region may be rectangular.

Optionally, the vector information includes a block vector or a motion vector.

Alternatively, when the processing device predicts the to-be-predicted pixel of the chrominance component and/or the luminance component in the image block to be currently processed based on the reference pixel of the known chrominance component or luminance component, the processing device may first acquire or determine the reference pixel of the known chrominance component or luminance component based on the block vector information within the same frame image in which the to-be-predicted pixel is located. Alternatively, the processing device may determine, using the block vector information, a reconstructed pixel in the reconstructed region as a reference pixel within the same frame of image in which the pixel to be predicted is located, to predict the chrominance component and/or the luminance component of the pixel to be predicted.

Alternatively, the processing apparatus may determine the best block vector BV (Block Vector) of the image block (block to be predicted) currently to be processed and the reference block corresponding to the best block vector by block search or block matching as the encoding end. The current image block to be processed is the image block where the pixel to be predicted is located. In this embodiment, the reference block corresponding to the best block vector is the encoded image block determined by the block mentioned above. As shown in fig. 12, the position information of the reference block corresponding to the block vector is (x 1, y 1), and the position information of the block to be predicted is (x 0, y 0), where x1=x0+x'; y1=y0+y ', block vector is (x ', y '). Alternatively, the process of the processing device performing the block search is similar to the process of the processing device performing the motion estimation search in image prediction by means of inter prediction. The difference between the process of block searching and the process of motion estimation searching is that the best block vector involved in performing the block searching at the encoding end processing device is the block vector corresponding to the reference block that is best matched to the block to be predicted in the current frame, which is obtained using the block matching algorithm in the reconstruction area of the frame where the pixel to be predicted is currently located, and the motion vector involved in performing the motion estimation searching at the encoding end processing device is the motion vector corresponding to the reference block that is best matched to the block to be predicted in the current frame, which is found using the block matching algorithm in the reference frame. Alternatively, the reference frame is an inter-frame reference frame, which is one or more frames used in the encoding process for predicting a current image (current frame). For example, the reference frame may be a previous frame, a subsequent frame, or other future or past frame of the current image. Alternatively, the reference Frame may be a P Frame (Predicted Frame) or a B Frame (Bi-directional Predicted Frame). Alternatively, the processing device may be used as a decoding end to determine the reference block corresponding to the block vector according to decoding the block vector from the bitstream.

Optionally, after determining the best block vector of the block to be predicted and the reference block corresponding to the best block vector, the processing device may determine the location information of the reference pixel in the reference block based on the location information of the block vector and the pixel combination to be predicted (formed by combining a plurality of adjacent third pixels). As shown in fig. 13, if the size of the pixel combination to be predicted is 2×2, the size of the reference luminance pixel combination made up of reference pixels (e.g., the first pixels) is 4*4, the position information of the pixel combination to be predicted is (x 01, y 01), the block vector is (x ', y'), and the position information of the reference luminance pixel combination is (x01+x '-1, y01+y' -1). In the above example, the position information of the reference luminance pixel combination is represented by a coordinate, which is a coordinate position corresponding to a pixel located in the upper left corner in the reference luminance pixel combination.

Optionally, the reference luminance pixel combination is rectangular in shape, and the reference luminance pixel combination includes a block vector matching block corresponding to the pixel combination to be predicted. The size of the block vector matching block corresponding to the pixel combination to be predicted is the same as the size of the pixel combination to be predicted, and the position information of the block vector matching block is (x01+x ', y01+y'). The reference luma pixel combination includes adjacent reference samples located around the block vector matching block in addition to the block vector matching block. The adjacent reference samples are not only the reference samples above and to the left of the block vector matching block, but also the reference samples below and to the right of the block vector matching block. In this embodiment, since the pixel combination to be predicted can be predicted by reference sampling below and to the right of the block vector matching block, the accuracy of prediction is improved.

Alternatively, in addition to the use of reconstructed pixels in the reference region adjacent to the block to be predicted and/or predicted pixels in the block to be predicted mentioned in the embodiment shown in fig. 11a to 11c, reconstructed pixels in the reference block mentioned in the embodiment shown in fig. 13 may be used. For example, some pixels in the reference pixel combination may be utilized. To distinguish reconstructed pixels in a reference block corresponding to a block vector from reconstructed pixels in a reference region adjacent to the block to be predicted. The reconstructed pixels in the reference region adjacent to the block to be predicted are hereinafter referred to as first reference pixels, the reconstructed pixels in the reference block corresponding to the block vector are referred to as second reference pixels, or the pixels determined or obtained using the reconstructed pixels in the reference block corresponding to the block vector are referred to as second pixels. It should be noted that, alternatively, the first reference pixel and/or the second reference pixel may be other regions or reconstructed pixels determined by other methods and different from each other. The first reference pixel and/or the second reference pixel are both reference pixels. If the first reference pixel or the second reference pixel is a pixel of a luminance component or a luminance pixel, it may be referred to as a reference luminance pixel; and/or, if the first reference pixel or the second reference pixel is a pixel of a chrominance component or a chrominance pixel, it may be referred to as a reference chrominance pixel.

As can be seen from the above, optionally, the second reference pixel may be determined or derived in the reference block by a block vector. Alternatively, as shown in fig. 14a, the reference pixels may include only the first reference pixel. The first reference pixel is a reference luminance pixel a, which is located above or to the left of the pixel to be predicted or the combination of luminance pixels to be predicted. The reference luminance pixel a is a reconstructed luminance pixel in a reference region adjacent to the block to be predicted or a prediction pixel in the block to be predicted. Alternatively, the reference pixels may include the first reference pixel and/or the second reference pixel. Similar to fig. 14a, the first reference pixel is a reference luminance pixel, which is located above or to the left of the luminance pixel to be predicted or the combination of luminance pixels to be predicted; and/or the second reference pixel is a reference luminance pixel, which is a reconstructed pixel in the reference block corresponding to the block vector. Alternatively, the first reference pixel may be the reference luminance pixel a shown in fig. 14B, and/or the second reference pixel may be the reference luminance pixel B shown in fig. 14B. Optionally, the reference luminance pixel B is a luminance pixel obtained by reconstructing a pixel in a reference block corresponding to the block vector. Alternatively, the reconstructed pixels in the reference block corresponding to the block vector may be filled with pixels below or to the right of the luminance pixel to be predicted or the luminance pixel combination to be predicted. Alternatively, the pixel values of the luminance pixels at the corresponding positions in the reference block corresponding to the block vector may be assigned to the pixels around the luminance pixel combination to be predicted. For example, the pixel value assignment of the neighboring reference pixel located below and/or right neighboring of the block vector matching block among the neighboring reference pixels shown in fig. 13 is assigned to the pixel below and/or right neighboring of the luminance pixel combination to be predicted, respectively. According to the raster scanning sequence, when the pixel combination to be predicted is predicted, the pixel value of the pixel below and/or the pixel to the right of the pixel combination to be predicted is unknown, and the filling operation in the embodiment can determine the pixel value of the pixel below and/or the pixel to the right of the pixel combination to be predicted when the pixel combination to be predicted is predicted, so that the accuracy of prediction is improved. It should be noted that, the pixel value of the lower and/or right pixel of the combination of pixels to be predicted obtained by the filling method is not necessarily the final predicted pixel value of the lower and/or right pixel of the combination of pixels to be predicted, and therefore, after determining the pixels to be predicted in the combination of pixels to be predicted, the pixel value of the lower and/or right pixel of the combination of pixels to be predicted may be determined by other prediction methods, and the pixel value obtained by other prediction methods may be used as the final predicted pixel value of the pixels. Alternatively, the pixel values of the pixels below and/or to the right of the combination of pixels to be predicted obtained by the filling manner are not necessarily the final predicted pixel values, but are equivalent to the pixel values of the pixels below and/or to the right of the combination of pixels to be predicted that are virtually obtained when the combination of pixels to be predicted is predicted, so the reference luminance pixel B shown in fig. 14B may be referred to as a virtual reference luminance pixel.

As shown in fig. 15, the processing apparatus obtains reference luminance pixels B by filling the pixel values of the reconstructed luminance pixels in the luminance pixels corresponding to the positions after the reconstructed luminance pixels obtained or determined by the block vector. Alternatively, the processing device may also obtain or determine reconstructed luminance pixels by means of the block vectors, and then combine these reconstructed luminance pixels as filled luminance pixels B with reference luminance pixels a to obtain all reference luminance pixels. Optionally, the location correspondence refers to: if one reconstructed brightness pixel is positioned on the right side of the block vector matching block, filling the brightness pixel at the position on the right side of the pixel combination to be predicted by using the reconstructed brightness pixel; and/or if one reconstructed luminance pixel is positioned below the block vector matching block, filling the luminance pixel at a position below the pixel combination to be predicted with the reconstructed luminance pixel. Alternatively, the processing device may also directly take the reference luminance pixels a (reconstructed luminance pixels in a reference region adjacent to the block to be predicted or predicted pixels in the block to be predicted) and/or the reconstructed luminance pixels obtained or determined by the block vector as the reference luminance pixels, so that these reference luminance pixels are substituted into the prediction model to be predicted to obtain the pixel values of the pixels to be predicted later. Alternatively, if the upper and/or left pixels of the pixel combination to be predicted are not available, the above-described filling manner may also be used to obtain pixel values for the upper and/or left pixels of the pixel combination to be predicted, so as to perform prediction processing for the pixel combination to be predicted.

Optionally, the processing device may further obtain or determine a reference pixel of the known chrominance component or luminance component in a reference image (also referred to as a reference frame) of the image in which the pixel to be predicted is located according to a motion vector information, so as to predict the chrominance component and/or luminance component of the pixel to be predicted. Alternatively, the reference pixels may be reconstructed pixels of the reference image. Alternatively, the reference image may be a previous frame image, a subsequent frame image, or other future or past frame of the image in which the pixel to be predicted is located.

Alternatively, the processing apparatus as the encoding end may perform a motion search in the reference image to determine the motion vector MV (Move Vector) of the block to be predicted and the reference block to which the motion vector corresponds. Alternatively, as shown in fig. 16, the position information of the reference block corresponding to the motion vector is (xb, yb), and the position information of the block to be predicted is (xa, ya). Wherein xb=xa+x'; yb=ya+y″ and the motion vector is (x ", y"). Alternatively, the processing device may be used as a decoding end, i.e. may directly determine the reference block corresponding to the motion vector according to the motion vector decoded from the bitstream.

Optionally, after determining the motion vector MV of the block to be predicted and the reference block corresponding to the motion vector, the processing device may determine the location information of the reference pixel in the reference block according to the location information of the motion vector MV and the pixel to be predicted (the combination of the single second pixel or the plurality of third pixels). Alternatively, as shown in fig. 17, if the size of the combination of the plurality of third pixels (i.e., the pixel combination to be predicted) is 2×2, the size of the combination of the plurality of first pixels (i.e., the reference luminance pixel combination) is 4*4, the position information of the pixel combination to be predicted is (xa 1, ya 1), the motion vector is (x ", y"), and the position information of the reference luminance pixel combination is (xa1+x "-1, ya 1+y" -1). Optionally, the position information of the reference luminance pixel combination is a coordinate position of an upper left corner pixel in the reference luminance pixel combination. Alternatively, the reference luminance pixel combination is rectangular in shape and contains a motion vector matching block of the pixel combination to be predicted. The size of the motion vector matching block of the pixel combination to be predicted is the same as the size of the pixel combination to be predicted, and the position information of the motion vector matching block is (xa1+x″, ya1+y″). Optionally, the position information of the motion vector matching block is a coordinate position of an upper left corner pixel in the motion vector matching block. The reference luminance pixel combination includes neighboring reference samples located around the motion vector matching block in addition to the motion vector matching block. The adjacent reference samples are not only the reference samples above and to the left of the motion vector matching block, but also the reference samples below and to the right of the motion vector.

In the present embodiment, the image processing method predicts the luminance component and/or the chrominance component by using the combination of at least one second pixel or a plurality of third pixels in the process of predicting the luminance component and/or the chrominance component for the pixel to be predicted by the processing apparatus, so that a plurality of results can be determined in one prediction process.

And/or when the pixel to be predicted is a plurality of third pixels, the plurality of third pixels are adjacent to each other or can be combined into a rectangle, and the processing device obtains respective predicted values (namely, the filter coefficients with the same value are utilized in the prediction model) through the same prediction model for the pixels in the rectangle combined, so that the image processing method predicts based on the unit of a rectangular block, and compared with the traditional pixel-by-pixel prediction mode, the calculation efficiency can be effectively improved, and the time consumed for image encoding and decoding is reduced.

Fourth embodiment

In the present embodiment, the image processing method will be described with the processing apparatus as the execution subject. On the basis of any of the above embodiments, a pixel to be predicted is determined or obtained according to the first parameter and the reference pixel.

Optionally, the first parameter is a filter coefficient.

Optionally, after determining the reference pixel of the pixel to be predicted by the above embodiments, the processing device may determine a prediction model for predicting the chrominance component and/or the luminance component for the pixel to be predicted, so as to perform weighting processing on the pixel value of the reference pixel by using the filter coefficient in the prediction model, so as to determine or obtain a prediction result of the chrominance component and/or the luminance component of the pixel to be predicted.

Optionally, before or in parallel with determining the reference pixel of the pixel to be predicted by the above embodiments, the processing device may further determine a prediction model for predicting the chrominance component and/or the luminance component for the pixel to be predicted, and determine or obtain the above prediction result by using the prediction model and the pixel value of the reference pixel.

Alternatively, the predictive model employed by the processing device may have a form as shown in equation 1 below:

pre luma=f (Luma 1, luma2, luma3, lumaN., lumaN) (formula 1

The Luma 1-LumaN are luminance pixels input by the processing device when predicting the luminance component of the pixel to be predicted, that is, the reference luminance pixels in the above embodiments. The premama is a pixel to be predicted (a luminance pixel to be predicted).

Alternatively, the prediction model employed by the processing device may also have a form as shown in the following equation 2:

pretuma=c0 x luma1+c1 x luma2+c2 x luma3+c3 x luma4+c4 x luma5+ c5+c6 p2+c7 p3+c8 p4+c9 p5+c10B (formula 2)

When the prediction model is a convolution filter, c 0-c 10 are filter coefficients, luma 1-Luma 5 are the reference brightness pixels, and P1-P5 are nonlinear terms. Alternatively, P1-P5 may be the square of the pixel value of the reference luminance pixels Luma 1-Luma 5. Alternatively, P1 may be the square of the pixel value of the reference luminance pixel Luma1 and scaled to the bit depth range. That is, p1= (luma1×luma1+midval) > > bitDepth; bitDepth is the bit depth to which the samples correspond and "> >" is the right shift symbol. For example, for 10-bit video content, P may be calculated by:

P = ( Luma1*Luma1 + 512 )>>10。

wherein P2-P5 are squares of pixel values of the reference luminance pixel Luma1, optionally scaled to a bit depth range.

Optionally, the offset term B may also be included in the above formula 2. Alternatively, the bias term B may be or other predetermined value (e.g., 512 may be set for 10-bit video).

Optionally, when the prediction model corresponding to the prediction mode adopted by the processing device is in the form shown in the above formula (2), the processing device needs to determine the filter coefficient in the prediction model for predicting the chrominance component and/or the luminance component of the pixel to be predicted subsequently.

Optionally, the image processing method further comprises at least one of:

Optionally, the first parameter is a filter coefficient and the first reference pixel is a coefficient-determining reference pixel. The coefficient determination reference pixels are used to determine filter coefficients.

Optionally, the manner in which the processing device is configured to determine the filter coefficients among the above-mentioned prediction models corresponds to at least one prediction model. That is, the manner in which the processing device determines the filter coefficients is different when different prediction modes are employed. Alternatively, the prediction modes employed by the processing device may be a conventional intra prediction mode, a block vector based intra prediction mode, and an inter prediction mode. Alternatively, the processing device may set an index or flag for the above-mentioned different prediction models in advance, and maintain a prediction mode list in itself or in a storage terminal/cloud platform to which itself is connected, and store the at least one index or flag in the prediction mode list. In this way, the processing device may determine the currently adopted prediction mode by acquiring the index or the flag and comparing the index or the flag in the prediction mode list, and further determine or obtain the filter coefficient and the predicted pixel to be predicted of the chrominance component and/or the luminance component in the prediction model according to the determination mode of the filter coefficient corresponding to the prediction mode, thereby obtaining the corresponding prediction result.

Optionally, the determining manner of the filter coefficients corresponding to the different prediction modes adopted by the processing device may include determining or obtaining the filter coefficients by determining the reference pixel according to the coefficients of the first component and/or the second component. Alternatively, when any of the above prediction modes is adopted, the processing device may select the coefficients of the corresponding number of first components and/or second components to determine the reference pixels based on the structure of the adopted prediction model, so that the filter coefficients in the prediction model are determined or derived by the reference pixels based on the coefficients.

Optionally, the coefficient determining reference pixel is located in at least one of the following areas:

a reference image block determined by the vector information;

Optionally, when determining the filter coefficient in the prediction model, the processing device may determine, as the reference area, the adjacent area of the image block where the pixel to be predicted is located, the adjacent area of the reference image block corresponding to the image block where the pixel to be predicted is located, the reference image block determined by the vector information (including the block vector or the motion vector) and/or the non-adjacent area of the image block where the pixel to be predicted is located, to determine at least one coefficient in the reference area according to a predetermined rule to determine the reference pixel, and then determine, based on the coefficient, the reference pixel to determine or obtain the filter coefficient in the prediction model.

Optionally, when determining the filter coefficient in the prediction model, the processing device may determine, as the reference area, an adjacent area of a co-located block of an image block where a pixel to be predicted is located, an adjacent area of a co-located block of a reference image block corresponding to the image block where the pixel to be predicted is located, a reference image block determined by the vector information (including a block vector or a motion vector) and/or a non-adjacent area of the co-located block of the image block where the pixel to be predicted is located, to determine at least one coefficient in the reference area according to a predetermined rule to determine a reference pixel, and then determine or obtain the filter coefficient in the prediction model based on the coefficient to determine the reference pixel. The co-located block of the reference picture block may be a block that is co-located with the reference picture block in another picture. For example, if a chroma reference image block is determined in a chroma image, the co-located block of the chroma reference image block may be the same chroma image block as the chroma reference image block in the luma image corresponding to the chroma image. Similarly, the co-located block of the image block where the pixel to be predicted is located is the image block with the same position as the image block where the pixel to be predicted is located.

Optionally, the number of reference pixels determined by the processing device is related to the structure of the prediction model. Alternatively, when the pixel combination to be predicted (the combination of the plurality of third pixels) and the reference luminance pixel corresponding to the pixel combination to be predicted (the plurality of first pixels in the luminance component prediction for the pixel to be predicted are combined into the inverted "L" shape) shown in fig. 9a are used, if the prediction model corresponding to the formula (2) is adopted, the number of coefficient determination reference pixels for determining the filter coefficient is 11. Alternatively, the processing device may determine other numbers of coefficients to determine the reference pixel based on different design requirements of the actual application.

Optionally, the embodiment of determining other pixel combinations to be predicted and corresponding reference luminance pixels in fig. 9b to 9f by the processing device is the same as the embodiment of determining the pixel combinations to be predicted and corresponding reference luminance pixels in fig. 9a, and will not be described in detail herein.

Alternatively, the above-described predetermined rule may determine at least one pixel to be selected among the pixels of the reference region, and then determine the reference pixel using the pixel of the average value of the at least one pixel to be selected as the coefficient. Alternatively, taking prediction of the luminance component for the pixel to be predicted as an example, the processing device may determine the reference luminance pixel using, as coefficients, the minimum luminance pixel, the second smallest luminance pixel, the maximum luminance pixel, the second largest luminance pixel, and a luminance pixel close to the average of the minimum luminance pixel and the maximum luminance pixel among the reference region.

Alternatively, the processing device may of course also employ other forms of predetermined rules, such as directly selecting luminance pixels at predetermined locations within the reference area as coefficients to determine the reference pixels, based on different design requirements of the actual application.

Optionally, the processing device further determines the coefficient determination reference pixel as a first coefficient determination reference pixel after determining the coefficient determination reference pixel, and determines at least one second coefficient determination reference pixel according to the at least one first coefficient determination reference pixel.

Optionally, the processing device uses the first coefficient determination reference pixel as the premama in the above formula (1) or formula (2) after using the above coefficient determination reference pixel as the first coefficient determination reference pixel, and determines the position of the reference pixel according to the position of the first coefficient determination reference pixel and the second coefficient determination reference pixel. As shown in fig. 18a to 18d, the processing device identifies a first coefficient determination reference pixel as a first reference luminance pixel in the prediction process of the luminance component for the pixel to be predicted, thereby determining the position of a second coefficient determination reference pixel (also referred to as a second reference luminance pixel) based on the position of the first luminance reference sample. Alternatively, as shown in fig. 18a, the processing apparatus may determine the upper right luminance pixel, the upper Fang Liangdu pixel, the upper left luminance pixel, the lower left luminance pixel of the first coefficient determination reference pixel as the second coefficient determination reference pixel. Alternatively, as shown in fig. 18b, the processing apparatus may also use an upper luminance pixel, an upper left luminance pixel, a luminance pixel on the left side of the upper left luminance pixel, a left second luminance pixel (a luminance pixel on the left side of the left luminance pixel), and a luminance pixel below the left second luminance pixel of the first reference luminance pixel as the second reference luminance pixel. Alternatively, as shown in fig. 18c, the processing apparatus may further use the second luminance pixel above the first reference luminance pixel (the luminance pixel above the upper luminance pixel), the luminance pixel to the right of the upper second luminance pixel, the luminance pixel to the left of the upper second luminance pixel, and the luminance pixel above the left luminance pixel as the second reference luminance pixel. Alternatively, as shown in fig. 18d, the processing apparatus may further use the upper second luminance pixel (luminance pixel above the upper luminance pixel) of the first reference luminance pixel, the luminance pixel to the left of the upper second luminance pixel, the upper second luminance pixel to the left of the upper second luminance pixel (second luminance pixel to the left of the upper second luminance pixel), the left second luminance pixel (luminance pixel to the left of the left second luminance pixel), and the luminance pixel above the left second luminance pixel as the second reference luminance pixel.

Optionally, the processing device determines the reference pixel by using at least one first coefficient determined in the same manner, and the positional correspondence relationship between each of the reference pixels and the corresponding at least one second coefficient determined reference pixel is the same. Alternatively, the positional relationship between each of the 11 first reference luminance pixels determined by the processing device and the corresponding 5 second reference luminance pixels is shown in fig. 18 a. Alternatively, the positional relationship between each of the 11 first reference luminance pixels and the corresponding 5 second reference luminance pixels determined by the processing device is shown in fig. 18 b. Alternatively, the positional relationship between each of the 11 first reference luminance pixels and the corresponding 5 second reference luminance pixels determined by the processing device is shown in fig. 18 c. Alternatively, the positional relationship between each of the 11 first reference luminance pixels and the corresponding 5 second reference luminance pixels determined by the processing device is shown in fig. 18 d.

Optionally, the processing device may determine the filter coefficients in the prediction model from the at least one first coefficient determination reference pixel and the at least one second coefficient determination reference pixel after determining the at least one first coefficient determination reference pixel and the at least one second coefficient determination reference pixel.

Optionally, it is assumed that the processing device determines 11 first coefficient determination reference pixels (i.e., first reference luminance pixels RefLuma 1-RefLuma 11), and determines 5 x 11 second coefficient determination reference pixels (i.e., second reference luminance pixels Luma1, 1-Luma 1,11, luma2, 1-Luma 2,11, luma3, 1-Luma 3,11, luma4, 1-Luma 4,11, luma5, 1-Luma 5, 11) based on the 11 first coefficient determination reference pixels. Thus, the processing apparatus can obtain the following rectangular form by substituting these second reference luminance pixels into the above-described formula (2):

PreLumai=c0*Lumai1 + c1*Lumai2+ c2*Lumai3+ c3*Lumai4 + c4*Lumai5+ c5*Pi1+ c6*Pi2+ c7*Pi3+ c8*Pi4+ c9*Pi5+ c10*B;i=1,2, ...,11

that is to say,formula (3)

Wherein, reluma is [ reluma1, reluma2, ], reluma11] ^T Which is a vector of predicted luminance pixels determined by the second reference luminance pixel.Is [ c0, c1 ], c9, c10] ^T Which is the filter coefficient (filter coefficients). L is a matrix of second reference luminance pixels.

The processing device then determines the filter coefficients in equation (3) above using a multiple linear regression method.

Alternatively, the processing device may determine the filter coefficients in the above formula (2) by minimizing a loss function of the predicted luminance pixels represented by the first reference luminance pixel and the second reference luminance pixel. That is to say,

Formula (4)

Wherein the reference luminance pixel RefLuma is [ RefLuma1, refLuma2, ], refLuma11] ^T Which is a vector of the first reference luminance pixels described above.

Alternatively, when the derivative of the above-mentioned loss function is 0, a minimum value is taken. The corresponding filter coefficient value is the result. The c in the formula (4) is derived to obtain the formula (5) as follows:

formula (5)

Optionally, after the filter coefficient in the above formula (2) is obtained, the processing device may determine a prediction model to perform a subsequent prediction operation of the pixel to be predicted, that is, perform a prediction process according to the pixel value of the at least one reference pixel and the prediction model to determine or obtain a chrominance component prediction result (e.g., a chrominance value) and/or a luminance component prediction result (e.g., a luminance value) of the pixel to be predicted.

Alternatively, taking the example of performing luminance component prediction for a to-be-predicted pixel combination formed by a plurality of third pixels, the processing device may determine, after determining reference pixels (for example, reference luminance pixels) corresponding to-be-predicted luminance pixels in the to-be-predicted pixel combination, by substituting the reference luminance pixels into the prediction model. Alternatively, the prediction value of the entire block to be predicted may be obtained or determined by sequentially obtaining or determining the luminance pixels to be predicted in the pixel combination to be predicted a to the pixel combination to be predicted D in fig. 8 from left to right and from top to bottom.

Alternatively, when the processing apparatus predicts pixels to be predicted of the chrominance component and/or the luminance component using the intra prediction mode based on the block vector, it is assumed that the processing apparatus is obtaining a prediction model with respect to a reference pixel (for example, a reference luminance pixel combination, or a single reference luminance pixel) as shown in formula (6):

pretuma=c0, luma1+c1, luma2+ & c15, luma16+c16, p2+ & c31, p16+c32, B; formula (6)

The processing device may also determine a reference region of the block to be predicted above and/or to the left of the current block to be processed (also referred to as the block to be predicted). As shown in fig. 19, the reference region is the illustrated reconstruction region. Thereafter, the processing device determines at least one coefficient-determining reference pixel as a first coefficient-determining reference pixel in the reference region according to a predetermined rule (the same number of determined first reference luminance pixels is related to the structure of the prediction model), and then determines at least one second coefficient-determining reference pixel based on the at least one first coefficient-determining reference pixel and the block vector. Whereby the reference pixel is determined from the at least one first coefficient and the reference pixel is determined from the at least one second coefficient to determine a filter coefficient to determine a prediction model.

Alternatively, the processing device may determine at least one coefficient determination reference pixel as the first coefficient determination reference pixel according to a predetermined rule identical or similar to the above. After determining the first coefficient determination reference pixel, a first image block (e.g., the luminance block shown in fig. 19) including the first coefficient determination reference pixel is determined. Alternatively, the processing device may directly determine the position of the reference pixel by using the first coefficient as the position of the first image block, and set the size of the first image block to 2×2, i.e. may determine the first image block. And then, the processing equipment can determine a block vector matching block A corresponding to the first image block and adjacent reference pixels a around the block vector matching block A according to the first image block and the block vector. Thus, the block vector matching block a and the adjacent reference pixel a around the block vector matching block a are used as the second coefficients to determine the reference pixel. Alternatively, the processing device may determine n first coefficient determination reference pixels and their corresponding second coefficient determination reference pixels in turn through the above procedure. Where n is equal to the number of filter coefficients in the prediction model. Alternatively, the processing device may determine the filter coefficients in the prediction model from the at least one first coefficient determination reference pixel and the at least one second coefficient determination reference pixel in the same manner as described above.

Alternatively, when the processing apparatus predicts the pixel to be predicted of the chrominance component and/or the luminance component using the intra prediction mode based on the block vector, the processing apparatus may further determine the reference pixel according to the first coefficient determination reference pixel and the block vector determination second coefficient determination reference pixel after determining the coefficient determination reference pixel based on the above process and determining the reference pixel as the first coefficient. The second coefficient determines that the reference pixel includes both the neighboring reference pixel and the block vector matching block. Optionally, the neighboring reference pixel determines a first portion of the reference pixel for the second coefficient, which is the first neighboring reference pixel neighboring the first coefficient-determining reference pixel, and the block vector matching block determines a second portion of the reference pixel for the second coefficient. Alternatively, the second coefficient determination reference pixel may not include the block vector matching block. Alternatively, the second coefficient determination reference pixel may not include the first neighboring reference pixel neighboring the first coefficient determination reference pixel.

Optionally, when the processing device predicts the chrominance component and/or the luminance component of the pixel to be predicted using the intra prediction mode based on the block vector, the processing device may further determine a first adjacent reference pixel adjacent to the first coefficient determination reference pixel according to the first coefficient determination reference pixel after determining the coefficient determination reference pixel based on the above process and determining the reference pixel as the first coefficient determination reference pixel, and match a second adjacent reference pixel adjacent to the block according to the block vector. As shown in fig. 20, the first coefficient determines the reference pixel as a first reference pixel, the first adjacent reference pixel as an adjacent reference pixel a1, and the second adjacent reference pixel as an adjacent reference pixel a2. Optionally, the first neighboring reference pixel determines a first portion of the reference pixel for the second coefficient, the second neighboring reference pixel determines a second portion of the reference pixel for the second coefficient, and the pixels in the block vector matching block determine a third portion of the reference pixel for the second coefficient. Alternatively, the second coefficient determination reference pixel may include only the first portion and the second portion described above, and not include the third portion. Alternatively, the second coefficient determination reference pixel may include only the first portion and the third portion described above, and not include the second portion. Alternatively, the second coefficient determination reference pixel may include only the second portion and the third portion described above, not the first portion.

Alternatively, the processing device may determine the coefficient determination reference pixel according to the same or similar predetermined rule as above and determine the reference luminance pixel as the first coefficient, and then the processing device determines a first image block including the first coefficient determination reference pixel. Alternatively, the processing device may directly determine the position of the reference pixel by using the first coefficient as the position of the first image block and set the size of the first image block to 2×2, so that the first image block may be determined.

Alternatively, as shown in fig. 20, the processing apparatus may determine the upper right luminance pixel, the upper Fang Liangdu pixel, the upper left luminance pixel, the lower left luminance pixel of the reference pixel (i.e., the first reference luminance pixel) as the first portion of the second coefficient determination reference pixel. I.e. these pixels are regarded as second reference luminance pixels (marked as neighboring reference samples a1 in fig. 20). Thereafter, the processing apparatus determines a block vector matching block a corresponding to the first luminance block and adjacent reference samples a2 around the block vector matching block a from the first image block (labeled as the first luminance block in fig. 20) and the block vector. Thus, the processing device may use the neighboring reference samples a around the block vector matching block a corresponding to the first luminance block as the second portion of the second reference luminance pixel. And/or the processing device may take the block vector matching block a as a third portion of the second reference luminance pixels. Alternatively, the second reference luminance pixel may include only the neighboring reference samples a around the block vector matching block a corresponding to the first luminance block, and not the luminance pixels in the block vector matching block a corresponding to the first luminance block. Alternatively, the processing device may sequentially determine n first reference luminance pixels and their corresponding second reference luminance pixels in the above manner. Where n is equal to the number of filter coefficients in the prediction model.

Alternatively, the processing device may determine the filter coefficients in the prediction model from the at least one first coefficient determination reference pixel and the at least one second coefficient determination reference pixel in the same manner as described above.

Optionally, when the processing device predicts the pixel to be predicted of the chrominance component and/or the luminance component based on the inter-prediction mode, the manner in which the processing device determines the filter coefficient in the above prediction model may be:

above and/or to the left of the current image block to be processed (block to be predicted), a reference region of the block to be predicted is determined. As shown in fig. 21, the reference region is the illustrated reconstruction region. Thereafter, the processing device determines at least one coefficient determination reference pixel in the reference region according to a predetermined rule, and determines the reference pixel as a first coefficient. Optionally, the determined coefficients determine a number of reference pixels and a structural correlation of the prediction model. Finally, the processing device determines at least one second coefficient determination reference pixel based on the at least one first coefficient determination reference pixel and the motion vector, such that the filter coefficients in the prediction model are determined based on the at least one first coefficient determination reference pixel and the at least one second coefficient determination reference pixel.

Alternatively, the processing device may determine the coefficient determination reference pixel as the first coefficient determination reference pixel using a predetermined rule identical or similar to that described above. And still determine the first image block containing the first coefficient determination reference pixel in the same manner as described above. And then, the processing equipment determines a motion vector matching block A ' corresponding to the first image block and adjacent reference samples a ' around the motion vector matching block A ' according to the first image block and the motion vector. And determining a reference pixel using the block vector matching block a ' and adjacent reference samples a ' around the block vector matching block a ' as second coefficients. In this way, n first coefficient determination reference pixels and their corresponding second coefficient determination reference pixels are sequentially determined (n is equal to the number of filter coefficients in the prediction model). Finally, the processing device may be adapted in the same manner as described above to determine the reference pixel from the at least one first coefficient and the at least one second coefficient to determine the filter coefficients in the prediction model.

Alternatively, when the processing device predicts the pixel to be predicted of the chrominance component and/or the luminance component based on the inter-prediction mode, the pixel to be predicted may be only one single second pixel, and when the pixel to be predicted is one single second pixel, the size of at least one reference pixel combination formed by combining the first pixels may be 3*3. Alternatively, the size of the reference pixel combination formed by the reference pixels may be 5*5 or other sizes in different possible embodiments based on different design requirements of practical applications.

Optionally, the coefficient determination reference pixel (e.g., the first coefficient determination reference pixel and/or the second coefficient determination reference pixel) is determined by the processing device in an adjacent region of the image block in which the pixel to be predicted is located, so that the filter coefficient used for performing the prediction processing to obtain the prediction result of the pixel to be predicted is determined based on the coefficient determination reference pixel. In this way, when the texture characteristics of the image block to be predicted and the texture characteristics of the adjacent areas are similar, the prediction result error for the pixel to be predicted can be made smaller by the filter coefficients in the prediction model determined by the adjacent areas.

Alternatively, the reference pixel is determined by the processing device selecting a first coefficient in the neighboring region of the image block where the pixel to be predicted is located, and the reference pixel is determined by determining a second coefficient using the block vector/motion vector and the selected first coefficient. The second coefficient determination reference pixel includes at least one of a block vector matching block and/or a motion vector matching block and its neighboring pixels. And determining a filter coefficient corresponding to a prediction result of the pixel to be predicted by determining a reference pixel based on the coefficient. In this way, when the texture characteristics of the image block to be predicted and the texture characteristics of the image block (i.e., the reference block) determined by the block vector/motion vector are similar, the filter coefficients in the prediction model can be determined by determining the reference pixel through the first coefficient, and the prediction result error for the pixel to be predicted can be made smaller.

Alternatively, the predictive model employed by the processing device may have a form as shown in equation 7 below:

prechroma=f (Chroma 1, chroma2, chroma3,., chromaN) (formula 7

Chroma 1-Chroma N are Chroma pixels input by processing equipment when Chroma component prediction is carried out on pixels to be predicted, namely reference Chroma pixels. The PreChroma is the pixel to be predicted (i.e., the chroma pixel to be predicted).

Alternatively, the predictive model employed by the processing device may have a form as shown in equation 8 below:

prechroma=f (Luma 1, luma2, luma3, lumaN., lumaN) (formula 8

Luma 1-LumaN are luminance pixels input by the processing device when the processing device predicts the chrominance component of the pixel to be predicted, i.e. the reference luminance pixels in the above embodiments. The PreChroma is the pixel to be predicted (i.e., the luminance pixel to be predicted).

It should be noted that, modifications and embodiments similar to those related to the prediction model corresponding to the formula (1) may be applied to the formulas (7) to (8) without departing from the spirit and basic operation principle of the present invention.

Through the technical scheme of the embodiment of the application, the residual error value of the residual error block corresponding to the prediction block can be reduced, so that the compression efficiency in the image coding and decoding process is improved.

The embodiment of the application also provides an intelligent terminal, which comprises a memory and a processor, wherein the memory stores an image processing program, and the image processing program realizes the steps of the image processing method in any embodiment when being executed by the processor.

The embodiment of the present application further provides a storage medium, on which an image processing program is stored, which when executed by a processor, implements the steps of the image processing method in any of the above embodiments.

The embodiments of the intelligent terminal and the storage medium provided in the present application may include all technical features of any one of the embodiments of the image processing method, and the expansion and explanation contents of the description are substantially the same as those of each embodiment of the method, which are not repeated herein.

The present embodiments also provide a computer program product comprising computer program code which, when run on a computer, causes the computer to perform the method in the various possible implementations as above.

The embodiments also provide a chip including a memory for storing a computer program and a processor for calling and running the computer program from the memory, so that a device on which the chip is mounted performs the method in the above possible embodiments.

It can be understood that the above scenario is merely an example, and does not constitute a limitation on the application scenario of the technical solution provided in the embodiments of the present application, and the technical solution of the present application may also be applied to other scenarios. For example, as one of ordinary skill in the art can know, with the evolution of the system architecture and the appearance of new service scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

The steps in the method of the embodiment of the application can be sequentially adjusted, combined and deleted according to actual needs.

The units in the device of the embodiment of the application can be combined, divided and pruned according to actual needs.

In this application, the same or similar term concept, technical solution, and/or application scenario description will generally be described in detail only when first appearing, and when repeated later, for brevity, will not generally be repeated, and when understanding the content of the technical solution of the present application, etc., reference may be made to the previous related detailed description thereof for the same or similar term concept, technical solution, and/or application scenario description, etc., which are not described in detail later. In this application, the descriptions of the embodiments are focused on, and the details or descriptions of one embodiment may be found in the related descriptions of other embodiments. The technical features of the technical solutions of the present application may be arbitrarily combined, and for brevity of description, all possible combinations of the technical features in the above embodiments are not described, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the present application.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or what contributes to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as above, comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to perform the method of each embodiment of the present application.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable devices. The computer instructions may be stored in a storage medium or transmitted from one storage medium to another storage medium, for example, from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage media may be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that contains an integration of one or more available media. Usable media may be magnetic media (e.g., floppy disks, storage disks, magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., solid State Disk (SSD)), among others.

The foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the claims, and all equivalent structures or equivalent processes using the descriptions and drawings of the present application, or direct or indirect application in other related technical fields are included in the scope of the claims of the present application.

Claims

1. An image processing method, comprising:

determining or obtaining a pixel to be predicted of a first component and/or a second component according to a reference pixel of the first component and a first parameter, if the pixel to be predicted is located inside a block to be predicted, taking a predicted pixel which is located in the block to be predicted and is adjacent to the pixel to be predicted as the reference pixel, wherein the reference pixel is determined according to an adjacent pixel of the pixel to be predicted or an adjacent pixel of a co-located pixel of the pixel to be predicted, wherein the adjacent pixel comprises an adjacent reconstructed pixel and/or an adjacent predicted pixel, the pixel to be predicted comprises a second pixel or a plurality of second pixels, the combination of the plurality of second pixels is rectangular, the first parameter is determined or obtained according to the first reference pixel of the first component and/or the second component, and the first reference pixel is located in at least one area:

2. The method of claim 1, wherein the manner in which adjacent pixels are determined further comprises:

adjacent pixels are determined from the encoded image block.

3. The method of claim 2, further comprising at least one of:

the determining of the reference pixel according to the adjacent pixels of the pixel to be predicted or the adjacent pixels of the co-located pixel of the pixel to be predicted comprises: determining a reference pixel according to adjacent pixels positioned above and/or to the left of the pixel to be predicted or the co-located pixel of the pixel to be predicted;

the determining neighboring pixels from the encoded image block comprises: and determining a pixel corresponding to the pixel position to be predicted in the encoded image block as a neighboring pixel.

4. A method as claimed in claim 3, wherein the encoded image blocks are image blocks determined according to a rate distortion optimization or image matching algorithm.

5. The method as recited in claim 1, further comprising:

the first parameter is determined in a manner corresponding to at least one prediction mode.

6. The method of claim 5, wherein the index or flag corresponding to the at least one prediction mode is located in a list of prediction modes.

7. The method of any one of claims 1 to 4, further comprising at least one of:

the first component is a luminance component or a chrominance component;

the second component is a luminance component or a chrominance component;

the first component and the second component are different pixel components;

the reference pixel includes at least one first pixel.

8. The method of claim 7, further comprising at least one of:

the plurality of second pixels are adjacent to each other;

the prediction results of the first component or the second component of each of the plurality of second pixels are the same.

9. A processing apparatus, comprising: a memory, a processor, the memory having stored thereon an image processing program which, when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 8.

10. A storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the image processing method according to any of claims 1 to 8.