WO2018059500A1 - 运动补偿预测方法和设备 - Google Patents

运动补偿预测方法和设备 Download PDF

Info

Publication number
WO2018059500A1
WO2018059500A1 PCT/CN2017/104030 CN2017104030W WO2018059500A1 WO 2018059500 A1 WO2018059500 A1 WO 2018059500A1 CN 2017104030 W CN2017104030 W CN 2017104030W WO 2018059500 A1 WO2018059500 A1 WO 2018059500A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
image
pixel point
spherical
sub
Prior art date
Application number
PCT/CN2017/104030
Other languages
English (en)
French (fr)
Inventor
马祥
杨海涛
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN201610896430.5A external-priority patent/CN107888928B/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP17854950.7A priority Critical patent/EP3515076B1/en
Publication of WO2018059500A1 publication Critical patent/WO2018059500A1/zh
Priority to US16/371,486 priority patent/US10779000B2/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/513—Processing of motion vectors
    • H04N19/517—Processing of motion vectors by encoding
    • H04N19/52—Processing of motion vectors by encoding by predictive encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/547—Motion estimation performed in a transform domain
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression

Definitions

  • the present application relates to the field of image processing and, more particularly, to a method and apparatus for motion compensated prediction.
  • the spherical image When processing a spherical panoramic image (hereinafter referred to as a spherical image), the spherical image is generally first projected into a two-dimensional planar panoramic image of a polyhedral format (hereinafter referred to as a two-dimensional image), and then the two-dimensional image of the polyhedral format is edited. Decoding operation.
  • the position of the reference pixel point of the current pixel point is first determined, and then the pixel value of the current pixel point is predicted according to the pixel value of the reference pixel point.
  • the pixel value of the current pixel point is predicted according to the pixel value of the position of the reference pixel point regardless of whether the reference pixel point is at the face where the current pixel point is located.
  • the reference pixel When the reference pixel is not in the plane where the current pixel is located, since the faces of the two-dimensional image in the polyhedral format are not in the same projection plane, the adjacent regions of the faces are deformed, resulting in a position according to the position of the reference pixel at this time.
  • the pixel value is less effective in predicting the pixel value of the current pixel.
  • the application provides a motion compensation prediction method and a motion prediction compensation device to improve the motion compensation prediction effect.
  • a motion compensation prediction method comprising: determining a position of an initial reference pixel of a current pixel point in a reference image, the current pixel point being located within a first sub-image of the current image;
  • the initial reference pixel is located outside the second sub-image of the reference image corresponding to the first sub-image, according to the position of the initial reference pixel and the spherical body corresponding to the reference image a spherical center position, determining that a target reference pixel point of the current pixel point corresponds to a spherical reference position of the spherical body surface; determining, according to the spherical reference position, that the target reference pixel point corresponds to a polyhedron corresponding to the reference image a polyhedral reference position of the surface, wherein the spherical body is cut into the polyhedron, a line connecting the polyhedron reference position and the spherical center passes the spherical reference position; and
  • the present application finds a target reference pixel point that can actually serve as a reference according to the position of the initial reference pixel point, and then predicts the pixel value of the current pixel point according to the target reference pixel point, thereby improving the motion compensation prediction. The accuracy.
  • the second sub-image is a sub-image in the reference image corresponding to the first sub-image in the current image, for example, the current image and the reference image are all polyhedrons are regular hexahedrons, and the current image and the reference image are latticed.
  • the formula is 4 ⁇ 3, assuming that the first sub-image of the current image corresponds to the Bottom plane of the regular hexahedron corresponding to the current image, then the second sub-image of the reference image also corresponds to the Bottom plane of the regular hexahedron corresponding to the reference image. .
  • the reference image corresponding polyhedron may be disposed on the reference image, wherein the anchoring surface of the polyhedron is the surface of the region of the second sub-image in the reference image, and the anchoring surface is the surface as the reference when the polyhedron is expanded, for example, in the unfolding
  • the Bottom face of the polyhedron is placed on a plane, and then the other faces in the polyhedron are folded onto the plane.
  • the position of the Bottom face on the plane is constant, and the other faces of the polyhedron The face folds into other areas of the plane, and Bottom is the anchoring face of the polyhedron.
  • determining a target reference pixel point of the current pixel point according to a location of the initial reference pixel point and a spherical center position of the spherical body corresponding to the reference image Corresponding to the spherical reference position on the surface of the spherical body comprising: determining the initial reference according to a position of the initial reference pixel point and an offset value of a position of the initial reference pixel point relative to the second sub image a pixel at a first position of a plane in which the first face of the polyhedron is located, wherein the first face is a face of the polyhedron corresponding to the second sub-image; according to the first position and the The spherical center position of the spherical body determines that the target reference pixel point corresponds to a spherical reference position on the surface of the spherical body, wherein a line connecting the first position and the spherical center position
  • the method further includes: determining, according to layout information of the reference image and a location of the initial reference pixel, whether the initial reference pixel is located in the Outside of the second sub-image in the reference image.
  • the pixel value of the current pixel can be directly predicted according to the initial reference pixel when the initial reference pixel is not within the second sub-image, It is no longer necessary to determine the target reference pixel based on the initial reference pixel.
  • determining a location of the target reference pixel in the reference image according to the polyhedral reference position comprises: according to the polyhedron reference location and the The layout information of the polyhedron determines the position of the target reference pixel in the reference image.
  • the layout information includes quantity information of a face of the polyhedron, arrangement mode information of a sub image of the reference image, and a sub image of the reference image Arranging order information, at least one of rotation information of a sub-image of the reference image.
  • the predicted value of the pixel value of the point includes: determining a pixel value of the target reference pixel point as a predicted value of a pixel value of the current pixel point.
  • the determining the current pixel according to a pixel value of the target reference pixel and/or a pixel value of a neighboring pixel of the target reference pixel a predicted value of a pixel value of the point comprising: performing weighting processing on a pixel value of the target reference pixel point and a pixel value of a neighboring pixel point of the target reference pixel point; and the target reference pixel to which the weighting is processed The pixel value at the position of the point is determined as the predicted value of the pixel value of the current pixel point.
  • the determining the current pixel according to a pixel value of the target reference pixel and/or a pixel value of a neighboring pixel of the target reference pixel a predicted value of the pixel value of the point including performing an interpolation operation on the target reference pixel point position according to the pixel value of the adjacent pixel point of the target reference pixel; determining the pixel value obtained by the interpolation operation as the current pixel point The predicted value of the pixel value.
  • the current image and the reference image are both two-dimensional images.
  • determining whether the initial reference pixel point is located outside the second sub-image in the reference image according to the layout information of the reference image and the position of the initial reference pixel point comprises: Determining, according to a location of the initial reference pixel point and an area where the second sub image of the reference image is located, whether the initial reference pixel point is located outside the second sub image in the reference image, wherein The area in which the second sub-image of the reference image is located is determined based on the layout information of the reference image.
  • the polyhedron corresponding to the two-dimensional image is a regular polyhedron.
  • the positive polyhedron may include a regular tetrahedron, a regular hexahedron, a regular octahedron, a regular dodecahedron, and an icosahedron.
  • the initial reference pixel when the initial reference pixel is located in a second sub-image of the reference image corresponding to the first sub-image, determining, according to the pixel value of the initial reference pixel The predicted value of the pixel value of the current pixel.
  • the pixel value of the current pixel may be directly predicted according to the pixel value of the initial reference pixel. Only when the initial reference pixel is not located in the second sub-image (at this time, since the adjacent region of the sub-image in the two-dimensional image is deformed, the pixel value of the current pixel cannot be directly predicted according to the initial reference pixel. ), it is necessary to find the target reference pixel according to the position of the initial reference pixel.
  • a motion compensation prediction method comprising: determining a position of an initial reference pixel of a current reference pixel in a reference image, the current pixel being located in a first sub-image of the current image Determining that the current pixel point is in an extended region of the second sub-image when the initial reference pixel is located outside a second sub-image of the reference image corresponding to the first sub-image a location of the target reference pixel, wherein the extended region of the second sub-image is located outside the second sub-image, the extended region includes a plurality of pixel points, and any one of the first pixels of the extended region
  • the pixel value is determined according to a pixel value of a second pixel point in the reference image, the second pixel point corresponding to a connection between the polyhedral reference position of the surface of the polyhedron corresponding to the reference image and the center of the sphere
  • the second pixel point corresponds to a spherical reference position of the spherical body corresponding to
  • the present application directly searches for the target reference pixel point that can actually serve as a reference in the extended area according to the current pixel point and the motion information of the current pixel point, and then performs the pixel value of the current pixel point according to the target reference pixel point.
  • the prediction can quickly find the target reference pixel and improve the accuracy of the motion prediction estimation.
  • the extended area of the second sub-image described above may be determined in advance according to the method in the above first aspect.
  • the method further includes determining the second according to a location of the first pixel point and a spherical center position of the spherical body corresponding to the reference image a pixel point corresponding to a spherical reference position of the surface of the spherical body; determining, according to the spherical reference position, a polyhedral reference position corresponding to a surface of the polyhedron corresponding to the reference image according to the spherical reference position; according to the polyhedral reference position, Determining a position of the second pixel in the reference image.
  • the location according to the first pixel point and Determining a spherical reference position of the second pixel point corresponding to the surface of the spherical body corresponding to a spherical center position of the spherical body corresponding to the reference image comprising: according to a position of the first pixel point and the first pixel point Determining a first position of the first pixel in a plane of the first face of the polyhedron relative to an offset value of the second sub-image, wherein the first face is in the polyhedron a face corresponding to the second sub-image; determining, according to the first position and a spherical center position of the spherical body, a spherical reference position corresponding to the second pixel point on the surface of the spherical body, the first A line connecting the position to the center of the ball passes the spherical reference position.
  • the method further includes: determining, according to layout information of the reference image and a location of the initial reference pixel, whether the initial reference pixel is located in the Outside of the second sub-image in the reference image.
  • a motion compensation prediction method comprising: determining an initial reference pixel position of a current pixel in a reference image, the current pixel being located in a first sub-image of the current image; When the initial reference pixel is located outside the second sub-image of the reference image corresponding to the first sub-image, determining the target reference pixel of the current pixel according to the initial reference pixel position a position of a surface of the polyhedron corresponding to the reference image, wherein a line connecting the position of the target reference pixel at the surface of the polyhedron and the position of the initial reference pixel at the first plane passes through a center point of the polyhedron
  • the first plane is a plane in which the plane corresponding to the second sub-image of the polyhedron is located; and the target reference pixel is determined in the reference image according to a position of the target reference pixel at a surface of the polyhedron a location according to a pixel value of the target reference pixel in the reference image and/or an adjacent
  • the present application finds a target reference pixel point that can actually serve as a reference according to the position of the initial reference pixel point, and then predicts the pixel value of the current pixel point according to the target reference pixel point, thereby improving the motion compensation prediction. The accuracy.
  • the determining, according to the initial reference pixel position, a location of a target reference pixel of the current pixel point on a surface of a polyhedron corresponding to the reference image including Determining, according to a location of the initial reference pixel point and layout information of the reference image, a position of the initial reference pixel in the first plane; according to a position of the initial reference pixel in the first plane And determining the position of the target reference pixel on the surface of the polyhedron and the layout information of the reference image.
  • the determining, according to the location of the target reference pixel on the surface of the polyhedron, the location of the target reference pixel in the reference image comprising: Determining a position of the target reference pixel in the reference image with a position of an initial reference pixel and a layout information of the reference image, the position of the target reference pixel at a surface of the polyhedron being at the initial reference A line connecting the pixel point to a center point of the polyhedron and an intersection of the surface of the polyhedron.
  • a motion compensated prediction apparatus comprising means for performing the method of the first aspect.
  • a motion compensated prediction apparatus comprising means for performing the method of the second aspect.
  • a motion compensation prediction apparatus including a third party for performing The module of the method.
  • a codec including a nonvolatile storage medium, and a central processing unit, the nonvolatile storage medium storing an executable program, the central processing unit and the nonvolatile The storage medium is connected, and the executable program is executed to implement the motion compensation prediction method and the extended content thereof as provided by the first aspect of the present application.
  • a codec including a nonvolatile storage medium, and a central processing unit, the nonvolatile storage medium storing an executable program, the central processing unit and the nonvolatile The storage medium is connected, and the executable program is executed to implement the motion compensation prediction method and the extended content thereof as provided by the second aspect of the present application.
  • a codec including a nonvolatile storage medium, and a central processing unit, the nonvolatile storage medium storing an executable program, the central processing unit and the nonvolatile The storage medium is connected, and the executable program is executed to implement the motion compensation prediction method and its extended content as provided by the third aspect of the present application.
  • a computer readable medium storing program code for execution by an image processing apparatus, the program code comprising instructions for performing the method of the first aspect.
  • a computer readable medium storing program code for execution by an image processing apparatus, the program code comprising instructions for performing the method of the second aspect.
  • a computer readable medium storing program code for execution by an image processing apparatus, the program code comprising instructions for performing the method of the third aspect.
  • the target reference pixel that can actually serve as a reference is found according to the position of the initial reference pixel, and then the current pixel is compared according to the target reference pixel.
  • the pixel values are predicted to improve the accuracy of the motion compensation prediction.
  • Figure 1 is a latitude and longitude view of a spherical image.
  • Figure 2 is a two-dimensional image of a polyhedral format.
  • Figure 3 is a schematic illustration of a spherical image projected onto a regular hexahedron.
  • FIG. 4 is a schematic diagram of a two-dimensional image obtained by developing a regular hexahedron and a regular hexahedron.
  • FIG. 5 is a schematic flowchart of a motion compensation prediction method according to an embodiment of the present application.
  • Fig. 6 is a schematic diagram of a current image and a reference image.
  • Figure 7 is a schematic illustration of a reference image of a different layout format.
  • Fig. 8 is a schematic view of a polyhedron and a spherical body corresponding to a reference image.
  • Fig. 9 is a schematic diagram of a reference image.
  • Fig. 10 is a schematic diagram of a spherical body corresponding to a reference image.
  • FIG. 11 is a schematic view of a polyhedron and a spherical body corresponding to a reference image.
  • Fig. 12 is a schematic view of a polyhedron and a spherical body corresponding to a reference image.
  • Figure 13 is a schematic illustration of a current image and a reference image.
  • Fig. 14 is a schematic diagram of a regular hexahedron corresponding to a reference image.
  • Fig. 15 is a schematic diagram of a regular hexahedron corresponding to a reference image.
  • Figure 16 is a schematic illustration of other polyhedrons corresponding to the reference image.
  • Figure 17 is a reference image of a different layout format.
  • Figure 18 is a schematic illustration of adjacent pixel points around a target reference pixel point.
  • FIG. 19 is a schematic flowchart of a motion compensation prediction method according to an embodiment of the present application.
  • 20 is a schematic diagram of an extended area of a current image, a reference image, and a reference image.
  • FIG. 21 is a schematic flowchart of a motion compensation prediction method according to an embodiment of the present application.
  • FIG. 22 is a schematic structural diagram of an application of a motion compensation prediction method according to an embodiment of the present application.
  • FIG. 23 is a schematic block diagram of a motion compensation prediction apparatus according to an embodiment of the present application.
  • FIG. 24 is a schematic block diagram of a motion compensation prediction apparatus according to an embodiment of the present application.
  • FIG. 25 is a schematic block diagram of a motion compensation prediction apparatus according to an embodiment of the present application.
  • 26 is a schematic flow chart of a decoding process of a panoramic video decoder.
  • FIG. 27 is a schematic flow chart of an encoding process of a panoramic video decoder.
  • FIG. 28 is a schematic block diagram of an image encoder in accordance with an embodiment of the present application.
  • VR video images In order to support video image content presentation in all directions, Virtual Reality (VR) video images usually contain 360-degree omnidirectional visual information in three dimensions, up and down, and can imagine VR video images from the inner center position of a globe to the appearance. Look at the map of the globe. Typically, we also refer to VR video images as panoramic video images (which may be referred to simply as spherical images).
  • the spherical image cannot be conveniently represented, stored, and indexed, before the spherical image is processed, the spherical image is usually developed to obtain a two-dimensional image, and then the two-dimensional image is compressed, processed, stored, transmitted, and the like. Among them, the process of expanding a spherical image to obtain a two-dimensional image is called projection.
  • the common two-dimensional image is called the latitude and longitude image.
  • the image adjacent to the north and south pole regions is stretched to a large extent, and there is serious distortion and data redundancy.
  • a spherical image can be projected into a polyhedron to convert the spherical image into a two-dimensional image in a polyhedral format.
  • a spherical image can be projected to a regular tetrahedron ((a) in FIG. 2), a regular hexahedron ((b) in FIG. 2), and a regular octahedron ((c) in FIG. 2).
  • a dodecahedron ((d) in FIG. 2), an icosahedron ((e) in FIG. 2), and a two-dimensional planar image obtained by projecting a spherical image into each of the above polyhedrons is sequentially in FIG. (f)-(k).
  • the specific process of projecting a spherical image into a polyhedron is: placing a spherical image in a polyhedron to make it a polyhedral inscribed ball; connecting a spherical or polyhedral body to a point on the spherical surface and extending it compared to a polyhedron,
  • the pixel at the position at the intersection of the polyhedron is the pixel of the corresponding point on the spherical image.
  • the spherical surface is inscribed in the regular hexahedron ABCDEFGH, and the spherical center O and M' are connected in order to obtain the pixel value at the M' point on the regular hexahedron.
  • the pixel at the M point is the pixel at the M' point.
  • all the pixels in the ABCD region on the plane A'B'C'D' can be obtained in the same way, wherein the pixels in the ABCD region form an ABCD face, and the plane A'B'C'D' is The projection plane of the ABCD surface.
  • each of the planar images on the surface of the polyhedron becomes an image of one region or a sub-image of the spherical image in the two-dimensional image.
  • Figure 4(a) The surface of the regular hexahedron in Fig. 4 is expanded into the image of (b) in Fig. 4.
  • the surface image of the top surface on the surface of the hexahedron will become the sub-image in the upper left corner of (b) in Fig. 4.
  • the surface image of the top surface of the image, the top surface in the spherical image refers to the area covered by the surface image of the Top surface. For any pixel in the Top face, the Top face is called the face of the pixel.
  • the face and the sub-image in the embodiment of the present application correspond to each other.
  • the Bottom face is a face of a certain two-dimensional image
  • the image in the Bottom face is the first sub-image
  • the Bottom face is the first sub-image.
  • the corresponding sub-image, the first sub-image is a sub-image corresponding to the Bottom surface.
  • each of the small rectangular areas is a face of the two-dimensional image
  • the image composed of the pixels in each rectangular area is a sub-image of the two-dimensional image, that is,
  • a face is a concept of a region
  • a subimage is an image.
  • the image shown by (f)-(k) in FIG. 2 may be directly processed, or the smallest area rectangle surrounded by the image may be selected.
  • the image in the area is to be processed, and the area other than the face containing the two-dimensional image in the rectangular area is filled with the default content or the like, for example, all gray, all black, or all white.
  • the image when encoding and decoding an image, the image is often divided into a plurality of image blocks of equal size, and then a reference block is found for each image block, and there may be different reference cases in the process of finding a reference block for the current image block. According to the reference direction, it can be divided into one-way prediction and two-way prediction.
  • Unidirectional prediction means that there is a reference image set in the current block (the elements in the set are selected reference images in the reconstructed image), and the coding block can select any one of the reference images in the set.
  • Bidirectional prediction means that there are two reference image sets in the current block (the elements of the two sets are respectively independently selected from the reconstructed image, the reference images in the two sets may be partially or completely the same), and the coding block may be from two Each of the sets selects a reference image.
  • the method of constructing the bidirectional or unidirectional and reference image sets is commonly agreed by the codec, or the encoding end transmits the used method to the decoding end, and the decoding end determines the used method according to the decoding result.
  • the bidirectional prediction method there will be two reference blocks for the current coding block, each of which requires an indication of motion information.
  • the decoding end needs to determine two reference blocks according to the two sets of motion information decoded.
  • the predicted value of the pixel value of the pixel in the current block is determined according to the pixel value of the pixel within the two reference blocks.
  • FIG. 5 is a schematic flowchart of a motion compensation prediction method according to an embodiment of the present application. The method includes:
  • the above current image and reference image are two-dimensional images of a polyhedral format obtained by spherical image conversion.
  • the current image when the current image is subjected to prediction processing, the current image may be divided into a plurality of image blocks, and then each image block is processed, and the current pixel point may be a pixel point in an image block in the current image. . Specifically, the current pixel to be processed may be located in a certain image block within the first sub-image.
  • the position of the initial reference pixel in the reference image may be determined according to the position of the current pixel point and the motion information of the current pixel point obtained by decoding the motion information code stream.
  • the reference image is determined based on the reference image indication information in the motion information
  • the position of the initial reference pixel point in the reference image is determined according to the motion vector information in the motion information and the position of the current pixel point.
  • the current reference pixel when the reference block is searched for the current block in one reference image, or when the reference pixel is searched for the current pixel, the current reference pixel The position of the point is the position of the initial reference pixel.
  • the polyhedron corresponding to the current image and the reference image is a regular hexahedron
  • (a) in FIG. 6 is the current image (the current pixel is on the current image)
  • (b) in FIG. 6 is the reference image (the initial reference pixel is in the reference)
  • the current image and the reference image are each composed of six sub-images, Top (Top), Front (front), Right (right), Rear (back), Left (left), and Bottom (bottom).
  • the sub-image here can be thought of as an array of pixels that are in the same projection plane when the spherical surface is projected onto the polyhedron.
  • each face of the polyhedron becomes a part of the two-dimensional image.
  • the current pixel is P
  • P is within the Bottom sub-image in the current image
  • T is within the Front sub-image in the reference image, that is, T is not in the Bottom sub-image in the reference image.
  • the first sub-image is a Bottom sub-image
  • the second sub-image is a Bottom sub-image, the first sub-image and the second sub-image. Is a sub-image at the corresponding position in the current image and the reference image.
  • the target reference pixel point corresponds to a polyhedral reference position of a surface of the polyhedron corresponding to the reference image, wherein the spherical body is cut in the polyhedron, and the polyhedron reference position and the location a line connecting the center of the sphere through the spherical reference position;
  • the spherical reference position of the target reference pixel corresponding to the surface of the spherical body may be determined according to the position of the initial reference pixel, and then the target reference is determined according to the spherical reference position.
  • the pixel is at the polyhedral reference position of the polyhedral surface, and then, based on the spherical reference position, the position of the target reference pixel in the reference image can be determined.
  • the current pixel may be predicted according to the initial reference pixel directly, instead of finding the target reference pixel. point.
  • the polyhedron corresponding to the reference image may be a polyhedron composed of the reference image, that is, a polyhedron formed by folding each sub-image of the reference image according to a certain rule.
  • the method further includes: determining, according to the layout information of the reference image and the position of the initial reference pixel, whether the initial reference pixel is located outside the second sub-image in the reference image.
  • the position of the initial reference pixel in the reference image may be determined based on the position of the current pixel and the motion information of the current pixel obtained by decoding the motion information bitstream.
  • the decoder may determine the reference image based on the reference image indication information in the motion information, and determine the position of the initial reference pixel in the reference image based on the motion vector information in the motion information and the position of the current pixel.
  • the position of the current reference pixel is the position of the initial reference pixel.
  • the layout information of the polyhedron corresponding to the reference image includes the quantity information of the face of the polyhedron corresponding to the reference image, the arrangement mode information of the sub image of the reference image, the arrangement order information of the sub image of the reference image, and the reference image At least one of rotation information of the sub image.
  • the number information of the faces of the polyhedron corresponding to the reference image may specifically be a polyhedron corresponding to the reference image.
  • the number information of the faces of the polyhedron may indicate that the corresponding image of the reference image is a regular hexahedron.
  • the arrangement mode information of the sub-images of the reference image refers to the arrangement manner of each sub-image in the reference image.
  • the reference image corresponds to a regular hexahedron, and the reference image includes 6 sub-images, and the 6 sub-images
  • the arrangement may be 4 ⁇ 3 type ((b) in FIG. 7), 3 ⁇ 2 type ((c) in FIG. 7, (d) in FIG. 7), and 6 ⁇ 1 type (in the middle of FIG. 7). (e), (f) in Fig. 7.
  • the arrangement order information of the sub-images of the reference image refers to the arrangement order of the respective sub-images of the reference image in the reference image, for example, (c) in FIG. 7 and (d) in FIG. 7 are both 3 ⁇ 2 type.
  • the image, in (c) of FIG. 7, the sub-images corresponding to the Front face are arranged in the lower left corner of the image, and in (d) of FIG. 7, the sub-images corresponding to the Front face are arranged in the first column. middle place.
  • the rotation information of the sub-image of the reference image may refer to the rotation angle of the sub-image of the reference image, assuming that the placement position of each sub-image in (c) of FIG. 7 is taken as a reference, then in (d) of FIG. 7, Front
  • the rotation angle of the sub-image corresponding to the face is -90 degrees.
  • the target reference pixel that can actually serve as a reference is found according to the position of the initial reference pixel, and then the current pixel is compared according to the target reference pixel.
  • the pixel values of the points are predicted to improve the accuracy of the motion compensation prediction.
  • the offset value of the position of the initial reference pixel point relative to the second sub-image may be the position of the initial reference pixel point relative to a certain point in the second sub-image (eg, the center point of the second sub-image) Position offset value.
  • the position of the initial reference pixel may be the position of the initial reference pixel in the coordinate system of the plane in which the second sub image is located, and the first position may be the initial reference pixel corresponding to the polyhedron (the polyhedron is the polyhedron corresponding to the reference image)
  • the position in the coordinate system of the plane in which the first face is located, the coordinate system of the plane in which the first face is located may be different from the coordinate system of the plane in which the second sub-image is located.
  • the reference hexahedron corresponding to the reference image is ABCD-EFGH
  • the ABCD surface in the regular hexahedron Corresponding to the second sub-image in the reference image, the plane where the ABCD surface is located is A'B'C'D'
  • the spherical surface corresponding to the reference image is cut in the regular hexahedron ABCD-EFGH
  • the M point is the target reference pixel corresponding to the spherical surface.
  • the spherical reference position on the body, the M' point is the target reference pixel corresponding to the polyhedron reference position on the polyhedron, according to the first position of the initial reference pixel (the first position is not shown in FIG. 8) and the position of the center O
  • the position of the M point can be determined, and the position of M' can be determined according to the position of the M point, and then the position of the target reference pixel in the reference image can be determined according to the position of M'.
  • the following describes a spherical reference position based on the position of the initial reference pixel point and the spherical center position of the spherical body in detail with reference to an example.
  • the reference image is a hexahedral panoramic image in a 4 ⁇ 3 format
  • the initial reference pixel point of the current pixel point is T
  • the current pixel point is in the Bottom plane in the current image
  • the R point is in the Bottom plane in the reference image.
  • 10 is a spherical body corresponding to a reference image.
  • the spherical body is cut in a polyhedron corresponding to the reference image (the polyhedron is not shown in FIG. 10), the spherical body is tangent to the Bottom surface of the polyhedron, and the Bottom surface of the polyhedron and Bottom in the reference image.
  • the face is corresponding.
  • the position of T' (i.e., the first position of the initial reference pixel point) can be determined.
  • A'B'C'D' is the plane where the ABCD plane is located, and the space rectangular coordinate system xyz is established with the spherical center of the spherical body as the coordinate origin.
  • the xOy plane and the plane A'B' are specified here.
  • C'D' is parallel
  • the projection point of the O point in the plane A'B'C'D' is O'
  • O'Q is parallel to the y-axis
  • O'P is parallel to the x-axis
  • PT' is perpendicular to O'P.
  • T'Q is perpendicular to O'Q
  • ON of the xy plane is parallel to O'T'
  • the first position of the initial reference pixel is T'
  • the spherical reference position of the target reference pixel on the spherical body is T".
  • the position of the T" point on the spherical surface can be determined according to the position of T'.
  • the lengths of PT' and QT' are known as L PT ' and L QT ' , respectively. From the geometric relationship in the figure:
  • the position of the T" point can be determined according to the radius of the spherical body and ⁇ O"ON and ⁇ WOT", that is, the spherical reference position of the target reference pixel on the multifaceted spherical body is determined.
  • the polyhedral reference position corresponding to the surface of the polyhedron corresponding to the target reference pixel according to the spherical reference position is described in detail below with reference to the second example.
  • the spherical reference position has been determined in FIG. 10, and then the polyhedron reference position is determined according to the spherical reference position.
  • the polyhedron corresponding to the reference image is ABCD-EFGH
  • the octahedral length of the polyhedron is a
  • O is the body of the polyhedron.
  • O' is the projection of O on the surface ABCD
  • J is the midpoint of the BC side
  • T" is the spherical reference position
  • the extension line of the OT" is the polyhedron at the H point
  • the ABCD is at the T point
  • the K is the T at the O.
  • 'J is the vertical projection on the extension line
  • S is the vertical projection of H point on BC
  • I is the vertical projection of T on the BC side.
  • the BC side is parallel to the x-axis.
  • the position of the H point can be determined, that is, the polyhedron reference position is determined. How to determine the length of SJ and SH is described below.
  • L SH is to determine L SH based on the lengths of L ST , L OO ' and L O ' T.
  • L SH can also be determined in accordance with other formulas, L SH determined according to the other formulas in the following steps:
  • L SH can also be determined based on L JK , L OO ' and L O'K .
  • the polyhedron corresponding to the reference image in (a) of FIG. 12 includes two faces ACB and DBC, and the angle between the face ACB and the face DBC is greater than 90 degrees (the polyhedron may also include other faces, which are not listed here).
  • the ACB plane is the projection plane of the face of the second sub-image in the reference image
  • the BC side is the edge of the second sub-image where the face is adjacent to the adjacent face.
  • J is the intersection of the O'K to the BC side of the perpendicular line O'K and the BC side
  • the extension line of the OT" is the polyhedron at the H point
  • the plane of the ACB plane is at the T point, O'T and BC While crossing the point S, OT and plane BCD intersect H point
  • H" is the vertical projection of H on O'K
  • L is the vertical projection of H" on O'K.
  • the position of O' and the length of OO' are determined, and the position of J is also determined if SJ is determined (with The HH "equal length" and the length of the SH determine the position of the intersection H of the OT and the surface BCD, that is, the polyhedral reference position of the target reference pixel at the polyhedral surface.
  • Example 1 the determination of the spherical body reference position from the first position and the determination of the polyhedral reference position from the spherical body reference position are described in Example 1 and Example 3. In fact, it is also possible to determine the polyhedron reference position directly from the first position. The following is a detailed description of the determination of the polyhedron reference position (that is, the position at which the target reference pixel is projected onto the surface of the polyhedron) based on the first position, in conjunction with Example 4 and Example 5.
  • FIG. 13 (a) in FIG. 13 is the current image, (b) in FIG. 13 is the reference image, P is the current pixel point in the current image, P is located on the Bottom side of the current image, and P' is the reference image. P is the pixel with the same position, P 'Bottom surface located in the reference image, T 1, T 2 initial reference pixel, and T 3 are the current pixel, which are located outside the Bottom of the reference image.
  • the regular hexahedron in FIG. 14 is a polyhedron corresponding to a reference image constructed with the Bottom face of the reference image as a bottom surface, and FIG. 14 shows three current reference pixels of T 1 , T 2 and T 3 in (b) of FIG. 13 .
  • Point as can be seen from FIG. 14, the corresponding pixel points of the three target reference pixels on the regular hexahedron are H 1 , H 2 and H 3 , respectively, which are located at T 1 , T 2 and T 3 and the O point respectively.
  • the position of the intersection of the regular hexahedron, so that the positions of H 1 , H 2 and H 3 can be determined based on the positions of T 1 , T 2 and T 3 .
  • T 1 , T 2 , and T 3 only show the case where the initial reference pixel points are at different positions, and in fact, only one reference pixel point exists at a time.
  • how to determine the position of the target reference pixel at the corresponding pixel of the polyhedron is determined according to the position of the initial reference pixel.
  • the position of the projection pixel point on which the target pixel point is projected onto the surface of the polyhedron is determined in detail based on the position of T 1 .
  • the regular hexahedron in FIG. 15 is a polyhedron corresponding to the reference image constructed by using the Bottom surface of the reference image as the bottom surface, that is, the anchoring surface of the regular hexahedron (the anchoring surface can be understood as the bottom surface when constructing the polyhedron, for example, for reference)
  • the anchoring surface can be understood as the bottom surface when constructing the polyhedron, for example, for reference
  • One side of the image serves as the bottom surface of the constructed polyhedron, then the bottom surface is the anchoring surface of the polyhedron) the Bottom surface of the reference image.
  • the hexahedron has a rib length of a
  • O is a body of a regular hexahedron
  • the T position is the position of the initial reference pixel point
  • the ABCD surface is the face of the reference image in the same position as the current pixel point
  • O' is O Projecting the vertical projection on the ABCD surface along the plane normal direction on the projection plane where the current pixel point ABCD is located
  • J is the midpoint of the BC side
  • K is the vertical projection of T on the O'J extension line
  • OT cross BCGF The surface is at point H (the position where the H point is located at the position of the projected pixel point of the initial reference pixel), the vertical projection of the H point on the BC is S, and I is the vertical projection of T on the BC side.
  • the lengths of OO' and O'J are both Assuming that the length of the JK line segment is x and the length of the KT line segment is y, it is obtained according to a similar triangle:
  • L SJ is the length of SJ
  • L SH is the length of SH.
  • the polyhedron corresponding to the reference image includes two faces ACB and DBC (the polyhedron may also include other faces, which are not listed here), wherein the face ACB is in the same position as the current pixel in the reference image.
  • the face where the pixel is located O is the body of the polyhedron, O' is the vertical projection of O on the ABC surface, the T position is the position of the initial reference pixel point, O'T and BC are intersected at the S point, OT and Face BCD is at point H, H" is the vertical projection of H on O'K, L is H" vertical projection on O'K, and I is the vertical projection of T on BC.
  • the formula in the fifth example is similar to the formula in the third example.
  • the angular reference parameter of the spherical reference position J is used in the process of calculating L SJ and L H"J , and in the fifth example Instead of J, the final result is directly derived, both of which determine the polyhedral reference position of the target reference pixel at the polyhedral surface.
  • the facet angle of the adjacent faces of the tetrahedron is less than 90°
  • the face angle of the adjacent faces of the regular hexahedron is 90°
  • the face angle of the adjacent faces of the regular octahedron and the more complicated polyhedron is greater than 90°.
  • the corresponding parameters can be obtained in a similar way, and finally L SJ and L H"J are obtained , thereby determining the position of the projected pixel point of the target reference pixel projected onto the surface of the polyhedron.
  • the projection pixel here
  • the position of the point refers to the position of the intersection of the initial reference pixel point and the line of the polyhedral body and the polyhedral surface.
  • determining a location of the target reference pixel in the reference image according to the polyhedron reference position includes: determining a location of the target reference pixel in the reference image according to the polyhedral reference position and the layout information of the polyhedron.
  • a point at which the target reference pixel is located at the surface of the polyhedron may be understood as a projected pixel point of the target reference pixel, such that the target reference pixel of the current pixel is determined in the reference image according to the position of the initial reference pixel.
  • the position of the medium includes: determining the position of the projected pixel point of the initial reference pixel on the surface of the polyhedron according to the position of the initial reference pixel point and the layout information of the reference image, and the position of the projected pixel point on the surface of the polyhedron is initially referenced The intersection of the line point with the center point of the polyhedron and the surface of the polyhedron; determining the position of the target reference pixel point in the reference image according to the position of the projected pixel point on the surface of the polyhedron and the layout information of the reference image.
  • a two-dimensional image in a regular hexahedron format is taken as an example to specifically describe how to determine the position of a target reference pixel in a reference image, as shown in (a) of FIG. 17, assuming a regular hexahedron corresponding to the reference image.
  • side length a the layout format of the reference image is 4x3, the target reference pixel projected current pixel to H front surface of the regular hexahedron 1 point, H 1 with respect to Front surface of the upper left corner and the apex respectively ⁇ x, ⁇ y.
  • the position of the target reference pixel in the reference image in the two-dimensional image of the different partial format is determined below.
  • the position of Q is (a- ⁇ y, 2a+ ⁇ x).
  • determining a predicted value of a pixel value of the current pixel point according to the pixel value of the target reference pixel point and/or the pixel value of the adjacent pixel point of the target reference pixel point, including: the target reference pixel point
  • the pixel value is determined as the predicted value of the pixel value of the current pixel.
  • the pixel value of the target reference pixel may be a pixel value of the location of the target reference pixel, and the pixel value of the target reference pixel is directly determined as the predicted value of the current pixel, which can reduce the calculation process.
  • determining a predicted value of a pixel value of the current pixel point according to the pixel value of the target reference pixel point and/or the pixel value of the adjacent pixel point of the target reference pixel point including: the target reference pixel point
  • the pixel value and the pixel value of the adjacent pixel of the target reference pixel are weighted; the pixel value of the position of the target reference pixel obtained by the weighting process is determined as the predicted value of the pixel value of the current pixel.
  • the weighting process may be a smoothing process on the target reference pixel and surrounding pixels, that is, averaging the pixel values of the plurality of pixels including the target reference pixel, and averaging the obtained pixel values.
  • the predicted value of the pixel value of the current pixel is a smoothing process on the target reference pixel and surrounding pixels, that is, averaging the pixel values of the plurality of pixels including the target reference pixel, and averaging the obtained pixel values.
  • the pixel value of the target reference pixel point may be directly used as the predicted value of the pixel value of the current pixel point, or the pixel value of the current pixel point is not separately applied to the pixel of the current pixel point.
  • the value is predicted, and the pixel values of the current pixel are predicted together according to the target reference pixel and the adjacent pixel around the target reference pixel.
  • determining a predicted value of a pixel value of the current pixel point according to a pixel value of the target reference pixel point and/or a pixel value of a neighboring pixel point of the target reference pixel point including: according to the target reference pixel point The pixel value of the adjacent pixel is interpolated at the target reference pixel position; the pixel value obtained by the interpolation operation is determined as the predicted value of the pixel value of the current pixel.
  • the accuracy of the pixel value of the target reference pixel may be first determined.
  • the accuracy of the pixel value of the target reference pixel is smaller than the motion vector accuracy of the current pixel, interpolation is needed.
  • the algorithm calculates the pixel value of the pixel around the target pixel, and uses the calculated pixel value as the target parameter. The pixel value of the pixel.
  • the accuracy of the pixel value of the target reference pixel is smaller than that of the current pixel The accuracy of the motion vector, so the interpolation algorithm is needed to recalculate the pixel value of the target reference pixel.
  • the coordinates of the projection point H are (i+u, j+v), where i, j are non-negative integers, and u and v are floating point numbers of the [0, 1) interval.
  • the position coordinates of the four integer pixels adjacent to H are P (i, j) , P (i, j+1) , P (i, j+1), and P (i+1, j+1), respectively .
  • P H P (i,j) (1-u)(1-v)+P (i,j+1) (u)(1-v)+P (i,j+1) (1-u) (v)+P (i+1,j+1) (u)(v) (19)
  • the coordinates of the projection point H are still (i+u, j+v), and the coordinates of the 16 adjacent points around the H point are respectively P (i-1, j-1) , P. (i-1, j+0) , P (i-1, j+1) , P (i-1, j+2) , P (i+0, j-1) , P (i+0, j +0) , P (i+0,j+1) , P (i+0,j+2) , P (i+1,j-1) , P (i+1,j+0) , P (i+1, j+1) , P (i+1, j+2) , P (i+2, j-1) , P (i+2, j+0) , P (i+2, j+ 1) and P (i+2, j+2) .
  • the above uses only the bilinear interpolation algorithm and the bicubic interpolation algorithm as an example to describe how to use the interpolation algorithm to determine the pixel at the position of the projection pixel in the embodiment of the present application.
  • the Lanczos interpolation algorithm, the nearest neighbor interpolation algorithm, and the like can also be used.
  • Some non-analytic interpolation methods based on information such as image structure.
  • the pixel value of the target reference pixel may be directly determined as the pixel value of the current pixel, regardless of whether the pixel value of the target reference pixel satisfies the corresponding requirement. .
  • the pixel value of the target reference pixel is determined to meet the preset requirement. When the pixel value of the target reference pixel does not meet the requirement, the target reference may be used according to the target reference.
  • Pixel values of adjacent pixel points of the pixel are interpolated to the target reference pixel point, and the obtained pixel value is used as a predicted value of the pixel value of the current pixel point, or the pixel value of the target reference pixel point and the target may be
  • the pixel value of the adjacent pixel of the reference pixel is weighted, and the result of the weighting process is used as the predicted value of the pixel value of the current pixel.
  • FIG. 19 is a schematic flowchart of a motion compensation prediction method according to an embodiment of the present application. The method shown in Figure 19 includes:
  • the second pixel point corresponds to a spherical reference of the spherical body corresponding to the reference image corresponding to the connection between the polyhedron reference position of the surface of the polyhedron corresponding to the reference image and the center of the sphere through the second pixel point Position, the spherical reference position is determined according to a position of the initial reference pixel point and a spherical center position of the spherical body, the spherical body being cut in the polyhedron;
  • the initial reference pixel when the initial reference pixel is located outside the second sub-image, directly searching for the target reference pixel that can be used as a reference function directly in the extended region according to the current pixel point and the motion information of the current pixel point. Then, the pixel value of the current pixel point is predicted according to the target reference pixel, and the target reference pixel point can be quickly found, and the accuracy of the motion prediction estimation can also be improved.
  • the extended region is not constructed for each sub-image in the reference image, but the predicted value of the pixel value of the current pixel point is directly calculated according to the pixel value of the initial reference pixel point, regardless of the initial value.
  • the position of the reference pixel in the reference image may be constructed for the sub-image in the reference image according to the method shown in FIG. 5, such that when the position of the initial reference pixel is not in the sub-image region corresponding to the current pixel in the reference image. And determining a position of the reference pixel point in the extended area according to the motion information, and using a pixel value of the reference pixel point in the extended area as a predicted value of the current pixel point pixel value.
  • the pixel value of the pixel in the extended area may be calculated according to the method shown in FIG. 5 above, so that when the pixel value of the current pixel is predicted, the target reference pixel can be directly found from the extended area. The prediction of the pixel value of the current pixel can be quickly realized.
  • the method shown in FIG. 19 further includes: determining, according to a position of the first pixel point and a spherical center position of the spherical body corresponding to the reference image, a spherical reference corresponding to the surface of the spherical body at the second pixel point. Position; determining, according to the spherical reference position, the second pixel point corresponding to the polyhedral reference position of the surface of the polyhedron corresponding to the reference image; determining the position of the second pixel point in the reference image according to the polyhedral reference position.
  • determining, according to the position of the first pixel point and the spherical center position of the spherical body corresponding to the reference image, determining that the second pixel point corresponds to the spherical reference position on the surface of the spherical body comprises:
  • the pixel value of any one of the above-mentioned extended regions can be determined by the method shown in FIG. 5, and the second pixel point is determined according to the position of the first pixel point corresponding to the spherical reference position of the sphere, and then according to the spherical surface.
  • the reference position determines that the second pixel corresponds to the polyhedral reference position of the polyhedron, and then determines the position of the second pixel in the reference image according to the polyhedral reference position, and finally determines the first image according to the position of the second pixel in the reference image.
  • the pixel value of the prime point is determined by the same method to determine the pixel value of each pixel in the extended region, so that the position of the target reference pixel in the extended region can be directly determined when performing motion compensation prediction, and the motion compensation prediction is improved. s efficiency.
  • the method shown in FIG. 19 further includes: determining, according to layout information of the reference image and a position of the initial reference pixel, whether the initial reference pixel is located outside the second sub image in the reference image .
  • determining whether the position of the initial reference pixel is outside the second sub-image may be after determining the position of the initial reference pixel in the reference image, and according to the position of the initial reference pixel and the reference image The spherical center position of the spherical body is determined before the spherical reference position is determined.
  • the initial reference pixel is located in the second sub-image in the reference image, and the current value is determined according to the pixel value of the initial reference pixel and/or the pixel value of the adjacent pixel of the target reference pixel.
  • the predicted value of the pixel value of the pixel that is, when the initial reference pixel is located within the second sub-image in the reference image, the pixel value of the current pixel may be directly predicted according to the first initial reference pixel.
  • the motion compensation prediction method shown in FIG. 20 is to first determine an extended region of a certain sub-image of the reference image (it may also determine an extended region of the plurality of sub-images in the reference image), and then determine that the reference pixel is in the extended region. The position, so that when the current pixel is predicted, the reference pixel can be directly searched from the extended region, and the current pixel is predicted according to the pixel value of the reference pixel.
  • the layout information of the polyhedron corresponding to the reference image (generally, the layout information of the polyhedron corresponding to the reference image is the same as the layout information of the current image, and therefore, can be used to determine the plane of the pixel in the current image) and the current pixel point
  • the position determines the face of the current pixel, finds the face of the reference image corresponding to the previous pixel, constructs an extended area for the corresponding face, and determines the pixel value of the pixel in the extended area.
  • the specific effect of the extended area is similar to expanding or filling the surface around the face. As shown in (a) of FIG. 20, the current pixel point P is on the Bottom plane, then the Bottom plane in the reference image needs to be constructed next.
  • a Bottom of the reference image may be used as a bottom surface to construct a picture
  • the spherical center position of the spherical body corresponding to the reference image is first. Determining a spherical reference position, determining a polyhedron reference position, finally determining a position in the reference image in the second pixel, and determining a pixel value of the first pixel according to the second pixel (a pixel value at the second pixel may be determined)
  • the pixel value of the first pixel is directly determined, and the pixel around the second pixel or the second pixel may be subjected to a certain operation, and the calculated pixel value is used as the pixel value of the first pixel.
  • the first pixel point is similar to the initial reference pixel point in the motion compensation prediction method of the embodiment of the present application shown in FIG. 5
  • the second pixel point is similar to the motion compensation prediction method of the embodiment of the present application shown in FIG. 5
  • the target reference pixel in .
  • the reference pixel of the current pixel is determined, and whether the reference pixel is in the same position as the current pixel is determined according to the surface of the current pixel and the position of the reference pixel. If the reference pixel is in the same position as the current pixel, then the pixel value of the reference pixel position can be directly used as the current pixel image. The predicted value of the prime value. Otherwise, the following operations are required: first, the corresponding position of the reference pixel in the extended area is determined according to the relative positional deviation of the reference pixel from the current pixel. As shown in (b) of FIG.
  • the reference pixel point T is not at the same position as the current pixel point P, and according to the relative positional deviation of the reference pixel point from the current pixel point, it can be known that the reference pixel point T should be located in the extension.
  • the pixel value at this position is taken as the predicted value of the current pixel pixel value.
  • the pixel points in the extended area may be integer pixels or fractional pixels. If it is a fractional pixel, the specific pixel value may be 1/2 precision, 1/4 precision, 1/8 precision, and the like.
  • the reference pixel After determining that the reference pixel is in the extended region position, if there is no pixel value at the location, the reference pixel needs to be interpolated.
  • Specific interpolation operations can use DCT interpolation, bilinear interpolation or other interpolation methods.
  • the method shown in FIG. 19 is different from the method shown in FIG. 5 in that, in FIG. 19, the pixel value of the pixel in the extended region is calculated in advance, so that it is more convenient to predict the current pixel point.
  • Finding the target reference pixel directly from the extended area eliminates the computational process of determining the position of the target reference pixel based on the position of the initial reference pixel.
  • the method shown in FIG. 5 is to determine the position of the target reference pixel point according to the position of the initial reference pixel point when predicting the pixel value of the current pixel point, and has a calculation process compared with the method shown in FIG. .
  • the other steps of the method shown in FIG. 19 are substantially the same as the other steps of the method shown in FIG. 5, and the repeated description is omitted as appropriate for the sake of brevity.
  • FIG. 21 is a schematic flowchart of a motion compensation prediction method according to an embodiment of the present application. The method shown in Figure 21 includes:
  • the target reference pixel of the current pixel is on the surface of the polyhedron corresponding to the reference image. a position in which a position of the target reference pixel on the surface of the polyhedron and a position of the initial reference pixel at the position of the first plane pass through a center point of the polyhedron, and the first plane is a face of the polyhedron corresponding to the second sub image Plane
  • the target reference pixel that can actually serve as a reference is found according to the position of the initial reference pixel, and then the current pixel is compared according to the target reference pixel.
  • the pixel values of the points are predicted to improve the accuracy of the motion compensation prediction.
  • the motion compensation prediction method shown in FIG. 5 is different from the motion compensation prediction method shown in FIG. 21 in that FIG. 5 is to determine the spherical surface according to the position of the initial reference pixel point and the spherical center position of the polyhedron. Reference The position is determined, and then the polyhedron reference position is determined according to the spherical reference position, and the position of the target reference pixel in the reference image is determined according to the polyhedron reference position, and FIG. 21 is to first determine the target reference pixel according to the position of the initial reference pixel.
  • the position of the surface of the polyhedron corresponding to the image is referenced, and then the position of the target reference pixel in the reference image is determined according to the position of the target reference pixel at the surface of the polyhedron.
  • the method shown in FIG. 5 determines the position of the target reference pixel in the reference image by three steps
  • the method shown in FIG. 16 determines the target reference pixel in the reference image by two steps. The location in . It should be understood that the relevant explanation and explanation of the motion compensation prediction method shown in FIG. 5 is equally applicable to the motion compensation prediction method shown in FIG. 16, and therefore, for the sake of brevity, the repeated description is appropriately omitted herein.
  • determining, according to the initial reference pixel position, a position of the target reference pixel of the current pixel point on a surface of the polyhedron corresponding to the reference image including: according to the position of the initial reference pixel point and the layout information of the reference image. Determining a position of the initial reference pixel in the first plane; determining a position of the target reference pixel on the surface of the polyhedron according to the position of the initial reference pixel at the first plane and the layout information of the reference image.
  • determining a location of the target reference pixel in the reference image according to the location of the target reference pixel at the polyhedral surface includes: determining the target reference according to the location of the initial reference pixel and the layout information of the reference image. The position of the pixel in the reference image, the position of the target reference pixel on the surface of the polyhedron at the intersection of the line connecting the initial reference pixel and the center point of the polyhedron with the surface of the polyhedron.
  • the motion compensation prediction method of the embodiment of the present application is applicable to two-dimensional images of various polyhedral formats, and that the surface of the two-dimensional image of the polyhedral format is rotated, the order of the faces is changed, and the spatial layout of the faces is changed.
  • the motion compensation prediction method of the embodiment of the present application is also applicable to the case where the format is changed.
  • the pixel value of the position of the target reference pixel point may be calculated when the pixel value of the current pixel point is predicted.
  • the pixel value of the target reference pixel of the pixel to be processed may also be calculated in advance, so that the pixel value of the target reference loudness point of the pixel to be processed may be directly obtained when the pixel to be processed is processed, thereby saving the time for processing the image.
  • the motion compensation prediction method in the embodiment of the present application is described in detail in the above two-dimensional image in the 4x3 format.
  • the motion compensation prediction method in the embodiment of the present application is also applicable.
  • the motion compensation prediction method of the embodiment of the present application is also applicable to the surface of the two-dimensional image with partial or full rotation, the arrangement order of different faces, and the arrangement method.
  • the pixel value of the reference pixel in the reference image may be directly used as the predicted value of the pixel value of the current pixel point, and the pixel value of the reference pixel point and the weight value of the pixel value of the surrounding pixel point may also be utilized. Or the pixel value obtained by other operations is used as the predicted value of the pixel value of the current pixel.
  • the motion compensation prediction method of the embodiment of the present application can be applied to the architecture shown in FIG. 22, and can be specifically applied in the process of encoding and decoding a two-dimensional panoramic image.
  • the existing VR video image capturing device is usually a ring-shaped or spherical multi-camera array, and each camera can collect images of different angles to obtain a multi-view video image of the current scene, and then, the VR video processing device has many
  • the perspective image is image mosaiced to obtain a three-dimensional spherical panorama, and then the three-dimensional spherical panorama is mapped to obtain a two-dimensional panoramic image, which facilitates input of subsequent processing, compression, transmission, storage, and the like.
  • the processed result is stored in the transmission server, and the VR head display device can acquire the image processing result stored in the transmission server through a network or the like, and perform a corresponding decoding operation to display the three-dimensional spherical panoramic image. It is.
  • the motion compensation prediction method of the embodiment of the present application is described in detail above with reference to FIG. 1-22.
  • the motion compensation prediction apparatus of the embodiment of the present application is described in detail below with reference to FIGS. 23-28. It should be understood that the motion compensation prediction apparatus in FIGS. 23-28 can perform the respective steps of the motion compensation prediction method in FIGS. 1-22, and the repeated description is appropriately omitted in order to avoid repetition.
  • the motion compensation prediction apparatus 400 includes:
  • the first determining unit 410 is configured to determine a position of an initial reference pixel of the current pixel in the reference image, where the current pixel is located in the first sub-image of the current image.
  • a second determining unit 420 configured to: when the initial reference pixel point is located outside the second sub-image of the reference image corresponding to the first sub-image, according to the location and location of the initial reference pixel Determining a spherical center position of the spherical body corresponding to the reference image, determining that the target reference pixel point of the current pixel point corresponds to a spherical reference position on the surface of the spherical body;
  • a third determining unit 430 configured to determine, according to the spherical reference position, that the target reference pixel point corresponds to a polyhedral reference position of a surface of the polyhedron corresponding to the reference image, wherein the spherical body is cut in the polyhedron, Connecting the polyhedron reference position to the center of the sphere through the spherical reference position;
  • a fourth determining unit 440 configured to determine, according to the polyhedron reference position, a location of the target reference pixel in the reference image
  • the prediction unit 450 is configured to determine a predicted value of a pixel value of the current pixel point according to a pixel value of the target reference pixel point and/or a pixel value of an adjacent pixel point of the target reference pixel point.
  • the present application finds a target reference pixel point that can actually serve as a reference according to the position of the initial reference pixel point, and then predicts the pixel value of the current pixel point according to the target reference pixel point. Improve the accuracy of motion prediction estimates.
  • the second determining unit 420 is specifically configured to:
  • the target reference pixel point corresponding to a spherical reference position on the surface of the spherical body, wherein the first position is connected to the spherical center position
  • the line passes the spherical reference position.
  • the motion compensation prediction apparatus 400 further includes:
  • the determining unit 460 is configured to determine, according to the layout information of the reference image and the position of the initial reference pixel, whether the initial reference pixel is located outside the second sub image in the reference image.
  • the fourth determining unit 440 is specifically configured to:
  • the layout information includes quantity information of a face of the polyhedron, arrangement information of a sub image of the reference image, and arrangement order information of a sub image of the reference image.
  • Reference image At least one of rotation information of the sub image.
  • the prediction unit 450 is specifically configured to:
  • a pixel value of the target reference pixel is determined as a predicted value of a pixel value of the current pixel.
  • the prediction unit 450 is specifically configured to:
  • the prediction unit 450 is specifically configured to:
  • the pixel value obtained by the interpolation operation is determined as a predicted value of a pixel value of the current pixel point.
  • FIG. 24 is a schematic block diagram of a motion compensation prediction apparatus according to an embodiment of the present application.
  • the motion compensation prediction apparatus 500 includes:
  • a first determining unit 510 configured to determine a position of an initial reference pixel of a current reference pixel in a reference image, where the current pixel is located in a first sub-image of the current image;
  • a second determining unit 520 configured to determine, when the initial reference pixel point is located outside the second sub-image of the reference image corresponding to the first sub-image, in the sub-image The position of the target reference pixel in the extended area
  • the extended area of the second sub-image is located outside the second sub-image, the extended area includes a plurality of pixel points, and the pixel value of any one of the first pixel points in the extended area is according to the reference Determining, by the pixel value of the second pixel in the image, the second pixel corresponding to the connection between the polyhedron reference position of the surface of the polyhedron corresponding to the reference image and the center of the sphere passing through the second pixel a spherical reference position of the spherical body corresponding to the reference image, the spherical reference position being determined according to a position of the initial reference pixel point and a spherical center position of the spherical body, the spherical body being cut in the polyhedron ;
  • the prediction unit 530 is configured to determine a predicted value of a pixel value of the current pixel point according to a pixel value of the target reference pixel point and/or a pixel value of a neighboring pixel point of the target reference pixel point.
  • the present application directly searches for the target reference pixel point that can be used as a reference directly in the extended area according to the current pixel point and the motion information of the current pixel point, and then according to the target reference pixel point pair.
  • the pixel values of the pixels are predicted, the target reference pixels can be quickly found, and the accuracy of the motion prediction estimation can be improved.
  • the motion compensation prediction apparatus further includes:
  • a third determining unit 540 configured to determine, according to a position of the first pixel point and a spherical center position of the spherical body corresponding to the reference image, that the second pixel point corresponds to a spherical reference position on a surface of the spherical body;
  • a fourth determining unit 550 configured to determine, according to the spherical reference position, that the second pixel point corresponds to a polyhedral reference position of a surface of the polyhedron corresponding to the reference image;
  • the fifth determining unit 560 determines a position of the second pixel in the reference image according to the polyhedron reference position.
  • the third determining unit 540 is specifically configured to:
  • the motion compensation prediction apparatus further includes:
  • the determining unit 570 is configured to determine, according to the layout information of the reference image and the position of the initial reference pixel point, whether the initial reference pixel point is outside the second sub image in the reference image.
  • FIG. 25 is a schematic block diagram of a motion compensation prediction apparatus according to an embodiment of the present application.
  • the motion compensation prediction apparatus 600 includes:
  • a first determining unit 610 configured to determine an initial reference pixel position of the current pixel in the reference image, where the current pixel is located in the first sub-image of the current image;
  • a second determining unit 620 when the initial reference pixel point is located outside the second sub-image of the reference image corresponding to the first sub-image, determining the current pixel according to the initial reference pixel position a position of the target reference pixel at a position of a surface of the polyhedron corresponding to the reference image, wherein a position of the target reference pixel at a surface of the polyhedron and a position of the initial reference pixel at a first plane a line passing through a center point of the polyhedron, the first plane being a plane of a face of the polyhedron corresponding to the second sub-image;
  • a third determining unit 630 configured to determine, according to a position of the target reference pixel on the surface of the polyhedron, a position of the target reference pixel in the reference image;
  • a prediction unit 640 configured to determine a predicted value of a pixel value of the current pixel point according to a pixel value of the target reference pixel point in the reference image and/or a pixel value of a neighboring pixel point of the target reference pixel point .
  • the present application finds a target reference pixel point that can actually serve as a reference according to the position of the initial reference pixel point, and then predicts the pixel value of the current pixel point according to the target reference pixel point. Improve the accuracy of motion prediction estimates.
  • the second determining unit 620 is specifically configured to:
  • the third determining unit is specifically configured to:
  • the motion compensation prediction method and the motion compensation prediction apparatus in the embodiments of the present application are described in detail above with reference to FIG. 1-25.
  • the motion compensation prediction method in the embodiment of the present application can be regarded as one of encoding or decoding processes.
  • the intermediate process or step, the panoramic video encoder or the panoramic video decoder may implement the motion compensation prediction method of the embodiment of the present application.
  • the decoding process of the panoramic video decoder and the encoding process of the panoramic video encoder will be described in detail below with reference to FIG. 26 and FIG. 27, respectively.
  • the current image Before encoding the current image, the current image is generally divided into several blocks of equal size, and then each block is encoded.
  • the layout information of the current image is generally transmitted as header information to the decoding end.
  • the header information is the preamble information, and the decoder first decodes the header information after receiving the encoded code stream, and then decodes the subsequent code stream according to the header information. It is assumed that the current image is divided into a plurality of image blocks at the time of encoding, and the decoding is sequentially performed in the same order. For the current block, if it is determined that it is the inter-frame coding mode, in order to obtain its reconstruction information, the reconstructed value of the pixel value of the pixel in the current block is obtained.
  • the process mainly includes intra prediction, inter prediction, transform, quantization, and entropy encoding.
  • the image may be first divided into image blocks before intra prediction and inter prediction, and after the residual is obtained, transform quantization is performed, and then entropy coding is performed and the code stream is output.
  • the image block here is an array of M ⁇ N sizes composed of pixel points (MM and N are integers greater than 0, M may be equal to N or not equal to N), and pixel values of respective pixel point positions are known.
  • Intra prediction is to predict the pixel value of a pixel in the current block by using the pixel value of the pixel in the reconstructed region in the current image.
  • the inter-frame prediction is to find a matching reference block for the current coding block in the current image in the reconstructed image, and use the pixel value of the pixel in the reference block as the prediction information of the pixel value of the pixel in the current coding block.
  • this process is called Motion Estimation (ME)
  • Motion vector (MV) information of the current block pointing to the reference block needs to be transmitted to the decoding end (also That is, the positional offset of the reference block position compared to the current block position, and the indication information of the image in which the reference block is located, and the like.
  • the motion estimation process needs to try multiple reference blocks in the reference picture for the current block, and finally determine which one or which reference blocks to use for prediction of the current block.
  • the residual prediction information (that is, the pixel value of the corresponding pixel in the corresponding reference block) may be subtracted from the pixel value of the pixel in the current coding block to obtain the residual. information.
  • the residual information is transformed by Discrete Cosine Transformation (DCT), and finally, the coded stream is obtained by using the quantization entropy coding.
  • DCT Discrete Cosine Transformation
  • decoding is equivalent to the inverse of the encoding.
  • the entropy decoding inverse quantization inverse transform is used to obtain the residual information, and according to the decoded code stream, it is determined whether the current block uses intra prediction or inter prediction; if it is intra coding, then the current pixel to be processed is used.
  • the pixel values of the pixels in the surrounding reconstructed area are constructed according to the intra prediction method used; if it is inter-frame coding, the motion information needs to be parsed, and the reference is determined in the reconstructed image according to the parsed motion information.
  • Block the pixel value of the pixel in the block is used as prediction information (this process is also called motion compensation); finally, the reconstruction information is obtained by using the prediction information plus the residual information.
  • 26 is a schematic flow chart of a decoding process of a panoramic video decoder. Specifically include:
  • the decoder can decode the motion information code stream and determine the current fast motion information according to the analysis result.
  • the motion information of the current block may also be determined by decoding the indication information of the motion information and the difference information of the motion information and the motion information, that is, the code stream of the motion information may not be directly parsed. It is possible to determine the motion information of the current block.
  • the decoding end may construct a motion information set (for example, the motion information set includes motion information of the reconstructed block adjacent to the time or space of the current block, and the construction method of the set is jointly agreed by the codec side), then decoding The terminal may parse the indication information of the predicted motion information, and then determine the predicted motion information from the set, and obtain the motion information of the current block according to the predicted motion information and the difference information of the predicted motion information and the motion information.
  • the decoding end also uses the predicted motion information determined according to the indication information of the predicted motion information as the motion information of the current block, which method is specifically used by the codec terminal, or the encoding end transmits the mode information used. To the decoding end, the decoding end determines which manner is used to determine the motion information of the current block according to the received mode information.
  • step 710 if it is a one-way reference, a set of motion information needs to be parsed. If it is a two-way reference, you need to parse two sets of motion information. Specifically, several sets of motion information are parsed together by the codec, or the encoder transmits the set number related information to the decoding end, and the decoding end determines according to the parsing result.
  • the decoder may determine a reference image of the current image in which the current block is located according to the running information of the current block, and determine a position of the reference block in the reference image.
  • the prediction value of the pixel value of the pixel in the current block is determined as the prediction value of the pixel value of each pixel in the current block, and the prediction value of the pixel value of the pixel is determined according to the embodiment of the present application.
  • the motion compensation prediction method is determined.
  • the prediction block of the current block may be obtained according to the predicted value of the pixel value of each pixel, and the prediction block includes the pixel of each pixel in the current block. The predicted value of the value.
  • step 720 if it is a one-way reference, it is only necessary to determine the position of a reference block in the reference image. For a pixel in the current block, only the position of one reference pixel needs to be determined, and then according to the present application.
  • the motion compensated prediction method of an embodiment determines a predicted value of a pixel pixel value within a current block.
  • the positions of the two reference pixels in the two reference blocks need to be determined separately.
  • the two predicted values of the pixel values of the pixel points within the current block may then be determined according to the motion compensated prediction method of the embodiment of the present application.
  • the two predicted values are then weighted or otherwise manipulated to obtain a predicted value of the pixel value of the pixel within the current block.
  • the specific operation mode is that the codec side has a common agreement, or the coding end transmits the used method to the decoding end, and the decoding end determines the used method according to the analysis result.
  • the predicted value obtained by the operation is used as the final predicted value of the pixel value of the current pixel.
  • the obtained predicted value may be smoothed to obtain a predicted value as a final predicted value of the pixel value of the current pixel point.
  • the pixel value of the current pixel point can be bi-directionally predicted, that is, the current pixel point can be predicted twice, and the obtained two predicted values are weighted to obtain a predicted value of the pixel value of the current pixel point.
  • the decoder decodes the coded code stream of the residual information of the pixel values of the pixel points in the current block, and obtains the residual information of the pixel values of the pixel points in the current block by using the inverse quantization and inverse transform methods. That is to say, the encoder decodes the coded code stream of the residual block of the current block, and then obtains the residual block of the current block by using the inverse quantization and inverse transform methods.
  • step 730 in order to determine the residual information of the pixel values of the pixel points in the current block, it is also possible to use only the inverse transform or only the inverse quantization method, which method is used for the codec end common agreement, or coding.
  • the terminal transmits the used method to the decoding end, and the decoding end determines the used method according to the analysis result.
  • the reconstructed value of the pixel value of each pixel in the current block may be obtained according to the residual value information of the pixel value of the pixel in the current block obtained in step 720 and the pixel value of the pixel in the current block obtained in step 730.
  • the prediction block of the current block may be obtained according to step 720, and then the prediction block is added to the residual block obtained in step 730 to obtain a reconstructed block of the current block, where the reconstructed block includes each pixel in the current block.
  • the reconstructed value of the pixel value may be obtained according to the residual value information of the pixel value of the pixel in the current block obtained in step 720 and the pixel value of the pixel in the current block obtained in step 730.
  • step 740 in order to obtain the reconstructed value of the pixel value of the pixel point in the current block, after using the prediction information plus the residual information, some other operations, such as deblocking filtering, etc., may also be required. Whether the other operations and other operations are required to be jointly agreed by the codec terminal, or the encoding end transmits the used method to the decoding end, and the decoding end determines the used method according to the analysis result.
  • the decoding process of the panoramic video decoder is described in detail above with reference to FIG. 26.
  • the decoding process is generally performed on the encoded code stream obtained by using the panoramic video encoder as an example. Carry out detailed instructions.
  • the layout information of the spherical image is generally determined after the acquisition or generation of the spherical image. Therefore, the encoder has already known the layout information of the spherical image before the encoder encodes the panoramic video.
  • the layout information of the spherical image is generally transmitted as header information.
  • the header information is a kind of preamble information, so that the decoder decodes the header information before receiving the encoded code stream, and after acquiring the layout information of the spherical image according to the header information, the subsequent code stream can be decoded.
  • FIG. 27 is a schematic flow chart of an encoding process of a panoramic video encoder. Specifically include:
  • the encoder first determines the motion information of the current block and then encodes the motion information. Specifically, the encoder selects a reference image for the current image in the reconstructed image, and searches for a matching block for the current block in a prescribed area of the reference image, and uses the matching block as a reference block of the current block, and in the process
  • the reference image used in and the motion vector information indicating the positional offset of the reference block with respect to the current block is used as motion information and encoded.
  • a plurality of blocks may be selected for the current block first, and which block is selected as the reference block at the end may be determined using a rate-distortion optimization criterion. For example, determining the number of bits of the motion information that needs to be encoded when the candidate reference block is used as the matching block, and the distortion value when using the pixel value of the pixel point in the current candidate reference block to predict the pixel value of the pixel point in the current block, using Lagrange The daily optimization method determines the cost of the candidate reference block. The candidate reference block with the least cost is then selected as the matching block. For the case of multiple reference images, it is necessary to perform the above operations for each image.
  • the motion information may not be directly encoded, but the indication information of the motion information is predicted, or the difference information of the motion information and the operation information is encoded.
  • the encoding end may construct a motion information set (for example, the motion information set includes motion information of the reconstructed block adjacent to the time or space of the current block, and the construction method of the set is jointly agreed by the codec side), then the encoding The terminal may select one of the motion information as the predicted motion information, and then obtain the difference information between the motion information of the current block and the predicted motion information, and predict the difference.
  • the indication information of the motion information and the difference information are encoded.
  • the decoding end can determine the motion information from the constructed motion information set according to the indication information, and use it as the motion information of the current block.
  • Which method is used for the common agreement of the codec side can also be selected by the rate distortion method, and the selected method is transmitted to the decoding end.
  • step 810 if it is a one-way reference, only one set of motion information is encoded. If it is a two-way reference, you need to encode two sets of motion information.
  • the specific coding several sets of motion information is jointly agreed by the codec end, and can also be determined by the rate distortion method, and the corresponding information is transmitted by the encoding end to the decoding end.
  • the unidirectional reference it is necessary to determine the position of a reference block in the reference image. For a current pixel in the current block, it is necessary to determine the position of a reference pixel, and then according to the motion compensation prediction method of the embodiment of the present application. The predicted value of the pixel value of the reference pixel is determined. If it is a bidirectional reference, it is necessary to determine the position of the two reference blocks of the current block in the respective reference images, that is, for one pixel in the current block, it is necessary to separately determine two reference pixels in the two reference blocks. position. The two predicted values of the pixel values of the pixel points within the current block may then be determined according to the motion compensated prediction method of the embodiment of the present application.
  • the two predicted values are then weighted or otherwise manipulated to obtain a predicted value of the pixel value of the pixel within the current block.
  • the specific operation mode is that the codec side has a common agreement, or the coding end transmits the used method to the decoding end, and the decoding end determines the used method according to the analysis result.
  • the encoder may determine a reference image of the current image in which the current block is located according to the running information of the current block, and determine a position of the reference block in the reference image.
  • the prediction value of the pixel value of the pixel in the current block is determined as the prediction value of the pixel value of each pixel in the current block, and the prediction value of the pixel value of the pixel is determined according to the embodiment of the present application.
  • the motion compensation prediction method is determined.
  • the predicted value obtained by the operation is used as the final predicted value of the pixel value of the current pixel.
  • the obtained predicted value may be smoothed to obtain a predicted value as a final predicted value of the pixel value of the current pixel point.
  • the pixel value of the current pixel point can be bi-directionally predicted, that is, the current pixel point can be predicted twice, and the obtained two predicted values are weighted to obtain a predicted value of the pixel value of the current pixel point.
  • the encoder subtracts the predicted value from the pixel value of the pixel in the current block to obtain the residual information of the pixel value of the pixel in the current block, and then processes the residual information of the pixel value of the pixel in the current block according to the transform quantization method. And encode the result after processing.
  • the rate distortion method determines and transmits the information to the decoder.
  • FIG. 28 is a schematic block diagram of an image encoder 900 according to an embodiment of the present application, including a prediction module 910, a transform quantization module 920, an entropy encoding module 930, a reconstruction module 940, and a filtering module 950.
  • the specific functions of each module are as follows:
  • the prediction module 910 is configured to generate prediction data. Prediction module 910 can generate one or more prediction units (PUs) that each no longer partition the CU. Each PU of a CU may be associated with a different block of pixels within a block of pixels of the CU. Prediction module 910 can generate predictive pixel blocks for each PU of the CU. Prediction module 910 can use intra prediction or inter prediction to generate predictive pixel blocks for the PU. If prediction module 910 uses intra prediction to generate a predictive pixel block for the PU, prediction module 910 can generate a predictive pixel block of the PU based on the decoded pixels of the picture associated with the PU.
  • PUs prediction units
  • the prediction module 910 can generate a predictive pixel block of the PU based on the decoded pixels of the one or more pictures that are different from the picture associated with the PU. .
  • Prediction module 910 can generate a residual pixel block of the CU based on the predictive pixel block of the PU of the CU.
  • the residual pixel block of the CU may indicate the difference between the sampled value in the predictive pixel block of the PU of the CU and the corresponding sampled value in the initial pixel block of the CU.
  • Transform quantization module 920 is for processing the predicted residual data.
  • Image encoder 900 may perform recursive quadtree partitioning on the residual pixel blocks of the CU to partition the residual pixel blocks of the CU into one or more smaller residual pixel blocks associated with the transform units (TUs) of the CU. Because the pixels in the pixel block associated with the TU each correspond to one luma sample and two chroma samples, each TU can be associated with one luma residual sample block and two chroma residual sample blocks.
  • Image encoder 900 may apply one or more transforms to the residual sample block associated with the TU to generate a coefficient block (ie, a block of coefficients).
  • the transform can be a DCT transform or a variant thereof.
  • the coefficient block is obtained by applying a one-dimensional transform in the horizontal and vertical directions to calculate a two-dimensional transform.
  • Image encoder 900 may perform a quantization procedure for each of the coefficients in the coefficient block. Quantization generally refers to the process by which the coefficients are quantized to reduce the amount of data used to represent the coefficients, thereby providing further compression.
  • Image encoder 900 may generate a set of syntax elements that represent coefficients in the quantized coefficient block.
  • Image encoder 900 may apply an entropy encoding operation (eg, a context adaptive binary arithmetic coding (CABAC) operation) to some or all of the above syntax elements by entropy encoding module 930.
  • CABAC context adaptive binary arithmetic coding
  • entropy encoding module 930 can binarize the syntax elements to form a binary sequence that includes one or more bits (referred to as "binary").
  • Entropy encoding module 930 can encode a portion of the binary using regular encoding, and can use bypass encoding to encode other portions of the binary.
  • image encoder 900 may apply inverse quantization and inverse transform to the transformed coefficient block by reconstruction module 940 to reconstruct the residual sample block from the transformed coefficient block.
  • Image encoder 900 may add the reconstructed residual sample block to a corresponding sample block of one or more predictive sample blocks to produce a reconstructed sample block.
  • image encoder 900 can reconstruct the block of pixels associated with the TU. The pixel block of each TU of the CU is reconstructed in this way until the entire pixel block reconstruction of the CU is completed.
  • image encoder 900 After image encoder 900 reconstructs the block of pixels of the CU, image encoder 900 performs a deblocking filtering operation through filtering module 950 to reduce the blockiness of the block of pixels associated with the CU. After the image encoder 900 performs the deblocking filtering operation, the image encoder 900 may use the sample adaptive offset (SAO) to modify the reconstructed pixel block of the CTB of the picture. After performing these operations, image encoder 900 may store the reconstructed blocks of pixels of the CU in a decoded picture buffer for use in generating predictive blocks of pixels for other CUs.
  • SAO sample adaptive offset
  • the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code via a computer-readable medium and executed by a hardware-based processing unit.
  • the computer readable medium can comprise a computer readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium comprising, for example, any medium that facilitates transfer of the computer program from one place to another in accordance with a communication protocol. .
  • computer readable media generally may correspond to (1) a non-transitory tangible computer readable storage medium, or (2) a communication medium such as a signal or carrier wave.
  • Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described in this application.
  • the computer program product can comprise a computer readable medium.
  • certain computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, flash memory, or may be used to store instructions or data structures. Any other medium in the form of the desired program code and accessible by the computer. Also, any connection is properly termed a computer-readable medium. For example, if you use coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (eg, infrared, radio, and microwave) to send commands from a website, server, or other remote source, coaxial cable , fiber optic cable, twisted pair, DSL, or wireless technologies (eg, infrared, radio, and microwave) are included in the definition of the media.
  • coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology eg, infrared, radio, and microwave
  • a magnetic disk and an optical disk include a compact disk (CD), a laser disk, an optical disk, a digital video disk (DVD), a flexible disk, and a Blu-ray disk, wherein the disk usually reproduces data magnetically, and the disk passes the laser Optically copy data. Combinations of the above should also be included within the scope of computer readable media.
  • processors such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits
  • DSPs digital signal processors
  • ASICs application specific integrated circuits
  • FPGAs field programmable logic arrays
  • processors may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein.
  • the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec.
  • the techniques can be fully implemented in one or more circuits or logic elements.
  • the techniques of the present application can be broadly implemented by a variety of devices or devices, including a wireless handset, an integrated circuit (IC), or a collection of ICs (eg, a chipset).
  • IC integrated circuit
  • Various components, modules or units are described in this application to emphasize functional aspects of the apparatus configured to perform the disclosed techniques, but are not necessarily required to be implemented by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or combined with suitable software and/or by a collection of interoperable hardware units (including one or more processors as described above). Or firmware to provide.
  • system and “network” are used interchangeably herein. It should be understood that the term “and/or” herein is merely an association relationship describing an associated object, indicating that there may be three relationships, for example, A and/or B, which may indicate that A exists separately, and A and B exist simultaneously. There are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual object is an "or" relationship.
  • B corresponding to A means that B is associated with A, and B can be determined according to A.
  • determining B from A does not mean that B is only determined based on A, and that B can also be determined based on A and/or other information.
  • the disclosed systems, devices, and methods may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the functions may be stored in a computer readable storage medium if implemented in the form of a software functional unit and sold or used as a standalone product.
  • the technical solution of the present application which is essential or contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product, which is stored in a storage medium, including
  • the instructions are used to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present application.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例提供一种运动补偿预测方法和运动补偿预测装置。该运动补偿预测方法包括：确定当前像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点的位置确定所述当前像素点的目标参考像素点在所述参考图像中的位置；根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。本申请实施例能够提高运动补偿预测的准确性。

Description

运动补偿预测方法和设备 技术领域
本申请涉及图像处理领域，并且更具体地，涉及一种运动补偿预测的方法和设备。
背景技术
在对球形全景图像(以下简称球形图像)进行处理时，一般是先将球形图像投影为多面体格式的二维平面全景图像(以下简称二维图像)，然后再对多面体格式的二维图像进行编解码操作。
在对多面体格式的二维图像进行运动补偿预测时，一般是先确定当前像素点的参考像素点的位置，然后根据参考像素点的像素值对当前像素点的像素值进行预测。在现有技术的运动补偿预测过程中，无论参考像素点是否在当前像素点所在的面，都根据参考像素点所在位置的像素值对当前像素点的像素值进行预测。当参考像素点不在当前像素点所在的面时，由于多面体格式的二维图像中各个面不处于同一个投影平面，各个面的相接区域存在形变，导致根据此时根据参考像素点所在位置的像素值对当前像素点的像素值进行预测的效果较差。
发明内容
本申请提供一种运动补偿预测方法和运动预测补偿装置，以提高运动补偿预测效果。
第一方面，提供一种运动补偿预测方法，所述方法包括：确定当前像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述当前像素点的目标参考像素点对应在所述球面体表面的球面参考位置；根据所述球面参考位置确定所述目标参考像素点对应在所述参考图像对应的多面体的表面的多面体参考位置，其中，所述球面体内切于所述多面体，所述多面体参考位置与所述球心的连线经过所述球面参考位置；根据所述多面体参考位置，确定所述目标参考像素点在所述参考图像中的位置；根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
当初始参考像素点位于第二子图像之外时，由于二维图像的子图像边界的几何形变的影响，导致直接根据初始参考像素点对当前像素点进行预测是不准确的，因此，在这种情况下，本申请是根据该初始参考像素点的位置寻找到真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，提高了运动补偿预测的准确性。
上述第二子图像是参考图像中与当前图像中的第一子图像处于对应位置的子图像，例如，当前图像和参考图像对应的都是多面体都是正六面体，并且当前图像和参考图像的格 式都是4×3，假设当前图像的第一子图像对应于当前图像对应的正六面体的Bottom(底)面，那么参考图像的第二子图像也对应于参考图像对应的正六面体的Bottom面。
上述参考图像对应多面体可以设置在参考图像上，这时多面体的锚定面为参考图像中的第二子图像的区域所在的面，锚定面是展开多面体时作为基准的面，例如，在展开多面体在时，将多面体的Bottom面置于某个平面上，然后将多面体中的其它面翻折到该平面上，这个时候Bottom面在该平面上的位置是不变的，而多面体的其它的面会翻折到该平面的其它区域，此时Bottom就是多面体的锚定面。
结合第一方面，在第一方面的某些实现方式中，根据所述初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述当前像素点的目标参考像素点对应在所述球面体表面的球面参考位置，包括：根据所述初始参考像素点的位置和所述初始参考像素点的位置相对于所述第二子图像的偏移值，确定所述初始参考像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；根据所述第一位置以及所述球面体的球心位置，确定所述目标参考像素点对应在所述球面体表面的球面参考位置，其中，所述第一位置与所述球心位置的连线经过所述球面参考位置。
结合第一方面，在第一方面的某些实现方式中，所述方法还包括：根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
通过判断初始参考像素点是否在参考图像中的第二子图像之外，能够在初始参考像素点不在第二子图像之内时，直接根据初始参考像素点对当前像素点的像素值进行预测，而不必再根据初始参考像素点来确定目标参考像素点。
结合第一方面，在第一方面的某些实现方式中，根据所述多面体参考位置，确定所述目标参考像素点在所述参考图像中的位置，包括：根据所述多面体参考位置以及所述多面体的布局信息，确定所述目标参考像素点在所述参考图像中的位置。
结合第一方面，在第一方面的某些实现方式中，所述布局信息包含所述多面体的面的数量信息，所述参考图像的子图像的排布方式信息，所述参考图像的子图像的排布顺序信息，所述参考图像的子图像的旋转信息中的至少一种。
结合第一方面，在第一方面的某些实现方式中，所述根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值，包括：将所述目标参考像素点的像素值确定为所述当前像素点的像素值的预测值。
结合第一方面，在第一方面的某些实现方式中，所述根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值，包括：对所述目标参考像素点的像素值以及所述目标参考像素点的临近像素点的像素值进行加权处理；将所述加权处理到的所述目标参考像素点所在位置的像素值确定为所述当前像素点的像素值的预测值。
结合第一方面，在第一方面的某些实现方式中，所述根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值，包括根据目标参考像素点的临近像素点的像素值在所述目标参考像素点位置进行插值运算；将所述插值运算得到的像素值确定为所述当前像素点的像素值的预测值。
在某些实现方式中，所述当前图像和所述参考图像均为二维图像。
在某些实现方式中，根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外，包括：根据所述初始参考像素点的位置以及所述参考图像的第二子图像所在的区域，判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外，其中，所述参考图像的第二子图像所在的区域是根据所述参考图像的布局信息确定的。
在某些实现方式中，所述二维图像对应的多面体为正多面体。其中，正多面体可以包括正四面体、正六面体、正八面体、正十二面体、正二十面体。
在某些实现方式中，当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像内时，根据所述初始参考像素点的像素值，确定所述当前像素点的像素值的预测值。
当初始参考像素点位于第二子图像内时，不必再根据初始参考像素点去寻找目标参考像素点，而是直接可以根据初始参考像素点的像素值，对当前像素点的像素值进行预测。只有当初始参考像素点不位于第二子图像内时(此时由于在二维图像中的子图像的相邻区域存在形变，不能直接根据该初始参考像素点对当前像素点的像素值进行预测)，才需要根据初始参考像素点的位置去寻找目标参考像素点。
第二方面，提供一种运动补偿预测方法，所述方法包括：确定当前参考像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，确定所述当前像素点在所述第二子图像的扩展区域中的目标参考像素点的位置，其中，所述第二子图像的扩展区域位于所述第二子图像之外，所述扩展区域包括多个像素点，所述扩展区域中任意一个第一像素点的像素值是根据所述参考图像中的第二像素点的像素值确定的，所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置与所述球心的连线经过所述第二像素点对应在所述参考图像对应的球面体的球面参考位置，所述球面参考位置是根据所述初始参考像素点的位置以及所述球面体的球心位置确定的，所述球面体内切于所述多面体；根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
当初始参考像素点位于第二子图像之外时，由于二维图像的面边界的几何形变的影响，导致直接根据初始参考像素点对当前像素点进行预测是不准确的，因此，在这种情况下，本申请是直接根据当前像素点以及当前像素点的运动信息直接在扩展区域中寻找真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，能够快速寻找到目标参考像素点，并且还能提高运动预测估计的准确性。
应理解，上述第二子图像的扩展区域可以是预先根据上述第一方面中的方法确定的。
结合第二方面，在第二方面的某些实现方式中，所述方法还包括：根据所述第一像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置；根据所述球面参考位置确定所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置；根据所述多面体参考位置，确定所述第二像素点在所述参考图像中的位置。
结合第二方面，在第二方面的某些实现方式中，所述根据所述第一像素点的位置以及 所述参考图像对应的球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置，包括：根据所述第一像素点的位置以及所述第一像素点的位置相对于所述第二子图像的偏移值，确定所述第一像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；根据所述第一位置以及所述球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置，所述第一位置与所述球心位置的连线经过所述球面参考位置。
结合第二方面，在第二方面的某些实现方式中，所述方法还包括：根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
第三方面，提供一种运动补偿预测方法，该方法包括：确定当前像素点在参考图像中的初始参考像素点位置，所述当前像素点位于所述当前图像的第一子图像内；当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点位置确定所述当前像素点的目标参考像素点在所述参考图像对应的多面体的表面的位置，其中，所述目标参考像素点在所述多面体的表面的位置与所述初始参考像素点在第一平面的位置的连线经过所述多面体的中心点，所述第一平面为所述多面体中与所述第二子图像对应的面所在的平面；根据所述目标参考像素点在多面体表面的位置确定所述目标参考像素点在所述参考图像中的位置；根据所述参考图像中所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
当初始参考像素点位于第二子图像之外时，由于二维图像的子图像边界的几何形变的影响，导致直接根据初始参考像素点对当前像素点进行预测是不准确的，因此，在这种情况下，本申请是根据该初始参考像素点的位置寻找到真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，提高了运动补偿预测的准确性。
结合第三方面，在第三方面某些实现方式中，所述根据所述初始参考像素点位置确定所述当前像素点的目标参考像素点在所述参考图像对应的多面体的表面的位置，包括：根据所述初始参考像素点的位置和所述参考图像的布局信息，确定所述初始参考像素点在所述第一平面的位置；根据所述初始参考像素点在所述第一平面的位置和所述参考图像的布局信息，确定所述目标参考像素点在所述多面体的表面的位置。
结合第三方面，在第三方面某些实现方式中，所述根据所述目标参考像素点在多面体表面的位置确定所述目标参考像素点在所述参考图像中的位置，包括：根据所述初始参考像素点的位置和所述参考图像的布局信息，确定所述目标参考像素点在所述参考图像中的位置，所述目标参考像素点在所述多面体的表面的位置在所述初始参考像素点与所述多面体的中心点的连线与所述多面体的表面的交点处。
第四方面，提供一种运动补偿预测装置，所述运动补偿预测装置包括用于执行第一方面的方法的模块。
第五方面，提供一种运动补偿预测装置，所述运动补偿预测装置包括用于执行第二方面的方法的模块。
第六方面，提供一种运动补偿预测装置，所述运动补偿预测装置包括用于执行第三方 面的方法的模块。
第七方面，提供一种编解码器，包括非易失性存储介质，以及中央处理器，所述非易失性存储介质存储有可执行程序，所述中央处理器与所述非易失性存储介质连接，并执行所述可执行程序以实现如本申请第一方面所提供的运动补偿预测方法及其扩展内容。
第八方面，提供一种编解码器，包括非易失性存储介质，以及中央处理器，所述非易失性存储介质存储有可执行程序，所述中央处理器与所述非易失性存储介质连接，并执行所述可执行程序以实现如本申请第二方面所提供的运动补偿预测方法及其扩展内容。
第九方面，提供一种编解码器，包括非易失性存储介质，以及中央处理器，所述非易失性存储介质存储有可执行程序，所述中央处理器与所述非易失性存储介质连接，并执行所述可执行程序以实现如本申请第三方面所提供的运动补偿预测方法及其扩展内容。
第十方面，提供一种计算机可读介质，所述计算机可读介质存储用于图像处理设备执行的程序代码，所述程序代码包括用于执行第一方面中的方法的指令。
第十一方面，提供一种计算机可读介质，所述计算机可读介质存储用于图像处理设备执行的程序代码，所述程序代码包括用于执行第二方面中的方法的指令。
第十二方面，提供一种计算机可读介质，所述计算机可读介质存储用于图像处理设备执行的程序代码，所述程序代码包括用于执行第三方面中的方法的指令。
本申请中，在初始参考像素点位于第二子图像之外时，根据该初始参考像素点的位置寻找到真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，提高了运动补偿预测的准确性。
附图说明
图1是球形图像的经纬图。
图2是多面体格式的二维图像。
图3是球形图像投影到正六面体的示意图。
图4是正六面体以及正六面体展开得到的二维图像的示意图。
图5是本申请实施例的运动补偿预测方法的示意性流程图。
图6是当前图像和参考图像的示意图。
图7是不同布局格式的参考图像的示意图。
图8是参考图像对应的多面体和球面体的示意图。
图9是参考图像的示意图。
图10是参考图像对应的球面体的示意图。
图11是参考图像对应的多面体和球面体的示意图。
图12是参考图像对应的多面体和球面体的示意图。
图13是当前图像和参考图像的示意图。
图14是参考图像对应的正六面体的示意图。
图15是参考图像对应的正六面体的示意图。
图16是参考图像对应的其它多面体的示意图。
图17是不同布局格式的参考图像。
图18是目标参考像素点周围的临近像素点的示意图。
图19是本申请实施例的运动补偿预测方法的示意性流程图。
图20是当前图像、参考图像以及参考图像的扩展区域的示意图。
图21是本申请实施例的运动补偿预测方法的示意性流程图。
图22是本申请实施例的运动补偿预测方法应用的架构示意图。
图23是本申请实施例的运动补偿预测装置的示意性框图。
图24是本申请实施例的运动补偿预测装置的示意性框图。
图25是本申请实施例的运动补偿预测装置的示意性框图。
图26是全景视频解码器的解码过程的示意性流程图。
图27是全景视频解码器的编码过程的示意性流程图。
图28是根据本申请实施例的图像编码器的示意性框图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
为了更好地理解本申请实施例的运动补偿预测方法，下面先结合图1至图3对本申请实施例的运动补偿预测方法所应用的场景进行简单的介绍。
为支持各个方向的视频图像内容呈现，虚拟现实(Virtual Reality，VR)视频图像通常包含三维空间上下左右360度全向的视觉信息，可以将VR视频图像想象为从一个地球仪的内部中心位置向外观看地球仪的地图。典型地，我们也将VR视频图像称为全景视频图像(可以简称为球形图像)。
由于球形图像无法方便的表示、存储和索引，因此，在对球形图像进行处理之前，通常要将球形图像展开得到二维图像，再对该二维图像进行压缩、处理、存储、传输等操作。其中，将球形图像展开得到二维图像的过程称为投影。
如图1所示，常见的二维图像称为经纬图，在图1中，邻近南北两极区域的图像经过拉伸幅度较大，存在严重的失真和数据冗余。
为克服经纬图中存在的缺陷，可以将球形图像投影到多面体中，从而将球形图像转换为多面体格式的二维图像。例如，如图2所示，可以将球形图像投影到正四面体(图2中的(a))、正六面体(图2中的(b))、正八面体(图2中的(c))、正十二面体(图2中的(d))、正二十面体(图2中的(e))，将球形图像投影到上述各个多面体中得到的二维平面图像依次为图2中的(f)-(k)所示。
将球形图像投影到多面体中的具体过程为：将球形图像置于多面体中，使其成为多面体的内切球；将球心或者多面体体心与球面上某点连线并延长相较于多面体，多面体上交点位置处的位置处的像素点便为球形图像上对应点的像素点。
下面结合图3以将球形图像投影到正六面体的过程为例，将球面内切于正六面体ABCDEFGH中，为了得到正六面体上M’点处的像素值，将球心O与M’连线，并交于球面的M点，则M点处的像素点即为M’点处的像素点。类似地，可以按照同样的方法得到平面A’B’C’D’上ABCD区域内所有像素点，其中，ABCD区域内像素点构成ABCD面(face)，平面A’B’C’D’为ABCD面的投影平面(projection plane)。
将球形图像投影到多面体，然后再将多面体表面展开为二维图像之后，多面体表面上的每个面图像成为二维图像中一个区域的图像或者球形图像的一个子图像。例如，图4(a) 中的正六面体表面展开成为图4中的(b)的图像，六面体表面上的Top面的面图像将会成为图4中的(b)中左上角的子图像，我们称这个子图像为球形图像的Top面的面图像，球形图像中的Top面指代Top面的面图像覆盖的区域。对于Top面中的任意一个像素点，Top面称为此像素点的所在面。
应理解，本申请实施例中的面与子图像是对应的，例如，Bottom面为某个二维图像的一个面，这个Bottom面中的图像是第一子图像，Bottom面是第一子图像对应的面，第一子图像是Bottom面对应的子图像。如图2中的(h)所示，其中的每个小的矩形区域就是二维图像的一个面，每个矩形区域中的像素点组成的图像就是二维图像的一个子图像，也就是说面是一个区域的概念，子图像是图像。
另外，在对上述二维图像进行编解码或者压缩等操作时，可以直接对图2中的(f)-(k)所示的图像进行处理，也可以选择将其包围在内的最小面积矩形区域内的图像作为处理对象，对该矩形区域中除了包含二维图像的面的之外的区域都填充为默认内容等，例如，全灰、全黑或者全白等。
应理解，在对图像进行编解码时经常将图像分成大小相等的多个图像块，然后为每个图像块寻找参考块，在为当前图像块寻找参考块的过程中可能存在不同的参考情况。依据参考方向可以分成单向预测和双向预测。单向预测是指当前块存在一个参考图像集合(该集合中的元素为已重建图像中选择的参考图像)，编码块可以选用集合中的任意一个参考图像。双向预测是指当前块存在两个参考图像集合(两个集合的元素分别是从已重建图像中独立选取的图像，这两个集合中的参考图像可以部分或者全部相同)，编码块可以从两个集合中各选择一个参考图像。具体使用双向还是单向以及参考图像集合的构建方法为编解码端共同约定，或者编码端将所用方法传输到解码端，解码端依据解码结果确定所用方法。当使用双向预测方法时，当前编码块会存在两个参考块，每个参考块各自需要运动信息指示。解码端需要根据解码出的两套运动信息确定两个参考块。根据两个参考块内像素点的像素值确定当前块内像素点像素值的预测值。
实际上在确定当前块的参考块，并根据参考块对当前块进行预测时实际上确定当前块中的当前像素点在参考块中的参考像素点，然后根据参考像素点的像素值来预测当前像素点的像素值的预测值。下面就对本申请实施例的运动补偿预测方法进行详细的介绍。
图5是本申请实施例的运动补偿预测方法的示意性流程图。该方法包括：
110、确定当前像素点的初始参考像素点在参考图像中的位置，当前像素点位于当前图像的第一子图像内。
应理解，上述当前图像和参考图像均为球形图像转换得到的多面体格式的二维图像。
应理解，在对当前图像进行预测处理时可以将该当前图像划分为多个图像块，然后对每个图像块进行处理，上述当前像素点可以是当前图像中的某个图像块中的像素点。具体地，上述待处理的当前像素点可以位于第一子图像内的某个图像块内。
具体地，在解码器中，可以根据当前像素点的位置以及通过解码运动信息码流得到的当前像素点的运动信息确定初始参考像素点在参考图像中的位置。例如，根据运动信息中的参考图像指示信息确定参考图像，根据运动信息中的运动矢量信息以及当前像素点的位置确定参考图像中初始参考像素点的位置。在编码器中，在运动搜索过程中，当在一个参考图像中为当前块搜索参考块时，或者在为当前像素点搜索参考像素点时，当前参考像素 点的位置即为初始参考像素点的位置。
假设当前图像和参考图像对应的多面体为正六面体，图6中的(a)为当前图像(当前像素点在当前图像上)，图6中的(b)为参考图像(初始参考像素点在参考图像上)，当前图像和参考图像均由六个子图像，这六个子图像分别是Top(上)、Front(前)、Right(右)、Rear(后)、Left(左)、Bottom(下)。这里的子图像可以认为是一些像素点组成的阵列，这些像素点在球面投影到多面体上时处于同一个投影平面。将多面体展开为二维图像之后，多面体的各个面则成为二维图像中的一部分。假设当前像素点为P，P在当前图像中的Bottom子图像内，假设初始参考像素点为T，T在参考图像中的Front子图像内，也就是说T不在参考图像中的Bottom子图像内。应理解，在图6中的(a)中，第一子图像就是Bottom子图像，在图6中的(b)中，第二子图像是Bottom子图像，第一子图像和第二子图像是当前图像和参考图像中处于对应位置的子图像。
120、当初始参考像素点位于参考图像中与第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述当前像素点的目标参考像素点对应在所述球面体表面的球面参考位置；
130、根据所述球面参考位置确定所述目标参考像素点对应在所述参考图像对应的多面体的表面的多面体参考位置，其中，所述球面体内切于所述多面体，所述多面体参考位置与所述球心的连线经过所述球面参考位置；
140、根据所述多面体参考位置，确定所述目标参考像素点在所述参考图像中的位置；
也就是说，为了确定目标参考像素点在参考图像中的位置，可以先根据初始参考像素点的位置确定目标参考像素点对应在球面体表面的球面参考位置，然后再根据球面参考位置确定目标参考像素点在多面体表面的多面体参考位置，接下来，再根据球面参考位置就可以确定目标参考像素点在参考图像中的位置了。
在这里，根据初始参考像素点的位置不太容易直接确定目标参考像素点在参考图像中的位置，而是通过借助参考图像对应的球面体以及多面体，通过球面参考位置和多面体参考位置可以逐步确定目标参考像素点在参考图像中的位置。
应理解，当初始参考像素点位于参考图像中与第一子图像对应位置的第二子图像之内时，可以根据直接根据初始参考像素点对当前像素点进行预测，而不用再寻找目标参考像素点。
上述参考图像对应的多面体可以是由该参考图像构成的多面体，也就是将该参考图像的各个子图像按照一定的规则折叠起来形成的多面体。
可选地，上述方法还包括：根据参考图像的布局信息和初始参考像素点的位置判断初始参考像素点是否位于参考图像中的第二子图像之外。
在判断初始参考像素点是否在第二子图像之外时，首先要先判断初始参考像素点在参考图像中的位置，在确定初始参考像素点的位置时，对于解码器和编码器，具体确定过程如下：
对于解码器，可以根据当前像素点的位置以及通过解码运动信息码流得到的当前像素点的运动信息确定初始参考像素点在参考图像中的位置。例如，解码器可以根据运动信息中的参考图像指示信息确定参考图像，根据运动信息中的运动矢量信息以及当前像素点的位置确定参考图像中初始参考像素点的位置。
对于编码器，在为当前块搜索参考块，或者在为当前像素点搜索参考像素点时，当前参考像素点的位置即为初始参考像素点的位置。
可选地，上述参考图像对应的多面体的布局信息包含参考图像对应的多面体的面的数量信息，参考图像的子图像的排布方式信息，参考图像的子图像的排布顺序信息，参考图像的子图像的旋转信息中的至少一种。
具体地，参考图像对应的多面体的面的数量信息具体可以为参考图像对应的是什么样的多面体，具体地，多面体的面的数量信息可以指示参考图像的对应的是正六面体。
参考图像的子图像的排布方式信息是指参考图像中的各个子图像的排布方式，如图7所示，参考图像对应的是正六面体，那么参考图像包含6个子图像，这6个子图像的排布可以是4×3型(图7中的(b))、3×2型(图7中的(c)、图7中的(d))、6×1型(图7中的中的(e)、图7中的(f))。
参考图像的子图像的排布顺序信息是指参考图像的各个子图像在参考图像中的排列顺序，例如，图7中的(c)和图7中的(d)都是3×2型的图像，在图7中的(c)中，Front面对应的子图像排列在图像的左下角，而在图7中的(d)中，Front面对应的子图像排列在第一列的中间位置。
参考图像的子图像的旋转信息可以指参考图像的子图像的旋转角度，假设以图7中的(c)中各个子图像的放置位置为基准，那么在图7中的(d)中，Front面对应的子图像的旋转角度为-90度。
150、根据目标参考像素点的像素值和/或目标参考像素点的临近像素点的像素值，确定当前像素点的像素值的预测值。
本申请实施例中，当初始参考像素点位于第二子图像之外时，根据初始参考像素点的位置寻找到真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，提高了运动补偿预测的准确性。
可选地，作为一个实施例，根据初始参考像素点的位置以及参考图像对应的球面体的球心位置，确定当前像素点的目标参考像素点对应在球面体表面的球面参考位置，包括：
根据初始参考像素点的位置和初始参考像素点的位置相对于第二子图像的偏移值，确定初始参考像素点在多面体中的第一面所在平面的第一位置，其中，第一面是多面体中与第二子图像对应的面；
根据第一位置以及球面体的球心位置，确定目标参考像素点对应在球面体表面的球面参考位置，其中，第一位置与球心位置的连线经过球面参考位置。
应理解，上述初始参考像素点的位置相对于第二子图像的偏移值可以是初始参考像素点的位置相对于第二子图像中某个点(例如，第二子图像的中心点)的位置偏移值。
初始参考像素点的位置可以是初始参考像素点在第二子图像所在的平面的坐标系中的位置，而第一位置可以是初始参考像素点对应在多面体(该多面体是与参考图像对应的多面体)中的第一面所在的平面的坐标系中的位置，第一面所在的平面的坐标系可能与第二子图像所在的平面的坐标系不同。这里是根据初始参考像素点在一个坐标系中的位置转换到另一个坐标系中的位置。如果参考图像对应的多面体中的第一面与参考图像中的第二子图像所在的面重合，那么初始参考像素点的位置与第一位置重合。
如图8所示，参考图像对应的正六面体为ABCD-EFGH，正六面体中的ABCD面与 参考图像中的第二子图像对应，ABCD面所在的平面为A’B’C’D’,参考图像对应的球面体内切在正六面体ABCD-EFGH中，M点为目标参考像素点对应在球面体上的球面参考位置，M’点为目标参考像素点对应在多面体上的多面体参考位置，根据初始参考像素点的第一位置(图8中未表示出第一位置)以及球心O的位置可以确定M点的位置，根据M点的位置可以确定M’的位置，接下来就可以根据M’的位置来确定目标参考像素点在参考图像中的位置了。
下面结合实例一对根据初始参考像素点的位置以及球面体的球心位置，确定球面参考位置进行详细的说明。
实例一：
如图9所示，参考图像为4×3格式的六面体全景图像，当前像素点的初始参考像素点为T，当前像素点在当前图像中的Bottom面，R点为参考图像中的Bottom面中的任意一点，T的位置与T相对于R的位置偏移是已经的。图10为参考图像对应的球面体，球面体内切在参考图像对应的多面体(图10中未画出该多面体)中，球面体与多面体的Bottom面相切，多面体的Bottom面与参考图像中的Bottom面是对应的。根据T的位置以及T相对于R的偏移值，可以确定T’的位置(也就是初始参考像素点的第一位置)。在图10中，A’B’C’D’为ABCD面所在的平面，以球面体的球心为坐标原点建立空间直角坐标系x-y-z，为了描述方便，这里规定x-O-y面与面A’B’C’D’平行，O点在面A’B’C’D’的投影像素点为O’，O’Q平行于y轴，O’P平行于x轴，PT’垂直于O’P，T’Q垂直于O’Q，x-y面上的ON平行于O’T’，初始参考像素点的第一位置为T’，目标参考像素点在球面体上的球面参考位置为T”。
由于球半径OO’长度已知，那么根据T’的位置就可以确定球面上的T”点的位置。具体来说，已知PT’和QT’的长度分别为LPT'和LQT'，由图中的几何关系可知：
Figure PCTCN2017104030-appb-000001
Figure PCTCN2017104030-appb-000002
Figure PCTCN2017104030-appb-000003
由于球面体的半径已知，因此，根据球面体的半径以及∠O"ON和∠WOT"就可以确定T”点的位置，也就是确定了目标参考像素点在球面体多面的球面参考位置。
下面结合实例二对根据球面参考位置确定目标参考像素点对应在多面体的表面的多面体参考位置进行详细的说明。
实例二：
图10中已经确定了球面参考位置，接下来根据球面参考位置确定多面体参考位置，在图11中，参考图像对应的多面体为ABCD-EFGH，多面体的棱长为a，O为多面体的体心，O’为O在面ABCD上的投影，J为BC边的中点，T”为球面参考位置，OT”的延长线交多面体于H点，交ABCD所在面于T点，K为T在O’J延长线上的垂直投影，S为H点在BC上的垂直投影，I为T在BC边上的垂直投影，为了便于描述，这里假设BC边平行于x轴。
如果确定了SJ和SH的长度，那么就可以确定H点的位置，也就是确定了多面体参考位置。下面描述如何确定SJ和SH的长度。
已知OO’和O’J的长度LOO'和LO'J均为
Figure PCTCN2017104030-appb-000004
此外，由于T”位置已知，那么∠O'OT也是已知的，并且∠QO'T＝∠O"ON，那么可知：
∠JO'S＝90°-∠QO'T     (4)
LSJ＝LO'J*tan∠JO'S     (5)
LO'T＝LOO'*tan∠O'OT     (6)
Figure PCTCN2017104030-appb-000005
Figure PCTCN2017104030-appb-000006
由公式(5)和公式(8)可知SJ和SH的长度，这样就确定了H点的位置,也就确定了多面体参考位置。
应理解，公式(8)是根据LST、LOO'以及LO'T的长度来确定LSH。实际上除了根据公式(8)确定LSH之外，还可以根据其它公式确定LSH，根据其它公式确定LSH的具体过程如下：
在图11中，根据三角形相似的几何关系可得：
Figure PCTCN2017104030-appb-000007
Figure PCTCN2017104030-appb-000008
由公式(b)和公式(c)可得：
Figure PCTCN2017104030-appb-000009
对公式(c)进一步推导可得：
Figure PCTCN2017104030-appb-000010
由公式(d)可知，还可以根据LJK、LOO'以及LO'K来确定LSH。
上文结合实例二对参考图像的多面体为正六面体的情况下如何确定多面体参考位置进行了详细的描述，下面结合实例三和实例四，对参考图像对应的多面体为其它多面体的情况下如何确定多面体参考位置进行了详细的描述进行详细的描述。
实例三：
图12中包含两个多面体，图12中的(a)中的多面体的相邻面之间的夹角大于90度，图12中的(b)中的多面体的相邻面之间的夹角小于90度。在图12中的(a)中参考图像对应的多面体包含两个面ACB和DBC，面ACB与面DBC的夹角大于90度(多面体还可以包含其它的面，这里不一一列出)，ACB面为参考图像中第二子图像所在面的投影平面，BC边为第二子图像所在面与相邻面的相接的边。J为O’到BC边的垂线O’K与BC边的交点，OT”的延长线交多面体于H点，并交ACB面所在平面于T点，O’T与BC 边交于S点，OT与面BCD交于H点，H”为H在O’K上的垂直投影，L为H”在O’K上的垂直投影。
由于多面体的几何形状已经确定(包括边长，相邻面的夹角，体心位置等)，那么O’的位置以及OO’的长度确定，J的位置也是确定的，如果确定了SJ(与HH”长度相等)和SH的长度就可以确定OT与面BCD的交点H的位置，也就是确定了目标参考像素点在多面体表面的多面体参考位置。
具体地，已知OO’的长度为LOO'，O’J的长度为LO'J，∠TO'K以及∠O'OT，∠O'SH＝θ，由几何关系可知：
LO'T＝LO'O*tan∠O'OT     (8)
LKT＝LO'T*sin∠TO'K     (9)
LJK＝LO'T*cos∠TO'K-LO'J     (10)
LO'K＝LO'J+LJK     (11)
Figure PCTCN2017104030-appb-000011
Figure PCTCN2017104030-appb-000012
Figure PCTCN2017104030-appb-000013
Figure PCTCN2017104030-appb-000014
Figure PCTCN2017104030-appb-000015
Figure PCTCN2017104030-appb-000016
LO'H'＝LO'S+LSH'      (18)
Figure PCTCN2017104030-appb-000017
Figure PCTCN2017104030-appb-000018
Figure PCTCN2017104030-appb-000019
根据上述公式(12)和(21)最终可以得到
Figure PCTCN2017104030-appb-000020
得到了LSJ和LH"J，进而可以确定多面体参考位置H。
在图12中的(b)中，参考图像对应的多面体的面ACB与面DBC的夹角小于90度，那么：
Figure PCTCN2017104030-appb-000021
LO'H'＝LO'S-LSH'      (23)
Figure PCTCN2017104030-appb-000022
Figure PCTCN2017104030-appb-000023
Figure PCTCN2017104030-appb-000024
根据上述公式(12)和(26)最终可以得到
Figure PCTCN2017104030-appb-000025
得到了LSJ和LH"J，进而可以多面体参考位置H。
应理解，在实例一和实例三中描述了根据第一位置确定球面体参考位置，以及根据球面体参考位置确定多面体参考位置。实际上，也可以直接根据根据第一位置确定多面体参考位置。下面结合实例四和实例五对根据第一位置直接确定多面体参考位置(也就是目标参考像素点投影到多面体表面的位置)进行详细的描述。
实例四：
在图13中，图13中的(a)为当前图像，图13中的(b)为参考图像，P为当前图像中的当前像素点，P位于当前图像的Bottom面，P′为参考图像中与P处于同位置的像素点，P′位于参考图像的Bottom面，T1、T2和T3均为当前像素点的初始参考像素点，它们均位于参考图像的Bottom面之外。
图14中的正六面体以参考图像的Bottom面为底面构建的与参考图像对应的多面体，图14示出了图13中的(b)中的T1、T2和T3三个当前参考像素点，由图14可知，三个目标参考像素点在该正六面体上的对应像素点分别为H1、H2和H3，它们分别位于T1、T2和T3与O点连线与正六面体的交点的位置，这样就可以根据T1、T2和T3的位置确定H1、H2和H3的位置。应理解，在图14中，T1、T2和T3只是示出了初始参考像素点位于不同位置的情况，事实上，同一时间只会存在一个参考像素点。
下面以图15为例，详细介绍如何根据初始参考像素点的位置确定目标参考像素点在多面体的对应像素点的位置。
具体以T1为例，对根据T1在的位置确定目标像素点投影到多面体的表面的投影像素点的位置进行详细的说明。
图15中的正六面体以参考图像的Bottom面为底面构建的与参考图像对应的多面体，也就是说该正六面体的锚定面(锚定面可以理解为构建多面体时的底面，例如，以参考图像的某个面作为构建的多面体的底面，那么该底面就是多面体的锚定面)该参考图像的Bottom面。该正六面体的棱长为a，O为正六面体的体心，T位置为初始参考像素点的位置，ABCD面为参考图像中与当前像素点处于同位置的像素所在的面，O’为O在当前像素点所在面ABCD所在投影平面上沿着面法线方向投影ABCD面上的垂直投影，J为BC边的中点，K为T在O’J延长线上的垂直投影，OT交BCGF面于H点(H点所在的位置就是初始参考像素点的投影像素点所在的位置)，H点在BC上的垂直投影为S，I为T在BC边上的垂直投影。由于正六面体的棱长为a，因此，OO’和O’J的长度均为
Figure PCTCN2017104030-appb-000026
假设JK线段的长度为x，KT线段长度为y，则根据相似三角形可得：
Figure PCTCN2017104030-appb-000027
Figure PCTCN2017104030-appb-000028
其中，LSJ为SJ的长度，LSH为SH的长度，在根据公式(1)和(2)得到SJ和SH的长度后就可以确定H点在正六面体的表面的位置。应理解，可以根据初始参考像素的位置坐标确定SJ和SH的长度，进而得到H点在正六面体的表面的位置。
进一步地，对公式(28)进行化简，可以得到：
Figure PCTCN2017104030-appb-000029
此外，在图15中，还可以直接根据三角形相似的几何关系来得到公式(e)，具体过程如下：
根据三角形相似的几何关系可得：
Figure PCTCN2017104030-appb-000030
Figure PCTCN2017104030-appb-000031
由公式(b)和公式(c)可得：
Figure PCTCN2017104030-appb-000032
将OO’的长度
Figure PCTCN2017104030-appb-000033
JK长度x,O’K长度
Figure PCTCN2017104030-appb-000034
代入公式(h)中即可得到公式(e)。
实例五：
如图16所示，参考图像对应的多面体包含两个面ACB和DBC(多面体还可以包含其它的面，这里不一一列出)，其中，面ACB为参考图像中与当前像素点处于同位置的像素点所在的面，O为多面体的体心，O’为O在ABC面上的垂直投影，T位置为初始参考像素点所在的位置，O’T与BC边交于S点，OT与面BCD交于H点，H”为H在O’K上的垂直投影，L为H”在O’K上的垂直投影，I为T在BC上的垂直投影。
如图16中的(a)所示，已知OO’的长度为LOO'，O’J的长度为LO'J，JK的长度为LJK，KT的长度为LKT，面ACB与面BCD的夹角为α(α大于90°)，∠O'SH＝θ，那么可以得到：
LO'K＝LO'J+LJK       (29)
Figure PCTCN2017104030-appb-000035
Figure PCTCN2017104030-appb-000036
Figure PCTCN2017104030-appb-000037
Figure PCTCN2017104030-appb-000038
Figure PCTCN2017104030-appb-000039
Figure PCTCN2017104030-appb-000040
LO'H'＝LO'S+LSH'       (36)
Figure PCTCN2017104030-appb-000041
Figure PCTCN2017104030-appb-000042
Figure PCTCN2017104030-appb-000043
当多面体的相邻面的夹角大于90°时，根据上述公式(30)和(39)最终可以得到
Figure PCTCN2017104030-appb-000044
得到了LSJ和LH"J，进而可以确定目标像素点投影到多面体的表面的位置。
在图16中的(b)中，面ACB与面BCD的夹角α小于90°。那么：
Figure PCTCN2017104030-appb-000045
LO'H'＝LO'S-LSH'       (41)
Figure PCTCN2017104030-appb-000046
Figure PCTCN2017104030-appb-000047
Figure PCTCN2017104030-appb-000048
当多面体的相邻面的夹角小于90°时，根据上述公式(30)和(44)最终可以得到
Figure PCTCN2017104030-appb-000049
得到了LSJ和LH"J，进而可以确定目标像素点投影到多面体的表面的位置。
应理解，实例五中的公式与实例三中的公式较为类似，在实例三计算LSJ和LH"J的过程中用到了球面参考位置J”在球面的角度参数，而在实例五中并未借助J”而是直接推导最终的结果，这两种实现方式都能确定目标参考像素点在多面体表面的多面体参考位置。
在一般的多面体中，四面体相邻面夹面角小于90°，正六面体相邻面夹面角为90°，正八面体以及更复杂的多面体相邻面夹面角大于90°。对于其他多面体格式的图像，可以按照类似方法得到相应参数，最终得到LSJ和LH"J，进而确定目标参考像素点投影到多面体的表面的投影像素点的位置。应理解，这里的投影像素点的位置为指初始参考像素点与多面体体心的连线与多面体表面的交点的位置。
可选地，作为一个实施例，根据多面体参考位置，确定目标参考像素点在参考图像中的位置，包括：根据多面体参考位置以及多面体的布局信息，确定目标参考像素点在参考图像中的位置。
可选地，可以将目标参考像素点在多面体的表面的位置的点理解为目标参考像素点的投影像素点，这样在根据初始参考像素点的位置确定当前像素点的目标参考像素点在参考图像中的位置时，具体包括：根据初始参考像素点的位置和参考图像的布局信息，确定初始参考像素点的投影像素点在多面体的表面的位置，投影像素点在多面体的表面的位置在初始参考像素点与多面体的中心点的连线与多面体的表面的交点处；根据投影像素点在多面体的表面的位置和参考图像的布局信息，确定目标参考像素点在参考图像中的位置。
下面结合图17，以正六面体格式的二维图像为例，具体描述如何确定目标参考像素点在参考图像中的位置，如图17中的(a)所示，假设参考图像对应的正六面体的边长为a，参考图像的布局格式为4x3，当前像素点的目标参考像素点投影到正六面体的front面的H1点，H1相对于Front面左上角顶点的距离分别为△x、△y。下面确定不同的局部格式的二维图像中目标参考像素点在参考图像中的位置。
在图17中的(b)中，图像的宽度为4a，高度为3a，Front面位于左下角，并且没有旋转，那么H1在参考图像中相对于参考图像左上角顶点Q的位置为(△x，a+△y)。
在图17中的(c)中，Front面仍位于左下角与图12中的(b)相同的位置，但是顺时针旋转了90度，则H1在参考图像中相对于参考图像左上角顶点Q的位置为(a-△y，2a+△x)。
可选地，作为一个实施例，根据目标参考像素点的像素值和/或目标参考像素点的临近像素点的像素值，确定当前像素点的像素值的预测值，包括：将目标参考像素点的像素值确定为当前像素点的像素值的预测值。
上述目标参考像素点的像素值可以是该目标参考像素点所在位置的像素值，将目标参考像素点的像素值直接确定为当前像素点的预测值，能够减少计算流程。
可选地，作为一个实施例，根据目标参考像素点的像素值和/或目标参考像素点的临近像素点的像素值，确定当前像素点的像素值的预测值，包括：对目标参考像素点的像素值以及目标参考像素点的临近像素点的像素值进行加权处理；将加权处理得到的目标参考像素点所在位置的像素值确定为当前像素点的像素值的预测值。
上述加权处理可以是对目标参考像素点以及周围的像素点进行平滑滤波处理，也就是对包含目标参考像素点在内的多个像素点的像素值进行平均处理，将得到的像素值的平均值作为当前像素点的像素值的预测值。
在对当前像素点的像素值进行预测时可以直接将将目标参考像素点的像素值作为当前像素点的像素值的预测值，或者，不单独适用当前像素点的像素值对当前像素点的像素值进行预测，而是根据目标参考像素点以及目标参考像素点周围临近的像素点共同对当前像素点的像素值进行预测。
可选地，作为一个实施例，根据目标参考像素点的像素值和/或目标参考像素点的临近像素点的像素值，确定当前像素点的像素值的预测值，包括：根据目标参考像素点的临近像素点的像素值在目标参考像素点位置进行插值运算；将插值运算得到的像素值确定为当前像素点的像素值的预测值。
在确定了目标参考像素点的像素值之后，可以先对目标参考像素点的像素值的精度进行判断，当目标参考像素点的像素值的精度小于当前像素点的运动矢量精度时，需要利用插值算法对目标像素点周围的像素点的像素值进行计算，将计算得出的像素值作为目标参 考像素点的像素值。具体来说，当目标参考像素点的像素值的精度为1/2像素精度，当前像素点的运动矢量精度为1/4像素精度时，目标参考像素点的像素值的精度小于当前像素点的运动矢量的精度，因此需要采用插值算法重新计算目标参考像素点的像素值。
具体地，如图18所示，投影点H的坐标为(i+u,j+v),其中i,j均为非负整数，u,v均为[0,1)区间的浮点数，H周围临近四个整数像素所在的位置坐标分别为P(i,j)、P(i,j+1)、P(i,j+1)和P(i+1,j+1)。
如果采用双线性内插算法，那么可以得到H处的像素值为：
PH＝P(i,j)(1-u)(1-v)+P(i,j+1)(u)(1-v)+P(i,j+1)(1-u)(v)+P(i+1,j+1)(u)(v)    (19)
而如果采用双三次插值算法，投影点H的坐标仍为(i+u,j+v),H点周围的16个相邻点的坐标分别为P(i-1,j-1)、P(i-1,j+0)、P(i-1,j+1)、P(i-1,j+2)、P(i+0,j-1)、P(i+0,j+0)、P(i+0,j+1)、P(i+0,j+2)、P(i+1,j-1)、P(i+1,j+0)、P(i+1,j+1)、P(i+1,j+2)、P(i+2,j-1)、P(i+2,j+0)、P(i+2,j+1)以及P(i+2,j+2)。
采用双三次插值算法，得到H处的像素值为：
PH＝[A]*[B]*[C]       (45)
[A]＝[S(u+1) S(u+0) S(u-1) S(u-2)]      (46)
Figure PCTCN2017104030-appb-000050
Figure PCTCN2017104030-appb-000051
Figure PCTCN2017104030-appb-000052
以上仅仅以双双线性插值算法和双三次插值算法为例对本申请实施例中如何利用插值算法确定投影像素位置处的像素进行了详细的说明，实际上还可以采用Lanczos插值算法、最邻近插值算法以及一些基于图像结构等信息的非解析插值方法等。
应理解，在确定了目标参考像素点的像素值后，可以不考虑目标参考像素点的像素值是否满足相应的要求，而是直接将目标参考像素点的像素值确定为当前像素点的像素值。另外，还可以先在确定了目标参考像素点的像素值后，对目标参考像素点的像素值是否满足预设要求进行判断，当目标参考像素点的像素值不满足要求时，可以根据目标参考像素点的临近像素点的像素值对所述目标参考像素点进行插值运算，将得到的像素值作为当前像素点的像素值的预测值，或者，还可以对目标参考像素点的像素值以及目标参考像素点的临近像素点的像素值进行加权处理，将加权处理得到的结果作为当前像素点的像素值的预测值。
图19是本申请实施例的运动补偿预测方法的示意性流程图。图19所示的方法包括：
210、确定当前参考像素点的初始参考像素点在参考图像中的位置，当前像素点位于当前图像的第一子图像内；
220、当初始参考像素点位于参考图像中与第一子图像对应位置的第二子图像之外时，确定当前像素点在第二子图像的扩展区域中的目标参考像素点的位置，其中，第二子图像的扩展区域位于第二子图像之外，扩展区域包括多个像素点，扩展区域中任意一个第一像素点的像素值是根据参考图像中的第二像素点的像素值确定的，所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置与所述球心的连线经过所述第二像素点对应在所述参考图像对应的球面体的球面参考位置，所述球面参考位置是根据所述初始参考像素点的位置以及所述球面体的球心位置确定的，所述球面体内切于所述多面体；
230、根据目标参考像素点的像素值和/或目标参考像素点的临近像素点的像素值，确定当前像素点的像素值的预测值。
本申请实施例中，当初始参考像素点位于第二子图像之外时，直接根据当前像素点以及当前像素点的运动信息直接在扩展区域中寻找真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，能够快速寻找到目标参考像素点，并且还能提高运动预测估计的准确性。
具体而言，在现有技术中，并未针对参考图像中的每个子图像构建扩展区域，而是直接根据初始参考像素点的像素值计算当前像素点的像素值的预测值，而不考虑初始参考像素点在参考图像中的位置。而在本申请实施例中，可以依照图5所示的方法为参考图像中的子图像构建扩展区域，这样当初始参考像素点的位置不处于参考图像中与当前像素点对应的子图像区域时，根据运动信息在扩展区域中确定参考像素点的位置，并将扩展区域中参考像素点的像素值作为当前像素点像素值的预测值。
应理解，上述扩展区域中的像素点的像素值可以根据上文中图5所示的方法进行计算，这样在对当前像素点的像素值进行预测时，能够直接从扩展区域中找到目标参考像素点，能够快速地实现对当前像素点的像素值的预测。
可选地，作为一个实施例，图19所示的方法还包括：根据第一像素点的位置以及参考图像对应的球面体的球心位置，确定第二像素点对应在球面体表面的球面参考位置；根据球面参考位置确定第二像素点对应在参考图像对应的多面体的表面的多面体参考位置；根据多面体参考位置，确定第二像素点在参考图像中的位置。
可选地，作为一个实施例，根据第一像素点的位置以及参考图像对应的球面体的球心位置，确定第二像素点对应在球面体表面的球面参考位置具体包括：
根据第一像素点的位置以及第一像素点的位置相对于第二子图像的偏移值，确定第一像素点在多面体中的第一面所在平面的第一位置，其中，第一面是多面体中与第二子图像对应的面；
根据第一位置以及球面体的球心位置，确定第二像素点对应在球面体表面的球面参考位置，第一位置与球心位置的连线经过球面参考位置。
也就是说上述扩展区域中的任意一个像素点像素值都可以采用图5所示的方法来确定，根据第一像素点的位置确定第二像素点对应在球体的球面参考位置，然后再根据球面参考位置确定第二像素点对应在多面体的多面体参考位置，然后根据多面体参考位置确定第二像素点在参考图像中的位置，最后根据第二像素点在参考图像中的位置再确定第一像 素点的像素值，采用同样的方法确定扩展区域中的每个像素点的像素值，这样在进行运动补偿预测时就可以直接确定扩展区域中的目标参考像素点的位置，提高了运动补偿预测的效率。
可选地，作为一个实施例，图19所示的方法还包括：根据参考图像的布局信息和初始参考像素点的位置判断初始参考像素点是否位于所述参考图像中的第二子图像之外。
应理解，判断初始参考像素点的位置是否在第二子图像之外可以是在确定了初始参考像素点在参考图像中的位置之后，并且根据初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定球面参考位置之前进行的。
可选地，作为一个实施例，初始参考像素点位于参考图像中的第二子图像之内，根据初始参考像素点的像素值和/或目标参考像素点的临近像素点的像素值，确定当前像素点的像素值的预测值。也就是说，当初始参考像素点位于参考图像中的第二子图像之内时可以直接根据第一初始参考像素点来对当前像素点的像素值进行预测。
具体来说，图20所示的运动补偿预测方法是先确定参考图像的某个子图像的扩展区域(也可以确定参考图像中的多个子图像的扩展区域)，然后确定参考像素点在扩展区域中的位置，这样就可以在对当前像素点进行预测时能够直接从扩展区域中寻找参考像素点，并根据该参考像素点的像素值对当前像素点进行预测。下面结合图20对上述过程进行详细说明，具体步骤如下：
201、在参考图像中为与当前像素点处于对应位置的面构建扩展区域。
根据参考图像对应的多面体的布局信息(一般情况下，参考图像对应的多面体的布局信息与当前图像的布局信息相同，因此，可以用来确定当前图像中的像素点所在的面)以及当前像素点的位置确定当前像素点的所在面，找到参考图像中与前像素点对应的面，并为该对应面构建扩展区域，并确定扩展区域中的像素点的像素值。扩展区域的具体效果类似于在面的周围对该面进行扩充或者填补，如图20中的(a)所示，当前像素点P处于Bottom面，那么接下来需要为参考图像中的Bottom面构建扩展区域，该扩展区域的范围如20中的(b)中的Bottom面外的虚线区域所示，在计算扩展区域中的像素点的像素值时可以以参考图像的Bottom为底面构建一个如图20中的(c)所示与该参考图像对应的多面体，如果将该多面体展开，那么多面体的Rear面、Front面、Left面、Right面等分别落在扩展区域的位置如图20中的(d)所示。对于扩展区域中的任一第一像素点，可以按照上文中图5所示的本申请实施例的运动补偿预测方法，根据第一像素点的位置，参考图像对应的球面体的球心位置先确定球面参考位置，再确定多面体参考位置，最终确定第二像素点中在参考图像中的位置，并根据第二像素点确定第一像素点的像素值(可以将第二像素点处的像素值直接确定为第一像素点的像素值，也可以对第二像素点或者第二像素点周围的像素点进行一定的运算，将运算得到的像素值作为第一像素点的像素值)。在这里，第一像素点类似于图5所示的本申请实施例的运动补偿预测方法中的初始参考像素点，第二像素点类似于图5所示的本申请实施例的运动补偿预测方法中的目标参考像素点。
202、确定当前像素点的在扩展区域中的目标参考像素点的位置。
首先确定当前像素点的参考像素点，并根据当前像素点所在的面以及参考像素点的位置判断参考像素点是否处于与当前像素点处于同位置的面。如果参考像素点是否处于与当前像素点处于同位置的面，那么就可以直接将参考像素点位置的像素值作为当前像素点像 素值的预测值。否则，则需要执行以下操作：首先根据参考像素点与当前像素点的相对位置偏移，确定参考像素点在扩展区域中的对应位置。如图20中的(b)所示，参考像素点T不处于与当前像素点P处于同位置的面，根据参考像素点与当前像素点的相对位置偏移，可知参考像素点T应位于扩展区域中Rear面对应的区域。如果扩展区域按照实际的相对空间位置进行放置，如图20中的(d)，则按照相对位置偏移确定参考像素点在扩展区域中的位置。否则，需要根据具体扩展区域的放置方法来确定参考像素点在扩展区域中的位置。
203、确定当前像素点的像素值的预测值。
确定参考像素点在扩展区域中的对应位置之后，将此位置上的像素值作为当前像素点像素值的预测值。
应理解，上述扩展区域内的像素点以为整数像素，也可以为分数像素，如果为分数像素时，具体像素值可以是1/2精度，1/4精度，1/8精度等。
另外，在确定了参考像素点在扩展区域位置之后，如果此位置没有像素值，则需要对该参考像素点进行插值运算。具体的插值运算方法可以使用DCT插值，双线性插值或者其他插值方法。
应理解，图19所示的方法与图5所示的方法的区别在于，图19中是预先计算出扩展区域中像素点的像素值，这样在对当前像素点进行预测时就能够更方便地从扩展区域中直接找到目标参考像素点，省去了根据初始参考像素点的位置确定目标参考像素点的位置的计算过程。而图5所示的方法是在对当前像素点的像素值进行预测时，再根据初始参考像素点的位置确定目标参考像素点的位置，与图19所示的方法相比多了一个计算过程。另外，图19所示的方法其它步骤与图5所示的方法的其它步骤基本相同，为了简洁，适当省略重复的描述。
图21是本申请实施例的运动补偿预测方法的示意性流程图。图21所示的方法包括：
310、确定当前像素点在参考图像中的初始参考像素点位置，当前像素点位于当前图像的第一子图像内；
320、当初始参考像素点位于参考图像中与第一子图像对应位置的第二子图像之外时，根据初始参考像素点位置确定当前像素点的目标参考像素点在参考图像对应的多面体的表面的位置，其中，目标参考像素点在多面体的表面的位置与初始参考像素点在第一平面的位置的连线经过多面体的中心点，第一平面为多面体中与第二子图像对应的面所在的平面；
330、根据目标参考像素点在多面体表面的位置确定目标参考像素点在参考图像中的位置；
340、根据参考图像中目标参考像素点的像素值和/或目标参考像素点的临近像素点的像素值，确定当前像素点的像素值的预测值。
本申请实施例中，当初始参考像素点位于第二子图像之外时，根据初始参考像素点的位置寻找到真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，提高了运动补偿预测的准确性。
应理解，本申请实施例中，图5所示的运动补偿预测方法与图21所示的运动补偿预测方法的区别在于图5中是根据初始参考像素点的位置和多面体的球心位置确定球面参 考位置，然后再根据球面参考位置确定多面体参考位置，最再根据多面体参考位置确定目标参考像素点在参考图像中的位置，而图21是先根据初始参考像素点的位置确定目标参考像素点在参考图像对应的多面体的表面的位置，然后再根据目标参考像素点在多面体表面的位置确定目标参考像素点在参考图像中的位置。两者比较话，图5所示的方法是通过三个步骤来确定目标参考像素点在参考图像中的位置，而图16所示的方法是通过两个步骤来确定目标参考像素点在参考图像中的位置。应理解，对图5中所示的运动补偿预测方法的相关解释和说明同样适用于对图16所示的运动补偿预测方法，因此，为了简洁，此处适当省去重复的描述。
可选地，作为一个实施例，根据初始参考像素点位置确定当前像素点的目标参考像素点在参考图像对应的多面体的表面的位置，包括：根据初始参考像素点的位置和参考图像的布局信息，确定初始参考像素点在第一平面的位置；根据初始参考像素点在第一平面的位置和参考图像的布局信息，确定目标参考像素点在多面体的表面的位置。
可选地，作为一个实施例，根据目标参考像素点在多面体表面的位置确定目标参考像素点在参考图像中的位置，包括：根据初始参考像素点的位置和参考图像的布局信息，确定目标参考像素点在参考图像中的位置，目标参考像素点在多面体的表面的位置在初始参考像素点与多面体的中心点的连线与多面体的表面的交点处。
应理解，本申请实施例的运动补偿预测方法适用于各种多面体格式的二维图像，而且对于多面体格式的二维图像的面发生了旋转，面的排列次序发生了变化，以及面的空间布局格式发生了改变的情况，本申请实施例的运动补偿预测方法也都适用。
应理解，在本申请实施例中，可以在对当前像素点的像素值进行预测时去计算目标参考像素点的位置的像素值。也可以预先计算出待处理像素的目标参考像素点的像素值，这样在对待处理像素进行处理时可以直接获取待处理像素的目标参考响度点的像素值，节省了处理图像的时间。
应理解，虽然上述文中的以六面体格式的二维图像为例，但是对于其他多面体格式二维图像，本方法同样适用。包括四面体，八面体以及其他多面体格式。
应理解，上文中是以4x3格式的二维图像对本申请实施例的运动补偿预测方法进行了详细的说明，对于其他类型格式的多面体二维图像，本申请实施例的运动补偿预测方法同样适用，其中，二维图像的面带有部分或者全部旋转，不同面的排布顺序和排布方法等情况也都适用本申请实施例的运动补偿预测方法。
应理解，在本实施例中可以直接将参考图像中参考像素点的像素值作为当前像素点像素值的预测值，也可以利用参考像素点的像素值及其周围像素点的像素值的加权值或者进行其他运算得到的像素值作为当前像素点的像素值的预测值。
本申请实施例的运动补偿预测方法可以应用在图22所示的架构中，并可以具体应用在对二维全景图像的编码和解码的过程中。在图22中，现有的VR视频图像采集设备通常为环状或球状的多摄像头阵列，各个摄像头可以采集不同角度的图像得到当前场景的多视角视频图像，接下来，VR视频处理设备对多视角图像做图像拼接得到三维球面全景图，再将三维球面全景图映射得到二维全景图像，便于后续处理、压缩、传输、存储等操作的输入。并将处理后的结果存储到传输服务器中，VR头显设备可以通过网络等方式获取传输服务器中存储的图像处理结果，进行相应的解码等操作后就可以显示三维球面全景图像 了。
上文结合图1-22详细的描述了本申请实施例的运动补偿预测方法，下面结合图23-28，详细描述本申请实施例的运动补偿预测装置。应理解，图23-28中的运动补偿预测装置能够执行图1-22中的运动补偿预测方法的各个步骤，为了避免重复，适当省略重复的描述。
图23是本申请实施例的运动补偿预测装置的示意性框图。该运动补偿预测装置400包括：
第一确定单元410，用于确定当前像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内。
第二确定单元420，用于当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述当前像素点的目标参考像素点对应在所述球面体表面的球面参考位置；
第三确定单元430，用于根据所述球面参考位置确定所述目标参考像素点对应在所述参考图像对应的多面体的表面的多面体参考位置，其中，所述球面体内切于所述多面体，所述多面体参考位置与所述球心的连线经过所述球面参考位置；
第四确定单元440，用于根据所述多面体参考位置，确定所述目标参考像素点在所述参考图像中的位置；
预测单元450，用于根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
本申请实施例中，当初始参考像素点位于第二子图像之外时，由于二维图像的面边界的几何形变的影响，导致直接根据初始参考像素点对当前像素点进行预测是不准确的，因此，在这种情况下，本申请是根据该初始参考像素点的位置寻找到真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，提高了运动预测估计的准确性。
可选地，作为一个实施例，所述第二确定单元420具体用于：
根据所述初始参考像素点的位置和所述初始参考像素点的位置相对于所述第二子图像的偏移值，确定所述初始参考像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；
根据所述第一位置以及所述球面体的球心位置，确定所述目标参考像素点对应在所述球面体表面的球面参考位置，其中，所述第一位置与所述球心位置的连线经过所述球面参考位置。
可选地，作为一个实施例，所述运动补偿预测装置400还包括：
判断单元460，用于根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
可选地，作为一个实施例，所述第四确定单元440具体用于：
根据所述多面体参考位置以及所述多面体的布局信息，确定所述目标参考像素点在所述参考图像中的位置。
可选地，作为一个实施例，所述布局信息包含所述多面体的面的数量信息，所述参考图像的子图像的排布方式信息，所述参考图像的子图像的排布顺序信息，所述参考图像的 子图像的旋转信息中的至少一种。
可选地，作为一个实施例，所述预测单元450具体用于：
将所述目标参考像素点的像素值确定为所述当前像素点的像素值的预测值。
可选地，作为一个实施例，所述预测单元450具体用于：
对所述目标参考像素点的像素值以及所述目标参考像素点的临近像素点的像素值进行加权处理；
将所述加权处理得到的所述目标参考像素点所在位置的像素值确定为所述当前像素点的像素值的预测值。
可选地，作为一个实施例，所述预测单元450具体用于：
根据目标参考像素点的临近像素点的像素值在所述目标参考像素点位置进行插值运算；
将所述插值运算得到的像素值确定为所述当前像素点的像素值的预测值。
图24是本申请实施例的运动补偿预测装置的示意性框图。该运动补偿预测装置500包括：
第一确定单元510，用于确定当前参考像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；
第二确定单元520，用于当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，确定所述当前像素点在所述子图像的扩展区域中的目标参考像素点的位置，
其中，所述第二子图像的扩展区域位于所述第二子图像之外，所述扩展区域包括多个像素点，所述扩展区域中任意一个第一像素点的像素值是根据所述参考图像中的第二像素点的像素值确定的，所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置与所述球心的连线经过所述第二像素点对应在所述参考图像对应的球面体的球面参考位置，所述球面参考位置是根据所述初始参考像素点的位置以及所述球面体的球心位置确定的，所述球面体内切于所述多面体；
预测单元530，用于根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
本申请实施例中，当初始参考像素点位于第二子图像之外时，由于二维图像的面边界的几何形变的影响，导致直接根据初始参考像素点对当前像素点进行预测是不准确的，因此，在这种情况下，本申请是直接根据当前像素点以及当前像素点的运动信息直接在扩展区域中寻找真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，能够快速寻找到目标参考像素点，并且还能提高运动预测估计的准确性。
可选地，作为一个实施例，所述运动补偿预测装置还包括：
第三确定单元540，用于根据所述第一像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置；
第四确定单元550，用于根据所述球面参考位置确定所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置；
第五确定单元560，根据所述多面体参考位置，确定所述第二像素点在所述参考图像中的位置。
可选地，作为一个实施例，所述第三确定单元540具体用于：
根据所述第一像素点的位置以及所述第一像素点的位置相对于所述第二子图像的偏移值，确定所述第一像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；
根据所述第一位置以及所述球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置，所述第一位置与所述球心位置的连线经过所述球面参考位置。
可选地，作为一个实施例，所述运动补偿预测装置还包括：
判断单元570，用于根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
图25是本申请实施例的运动补偿预测装置的示意性框图。该运动补偿预测装置600包括：
第一确定单元610，用于确定当前像素点在参考图像中的初始参考像素点位置，所述当前像素点位于所述当前图像的第一子图像内；
第二确定单元620，当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点位置确定所述当前像素点的目标参考像素点在所述参考图像对应的多面体的表面的位置，其中，所述目标参考像素点在所述多面体的表面的位置与所述初始参考像素点在第一平面的位置的连线经过所述多面体的中心点，所述第一平面为所述多面体中与所述第二子图像对应的面所在的平面；
第三确定单元630，用于根据所述目标参考像素点在多面体表面的位置确定所述目标参考像素点在所述参考图像中的位置；
预测单元640，用于根据所述参考图像中所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
本申请实施例中，当初始参考像素点位于第二子图像之外时，由于二维图像的面边界的几何形变的影响，导致直接根据初始参考像素点对当前像素点进行预测是不准确的，因此，在这种情况下，本申请是根据该初始参考像素点的位置寻找到真正能够起到参考作用的目标参考像素点，然后根据目标参考像素点对当前像素点的像素值进行预测，提高了运动预测估计的准确性。
可选地，作为一个实施例，所述第二确定单元620具体用于：
根据所述初始参考像素点的位置和所述参考图像的布局信息，确定所述初始参考像素点在所述第一平面的位置；
根据所述初始参考像素点在所述第一平面的位置和所述参考图像的布局信息，确定所述目标参考像素点在所述多面体的表面的位置。
可选地，作为一个实施例，所述第三确定单元具体用于：
根据所述初始参考像素点的位置和所述参考图像的布局信息，确定所述目标参考像素点在所述参考图像中的位置，所述目标参考像素点在所述多面体的表面的位置在所述初始参考像素点与所述多面体的中心点的连线与所述多面体的表面的交点处。
上文结合图1-25对本申请实施例的运动补偿预测方法和运动补偿预测装置进行了详细的说明，事实上，本申请实施例的运动补偿预测方法可以看成是编码或者解码过程中的一个中间过程或者步骤，全景视频编码器或者全景视频解码器可以实现本申请实施例的运动补偿预测方法。下面结合图26和图27分别对全景视频解码器的解码过程和全景视频编码器的编码过程进行详细的介绍。
在结合图26和图27介绍全景视频解码器的解码过程和全景视频编码器的编码过程之前，首先对编码和解码进行一些简单的介绍。
在对当前图像进行编码之前，一般是当前图像分成大小相等的若干块，然后再对每个块进行编码操作。在对当前图像进行编码过程中，当前图像的布局信息一般作为头信息传输到解码端。头信息为先导信息，解码器在接收到编码码流后会先解码该头信息，然后根据该头信息对后续的码流进行解码。假设，在编码时当前图像被划分成多个图像块依次进行编码，解码则按照同样的顺序依次进行解码。对于当前块，如果已确定其为帧间编码方式，为了得到其重建信息，要得到当前块内像素点的像素值的重建值。
对于编码来说，该过程主要包括帧内预测(Intra Prediction)、帧间预测(Inter Prediction)、变换(Transform)、量化(Quantization)、熵编码(Entropy encode)等环节。在对图像进行编码时，可以先将图像划分为图像块之后再进行帧内预测和帧间预测，并且在得到残差之后进行变换量化，最终进行熵编码并输出码流。这里的图像块为像素点组成的M×N大小的阵列(MM和N均为大于0的整数，M可以等于N，也可以不等于N)，并且已知各个像素点位置的像素值。
帧内预测是利用当前图像内已重建区域内像素点的像素值对当前块内像素点的像素值进行预测。而帧间预测则是在已重建的图像中，为当前图像中的当前编码块寻找匹配的参考块，将参考块中的像素点的像素值作为当前编码块中像素点的像素值的预测信息或者预测值(以下不再区分信息和值)(此过程称为运动估计(Motion estimation,ME))，并需要向解码端传输当前块指向参考块的运动矢量(motion vector,MV)信息(也就是参考块位置相比于当前块位置的位置偏移)，以及参考块所在图像的指示信息等。运动估计过程需要为当前块在参考图像中尝试多个参考块，最终确定使用哪一个或者哪几个参考块用对当前块进行预测可以使用率失真优化(Rate-distortion optimization，RDO)方法或者其他方法来确定。
在利用帧内预测或者帧间预测得到预测信息之后，可以用当前编码块内像素点的像素值减去对应的预测信息(也就是对应的参考块中对应的像素点的像素值)得到残差信息。在得到残差信息后再利用离散余弦变换(Discrete Cosine Transformation，DCT)等方法对残差信息进行变换，最后再使用量化熵编码得到码流。
对于解码来说，解码则相当于编码的逆过程。解码时，首先利用熵解码反量化反变换得到残差信息，并根据解码码流确定当前块使用的是帧内预测还是帧间预测；如果是帧内编码，那么就利用当前待处理的像素点周围已重建区域内像素点的像素值按照所使用的帧内预测方法构建预测信息；如果是帧间编码，则需要解析出运动信息，并根据解析出的运动信息在已重建的图像中确定参考块，将块内像素点的像素值作为预测信息(此过程也被称为运动补偿)；最后再使用预测信息加上残差信息便可以得到重建信息。
图26是全景视频解码器的解码过程的示意性流程图。具体包括：
710、确定当前图像中的当前块的运动信息。
具体来说，解码器可以解码运动信息码流，根据解析结果确定当前快的运动信息。
应理解，在步骤710中，还可以通过解码预测运动信息的指示信息以及预测运动信息与运动信息的差值信息来确定当前块的运动信息，也就是说可以不直接解析运动信息的编码码流就可以确定当前块的运动信息。例如，解码端可以构建一个运动信息集合(例如，该运动信息集合包含与当前块的时间或者空间相邻的已重建块的运动信息，该集合的构建方法由编解码端共同约定)，那么解码端可以解析预测运动信息的指示信息，然后从该集合中确定预测运动信息，并根据预测运动信息以及预测运动信息与运动信息的差值信息获得当前块的运动信息。此外，应理解，解码端还以将根据预测运动信息的指示信息确定的预测运动信息作为当前块的运动信息，具体使用哪一种方法由编解码端约定，或者编码端将所用的方式信息传输到解码端，解码端根据接收到的方式信息确定使用哪种方式来确定当前块的运动信息。
另外，在步骤710中，如果是单向参考，需解析一套运动信息。如果是双向参考，需解析两套运动信息。具体解析几套运动信息为编解码端共同约定，或者编码端将所用套数相关信息传输到解码端，解码端依据解析结果确定。
720、确定当前块内像素点的像素值的预测值。
解码器可以根据当前块的运行信息来确定当前块所在的当前图像的参考图像，并确定参考块在参考图像中的位置。事实上，确定当前块内像素点的像素值的预测值就是确定当前块内的每个像素点的像素值的预测值，确定像素点的像素值的预测值具体可以根据本申请实施例中的运动补偿预测方法来确定。
应理解，在得到当前块每个像素点的像素值的预测值之后，可以根据每个像素点的像素值的预测值得到当前块的预测块，预测块包含当前块内每个像素点的像素值的预测值。
在步骤720中，如果是单向参考，则只需要确定一个参考块在所在参考图像中的位置，对于当前块内的一个像素点，只需要确定一个参考像素点的位置，然后可以根据本申请实施例的运动补偿预测方法来确定当前块内像素点像素值的预测值。
如果是双向参考，则需要确定当前块的两个参考块在各自参考图像中的位置，对于当前块内的一个像素点，需要分别确定两个参考像素点在两个参考块中的位置。然后可以根据本申请实施例的运动补偿预测方法来确定当前块内像素点的像素值的两个预测值。然后将这两个预测值进行加权或者其他操作得到当前块内像素点像素值的预测值。具体操作方式为编解码端共同约定，或者编码端将所用方法传输到解码端，解码端依据解析结果确定所用方法。
此外，在得到当前块中的像素点的像素值的预测值之后，可以先对该预测值进行其他的操作，将操作得到的预测值作为当前像素点的像素值的最终预测值。例如，可以将得到的预测值经过平滑滤波后得到预测值作为当前像素点的像素值的最终预测值。具体是否使用以及使用哪一种方式为编解码端共同约定，或者编码端将所用方法传输到解码端，解码端依据解析结果确定所用方法。另外，还可以对当前像素点的像素值进行双向预测，也就是说可以对当前像素点进行两次预测，再将得到的两个预测值进行加权得到当前像素点的像素值的预测值。
730、确定当前块内像素点的像素值的残差信息。
解码器解码当前块内像素点的像素值的残差信息的编码码流，并利用反量化和反变换方法得到当前块内像素点的像素值的残差信息。也就是说编码器解码当前块的残差块的编码码流，然后利用反量化和反变换方法得到当前块的残差块。
应理解，在步骤730中，为了确定当前块内像素点的像素值的残差信息，也可以仅使用反变换或者仅使用反量化方法，具体使用何种方法为编解码端共同约定，或者编码端将所用方法传输到解码端，解码端依据解析结果确定所用方法。
740、确定当前块内像素点的像素值的重建值。
可以根据步骤720中得到的当前块内像素点的像素值与步骤730中得到的当前块内像素点的像素值的残差信息，得到当前块内每个像素点的像素值的重建值。另外，还可以根据步骤720得到当前块的预测块，然后将该预测块与步骤730中得到的残差块相加即可得到当前块的重建块，该重建块包括当前块内每个像素点的像素值的重建值。
应理解，在步骤740中，为了得到当前块内像素点的像素值的重建值，在使用预测信息加上残差信息之后，也可以需要做一些其他操作，例如去块滤波等。具体是否需要其他操作以及其他何种操作为编解码端共同约定，或者编码端将所用方法传输到解码端，解码端依据解析结果确定所用方法。
上文结合图26对全景视频解码器的解码过程进行了详细的说明，实际上解码过程一般是对编码得到的编码码流进行的，下面结合图27以全景视频编码器为例，对编码过程进行详细的说明。
球形图像的布局信息一般在球形图像的采集或者生成之后便可确定了。因此，在编码器对全景视频进行编码之前，编码器便已获知了球形图像的布局信息。球形图像的布局信息一般作为头信息进行传输。头信息是一种先导信息，这样解码器在接收到编码码流之前，先解码头信息，根据头信息获取了球形图像的布局信息后就可以对后续的码流进行解码。
图27是全景视频编码器的编码过程的示意性流程图。具体包括：
810、确定当前块的运动信息。
编码器首先确定当前块的运动信息，然后对运动信息进行编码。具体来说，编码器在重建图像中为当前图像选择一个参考图像，并在参考图像的规定区域内为当前块寻找一个匹配块，并将该匹配块作为当前块的参考块，并在这个过程中所用的参考图像以及指示参考块相对于当前块的位置偏移的运动矢量信息作为运动信息，并对其进行编码。
在参考图像中的规定区域内为当前块寻找参考块时，可以先为当前块选择多个块，最后选用哪一个块作为参考块可以使用率失真优化准则来确定。例如，确定候选参考块作为匹配块时需要编码的运动信息的比特数，以及使用当前候选参考块内像素点的像素值来预测当前块内像素点的像素值时的失真值，利用拉格朗日优化方法确定该候选参考块的代价。然后选择代价最小的候选参考块作为匹配块。而对于多个参考图像的情况，需要对每一个图像均执行上述操作。
应理解，在步骤810中，还可以不直接对运动信息进行编码，而是对预测运动信息的指示信息，或者预测运动信息与运行信息的差值信息进行编码。例如，编码端可以构建一个运动信息集合(例如，该运动信息集合包含与当前块的时间或者空间相邻的已重建块的运动信息，该集合的构建方法由编解码端共同约定)，那么编码端可以选定其中一个运动信息作为预测运动信息，然后得到当前块的运动信息与预测运动信息的差值信息，对预测 运动信息的指示信息以及差值信息进行编码。另外，也可以只对预测运动信息的指示信息进行编码，使得解码端能够根据指示信息从所构建的运动信息集合中确定运动信息，并将其作为当前块的运动信息。具体使用哪一种方法为编解码端共同约定，也可以通过率失真方法选用，并将选用的方法传输到解码端。
在步骤810中，如果是单向参考，则只需编码一套运动信息。如果是双向参考，需编码两套运动信息。具体编码几套运动信息由编解码端共同约定，也可以通过率失真方法确定，并由编码端将相应信息传输到解码端。
在单向参考时，需要确定一个参考块在所在参考图像中的位置，对于当前块内的一个当前像素点，需要确定一个参考像素点的位置，然后根据本申请实施例的运动补偿预测方法来确定参考像素点的像素值的预测值。如果是双向参考，则需要确定当前块的两个参考块在各自参考图像中的位置，也就是说对于当前块内的一个像素点，需要分别确定两个参考像素点在两个参考块中的位置。然后可以根据本申请实施例的运动补偿预测方法来确定当前块内像素点的像素值的两个预测值。然后将这两个预测值进行加权或者其他操作得到当前块内像素点像素值的预测值。具体操作方式为编解码端共同约定，或者编码端将所用方法传输到解码端，解码端依据解析结果确定所用方法。
820、确定当前块内像素点的像素值的预测值
编码器可以根据当前块的运行信息来确定当前块所在的当前图像的参考图像，并确定参考块在参考图像中的位置。事实上，确定当前块内像素点的像素值的预测值就是确定当前块内的每个像素点的像素值的预测值，确定像素点的像素值的预测值具体可以根据本申请实施例中的运动补偿预测方法来确定。
此外，在得到当前块中的像素点的像素值的预测值之后，可以先对该预测值进行其他的操作，将操作得到的预测值作为当前像素点的像素值的最终预测值。例如，可以将得到的预测值经过平滑滤波后得到预测值作为当前像素点的像素值的最终预测值。具体是否使用以及使用哪一种方式为编解码端共同约定，或者编码端将所用方法传输到解码端，解码端依据解析结果确定所用方法。另外，还可以对当前像素点的像素值进行双向预测，也就是说可以对当前像素点进行两次预测，再将得到的两个预测值进行加权得到当前像素点的像素值的预测值。
830、确定当前块内像素点的像素值的残差信息。
编码器使用当前块内像素点的像素值减去其预测值，得到当前块内像素点像素值的残差信息，然后根据变换量化方法对当前块内像素点的像素值的残差信息进行处理，并对处理之后的结果进行编码。
应理解，在对当前块内像素点的像素值的残差信息进行编码时，也可以仅使用变换或者仅使用量化方法，具体使用何种方法为编解码端共同约定，也可以编码端通过通过率失真方法确定，并将信息传输到解码端。
图28是根据本申请实施例的图像编码器900的示意性框图，包括预测模块910、变换量化模块920、熵编码模块930、重建模块940以及滤波模块950，各个模块的具体作用如下的：
预测模块910用于产生预测数据。预测模块910可产生每一不再分割CU的一个或多个预测单元(PU)。CU的每一个PU可与CU的像素块内的不同像素块相关联。预测模块 910可针对CU的每一个PU产生预测性像素块。预测模块910可使用帧内预测或帧间预测来产生PU的预测性像素块。如果预测模块910使用帧内预测来产生PU的预测性像素块，则预测模块910可基于与PU相关联的图片的解码后的像素来产生PU的预测性像素块。如果预测模块910使用帧间预测来产生PU的预测性像素块，则预测模块910可基于不同于与PU相关联的图片的一个或多个图片的解码后的像素来产生PU的预测性像素块。预测模块910可基于CU的PU的预测性像素块来产生CU的残余像素块。CU的残余像素块可指示CU的PU的预测性像素块中的采样值与CU的初始像素块中的对应采样值之间的差。
变换量化模块920用于对经过预测的残差数据进行处理。图像编码器900可对CU的残余像素块执行递归四叉树分割以将CU的残余像素块分割成与CU的变换单元(TU)相关联的一个或多个较小残余像素块。因为与TU相关联的像素块中的像素各自对应一个亮度采样及两个色度采样，所以每一个TU可与一个亮度的残余采样块及两个色度的残余采样块相关联。图像编码器900可将一个或多个变换应用于与TU相关联的残余采样块以产生系数块(即，系数的块)。变换可以是DCT变换或者它的变体。采用DCT的变换矩阵，通过在水平和竖直方向应用一维变换计算二维变换，获得所述系数块。图像编码器900可对系数块中的每一个系数执行量化程序。量化一般指系数经量化以减少用以表示系数的数据量，从而提供进一步压缩的过程。
图像编码器900可产生表示量化后系数块中的系数的语法元素的集合。图像编码器900通过熵编码模块930可将熵编码操作(例如，上下文自适应二进制算术译码(CABAC)操作)应用于上述语法元素中的部分或者全部。为将CABAC编码应用于语法元素，熵编码模块930可将语法元素二进制化以形成包括一个或多个位(称作“二进位”)的二进制序列。熵编码模块930可使用规则(regular)编码来编码二进位中的一部分，且可使用旁通(bypass)编码来编码二进位中的其它部分。
除熵编码系数块的语法元素外，图像编码器900通过重建模块940，可将逆量化及逆变换应用于变换后的系数块，以从变换后的系数块重建残余采样块。图像编码器900可将重建后的残余采样块加到一个或多个预测性采样块的对应采样块，以产生重建后的采样块。通过重建每一色彩分量的采样块，图像编码器900可重建与TU相关联的像素块。以此方式重建CU的每一TU的像素块，直到CU的整个像素块重建完成。
在图像编码器900重建构CU的像素块之后，图像编码器900通过滤波模块950，执行消块滤波操作以减少与CU相关联的像素块的块效应。在图像编码器900执行消块滤波操作之后，图像编码器900可使用采样自适应偏移(SAO)来修改图片的CTB的重建后的像素块。在执行这些操作之后，图像编码器900可将CU的重建后的像素块存储于解码图片缓冲器中以用于产生其它CU的预测性像素块。
在一个或多个实例中，所描述的功能可以硬件、软件、固件或其任何组合来实施。如果以软件实施，则功能可作为一个或多个指令或代码而存储于计算机可读媒体上或经由计算机可读媒体而发送，且通过基于硬件的处理单元执行。计算机可读媒体可包含计算机可读存储媒体(其对应于例如数据存储媒体等有形媒体)或通信媒体，通信媒体包含(例如)根据通信协议促进计算机程序从一处传送到另一处的任何媒体。以此方式，计算机可读媒体大体上可对应于(1)非瞬时的有形计算机可读存储媒体，或(2)例如信号或载波等通信媒体。 数据存储媒体可为可由一个或多个计算机或一个或多个处理器存取以检索指令、代码及/或数据结构以用于实施本申请中所描述的技术的任何可用媒体。计算机程序产品可包含计算机可读媒体。
通过实例而非限制，某些计算机可读存储媒体可包括RAM、ROM、EEPROM、CD-ROM或其它光盘存储器、磁盘存储器或其它磁性存储装置、快闪存储器，或可用以存储呈指令或数据结构的形式的所要程序代码且可由计算机存取的任何其它媒体。而且，任何连接可适当地称为计算机可读媒体。举例来说，如果使用同轴电缆、光缆、双绞线、数字用户线(DSL)或无线技术(例如，红外线、无线电及微波)而从网站、服务器或其它远程源发送指令，则同轴电缆、光缆、双绞线、DSL或无线技术(例如，红外线、无线电及微波)包含于媒体的定义中。然而，应理解，计算机可读存储媒体及数据存储媒体不包含连接、载波、信号或其它瞬时媒体，而是有关非瞬时有形存储媒体。如本文中所使用，磁盘及光盘包含压缩光盘(CD)、激光光盘、光学光盘、数字影音光盘(DVD)、软性磁盘及蓝光光盘，其中磁盘通常以磁性方式复制数据，而光盘通过激光以光学方式复制数据。以上各物的组合还应包含于计算机可读媒体的范围内。
可由例如一个或多个数字信号处理器(DSP)、通用微处理器、专用集成电路(ASIC)、现场可编程逻辑阵列(FPGA)或其它等效集成或离散逻辑电路等一个或多个处理器来执行指令。因此，如本文中所使用的术语“处理器”可指代前述结构或适于实施本文中所描述的技术的任何其它结构中的任一者。另外，在一些方面中，可将本文中所描述的功能性提供于经配置以用于编码及解码的专用硬件及/或软件模块内，或并入于组合式编解码器中。而且，所述技术可完全实施于一个或多个电路或逻辑元件中。
本申请的技术可以广泛地由多种装置或设备来实施，所述装置或设备包含无线手持机、集成电路(IC)或IC集合(例如，芯片组)。在本申请中描述各种组件、模块或单元以强调经配置以执行所揭示技术的装置的功能方面，但未必要求通过不同硬件单元来实现。确切地说，如上文所描述，各种单元可组合于编解码器硬件单元中，或通过交互操作性硬件单元(包含如上文所描述的一个或多个处理器)的集合结合合适软件及/或固件来提供。
应理解，说明书通篇中提到的“一个实施方式”或“一实施方式”意味着与实施方式有关的特定特征、结构或特性包括在本申请的至少一个实施方式中。因此，在整个说明书各处出现的“在一个实施方式中”或“在一实施方式中”未必一定指相同的实施方式。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施方式中。
在本申请的各种实施方式中，应理解，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施方式的实施过程构成任何限定。
另外，本文中术语“系统”和“网络”在本文中常可互换使用。应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
在本申请所提供的实施方式中，应理解，“与A相应的B”表示B与A相关联，根据A可以确定B。但还应理解，根据A确定B并不意味着仅仅根据A确定B，还可以根据A和/或其它信息确定B。
本领域普通技术人员可以意识到，结合本文中所公开的实施方式描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (24)

  1. 一种运动补偿预测方法，其特征在于，包括：
    确定当前像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；
    当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述当前像素点的目标参考像素点对应在所述球面体表面的球面参考位置；
    根据所述球面参考位置确定所述目标参考像素点对应在所述参考图像对应的多面体的表面的多面体参考位置，其中，所述球面体内切于所述多面体，所述多面体参考位置与所述球心的连线经过所述球面参考位置；
    根据所述多面体参考位置，确定所述目标参考像素点在所述参考图像中的位置；
    根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
  2. 如权利要求1所述的方法，其特征在于，根据所述初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述当前像素点的目标参考像素点对应在所述球面体表面的球面参考位置，包括：
    根据所述初始参考像素点的位置和所述初始参考像素点的位置相对于所述第二子图像的偏移值，确定所述初始参考像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；
    根据所述第一位置以及所述球面体的球心位置，确定所述目标参考像素点对应在所述球面体表面的球面参考位置，其中，所述第一位置与所述球心位置的连线经过所述球面参考位置。
  3. 如权利要求1或2所述的方法，其特征在于，所述方法还包括：
    根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
  4. 如权利要求1-3中任一项所述的方法，其特征在于，根据所述多面体参考位置，确定所述目标参考像素点在所述参考图像中的位置，包括：
    根据所述多面体参考位置以及所述多面体的布局信息，确定所述目标参考像素点在所述参考图像中的位置。
  5. 如权利要求1-4中任一项所述的方法，其特征在于，所述布局信息包含所述多面体的面的数量信息，所述参考图像的子图像的排布方式信息，所述参考图像的子图像的排布顺序信息，所述参考图像的子图像的旋转信息中的至少一种。
  6. 如权利要求1-5中任一项所述的方法，其特征在于，所述根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值，包括：
    将所述目标参考像素点的像素值确定为所述当前像素点的像素值的预测值。
  7. 如权利要求1-5中任一项所述的方法，其特征在于，所述根据所述目标参考像素 点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值，包括：
    对所述目标参考像素点的像素值以及所述目标参考像素点的临近像素点的像素值进行加权处理；
    将所述加权处理得到的所述目标参考像素点所在位置的像素值确定为所述当前像素点的像素值的预测值。
  8. 如权利要求1-5中任一项所述的方法，其特征在于，所述根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值，包括：
    根据目标参考像素点的临近像素点的像素值在所述目标参考像素点位置进行插值运算；
    将所述插值运算得到的像素值确定为所述当前像素点的像素值的预测值。
  9. 一种运动补偿预测方法，其特征在于，包括：
    确定当前参考像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；
    当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，确定所述当前像素点在所述第二子图像的扩展区域中的目标参考像素点的位置，其中，所述第二子图像的扩展区域位于所述第二子图像之外，所述扩展区域包括多个像素点，所述扩展区域中任意一个第一像素点的像素值是根据所述参考图像中的第二像素点的像素值确定的，所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置与所述参考图像对应的球面体的球心的连线经过所述第二像素点对应在所述参考图像对应的球面体的球面参考位置，所述球面参考位置是根据所述初始参考像素点的位置以及所述球面体的球心位置确定的，所述球面体内切于所述多面体；
    根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
  10. 如权利要求9所述的方法，其特征在于，所述方法还包括：
    根据所述第一像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置；
    根据所述球面参考位置确定所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置；
    根据所述多面体参考位置，确定所述第二像素点在所述参考图像中的位置。
  11. 如权利要求10所述的方法，其特征在于，所述根据所述第一像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置，包括：
    根据所述第一像素点的位置以及所述第一像素点的位置相对于所述第二子图像的偏移值，确定所述第一像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；
    根据所述第一位置以及所述球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置，所述第一位置与所述球心位置的连线经过所述球面参考位置。
  12. 如权利要求9-11中任一项所述的方法，其特征在于，所述方法还包括：
    根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
  13. 一种运动补偿预测装置，其特征在于，包括：
    第一确定单元，用于确定当前像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；
    第二确定单元，用于当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，根据所述初始参考像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述当前像素点的目标参考像素点对应在所述球面体表面的球面参考位置；
    第三确定单元，用于根据所述球面参考位置确定所述目标参考像素点对应在所述参考图像对应的多面体的表面的多面体参考位置，其中，所述球面体内切于所述多面体，所述多面体参考位置与所述球心的连线经过所述球面参考位置；
    第四确定单元，用于根据所述多面体参考位置，确定所述目标参考像素点在所述参考图像中的位置；
    预测单元，用于根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
  14. 如权利要求13所述的运动补偿预测装置，其特征在于，所述第二确定单元具体用于：
    根据所述初始参考像素点的位置和所述初始参考像素点的位置相对于所述第二子图像的偏移值，确定所述初始参考像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；
    根据所述第一位置以及所述球面体的球心位置，确定所述目标参考像素点对应在所述球面体表面的球面参考位置，其中，所述第一位置与所述球心位置的连线经过所述球面参考位置。
  15. 如权利要求13或14所述的运动补偿预测装置，其特征在于，所述运动补偿预测装置还包括：
    判断单元，用于根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
  16. 如权利要求13-15中任一项所述的运动补偿预测装置，其特征在于，所述第四确定单元具体用于：
    根据所述多面体参考位置以及所述多面体的布局信息，确定所述目标参考像素点在所述参考图像中的位置。
  17. 如权利要求13-16中任一项所述的运动补偿预测装置，其特征在于，所述布局信息包含所述多面体的面的数量信息，所述参考图像的子图像的排布方式信息，所述参考图像的子图像的排布顺序信息，所述参考图像的子图像的旋转信息中的至少一种。
  18. 如权利要求13-17中任一项所述的运动补偿预测装置，其特征在于，所述预测单元具体用于：
    将所述目标参考像素点的像素值确定为所述当前像素点的像素值的预测值。
  19. 如权利要求13-17中任一项所述的运动补偿预测装置，其特征在于，所述预测单元具体用于：
    对所述目标参考像素点的像素值以及所述目标参考像素点的临近像素点的像素值进行加权处理；
    将所述加权处理得到的所述目标参考像素点所在位置的像素值确定为所述当前像素点的像素值的预测值。
  20. 如权利要求13-17中任一项所述的运动补偿预测装置，其特征在于，所述预测单元具体用于：
    根据目标参考像素点的临近像素点的像素值在所述目标参考像素点位置进行插值运算；
    将所述插值运算得到的像素值确定为所述当前像素点的像素值的预测值。
  21. 一种运动补偿预测装置，其特征在于，包括：
    第一确定单元，用于确定当前参考像素点的初始参考像素点在参考图像中的位置，所述当前像素点位于所述当前图像的第一子图像内；
    第二确定单元，用于当所述初始参考像素点位于所述参考图像中与所述第一子图像对应位置的第二子图像之外时，确定所述当前像素点在所述子图像的扩展区域中的目标参考像素点的位置，其中，所述第二子图像的扩展区域位于所述第二子图像之外，所述扩展区域包括多个像素点，所述扩展区域中任意一个第一像素点的像素值是根据所述参考图像中的第二像素点的像素值确定的，所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置与所述球心的连线经过所述第二像素点对应在所述参考图像对应的球面体的球面参考位置，所述球面参考位置是根据所述初始参考像素点的位置以及所述球面体的球心位置确定的，所述球面体内切于所述多面体；
    预测单元，用于根据所述目标参考像素点的像素值和/或所述目标参考像素点的临近像素点的像素值，确定所述当前像素点的像素值的预测值。
  22. 如权利要求21所述的运动补偿预测装置，其特征在于，所述运动补偿预测装置还包括：
    第三确定单元，用于根据所述第一像素点的位置以及所述参考图像对应的球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置；
    第四确定单元，用于根据所述球面参考位置确定所述第二像素点对应在所述参考图像对应的多面体的表面的多面体参考位置；
    第五确定单元，根据所述多面体参考位置，确定所述第二像素点在所述参考图像中的位置。
  23. 如权利要求22所述的运动补偿预测装置，其特征在于，所述第三确定单元具体用于：
    根据所述第一像素点的位置以及所述第一像素点的位置相对于所述第二子图像的偏移值，确定所述第一像素点在所述多面体中的第一面所在平面的第一位置，其中，所述第一面是所述多面体中与所述第二子图像对应的面；
    根据所述第一位置以及所述球面体的球心位置，确定所述第二像素点对应在所述球面体表面的球面参考位置，所述第一位置与所述球心位置的连线经过所述球面参考位置。
  24. 如权利要求21-23中任一项所述的运动补偿预测装置，其特征在于，所述运动补偿预测装置还包括：
    判断单元，用于根据所述参考图像的布局信息和所述初始参考像素点的位置判断所述初始参考像素点是否位于所述参考图像中的所述第二子图像之外。
PCT/CN2017/104030 2016-09-30 2017-09-28 运动补偿预测方法和设备 WO2018059500A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP17854950.7A EP3515076B1 (en) 2016-09-30 2017-09-28 Motion compensation prediction method and device
US16/371,486 US10779000B2 (en) 2016-09-30 2019-04-01 Motion-compensated prediction method and apparatus

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201610876864 2016-09-30
CN201610876864.9 2016-09-30
CN201610896430.5A CN107888928B (zh) 2016-09-30 2016-10-13 运动补偿预测方法和设备
CN201610896430.5 2016-10-13

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/371,486 Continuation US10779000B2 (en) 2016-09-30 2019-04-01 Motion-compensated prediction method and apparatus

Publications (1)

Publication Number Publication Date
WO2018059500A1 true WO2018059500A1 (zh) 2018-04-05

Family

ID=61763281

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/104030 WO2018059500A1 (zh) 2016-09-30 2017-09-28 运动补偿预测方法和设备

Country Status (1)

Country Link
WO (1) WO2018059500A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101521823A (zh) * 2009-03-27 2009-09-02 北京航空航天大学 空间相关的全景数据压缩方法
US20150341552A1 (en) * 2014-05-21 2015-11-26 Here Global B.V. Developing a Panoramic Image
WO2016064862A1 (en) * 2014-10-20 2016-04-28 Google Inc. Continuous prediction domain
CN105681805A (zh) * 2016-01-19 2016-06-15 北京大学深圳研究生院 视频编码、解码方法及其帧间预测方法和装置
CN105959702A (zh) * 2016-05-30 2016-09-21 北京奇艺世纪科技有限公司 一种球面视频编码方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101521823A (zh) * 2009-03-27 2009-09-02 北京航空航天大学 空间相关的全景数据压缩方法
US20150341552A1 (en) * 2014-05-21 2015-11-26 Here Global B.V. Developing a Panoramic Image
WO2016064862A1 (en) * 2014-10-20 2016-04-28 Google Inc. Continuous prediction domain
CN105681805A (zh) * 2016-01-19 2016-06-15 北京大学深圳研究生院 视频编码、解码方法及其帧间预测方法和装置
CN105959702A (zh) * 2016-05-30 2016-09-21 北京奇艺世纪科技有限公司 一种球面视频编码方法和装置

Similar Documents

Publication Publication Date Title
CN107888928B (zh) 运动补偿预测方法和设备
TWI650996B (zh) 視頻編碼或解碼方法及裝置
JP6690061B2 (ja) 動き補償予測方法およびデバイス
WO2018095087A1 (zh) 一种去块滤波方法及终端
US10986371B2 (en) Sample adaptive offset filtering method for reconstructed projection-based frame that employs projection layout of 360-degree virtual reality projection
CN111279673B (zh) 具有电子卷帘快门校正的图像拼接的系统和方法
KR101131756B1 (ko) 도메인 변환을 이용한 메시 기반 비디오 압축
WO2017215587A1 (zh) 一种视频图像的编解码方法和装置
KR20190052013A (ko) 적응적 인트라 최고 확률 모드들에 의한 전방향성 비디오 코딩을 위한 방법 및 장치
KR102254986B1 (ko) 구면 투영부들에 의한 왜곡을 보상하기 위한 등장방형 객체 데이터의 프로세싱
WO2018107800A1 (zh) 一种运动矢量解码方法及解码器
WO2018068747A1 (zh) 运动补偿预测方法和设备
US20210150665A1 (en) Image processing method and device
US11330268B2 (en) Apparatus and methods for encoding and decoding a video signal
TWI756526B (zh) 用於重構的基於投影的幀的樣本適應性偏移濾波方法
WO2018041005A1 (zh) 运动补偿预测方法和设备
JP2023514215A (ja) 画像およびビデオ圧縮のイントラ予測
WO2018059500A1 (zh) 运动补偿预测方法和设备
US11915373B1 (en) Attribute value compression for a three-dimensional mesh using geometry information to guide prediction
CN118077203A (zh) 具有明确信示的扩展旋转的经扭曲的运动补偿

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17854950

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017854950

Country of ref document: EP

Effective date: 20190415