WO2018066986A1 - Procédé et appareil de codage et de décodage de données d'images - Google Patents

Procédé et appareil de codage et de décodage de données d'images Download PDF

Info

Publication number
WO2018066986A1
WO2018066986A1 PCT/KR2017/011147 KR2017011147W WO2018066986A1 WO 2018066986 A1 WO2018066986 A1 WO 2018066986A1 KR 2017011147 W KR2017011147 W KR 2017011147W WO 2018066986 A1 WO2018066986 A1 WO 2018066986A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
information
resizing
encoding
decoding
Prior art date
Application number
PCT/KR2017/011147
Other languages
English (en)
Korean (ko)
Inventor
김기백
Original Assignee
김기백
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 김기백 filed Critical 김기백
Publication of WO2018066986A1 publication Critical patent/WO2018066986A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/11—Selection of coding mode or of prediction mode among a plurality of spatial predictive coding modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/129—Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/13—Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/625—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using discrete cosine transform [DCT]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness

Definitions

  • the present invention relates to video data encoding and decoding technology, and more particularly, to a method and apparatus for processing encoding and decoding of 360 degree video for an immersive media service.
  • An object of the present invention for solving the above problems is to provide a method for improving the image setting process in the initial stage of encoding and decoding. More specifically, the present invention provides an encoding and decoding method and apparatus for improving an image setting process considering a 360 degree image characteristic.
  • One aspect of the present invention for achieving the above object provides a decoding method of a 360-degree image.
  • the method of decoding a 360-degree image receiving a bitstream encoded with a 360-degree image, generating a predictive image by referring to syntax information obtained from the received bitstream, the bit of the generated prediction image
  • the method may include obtaining a decoded image by combining the stream with a residual image obtained by inverse quantization and inverse transform, and reconstructing the decoded image into a 360 degree image according to a projection format.
  • the syntax information may include projection format information about the 360 degree image.
  • the projection format information may include an ERP (Equi-Rectangular Projection) format in which the 360-degree image is projected in a two-dimensional plane, a CMP (CubeMap Projection) format in which the 360-degree image is projected into a cube, and the 360-degree octahedron It may be information indicating at least one of an OHta (OctaHedron Projection) format projected on the screen and an IcoSahedral Projection (ISP) format on which the 360 degree image is projected on a polyhedron.
  • ERP Equi-Rectangular Projection
  • CMP CubeMap Projection
  • ISP IcoSahedral Projection
  • the reconstructing may include obtaining placement information based on regional packing by referring to the syntax information, and rearranging each block of the decoded image according to the placement information. have.
  • the generating of the prediction image may include performing image expansion on a reference picture obtained by reconstructing the bitstream, and generating a prediction image by referring to the reference image on which the image expansion is performed. It may include.
  • the performing of the image expansion may include performing an image expansion based on a division unit of the reference image.
  • the extended region may be individually generated for each division unit by using the boundary pixels of the division unit.
  • the extended area may be generated using a boundary pixel of a division unit that is spatially adjacent to the division unit to be extended or a division pixel of a division unit having a continuity of the image and the division unit to be extended.
  • the extended image may be generated with respect to the combined region by using boundary pixels of a region in which two or more spatially adjacent division units are combined among the division units. Can be.
  • the extended region may be generated between the adjacent division units by using all the adjacent pixel information of the spatially adjacent division units among the division units.
  • the extended area may be generated using an average value of adjacent pixels of each of the spatially adjacent division units.
  • the generating of the prediction image may include configuring a reference pixel using a boundary pixel adjacent to the current block and having a different color space from a current block to be decoded according to the prediction mode obtained from the syntax information.
  • the method may include generating a prediction block with respect to the current block by performing intra prediction on the screen.
  • the color space may include a color difference component (Cb, Cr) and a luminance component (Y).
  • the reference pixel may be configured using at least one of TL to BR of luminance components.
  • the reference pixel may be configured by a weighted average value of at least two of TL to Br of the luminance component.
  • the prediction mode may include at least one of horizontal, vertical, DC, planar, diagonal, luma mode, and correlation mode.
  • compression performance can be improved.
  • the 360-degree image can improve the compression performance.
  • FIG. 1 is a block diagram of an image encoding apparatus according to an embodiment of the present invention.
  • FIG. 2 is a block diagram of an image decoding apparatus according to an embodiment of the present invention.
  • 3 is an exemplary diagram of image information divided by layers in order to compress an image.
  • FIG. 4 is a conceptual diagram illustrating various examples of image segmentation according to an embodiment of the present invention.
  • FIG. 5 is another exemplary diagram of an image segmentation method according to an exemplary embodiment.
  • FIG. 6 is an exemplary diagram for a method of adjusting a size of a general image.
  • FIG. 7 is an exemplary view of image resizing according to an embodiment of the present invention.
  • FIG. 8 is an exemplary view illustrating a method of configuring an extended area in an image resizing method according to an embodiment of the present invention.
  • FIG. 9 is an exemplary view illustrating a method of configuring an area to be deleted and an area to be reduced and generated in an image resizing method according to an embodiment of the present invention.
  • FIG. 10 is an exemplary view of image reconstruction according to an embodiment of the present invention.
  • FIG. 11 is an exemplary diagram illustrating an image before and after an image setting process according to an embodiment of the present invention.
  • FIG. 12 illustrates an example of size adjustment for each segmentation unit in an image according to an exemplary embodiment.
  • FIG. 13 is an exemplary diagram of a size adjustment or setting set of a division unit in an image.
  • FIG. 14 is an exemplary diagram illustrating an image scaling process and a scaling process of segmentation units in an image.
  • 15 is an exemplary diagram illustrating a three-dimensional space and a two-dimensional planar space representing a three-dimensional image.
  • 16A to 16D are conceptual views illustrating a projection format according to an embodiment of the present invention.
  • FIG. 17 is a conceptual view of implementing a projection format included in a rectangular image according to an embodiment of the present invention.
  • FIG. 18 is a conceptual diagram of a method of converting a projection format into a rectangular shape according to an embodiment of the present invention, and repositioning the surface to exclude a meaningless area.
  • FIG. 19 is a conceptual view illustrating a wrapping process for each region in a rectangular image of a CMP projection format according to an embodiment of the present invention.
  • 20 is a conceptual diagram for 360-degree image segmentation according to an embodiment of the present invention.
  • 21 is an exemplary diagram of 360 degree image segmentation and image reconstruction according to an embodiment of the present invention.
  • FIG. 22 illustrates an example of dividing an image projected by CMP or a packaged image into tiles.
  • FIG. 23 is a conceptual diagram illustrating an example of adjusting the size of a 360 degree image according to an embodiment of the present invention.
  • 24 is a conceptual diagram illustrating continuity between surfaces in a projection format (eg, CMP, OHP, ISP) according to an embodiment of the present invention.
  • a projection format eg, CMP, OHP, ISP
  • FIG. 25 is a conceptual diagram illustrating the continuity of the surface of FIG. 21C, which is an image obtained through an image reconstruction process or a region-based wrapping process in a CMP projection format.
  • FIG. 26 is an exemplary diagram for explaining an image size adjustment in a CMP projection format according to an embodiment of the present invention.
  • FIG. 27 is an exemplary diagram for explaining a size adjustment for a packaged image converted to a CMP projection format according to an embodiment of the present invention.
  • FIG. 28 is an exemplary diagram for describing a data processing method in resizing a 360 degree image according to an embodiment of the present invention.
  • 29 is an exemplary diagram illustrating a tree-based block form.
  • FIG. 30 shows an example of a type based block shape.
  • FIG. 31 is an exemplary view illustrating various block types that may be acquired by a block divider of the present invention.
  • FIG. 31 is an exemplary view illustrating various block types that may be acquired by a block divider of the present invention.
  • 32 is an exemplary diagram for describing tree based partitioning according to an embodiment of the present invention.
  • 33 is an exemplary diagram for explaining tree-based partitioning according to an embodiment of the present invention.
  • 34 is an exemplary diagram for describing a configuration of a reference pixel used for intra prediction.
  • 35 is an exemplary diagram illustrating an intra prediction mode of HEVC.
  • 36 is a conceptual diagram of using a block in which encoding of another color space is completed, as a prediction block, according to an embodiment of the present invention.
  • 37 is a conceptual diagram of a configuration of a color difference component and a luminance component corresponding thereto in some color formats.
  • 38A to 38C are diagrams illustrating expansion according to an integer multiple of the size of a block according to an embodiment of the present invention.
  • 39A to 39C are exemplary views illustrating generation of partition information according to block division of FIGS. 38A to 38C.
  • first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.
  • the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.
  • the video encoding device and the decoding device include a personal computer (PC), a notebook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), and a PlayStation Portable (PSP).
  • PC personal computer
  • PDA personal digital assistant
  • PMP portable multimedia player
  • PSP PlayStation Portable
  • Wireless Communication Terminal Smart Phone, TV, Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Head It may be a user terminal such as a head mounted display (HMD) or smart glasses, or a server terminal such as an application server and a service server, and a communication device such as a communication modem for communicating with various devices or wired / wireless communication networks.
  • HMD head mounted display
  • server terminal such as an application server and a service server
  • a communication device such as a communication modem for communicating with various devices or wired / wireless communication networks.
  • Program to encode or decode an image, or to predict an intra-screen or inter-screen for encoding or decoding It may include various devices including a memory for storing RAM and data, and a processor for executing and controlling a program by executing a program.
  • the image encoded in the bitstream by the video encoding apparatus is real-time or non-real-time through the wired or wireless communication network (Network), such as the Internet, local area wireless communication network, wireless LAN network, WiBro network, mobile communication network, etc. It may be transmitted to an image decoding apparatus through various communication interfaces such as a universal serial bus (USB), and the like, may be decoded by the image decoding apparatus, reconstructed, and played back.
  • Network such as the Internet, local area wireless communication network, wireless LAN network, WiBro network, mobile communication network, etc.
  • USB universal serial bus
  • an image encoded in a bitstream by the image encoding apparatus may be transferred from the encoding apparatus to the decoding apparatus through a computer-readable recording medium.
  • the above-described image encoding apparatus and the image decoding apparatus may be separate apparatuses, but may be made of one image encoding / decoding apparatus according to implementation.
  • some components of the image encoding apparatus may be implemented to include at least the same structure or to perform at least the same function as substantially the same technical elements as some components of the image decoding apparatus.
  • the image decoding apparatus corresponds to a computing device applying the image encoding method performed by the image encoding apparatus to the decoding, the following description will focus on the image encoding apparatus.
  • the computing device may include a memory for storing a program or software module for implementing an image encoding method and / or an image decoding method, and a processor connected to the memory and executing a program.
  • the image encoding apparatus may be referred to as an encoder, and the image decoding apparatus may be referred to as a decoder.
  • an image may be configured as a series of still images, and the still images may be classified into a group of pictures (GOP), and each still image may be referred to as a picture.
  • the picture may represent one of a frame and a field in a progressive signal and an interlace signal, and when the encoding / decoding is performed in units of frames, the image may be a 'frame', a field. When executed in units, it can be represented as a 'field'.
  • the present invention assumes and describes a progressive signal, it may be applicable to an interlaced signal.
  • a unit such as a GOP or a sequence may exist, and each picture may be divided into predetermined regions such as slices, tiles, blocks, and the like.
  • one GOP may include units such as an I picture, a P picture, and a B picture.
  • An I picture may refer to a picture that is encoded / decoded by itself without using a reference picture, and a P picture and a B picture use a reference picture to perform a process such as motion estimation and motion compensation. This may mean a picture to be encoded / decoded by performing.
  • I picture and P picture can be used as reference picture
  • B picture I picture and P picture can be used as reference picture, but the above definition is also changed by the setting of encoding / decoding. Can be.
  • the picture referred to for encoding / decoding is referred to as a reference picture, and the referred block or pixel is referred to as a reference block and a reference pixel.
  • the reference data may be not only pixel values of a spatial domain but also coefficient values of a frequency domain and various encoding / decoding information generated and determined during an encoding / decoding process.
  • intra prediction prediction information or motion related information in the prediction unit, transform related information in the transform unit / inverse transform unit, quantization information in the quantization unit / dequantizer, and encoding / decoding related information in the encoder / decoder ( Context information), the filter related information, etc. may correspond to the in-loop filter unit.
  • the minimum unit constituting the image may be a pixel, and the number of bits used to represent one pixel is called a bit depth.
  • the bit depth may be 8 bits and may support more bit depths according to encoding settings.
  • the bit depth may support at least one bit depth according to a color space.
  • at least one color space may be configured according to a color format of an image. It may consist of one or more pictures with a certain size or one or more pictures with other sizes, depending on the color format. For example, in the case of YCbCr 4: 2: 0, it may be composed of one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example).
  • the composition ratio may have a length of 1: 2.
  • 4: 4: 4 may have the same ratio of width and length.
  • the setting independent of each color space may mean having a setting of the corresponding color space irrespective of the composition ratio of each component or independently. In the present invention, depending on the encoder / decoder, some components may have independent settings or dependent settings.
  • Configuration information or syntax elements required in the video encoding process may be determined at the unit level of a video, sequence, picture, slice, tile, block, and the like. These include VPS (Video Parameter Set), SPS (Sequence Parameter Set), Setting information transmitted from the encoder by parsing in the same level unit in the bit stream in units such as Picture Parameter Set (PPS), Slice Header, Tile Header, Block Header, etc. It can be used to reconstruct the image decoding process. In addition, related information may be transmitted in a bitstream, parsed, and used in the form of Supplementary Enhancement Information (SEI) or Metadata. Each parameter set has a unique ID value, and a lower parameter set may have an ID value of a higher parameter set to be referred to.
  • SEI Supplementary Enhancement Information
  • the lower parameter set may refer to information of an upper parameter set having a matching ID value among one or more higher parameter sets.
  • a unit corresponding to a case in which any one of the examples of the various units mentioned above includes one or more other units may be referred to as an upper unit, and the included unit may be referred to as a lower unit.
  • each of the units may include the contents of the independent settings or the contents of the settings depending on the previous, subsequent or higher units.
  • the dependent setting may be understood to indicate setting information of a corresponding unit as flag information (for example, 1 if a bit is 1 and 0 if not) according to a setting of a higher unit before and after.
  • the setting information in the present invention will be described mainly on the example of the independent setting, but the example of adding or replacing the content of the relationship depending on the setting information of the previous or subsequent units or higher units of the current unit May be included.
  • 1 is a block diagram of an image encoding apparatus according to an embodiment of the present invention.
  • 2 is a block diagram of an image decoding apparatus according to an embodiment of the present invention.
  • the image encoding apparatus may include a predictor, a subtractor, a transformer, a quantizer, an inverse quantizer, an inverse transformer, an adder, an in-loop filter, a memory, and / or an encoder.
  • the image decoding apparatus may include a decoder, a predictor, an inverse quantizer, an inverse transform unit, an adder, an in-loop filter unit, and / or a memory, and some of the above components are necessarily included. Not all of them may be included, optionally, some or all of them may be included, and some additional components not shown may be included.
  • the image encoding apparatus and the image decoding apparatus may be separate apparatuses, but may be made of one image encoding / decoding apparatus according to implementation.
  • some components of the image encoding apparatus may be implemented to include at least the same structure or to perform at least the same function as substantially the same technical elements as some components of the image decoding apparatus. Therefore, detailed descriptions of the corresponding technical elements and their operating principles will be omitted.
  • the image decoding apparatus corresponds to a computing device applying the image encoding method performed by the image encoding apparatus to the decoding, the following description will focus on the image encoding apparatus.
  • the image encoding apparatus may be referred to as an encoder, and the image decoding apparatus may be referred to as a decoder.
  • the prediction unit may be implemented using a prediction module, which is a software module, and may generate a prediction block by using an intra prediction method or an inter prediction method for a block to be encoded.
  • the predictor generates a prediction block by predicting a current block to be currently encoded in the image. That is, the prediction unit includes a prediction block having a predicted pixel value of each pixel generated by predicting a pixel value of each pixel of the current block to be encoded in the image according to intra prediction or inter prediction.
  • the prediction unit may transmit information necessary for generating the prediction block to the encoder to encode information on the prediction mode, and store the information according to the bitstream and transmit the information to the decoder, and the decoder of the decoder After parsing the information, the information about the prediction mode can be restored and used for intra prediction or inter prediction.
  • the subtraction unit subtracts the prediction block from the current block to generate a residual block. That is, the subtractor calculates a difference between the pixel value of each pixel of the current block to be encoded and the predicted pixel value of each pixel of the prediction block generated through the predictor to generate a residual block that is a residual signal in the form of a block. .
  • the transform unit may convert a signal belonging to a spatial domain into a signal belonging to a frequency domain, and a signal obtained through a transformation process is called a transformed coefficient.
  • a transform block having a transform coefficient may be obtained by transforming the residual block having the residual signal received from the subtractor, but the input signal is determined according to an encoding setting, which is not limited to the residual signal.
  • the transform unit can transform the residual block using transformation techniques such as a Hadamard transform, a Discrete Sine Transform (DST Based-Transform), or a Discrete Cosine Transform (DCT Based-Transform).
  • transformation techniques such as a Hadamard transform, a Discrete Sine Transform (DST Based-Transform), or a Discrete Cosine Transform (DCT Based-Transform).
  • DST Based-Transform a Discrete Sine Transform
  • DCT Based-Transform Discrete Cosine Transform
  • At least one of the transform schemes may be supported, and at least one detailed transform scheme may be supported in each transform scheme.
  • the at least one detailed transformation technique may be a transformation technique in which a part of the basis vector is configured differently in each transformation technique.
  • DST-based and DCT-based transformations can be supported as a conversion technique.
  • DST, DST-I, DST-II, DST-III, DST-V, DST-VI, DST-VII, DST Detailed conversion techniques such as -VIII can be supported, and detailed conversion techniques such as DCT-I, DCT-II, DCT-III, DCT-V, DCT-VI, DCT-VII, and DCT-VIII are supported for DCT. Can be.
  • One of the transforms may be set as a basic transform technique, and an additional transform technique (for example, multiple transform schemes) Conversion techniques).
  • additional transform techniques for example, multiple transform schemes
  • Whether to support additional transform schemes may be determined in units of sequence, picture, slice, tile, etc. so that related information may be generated in the above units, and when support for additional transform schemes is supported, transform scheme selection information is determined in units such as blocks. Related information may be generated.
  • the conversion can be performed in the horizontal / vertical direction.
  • a pixel value of a spatial domain may be transformed into a frequency domain by performing a one-dimensional transformation in a horizontal direction using a basis vector in the transformation and a total two-dimensional transformation by performing a one-dimensional transformation in a vertical direction.
  • the conversion in the horizontal / vertical direction can be performed adaptively.
  • whether to perform adaptive operation may be determined according to at least one encoding setting.
  • DCT-I may be applied in the horizontal direction and DST-I in the vertical direction when the prediction mode is the horizontal mode
  • DCT-VI can be applied in the vertical direction, DCT-II in the horizontal direction, DCT-V in the vertical direction, and in the horizontal direction in the case of Diagonal down left.
  • DST-I, DST-VI may be applied in the vertical direction.
  • the size and shape of each transform block may be determined according to the encoding cost for each candidate of the size and shape of the transform block, and information such as image data of the determined transform block and the size and shape of the determined transform block may be encoded.
  • a square transform among the transform forms may be set as a basic transform form, and an additional transform form (for example, a rectangular form) may be supported.
  • an additional transform form for example, a rectangular form
  • Whether to support the additional transform type may be determined in units of a sequence, a picture, a slice, a tile, and the like, and related information may be generated in the unit, and the transform type selection information may be determined in a unit such as a block to generate related information.
  • support in the form of transform blocks may be determined according to encoding information.
  • the encoding information may correspond to a slice type, an encoding mode, a size and shape of a block, a block division scheme, and the like. That is, one transform type may be supported according to at least one encoded information, and a plurality of transform types may be supported according to at least one encoded information.
  • the former may be an implicit situation, and the latter may be an explicit situation.
  • adaptive selection information indicating an optimal candidate group among a plurality of candidate groups may be generated and recorded in the bitstream.
  • the encoded information when the encoded information is explicitly generated, it may be understood that the information is stored in the bitstream in various units, and the decoder parses the related information in various units and restores the decoded information. have.
  • the encoder and the decoder are processed by the same process and rule.
  • the rectangular support may be determined based on the slice type.
  • the supported transformation may be a square transformation
  • the transformation may be a square or rectangular transformation.
  • a rectangular transform support may be determined according to an encoding mode.
  • the supported conversion form may be a square form
  • the supported transformation form may be a square form or a rectangular form.
  • the rectangular support may be determined based on the size and shape of the block.
  • the transform type supported in blocks of a predetermined size or more may be a square type transform, and the transform type supported in a block smaller than a predetermined size may be a square or rectangular type of transform.
  • the rectangular support may be determined based on the block division scheme.
  • the supported transform type may be a square type transform, or a block obtained through a binary tree splitting method.
  • the type of transform supported may be a square or rectangular type of transform.
  • the above example is an example of transform type support according to one piece of encoding information, and a plurality of pieces of information may be combined to participate in additional transform type support setting.
  • the above example is not limited to the above, but is an example of additional transform type support according to various encoding settings, and various modification examples may be possible.
  • the conversion process may be omitted depending on the encoding setting or the characteristics of the image.
  • a conversion process (including an inverse process) may be omitted according to an encoding setting (assuming lossless compression environment in this example).
  • the conversion process may be omitted when the compression performance through the conversion is not performed according to the characteristics of the image.
  • the omitted transform may be an entire unit, or one of horizontal and vertical units may be omitted, and it may be determined whether such omission is supported according to the size and shape of the block.
  • the conversion skip flag when the conversion skip flag is 1, the conversion is not performed in the horizontal and vertical directions, and when 0, the conversion may be performed in the horizontal and vertical directions.
  • the first conversion skip flag when the first conversion skip flag is 1, the conversion is not performed in the horizontal direction, and when 0 is converted, the conversion is performed in the horizontal direction, and the second conversion skip flag is performed. If 1, no transformation is performed in the vertical direction. If 0, the transformation is performed in the vertical direction.
  • the conversion omission may be supported.
  • the conversion omission may not be supported.
  • M (m) and N (n) may be the same or different.
  • the transformation related setting may be determined in units of a sequence, a picture, a slice, and the like.
  • the transformation scheme setting may be determined according to at least one encoding information.
  • the encoding information may correspond to a slice type, an encoding mode, a size and shape of a block, a prediction mode, and the like.
  • support of a transformation scheme may be determined according to an encoding mode.
  • the conversion technique supported may be DCT-I, DCT-III, DCT-VI, DST-II, DST-III.
  • the conversion technique supported is DCT-II, DCT-III, DST. -III.
  • support of the conversion scheme may be determined according to the slice type.
  • the conversion schemes supported may be DCT-I, DCT-II, or DCT-III
  • the transformation schemes may be DCT-V, DST-V, or DST-VI.
  • the conversion scheme supported may be DCT-I, DCT-II, or DST-III.
  • support of a transformation scheme may be determined according to a prediction mode.
  • the conversion scheme supported in prediction mode A may be DCT-I, DCT-II
  • the conversion scheme supported in prediction mode B may be DCT-I, DST-I
  • the conversion scheme supported in prediction mode C is DCT. May be -I.
  • the prediction modes A and B may be a directional mode and the prediction mode C may be a non-directional mode.
  • support of the conversion scheme may be determined according to the size and shape of the block.
  • Conversion schemes supported on blocks of a certain size may be DCT-II
  • transformation schemes supported on blocks of a certain size may be DCT-II, DST-V, and supported on blocks above and below a certain size.
  • the conversion technique may be DCT-I, DCT-II, or DST-I.
  • the conversion scheme supported in the square form may be DCT-I, DCT-II
  • the transformation scheme supported in the rectangular form may be DCT-I, DST-I.
  • the above example is an example of supporting a transformation scheme according to one piece of encoding information, and a plurality of pieces of information may be combined to participate in additional transformation scheme support setting. It is not limited only to the above examples, but modifications to other examples may be possible.
  • the transformer may transmit information necessary to generate a transform block to the encoder to encode the information, and store the information in the bitstream and transmit the information to the decoder, and the decoder of the decoder parses the information and inversely transforms the information. Can be used for the process.
  • the quantization unit may quantize the input signal, and the signal obtained through the quantization process is called a quantized coefficient.
  • a quantization block having a quantization coefficient may be obtained by quantizing a residual block having a residual transform coefficient received from a transform unit.
  • the received signal is determined according to an encoding setting, which is not limited to the residual transform coefficient.
  • the quantization unit may quantize the transformed residual block using a quantization technique such as dead zone uniform threshold quantization, quantization weighted matrix, and the like, but is not limited thereto. Quantization techniques can be used. Whether to support the additional quantization scheme is determined in units of sequence, picture, slice, tile, etc. so that the related information can be generated in the unit, and when the support for additional quantization scheme is supported, the quantization scheme selection information is determined in units such as blocks. Related information may be generated.
  • a quantization technique such as dead zone uniform threshold quantization, quantization weighted matrix, and the like, but is not limited thereto. Quantization techniques can be used. Whether to support the additional quantization scheme is determined in units of sequence, picture, slice, tile, etc. so that the related information can be generated in the unit, and when the support for additional quantization scheme is supported, the quantization scheme selection information is determined in units such as blocks. Related information may be generated.
  • the quantization scheme configuration may be determined according to at least one encoding information.
  • the encoding information may correspond to a slice type, an encoding mode, a size and shape of a block, a prediction mode, and the like.
  • the quantization unit may set a quantization weight matrix according to an encoding mode to a weight matrix applied according to inter prediction / intra prediction.
  • the weight matrix applied according to the intra prediction mode may be different.
  • the quantization weight matrix may be a quantization matrix in which some of the quantization components are configured differently, when it is assumed that the size of the block is equal to the size of the quantization block.
  • the quantization process may be omitted according to the encoding setting or the characteristics of the image.
  • the quantization process (including the inverse process) may be omitted according to the encoding setting (assuming lossless compression environment in this example).
  • the quantization process may be omitted when the compression performance through quantization is not exhibited according to the characteristics of the image.
  • the omitted region may be an entire region or a partial region may be omitted, and it may be determined whether such omission is supported according to the size and shape of the block.
  • Information about a quantization parameter may be generated in units of a sequence, a picture, a slice, a tile, a block, and the like.
  • a basic QP may be set in an upper unit where QP information is first generated ⁇ 1>, and a QP may be set to a value that is the same as or different from the QP set in the upper unit as the lower unit ⁇ 2>.
  • the QP may be finally determined.
  • a unit such as a sequence or a picture may be an example corresponding to ⁇ 1>
  • a unit such as a slice, a tile, a block, etc. may be a ⁇ 2>
  • a unit such as a block may be an example corresponding to ⁇ 3>.
  • Information on the QP may be generated based on the QP in each unit.
  • the preset QP may be set as a prediction value to generate difference information with the QP in each unit.
  • the difference information with the QP in the current unit may be generated by setting a QP obtained based on at least one of the QP set in the higher unit, the QP set in the previous same unit, or the QP set in the neighboring unit as a prediction value.
  • the difference information between the QP in the current unit may be generated by setting the QP obtained based on the QP set in the higher unit and the at least one encoded information as a prediction value.
  • the same previous unit may be a unit that may be defined according to the coding order of each unit, and neighboring units may be spatially adjacent units, and encoding information may include a slice type, an encoding mode, a prediction mode, and position information of the corresponding unit. And so on.
  • the QP of the current unit may generate difference information by setting the QP of the upper unit as a prediction value.
  • the difference information between the QP set in the slice and the QP set in the picture may be generated or the difference information between the QP set in the tile and the QP set in the picture may be generated.
  • the difference information between the QP set in the block and the QP set in the slice or tile may be generated.
  • the difference information between the QP set in the sub-block and the QP set in the block can be generated.
  • the QP of the current unit may generate difference information by setting a QP obtained based on a QP of at least one neighboring unit or a QP obtained based on a QP of at least one previous unit as a prediction value. Based on the QPs of neighboring blocks such as the left, upper left, lower left, upper and upper right of the current block, difference information with the obtained QP may be generated. Alternatively, difference information with the QP of the encoded picture before the current picture may be generated.
  • the QP of the current unit may generate difference information by setting a QP obtained based on the higher unit QP and at least one piece of encoding information as a prediction value.
  • the difference information between the QP of the current block and the QP of the slice corrected according to the slice type (I / P / B) may be generated.
  • the difference information between the QP of the current block and the QP of the tile corrected according to the encoding mode (Intra / Inter) may be generated.
  • the difference information between the QP of the current block and the QP of the picture corrected according to the prediction mode (directional / non-directional) may be generated.
  • difference information between the QP of the current block and the QP of the picture corrected according to the position information (x / y) may be generated.
  • the correction may mean adding or subtracting to the higher unit QP used for prediction in an offset form.
  • at least one offset information may be supported according to an encoding setting, and implicitly processed or explicitly related information may be generated according to a preset process. It is not limited only to the above examples, but modifications to other examples may be possible.
  • the example may be a possible example when a signal indicating QP variation is provided or activated.
  • difference information is not generated and the predicted QP may be determined as a QP of each unit.
  • difference information is generated and when the value is 0, the predicted QP may also be determined as a QP of each unit.
  • the quantization unit may transmit information necessary for generating a quantization block to the encoding unit and encode the information.
  • the quantization unit stores the information in a bitstream and transmits the information to the decoder, and the decoder of the decoder parses the information and dequantizes it. Can be used for the process.
  • the residual block may be transformed from the residual signal to generate a residual block having transform coefficients, and the quantization process may not be performed. Not only may the quantization process be performed without converting the residual signal into transform coefficients, but neither the transformation nor the quantization process may be performed. This may be determined according to the encoder setting.
  • the encoder scans the generated quantization coefficients, transform coefficients, or residual signals according to at least one scan order (eg, zigzag scan, vertical scan, horizontal scan, etc.), and then encodes the quantization coefficients, transform coefficients, Alternatively, the signal sequence may be generated and encoded by using at least one entropy coding technique.
  • the information about the scan order may be determined according to an encoding setting (eg, an encoding mode, a prediction mode, etc.), and may implicitly or explicitly generate related information. For example, one of a plurality of scanning orders may be selected according to the intra prediction mode.
  • encoded data including encoded information transmitted from each component may be generated and output as a bitstream, which may be implemented by a multiplexer (MUX).
  • MUX multiplexer
  • the coding scheme uses exponential Golomb, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC).
  • CABAC Context Adaptive Binary Arithmetic Coding
  • the entropy encoding apparatus When performing entropy encoding (assuming CABAC in this example) for syntax elements such as the residual block data and information generated during the encoding / decoding process, the entropy encoding apparatus is a binarizer or a context modeler. It may include a binary arithmetic coder. In this case, the binary arithmetic coding unit may include a regular coding unit and a bypass coding unit.
  • the binarization unit may output a bin string composed of 0 or 1 by binarizing the syntax elements.
  • the bin represents a bit composed of 0 or 1 and may be encoded through a binary arithmetic encoder.
  • one of the regular coding unit and the bypass coding unit may be selected based on the probability of occurrence of 0 and 1, which may be determined according to the encoding / decoding setting. If the syntax element is data having the same frequency of 0 and 1, the bypass coding unit may be used, otherwise the regular coding unit may be used.
  • binarization process for the syntax element occurring in the present invention may be performed including not only the binarization mentioned in the above example but also other additional binarization methods.
  • the inverse quantization unit and the inverse transform unit may be implemented by inversely performing the processes of the transform unit and the quantization unit.
  • the inverse quantization unit may inverse quantize the quantized transform coefficients generated by the quantization unit, and the inverse transform unit may inverse transform the inverse quantized transform coefficients to generate a reconstructed residual block.
  • the adder reconstructs the current block by adding the prediction block and the reconstructed residual block.
  • the reconstructed block can be stored in memory and used as reference data (prediction section, filter section, etc.).
  • the in-loop filter unit may include at least one post-processing filter process such as a deblocking filter, a pixel adaptive offset (SAO), an adaptive loop filter (ALF), and the like.
  • the deblocking filter may remove block distortion occurring at boundaries between blocks in the reconstructed image.
  • the ALF may perform filtering based on a value obtained by comparing the restored image with the input image. In detail, after the block is filtered through the deblocking filter, the filtering may be performed based on a value obtained by comparing the reconstructed image with the input image. Alternatively, filtering may be performed based on a value obtained by comparing the restored image with the input image after the block is filtered through SAO.
  • the SAO restores the offset difference based on a value obtained by comparing the reconstructed image with the input image, and may be applied in the form of a band offset (BO) and an edge offset (EO).
  • the SAO adds an offset from the original image in units of at least one pixel to the reconstructed image to which the deblocking filter is applied, and may be applied in the form of BO, EO, and the like.
  • an offset with respect to an original image is added to the image reconstructed after the block is filtered through the ALF, and may be applied in the form of BO, EO, and the like.
  • setting information on whether each post-processing filter is supported may be generated in units of a sequence, a picture, a slice, a tile, and the like.
  • setting information on whether each post-processing filter is performed may be generated in units of a picture, a slice, a tile, and a block.
  • the range to which the filter is applied may be divided into the inside of the image and the boundary of the image, and setting information considering the same may be generated.
  • information related to the filtering operation may be generated in units of a picture, a slice, a tile, a block, and the like.
  • the information may be implicitly or explicitly processed, and the filtering may be performed by an independent filtering process or a dependent filtering process according to color components, which may be determined according to an encoding setting.
  • the in-loop filter may transmit the filtering related information to an encoder to encode the information.
  • the in-loop filter stores the information in a bitstream and transmits the information to the decoder, and the decoder of the decoder parses the information about the in-loop filter. Applicable to
  • the memory may store the restored block or picture.
  • the reconstructed block or picture stored in the memory may be provided to a prediction unit that performs intra prediction or inter prediction.
  • the encoder may process a storage space in the form of a queue of a compressed bitstream as a coded picture buffer (CPB), and decode the space for storing the decoded image in picture units. It can be processed by placing it in a decoded picture buffer (DPB).
  • CPB coded picture buffer
  • decoding units are stored according to decoding order, and the decoding unit can be emulated in the encoder, and the compressed bitstream can be stored in the emulation process, and the bitstream output from the CPB is restored through the decoding process.
  • the image is stored in the DPB, and the pictures stored in the DPB can be referenced later during the image encoding and decoding process.
  • the decoder may be implemented by inversely performing the process of the encoder. For example, a quantization coefficient sequence, a transform coefficient sequence, or a signal sequence may be received from the bitstream and decoded.
  • the decoded data including the decoding information may be parsed and transmitted to each component.
  • an image setting process applied to an image encoding / decoding apparatus will be described. This may be an example applied to the step before performing the encoding / decoding (image initial setting), but some processes may be applied at other stages (eg, after the encoding / decoding is performed or the encoding / decoding internal stage, etc.). This may be an example.
  • the image setting process may be performed in consideration of a network and a user environment such as characteristics of multimedia content, bandwidth, performance and accessibility of a user terminal, and the like. For example, image segmentation, image size adjustment, image reconstruction, etc. may be performed according to the encoding / decoding.
  • the image setting process to be described later will be described based on the rectangular image, but the present invention is not limited thereto and may be applicable to the polygonal image.
  • the same image setting or different image settings may be applied, which may be determined according to the encoding / decoding setting. For example, after confirming the information on the shape of the image (for example, rectangular or non-rectangular), the information on the image setting may be configured accordingly.
  • an independent setting may include an example of separately setting a decoding / decoding setting in each color space, and an example that is applied to another color space even when one color space is described (for example, , When generating M from the luminance component generates N from the chrominance component).
  • M in the luminance component may include M / 2) in the chrominance component, and it may be derived by assuming that the example is applied to each color space without special description. This is not limited only to the above examples and may be a description commonly applied to the present invention.
  • Some components in the following example may be content that can be applied to various encoding techniques, such as encoding in the spatial domain, encoding in the frequency domain, block-based encoding, object-based encoding, and the like.
  • partitioning may be performed for error robustness in order to prevent damage due to packet damage during transmission.
  • segmentation may be performed for the purpose of classifying regions having different properties in the same image according to characteristics, types, and the like of the image.
  • a segmentation process may be performed and an inverse process may be performed.
  • an inverse process may be performed.
  • a description will be given based on the division process, but the contents of the division inversion process may be reversed in the division process.
  • 3 is an exemplary diagram of image information divided by layers in order to compress an image.
  • 3a is an exemplary diagram in which a sequence of images is composed of a plurality of GOPs.
  • One GOP may include an I picture, a P picture, and a B picture as in 3b.
  • One picture may be composed of a slice, a tile, or the like as in 3c.
  • a slice, a tile, etc. may be composed of a plurality of basic coding units as in 3d, and the basic coding unit may be configured as at least one lower coding unit as shown in FIG. 3e.
  • the image setting process according to the present invention will be described based on an example applied to units of a picture, a slice, a tile, and the like, such as 3b and 3c.
  • FIG. 4 is a conceptual diagram illustrating various examples of image segmentation according to an embodiment of the present invention.
  • each block may be a basic coding unit (or a maximum coding unit) obtained through the picture divider, or may be a basic unit applied in a division unit described later.
  • the 4b is a conceptual diagram of dividing an image into at least one of a horizontal direction and a vertical direction.
  • the divided regions T 0 to T 3 may be referred to as tiles, and each region may perform encoding / decoding independent or dependent on other regions.
  • the 4c is a conceptual diagram of dividing an image into a bundle of consecutive blocks.
  • the divided regions S 0 and S 1 may be referred to as slices, and each region may be a region that performs encoding / decoding independent or dependent on other regions.
  • the bundle of contiguous blocks may be defined according to the scan order, and generally, the raster scan order is not limited thereto but may be determined according to the encoding / decoding setting.
  • the 4d is a conceptual diagram of dividing an image into a bundle of blocks with arbitrary settings according to user definition.
  • the divided regions A 0 to A 2 may be referred to as arbitrary partitions, and each region may be an area that performs encoding / decoding independent or dependent on other regions.
  • Independent encoding / decoding may mean that data of other units cannot be referred to when performing encoding / decoding of some units (or regions).
  • the information used or generated in the texture encoding and the entropy encoding of some units are independently encoded without being referred to each other, and the parser and the reconstruction information of the other units are used for decoding and entropy decoding of the units of the decoder. May not be referenced to each other.
  • whether data of another unit (or region) can be referred to may be limited in a spatial region (for example, regions within one image), but may be temporal region (for example, depending on the encoding / decoding setting). Limited settings can also be made in consecutive images or frames). For example, some units of the current image and some units of another image may be referred to when there is a continuity or a condition having the same encoding environment, otherwise the reference may be limited.
  • dependent encoding / decoding may mean that other units of data may be referenced when performing encoding / decoding of some units.
  • the information used or generated in the texture encoding and the entropy encoding of some units are reliably encoded by referring to each other.
  • the parsing information and the reconstruction information of other units may be used for the texture decoding and the entropy decoding of some units.
  • the setting may be the same as or similar to that of general encoding / decoding.
  • the image may be divided for the purpose of identifying a region (a surface ⁇ Face>, etc. generated according to the projection format in this example) according to a characteristic, a type, etc. (eg, a 360 degree image) of the image.
  • Some units (slices, tiles, etc.) in the above example may have independent sub / decoding settings (e.g., independent slice segments), and some units depend on sub / decoding settings (e.g., dependent slice segments). ), And the present invention will be described based on independent encoding / decoding.
  • the basic coding units obtained through the picture splitter may be divided into basic coding blocks according to color spaces, and sizes and shapes may be determined according to characteristics and resolutions of an image.
  • Supported block sizes or shapes are N ⁇ N square shapes (2 n x 2 n . 256x256, 128x128, 64x64, 32x32, 16x16, 8x8, etc.) whose length and width are represented by an exponential power of 2 (2 n ).
  • n may be an integer between 3 and 8), or may be an M x N rectangular shape (2 m x 2 n ).
  • the input video can be divided into sizes such as 128x128 for 8k UHD video, 64x64 for 1080p HD video, and 16x16 for WVGA video, and 256x256 for 360-degree video, depending on the video type.
  • the input image can be divided by the size of.
  • the basic coding unit may be divided and encoded / decoded into lower coding units, and information about the basic coding unit may be transmitted in a bitstream in units of a sequence, a picture, a slice, a tile, and the like, which is parsed by a decoder. Related information can be restored.
  • An image encoding method and a decoding method may include the following image segmentation step.
  • the image segmentation process may include an image segmentation indicating step, an image segmentation type identification step, and an image segmentation performing step.
  • the video encoding apparatus and the decoding apparatus may be configured to include an image segmentation indicating step, an image segmentation type identifying step, an image segmentation indicating unit implementing an image segmentation performing step, an image segmentation type identifying unit, and an image segmentation performing unit.
  • an associated syntax element may be generated, and in the case of decoding, an associated syntax element may be parsed.
  • the image segmentation indicator may be omitted, and the image segmentation type identification unit may be a process of confirming information regarding the size and shape of the block.
  • the division may be performed by the basic coding unit.
  • a block it may be a unit in which division is always performed, while in other division units (tile, slice, etc.), whether or not to divide may be determined according to a sub / decoding setting.
  • the picture division unit may be configured to perform division of another unit after division of a block unit as a default setting. In this case, block division may be performed based on the size of the picture.
  • block division may be performed based on the size of the division unit. This can be determined through explicit or implicit processing, depending on the encoding / decryption settings. In the following example, the former case is assumed, and description will be made based on units other than the block.
  • the image segmentation instructing step it may be determined whether to perform image segmentation. For example, when the signal indicating the image segmentation (for example, tiles_enabled_flag) is confirmed, the segmentation may be performed. If the signal indicating the image segmentation is not confirmed, the segmentation may not be performed or other encoding / decoding information. You can perform division by checking.
  • the signal indicating the image segmentation for example, tiles_enabled_flag
  • the signal indicating the image segmentation is not confirmed, it may mean that the segmentation is not performed or the segmentation is performed in at least one unit. , first_slice_segment_in_pic_flag).
  • the signal is a signal for indicating whether to divide the image into a plurality of units, and according to the signal, it may be determined whether the image is divided. For example, when tiles_enabled_flag is a signal indicating whether to split an image, when tiles_enabled_flag is 1, it may mean that the tile_enabled_flag is split into a plurality of tiles, and when it is 0, it may mean that it is not split.
  • first_slice_segment_in_pic_flag is not a signal indicating whether the image is divided, but a signal indicating whether the image is the first slice segment in the image, but through this, whether the data is divided into two or more units (for example, when the flag is 0, It means that it is divided into slices).
  • a signal indicating image segmentation in a tile may not be provided, and a signal indicating image segmentation in a slice may be provided.
  • a signal indicating image segmentation may be provided based on the type, characteristic, etc. of the image.
  • the image segmentation type may be identified.
  • the image segmentation type may be defined by a method of performing segmentation, segmentation information, and the like.
  • the tiles may be defined in units obtained by dividing in the horizontal and vertical directions. In detail, it may be defined as a bundle of adjacent blocks in a rectangular space partitioned by at least one horizontal or vertical dividing line crossing the image.
  • the splitting information about the tiles may include boundary position information of the columns and columns, tile number information of the columns and columns, and tile size information.
  • the information on the number of tiles may include the number of rows of tiles (for example, num_tile_columns) and the number of columns (for example, num_tile_rows), and through this, the number of tiles may be divided into (number of rows ⁇ number of columns). Can be.
  • the size information of the tile may be obtained based on the number of tiles.
  • the size of the tiles may be equal or unequal, which may be determined implicitly or explicitly under related rules (eg, uniform_spacing_flag) may be generated.
  • the size information of the tile may include the size information of each column and the column of the tile (for example, column_width_tile [i], row_height_tile [i]) or may include the horizontal and vertical size information of each tile.
  • the size information may be additional information that may be additionally generated according to whether the tile sizes are equal (for example, when uniform_spacing_flag is 0, which means non-uniform division).
  • a slice may be defined as a bundle unit of consecutive blocks. In detail, it may be defined as a bundle of consecutive blocks based on a preset scan order (raster scan in this example).
  • the split information about the slice may include the number information of the slice, location information (eg, slice_segment_address) of the slice, and the like.
  • the position information of the slice may be position information of a predetermined block (for example, the first order in the scan order in the slice), and the position information may be scan order information of the block.
  • any divided region can have various division settings.
  • a division unit may be defined as a bundle of spatially adjacent blocks, and the division information about the partition unit may include size, shape, location information, and the like of the division unit. This is some examples of any divided region, and various divided forms may be possible as shown in FIG. 5.
  • FIG. 5 is another exemplary diagram of an image segmentation method according to an exemplary embodiment.
  • the image may be divided into a plurality of areas at least one block interval in the horizontal or vertical direction, and the division may be performed based on the block position information.
  • 5a shows an example in which division is performed based on column information of each block in the horizontal direction (A 0 , A 1 )
  • 5b indicates division based on column and column information of each block in the horizontal and vertical directions.
  • Example (B 0 ⁇ B 3 ) is shown.
  • the splitting information may include the number of splitting units, block interval information, splitting direction, and the like, and when some of the splitting information is implicitly included according to a preset rule, some splitting information may not be generated.
  • an image may be divided into a bundle of consecutive blocks based on a scan order.
  • An additional scan order other than the raster scan order of the existing slice may be applied to the image segmentation.
  • 5c shows an example (C 0 , C 1 ) in which the scan is performed clockwise or counterclockwise about the starting block (C 0 , C 1 ), and 5d shows the scanning in the vertical direction about the starting block (Vertical).
  • An example (D 0 , D 1 ) is shown.
  • the partitioning information may include partition unit count information, position information of the partition unit (for example, the first order in the scan order within the partition unit), information about the scan order, and the like, and implicitly according to a preset rule. If included, some partitioning information may not be generated.
  • the image may be divided into horizontal and vertical dividing lines.
  • Existing tiles may be divided by horizontal or vertical dividing lines and accordingly have a quadrangular dividing form, but it may be impossible to divide an image across the image by dividing lines.
  • segmentation eg, a segmentation line forming a right boundary of E1, E3, and E4 and a left boundary of E5
  • a partial division of the image may be possible.
  • An example of segmentation that crosses an image along a line (for example, a segmentation line forming a lower boundary of E2 and E3 and an upper boundary of E4) may not be possible.
  • division may be performed on a block-by-block basis (eg, division after block division is performed first) or division may be performed by the horizontal or vertical division line or the like (eg, division regardless of block division). Division by a line), whereby each division unit may not be composed of an integer multiple of the block.
  • partition information different from the existing tile may be generated, and the partition information may include the number information of the partition unit, the location information of the partition unit, the size information of the partition unit, and the like.
  • the position information of the division unit may be generated based on a preset position (eg, the upper left of the image), and the position information (eg, measured in pixel units or block units).
  • the size information may generate horizontal and vertical size information (for example, measured in pixel units or block units) of each division unit.
  • the division having any setting according to the user definition may be performed by applying a new division method, or may be performed by changing some configurations of the existing division. That is, it may be supported as a partitioning form that replaces or adds to the existing partitioning method, and some settings are changed in the existing partitioning method (slice, tile, etc.) (for example, follow a different scanning order, or have a rectangular shape. Other partitioning methods, and accordingly, different partitioning information generation, dependent encoding / decoding features, etc.).
  • settings for configuring additional division units may be supported, and additional division unit types (eg, rectangular spaces).
  • an image segmentation method may be supported based on the type, characteristic, etc. of an image. For example, some segmentation methods (eg, a surface of a 360 degree image) may be supported according to the type, characteristic, etc. of an image, and segmentation information may be generated based on the segmentation method.
  • the image may be segmented based on the identified segmentation type information. That is, partitioning may be performed in a plurality of partitioning units based on the identified partition type, and encoding / decoding may be performed based on the obtained partitioning unit.
  • the division unit may be determined whether the division unit has a sub / decoding setting according to the division type. That is, setting information necessary for the encoding / decoding process of each division unit may be allocated in an upper unit (eg, a picture) or may have independent encoding / decoding settings of a division unit.
  • slices can have independent sub / decoding settings of division units (eg, slice headers), and tiles can not have independent sub / decoding settings of subdivision units and pictures / decoding settings ( For example, it may have a setting dependent on PPS).
  • the information generated in relation to the tile may be split information, which may be included in the encoding / decoding setting of the picture.
  • the present invention is not limited only to the above cases, and other modifications may be possible.
  • the encoding / decoding setting information for the tile may be generated in units of a video, a sequence, a picture, and the like, and at least one encoding / decoding setting information may be generated in an upper unit to refer to one of them.
  • independent encoding / decoding setting information (for example, a tile header) may be generated in units of tiles. This is different from following one encoding / decoding setting determined in the upper unit in that the encoding / decoding is performed by setting at least one encoding / decoding setting in units of tiles. That is, one sub / decoding setting may be followed in all tiles, or at least one tile may perform sub / decoding along different sub / decoding settings from other tiles.
  • partition information may be generated in a higher unit and sub / decoding may be performed according to one sub / decoding configuration of a higher unit.
  • partition information may be generated in an upper unit, and an independent sub / decoding setting for each partition unit may be generated in an upper unit, and the sub / decoding may be performed accordingly.
  • partition information may be generated in an upper unit, a plurality of sub / decoding setting information may be supported in an upper unit, and encoding / decoding may be performed according to the sub / decoding setting referenced in each partition unit.
  • partition information may be generated in a higher unit, independent sub / decoding settings may be generated in a corresponding partition unit, and encoding / decoding may be performed accordingly.
  • an independent encoding / decoding configuration including division information may be generated in a corresponding division unit, and encoding / decoding may be performed accordingly.
  • the sub / decoding setting information includes information about tile type, reference picture list information, quantization parameter information, inter prediction prediction information, in-loop filtering setting information, in-loop filtering control information, scan order, and whether to perform sub / decoding. It may include information necessary for encoding / decoding of tiles.
  • the encoding / decoding setting information may explicitly generate related information, or the encoding / decoding setting may be implicitly determined according to the format, characteristics, etc. of the image determined in the higher unit. In addition, related information may be explicitly generated based on the information obtained through the setting.
  • the following is an example of performing image segmentation in the encoding / decoding apparatus according to an embodiment of the present invention.
  • a segmentation process may be performed on the input image before encoding starts. After segmentation is performed using segmentation information (eg, image segmentation information, segmentation unit setting information, etc.), an image may be encoded in segmentation units. After encoding is completed, the data may be stored in a memory, and the image encoded data may be stored in a bitstream and transmitted.
  • segmentation information eg, image segmentation information, segmentation unit setting information, etc.
  • the partitioning process may be performed before the decoding starts. After segmentation is performed using segmentation information (eg, image segmentation information, segmentation unit setting information, and the like), image decoding data may be parsed and decoded in segmentation units. After decoding, the data may be stored in a memory, and a plurality of division units may be merged into one to output an image.
  • segmentation information eg, image segmentation information, segmentation unit setting information, and the like
  • image decoding data may be parsed and decoded in segmentation units. After decoding, the data may be stored in a memory, and a plurality of division units may be merged into one to output an image.
  • division may be performed on an image, and division may be performed on a division unit of an image, and the division may be the same division process (for example, slice / slice, tile / tile, etc.) or another division process ( For example, slice / tile, tile / slice, tile / surface, surface / tile, slice / surface, surface / slice, etc.).
  • the subsequent partitioning process may be performed based on the preceding partitioning result, and the partitioning information generated in the subsequent partitioning process may occur based on the preceding partitioning result.
  • dividing processes A may be performed, and the dividing process may be another dividing process (eg, a slice / surface, a tile / surface, etc.).
  • the subsequent partitioning process may be performed based on the preceding partitioning result, or the partitioning process may be performed independently of the preceding partitioning result, and the partitioning information generated in the subsequent partitioning process may be performed by the preceding partitioning result. Can occur based on or occur independently.
  • the multi-segmentation process of the image may be determined according to the encoding / decoding setting, and various modification examples may be possible without being limited to the above examples.
  • the generated information will be recorded in the bitstream in at least one unit of a sequence, a picture, a slice, a tile, etc. in the encoder, and the decoder will parse related information from the bitstream. That is, it may be stored in one unit and may be recorded in a plurality of units.
  • a syntax element for whether some information is supported or a syntax element for activation or the like may be generated in some units (eg, an upper unit), and in some units (eg, a lower unit). The same or similar information as the case may be generated. That is, even when the related information is supported and set in the upper unit, it may have individual settings in the lower unit. This is not limited only to the above examples and may be a description commonly applied in the present invention. It may also be included in the bitstream in the form of SEI or metadata.
  • image scaling such as overall expansion and reduction of an image may be performed.
  • image size adjustment such as partial expansion and reduction of an image may be performed.
  • Image scaling may be performed by various examples, which may be performed for adaptability to an encoding environment, may be performed for encoding uniformity, may be performed for encoding efficiency, or may be performed for image quality improvement.
  • the image may be performed according to the type, characteristic, or the like of the image.
  • a resizing process may be performed in a process (for example, hierarchical coding, 360 degree video coding, etc.) that is performed according to characteristics, types, and the like of an image.
  • a process for example, hierarchical coding, 360 degree video coding, etc.
  • the resizing process may be performed in the initial stage of encoding / decoding.
  • the resizing process may be performed before performing the encoding / decoding, and may encode / decode the image to be resized.
  • a resizing process may be performed before a prediction step (intra-prediction or inter-screen prediction) or prediction.
  • image information in a prediction step for example, pixel information referred to in intra prediction, information related to intra prediction mode, reference image information used for inter prediction, information related to inter prediction mode, etc.can be used.
  • the resizing process may be performed before performing the filtering step or the filtering.
  • image information in the filtering step for example, pixel information applied to the deblocking filter, pixel information applied to the SAO, SAO filtering related information, pixel information applied to the ALF, ALF filtering related information, etc.can be used.
  • the image may or may not be changed to the pre-sizing image (the size perspective of the image) through the resizing process, which may be changed by the sub / decoding setting (e.g., And the nature of the work performed).
  • the resizing reverse process may be reduction
  • the resizing reverse process may be expansion.
  • the resizing process may be performed in a subsequent step to acquire an image before resizing.
  • the reversing process may not be performed in a subsequent step.
  • the image resizing process may be performed alone or inversely.
  • the following example will be described based on the resizing process.
  • the description of the resizing reverse process may be omitted in order to prevent duplicate explanation, but the person skilled in the art may recognize the same as that described in the literary manner. It is obvious.
  • FIG. 6 is an exemplary diagram for a method of adjusting a size of a general image.
  • an extended image P 0 + P 1 may be obtained by additionally including a partial region P 1 in the initial image (or the image before resizing. P 0. Thick solid line).
  • a reduced image S 0 may be obtained from the initial image S 0 + S 1 except for a partial region S 1 .
  • a partial area (T 1) to a further comprise and sizing to exclude some regions (T 2) image (T 0 + T 1) in the have to obtain an initial image (T 0 + T 2) a partial area (T 1) to a further comprise and sizing to exclude some regions (T 2) image (T 0 + T 1) in the have.
  • FIG. 7 is an exemplary view of image resizing according to an embodiment of the present invention.
  • a method of expanding an image during the resizing process may be described, and referring to 7b, a method of reducing an image may be described.
  • the image before scaling is S0
  • the image after scaling is S1
  • the image before scaling is T0
  • the image after scaling is T1.
  • the image When the image is expanded as shown in 7a, it can be expanded in the up, down, left, and right directions (ET, EL, EB, ER). RT, RL, RB, RR).
  • the up, down, left, and right directions in the expansion may correspond to the down, up, right, and left directions in the reduction, respectively. It should be understood that this includes.
  • the RC and BC regions may be obtained while the BR region may be obtained according to the encoding / decoding. That is, the case where the TL, TR, BL, and BR regions are acquired may or may not be possible.
  • the edge regions TL, TR, BL, and BR regions
  • An image resizing process may be performed in at least one direction. For example, it may be performed in all up, down, left, and right directions, and two or more selected directions of up, down, left, and right directions (left + right, up + down, up + left, up + right, down). + Left, down + right, up + left + right, down + left + right, up + down + left, up + down + right, etc., one of the up, down, left, right direction Only may be performed.
  • the size of the image may be adjusted in left + right, up + down, left + right, left + right, and right + left directions, which may be symmetrically extended at both ends based on the center of the image. It may be possible to adjust the size in the upper left + upper right, the lower left + right direction, the horizontal + symmetrical expansion of the image may be possible to adjust the size in the upper + lower, upper left + left, upper right + lower direction, and other sizes may be possible.
  • the image before resizing (S0, T0) is P_Width (width) ⁇ P_Height (height)
  • the image size after resizing (S1, T1) is P'_Width (width) ⁇ P'_Height (height)
  • the image size after scaling is (P_Width + Var_L + Var_R) ⁇ (P_Height + Var_T + Var_B).
  • Var_L, Var_R, Var_T, and Var_B which are scaling values in the left, right, up, and down directions, may be Exp_L, Exp_R, Exp_T, and Exp_B (Exp_x in this example is positive) in the image expansion (FIG.
  • -Rec_L, -Rec_R, -Rec_T, and -Rec_B if Rec_L, Rec_R, Rec_T, Rec_B is defined as a positive number, it may be represented as a negative number according to the image reduction).
  • the coordinates of the upper left, upper right, lower left and lower right of the image before resizing are (0,0), (P_Width-1,0), (0, P_Height-1), (P_Width-1, P_Height-1),
  • the coordinates of the image after scaling can be expressed as (0,0), (P'_Width-1,0), (0, P'_Height-1), (P'_Width-1, P'_Height-1) have.
  • the size of the area that is changed (or acquired, removed) by resizing (in this example, TL to BR.i is the index that distinguishes TL to BR) can be M [i] ⁇ N [i], which is Var_X ⁇ Var_Y (In this example, X is L or R, Y is T or B). M and N can have various values and can be the same or independent settings for i regardless of i. Various cases thereof will be described later.
  • S1 may be configured to include all or a portion of TL to BR (upper left to lower right) generated through expansion along various directions in S0.
  • T1 may be configured to exclude all or a portion of TL to BR that are removed through reduction in various directions.
  • the image when the expansion is performed in the up, down, left, and right directions of the existing image S0, the image may be configured to include the TC, BC, LC, and RC regions obtained according to each scaling process. It may additionally include TL, TR, BL, BR regions.
  • the image when performing the expansion in the upward (ET) direction, the image may be configured to include the TC region in the existing image S0, and may be TL or TR according to the expansion (EL or ER) in at least one other direction. It can include an area.
  • the image when the expansion is performed in the lower (EB) direction, the image may be configured to include the BC region in the existing image S0, and may be BL or BR according to the expansion (EL or ER) in at least one other direction. It can include an area.
  • the image when the expansion is performed in the left (EL) direction, the image may be configured to include the LC region in the existing image S0, and may be TL or BL according to the expansion (ET or EB) in at least one other direction. It can include an area.
  • the image when the expansion is performed in the right (ER) direction, the image may be configured to include the RC region in the existing image S0 and may be TR or BR according to the expansion (ET or EB) in at least one other direction. It can include an area.
  • there may be a setting for example, spa_ref_enabled_flag or tem_ref_enabled_flag
  • a setting for example, spa_ref_enabled_flag or tem_ref_enabled_flag
  • Encoding / decoding of the pre-scaling image S0 and T1 and the region (TC, BC, LC, RC, TL, TR, BL, BR region) added or deleted at the time of resizing may be performed as follows. .
  • the data of the image before resizing and the data of the region to be added or deleted (the data to be encoded / decoded. Pixel value or prediction related information, etc.) ) Can be referenced to each other spatially or temporally.
  • the data before the resizing and data of the region to be added or deleted may be spatially referenced, while the data of the image before the resizing may be temporally referred to, and the data of the region to be added or deleted may be temporally referred to. Can't.
  • a setting may be set to limit the referability of the region to be added or deleted.
  • the setting information on the referability of the region to be added or deleted may be explicitly created or implicitly determined.
  • An image scaling process may include an image scaling instruction step, an image scaling type identification step, and / or an image scaling step.
  • the image encoding apparatus and the decoding apparatus may include an image scaling instruction step, an image scaling type identification step, an image scaling instruction unit for implementing the image scaling step, an image scaling type identification unit, and an image scaling unit. have.
  • an associated syntax element may be generated, and in the case of decoding, an associated syntax element may be parsed.
  • resizing may be performed when a signal indicative of image resizing (eg, img_resizing_enabled_flag) is checked, and resizing is not performed when a signal indicative of image resizing is not confirmed or another
  • the resizing can be performed by checking the encoding / decoding information.
  • a signal indicating the scaling may be implicitly activated or deactivated according to a sub / decoding setting (for example, characteristics and types of the image).
  • the size related information may be generated or the size related information may be implicitly determined.
  • the signal is a signal indicating whether to scale the image, and it is possible to check whether the image is scaled according to the signal.
  • a signal indicating image scaling eg, img_resizing_enabled_flag
  • image scaling can be performed, and the signal is disabled.
  • img_resizing_enabled_flag 0
  • this may mean that image resizing is not performed.
  • the resizing may not be performed or the resizing of the corresponding image may be determined by another signal.
  • the image when dividing an input image into blocks, the image is resized according to whether the size of the image (e.g., width or height) is an integer multiple of the size of the block (e.g., width or length).
  • the size adjustment may be performed when the width of the image is not an integer multiple of the width of the block or when the length of the image is not an integer multiple of the length of the block.
  • the scaling information eg, scaling direction, scaling value, etc.
  • the sub / decoding information eg, the size of the image, the size of the block, etc.
  • the image may be resized according to characteristics, types (eg, 360 degree images) of the image, and the resizing information may be explicitly generated or assigned to a preset value.
  • characteristics e.g, 360 degree images
  • the resizing information may be explicitly generated or assigned to a preset value.
  • the image scaling type may be identified.
  • the image scaling type may be defined by a method of performing scaling, size adjustment information, and the like. For example, scaling through a scale factor and scaling through an offset factor may be performed. It is also possible to apply a mixture of the above manner is not limited to this. For convenience of explanation, the following description will focus on scaling through a scale factor and an offset factor.
  • scaling may be performed by multiplying or dividing based on the size of an image.
  • Information about the operation of scaling may be explicitly generated, and the expansion or contraction process may be performed according to the information.
  • the resizing process may be performed by a predetermined operation (for example, one of expansion or reduction) according to the encoding / decoding, and in this case, information on the operation of resizing may be omitted.
  • the resizing of the image may be performed by a preset operation.
  • the scaling direction may be at least one direction selected from up, down, left, and right directions. At least one scale factor may be needed according to the scaling direction. That is, one scale factor (unidirectional, in this example) may be required in each direction, and one scale factor (two-way, in this example) may be needed in the horizontal or vertical direction, one in the overall direction of the image. A scale factor of (in this example, omni-directional) may be required.
  • the resizing direction is not limited to the above example but may be modified in other examples.
  • the scale factor may have a positive value and different range information may be set according to a negative / decoding setting.
  • the scale factor when information is generated by mixing a scaling operation and a scale factor, the scale factor may be used as a value to be multiplied. If greater than 0 or less than 1, it may mean a reduction operation. If greater than 1, it may mean an expansion operation, and if it is 1, it may mean that no scaling is performed.
  • the scale factor information is generated separately from the scaling operation, the scale factor may be used as a multiplied value in the case of the extension operation, and the scale factor may be used as a divided value in the case of the reduction operation.
  • the scaling direction is ER, EB (or RR, RB), and the scaling value is Var_L (Exp_L or Rec_L) and Var_T (Exp_T or Rec_T) are 0, and Var_R (Exp_R or Rec_R) and Var_B (Exp_B or Rec_B) can be expressed as P_Width ⁇ (sc-1), P_Height ⁇ (sc-1). have. Therefore, the image after the resizing may be (P_Width ⁇ sc) ⁇ (P_Height ⁇ sc).
  • each scale factor (sc_w, sc_h in this example) is used according to the horizontal or vertical direction of the image, and the resizing direction is left + right and up + down directions (up + down + left + right when both are operated).
  • the scaling direction is ET, EB, EL, ER, and the scaling values Var_T and Var_B are P_Height ⁇ (sc_h-1) / 2, and Var_L and Var_R are P_Width ⁇ (sc_w-1) / 2 days. have. Therefore, the image after the resizing may be (P_Width ⁇ sc_w) ⁇ (P_Height ⁇ sc_h).
  • the scaling may be performed by adding or subtracting based on the size of the image.
  • the scaling may be performed by adding or subtracting based on the encoding / decoding information of the image.
  • the scaling may be performed by adding or subtracting independently. That is, the resizing process may be dependent or independent setting.
  • Information about the resizing operation may be explicitly generated, and the expansion or contraction process may be performed according to the corresponding information.
  • the resizing operation may be performed by a predetermined operation (for example, one of expansion or reduction) according to the encoding / decoding, and in this case, information on the resizing operation may be omitted.
  • the resizing of the image may be performed by a preset operation.
  • the scaling direction may be at least one of up, down, left, and right directions. At least one offset factor may be needed according to the scaling direction. That is, one offset factor (in this example, unidirectional) may be needed in each direction, and one offset factor (in this example, symmetrical bidirectional) may be needed, depending on the horizontal or vertical direction, and the partial Depending on the combination, one offset factor (in this example, asymmetric bidirectional) may be needed, and one offset factor (in this example, omni) in accordance with the overall direction of the image.
  • the resizing direction is not limited to the above example but may be modified in other examples.
  • the offset factor may have a positive value or may have a positive value and a negative value, and may have different range information according to a negative / decoding setting. For example, when information is generated by mixing a scaling operation and an offset factor (assuming positive and negative values in this example), the offset factor may be used as a value that is added or subtracted according to the sign information of the offset factor. . If the offset factor is greater than zero, it may mean an extension operation. If the offset factor is larger than zero, it may mean a reduction operation. If the offset factor is greater than zero, it may mean that no scaling is performed.
  • the offset factor when generating offset factor information separately from the resizing operation (assuming a positive value in this example), the offset factor may be used as a value added or subtracted according to the resizing operation. If it is greater than zero, the expansion or contraction operation may be performed according to the resizing operation, and if it is 0, this may mean that the resizing is not performed.
  • the scaling direction is ET, EB, EL, ER (or RT). , RB, RL, RR), and the scale values Var_T, Var_B, Var_L, and Var_R may be os.
  • the image size after resizing may be (P_Width + os) ⁇ (P_Height + os).
  • each offset factor (os_w, os_h) is used according to the horizontal or vertical direction of the image, and the resizing direction is left + right, up + down direction (up + down + left + right when both are operated)
  • the scaling directions are ET, EB, EL, and ER (or RT, RB, RL, RR), and the scaling values Var_T and Var_B are os_h, and Var_L and Var_R may be os_w.
  • the image size after resizing may be ⁇ P_Width + (os_w ⁇ 2) ⁇ ⁇ ⁇ P_Height + (os_h ⁇ 2) ⁇ .
  • the resizing direction is down and right directions (bottom + right when working together) and each offset factor (os_b, os_r) is used according to the resizing direction
  • the resizing direction is EB, ER (or RB, RR)
  • the scale value Var_B may be os_b
  • Var_R may be os_r.
  • the image size may be (P_Width + os_r) ⁇ (P_Height + os_b).
  • each offset factor (os_t, os_b, os_l, os_r) is used according to each direction of the image, and the resizing directions are up, down, left, and right directions (up + down + left + right when all operate).
  • the scaling direction is ET, EB, EL, ER (or RT, RB, RL, RR), and the scaling values Var_T may be os_t, Var_B is os_b, Var_L is os_l, and Var_R may be os_r.
  • the image size may be (P_Width + os_l + os_r) ⁇ (P_Height + os_t + os_b).
  • the offset factor refers to a case in which the offset factor is used as it is, and may be an example of scaling performed independently.
  • an offset factor may be used as an input variable of the scaling value.
  • an offset factor may be assigned as an input variable and a scaling value may be obtained through a series of processes according to a sub / decoding setting, which may be preset information (for example, image size, sub / decoding information, etc.). It may be an example of the scaling performed based on) or an example of the scaling performed depending on.
  • the offset factor may be a multiple of a preset value (an integer in this example) (e.g., 1, 2, 4, 6, 8, 16, etc.) or an exponent (e.g., 1, 2, 4, 8). Exponent of 2, such as, 16, 32, 64, 128, 256, etc.).
  • it may be a multiple or an index of a value obtained based on a negative / decoding setting (for example, a value set based on a motion search range of inter-screen prediction).
  • it may be a multiple or an index of a unit obtained from the picture division unit (assuming A ⁇ B in this example).
  • it may be a multiple of a unit obtained from the picture division unit (assuming E ⁇ F because it is the same as a tile in this example).
  • the value may be a value smaller than or equal to the width and length of the unit obtained from the picture divider.
  • the multiples or exponents of the above examples may also include cases where the value is 1, and the present invention is not limited to the above examples and modifications to other examples may also be possible.
  • Var_x may be 2 ⁇ n or 2 n .
  • individual offset factors may be supported according to color components, and offset factors for some color components may be supported to derive offset factor information for other color components. For example, if an offset factor A is explicitly generated for the luminance component (assuming that the ratio of the luminance component and the chrominance component is 2: 1 in this example), then the offset factor A / 2 for the chrominance component is Can be obtained implicitly. Alternatively, when the offset factor A for the chrominance component is explicitly generated, the offset factor 2A for the luminance component can be obtained implicitly.
  • Information about the scaling direction and the scaling value may be explicitly generated, and the scaling process may be performed according to the corresponding information. In addition, it may be implicitly determined according to the encoding / decoding setting, and accordingly, a resizing process may be performed. At least one preset direction or adjustment value may be allocated, in which case the related information may be omitted. In this case, the encoding / decoding setting may be determined based on the characteristic, type, encoding information, and the like of the image. For example, at least one scaling direction according to at least one scaling operation may be predetermined, at least one scaling value according to at least one scaling operation may be predetermined, and at least one scaling direction At least one scaling value according to may be predetermined.
  • the scaling direction and the scale value in the reverse scale process may be derived from the scale direction and the scale value applied in the scale process.
  • the size adjustment value implicitly determined may be one of the above-described examples (examples in which various size adjustment values are obtained).
  • the case of multiplication or division has been described, but may be implemented as a shift operation according to the implementation of the encoder / decoder.
  • the left shift operation may be performed, and in the case of division, the right shift operation may be implemented. This is not limited only to the above examples and may be a description commonly applied in the present invention.
  • image scaling may be performed based on the identified scaling information. That is, the image may be resized based on information such as a resizing type, a resizing operation, a resizing direction, and a resized value, and encoding / decoding may be performed based on the acquired image after resizing. .
  • the resizing may be performed using at least one data processing method.
  • the resizing may be performed using at least one data processing method in an area to be resized according to the resizing type and the resizing operation. For example, depending on the type of scaling, one can determine how to populate the data when scaling is extended, or how to remove data when scaling is shrinking.
  • image scaling may be performed based on the scaling information identified in the image scaling step.
  • the image resizing may be performed based on the resizing information and the data processing method in the image resizing step.
  • the difference between the two cases may be to adjust only the size of the image to be encoded / decoded or to consider data processing of the size and the region to be resized.
  • Whether to perform the image resizing including the data processing method may be determined according to the application stage, the position, and the like of the resizing process. In the following example, a description will be given of an example of performing size adjustment based on a data processing method, but the present invention is not limited thereto.
  • scaling can be done using various methods for scaling up and down.
  • the resizing may be performed using at least one data filling method, and in the case of reduction, the resizing may be performed using the method of removing at least one data.
  • new data or data of an existing image may be directly or deformed to be filled in the resizing area (extension), and the resize area (reduction) may be simply removed or removed through a series of processes. Can be applied and removed.
  • scaling can be done by applying upsampling, and scaling is done by applying downsampling to scale.
  • upsampling filter can be used for expansion
  • at least one downsampling filter can be used for reduction
  • the filters applied to horizontal and vertical can be the same, and applied to horizontal and vertical.
  • the filter may be different.
  • new data may not be created or removed in the resizing area, but may be rearranged using an interpolation method such as interpolation. It may be divided into filters used for the sampling.
  • expansion can be scaled using at least one data padding method, and reduction can be scaled using at least one data removal method. Can be performed.
  • a description will be given focusing on a data processing method in the case of performing scaling through an offset factor.
  • one preset data processing method may be used for the resized area, but at least one data processing method may be used for the resized area as in the following example, and selection information on the data processing method may be generated.
  • the resizing may be performed through a fixed data processing method and the latter in an adaptive data processing method.
  • some units of an area added or deleted upon resizing eg, TL, TC, TR, ..., BR in FIGS. 7A and 7B
  • the data processing method may be applied to each of a combination of TL to BR of FIG. 7A and FIG. 7B or a part thereof.
  • FIG. 8 is an exemplary view illustrating a method of configuring an extended area in an image resizing method according to an embodiment of the present invention.
  • the images may be classified into TL, TC, TR, LC, C, RC, BL, BC, and BR regions for convenience of explanation, and the upper left, upper, right upper, left, center, right, lower, It can correspond to the lower right position.
  • TL TL
  • TR TR
  • LC LC
  • C C
  • RC RC
  • BL BC
  • BR BR
  • the area added according to the expansion of the image may be configured in various ways.
  • the region may be filled with an arbitrary value or filled with reference to some data of the image.
  • regions A0 and A2 extending to arbitrary pixel values may be filled.
  • Arbitrary pixel values can be determined using various methods.
  • an arbitrary pixel value may be one pixel belonging to a pixel value range (for example, 0 to 1 ⁇ (bit_depth) ⁇ 1) that can be expressed as a bit depth.
  • bit_depth a pixel value range
  • it may be a minimum value, a maximum value, a median value of the pixel value range (for example, 1 ⁇ (bit_depth ⁇ 1), etc.) and the like (where bit_depth is a bit depth).
  • the arbitrary pixel value is a pixel value range (eg, min P to max P of pixels belonging to an image).
  • min P , max P are the minimum and maximum pixels among the pixels belonging to the image.
  • min P may be greater than or equal to 0 and max P may be one pixel belonging to 1 ⁇ less than or equal to (bit_depth) ⁇ 1 ⁇ .
  • bit_depth bit_depth
  • any pixel value may be a minimum value, a maximum value, a median value, an average (at least two pixels), a sum of weights, and the like of the pixel value range.
  • an arbitrary pixel value may be a value determined in a pixel value range belonging to a partial region belonging to an image.
  • some areas may be TR + RC + BR.
  • some areas may have 3 ⁇ 9 of TR, RC, and BR as corresponding areas, or 1 ⁇ 9 ⁇ assuming rightmost line> as corresponding areas, which may be subject to encoding / decoding.
  • the partial region may be a unit divided from the picture divider.
  • any pixel value may be a minimum value, a maximum value, a median value, an average (at least two pixels), a weighted sum, and the like, of the pixel value range.
  • the area A1 added according to the image expansion is pattern information generated using a plurality of pixel values (for example, it is assumed that a pattern uses a plurality of pixels. It is not necessary to follow a predetermined rule. Can be filled using).
  • the pattern information may be defined according to the encoding / decoding setting, or may generate related information, and may fill an extended area using at least one pattern information.
  • an area added according to image expansion may be configured by referring to pixels of some regions belonging to an image.
  • the area to be added can be configured by copying or padding the pixels (hereinafter referred to as reference pixels) in the area adjacent to the area to be added.
  • the pixel in the region adjacent to the added region may be a pixel before encoding or a pixel after encoding (or decoding).
  • the reference pixel may mean a pixel of the input image, and when the resizing is performed in the intra prediction screen generation step, the reference image generation step, and the filtering step, etc.
  • the reference pixel may mean a pixel of the reconstructed image. In this example, it is assumed that the pixel closest to the region to be added is used, but the present invention is not limited thereto.
  • the area A0 extending in the left or right direction related to the horizontal resizing of the image may be configured by padding (Z0) the outer pixels adjacent to the area A0 in the horizontal direction and related to vertical resizing of the image.
  • the region A1 extending in the up or down direction may be configured by padding Z1 the outer pixel adjacent to the region A1 extending in the vertical direction.
  • the area A2 extending in the lower right direction may be configured by padding Z2 the outer pixel adjacent to the area A2 extending in the diagonal direction.
  • an extended area B'0 to B'2 may be configured by referring to data B0 to B2 of a partial area belonging to an image.
  • 8d unlike 8c, it may be distinguished in that it can refer to an extended region and a region which is not adjacent to each other.
  • the expanded region may be filled with reference to the pixel of the highly correlated region.
  • location information, area size information, and the like of a highly correlated region may be generated.
  • a region of high correlation exists through encoding / decoding information such as characteristics and types of the image, and implicitly confirms location information and size information of a region of high correlation (for example, in a 360-degree image). If you can, you can populate the area with data from that area. In this case, location information, area size information, etc. of the corresponding area may be omitted.
  • the pixel of the area B2 opposite to the expanded area in the left or right direction related to the horizontal scaling is referred to.
  • To fill in the expanded area in the case of the area B'2 extending in the left or right direction related to the horizontal scaling of an image, the pixel of the area B2 opposite to the expanded area in the left or right direction related to the horizontal scaling is referred to.
  • the pixel of the area B1 opposite to the extended area in the up or right direction related to the vertical resizing is referred to.
  • the pixel of the area B1 opposite to the extended area in the up or right direction related to the vertical resizing is referred to.
  • the image is extended with reference to the pixels in the areas B0 and TL opposite to the expanded area. You can fill the area to be.
  • the data of the corresponding area may be copied and filled as it is, or the data of the area may be filled after performing a conversion process based on the characteristics and types of the image.
  • copying as it is it may mean that the pixel value of the corresponding area is used as it is, and when it is converted, it may mean that the pixel value of the corresponding area is not used as it is. That is, a change may be made in at least one pixel value of the corresponding region through the conversion process, and may be filled in the expanded region, or at least one acquisition position of some pixels may be different.
  • the data of CxD may be used instead of the data of AxB of the corresponding area to fill AxB which is an extended area.
  • at least one motion vector applied to the pixel to be filled may be different.
  • the above example may be an example that occurs when a plurality of surfaces are formed according to a projection format in a 360-degree image when filling in an area that is expanded using data of another surface.
  • the data processing method for filling the region extended by the size adjustment of the image is not limited to the above example and may be improved and changed or an additional data processing method may be used.
  • Candidate groups for a plurality of data processing methods may be supported according to encoding / decoding, and data processing method selection information among a plurality of candidate groups may be generated and recorded in a bitstream.
  • one data processing method may be selected from a method of filling using a preset pixel value, a method of copying and filling an outer pixel, a method of copying and filling a portion of an image, and a method of converting and filling a portion of an image.
  • selection information may be generated.
  • an implicit data processing method may be determined.
  • the data processing method applied to the entire area (in this example, TL to BR of FIG. 7A) extended by the resizing of an image may be filled using a preset pixel value, a method of copying and filling an outer pixel, The method may be one of a method of copying and filling a partial region of the image, a method of converting and filling a partial region of the image, and other methods, and selection information may be generated.
  • one preset data processing method applied to the entire area may be determined.
  • the data processing method applied to the region may be filled using a preset pixel value.
  • the method may be one of a method, a method of copying and filling an outer pixel, a method of copying and filling a partial region of an image, a method of converting and filling a partial region of an image, and other methods, and selection information may be generated.
  • one preset data processing method applied to at least one area may be determined.
  • FIG. 9 is an exemplary view illustrating a method of configuring a region to be deleted and a region to be generated according to image size reduction in an image size adjusting method according to an embodiment of the present invention.
  • the area to be deleted in the image reduction process may not only be removed but also removed after a series of utilization processes.
  • the image A may be subdivided and referred to as TL to BR as in FIG. 8A.
  • the partial regions A0 to A2 may be removed, and may be used as reference information when encoding / decoding the image A.
  • FIG. For example, some areas A0 to A2 deleted in the restoration process or the correction process of the partial area of the reduced image A generated may be utilized.
  • a weighted sum, an average, and the like of two regions may be used.
  • the restoration or correction process may be a process that can be applied when the two regions have a high correlation.
  • the area B′2 deleted as the image is reduced in the left or right direction related to the horizontal resizing of the image may be formed on the opposite side (B2, LC) of the region reduced in the left or right direction related to the horizontal resizing.
  • the area can be removed from memory after it has been used to restore or correct the pixels.
  • the area B'1 deleted in the up or down direction related to the vertical resizing of the image may be a negative / neutral portion of the areas B1 and TR opposite to the reduced area in the up or down direction related to the vertical resizing.
  • the area can be removed from memory.
  • an area B'0 that is reduced by adjusting some size of the image may be a negative / decoding process (restore) of the areas B0 and TL opposite to the area that is reduced.
  • the area can be removed from memory after the calibration process.
  • the continuity exists at both edges of the image, and thus the case of using the data for restoring or correcting the data in the symmetrical position with the resizing direction is not limited thereto. It can also be removed from memory after it is used for data restoration or correction.
  • the data processing method for eliminating the reduced region of the present invention is not limited to the above examples, and may be improved and changed or additional data processing methods may be used.
  • candidate groups for a plurality of data processing methods may be supported, and selection information about the plurality of data processing methods may be generated and recorded in the bitstream. For example, one data processing method may be selected from a method of simply removing a size to be resized, a method of removing and then using the size to be resized in a series of processes, and generating selection information.
  • an implicit data processing method may be determined.
  • the data processing method applied to the entire area (in this example, TL to BR in FIG. 7B of FIG. 7) that is deleted as the image is reduced due to the resizing of the image is a simple removal method, and used after a series of processes.
  • the method may be one of other methods, and selection information may be generated.
  • an implicit data processing method may be determined.
  • the data processing method applied to an individual region (in this example, each of TL to BR in FIG. 7B) reduced by resizing of an image may be a simple removal method, a method of removing the image after using it in a series of procedures, and other methods. One may be selected, and selection information may be generated. In addition, an implicit data processing method may be determined.
  • a method of filling a region to be enlarged by using some data of the image is selected, and a method of removing and then removing the region to be reduced in the reverse process thereof is used to restore or correct some data of the image.
  • a method of filling a region to be enlarged using a copy of an outer pixel is selected, and a method of simply removing a region to be reduced in the reverse process is selected. That is, the data processing method in the reverse process may be determined based on the data processing method selected in the image resizing process.
  • an image resizing process and an inverse process data processing method may have an independent relationship. That is, the data processing method in the reverse process may be selected regardless of the data processing method selected in the image scaling process. For example, a method of filling a region to be enlarged using some data of an image may be selected, and a method of simply removing a region to be reduced in the reverse process may be selected.
  • the data processing method in the image resizing process may be implicitly determined according to the encoding / decoding setting, and the data processing method in the reverse process may be implicitly determined according to the encoding / decoding setting.
  • the data processing method in the image resizing process may be explicitly generated, and the data processing method in the reverse process may be explicitly generated.
  • the data processing method in the image resizing process may be explicitly generated, and the data processing method in the reverse process may be implicitly determined based on the data processing method.
  • the resizing process is expanded, and the resizing process is reduced.
  • the difference between “image before resizing” and “image after resizing” may refer to the size of the image, and the size related information may be explicitly generated or some information may be generated depending on the encoding / decoding setting. Information can be implicit.
  • the scaling related information may include information about the scaling process and the scaling reverse process.
  • a size adjustment process may be performed on an input image before encoding starts.
  • Image after resizing using resizing information e.g. resizing behavior, resizing direction, resizing value, data processing method, etc.
  • the data processing method is used in the resizing process
  • the data may be stored in the memory, and the image encoded data (in this example, the image after the resizing) may be stored in the bitstream and transmitted.
  • the resizing process may be performed before the decoding starts. After the resizing is performed using the resizing information (for example, resizing operation, resizing direction, resizing value, etc.), the decoded image data may be parsed and decoded after resizing. After the decoding is completed, it can be stored in the memory, and the output image can be changed to the image before resizing by performing a scaling inverse process (in this example, a data processing method, which is used in the scaling inverse process).
  • a scaling inverse process in this example, a data processing method, which is used in the scaling inverse process.
  • a resizing process may be performed on a reference image before encoding starts.
  • Image after resizing using resizing information e.g. resizing behavior, resizing direction, resizing value, data processing method, etc.
  • the data processing method is used in the resizing process
  • the reference image after resizing in this example can be stored in the memory, and the image can be encoded using this.
  • the image encoded data meaning the encoding using the reference image in this example
  • a resizing process may be performed as described above.
  • the resizing process may be performed on the reference picture before decoding starts.
  • Image after resizing using resizing information e.g. resizing behavior, resizing direction, resizing value, data processing method, etc.
  • the data processing method is used in the resizing process
  • the reference image may be stored in a memory, and the image decoded data (the same as that encoded by using the reference image in the encoder in this example) may be parsed and decoded.
  • an output image may be generated, and when the decoded image is included as a reference image and stored in a memory, the resizing process may be performed as described above.
  • the encoding is completed except for the filtering process.
  • the image resizing process may be performed before the filtering of the image (assuming a deblocking filter in this example) starts.
  • Image after resizing using resizing information e.g. resizing behavior, resizing direction, resizing value, data processing method, etc.
  • the data processing method is used in the resizing process
  • the resizing process can be performed to change the image before resizing.
  • the decoding is completed except for the filtering process.
  • the resizing process may be performed on the image before the filtering of the image is started.
  • Image after resizing using resizing information e.g. resizing behavior, resizing direction, resizing value, data processing method, etc.
  • the data processing method is used in the resizing process
  • the resizing process can be performed to change the image before resizing.
  • the scaling process and the scaling inverse process may be performed, and in other cases (second example), only the scaling process may be performed.
  • the scaling process in the encoder and the decoder may be the same, and in other cases (the first example), the scaling process in the encoder and the decoder is the same or May not be the same.
  • the difference in the resizing process in the encoder / decoder may be a resizing step.
  • the resizing step may include a resizing step that takes into account the scaling of the image and data processing of the region to be resized, and in some cases (decoder in this example) It may include the step of performing the scaling to consider the scaling.
  • the former data processing may correspond to the data processing of the latter resizing process.
  • the resizing process may be a process applied only to the corresponding step, and the resizing area may not be stored in the memory.
  • the filter can be stored in temporary memory for use in the filtering process, and the region can be removed by resizing. In this case, there is no change in the size of the image due to resizing. Can be. It is not limited to the above examples but modifications to other examples may be possible.
  • the size of the image may be changed through the resizing process, and thus the coordinates of some pixels of the image may be changed through the resizing process, which may affect the operation of the picture divider.
  • the block-based division may be performed based on the image before the resizing or the block-based division may be performed based on the image after the resizing.
  • division of some units for example, tiles, slices, etc.
  • division of some units may be performed based on the image before resizing
  • division of some units may be performed based on the image after resizing, which may be encoded / decoded. It can be determined according to the setting.
  • the picture division unit operates based on the image after the resizing (for example, the image dividing process after the resizing process) will be described. However, other modifications are also possible. The case for the example will be explained.
  • the generated information will be recorded in the bitstream in at least one unit of a sequence, a picture, a slice, a tile, etc. in the encoder, and the decoder will parse related information from the bitstream. It may also be included in the bitstream in the form of SEI or metadata.
  • image reconstruction may be performed for the purpose of improving encoding efficiency of the image
  • image reconstruction may be performed for the purpose of considering the network and user environment
  • image reconstruction may be performed according to the type and characteristics of the image. can do.
  • the image reconstruction process may be performed by the reconstruction process alone or may be performed in reverse.
  • the reconstruction process will be described.
  • the reconstruction process may be reversed in the reconstruction process.
  • FIG. 10 is an exemplary view of image reconstruction according to an embodiment of the present invention.
  • 10a to 10d may include a rotation including zero degrees in the image (for example, a candidate group generated by sampling 360 degrees into k intervals. K is 2 or 4). , 8, etc., and in the present example, it is an exemplary diagram in which 4 is applied and 10e to 10h are exemplary diagrams in which inversion (or symmetry) is applied based on 10a or 10b to 10d. .
  • the starting position or the scanning order of the image may be changed, or the preset starting position and the scanning order may be followed regardless of whether or not to reconstruct the image, which may be determined according to the encoding / decoding setting.
  • the case corresponds to a preset start position (eg, the upper left position of the image) and a scanning order (eg, raster scan) regardless of whether the image is reconstructed or not.
  • the image encoding method and the decoding method may include the following image reconstruction step.
  • the image reconstruction process may include an image reconstruction instruction step, an image reconstruction type identification step, and an image reconstruction step.
  • the image encoding apparatus and the decoding apparatus may be configured to include an image reconstruction instruction step, an image reconstruction type identification step, an image reconstruction instructor implementing an image reconstruction operation step, an image reconstruction type identification unit, and an image reconstruction execution unit.
  • an associated syntax element may be generated, and in the case of decoding, an associated syntax element may be parsed.
  • the image reconstruction instruction step it may be determined whether to perform image reconstruction. For example, reconstruction may be performed when a signal indicating image reconstruction (for example, convert_enabled_flag) is confirmed, and reconstruction is not performed when a signal indicating image reconstruction is not confirmed, or other encoding / decoding information. You can check the reconstruction. In addition, even if a signal for instructing image reconstruction is not provided, a signal for instructing reconstruction may be implicitly activated or deactivated according to a sub / decoding setting (for example, characteristics, types, etc. of an image). If so, the reconfiguration related information may be generated or the reconfiguration related information may be implicitly determined.
  • a signal indicating image reconstruction for example, convert_enabled_flag
  • a signal indicating image reconstruction e.g. convert_enabled_flag
  • convert_enabled_flag 1
  • reconstruction is not performed or whether the reconstruction of the corresponding image is reconstructed by another signal.
  • reconstruction may be performed according to the characteristics, types, and the like (eg, 360 degree images) of the image, and the reconstruction information may be explicitly generated or assigned to a preset value.
  • the present invention is not limited only to the above example, and modifications to other examples may be possible.
  • the image reconstruction type may be identified.
  • the image reconstruction type may be defined by a method of performing reconstruction, reconstruction mode information, or the like.
  • the method of performing the reconstruction (eg, convert_type_flag) may include rotation, inversion, and the like.
  • the reconstruction mode information may include a mode (eg, convert_mode) in the method of performing the reconstruction.
  • the reconfiguration related information may include a method of performing reconfiguration and mode information. That is, it may be composed of at least one syntax element. In this case, the number of candidate groups of mode information according to the method of performing each reconstruction may be the same or different.
  • it may include candidates having a constant interval difference (90 degrees in this example), such as 10a to 10d.
  • 10a to 10d When 10a is 0 degrees rotation, 10b to 10d is 90 degrees, 180 degrees, An example of applying a 270 degree rotation (in this example, measuring an angle clockwise).
  • the inversion may include candidates such as 10a, 10e, and 10f.
  • 10e and 10f may be examples of applying left and right inversion and up and down inversion respectively.
  • the above example describes the case of the setting for the rotation with a certain interval and the setting for the partial inversion, but is not limited to the above example only for the image reconstruction and may include examples such as different interval difference, different inversion operation, etc. This may be determined according to the negative / decryption setting.
  • the method may include integrated information (eg, convert_com_flag) generated by mixing a method of performing reconstruction and mode information according thereto.
  • the reconfiguration-related information may be composed of information in which a method of performing reconstruction and mode information are mixed.
  • the integrated information may include candidates such as 10a to 10f, which are examples of applying 0 degree rotation, 90 degree rotation, 180 degree rotation, 270 degree rotation, left and right reverse, and up and down inversion based on 10a. have.
  • the integrated information may include candidates such as 10a to 10h, which are 0 degrees rotation, 90 degrees rotation, 180 degrees rotation, 270 degrees rotation, left and right reverse, upside down, 90 degrees rotation based on 10a.
  • This can be an example of applying reverse (or 90 degree rotation after left and right reverse), up and down reverse (or 90 degree rotation after upside down), or 0 degree rotation, 90 degree rotation, 180 degree rotation, 270 degree rotation, left and right reverse.
  • It may be an example of applying left and right inversion after rotating 180 degrees (rotating 180 degrees after inverting left and right), right and left inversion after rotating 90 degrees (90 degrees after turning left and right), and inverting left and right after turning 270 degrees (rotating 270 degrees after inverting left and right).
  • the candidate group may include a mode in which rotation is applied and a mode in which rotation and inversion are mixed in a mode in which rotation is applied.
  • the mixed configured mode may simply include mode information in a method of performing reconstruction, and may include a mode generated by mixing mode information in each method.
  • the method may include a mode generated by mixing at least one mode of some methods (eg, rotation) and at least one mode of some methods (eg, inversion), and the example may include one of some methods.
  • a plurality of modes of some methods are generated mixed (in this example, 90 degrees rotation + a plurality of inversion / left and right inversion + a plurality of rotations).
  • the mixed information may be configured to include the case where the reconstruction is not applied (10a in this example) as a candidate group, and the case where the reconstruction is not applied as the first candidate group (for example, assigning 0 as an index). It may include.
  • the reconfiguration related information may be composed of mode information according to a method of performing a preset reconfiguration. That is, the information on how to perform the reconstruction may be omitted, and may be composed of one syntax element related to the mode information.
  • it can be configured to include candidates such as 10a-10d regarding rotation.
  • it may be configured to include candidates such as 10a, 10e, and 10f regarding inversion.
  • the size of the image before and after the image reconstruction process may be the same or at least one length may be different, and this may be determined according to the encoding / decoding setting.
  • the reconstruction of the image may be a process of relocating pixels in the image (in this example, in the reconstruction inverse of the image, a pixel relocation inverse may be performed. This may be reversed in the pixel repositioning process). The location can be changed.
  • the pixel relocation may be performed according to a rule based on image reconstruction type information.
  • the rearrangement process of the pixels may be affected by the size and shape of the image (for example, square or rectangular).
  • the horizontal and vertical lengths of the image before the reconstruction process and the horizontal and vertical lengths of the image after the reconstruction process may serve as variables in the rearrangement process of the pixels.
  • the ratio between the width of the image before the reconstruction and the image width after the reconstruction, the ratio of the width of the image before the reconstruction and the length of the image after the reconstruction, and the length and reconstruction of the image before the reconstruction At least one ratio information (e.g., the former / the latter or the latter / the former, etc.) of the ratio between the horizontal length of the image after the process, the vertical length of the image before the reconstruction process, and the vertical length of the image after the reconstructing process is performed. Can act as a variable in the relocation process.
  • the ratio between the horizontal length and the vertical length of the image may act as a variable in the pixel rearrangement process.
  • the ratio between the image length before the image reconstruction process and the image length after the reconstruction process may serve as a variable in the pixel rearrangement process.
  • image reconstruction may be performed based on the identified reconstruction information. That is, the reconstruction of the image may be performed based on information such as the reconstruction type, the reconstruction mode, and the encoding / decoding may be performed based on the obtained reconstructed image.
  • the following is an example of performing image reconstruction in the encoding / decoding apparatus according to an embodiment of the present invention.
  • a reconstruction process may be performed on the input image before encoding starts.
  • the reconstructed image may be encoded after reconstruction using the reconstruction information (eg, image reconstruction type, reconstruction mode, etc.).
  • the data may be stored in a memory, and the image encoded data may be stored in a bitstream and transmitted.
  • the reconstruction process may be performed before the decoding starts. After the reconstruction is performed using the reconstruction information (for example, image reconstruction type, reconstruction mode, etc.), the image decoded data may be parsed and decoded. After decoding, the data may be stored in the memory, and the image may be output after changing to the image before reconstruction by performing the reconstruction process.
  • the reconstruction information for example, image reconstruction type, reconstruction mode, etc.
  • the generated information will be recorded in the bitstream in at least one unit of a sequence, a picture, a slice, a tile, etc. in the encoder, and the decoder will parse related information from the bitstream. It may also be included in the bitstream in the form of SEI or metadata.
  • Table 1 shows an example of syntax elements related to segmentation during image setting.
  • the description will focus on additional syntax elements.
  • the syntax elements of the example described below are not limited to any specific unit and may be syntax elements supported in various units such as a sequence, a picture, a slice, a tile, and the like. Alternatively, it may be a syntax element included in SEI, metadata, and the like.
  • the types of syntax elements, syntax element order, and conditions supported in the following examples are limited only to the present example and may be changed and determined according to encoding / decoding settings.
  • not performing encoding means not generating encoded data in a corresponding tile (assuming that the region is processed according to a predetermined rule, etc. in this example). It is applicable to a meaningless region in some projection formats of a 360 degree image. May mean.
  • Not performing decryption means that the decrypted data in the tile is no longer parsed (in this example, the region is assumed to be processed according to a predetermined rule).
  • not parsing the decoded data any more may mean that the encoded data does not exist in the unit and no longer parses the encoded data, but may also mean that the encoded data is not parsed any more by the flag. .
  • Header information of a tile unit may be supported according to whether tiles are decoded or decoded.
  • tile division setting is not limited to the above case, and modifications to other examples may also be possible.
  • Table 2 shows examples of syntax elements related to reconstruction during image setup.
  • convert_type_flag means mixed information about a reconstruction method and mode information. It may be determined as one of a plurality of candidate groups regarding a method of applying rotation, a method of applying inversion, and a method of applying rotation and inversion.
  • Table 3 shows an example of syntax elements related to scaling during image setting.
  • pic_width_in_samples and pic_height_in_samples mean syntax elements related to the horizontal and vertical lengths of an image, and the size of the image may be confirmed by the syntax elements.
  • img_resizing_enabled_flag means a syntax element for resizing the image.
  • resizing_mov_flag means a syntax element for the resizing operation. For example, it can be determined as one of expansion or contraction.
  • width_scale and height_scale refer to scale factors related to horizontal scaling and vertical scaling in scaling through scale factors.
  • top_height_offset and bottom_height_offset are the up and down offset factors related to horizontal scaling during scaling with the offset factor
  • left_width_offset and right_width_offset are the left and right directions associated with vertical scaling during scaling through the offset factor.
  • the offset factor is the offset factor.
  • the size of the image may be updated after the scaling through the scaling related information and the image size information.
  • resizing_type_flag means a syntax element for a data processing method of a size to be resized. Depending on the resizing method and the resizing operation, the number of candidate groups for the data processing method may be the same or different.
  • the image setting process applied to the above-described image encoding / decoding apparatus may be performed individually or a plurality of image setting processes may be mixed. In the following example, a case where a plurality of image setting processes are mixed and performed will be described.
  • 11 is an exemplary diagram illustrating an image before and after an image setting process according to an embodiment of the present invention.
  • 11a is an example before performing image reconstruction on the divided image (eg, an image projected by 360 degree image encoding)
  • 11b is an example after performing image reconstruction on the divided image (eg, , Packed video in 360-degree video encoding). That is, 11a may be understood as an exemplary view after performing the image setting process and 11b before performing the image setting process.
  • the image reconstruction is performed after the image segmentation is performed.
  • the image segmentation may be performed after the image reconstruction is performed according to the encoding / decoding, and may be modified in other cases. Can be.
  • the above-described image reconstruction process (including an inverse process) may be applicable to the same or similar to the reconstruction process of the division unit in the image in the present embodiment.
  • the reconstruction of the image may or may not be performed on every division unit in the image, or may be performed on some division units.
  • the division unit before reconstruction eg, some of P0 through P5
  • the division unit after reconstruction eg some of S0 through S5
  • a case of performing various image reconstruction will be described with an example to be described later.
  • it is assumed that a unit of an image is a picture, a unit of a split image is a tile, and a division unit is a square.
  • some units eg, sps_con
  • whether to perform reconstruction of a division unit in an image may be determined according to a signal (eg, pps_convert_enabled_flag) indicating the reconstruction of the image.
  • whether to perform reconstruction of all division units in an image may be determined according to the signal.
  • a signal may be generated to indicate reconstruction of one image in the image.
  • whether to perform reconstruction of a division unit in an image may be determined according to a signal (eg, tile_convert_flag [i]) indicating a reconstruction of the image.
  • a signal indicating the reconstruction of at least one image (eg, generated by the number of division units) may be generated.
  • whether to perform the reconstruction of the image may be determined according to a signal (for example, pps_convert_enabled_flag) indicating the reconstruction of the image, and the image according to the signal (for example, tile_convert_flag [i]) indicating the reconstruction of the image.
  • Whether or not to perform reconstruction of the subdivision unit may be determined.
  • some signals for example, tile_convert_flag [i]
  • a signal for instructing reconstruction of the plurality of images may be generated.
  • image reconstruction related information may occur.
  • image reconstruction related information a case related to various image reconstruction information will be described.
  • reconstruction information applied to an image may occur.
  • one reconstruction information may be used as reconstruction information of all division units in an image.
  • reconstruction information applied to a division unit in an image may occur.
  • at least one reconstruction information may be used as reconstruction information of some division units in an image. That is, one reconstruction information may be used as reconstruction information of one division unit or one reconstruction information may be used as reconstruction information of a plurality of division units.
  • a signal eg, pps_convert_enabled_flag
  • reconstruction information commonly applied to the division unit in the image
  • a signal eg, pps_convert_enabled_flag
  • reconstruction information that is individually applied to the division unit in the image
  • a signal eg, tile_convert_flag [i]
  • reconstruction information that is individually applied to the division unit in the image may occur.
  • a signal eg, tile_convert_flag [i]
  • reconstruction information commonly applied to the division unit in the image may occur.
  • reconstruction information implicit or explicit processing may be performed according to the encoding / decoding.
  • reconstruction information may be allocated to a preset value according to characteristics, types, etc. of an image.
  • P0 to P5 of 11a may correspond to S0 to S5 of 11b and a reconstruction process may be performed on a divided unit. For example, you can assign to SO without performing reconstruction on P0, assign it to S1 by applying a 90-degree rotation to P1, assign it to S2 by applying a 180-degree rotation to P2, and assign it to P3. It may be assigned to S3 by applying left and right inversion, and may be assigned to S4 by applying left and right inversion after P90 rotation to P4, and may be assigned to S5 by applying left and right inversion after P180 rotation to P5.
  • At least one reconstruction method may be performed without reconstructing the division unit of the image, reconstruction applying rotation, reconstruction applying rotation, and reconstruction applying rotation and inversion.
  • an additional reconstruction process such as rearrangement of the division unit may be performed.
  • the image reconstruction process of the present invention may be configured to include rearrangement of the image in division units in addition to rearrangement of pixels in the image, and may include some syntax elements as shown in Table 4 (eg, part_top, part_left, part_width). , part_height, etc.). This means that image segmentation and image reconstruction may be mixed and understood.
  • Table 4 eg, part_top, part_left, part_width).
  • P0 to P5 of 11a may correspond to S0 to S5 of 11b, and a reconstruction process may be performed on a divided unit. For example, you can assign it to S0 without performing a reconstruction on P0, assign it to S2 without performing a reconstruction on P1, assign it to S1 by applying a 90-degree rotation to P2, and right and left on P3. It can be assigned to S4 by applying inversion, can be assigned to S5 by applying left and right reversal after P90 rotation to P4, and can be assigned to S3 by applying 180 degree rotation after inverting to P5, but not limited to this. Examples of various modifications may also be possible.
  • P_Width and P_Height of FIG. 7 may correspond to P_Width and P_Height of FIG. 11, and P'_Width and P'_Height of FIG. 7 may correspond to P'_Width and P'_Height of FIG. 11.
  • the image size after resizing may be expressed as P'_Width ⁇ P'_Height, (P_Width + Exp_L + Exp_R) ⁇ (P_Height + Exp_T + Exp_B), and in FIG.
  • the image size after resizing is P'_Width ⁇ P'_Height, (P_Width + Var0_L + Var1_L + Var2_L + Var0_R + Var1_R + Var2_R) ⁇ (P_Height + Var0_T + Var1_T + Var0_B + Var1_B) or (Sub_P0_Width + Sub_P1_Width + Var_P__2 + Var_P2_V2_2) Var1_R + Var2_R) ⁇ (Sub_P0_Height + Sub_P1_Height + Var0_T + Var1_T + Var0_B + Var1_B).
  • the image reconstruction may perform rearrangement of pixels in the division unit of the image, perform rearrangement of the division unit in the image, and perform rearrangement of the division unit in the image as well as rearrangement of the pixel in the division unit of the image.
  • the pixel rearrangement of the division unit may be performed, and then the rearrangement of the division unit may be performed, or the rearrangement of the division unit pixel may be performed.
  • Relocation of the division unit in the image may be determined according to a signal indicating reconstruction of the image.
  • a signal may be generated for relocation of division units in an image.
  • the signal may occur when a signal indicating reconstruction of the image is activated.
  • implicit or explicit processing may be performed depending on the encoding / decoding. The implied case may be determined according to the characteristics, types, and the like of the image.
  • the information on the relocation of the segmentation unit in the image may be implicitly or explicitly processed according to the encoding / decoding, and may be determined according to the characteristics, types, etc. of the image. That is, each division unit may be arranged according to the arrangement information of the preset division unit.
  • the following is an example of performing reconstruction of a segmentation unit in an image in an encoding / decoding apparatus according to an embodiment of the present invention.
  • a segmentation process may be performed using segmentation information on an input image before encoding starts.
  • the reconstruction process may be performed using the reconstruction information in the division unit, and the image reconstructed in the division unit may be encoded.
  • the data After encoding is completed, the data may be stored in a memory, and the image encoded data may be stored in a bitstream and transmitted.
  • the partitioning information may be used to perform the partitioning process.
  • the reconstruction process may be performed using the reconstruction information in the division unit, and the image decoded data may be parsed and decoded in the division unit in which the reconstruction is performed.
  • the data may be stored in a memory, and after performing a reconstruction process of the division units, the division units may be merged into one to output an image.
  • P0 to P5 of FIG. 12 correspond to P0 to P5 of FIG. 11, and S0 to S5 of FIG. 12 correspond to S0 to S5 of FIG. 11.
  • the image resizing is performed after the image segmentation is performed.
  • the image segmentation may be performed after the image resizing is performed according to the sub / decoding setting. Modifications may also be possible.
  • the above-described image resizing process (including the reverse process) may be the same as or similar to the resizing process of the division unit in the image in the present embodiment.
  • TL to BR of FIG. 7 may correspond to TL to BR of the division units SX (S0 to S5) of FIG. 12, and S0 and S1 of FIG. 7 may correspond to PX and SX of FIG. 12.
  • P_Width and P_Height of FIG. 7 may correspond to Sub_PX_Width and Sub_PX_Height of FIG. 12
  • P'_Width and P'_Height of FIG. 7 may correspond to Sub_SX_Width and Sub_SX_Height of FIG. 12
  • Exp_L and Exp_R of FIG. Exp_T and Exp_B may correspond to VarX_L, VarX_R, VarX_T, and VarX_B of FIG. 12, and other factors may also correspond.
  • the resizing of the image may or may not be performed on every division unit in the image, or may be performed on some division units.
  • a case of adjusting various image sizes will be described with an example to be described later.
  • the scaling operation is expanded and scaled in such a manner that the offset factor, the scaling direction is set in the up, down, left and right directions, and the scaling direction is operated by the scaling information. It is assumed that a unit of picture is a picture and a unit of a split image is a tile.
  • whether to perform image resizing may be determined in some units (eg, sps_img_resizing_enabled_flag. Or SEI or metadata).
  • whether to perform the resizing of the image may be determined in some units (for example, tile_resizing_flag [i] .i is a split unit index), which is first generated in the unit (tile in this example) or in higher units. It may be possible if activated.
  • whether to adjust the size of the partial image may be implicitly determined according to the encoding / decoding setting, and thus related information may be omitted.
  • whether or not to adjust the size of the division unit in the image may be determined according to a signal (eg, pps_img_resizing_enabled_flag) indicating the size of the image.
  • a signal indicating size adjustment of one image may occur.
  • whether or not to adjust the size of the division unit in the image may be determined according to a signal (eg, tile_resizing_flag [i]) indicating the size of the image.
  • a signal eg, tile_resizing_flag [i]
  • it may be determined whether to adjust the size of some division units in the image according to the signal.
  • a signal eg, generated as many as the number of division units
  • size adjustment of at least one image may be generated.
  • whether or not to perform the resizing of the image may be determined according to a signal indicating the resizing of the image (for example, pps_img_resizing_enabled_flag), and a signal indicating the resizing of the image (for example, tile_resizing_flag [i]).
  • a signal indicating the resizing of the image for example, pps_img_resizing_enabled_flag
  • tile_resizing_flag [i] indicating the resizing of the image.
  • resizing information applied to an image may occur.
  • one scaling information or a set of scaling information may be used as scaling information of all division units in an image.
  • one scaling information commonly applied to the up, down, left, and right directions of a division unit in an image or a scaling value applied to all of the scaling directions supported or allowed in the division unit.
  • 1 information or one set of resizing information or, as many as the number of resizing directions supported or allowed in a division unit. Up to four pieces of information in this example) are applied to the up, down, left, and right directions, respectively. May occur.
  • resizing information applied to a division unit in an image may occur.
  • at least one of the scaling information or the scaling information set may be used as the scaling information of some division unit in the image. That is, one scale information or a set of scale information may be used as scale information of one division unit or may be used as scale information of a plurality of division units.
  • one scaling information commonly applied to the up, down, left, and right directions of one division unit in an image, or one scaling information set respectively applied to the up, down, left, and right directions may occur.
  • one size adjustment information commonly applied to the up, down, left, and right directions of the plurality of division units in the image or one set of the adjustment information respectively applied to the up, down, left, and right directions may occur.
  • the configuration of the scale set means scale value information for at least one scale direction.
  • resizing information that is commonly applied to division units in an image may occur.
  • resizing information may be separately applied to division units in an image.
  • An example to be described below may be described through a combination with an example of performing image resizing.
  • scaling information commonly applied to a division unit in the image may occur.
  • a signal (eg, pps_img_resizing_enabled_flag) indicating size adjustment of the image is activated
  • scaling information applied to a division unit in the image may be generated.
  • a signal (eg, tile_resizing_flag [i]) indicating size adjustment of the image is activated
  • resizing information applied to a division unit in the image may be generated.
  • scaling information commonly applied to a division unit in the image may occur.
  • the image resizing direction and the resizing information may be implicitly or explicitly processed according to the encoding / decoding.
  • the resizing information may be allocated to a preset value according to the characteristics, types, etc. of the image.
  • the resizing direction in the resizing process of the present invention described above is at least one of up, down, left, and right directions, and the resizing direction and the resizing information can be explained explicitly or implicitly. That is, some directions may implicitly be pre-determined with a scale value (including zero, ie no adjustment) and some directions may be explicitly assigned a scale value (with zero, ie no adjustment).
  • the scaling direction and the scaling information may be set to be implicitly or explicitly processed, and may be applied to the division unit in the image.
  • a setting may be applied to one division unit in the image (occurring by division units in this example) or a setting may be applied to a plurality of division units in the image or may be applied to all division units in the image. May be generated (one setting in this example), and at least one setting may be generated in the image (for example, one setting may be generated by the number of division units).
  • One setting set may be defined by collecting setting information applied to the division unit in the image.
  • FIG. 13 is an exemplary diagram of a size adjustment or setting set of a division unit in an image.
  • P0 is up and left directions a2 and a0
  • P1 is up direction a2
  • P2 is up and right directions a2 and a1
  • P3 is down and left directions a3 and a0
  • P4 may be adjusted in the downward direction a3
  • P5 may be adjusted in the downward direction and the right directions a3 and a1, and may not be adjustable in other directions.
  • some directions of the division unit can be explicitly processed for resizing, and some directions of the division unit (left and right in this example) are boundary of the image. If it matches, then explicit processing (bold solid lines in this example) and implicit processing if it does not match (thin solid lines in this example) can be done.
  • P0 is up, down, left direction (b2, b3, b0)
  • P1 is up, down direction (b2, b3)
  • P2 is up, down, right direction (b2, b3, b1)
  • P3 is Up, down, left direction (b3, b4, b0)
  • P4 can be adjusted in the up, down, right direction (b3, b4, b1), up, down (b3, b4), P5,
  • the other direction may not be resizable.
  • some directions of the division unit may perform explicit processing of resizing, and some directions of the division unit (up and down in this example) may be defined as boundaries of the image. If it matches (thick solid line in this example), explicit processing can be done, and if it does not match (thin line in this example), implicit processing can be performed.
  • P0 is the up, left, and right directions (c4, c0, c1)
  • P1 is the up, left, and right directions (c4, c1, c2)
  • P2 is the up, left, and right directions (c4, c2, c3).
  • P3 is in the down, left and right directions (c5, c0, c1)
  • P4 is in the down, left and right directions (c5, c1, c2)
  • P5 is in the down, left and right directions (c5, c2, c3). Resizing may be possible and other directions may be impossible.
  • the setting related to the scaling of an image may have various cases.
  • a plurality of setting sets may be supported to explicitly generate setting set selection information, or a preset setting set may be implicitly determined according to a sub / decoding setting (for example, characteristics, types, etc. of an image).
  • FIG. 14 is an exemplary diagram illustrating an image scaling process and a scaling process of segmentation units in an image.
  • an image resizing process and a reverse process may be performed with e and f, and a size reversal process and reverse process may be performed with d and g. That is, the resizing process may be performed on the image, the resizing of the division unit in the image may be performed, and the order of the resizing process may not be fixed. This means that multiple scaling processes may be possible.
  • the image resizing process can be classified into image resizing (or resizing before splitting) and segmentation unit (or resizing after splitting) in the image. Both of the divisions in the image may not be resized, either one of them, or both, depending on the sub / decoding settings (e.g., characteristics, type, etc. of the image). Can be determined accordingly.
  • the resizing of the image may be performed in at least one of the up, down, left, and right directions of the image, and at least one of the division units in the image.
  • the size adjustment of the division unit may be performed, and in this case, the size adjustment may be performed in at least one of the up, down, left, and right directions of the division unit to which the adjustment is performed.
  • the size of the image A before resizing is P_Width ⁇ P_Height
  • the size of the image after the first resizing (or before the second resizing, B) is P'_Width ⁇ P'_Height, second.
  • the size of the image after resizing (or the image after final resizing, C) may be defined as P '' _ Width ⁇ P '' _ Height.
  • the image before resizing refers to an image on which no scaling is performed
  • the image after first resizing refers to an image on which some resizing has been performed
  • the image after second resizing Means an image on which all the resizing is performed.
  • the image B after the primary size adjustment refers to an image in which the size of the division unit in the image is adjusted as shown in FIGS. 13A to 13C
  • the image C after the second size adjustment is shown in FIG. 7A.
  • it may mean an image on which the resizing is performed on the entire first-sized image B, and vice versa.
  • Various modifications may be possible without being limited to the above examples.
  • P'_Width may be obtained through at least one scaling value in a left or right direction that is horizontally resizable with P_Width, and P'_Height is resized vertically with P_Height. It can be obtained through at least one scaling value in the adjustable up or down direction.
  • the scaled value may be a scaled value generated in a division unit.
  • P '' _ Width may be obtained through P'_Width and at least one scaling value in the left or right direction that is horizontally resizable, and P '' _ Height is P '_Height and at least one scaling value of the vertical or vertical direction can be obtained.
  • the scale value may be a scale value generated in the image.
  • the size of the image after the scaling may be obtained through the size of the image before the scaling and at least one scaling value.
  • Information about a data processing method may occur in an area in which an image is resized.
  • a case of various data processing methods will be described through an example to be described later.
  • the data processing method generated in the resizing process may be the same as or similar to that of the resizing process.
  • the data processing method in the process may be described through various combinations in the following case.
  • a data processing method applied to an image may occur.
  • one data processing method or a set of data processing methods may be used as a data processing method of all division units in the image (assuming that all division units are scaled in this example).
  • one data processing method that is commonly applied to the up, down, left, and right directions of a division unit in an image (or a data processing method that is applied to all the resize directions supported or allowed in a division unit, etc.) 1 information) or one set of data processing methods (or, as many as the number of resizing directions supported or allowed in the division unit. Up to four pieces of information in this example) are applied to the up, down, left, and right directions, respectively. May occur.
  • a data processing method applied to a division unit in an image may occur.
  • at least one data processing method or a set of data processing methods may be used as a data processing method of some division units (assuming a division unit to be scaled in this example) in an image. That is, one data processing method or a set of data processing methods may be used as a data processing method of one division unit or may be used as a data processing method of a plurality of division units.
  • one data processing method commonly applied to the up, down, left, and right directions of one division unit in an image, or one set of data processing methods respectively applied to the up, down, left, and right directions may occur. have.
  • one data processing method commonly applied to the up, down, left, and right directions of a plurality of division units in an image, or one data processing method information set respectively applied to the up, down, left, and right directions may occur.
  • the configuration of the data processing method set means a data processing method for at least one scaling direction.
  • a data processing method commonly applied to division units in an image may be used.
  • a data processing method that is individually applied to the division unit in the image may be used.
  • the data processing method may use a preset method.
  • the preset data processing method may include at least one method, which corresponds to an implicit case, and may explicitly generate selection information about the data processing method, which may include encoding / decoding settings (eg, characteristics of an image, Type, etc.).
  • a data processing method commonly applied to the division unit in the image may be used, and a preset method or one of a plurality of data processing methods may be selected.
  • a data processing method that is individually applied to a division unit in an image may be used, and a preset method may be used or one of a plurality of data processing methods may be selected according to the division unit.
  • Some areas TL to BR of some units may store data of some areas tl to br of some units (P0 to P5 of FIGS. 12A to 12F). Can be used to perform resizing.
  • the some units may be the same (for example, S0 and P0) or regions that are not the same (for example, S0 and P1). That is, the area to be resized (TL to BR) can be filled using some data (tl to br) of the corresponding partitioning unit, and the area to be resized can be filled using some data of a partitioning unit different from the partitioning unit. have.
  • the area TL to BR of the current division unit may be resized using tl to br data of the current division unit.
  • S0 TL is P0 tl data
  • S1 RC is P1 tr + rc + br data
  • S2 BL + BC is P2 bl + bc + br data
  • S3 TL + LC + BL can be filled using the data of tl + lc + bl of P3.
  • the area TL to BR of the current division unit may be resized using tl ⁇ br data of the division unit spatially adjacent to the current division unit.
  • TL + TC + TR of S4 gives bl + bc + br data of P1 in the upward direction
  • BL + BC of S2 gives tl + tc + tr data of P5 in the downward direction
  • LC + BL of S2 The tl + rc + bl data of P1 in the left direction, the tl + lc + bl data of P4 in the right direction, and the tl + lc + bl data in P in the left direction can be filled using tl + rc + bl data in the left direction.
  • the area TL to BR of the current division unit may be resized using tl ⁇ br data of the division unit that is not spatially adjacent to the current division unit.
  • data of both boundary regions (eg, left, right, up and down) of an image may be acquired.
  • S3 LC is obtained using S5 tr + rc + br data
  • S2 RC uses S0 tl + lc data
  • S4 BC uses S1 tc + tr data
  • S1 TC uses S4 bc data can do.
  • BC of S1 can be obtained using tl + lc + bl data of S3, RC of S3 and tl + tc data of S1, and RC of S5 using bc data of S0.
  • Some areas TL to BR of some units may be used to restore or correct some areas tl to br of some units P0 to P5.
  • the some units may be the same (for example, S0 and P0) or regions that are not the same (for example, S0 and P2). That is, the resized area may be used and removed for restoring some data of the divided unit, and the resized area may be used and removed for restoring some data of another partitioned unit.
  • the detailed example is omitted since it can be derived inversely from the expansion process.
  • the above example may be applied when there is highly correlated data in a region to be resized, and information about a position referred to the resizing may be explicitly generated or may be obtained implicitly according to a preset rule. You can mix this to see related information. This may be an example applied when data is acquired from another region where continuity exists in encoding of a 360 degree image.
  • the following is an example of adjusting the size of a division unit in an image in an encoding / decoding apparatus according to an embodiment of the present invention.
  • a segmentation process may be performed on the input image before encoding starts.
  • the resizing process may be performed using the resizing information for each division unit, and the image may be encoded after the resizing unit is resized.
  • the data may be stored in a memory, and the image encoded data may be stored in a bitstream and transmitted.
  • the partitioning information may be used to perform the partitioning process.
  • the resizing process may be performed using the resizing information in the division unit, and the image decoded data may be parsed and decoded in the division unit in which the resizing is performed.
  • the data may be stored in the memory, and after performing the reverse process of resizing the division units, the division units may be merged into one to output an image.
  • Image reconstruction may be performed after image resizing is performed or image resizing may be performed after image reconstruction is performed.
  • image segmentation, image reconstruction, and image resizing may be possible.
  • image resizing and image reconstruction may be performed, and the order of image setting is not fixed but may be changed, which may be determined according to the encoding / decoding setting.
  • the image setting process will be described for the case in which image reconstruction is performed after image segmentation is performed and image size adjustment is performed.
  • other procedures may be possible and may be changed in other cases according to the encoding / decoding setting.
  • Combinations with image settings may also be possible.
  • the image setting process may be performed sequentially, but all or part of the setting process may be simultaneously performed.
  • a plurality of processes may be performed in some image setting processes according to encoding / decoding settings (for example, characteristics and types of images). The following is an example of various combinations of the image setting process.
  • P0 to P5 of FIG. 11A may correspond to S0 to S5 of FIG. 11B, and the reconstruction process (relocation of pixels in this example) and the resizing process (same size adjustment to the division unit in this example) may be performed in each division unit. Can be performed.
  • scaling may be applied to S0 to S5 by applying a scaling through an offset to P0 to P5.
  • S0 without reconstruction at P0
  • S1 by applying a 90 degree rotation to P1
  • S2 by applying a 180 degree rotation to P2
  • S3 can be assigned to S4 by applying left and right inversion to P4
  • S5 by applying up and down reversal to P5.
  • P0 to P5 of FIG. 11A may correspond to the same or different positions S0 to S5 of FIG. 11B, and reconstruct the division unit (relocation of the pixel and the division unit in this example), and the scaling process (in this example). The same scaling may be performed for each division unit.
  • scaling through scale may be applied to P0 to P5 and allocated to S0 to S5. It can also be assigned to S0 without reconstruction at P0, to S2 without reconstruction at P1, to S1 by applying a 90 degree rotation to P2, and to left and right inversion at P3. It can be assigned to S4 by applying the left and right inversion after P90 rotation to P4 can be assigned to S5, and can be assigned to S3 by applying a 180-degree rotation after P90 to the P5.
  • P0 to P5 of FIG. 11A may correspond to E0 to E5 of FIG. 5E, and the reconstruction process (relocation of pixels and division units in this example) and the resizing process (not identical to the division units in this example). Unscaling) may be performed. For example, you can assign to P0 without scaling and reconstruction at P0, scale with P1 at scale, assign to E1 without reconstruction, and resize at P2. Can be assigned to E2 by performing a reconstruction without performing resizing on P3 and assigning to E4 without performing a reconstruction, and performing a reconstruction without performing resizing on P4 and assigning to E5. Can be scaled through the offset to P5 and reconstructed to assign to E3.
  • the absolute position or the relative position in the image of the division unit before and after the image setting process may be maintained or changed, which may be determined according to the sub / decoding setting (for example, characteristics, types, etc. of the image).
  • the sub / decoding setting for example, characteristics, types, etc. of the image.
  • a combination of various image setting processes may be possible, and modifications to various examples may be possible without being limited to the above examples.
  • the generated information will be recorded in the bitstream in at least one unit of a sequence, a picture, a slice, a tile, etc. in the encoder, and the decoder will parse related information from the bitstream. It may also be included in the bitstream in the form of SEI or metadata.
  • syntax elements related to a plurality of image settings The following shows an example of syntax elements related to a plurality of image settings.
  • the description will focus on additional syntax elements.
  • the syntax elements of the example described below are not limited to any specific unit and may be syntax elements supported in various units such as a sequence, a picture, a slice, a tile, and the like. Alternatively, it may be a syntax element included in SEI, metadata, and the like.
  • parts_enabled_flag means a syntax element for splitting some units.
  • num_partitions means a syntax element for the number of partition units, and a value of 1 plus the number of partition units.
  • part_top [i] and part_left [i] mean syntax elements for position information of a division unit, and mean a horizontal and vertical start position of the division unit (eg, an upper left position of the division unit).
  • part_width [i] and part_height [i] mean syntax elements for the size information of the partition unit, and indicate the width and length of the partition unit.
  • the start position and the size information may be set in a pixel unit or a block unit.
  • the syntax element may be a syntax element that may occur in the image reconstruction process or may be a syntax element that may occur when the image segmentation process and the image reconstruction process are mixed.
  • part_header_enabled_flag means a syntax element of whether to support the part / decryption setting in the partition unit.
  • the above example is not limited thereto as an example of syntax elements related to size adjustment and reconstruction in the division unit of the image setting to be described later, and other division units and settings of the present invention may be changed and applied.
  • This example is described under the assumption that the resizing and reconstruction is performed after the division is performed, but the present invention is not limited thereto and may be changed and applied according to another image setting order.
  • the types of syntax elements, syntax element order, and conditions supported in the following examples are limited only to the present example and may be changed and determined according to encoding / decoding settings.
  • Table 5 shows an example of syntax elements related to reconstruction in division units during image setting.
  • part_convert_flag [i] means a syntax element for reconstructing the partition unit.
  • convert_type_flag [i] means mode information on reconstruction of the division unit and may be information on relocation of pixels.
  • syntax elements may occur for additional reconstructions such as split unit relocation.
  • relocation of the partition unit may be performed through part_top and part_left, which are syntax elements related to the above-described image segmentation, or syntax elements (for example, index information, etc.) related to partition unit relocation may occur.
  • Table 6 shows an example of syntax elements related to size adjustment in division units during image setting.
  • part_resizing_flag [i] means a syntax element for adjusting image size in division units.
  • width_scale [i] and height_scale [i] mean scale factors related to horizontal scaling and vertical scaling in scaling through scale factors in subdivision units.
  • top_height_offset [i] and bottom_height_offset [i] are the top and bottom offset factors associated with scaling in the offset unit through the offset factor
  • left_width_offset [i] and right_width_offset [i] are the offset factor in the division unit.
  • resizing_type_flag [i] [j] means a syntax element for a data processing method of a region to be resized in a division unit.
  • the syntax element refers to a method of processing data that is individual in the direction of resizing.
  • syntax elements may be generated for individual data processing methods in areas that are scaled in the up, down, left, and right directions. This may be generated based on the resizing information (e.g., can only occur when resized in some direction).
  • the above-described process of image setting may be a process applied according to characteristics, types, and the like of the image.
  • the above-described image setting process may be applied in the same manner or may be changed.
  • the description will be mainly focused on the case where additional or change is applied in the above example.
  • a video generated through a 360 degree camera has a different characteristic from that obtained through a general camera. Has a different encoding environment.
  • a 360-degree image has no boundary portion having discontinuous characteristics and data of all regions may have continuity.
  • an image may be reproduced in front of an eye through a lens to request a high quality image, and when the image is acquired through a stereoscopic camera, processed image data may increase.
  • various image setting processes considering a 360 degree image may be performed.
  • the 360 degree camera may be a camera having a plurality of cameras or a plurality of lenses and sensors, and the camera or lens may cover all directions around the arbitrary center point captured by the camera.
  • the 360 degree image may be encoded using various methods. For example, encoding may be performed using various image processing algorithms in three-dimensional space, or encoding may be performed using various image processing algorithms by converting to two-dimensional space. In the present invention, a method of performing encoding / decoding by converting a 360 degree image into a two-dimensional space will be described.
  • 360 degree image encoding apparatus may be configured to include all or part of the configuration according to Figure 1, pre-processing for performing the pre-processing (Stitching, Projection, Region-wise Packing) for the input image It may further comprise a wealth.
  • the 360-degree image decoding apparatus may include all or part of the configuration according to Figure 2, the post-processing unit for performing the post-processing (Rendering) before being decoded and reproduced as the output image It may further include.
  • encoding may be performed to transmit a bitstream.
  • the parser parses the bitstream transmitted from the decoder to perform decoding.
  • the output image can be generated.
  • the bitstream may include information generated during the preprocessing process and information generated during the encoding process, and may be transmitted.
  • the decoder may parse and use the decoder in the decoding process and the postprocessing process.
  • the input image may be stitched and projected to a three-dimensional projection structure in a sphere unit, and the image data on the three-dimensional projection structure is converted into a two-dimensional image through the above process. Can be projected.
  • the projected image may include all or a part of 360 degree contents according to the encoding setting.
  • the position information of the region (or the pixel) to be disposed in the center of the projected image may be generated by an implicitly preset value or may be explicitly generated.
  • the range and position information of the included region may be generated.
  • range information eg, width and length
  • location information eg, measured based on an upper left side of the image
  • ROI region of interest
  • the 360-degree image can view all contents in the up, down, left, and right directions, but the user's gaze may be limited to a part of the image, and may be set as the ROI in consideration of this.
  • the region of interest may be set to have good quality and resolution, and other regions may be set to have lower quality and resolution than the region of interest.
  • the single stream transmission method (Single Stream) of the 360-degree image transmission method may transmit the entire image or viewport image as a separate single bitstream for the user.
  • the entire video having different video quality is transmitted as a multi-bit stream so that the video quality can be selected according to the user environment and the communication situation.
  • a tile may be selected according to a user environment and a communication situation by transmitting a partially encoded partial image of a tile unit in a multiple bit stream.
  • the 360 degree image encoder generates and transmits a bitstream having two or more qualities
  • the 360 degree image decoder may set a region of interest according to the user's gaze and selectively decode according to the region of interest. That is, the head tracking or eye tracking system may set the place where the user's gaze stays as the region of interest and render only the necessary parts.
  • the projected image may be converted into a packed image by performing a region-wise packing process.
  • the regional pavement process may include dividing the projected image into a plurality of regions, wherein each divided region may be disposed (or rearranged) in the packaged image according to the regional pavement setting.
  • the regional packaging may be performed for the purpose of increasing spatial continuity when converting a 360 degree image into a 2D image (or a projected image), and may reduce the size of the image through the regional packaging. In addition, it may be performed for the purpose of reducing image quality deterioration occurring during rendering, enabling viewport-based projection, and providing other types of projection formats.
  • Regional packing may or may not be performed according to encoding settings, and may be performed based on a signal indicating whether or not to be performed (for example, regionwise_packing_flag, and in the following example, region-specific packing information may occur only when regionwise_packing_flag is activated). Can be determined.
  • setting information or mapping information or the like in which some regions of the projected image are allocated (or arranged) to some regions of the wrapped image may be displayed (or generated).
  • the projected image and the packaged image may be the same image.
  • stitching, projection, and regional wrapping were defined as individual processes, but some (eg, stitching + projection, projection + regional wrapping) or all (eg stitching + projection + regional wrapping) It can be defined as the process of.
  • At least one packaged image may be generated with respect to the same input image according to the setting of the stitching, the projection, and the wrapping process for each region.
  • at least one encoded data of the same projected image may be generated according to the setting of the packing process for each region.
  • the packed image may be divided by performing a tiling process.
  • tiling is a process of dividing and transmitting an image into a plurality of regions, which may be an example of the 360 degree image transmission method.
  • tiling may be performed for the purpose of partial decoding in consideration of a user environment and the like, and may be performed for the efficient processing of massive data of a 360 degree image.
  • all of the images may be decoded for decoding the ROI.
  • it may be efficient to decode only the ROI.
  • the division may be divided into tiles, which are division units according to an existing coding scheme, or may be divided into various division units (square division, block, etc.) described in the present invention.
  • the division unit may be a unit that performs independent encoding / decoding. Tiling may be performed based on the projected image or the packed image or may be performed independently. That is, the screen may be divided based on the surface boundary of the projected image, the surface boundary of the packaged image, the pavement setting, and the like, and may be divided independently for each division unit. This may affect the generation of the partition information during the tiling process.
  • the projected image or the packed image may be encoded.
  • the encoded data and the information generated during the preprocessing may be included in the bitstream and transmitted to the 360 degree image decoder.
  • Information generated during the preprocessing may be included in the bitstream in the form of SEI or metadata.
  • the bitstream may include at least one piece of encoded data and at least one preprocessing information different from one part of the encoding process or some part of the preprocessing.
  • This may be an object for constructing a decoded image by mixing a plurality of encoded data (encoded data + preprocessing information) according to a user environment in a decoder.
  • a decoded video can be constructed by selectively combining a plurality of encoded data.
  • the process can be performed in two separate for application in the binocular system, the process can be performed for an additional depth image.
  • 15 is an exemplary diagram illustrating a three-dimensional space and a two-dimensional planar space representing a three-dimensional image.
  • 3DoF Degree of Freedom
  • 3DoF Degree of Freedom
  • 3DoF means the degree of freedom in space
  • 3DoF means the degree of freedom including rotation around the X, Y, and Z axes, such as 15a
  • 6DoF permits 3DoF additional movement along the X, Y, and Z axes. It means freedom.
  • the video encoding apparatus and the decoding apparatus of the present invention will be mainly described with respect to the case of 3DoF, and in the case of supporting 3DoF or more (3DoF +), it may be combined or changed with an additional process or apparatus not shown in the present invention.
  • Yaw ranges from - ⁇ (-180 degrees) to ⁇ (180 degrees)
  • Pitch ranges from - ⁇ / 2 rad (or -90 degrees) to ⁇ / 2 rad (or 90 degrees).
  • the Roll can range from - ⁇ / 2 rad (or -90 degrees) to ⁇ / 2 rad (or 90 degrees).
  • ⁇ is the longitude and latitude in the map representation of the earth
  • (x, y, z) in the three-dimensional space is the ( , ⁇ ).
  • the coordinates of the three-dimensional space can be derived from the two-dimensional space coordinates according to the conversion equation of).
  • ( , ⁇ ) can be converted to (x, y, z).
  • x, y, z tan -1 (-Z / X)
  • ⁇ sin - 1 (Y / (X 2 + Y 2 + Z 2 ) 1/2 ) to derive two-dimensional space coordinates from three-dimensional space coordinates have.
  • the pixels of the 3D space When the pixels of the 3D space are correctly converted to the 2D space (for example, integer unit pixels of the 2D space), the pixels of the 3D space may be mapped to the pixels of the 2D space.
  • the 2D pixels When the pixels of the 3D space are not correctly converted to the 2D space (for example, the decimal unit pixels of the 2D space), the 2D pixels may be mapped to the pixels obtained by performing interpolation.
  • the used interpolation may include Nearest neighbor interpolation, Bi-linear interpolation, B-spline interpolation, Bi-cubic interpolation, or the like.
  • the relevant information may be explicitly generated by selecting one of the plurality of interpolation candidates, or an interpolation method may be implicitly determined according to a preset rule.
  • an interpolation filter preset according to a 3D model, a projection format, a color format, and a slice / tile type may be used.
  • information about filter information eg, filter coefficients
  • ( ,) may be sampled (i, j) based on the size of the image (horizontal length, vertical length), i may range from 0 to P_Width-1, and j may range from 0 to P_Height-1.
  • the center point can be set in three-dimensional space, and the position information on the center point can be explicitly created or set to an implicitly preset value. For example, center position information in Yaw, center position information in Pitch, center position information in Roll, and the like can be generated. If a value for the information is not specified separately, each value may be assumed to be zero.
  • a partial region of the 360-degree image may be targeted, and position information (eg, Some locations belonging to this location (in this example, location information on the center point), range information, and the like may be explicitly generated or implicitly pre-set location and range information may be followed.
  • position information eg, Some locations belonging to this location (in this example, location information on the center point), range information, and the like may be explicitly generated or implicitly pre-set location and range information may be followed.
  • center position information in Yaw, center position information in Pitch, center position information in Roll, range information in Yaw, range information in Pitch, range information in Roll, etc. can be generated.
  • an area it may be at least one area, and accordingly, location information and range information of a plurality of areas may be processed. If a value for the information is not specified, it may be assumed to be the entire 360 degree image.
  • H0 to H6 and W0 to W5 in 15a each represent some latitude and longitude at 15b, and the coordinates of 15b can be expressed as (C, j), (i, C) (C is the longitude or latitude component). ).
  • C the longitude or latitude component.
  • the position information (for example, the x, y component or a range defined by x and y) is used to determine the sub / decoding information. It may be included as an example.
  • the image acquired by the 360 degree camera may be converted into a two-dimensional space.
  • a 360-degree image may be mapped using the 3D model, and various 3D models such as a sphere, a cube, a cylinder, a pyramid, and a polyhedron may be used.
  • a projection process according to the projection format based on the model may be performed.
  • 16A to 16D are conceptual views illustrating a projection format according to an embodiment of the present invention.
  • 16A illustrates an ERP (Equi-Rectangular Projection) format in which a 360 degree image is projected onto a two-dimensional plane.
  • 16B illustrates a CMP (CubeMap Projection) format in which a 360 degree image is projected onto a cube.
  • FIG. 16C illustrates an OHP (OctaHedron Projection) format in which a 360 degree image is projected on an octahedron.
  • 16D illustrates an ISP (IcoSahedral Projection) format in which a 360 degree image is projected on a polyhedron.
  • the present invention is not limited thereto, and various projection formats may be used. The left side of FIGS.
  • 16A to 16D shows an example of a 3D model, and the right side converts to a 2D space through a projection process.
  • the projection format it may have various sizes and shapes, and each shape may be formed of faces or faces, and the surfaces may be represented by circles, triangles, squares, and the like.
  • the projection format may be defined by the three-dimensional model, the setting of the surface (for example, the number of surfaces, the shape of the surface, the configuration of the shape of the surface, etc.), the setting of the projection process, and the like. If at least one of the above definitions is different, it can be viewed in a different projection format.
  • ERP consists of a sphere model (three-dimensional model), one surface (number of surfaces), and a rectangular surface (surface shape), but some of the settings in the projection process (for example, three-dimensional space). Equations used when converting from to two-dimensional space, etc.
  • the remaining projection settings are the same, and elements that make the difference in at least one pixel of the projected image during the projection process are different and are classified into different formats such as ERP1 and EPR2.
  • CMP consists of a cube model, six surfaces, and a rectangular surface, but CMP1, if some of the settings in the projection process (for example, the sampling method when converting from three-dimensional space to two-dimensional) are different. Can be classified into other formats such as CMP2.
  • projection format identification information (or projection format information) may be explicitly generated.
  • the projection format identification information may be configured through various methods.
  • the projection format may be identified by assigning index information (for example, proj_format_flag) to the plurality of projection formats. For example, 0 for ERP, 1 for CMP, 2 for OHP, 3 for ISP, 4 for ERP1, 5 for CMP1, 6 for OHP1, 7 for ISP1, 8 for CMP compact, You can assign 9 times for OHP compact, 10 times for ISP compact, and 11 times for other formats.
  • index information for example, proj_format_flag
  • the projection format may be identified by at least one element information constituting the projection format.
  • the element information constituting the projection format may include three-dimensional model information (for example, 3d_model_flag. 0 is a sphere, 1 is a cube, 2 is a cylinder, 3 is a pyramid, 4 is a polyhedron 1, 5 is a polyhedron 2), Information about the number of surfaces (for example, starting from num_face_flag. 1 and increasing in increments of 1 or by assigning the number of surfaces generated in the projection format as index information, 0 is 1, 1 is 3, 2 is 6, 3 is 8, 4 is 20, etc., surface shape information (e.g. shape_face_flag.
  • projection Process setting information (eg, 3d_2d_convert_idx, etc.) may be included.
  • the projection format may be identified by projection format index information and element information constituting the projection format.
  • projection format index information can be assigned 0 for ERP, 1 for CMP, 2 for OHP, 3 for ISP, and 4 for other formats.
  • the projection format eg, ERP, ERP1, CMP, CMP1, OHP, OHP1, ISP, ISP1, etc.
  • the projection format may be identified together with the projection process setting information.
  • the projection format for example, ERP, CMP, CMP compact, OHP, OHP compact, ISP, ISP compact, etc.
  • the element information constituting the projection format may be identified together with the element information constituting the projection format (whether by region or not).
  • the projection format may be identified by projection format index information, may be identified by at least one projection format element information, and may be identified by projection format index information and at least one projection format element information.
  • This may be defined according to the encoding / decoding setting, and the present invention will be described on the assumption that it is identified by the projection format index.
  • a description will be given of a case of a projection format represented by surfaces having the same size and shape, but a configuration in which the size and shape of each surface is not the same may be possible.
  • the configuration of each surface may be the same as or different from those of Figs. 16A to 16D, and the number of each surface is used as a symbol for identifying each surface and is not limited to a specific order.
  • ERP is one surface + rectangle
  • CMP is six surfaces + rectangle
  • OHP is eight surfaces + triangle
  • ISP is 20 surfaces + triangle.
  • the projection format may be divided into one surface (eg, ERP) or a plurality of surfaces (eg, CMP, OHP, ISP, etc.).
  • each surface may be divided into a square and a triangle.
  • the division may be an example of the type, characteristics, and the like of the image in the present invention, which may be applied when the encoding / decoding setting according to the projection format is different.
  • the type of image may be a 360 degree image
  • the characteristic of the image may be one of the above-described divisions (for example, each projection format, a projection format of one surface or a plurality of surfaces, a projection format in which the surface is rectangular or not), and the like. Can be.
  • the two-dimensional plane coordinate system (for example, (i, j)) may be defined on each surface of the two-dimensional projection image, and the characteristics of the coordinate system may vary depending on the projection format and the position of each surface.
  • one two-dimensional plane coordinate system, and other projection formats may have a plurality of two-dimensional plane coordinate systems according to the number of surfaces.
  • the coordinate system may be represented by (k, i, j), where k may be index information of each surface.
  • FIG. 17 is a conceptual view of implementing a projection format included in a rectangular image according to an embodiment of the present invention.
  • 17a to 17c may be understood to implement the projection format of FIGS. 16b to 16d as a rectangular image.
  • each image format may be configured in a rectangular form for encoding / decoding of a 360 degree image.
  • one coordinate system may be used as it is, but in the case of other projection formats, the coordinate system of each surface may be integrated into one coordinate system, and a detailed description thereof will be omitted.
  • an area filled with meaningless data such as a blank or a background occurs in the process of constructing a rectangular image. That is, it may be composed of an area including actual data (surface in this example, an active area) and an insignificant area (in this example, an inactive area) filled to form a rectangular image. This may be degraded due to an increase in the amount of encoded data due to the increase in the size of the image due to the meaningless region as well as the encoding / decoding of the actual image data.
  • a process for constituting an image with an area including real data without excluding a meaningless area may be additionally performed.
  • FIG. 18 is a conceptual diagram of a method of converting a projection format into a rectangular shape according to an embodiment of the present invention, and repositioning the surface to exclude a meaningless area.
  • this process may be defined as a regional packaging process (CMP compact, OHP compact, ISP compact, etc.).
  • CMP compact, OHP compact, ISP compact, etc. may be divided and rearranged (OHP compact, ISP compact, etc.).
  • This may be performed for the purpose of improving encoding performance through not only removing meaningless regions but also efficient placement of surfaces. For example, when the arrangement of images is continuous between surfaces (for example, B2-B3-B1, B5-B0-B4, etc. in 18a), encoding performance may be improved by improving prediction accuracy at the time of encoding.
  • the regional packaging according to the projection format is only an example in the present invention and is not limited thereto.
  • FIG. 19 is a conceptual view illustrating a wrapping process for each region in a rectangular image of a CMP projection format according to an embodiment of the present invention.
  • the CMP projection format may be arranged as 6 ⁇ 1, 3 ⁇ 2, 2 ⁇ 3, 1 ⁇ 6.
  • it may be arranged as 19d to 19e.
  • CMP is taken as an example, but may be applied to other projection formats without being limited to CMP.
  • the surface layout of the image acquired through the regional packaging may conform to a predetermined rule according to the projection format or explicitly generate information about the layout.
  • the 360-degree image encoding / decoding apparatus may include all or a part of the image encoding / decoding apparatus according to FIGS. 1 and 2, and in particular, a format conversion unit for converting and inversely converting a projection format. And a format inverse transform unit may be further included in the image encoding apparatus and the image decoding apparatus, respectively. That is, in the image encoding apparatus of FIG. 1, the input image may be encoded through a format converter, and after the bitstream is decoded in the image decoding apparatus of FIG. 2, an output image may be generated through the inverse format converter.
  • the input image will be described on the premise that the input image is the same image as the two-dimensional projection image or the package image obtained by performing the preprocessing process in the 360-degree encoding apparatus. That is, the input image may be an image obtained by performing a projection process or a region-specific packaging process according to some projection formats.
  • the projection format previously applied to the input image may be one of various projection formats, may be regarded as a common format, and may be referred to as a first format.
  • the format conversion unit may perform conversion to a projection format other than the first format.
  • a format to be converted may be referred to as a second format.
  • ERP may be set to a first format and conversion may be performed to a second format (eg, ERP2, CMP, OHP, ISP, etc.).
  • the ERP2 may have the same three-dimensional model, the configuration of the surface, etc., but may have a different EPR format with some settings.
  • some of the image setting processes described below may be applied.
  • the first format and the second format may be one of various projection formats, and the present invention is not limited to the above example and may be changed to another case.
  • interpolation may be performed because pixels (integer pixels) of the converted image may be obtained not only from integer units but also from decimal units in the converted image due to different coordinate system characteristics between projection formats.
  • the interpolation filter used may use the same or similar filters as described above.
  • the interpolation filter may select one of the plurality of interpolation filter candidates to explicitly generate related information, or may be implicitly determined according to a preset rule. For example, a preset interpolation filter may be used according to a projection format, a color format, a slice / tile type, and the like.
  • information on filter information eg, filter coefficients
  • the projection format may be defined including a region-specific package. That is, the projection process and the regional packaging process may be performed in the format conversion process. Alternatively, after the format conversion unit, a process such as packing by region may be performed before encoding.
  • the generated information will be recorded in the bitstream in at least one unit of a sequence, a picture, a slice, a tile, etc. in the encoder, and the decoder will parse related information from the bitstream. It may also be included in the bitstream in the form of SEI or metadata.
  • the image setting process according to the present invention may be applied to a preprocessing process, a postprocessing process, a format conversion process, a format inverse conversion process, and the like in a 360 degree image encoding / decoding device as well as a general encoding / decoding process.
  • the image setting process to be described later will be described based on the 360 degree image encoding apparatus, and can be described including the contents of the image setting described above. Duplicate explanations in the above-described image setting process will be omitted.
  • an example to be described later will be described based on an image setting process, and an image setting reverse process may be induced in the image setting process in reverse, and in some cases, it may be confirmed through various embodiments of the present invention described above.
  • the image setting process in the present invention may be performed in the projection step of the 360-degree image, may be performed in the region-specific packaging step, may be performed in the format conversion step, may be performed in other steps.
  • FIG. 20 is a conceptual diagram for 360-degree image segmentation according to an embodiment of the present invention.
  • FIG. 20 a case of an image projected by ERP will be described.
  • 20a represents an image projected by the ERP, and division may be performed using various methods.
  • the description will be focused on slices and tiles, and it is assumed that W0 to W2, H0, and H1 are divided boundary lines of slices or tiles and follow raster scan order.
  • An example to be described below will be described based on slices and tiles.
  • the present invention is not limited thereto and other division methods may be applied.
  • division may be performed in units of slices and may have division boundaries of H0 and H1.
  • division may be performed in units of tiles, and may have division boundaries of W0 to W2 and H0 and H1.
  • FIG. 20b shows an example of dividing the image projected by the ERP into tiles (assuming the same tile segmentation boundary (W0 to W2, H0, and H1 are all active) as shown in FIG. 20A).
  • the P area is the entire image
  • the V area is the area or viewport where the user's eyes stay
  • various methods may exist to provide an image corresponding to the viewport.
  • an area corresponding to the viewport may be obtained by decoding the entire image (for example, tiles a to l).
  • the entire image may be decoded, and when partitioned, tiles a to l (in this example, A + B region) may be decoded.
  • an area corresponding to the viewport may be obtained by decoding the area belonging to the viewport.
  • the tiles f, g, j, and k may be decoded to obtain an area corresponding to the viewport in the reconstructed image.
  • the former may be referred to as full decoding (or Viewport Independent Coding), and the latter may be referred to as partial decoding (or Viewport Dependent Coding).
  • the latter case may be an example that may occur in a 360-degree image having a large amount of data, and a tile-based segmentation method may be frequently used rather than a slice-based segment in that a segmentation region may be flexibly obtained.
  • the referenceability of the partition unit may be limited spatially or temporally (implicitly in this example), and encoding / decoding may be performed to take this into account.
  • An example to be described later will be described with reference to the case of full decoding, but for the purpose of preparing for the case of partial decoding, a 360-degree image division will be described based on a tile (or a rectangular division method of the present invention). The contents of the example may be the same or changed to other division units.
  • FIG. 21 is an exemplary diagram of 360 degree image segmentation and image reconstruction according to an embodiment of the present invention.
  • FIG. 21 a case of an image projected by CMP will be described.
  • 21a represents an image projected by CMP, and division may be performed using various methods. Assume that W0 to W2, H0, and H1 are the dividing boundaries of the surface, slice, and tile and follow the raster scan order.
  • division may be performed in units of slices and may have division boundaries of H0 and H1.
  • division may be performed in units of tiles, and may have division boundaries of W0 to W2 and H0 and H1.
  • the division may be performed by the surface unit, and may have a division boundary of W0 to W2 and H0 and H1. In this example, it is assumed that the surface is part of the division unit.
  • the surface is used to classify or classify regions having different properties (for example, planar coordinate systems of each surface) in the same image according to the characteristics, types (360-degree image, projection format, etc.) of the image. It may be a division unit performed (dependent sub / decoding in this example), and a slice or tile may be a division unit (independent example / decoding in this example) performed for the purpose of segmenting an image according to a user definition. .
  • the surface may be a unit divided by a predetermined definition (or derived from the projection format information) in the projection process according to the projection format, the slice, the tile explicitly generates the segmentation information according to the user definition and the divided unit Can be.
  • the surface may have a partitioned shape in a polygonal shape including a quadrangle depending on the projection format
  • the slice may have any partitioned shape that cannot be defined as a rectangle or a polygon
  • the tile may have a partitioned shape of a rectangle.
  • Setting of the division unit may be limited and defined for the purpose of explanation of the present example.
  • the surface has been described as a division unit classified for the purpose of area division, but may be a unit for performing independent sub / decoding to at least one surface unit according to the sub / decoding setting, or in combination with a tile or a slice. It can have a setting to perform independent encoding / decoding.
  • explicit information about the tile and the slice may be generated when the tile and the slice are combined, or an implicit case where the tile and the slice are combined based on the surface information may occur.
  • explicit information about tiles and slices may be generated based on surface information.
  • image segmentation process (surface in this example) is performed, and image segmentation may implicitly omit segmentation information (obtaining segmentation information from projection format information).
  • image segmentation may implicitly omit segmentation information (obtaining segmentation information from projection format information).
  • This example is an example of a dependent encoding / decoding setup, and may be an example in which referenceability between surface units is not limited.
  • one image segmentation process (surface in this example) is performed, and the image segmentation may explicitly generate segmentation information.
  • This example is an example of a dependent encoding / decoding setup, and may be an example in which referenceability between surface units is not limited.
  • a plurality of image segmentation processes are performed, and some image segmentation (surfaces in this example) may implicitly omit or explicitly generate segmentation information, and partially segment the images. (Tile in this example) may explicitly generate split information.
  • some image segmentation process surface in this example precedes some image segmentation process (tile in this example).
  • a plurality of image segmentation processes are performed, and some image segmentation (surfaces in this example) may implicitly omit or explicitly generate segmentation information, and some image segmentation (tiles in this example) may be partially Segmentation information can be explicitly generated based on image segmentation (surface in this example).
  • some image segmentation process surface in this example
  • some image segmentation process precedes some image segmentation process (tile in this example).
  • the split information is explicitly generated, but there may be a difference in the split information configuration.
  • a plurality of image segmentation processes are performed, and some image segmentation (surface in this example) may implicitly omit segmentation information, and some image segmentation (tile in this example) is part of image segmentation (this example).
  • Splitting information can be omitted implicitly, based on the For example, individual surface units can be set in tile units, or multiple surface units (in this example, tied if neighboring surfaces are continuous, otherwise unbundled. B2-B3-B1 and B4 at 18a) -B0-B5) may be set in units of tiles. It is possible to set the surface unit to tile unit according to the preset rule.
  • This example is an example of independent encoding / decoding, and may be an example in which referenceability between surface units is limited. That is, in some cases (assuming the case of the first example), the split information is implicitly processed, but there may be a difference in encoding / decoding settings.
  • the above example may be a description of a case in which a segmentation process may be performed in a projection step, a regional packaging step, and an initial stage of encoding / decoding, and may be an image segmentation process occurring in other encoding / decoding machines.
  • a rectangular image may be formed by including the region B including the data in the region A including the data.
  • the position, size, shape, number, etc. of the A and B areas are information that can be checked by the projection format or the like, or when the information about the projected image is explicitly generated.
  • Information, image reconstruction information, etc. may be represented as related information. For example, as shown in Tables 4 and 5, information about some areas of the projected image (for example, part_top, part_left, part_width, part_height, part_convert_flag, etc.) may be indicated, and is not limited to this example. For example, it may be an example applicable to different projection formats, different projection settings, and the like.
  • the region B may be configured as one image together with the region A to perform encoding / decoding.
  • the division may be performed in consideration of the characteristics of each region to set different encoding / decoding settings.
  • the encoding / decoding for the B region may not be performed through information on whether to perform encoding / decoding (for example, tile_coded_flag when the division unit is a tile).
  • the corresponding area may be restored to predetermined data (any pixel value in this example) according to a preset rule.
  • the region B and the decoding / decoding setting in the above-described image segmentation process may be different from the region A.
  • the area can be removed by performing a regional packaging process.
  • the packaged image may be an image of a surface relocation process or a region-based packaging process, and may be an image obtained by performing image segmentation and image reconstruction.
  • a rectangular shape may be formed by including an area including data.
  • the location, size, shape, number, etc. of each region is information that can be checked by a preset setting or information that can be checked when explicitly generating information about a packaged image.
  • the relevant information may be indicated as information, image reconstruction information, and the like. For example, as shown in Table 4 and Table 5, information (eg, part_top, part_left, part_width, part_height, part_convert_flag, etc.) of some areas of the wrapped image may be indicated.
  • the packed image may be segmented using various segmentation methods. For example, partitioning may be performed in units of slices and may have a partition boundary of H0. Alternatively, division may be performed in units of tiles, and may have division boundaries of W0, W1, and H0. Alternatively, the division may be performed by the surface unit, and may have a division boundary of W0, W1, and H0.
  • the image segmentation and image reconstruction processes of the present invention may be performed on the projected image.
  • the reconstruction process may rearrange not only the pixels in the surface but also the surface in the image. This may be an example where the image is divided or configured into a plurality of surfaces. An example to be described later will be described based on the case where the tile is divided based on the surface unit.
  • SX, Y (S0,0 ⁇ S3,2) of 21a is S'U, V (S'0,0 ⁇ S'2,1) of 21b.
  • X, Y may be the same as or different from U, V.
  • the reconstruction process may be performed on the surface unit.
  • S2,1, S3,1, S0,1, S1,2, S1,1, S1,0 are S'0,0, S'1,0, S'2,0, S'0, 1, S'1,1, S'2,1 can be assigned (or surface rearrangement).
  • S2,1, S3,1, S0,1 does not perform reconstruction (or pixel rearrangement), and S1,2, S1,1, S1,0 may perform reconstruction by applying a 90 degree rotation. This may be represented as shown in FIG. 21C.
  • the symbols S1, 0, S1, 1, S1, and 2 displayed side by side in FIG. 21C may be images lying sideways in accordance with the symbols in order to maintain continuity of the images.
  • Surface reconstruction can be either implicit or explicit, depending on the encoding / decoding settings.
  • the implicit case may be performed according to a preset rule in consideration of the type of image (360 degree image in this example) and the characteristics (projection format in this example).
  • 21c S'0,0 and S'1,0, S'1,0 and S'2,0, S'0,1 and S'1,1, S'1,1 and S ' 2, 1 is an image continuity (or correlation) between the two surfaces based on the surface boundary
  • 21c may be an example configured to have a continuity between the top three surfaces and the bottom three surfaces.
  • reconstruction may be performed for the purpose of increasing image continuity between surfaces in order to efficiently reconstruct the surface in the process of packaging by region. Reconstruction of such surfaces can be preset and processed.
  • the reconstruction process may be performed through explicit processing and reconfiguration information may be generated.
  • a regional packaging process might lead to M ⁇ N configurations (e.g., 6 ⁇ 1, 3 ⁇ 2, 2 ⁇ 3, 1 ⁇ 6, etc. for the CMP compact, assuming a 3 ⁇ 2 configuration in this example).
  • information about the information e.g, implicitly acquired information or explicitly generated information
  • information about the surface may be generated after the surface reconstruction according to the M ⁇ N configuration.
  • index information or position information in an image
  • mode information for reconstruction may be allocated in the case of pixel relocation in the surface.
  • Index information may be defined as shown in 18a to 18c of Figure 18, S21, 21A to 21C in SX, Y or S'U, V is the position information indicating the horizontal and vertical on each surface (for example, S [i] [j]) or a piece of position information (eg, the assumption that the position information is allocated in the raster scan order from the upper left surface of the image. S [i]) and an index of each surface may be assigned thereto.
  • S'0,0 is surface 2
  • S'1,0 is surface 3
  • S'2,0 is 1
  • S'0,1 is assigned to surface 5
  • S'1,1 is assigned to surface 0,
  • S'2,1 is assigned to surface index 4.
  • S [0] is surface 2
  • S [1] is surface 3
  • S [2] is surface 1
  • S [3] is surface 5
  • S [4] assigns surface 0
  • S [5] assigns surface index 4.
  • S'0,0 to S'2,1 are referred to as a to f.
  • the image may be represented by position information representing horizontal and vertical units of pixels or blocks based on the upper left end of the image.
  • the scanning order of the surfaces may or may not be the same in the image according to the reconstruction setting.
  • one scan order eg, raster scan
  • the scan order of a, b, and c may be the same, and the scan order of d, e, and f may not be the same.
  • the scan order may follow the order of (1,0)-> (1,1)-> (0,0)-> (0,1). This may be determined according to the reconstruction setting of the image, and may have the same setting in other projection formats.
  • the image segmentation process may set individual surface units as tiles.
  • the surfaces a to f may be set in tile units, respectively.
  • a plurality of surface units may be set as tiles.
  • surfaces a to c may be set to one tile, and d to f may be set to one tile.
  • the configuration may be determined based on surface properties (eg, continuity between surfaces, etc.) and tile settings of surfaces different from the above example may be possible.
  • the following is an example of split information according to a plurality of video segmentation processes.
  • the partitioning information on the surface is omitted, the unit outside the surface is tile, and the partitioning information is processed in various ways.
  • image segmentation information may be obtained based on surface information and may be implicitly omitted.
  • individual surfaces may be set as tiles or a plurality of surfaces may be set as tiles.
  • it may be determined according to a predetermined rule based on surface information (for example, continuity or correlation).
  • image segmentation information may be explicitly generated regardless of surface information.
  • the split information may be generated by the above-described method of image segmentation.
  • the number of columns and the number of columns of tiles can range from 0 to the width of the image / width of the block (in this example, the unit obtained from the picture divider), from 0 to the length of the image. Can be up to the vertical length of the block.
  • additional partitioning information eg, uniform_spacing_flag, etc.
  • the boundary of the surface and the boundary of the division unit may or may not coincide according to the division setting.
  • image segmentation information may be explicitly generated based on surface information.
  • the surface information in this example, the range of number of columns is 0-2, the range of number of columns is 0, 1.
  • Partitioning information can be generated based on the configuration of the inner surface 3x2). For example, the range of the number of columns and the number of columns of the tile may range from 0 to 2 and from 0 to 1.
  • additional splitting information eg, uniform_spacing_flag, etc.
  • syntax element setting for example, binarization setting, etc.
  • Ranges are limited and other binarizations may be used for smaller ranges.
  • FIG. 22 illustrates an example of dividing an image projected by CMP or a packaged image into tiles.
  • FIG. 23 is a conceptual diagram illustrating an example of adjusting the size of a 360 degree image according to an embodiment of the present invention. In this case, it is assumed that the image projected by ERP. In addition, the example mentioned later demonstrates focusing on the case of expansion.
  • the projected image can be scaled by the scale factor and scaled by the offset factor according to the image scaling type.
  • the image before scaling is P_Width ⁇ P_Height and the image after scaling is P'_Width ⁇ P'_Height Can be.
  • the horizontal length (P_Width ⁇ a) and the vertical length (P_Height ⁇ b) of the image can be obtained after the scale factor (horizontal a, vertical b in this example) is adjusted to the horizontal and vertical lengths of the image. have.
  • the image's width (P_Width + L + R) and the height (P_Height + T + B) can be obtained.
  • the resizing may be performed using a preset method, or the resizing may be performed by selecting one of a plurality of methods.
  • a data processing method may include a method of filling using a preset pixel value, a method of copying and filling an outer pixel, a method of copying and filling a portion of an image, and a method of converting and filling a portion of an image. have.
  • the size may be adjusted in consideration of a characteristic in which continuity exists at the boundary of the image.
  • the outer boundary does not exist in three-dimensional space, but when transformed into two-dimensional space through the projection process, the outer boundary region may exist. This means that the data of the boundary area has continuity data outside the boundary, but may have a boundary due to spatial characteristics. Considering these characteristics, the scaling can be performed. At this time, the continuity can be confirmed according to the projection format.
  • the ERP may be an image having characteristics in which both edges are continuous stones. In this example, it is assumed that the left and right borders of the image are continuous and the upper and lower borders are continuous.
  • the data processing method is a method of copying and filling a part of an image and a method of converting and filling a part of an image. Explain the center.
  • the area to be scaled (LC or TL + LC + BL in this example) is the area of the right side of the image (tr + rc + br in this example) where continuity exists. Can be populated using data.
  • the area to be resized (RC or TR + RC + BR in this example) is the data of the left region (tl + lc + bl in this example) with continuity to the right.
  • the area to be resized (TC or TL + TC + TR in this example) is the data of the lower region of the image (bl + bc + br in this example) where continuity exists.
  • data may be filled using data of an area to be scaled (BC or BL + BC + BR in this example).
  • the area to be resized is (-m, y) to (-1, y) with the coordinate reference of the image before resizing (in this example, x is 0 to P_Width-1). ) Or (P_Width, y) to (P_Width + m-1, y) in the range (resize to the left).
  • x refers to the coordinates of the area to be scaled based on the image coordinates before resizing
  • x ' refers to the coordinates of the area referenced to the area to be scaled based on the image coordinates before resizing. For example, if you scale to the left and m is 4 and the width of the image is 16, (-4, y) is (12, y) and (-3, y) is (13, y). In this case, (-2, y) may obtain corresponding data from (14, y) and (-1, y) from (15, y).
  • the size to be resized is (x, -n) to (x, -1 in the coordinates of the image before resizing (in this example, y is 0 to P_Height-1). ) Or (x, P_Height) to (x, P_Height + n-1).
  • y refers to the coordinates of the area to be scaled based on the image coordinates before resizing
  • y ' refers to the coordinates of the area referred to the area to be scaled to the image coordinates before the resizing. For example, if the image is scaled upward and n is 4 and the vertical length of the image is 16, (x, -4) is at (x, 12) and (x, -3) is at (x, 13). , (x, -2) may obtain corresponding data from (x, 14) and (x, -1) from (x, 15).
  • the image may be adjusted based on the image coordinate reference (in this example, x is 0 to P'_Width-1 and y is 0 to P'_Height-1).
  • the example may be an example that may be applied to a coordinate system of latitude and longitude.
  • the size may be adjusted by m to the left of the image.
  • the size may be adjusted by n to the right of the image.
  • the image may be resized by o toward the upper side of the image.
  • the size may be adjusted by p toward the lower side of the image.
  • the image may be resized by m to the left and n to the right.
  • the image may be resized by o toward the top of the image and p by the bottom of the image.
  • the size of the image may be adjusted by m to the left, n to the right, and o by the upper side.
  • the image may be resized by m to the left of the image, n to the right of the image, and p by the lower side of the image.
  • the image may be resized by m to the left of the image, by o to the top, and by p to the bottom.
  • the image may be resized by n to the right of the image, o by the upper side, and p by the lower side.
  • the image may be resized by m to the left, n to the right, o by the top, and p by the bottom.
  • At least one resizing may be performed as in the above example, and the resizing of the image may be performed implicitly according to the encoding / decoding setting, or the resizing of the image may be explicitly generated based on the resizing information. Can be performed. That is, m, n, o, and p in the above example may be set to a preset value, or may be explicitly generated by the resizing information, some may be set to a predetermined value, and some may be explicitly created.
  • the above example has been described focusing on the case of acquiring data from a partial region of an image, but other methods are also applicable.
  • the data may be pixels before encoding or pixels after encoding, and may be determined according to characteristics of an image or a step in which scaling is performed.
  • the data refers to input pixels such as a projection image and a packaging image, and includes a post-processing step, an intra prediction screen pixel generation step, a reference image generation step,
  • the data may mean a reconstructed pixel.
  • the resizing may be performed by using a data processing method individually for each sized area.
  • 24 is a conceptual diagram illustrating continuity between surfaces in a projection format (eg, CMP, OHP, ISP) according to an embodiment of the present invention.
  • a projection format eg, CMP, OHP, ISP
  • FIGS. 24A to 24C are spatially adjacent when there is a continuity when converted to two-dimensional space through a projection process (A), and spatially adjacent And if there is no continuity (B), there is no spatially adjacent and there is continuity (C), there is no spatially adjacent and there is no continuity (D).
  • the general image has a difference from being classified as being spatially adjacent and having continuity (A), and not being spatially adjacent and having no continuity (D). In this case, when the continuity is present, some examples (A or C) are applicable.
  • a case in which there is a spatially adjacent and continuity may appear as b0 to b4, and a case in which there is no spatially adjacent and there is continuity is B0.
  • B6 it means a case of adjacent regions in a three-dimensional space, and coding performance can be improved by using the properties of continuity between b0 to b4 and B0 to B6 in the encoding process.
  • FIG. 25 is a conceptual diagram illustrating the continuity of the surface of FIG. 21C, which is an image obtained through an image reconstruction process or a region-based wrapping process in a CMP projection format.
  • 21c of FIG. 21 is a rearrangement of 21a in which a 360-degree image is unfolded in a cube shape, the continuity of the surface of FIG. 21a is maintained. That is, as in 25a, the surfaces S2,1 are continuous with each other S1,1 and S3,1 from side to side, and with the S1,0 surface rotated 90 degrees up and down and with the S1,2 surface rotated -90 degrees have.
  • the continuity between the surfaces may be defined according to the projection format setting or the like, and examples of other modifications may be possible without being limited to the above examples. Examples to be described later will be described under the assumption that continuity as shown in FIGS. 24 and 25 exists.
  • FIG. 26 is an exemplary diagram for explaining an image size adjustment in a CMP projection format according to an embodiment of the present invention.
  • 26a is an example of resizing an image
  • 26b is an example of resizing in surface units (or divided units)
  • 26c is an example of resizing (or multiple resizing) units in an image and a surface unit. Indicates.
  • the projected image can be scaled by the scale factor and scaled by the offset factor according to the image scaling type.
  • the image before scaling is P_Width ⁇ P_Height and the image after scaling is P'_Width ⁇ P'_Height
  • the size of the surface may be F_Width ⁇ F_Height.
  • the size may be the same or different depending on the surface, and the horizontal and vertical lengths of the surface may be the same or different. However, in the present example, all surfaces in the image have the same size and have a square shape for convenience of description. In addition, it demonstrates on the assumption that the magnitude adjustment values (WX, HY in this example) are the same.
  • the data processing method in the following example will be described based on the case of the offset factor, and the data processing method will be described based on a method of copying and filling a partial region of an image and a method of converting and filling a partial region of an image.
  • the above setting may be equally applied to FIG. 27.
  • the boundary of the surface may have continuity with the boundary of the other surface.
  • it may be classified into a case where the continuity of the image exists spatially adjacent to the 2D plane (the first example) and a case where the continuity of the image exists while the continuity of the image exists (the second example) in the 2D plane.
  • the upper, left, right, and lower regions of S1, 1 are S1,0, S0,1, S2,1, S1,2, lower, right, left
  • the image may be continuous (in the case of the first example) while being spatially adjacent to the upper region.
  • the left and right regions of S1,0 are not spatially adjacent to the upper regions of S0,1 and S2,1, but images of each other may be continuous (when the second example).
  • the left region of S0,1 and the right region of S3,1 are not spatially adjacent to each other, the images of each other may be continuous (in the case of the second example).
  • the left region and the right region of S1, 2 may be continuous with each other (if the second example) with the lower region of S0,1 and S2,1. This may be a limiting example in this example, and may be a different configuration from the above depending on the projection format definition and setting.
  • S0,0 to S3,2 of FIG. 26A are referred to as a to l.
  • 26a may be an example of filling by using data of an area in which continuity exists in an external boundary direction of an image.
  • the area to be resized from the area A where no data exists (in this example, a0 to a2, c0, d0 to d2, i0 to i2, k0, l0 to l2) can be filled with a predetermined value or outer pixel padding.
  • the area (B0, e0, h0, j0) that is resized from the B area including the actual data may be filled using data of an area (or surface) in which the continuity of the image exists.
  • b0 can be filled using data on the upper side of the surface h, e0 on the right side of the surface h, h0 on the left side of the surface e, and j0 on the lower side of the surface h.
  • b0 may be an example of filling the data of the lower side of the surface obtained by applying a 180 degree rotation to the surface h
  • j0 may be an example of filling using the upper data of the surface obtained by applying the 180 degree rotation to the surface h.
  • an adjustment process eg, rotation, etc.
  • 26b may be an example of filling by using data of an area in which continuity exists in an inner boundary direction of an image.
  • the scaling operation performed according to the surface may be different.
  • Region A may perform a reduction process and region B may perform an expansion process.
  • scaling reduction in this example
  • region B may perform an expansion process.
  • scaling reduction in this example
  • w0 the case of surface a
  • scaling expansion in this example
  • h0 the case of surface a
  • scaling (expansion in this example) may be performed upward by h0.
  • the area to be resized (in this example, b0, e0, be, b1, bg, g0, h0, e1, ej, j0, gi, g1, j1, h1) shows the point of reduction from the area A where no data exists. In consideration, it may be simply removed, or may be newly filled with data of an area in which continuity exists in consideration of an extension from the B area including actual data.
  • b0 is the upper side of surface e
  • e0 is the left side of surface b
  • be is the left side of surface b or the upper side of surface e or the left side of surface b and the upper side of surface e
  • b1 is the upper side of surface g
  • bg is the left side of surface b or the upper side of surface g or the right side of surface b and the upper side of surface g
  • g0 is the right side of surface b
  • h0 is the upper side of surface b
  • e1 is the left side of surface j
  • ej is surface e
  • j0 is the lower side of the surface e
  • gj is the lower side of the surface g or the lower side of the surface j or the lower side of the surface g and the right side of the surface j
  • the data of the area may be copied and filled, or data obtained after the process of converting the data of the area based on the characteristics and types of the image may be obtained.
  • a coordinate system eg, a two-dimensional plane coordinate system
  • each surface is converted to (x, y, C) or (x, C, z) or (C, y, z) in (x, y, z) in three-dimensional space.
  • the above example shows a case where data of a surface different from that of a surface is acquired in an area to be scaled of some surface. That is, there is a possibility that the continuity may be distorted based on the scaling boundary when the data of the other surface having different coordinate system characteristics is copied and filled as it is to adjust the size of the current surface.
  • the data of another surface acquired according to the coordinate system characteristics of the current surface may be converted and filled in an area to be resized. In the case of conversion is also an example of a data processing method is not limited thereto.
  • the distorted continuity (or radically changing continuity) is included in the boundary area between the resized area e and the resized area e0. can do.
  • a continuous feature is changed based on a boundary, and the edges have a straight shape as if the edges were bent based on the boundary.
  • the region to be resized When transforming and filling data of a partial region of an image into a region to be resized, it may include a continuity that gradually changes in a boundary region between the region to be resized and the region to be resized.
  • the data processing method of the present invention fills an area to be resized with data obtained by performing a conversion process based on the characteristics, types, etc. of an image in a resizing process (expansion in this example). It may be an example.
  • 26c may be an example of filling by using data of an area in which continuity exists in the direction of the boundary (inner boundary and outer boundary) of the image by combining the image resizing process according to 26a and 26b.
  • the resizing process of this example can be derived from 26a and 26b, so a detailed description thereof will be omitted.
  • 26a may be an example of adjusting the size of the image
  • 26b may be an example of adjusting the size of the division unit in the image
  • 26c may be an example of a plurality of resizing processes in which the resizing process of the image and the resizing of the division unit in the image are performed.
  • a resizing may be performed on an image (first format in this example) obtained through a projection process, and an image (second format in this example) acquired through a format conversion process. ), You can perform the resizing (area D in this example).
  • the image is adjusted in size (in this example, the entire image) on the image projected by ERP, and the size adjustment (surface unit in this example) is performed after acquiring the image projected in CMP through the format converter.
  • the above example is an example of performing a plurality of size adjustments, but is not limited thereto and may be modified in other cases.
  • FIG. 27 is an exemplary diagram for explaining a size adjustment for a packaged image converted to a CMP projection format according to an embodiment of the present invention.
  • the boundary of the surface may have continuity with the boundary of another surface.
  • the offset factors W0 to W5 and H0 to H3 may have various values. For example, it may be derived from a preset value, a motion search range of inter-screen prediction, a unit obtained from a picture splitter, or the like, or else.
  • the unit obtained from the picture divider may include a surface. That is, the size adjustment value may be determined based on F_Width and F_Height.
  • 27a is an example of filling by using data of a region where continuity exists in each region to be extended by performing a sizing (in this example, the up, down, left, and right directions of each surface) for each surface.
  • a sizing in this example, the up, down, left, and right directions of each surface
  • continuous data may be filled in a0 to a6 outside of the surface a
  • continuous data may be filled in b0 through b6 outside of the surface b.
  • 27b is an example of filling by using data of a region where continuity exists in a region that is extended by performing size adjustment (up, down, left, and right directions of the plurality of surfaces in this example). For example, it can extend to a0-a4, b0-b1, and c0-c4 about the outer surface with respect to the surface a, b, and c.
  • 27c may be an example of filling by using data of a region in which continuity exists in an extended region by adjusting the size of the entire image (up, down, left, and right directions of the entire image in this example). For example, it is possible to extend a0 to a2, b0, c0 to c2, d0 to d2, e0 and f0 to f2 with respect to the entire outline of the image including the surfaces a to f.
  • the size adjustment may be performed by one surface unit, the size adjustment may be performed by a plurality of surface units having continuity, and the size adjustment may be performed by the whole surface unit.
  • the region to be scaled in the above example may be filled using data of the region (or surface) where continuity exists as shown in FIG. 24A. That is, the data to be sized can be filled using the data on the upper, lower, left and right sides of the surfaces a to f.
  • FIG. 28 is an exemplary diagram for describing a data processing method in resizing a 360 degree image according to an embodiment of the present invention.
  • the area B to be resized (a0 to a2, ad0, b0, c0 to c2, cf1, d0 to d2, e0, f0 to f2) has continuity among pixel data belonging to a to f. You can fill it with data from an existing area.
  • the C region (ad1, be, cf0), which is another sized region, may be mixed with data of the region to be scaled and data of the region that is spatially adjacent but does not have continuity.
  • the region C may be sized between two selected regions a to f (for example, a and d, b and e, c and f), and thus data of the two regions may be mixed and filled.
  • surface b and surface e may have a relationship that is spatially adjacent but without continuity. Resizing can be performed using the data of surface b and the data of surface e in the region be to be scaled between surface b and surface e.
  • the be region may be filled with a value obtained by averaging the data of the surface b and the data of the surface e, or may be filled with a value obtained through a weighted sum based on a distance.
  • the pixels used for data to be filled in the areas to be adjusted in the surface b and the surface e may be boundary pixels of each surface, but may also be internal pixels of the surface.
  • an area to be adjusted between division units of an image may be filled with data generated by mixing data of both units.
  • the data processing method may be a method supported in some situations (when resizing is performed in a plurality of areas in this example).
  • FIG. 27A and 27B separately configure regions in which size adjustment is performed between division units for each division unit separately (for example, a6 and d1 are configured for a and d, respectively, in FIG. 27A), but divisions are made in FIG. 28. Regions in which the size adjustment is performed between the units may be configured one by one for adjacent division units (one ad1 for a and d).
  • the method may be included as a candidate group of the data processing method in FIGS. 27A to 27C, and size adjustment may be performed using a data processing method different from the above example in FIG. 28.
  • an implicitly preset data processing method may be used for the area to be scaled, or one of a plurality of data processing methods may be used to explicitly generate related information.
  • the preset data processing method includes a method of filling using an arbitrary pixel value, a method of copying and filling an outer pixel, a method of copying and filling a portion of an image, a method of converting and filling a portion of an image, and a plurality of regions of an image. It may be one of data processing methods such as a method of filling derived data. For example, if a region to be resized is located inside the image (e.g. a packed image) and both regions (e.g.
  • a data processing method of filling the derived data may be applied to fill an area to be resized.
  • size adjustment may be performed by selecting one of the plurality of data processing methods, and selection information thereof may be explicitly generated. This may be an example that may be applied to a general image as well as a 360 degree image.
  • the generated information will be recorded in the bitstream in at least one unit of a sequence, a picture, a slice, a tile, etc. in the encoder, and the decoder will parse related information from the bitstream. It may also be included in the bitstream in the form of SEI or metadata.
  • SEI Session Initiation Protocol
  • the process of segmentation, reconstruction, and resizing in a 360 degree image has been described based on some projection formats such as ERP and CMP, but the present invention is not limited thereto and may be applied to other projection formats in the same manner or in a different manner.
  • the image setting process applied to the 360-degree image encoding / decoding apparatus described above can be applied not only to the encoding / decoding process but also to a preprocessing process, a post-processing process, a format conversion process, and a format inverse conversion process.
  • the projection process may include an image setting process.
  • at least one image setting process may be included and the projection process may be performed.
  • the division may be performed on an area (or surface) basis based on the projected image. Depending on the projection format, division may be performed into one region or a plurality of regions. Partition information according to the partition may be generated.
  • scaling of the projected image may be performed or scaling of the projected area may be performed. In this case, the size may be adjusted to at least one area. Size adjustment information according to the size adjustment may be generated.
  • reconstruction (or surface placement) of the projected image may be performed or reconstruction of the projected area may be performed. At this time, the reconstruction may be performed on at least one region. Reconstruction information according to the reconstruction may be generated.
  • the regional packaging process may include an image setting process.
  • at least one image setting process may be included and a region-specific packaging process may be performed.
  • the segmentation process may be performed in units of regions (or surfaces) based on the packed image.
  • the division may be performed in one region or a plurality of regions according to the regional packaging setting.
  • Partition information according to the partition may be generated.
  • scaling of the wrapped image may be performed or scaling of the wrapped area may be performed. In this case, the size may be adjusted to at least one area. Size adjustment information according to the size adjustment may be generated.
  • reconstruction of the packed image may be performed or reconstruction of the packed area may be performed. At this time, the reconstruction may be performed on at least one region. Reconstruction information according to the reconstruction may be generated.
  • the projection process may be all or part of the image setting process, and may include image setting information. This may be setting information of the projected image. In detail, it may be setting information of regions in the projected image.
  • the packaging process for each region may be all or part of the image setting process and may include image setting information.
  • This may be setting information of the packaged image.
  • it may be setting information of regions in the packaged image.
  • it may be mapping information of the projected image and the wrapped image (for example, refer to description related to FIG. 11.
  • P0 to P1 may be understood as projected images and S0 to S5 may be understood to be packed images).
  • the information may be mapping information of a partial region in the projected image and a partial region in the packaged image. That is, the setting information may be set to be allocated to some areas of the wrapped image from some areas of the projected image.
  • the information may be represented as information obtained through the aforementioned various embodiments in the image setting process of the present invention.
  • the setting information of the projected image is pic_width_in_samples, pic_height_in_samples, part_top [i], part_left [i], part_width [i], part_height [i] and the like
  • the setting information of the packed image is pic_width_in_samples, pic_height_in_samples, part_top [i], part_left [i], part_width [i], part_height [i], convert_type_flag [i], part_resizing_flag [i ], top_height_offset [i], bottom_height_offset [i], left_width_offset [i], right_width_offset [i], resizing_type_flag [i], and the like.
  • the above example may be an example of explicitly
  • a part of the image setting process may be included in the projection process according to the projection format or the packaging process by region as a predetermined operation.
  • the size is adjusted by converting and filling the data of the region to be adjusted in the regions extended by m, n, o, and p in the up, down, left, and right directions, respectively, and the region in which the continuity exists.
  • the process of making adjustments may be implicitly included.
  • the projection format of the above example may be an example of replacing an existing projection format or may be an example of an additional projection format (for example, ERP1, CMP1) in addition to the existing projection format, and is not limited to the above example and various image setting procedures of the present invention. May be alternatively combined, and similar applications may be possible for other formats.
  • ERP1, CMP1 additional projection format
  • a block divider may be further included.
  • Information about a basic coding unit may be obtained from a picture splitter, and the basic coding unit may mean a basic (or start) unit for prediction, transformation, quantization, etc. in an image encoding / decoding process.
  • the coding unit may include one luminance coding block and two color difference coding blocks according to a color format (YCbCr in this example), and the size of each block may be determined according to the color format.
  • the description will be made based on the block (luminance component in this example). In this case, it is assumed that the block is a unit that can be obtained after each unit is determined, and it will be described on the assumption that similar settings are applicable to other types of blocks.
  • the block divider may be set in relation to each component of the image encoding apparatus and the decoding apparatus, and the size and shape of the block may be determined through this process.
  • the set block may be defined differently according to the configuration unit, and the prediction block in the prediction unit, the transform block in the transform unit, and the quantization block in the quantization unit may correspond thereto.
  • block units according to other components may be further defined.
  • the size and shape of the block may be defined by the width and length of the block.
  • a block may be expressed as M ⁇ N, and a maximum value and a minimum value of each block may be obtained within a range.
  • the block type supports squares, and if the maximum value of the block is 256 ⁇ 256 and the minimum value is 8 ⁇ 8, a block of size 2 m ⁇ 2 m (in this example, m is an integer from 3 to 8).
  • 8 ⁇ 8, 16 ⁇ 16, 32 ⁇ 32, 64 ⁇ 64, 128 ⁇ 128, 256 ⁇ 256, or a block of size 2m ⁇ 2m (in this example, m is an integer from 4 to 128) or A block of size m ⁇ m (in this example, m is an integer from 8 to 256) can be obtained.
  • the block shape is square and rectangular, and has the same range as the above example, a block of size 2 m ⁇ 2 n (m and n are integers from 3 to 8 in this example.
  • a block obtainable may be determined according to a sub / decoding setting (for example, a type of a block, a division scheme, a division setting, etc.).
  • a coding block may obtain a 2 m ⁇ 2 n block
  • a prediction block may be a 2 m ⁇ 2 n or m ⁇ n size block
  • a transform block may have a 2 m ⁇ 2 n size block.
  • information such as a block size and a range (for example, information related to an index and a multiple) may be generated.
  • the above range (in this example, the maximum value and the minimum value) may be determined.
  • range information of blocks may be explicitly generated, and in some blocks, range information of blocks may be implicitly determined.
  • related information may be explicitly generated in an encoding and transform block, and related information may be implicitly processed in a prediction block.
  • At least one range information may be generated.
  • information on a range may generate information on a maximum value and a minimum value.
  • the method may be generated based on a difference between the maximum value and a predetermined minimum value (eg, 8) (eg, generated based on the setting.
  • the difference value information of the exponent of the maximum value and the minimum value may be generated.
  • range information may be obtained based on a sub / decoding setting (for example, a type of a block, a division method, a division setting, etc.).
  • a sub / decoding setting for example, a type of a block, a division method, a division setting, etc.
  • the splitting setting of the prediction block in the coding block (the maximum size M ⁇ N of the coding block in this example, the minimum size m ⁇ n in this example) (quad tree split + split depth 0 in this example)
  • the maximum value and the minimum value information can be obtained with a candidate group obtainable as (M ⁇ N and m / 2 ⁇ n / 2 in this example).
  • the size and shape of the initial (or starting) block of the block division may be determined from higher units.
  • a basic coding block obtained from a picture splitter may be an initial block
  • a coding block in the case of a prediction block, a coding block may be an initial block
  • a transform block in the case of a transform block
  • a coding block or a prediction block may be an initial block. It can be determined according to the negative / decryption setting.
  • the prediction block may be an upper unit of the transform block
  • Inter the prediction block may be a unit independent of the transform block.
  • the initial block may be divided into blocks of small size as a start unit of the partition.
  • the block may be determined as an initial block of lower units.
  • the former case may be a coding block
  • the latter case (subunit) may be a prediction block or a transform block.
  • a partitioning process for searching for a block having an optimal size and shape like the upper unit may be performed.
  • the block division unit may divide the basic coding unit (or the largest coding unit) into at least one coding unit (or the lower coding unit).
  • the coding unit may perform division in at least one prediction unit and may perform division in at least one transformation unit.
  • the coding unit may perform splitting into at least one coding block, the coding block may perform splitting into at least one prediction block, and may split into at least one transform block.
  • the prediction unit may perform division into at least one prediction block, and the transform unit may perform division into at least one transform block.
  • mode information for example, partition information
  • the mode information may be transmitted in the bitstream together with information (for example, prediction related information and transformation related information) generated in the component to which the block belongs, and transmitted to the decoder. Can be used in the video decoding process.
  • the block divider may support various split schemes. For example, it may support tree-based partitioning or type-based partitioning, and other methods may be applied. In the case of tree-based partitioning, partition information may be generated using a partition flag, and in the case of type-based partitioning, partition information may be generated using index information for a block type included in a preset candidate group.
  • 29 is an exemplary diagram illustrating a tree-based block form.
  • 29a is one 2N ⁇ 2N without splitting
  • 29b is some splitting flag (horizontal splitting of the binary tree in this example)
  • two 2N ⁇ N, 29c is some splitting flag (vertical splitting in the binary tree in this example).
  • Two N ⁇ 2N and 29d represent an example in which four N ⁇ Ns are obtained through some splitting flags (four divisions of a quad tree or horizontal and vertical divisions of a binary tree in this example).
  • the shape of the block to be obtained may be determined according to the tree type used for splitting. For example, when quad tree splitting is performed, candidate blocks obtainable may be 29a or 29d.
  • candidate blocks obtainable may be 29a, 29b, 29c, and 29d.
  • one splitting flag is supported, and when the corresponding flag is '0', 29a and 29d may be obtained.
  • a plurality of splitting flags are supported, one of which may be a flag indicating whether to split, one of which may be a flag indicating whether to split horizontally or vertically, and one of which is a splitting of horizontal / vertical splitting. It may be a flag indicating whether to allow duplication.
  • the candidate blocks obtainable when overlapping is allowed may be 29a, 29b, 29c, and 29d, and the candidate blocks obtainable when duplication is not allowed may be 29a, 29b, or 29c.
  • the quad tree may be a basic tree-based partitioning scheme.
  • a tree splitting scheme (a binary tree in this example) may be included in the tree-based partitioning scheme.
  • a flag allowing additional tree splitting is activated implicitly or explicitly, a plurality of tree splitting may be performed.
  • Tree-based partitioning may be a method in which recursive partitioning is possible. That is, the partitioned block may be set as an initial block again and tree-based partitioning may be performed, which may be determined according to partitioning settings such as a partition range and a partitioning allowable depth. This may be an example of a hierarchical partitioning scheme.
  • FIG. 30 shows an example of a type based block shape.
  • the block is divided into 1 division (30a in this example), 2 divisions (30b, 30c, 30d, 30e, 30f, and 30g in this example), and 4 divided forms (30h in this example).
  • Candidate groups can be configured through various configurations.
  • the candidate group may be configured with a, b, c, n or a, b to g, n or a, n, q, and the like of FIG. 31, and the present disclosure is not limited thereto.
  • An example may be possible.
  • the blocks supported when the flag allowing the symmetric partition is activated may be FIGS.
  • Type-based partitioning may be a method of supporting one partitioning. Compared with tree-based partitioning, blocks obtained through type-based partitioning may not be able to further partition further. This may be an example where the splitting allowance depth is zero (eg, single layer splitting).
  • FIG. 31 is an exemplary view illustrating various block types that may be acquired by a block divider of the present invention.
  • FIG. 31 is an exemplary view illustrating various block types that may be acquired by a block divider of the present invention.
  • blocks 31a to 31s may be obtained according to a partitioning setting and a partitioning scheme, and an additional block type not shown may be possible.
  • asymmetric partitioning may be allowed for tree-based partitioning.
  • a block such as those shown in FIGS. 31B and 31C (in this example, may be divided into a plurality of blocks) may be possible, or a block such as FIGS. 31B to 31G (in this example, divided into a plurality of blocks). May be possible.
  • the candidate blocks obtainable may be 31b or 31c (this example assumes not allowing horizontal and vertical overlapping splitting) and is asymmetric.
  • the candidate blocks obtainable when the flag allowing division is activated may be 31b, 31d, 31e (horizontal division in this example) or 31c, 31f, 31g (vertical division in this example).
  • This example may correspond to a case in which a division direction is determined by a horizontal or vertical division flag and a block shape is determined according to an asymmetric allowable flag.
  • the present invention is not limited thereto and may be modified in other examples.
  • additional tree splitting may be allowed for tree-based splitting.
  • it may be possible to divide a triple tree, a quad type tree, an octa tree, and the like, thereby obtaining n partition blocks (3, 4, 8. n are integers in this example).
  • the supported block when divided into a plurality of blocks in this example
  • the supported block may be 31h to 31m
  • the supported block may be 31n to 31p
  • the supported block May be 31q.
  • Whether to support the tree-based partitioning may be implicitly determined or explicitly related information may be generated according to the encoding / decoding.
  • a binary tree blocks such as FIGS. 31B and 31C may be possible, and when a binary tree and a triple tree are mixed (this example partially overlaps the use range of the binary tree and the use range of the triple tree).
  • a block such as 31b, 31c, 31i, and 31l may be possible.
  • the candidate blocks obtainable may be 31b or 31c, and if enabled, the candidate blocks obtainable are 31b, 31i or 31b, 31h, 31i, 31j (horizontal division in this example) or 31c, 31l or 31c, 31k, 31l, 31m (vertical division in this example).
  • This example may correspond to a case in which a division direction is determined by a horizontal or vertical division flag and a block shape is determined according to a flag that allows additional division.
  • the present invention is not limited thereto and may be modified in other examples.
  • a non-rectangular partition may be allowed for a type-based block.
  • partitioning such as 31r and 31s may be possible.
  • blocks of 31a, 31b, 31c, 31h, 31r, 31s, or 31a to 31h, 31r, 31s may be supported.
  • a block supporting n divisions such as 31h to 31m (for example, n is an integer. In this example, 3 except 1, 2, and 4) may be included in the candidate group.
  • the division method may be determined according to the negative / decoding settings.
  • the partitioning scheme may be determined according to the type of block.
  • coding blocks and transform blocks may use tree-based partitioning
  • prediction blocks may use type-based partitioning.
  • a combination of two types of splitting schemes may be used.
  • the prediction block may use a partitioning scheme that mixes tree-based partitioning and type-based partitioning, and the partitioning scheme applied to at least one range of the block may be different.
  • the partitioning scheme may be determined according to the size of the block. For example, some ranges (e.g., axb to cxd. If the latter is larger) between the maximum and minimum values of the block include tree-based partitioning, and some ranges (e.g. e X f to g x h) may be divided based on type. In this case, range information according to the division scheme may be explicitly generated or may be determined implicitly.
  • a partitioning scheme may be determined according to the shape of a block (or a block before partitioning). For example, if the block shape is square, tree-based partitioning and type-based partitioning may be possible. Alternatively, tree-based partitioning may be possible when the block is rectangular.
  • the division setting may be determined according to the encoding / decoding setting.
  • the division setting may be determined according to the type of the block.
  • a coding tree and a prediction block may use quad tree, and a transform block may use binary tree partition.
  • the partitioning allowable depth may be m
  • the partitioning allowable depth is n
  • the partitioning allowable depth may be set to o, and m, n and o may or may not be the same.
  • the division setting may be determined according to the size of the block.
  • some ranges of blocks e.g., axb to cxd
  • some ranges e.g. exf to gxh.
  • Cxd in this example is g Binary tree splitting may be possible
  • the range may include all ranges between the maximum value and the minimum value of the block, and the range may have settings that do not overlap each other or may have settings that overlap.
  • the minimum value of some ranges may be equal to the maximum value of some ranges, or the minimum value of some ranges may be less than the maximum value of some ranges.
  • a partitioning scheme having a higher maximum value may have priority. That is, whether or not to perform the partitioning method having a lower priority may be determined according to the partitioning result in the partitioning method having the priority. In this case, range information according to the tree type may be explicitly generated or may be determined implicitly.
  • Type-based partitioning with some candidate groups in some ranges may be possible.
  • the range may include all ranges between the maximum value and the minimum value of the block, and the range may have a setting that does not overlap each other.
  • the division setting may be determined according to the shape of the block. For example, quadtree splitting may be possible when a block has a square shape. Alternatively, binary tree splitting may be possible when a block is rectangular.
  • the division setting may be determined according to sub / decoding information (eg, slice type, color component, encoding mode, etc.). For example, if the slice type is I, quad tree (or binary tree) splitting may be for some ranges (e.g. axb to cxd), and for P some ranges (e.g. exf to g). Xh), in the case of B, it may be possible in some range (eg, i ⁇ j to k ⁇ l).
  • the partitioning allowable depth m of the quad tree (or binary tree) partitioning may be set to n, the partitioning allowable depth n for P, and the partitioning allowable depth o for B, and m and n and o may be the same. May not be the same.
  • Some slice types may have the same configuration as other slices (for example, P and B slices).
  • the quadtree (or binary tree) division allowable depth may be set to m, and in the case of a chrominance component, m and n may or may not be the same.
  • the range of quad tree (or binary tree) splitting e.g., axb to cxd
  • e x f to g x h may or may not be the same.
  • the quadtree (or binary tree) splitting depth may be m, n may be n (assuming n is greater than m in this example), and m and n may or may not be the same. Can be.
  • the range of quad tree (or binary tree) splitting and the encoding mode when Inter is the range of quad tree (or binary tree) splitting may or may not be the same.
  • information on whether to support adaptive segmentation candidate group configuration based on encoding / decoding information may be explicitly generated or implicitly determined.
  • the division method and the division setting are determined according to the encoding / decoding setting.
  • the above example shows some cases according to each element, and variations to other cases may also be possible.
  • the division method and the division setting may be determined according to the combination by a plurality of elements.
  • the division scheme and the division setting may be determined based on the type, size, shape, sub / decoding information, and the like of the block.
  • elements involved in the partitioning scheme, setting, etc. may be determined implicitly or explicitly to generate information to determine whether to allow the adaptive case as in the above example.
  • the splitting depth of the splitting setup means the number of times of spatial division based on the initial block (in this example, the splitting depth of the initial block is 0), and the splitting depth may be divided into smaller blocks as the splitting depth increases. This may vary the depth related setting according to the division method. For example, among tree-based partitioning methods, one common depth may be used for the quadtree splitting depth and the binary tree splitting depth, and individual depths according to the tree types may be used.
  • the split depth when the individual split depth is used according to the type of the tree, the split depth may be set to 0 at the split start position of the tree (the block before performing the split in this example).
  • the split depth can be calculated around the start of splitting, not based on the split range of each tree (maximum value in this example).
  • 32 is an exemplary diagram for describing tree based partitioning according to an embodiment of the present invention.
  • 32a shows an example of quad tree and binary tree splitting. Specifically, the upper left block of 32a shows a quad tree split, the upper right block and the lower left block show a quad tree and a binary tree split, and the lower right block shows a binary tree split.
  • the solid line (Quad1 in this example) represents the boundary line divided into quad trees
  • the dotted line (Binary1 in this example) represents the boundary line divided into binary trees
  • the thick solid line (Binary2 in this example) represents the boundary line divided into binary trees. it means.
  • the difference between the dashed line and the thick solid line is the difference in the division method.
  • the upper left block has a quadtree splitting depth of 3.
  • the current block is N ⁇ N
  • splitting is performed until either horizontal or vertical reaches (N >> 3), but the splitting information is (N> Generate split information up to> 2), which is common to the examples described below: assume that the maximum and minimum values of the quad tree are N ⁇ N and (N >> 3) ⁇ (N >> 3).
  • the tree split may be divided into four blocks each having a length of 1/2 of a length and a length.
  • the partition flag may have a value of '1' when partitioning is activated and '0' when partitioning is inactive. According to the setting, the division flag of the upper left block may occur like the upper left block of 32b.
  • the upper right block has a quadtree splitting allowance depth of 0.
  • the binary tree splitting allowance depth is 4.
  • the maximum and minimum values of quadtree splitting are N ⁇ N, (N >> 2) ⁇ (N >> 2).
  • the maximum and minimum values of the binary tree are (N >> 1) ⁇ (N >> 1), (N >> 3) ⁇ (N >> 3)).
  • the idol block performs quad tree splitting on the initial block.
  • the block may be divided into four blocks each having a length of 1/2 of a length and a length.
  • the size of the partitioned block is (N >> 1) ⁇ (N >> 1), which means that binary tree splitting (in this example is larger than the minimum value of the quadtree splitting, but the limiting splitting depth is limited), depends on the settings in this example.
  • this example may be an example in which overlapping of quad tree partitioning and binary tree partitioning is impossible.
  • the partition information of the binary tree of this example may be composed of a plurality of partition flags. Some flags can be horizontal split flags (corresponding to x of x / y in this example), and some flags can be vertical split flags (corresponding to x / y in y in this example). It may have a similar setting (eg whether it is active or not). In this example, the two flags may be activated in duplicate.
  • flag information is generated with '-' in the figure, '-' may be an implicit processing of flags that may occur when additional splitting is not possible due to conditions such as maximum, minimum, and splitting depths according to tree splitting. have. According to the setting, the division flag of the upper right block may occur like the upper right block of 32b.
  • the lower left block has a quadtree splitting allowance depth 3.
  • the binary tree splitting allowance depth is 2.
  • the maximum and minimum values of quadtree splitting are N ⁇ N, (N >> 3) ⁇ (N >> 3).
  • the maximum and minimum values of binary tree partitions are (N >> 2) ⁇ (N >> 2), (N >> 4) ⁇ (N >> 4) .
  • the partition priority is given to quad tree partitions.
  • the quadtree splitting is performed on the initial block, the lower left block may be divided into four blocks each having a length of 1/2 of the horizontal and vertical lengths.
  • the size of the divided block is (N >> 1) ⁇ (N >> 1), which may be quad tree splitting and binary tree splitting according to the configuration of the present example.
  • this example may be an example in which overlapping of quad tree partitioning and binary tree partitioning is possible.
  • whether or not to perform binary tree partitioning may be determined according to the quad tree partitioning result given priority.
  • the quad tree split is performed, the binary tree split is not performed.
  • the binary tree split may be performed. If the quad tree split is not performed, even if the split is possible under the above configuration, the quad tree split may not be possible further.
  • the partition information of the binary tree of this example may be composed of a plurality of partition flags.
  • Some flags may be split flags (corresponding to x in x / y in this example) and some flags may be split-direction flags (corresponding to y in x / y in this example, where x may determine whether to generate y information).
  • the split flag may have a setting similar to quad tree split. In this example, the horizontal division and the vertical division do not overlap each other.
  • '-' may have a setting similar to the above example. According to the setting, the division flag of the lower left block may be generated like the lower left block of 32b.
  • the lower right block assumes that the binary tree partitioning allowed depth is 5.
  • the maximum and minimum values of the binary tree split are N ⁇ N, (N >> 2) ⁇ (N >> 3)).
  • it may be split into two blocks having a length of 1/2 of a horizontal or vertical length.
  • the division flag setting of this example may be the same as the lower left block.
  • '-' may have a setting similar to the above example.
  • This example shows a case where the minimum horizontal and vertical values of the binary tree are set differently.
  • the partition flag of the lower right block may occur like the lower right block of 32b.
  • the division method and the division setting according to the division can be determined, and the division process can be performed accordingly. Can be.
  • 33 is an exemplary diagram for explaining tree-based partitioning according to an embodiment of the present invention.
  • the thick solid line L0 denotes the largest coding block
  • the block partitioned by the thick solid line and the other lines L1 to L5 means the divided coding block
  • the number inside the block is divided.
  • the position of the sub block (following the Raster Scan order in this example), and the number of '-' may mean the division depth of the block
  • the number of boundary lines between the blocks may mean the number of divisions. For example, if it is divided into four (quad tree in this example), then in the order of UL (0) -UR (1) -DL (2) -DR (3), if it is divided into two (binary tree in this example) L Or U (0) -R or D (1), which can be defined at each division depth.
  • An example to be described later illustrates a case where the obtainable coding block is limited.
  • the maximum coding block of 33a is 64 ⁇ 64
  • the minimum coding block is 16 ⁇ 16
  • quad tree partitioning is used.
  • 2-3-0, 2-3-1, 2-3-2 because 2-0, 2-1, 2-2 blocks (16 ⁇ 16 size in this example) are equal to the minimum coding block size.
  • block division information is not generated in blocks 2-0, 2-1, 2-2, and 2-3 because the obtainable block has 16 ⁇ 16 blocks, that is, one candidate group.
  • the maximum coding block of 33b is 64 ⁇ 64
  • the minimum coding block is 8 horizontally or vertically
  • the allowable splitting depth is 3.
  • the 1-0-1-1 (16 ⁇ 16 size in this example, division depth is 3) block satisfies the minimum coding block condition, it may be divided into smaller blocks.
  • it since it is equal to the allowable dividing depth, it may not be divided into higher dividing depth blocks (1-0-1-0-0, 1-0-1-0-1 blocks in this example).
  • the block partitioning information is not generated because the obtainable block has 16x8 blocks, that is, one candidate group.
  • quad tree partitioning or binary tree partitioning may be supported depending on the encoding / decoding.
  • quad tree splitting and binary tree splitting may be supported.
  • one of the above schemes or a mixture thereof may be supported depending on the block size, the partition depth, and the like.
  • the block belongs to the first block range a quad tree may be supported, and if the block belongs to the second block range, binary tree partitioning may be supported.
  • the plurality of division schemes may have at least one setting such as a maximum coding block size, a minimum coding block size, an allowable division depth, and the like.
  • the range may or may not be set by overlapping the ranges of each other.
  • a setting in which one range includes another range may also be possible. Settings for this may be determined according to individual or mixed factors such as a slice type, an encoding mode, and a color component.
  • the division setting may be determined according to the slice type.
  • the partition configuration supported may include partitioning in the range of 128 ⁇ 128 to 32 ⁇ 32 for a quad tree and partitioning in the range of 32 ⁇ 32 to 8 ⁇ 8 for a binary tree.
  • the block division setting supported may support partitioning in the range of 128 ⁇ 128 to 32 ⁇ 32 for quad trees and partitioning in the range of 64 ⁇ 64 to 8 ⁇ 8 for binary trees.
  • the division setting may be determined according to an encoding mode.
  • the partition setting supported may support two partitions in the 64 ⁇ 64 to 8 ⁇ 8 range and an allowable partition depth in the binary tree.
  • the supported partition configuration can support 3 in the range of 32 ⁇ 32 to 8 ⁇ 8 and allowable partition depth in the binary tree.
  • the division setting may be determined according to the color component.
  • the luminance component a division in the range of 256 ⁇ 256 to 64 ⁇ 64 in the case of the quad tree and a division in the range of 64 ⁇ 64 to 16 ⁇ 16 in the case of the binary tree can be supported.
  • the same setting as the luminance component in the quad tree in this example, the length of each block is proportional to the color difference format
  • the binary tree in the range of 64 ⁇ 64 to 4 ⁇ 4
  • the range in the same luminance component may support division in 128 ⁇ 128 to 8 ⁇ 8.4: 2: 0).
  • some blocks may be combined with other blocks to perform one partitioning process.
  • a partitioning process is performed to obtain an optimal block size and shape, which is optimized for the transform block as well as the optimal size and shape of the coding block. Size and shape.
  • the coding block and the transform block may be combined in one unit, the prediction block and the transform block may be combined in one unit, the coding block, the prediction block and the transform block may be combined in one unit, Combination of other blocks may be possible.
  • the generated information will be recorded in the bitstream in at least one unit of a sequence, a picture, a slice, a tile, etc. in the encoder, and the decoder will parse related information from the bitstream.
  • the prediction unit may be classified into intra prediction and inter prediction, and intra prediction and inter prediction may be defined as follows.
  • the intra prediction may be a technique of generating a prediction value from an area where sub / decoding of the current image (eg, a picture, a slice, a tile, etc.) is completed, and the inter screen prediction may include at least one sub / decoding before the current image. It may be a technique of generating a prediction value from a completed image (eg, a picture, a slice, a tile, etc.).
  • the intra prediction may be a technique of generating a prediction value from an area where the encoding / decoding of the current image is completed, but some prediction methods (for example, a method of generating the prediction value from the reference image).
  • Block Matching, Template Matching, etc. ⁇ may be a prediction that excludes
  • the inter prediction may be a technique of generating a prediction value from at least one sub / decoded image.
  • the completed image may be configured to include the current image.
  • the intra prediction may be configured as follows.
  • the intra prediction of the prediction unit may include a reference pixel construction step, a prediction block generation step, a prediction mode determination step, and a prediction mode encoding step.
  • the image encoding apparatus may be configured to include a reference pixel constructing unit, a predictive block generating unit, and a prediction mode encoding unit for implementing a reference pixel constructing step, a predictive block generating step, a prediction mode determining step, and a prediction mode encoding step.
  • FIG. 34 is an exemplary diagram for describing a configuration of a reference pixel used for intra prediction.
  • the size and shape (M ⁇ N) of the current block on which prediction is performed may be obtained from the block splitter, and will be described under the assumption that it is supported in the range of 4 ⁇ 4 to 128 ⁇ 128 for intra prediction.
  • Intra prediction may be generally performed in units of prediction blocks, but may be performed in units of coding blocks, transform blocks, or the like according to the setting of the block partitioner.
  • the reference pixel configuration unit may configure a reference pixel used for prediction of the current block.
  • the reference pixel may be managed through a temporary memory (for example, an array ⁇ Array>, a primary or a secondary array, etc.), generated and removed for each intra-prediction process of the block, and the size of the temporary memory is referred to. It may be determined according to the configuration of the pixel.
  • a temporary memory for example, an array ⁇ Array>, a primary or a secondary array, etc.
  • the reference pixels used for prediction of the current block may include adjacent pixels (Ref_L, Ref_T, Ref_TL, Ref_TR, and Ref_BL in FIG. 34) of the left, top, left, top, and bottom blocks.
  • the reference pixel is generally composed of pixels (FIG. 34 a) of the neighboring block closest to the current block, but may also be configured of other pixels (pixels of the outer line of FIG. 34 b). .
  • a description will be given based on a case in which a reference pixel is composed of pixels of the nearest neighbor block.
  • the reference pixels are p (0, -1) to p (M + N-1, -1), p (-1,0) to p (-1, N + M-1), p (-1 , -1) may be present.
  • 35 is an exemplary diagram illustrating an intra prediction mode of HEVC.
  • 35 prediction modes are shown and may be classified into 33 directional modes and 2 non-directional modes (DC, Planar).
  • the directional mode may be identified by inclination (for example, dy / dx) or angle information.
  • the above example may mean a prediction mode candidate group for a luminance component or a chrominance component.
  • some prediction modes eg, DC, planar, vertical, horizontal, diagonal modes, etc.
  • the corresponding mode may be included as the prediction mode of the chrominance component, or a mode derived from the corresponding mode may be included as the prediction mode.
  • a reconstruction block of another color space in which encoding / decoding is completed may be used for prediction of the current block by using correlation between color spaces, and may include a prediction mode supporting the same.
  • a reconstructed block of luminance components corresponding to the current block may be generated as a prediction block of the current block.
  • a reconstructed block of some color difference components Cb corresponding to the current block of some color difference components Cr may be generated as a prediction block.
  • a reconstructed block of another color space may be generated as a prediction block as it is.
  • the reconstructed block may be generated as a prediction block in consideration of the correlation between color spaces.
  • the prediction block generator may generate a prediction block according to at least one prediction mode, and use a reference pixel based on the prediction mode.
  • the reference pixel may be used in a method such as extrapolation (directional mode) according to a prediction mode, and may be used in a method such as interpolation or average (DC) or copy (non-directional mode). have.
  • the prediction mode determiner performs a process for selecting an optimal mode among a plurality of prediction mode candidate groups.
  • block distortion eg, Distortion of current and reconstructed blocks.
  • a mode that is optimal in terms of coding cost may be determined using a sum-of-absolute difference (SAD), a sum of square difference (SSD), etc., and a rate-distortion technique that takes into account the amount of bits generated according to a corresponding mode.
  • the prediction block generated based on the prediction mode determined through the above process may be transmitted to the subtractor and the adder.
  • the prediction mode encoder may encode the prediction mode selected by the prediction mode determiner.
  • index information corresponding to the prediction mode may be encoded, or the information about the prediction mode may be predicted. That is, the former means a method of encoding the prediction mode as it is without prediction, and the latter means a method of encoding the mode prediction information and the information obtained based on the prediction mode.
  • the former is an example that can be applied to the chrominance component, the latter in the luminance component, it is not limited to this may also be possible in other cases.
  • the prediction related information generated by the prediction mode encoder may be transmitted to the encoder and may be included in the bitstream.
  • the intra prediction may be configured as follows.
  • the intra prediction of the prediction unit may include a prediction mode decoding step, a reference pixel construction step, and a prediction block generation step.
  • the image decoding apparatus may be configured to include a prediction mode decoding unit, a reference pixel construction unit, and a prediction block generation unit that implement the prediction mode decoding step, the reference pixel construction step, and the prediction block generation step.
  • the prediction mode decoder may receive the prediction mode from the decoder to restore the prediction mode of the current block.
  • the prediction mode may be restored from index information corresponding to the prediction mode candidate group or prediction related information of the prediction mode. That is, the former means a method of decoding the prediction mode as it is without prediction, and the latter means a method of decoding the prediction mode and the information obtained based on the prediction mode by performing prediction.
  • the reference pixel component may configure the reference pixel in the same manner as the reference pixel component of the encoder, and a detailed description thereof will be omitted.
  • the prediction block generator may generate the prediction block in the same manner as the prediction block generator of the encoder, and a detailed description thereof will be omitted.
  • the prediction block generated through this process may be transmitted to the adder.
  • a prediction block may be generated using pixels of a spatially neighboring neighboring block, and a reconstruction block of another color space in which sub / decoding is completed may be generated as a prediction block by using correlation between color spaces.
  • This may be an example of being limited to the color difference component, and in the color difference component, a reconstructed block of some color spaces (eg, Cb) may be referred to as a prediction block in another color space (eg, Cr).
  • Cb some color spaces
  • Cr another color space
  • 36 is a conceptual diagram of using a block in which encoding of another color space is completed, as a prediction block, according to an embodiment of the present invention.
  • 37 is a conceptual diagram of a configuration of a color difference component and a luminance component corresponding thereto in some color formats.
  • a prediction block of a current block may be generated by using spatially adjacent reference pixels (c0 to c16 of FIG. 36) in intra-picture prediction of a chrominance component or of luminance component.
  • a prediction block of the current block may be generated using a reference pixel (L in FIG. 36) of a block having the same coding belonging to the same position.
  • the reference pixel When referring to the reconstruction block of the luminance component in the chrominance component, the reference pixel may be obtained according to the color configuration ratio shown in FIG. 37.
  • the prediction pixel may be obtained by using (copying) at least one reference pixel of TL to BR of the luminance component for prediction of the color difference component C.
  • a plurality of pixels of the TL to BR may be used (weighted average) to obtain the prediction pixels.
  • the reconstruction block of the luminance component may be used as it is for the prediction of the chrominance component, or the reconstruction block of the luminance component obtained using the correlation of the color space may be used for the prediction of the chrominance component.
  • Pred_c may be a prediction pixel of a chrominance component
  • Rec_L may be a reconstructed pixel of a luminance component
  • a and b may be parameters representing a correlation of a color space.
  • the correlation parameter may explicitly generate related information or may obtain it implicitly.
  • the parameter information it may be obtained from a reference pixel of an adjacent block of the current block of the color difference component and a reference pixel of an adjacent block of the corresponding block of the luminance component.
  • a prediction mode candidate group of chrominance components may be configured according to whether or not a prediction mode (correlation mode) using a reconstruction block of another color space is used as a reference pixel.
  • the prediction mode candidate group of the chrominance components may be configured in vertical, horizontal, DC, planar, and diagonal modes.
  • the prediction mode candidate group of the color difference component may be configured as vertical, horizontal, DC, planar, and luma modes.
  • the prediction mode candidate group of the chrominance components may be configured as vertical, horizontal, DC, planar, and correlation modes.
  • the prediction mode candidate group of the chrominance components may include a vertical, horizontal, DC, luma mode, and correlation mode.
  • the luma mode may be a mode used as a prediction mode of the reconstruction block of the luminance component.
  • the mode used for the reconstruction block of the luminance component and the mode using the reconstruction block of the luminance component as reference pixels are mixed configurations.
  • a prediction mode candidate group is a prediction mode candidate group.
  • the above example is an example of using the reference pixel of the prediction block by using the correlation of the color space, and examples of other applications and modifications not limited to the above example may be possible.
  • An image resizing process may be performed at an initial stage of encoding, and the size of an encoded picture may be adjusted.
  • the resizing may be performed implicitly without a signal indicating the resizing of the image in the resizing process.
  • the signal indicating the size adjustment may be implicitly activated. That is, whether to perform the scaling may be determined depending on whether the horizontal size of the image is an integer multiple of the horizontal size of the block or the vertical size of the image is an integer multiple of the vertical size of the block.
  • Some setting information for adjusting the size of the image as in the above example may be implicitly determined.
  • the resizing operation, the resizing direction, and the like may be implicitly determined, and the data processing method will be described in an example to be described later.
  • 38A to 38C are diagrams illustrating expansion according to an integer multiple of the size of a block according to an embodiment of the present invention.
  • FIG. 38A illustrates a case in which the horizontal and vertical sizes of the image are not integer multiples of the block
  • FIG. 38B illustrates a case in which the horizontal size of the image is not an integer multiple of the block
  • FIG. 38C illustrates a case in which the vertical size of the image is not an integer multiple of the block.
  • the entire block S1 + S2, T1 + T2, U1 + U2 means the largest coding block.
  • the extended area of the above example may be filled by using a predetermined value or a method of image outer pixel padding.
  • the data may be filled in an extended region by using data of some region in the image having continuity.
  • One preset data processing method may be used or one of a plurality of data processing methods may be selected according to the encoding / decoding setting.
  • the encoding / decoding information may be generated differently according to the data processing method.
  • the coded / decoded information in the example described below will be described assuming split information.
  • FIG. 38A may be an example using a quad tree-based splitting scheme
  • FIGS. 38B and 38C may be examples using a binary tree-based splitting scheme, and lines in each drawing may be partitioned lines according to the splitting result.
  • 39A to 39C are exemplary views illustrating generation of partition information according to block division of FIGS. 38A to 38C. In detail, partition information generated according to quad tree and binary tree split is shown.
  • splitting is performed to generate split information '1'.
  • split depth 1
  • splitting is performed on the upper left block to generate split information '1', and splitting is not performed on the remaining blocks, thereby splitting information '0' may be generated.
  • the dividing depth 2
  • the same dividing result (1-0-0-0) as in the present example when the dividing depth is 1 is obtained.
  • partition information is combined, when the bitstream is configured in increasing order of division depth, partition information of '1-1-0-0-0-1-0-0-0-0' may be generated.
  • 39B and yc when performing the same division as the above example, '0-1-0-1-0-0-0-1-0-0' and '1-0-1-0-0', respectively.
  • Split information of -0-0-0-0-0 ' may be generated.
  • All or part of the bitstream may be encoded according to a data processing method of an extended region.
  • S1, T1, and U1 may be basic regions of an image (or regions having meaningful data), and S2, T2, and U2 may be used to define an image. It may be an extended area (or an area with meaningless data).
  • an optimal partitioning process for example, a process of generating the least partitioning information for obtaining S1 may be performed for the purpose of reducing the complexity of the encoding cost determining process and reducing the amount of bits generated. This may be a partitioning process performed to acquire the S1 region.
  • splitting information '1' generated because splitting is essentially performed to obtain S1 may be omitted.
  • splitting information '1' which occurs because splitting is essentially performed to obtain S1
  • the partition information '0' generated by not performing division may be omitted.
  • the split depth is 2 since the upper left block and the lower left block are regions including S1, the partitioning is performed in consideration of the characteristics of the image, so that the split information '1' and '0' of the upper left and lower blocks including S1 are divided. Is composed of split information, and since split information of the upper right block and the lower right block is an area irrelevant to obtaining S1, the partition information '0' generated by not performing partitioning may be omitted.
  • the partitioning information generated through the partitioning process is '1-1-0-0-0-1-0-0-0-0', except for the information omitted in the partitioning process, the '1-1-0' Only split information corresponding to parentheses in -0-0- (1-0) -0-0-0 'can be encoded. 39B and 39C, split information may be obtained and encoded through the above process.
  • all of the split information generated without the omitted information may be encoded, which may be determined according to the encoding / decoding setting.
  • a preset data processing method for example, a simple removal method
  • a post-processing removal method eg, removal after use in a process of correcting or restoring a part of an image
  • a plurality of data processing methods may be used when performing the reduction, which is a resizing process.
  • a simple post removal method as well as a series of post removal methods may be used, which may be used for the restoration or correction process of the acquired area because the data of the extended area is obtained from the data of some areas in the image.
  • the image may be divided into the basic regions S1, T1, and U1 and the expanded regions S2, T2, and U2.
  • block division may be performed. That is, information omitted in the block division process in the region extended by the outer pixel padding may not occur in this example.
  • the present invention is not limited to the above example, and the same or modified application of other encoding information may be possible, which may be determined according to a data processing method used in an extended region. Can be.
  • a method of decoding a 360 degree image comprises: receiving a bitstream encoded with a 360 degree image, generating a predictive image by referring to syntax information obtained from the received bitstream; Combining the generated prediction image with a residual image obtained by inverse quantization and inverse transformation of the bitstream to obtain a decoded image, and reconstructing the decoded image into a 360 degree image according to a projection format.
  • the syntax information may include projection format information about the 360 degree image.
  • the projection format information may include an ERP (Equi-Rectangular Projection) format in which the 360-degree image is projected in a two-dimensional plane, a CMP (CubeMap Projection) format in which the 360-degree image is projected into a cube, and the 360-degree octahedron It may be information indicating at least one of an OHta (OctaHedron Projection) format projected on the screen and an IcoSahedral Projection (ISP) format on which the 360 degree image is projected on a polyhedron.
  • ERP Equi-Rectangular Projection
  • CMP CubeMap Projection
  • ISP IcoSahedral Projection
  • the reconstructing may include obtaining placement information based on regional packing by referring to the syntax information, and rearranging each block of the decoded image according to the placement information. have.
  • the generating of the prediction image may include performing image expansion on a reference picture obtained by reconstructing the bitstream, and generating a prediction image by referring to the reference image on which the image expansion is performed. It may include.
  • the performing of the image expansion may include performing an image expansion based on a division unit of the reference image.
  • the extended region may be individually generated for each division unit by using the boundary pixels of the division unit.
  • the extended area may be generated using a boundary pixel of a division unit that is spatially adjacent to the division unit to be extended or a division pixel of a division unit having a continuity of the image and the division unit to be extended.
  • the extended image may be generated with respect to the combined region by using boundary pixels of a region in which two or more spatially adjacent division units are combined among the division units. Can be.
  • the extended region may be generated between the adjacent division units by using all the adjacent pixel information of the spatially adjacent division units among the division units.
  • the extended area may be generated using an average value of adjacent pixels of each of the spatially adjacent division units.
  • the generating of the prediction image may include configuring a reference pixel using a boundary pixel adjacent to the current block and having a different color space from a current block to be decoded according to the prediction mode obtained from the syntax information.
  • the method may include generating a prediction block with respect to the current block by performing intra prediction on the screen.
  • the color space may include a color difference component (Cb, Cr) and a luminance component (Y).
  • the reference pixel may be configured using at least one of TL to BR of luminance components.
  • the reference pixel may be configured by a weighted average value of at least two of TL to Br of the luminance component.
  • the prediction mode may include at least one of horizontal, vertical, DC, planar, diagonal, luma mode, and correlation mode.
  • the methods according to the invention can be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium.
  • Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like.
  • the program instructions recorded on the computer readable medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in computer software.
  • Examples of computer readable media may include hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.
  • Examples of program instructions may include high-level language code that can be executed by a computer using an interpreter, as well as machine code such as produced by a compiler.
  • the hardware device described above may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.
  • the above-described method or apparatus may be implemented by combining all or part of the configuration or function, or may be implemented separately.

Abstract

L'invention concerne un procédé pour décoder une image à 360 degrés comprend les étapes consistant à : recevoir un train de bits obtenu par codage d'une image à 360 degrés ; combiner l'image de prédiction générée avec une image résiduelle obtenue par déquantification et transformation inverse du train de bits, de manière à obtenir une image décodée ; et reconstruire l'image décodée en une image à 360 degrés selon un format de projection. Ici, l'étape de génération de l'image de prédiction comprend les étapes consistant à : configurer un pixel de référence à l'aide d'un pixel limite qui a un espace de couleur différent d'un bloc courant à décoder et qui est adjacent au bloc courant, selon un mode de prédiction obtenu à partir des informations de syntaxe ; et effectuer une prédiction intra à l'aide du pixel de référence, de façon à générer un bloc de prédiction pour le bloc courant.
PCT/KR2017/011147 2016-10-06 2017-10-10 Procédé et appareil de codage et de décodage de données d'images WO2018066986A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
KR20160129413 2016-10-06
KR10-2016-0129413 2016-10-06
KR20170090625 2017-07-17
KR10-2017-0090625 2017-07-17

Publications (1)

Publication Number Publication Date
WO2018066986A1 true WO2018066986A1 (fr) 2018-04-12

Family

ID=61831863

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2017/011147 WO2018066986A1 (fr) 2016-10-06 2017-10-10 Procédé et appareil de codage et de décodage de données d'images

Country Status (1)

Country Link
WO (1) WO2018066986A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110958443A (zh) * 2019-12-16 2020-04-03 宁波大学 一种360度视频帧间快速编码方法
WO2020101451A1 (fr) * 2018-11-15 2020-05-22 한국전자통신연구원 Procédé et appareil pour effectuer un codage/décodage au moyen d'une technique de prédiction inter/intra basée sur une région
CN116248895A (zh) * 2023-05-06 2023-06-09 上海扬谷网络科技有限公司 虚拟现实全景漫游的视频云转码方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20060015223A (ko) * 2004-08-13 2006-02-16 경희대학교 산학협력단 영상 코딩 방법, 코딩 장치, 영상 디코딩 방법 및 디코딩장치
KR20060050350A (ko) * 2004-08-13 2006-05-19 경희대학교 산학협력단 파노라마 영상의 움직임 추정 및 보상
KR20140088099A (ko) * 2011-11-25 2014-07-09 인포브릿지 피티이 엘티디 색차 영상 복호화 장치
KR20150068299A (ko) * 2013-12-09 2015-06-19 씨제이씨지브이 주식회사 다면 영상 생성 방법 및 시스템
KR20160032909A (ko) * 2014-09-17 2016-03-25 한화테크윈 주식회사 멀티 영상 전처리 장치 및 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20060015223A (ko) * 2004-08-13 2006-02-16 경희대학교 산학협력단 영상 코딩 방법, 코딩 장치, 영상 디코딩 방법 및 디코딩장치
KR20060050350A (ko) * 2004-08-13 2006-05-19 경희대학교 산학협력단 파노라마 영상의 움직임 추정 및 보상
KR20140088099A (ko) * 2011-11-25 2014-07-09 인포브릿지 피티이 엘티디 색차 영상 복호화 장치
KR20150068299A (ko) * 2013-12-09 2015-06-19 씨제이씨지브이 주식회사 다면 영상 생성 방법 및 시스템
KR20160032909A (ko) * 2014-09-17 2016-03-25 한화테크윈 주식회사 멀티 영상 전처리 장치 및 방법

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020101451A1 (fr) * 2018-11-15 2020-05-22 한국전자통신연구원 Procédé et appareil pour effectuer un codage/décodage au moyen d'une technique de prédiction inter/intra basée sur une région
US11706408B2 (en) 2018-11-15 2023-07-18 Electronics And Telecommunications Research Institute Method and apparatus for performing encoding/decoding by using region-based inter/intra prediction technique
CN110958443A (zh) * 2019-12-16 2020-04-03 宁波大学 一种360度视频帧间快速编码方法
CN110958443B (zh) * 2019-12-16 2021-06-29 宁波大学 一种360度视频帧间快速编码方法
CN116248895A (zh) * 2023-05-06 2023-06-09 上海扬谷网络科技有限公司 虚拟现实全景漫游的视频云转码方法及系统

Similar Documents

Publication Publication Date Title
WO2018066988A1 (fr) Procédé et appareil de codage et de décodage de données d'images
WO2018066983A1 (fr) Procédé et appareil de codage et de décodage de données d'image
WO2018066982A1 (fr) Procédé et appareil de codage et de décodage de données d'image
WO2020190075A1 (fr) Dispositif d'émission de données de nuage de points, procédé d'émission de données de nuage de points, dispositif de réception de données de nuage de points et procédé de réception de données de nuage de points
WO2018066991A1 (fr) Procédé et appareil de codage/décodage d'images
WO2019235797A1 (fr) Procédé et dispositif de traitement de signal vidéo à l'aide d'une transformée réduite
WO2018048078A1 (fr) Procédé de codage/décodage d'image multivue synchronisée à l'aide d'informations de structure spatiale et appareil associé
WO2020189943A1 (fr) Dispositif d'émission de données de nuage de points, procédé d'émission de données de nuage de points, dispositif de réception de données de nuage de points et procédé de réception de données de nuage de points
WO2019017651A1 (fr) Procédé et appareil de codage/décodage d'image
WO2020222617A1 (fr) Procédé de codage/décodage de signal vidéo et dispositif associé
WO2020009434A1 (fr) Procédé et dispositif de traitement de signal vidéo sur la base d'une transformation secondaire
WO2019235887A1 (fr) Procédé de réalisation d'un codage d'indice de transformation sur la base du mode de prédiction intra et dispositif correspondant
WO2020213963A1 (fr) Procédé de codage/décodage de signal d'image, et dispositif associé
WO2021066598A1 (fr) Procédé de codage d'image basé sur une transformée, et appareil associé
WO2020005045A1 (fr) Procédé et appareil d'inter-prédiction basée sur un mode de fusion
WO2018066990A1 (fr) Procédé et appareil de codage/décodage de données d'images
WO2019009590A1 (fr) Procédé et dispositif de décodage d'image au moyen d'une unité de partition comprenant une région supplémentaire
WO2019190140A1 (fr) Procédé et dispositif de codage/décodage d'image
WO2021002558A1 (fr) Dispositif de transmission de données de nuage de points, procédé de transmission de données de nuage de points, dispositif de réception de données de nuage de points et procédé de réception de données de nuage de points
WO2020005046A1 (fr) Procédé et appareil d'encodage et de décodage d'images
WO2018066986A1 (fr) Procédé et appareil de codage et de décodage de données d'images
WO2018066980A1 (fr) Procédé et appareil de codage/décodage de données d'image
WO2021029511A1 (fr) Dispositif de transmission de données de nuage de points, procédé de transmission de données de nuage de points, dispositif de réception de données de nuage de points et procédé de réception de données de nuage de points
WO2019009667A1 (fr) Procédé de traitement d'image synchronisée et appareil associé
WO2018066987A1 (fr) Procédé et appareil de codage/décodage d'images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17858780

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17858780

Country of ref document: EP

Kind code of ref document: A1