KR20180028300A - A method for providing media bitstream of a syncronized multi view video by using spatial layout information - Google Patents

Abstract

A method for operating a media stream providing apparatus according to an embodiment of the present invention includes the steps of: acquiring spatial layout information of a synchronized multi-perspective image bitstream; receiving a bitstream request including a perspective index from a client system; and transmitting a perspective media bitstream corresponding to the perspective index to the client system based on the spatial layout information. Accordingly, the present invention can efficiently reduce complexity and a bandwidth.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and a device for providing a synchronized multi-view media stream using spatial structure information,

The present invention relates to a method and apparatus for encoding / decoding an image. More particularly, the present invention relates to providing a synchronized multi-view media stream using spatial structure information, a visual decoding method, a coding method, and an apparatus therefor.

Recently, as digital image processing and computer graphics technology are developed, researches on virtual reality (VR) technology, which reproduces the real world and realizes it, is being actively studied.

In particular, a recent VR system such as an HMD (Head Mounted Display) not only can provide a three-dimensional stereoscopic image in both sides of a user, but also can track the viewpoint in all directions. Therefore, (VR) image contents.

However, since the 360 VR contents are composed of the multi-view image information of the simultaneous omnidirectionally synchronized spatio-temporal and binocular images, two large images synchronized with respect to the binocular space at all viewpoints in the production and transmission of the image And compresses and transmits them. This increases the complexity and the bandwidth burden. In particular, in a decoding apparatus, an area that is not actually viewed beyond a user's point of view is decoded, thus wasting an unnecessary process.

Accordingly, there is a need for an efficient encoding method in terms of reducing the amount of data to be transferred and the complexity of the video and also in terms of battery consumption of the bandwidth and decoding apparatus.

The present invention has been made to solve the above problems and it is an object of the present invention to provide a method and apparatus for efficiently encoding / decoding a synchronized multi-view image such as a 360-degree camera or a VR image using spatial structure information of synchronized multi- And the like.

According to another aspect of the present invention, there is provided a method of operating a media stream providing apparatus, the method comprising: acquiring SPATIAL LAYOUT INFORMATION of a synchronized multi-view video bitstream; Receiving a bitstream request including a PERSPECTIVE index from a client system; And transmitting to the client system a point-to-point media bit stream corresponding to the point-in-time index based on the spatial structure information.

In addition, as a technical means for accomplishing the above technical object, a media stream providing apparatus according to an embodiment of the present invention includes a spatial structure information managing unit for obtaining SPATAL LAYOUT INFORMATION of a synchronized multi- ; And transmitting, to the client system, a point-in-time media bit stream (PERSPECTIVE MEDIA BITSTREAM) corresponding to the point-in-time index, based on the spatial structure information, when receiving a bit stream request including a PERSPECTIVE index from the client system And a stream providing section.

As a technical means for accomplishing the above technical object, a method of decoding a media stream according to an embodiment of the present invention includes receiving SPATAL LAYOUT INFORMATION of a synchronized multi-view video bit stream from a server ; Acquiring a viewpoint index corresponding to a user's viewpoint (PERSPECTIVE) and transmitting the acquired viewpoint index to the server; Receiving a first point-of-view media bit stream (PERSPECTIVE MEDIA BITSTREAM) corresponding to the viewpoint index from the server; And decoding the received first view media bit stream.

The method may be implemented by a computer-readable recording medium storing a program for execution on a computer.

According to the embodiment of the present invention, the spatial data structure optimized for coding and transmission can be extracted and signaled from the synchronized multi-view image, thereby effectively reducing the transmission data amount, bandwidth, and complexity of the image.

In addition, the decoding unit can receive the synchronized multi-view image at each viewpoint, and when the synchronized multi-view image is received, it is possible to perform optimized decoding for each view according to the signaling information, thereby reducing system waste It is possible to provide an efficient encoding / decoding method and apparatus in terms of complexity and battery consumption.

In addition, according to the embodiment of the present invention, it is possible to support spatial structure information on various types of synchronized images, thereby enabling proper image reproduction according to the specifications of a decoding apparatus, thereby improving device compatibility.

1 shows an overall system architecture according to an embodiment of the present invention.
FIG. 2 is a block diagram of a time-synchronized multi-view image encoding apparatus according to an embodiment of the present invention. Referring to FIG.
3 to 6 are views showing an example of a spatial structure of a synchronized multi-view image according to an embodiment of the present invention.
7 to 9 are diagrams for explaining a signaling method of spatial structure information according to various embodiments of the present invention.
10 is a diagram for explaining a configuration of spatial structure information according to an embodiment of the present invention.
11 to 12 are views for explaining a type index table of spatial structure information according to an embodiment of the present invention.
13 is a view for explaining a view information table of spatial structure information according to an embodiment of the present invention.
14 is a flowchart illustrating a decoding method according to an embodiment of the present invention.
FIGS. 15 and 16 are diagrams illustrating the determination of the scanning order at the decoding end according to the signaling of the spatial structure information according to the embodiment of the present invention.
17 is a diagram for explaining independent sub-images and dependent sub-images that are classified according to signaling of spatial structure information.
FIGS. 18 to 19 show that boundary regions between sub-images are decoded by referring to independent sub-images according to spatial structure information.
20 to 24 illustrate a decoding system and its operation according to an embodiment of the present invention.
25 to 30 are views showing a server system and its operation according to an embodiment of the present invention.
31 to 32 are diagrams for explaining encoding and decoding processing according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when a part is referred to as being "connected" to another part, it is not limited to a case where it is "directly connected" but also includes the case where it is "electrically connected" do.

Throughout this specification, when a member is located on another member, this includes not only when a member is in contact with another member but also when another member is present between the two members.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise. The terms "about "," substantially ", etc. used to the extent that they are used throughout the specification are intended to be taken to mean the approximation of the manufacturing and material tolerances inherent in the stated sense, Accurate or absolute numbers are used to help prevent unauthorized exploitation by unauthorized intruders of the referenced disclosure. The word " step (or step) "or" step "used to the extent that it is used throughout the specification does not mean" step for.

Throughout the present specification, the term of a combination of these in the expression of a mark-up form means a combination or combination of at least one selected from the group consisting of the constituents described in the expression of the mark-up form, Quot; is meant to include one or more selected from the group consisting of

In an embodiment of the present invention, as an example of a method of encoding a synchronized image, standardized jointly by Moving Picture Experts Group (MPEG) and Video Coding Experts Group (VCEG) having the highest coding efficiency among the video coding standards developed so far Coding may be performed using one High Efficiency Video Coding (HEVC), but the present invention is not limited thereto.

Generally, the encoding apparatus includes an encoding process and a decoding process, and the decoding apparatus has a decoding process. The decoding process of the decoding apparatus is the same as the decoding process of the encoding apparatus. Therefore, the encoding apparatus will be mainly described below.

1 shows an overall system architecture according to an embodiment of the present invention.

Referring to FIG. 1, an overall system according to an embodiment of the present invention includes a preprocessing apparatus 10, an encoding apparatus 100, a decrypting apparatus 200, and a post-processing apparatus 20.

The system according to an embodiment of the present invention includes a preprocessing apparatus 10 for preprocessing a plurality of viewpoint images through an operation such as merging or stitching to obtain a synchronized video frame, And outputs the bit stream; a decoding unit (200) for decoding the synchronized video frame by receiving the bit stream; and a decoding unit And a post-processing device 20 for outputting the output of the post-processing device 20 to the display device.

Here, the input image may include a separate individual image, for example, sub-image information at various time points, in which one or more cameras are photographed in time and space synchronized state. Accordingly, the preprocessing apparatus 10 can acquire the synchronized image information by spatially merging or stitching the acquired multi-view sub-image information according to time.

The encoding apparatus 100 scans and predictively encodes the synchronized image information to generate a bitstream, and the generated bitstream can be transmitted to the decoding apparatus 200. In particular, the encoding apparatus 100 according to the embodiment of the present invention can extract spatial structure information from the synchronized image information, and can signal the signal to the decoding apparatus 200.

Here, the spatial layout information may include basic information about the attributes and the layout of each sub-image, as one or more sub-images are merged into one video frame from the preprocessing apparatus 10 . Further, it may further include additional information on the relationship between each of the sub images and the sub images, which will be described later.

Accordingly, the spatial structure information according to the embodiment of the present invention can be transmitted to the decoding apparatus 200. Then, the decoding apparatus 200 can determine the decoding target and decoding order of the bitstream by referring to the spatial structure information and the user viewpoint information, which can lead to efficient decoding.

Then, the decoded video frame is divided into sub-images for each display through the post-processing device 20 and provided to a plurality of synchronized display systems such as an HMD, A multi-view image can be provided.

FIG. 2 is a block diagram of a time-synchronized multi-view image encoding apparatus according to an embodiment of the present invention. Referring to FIG.

2, an encoding apparatus 100 according to an exemplary embodiment of the present invention includes a synchronized multi-view image acquisition unit 110, a spatial structure information generation unit 120, a spatial structure information signaling unit 130, And a transmission and processing unit 150.

The synchronized multi-view image acquisition unit 110 acquires a synchronized multi-view image using synchronized multi-view image acquisition means such as a 360-degree camera. The synchronized multi-view image may include a plurality of time-and-space-synchronized subimages and may be received from the preprocessor 10 or may be received from a separate external input device.

The spatial structure information generation unit 120 divides the synchronized multi-view image into video frames in units of time, and extracts spatial structure information on the video frames. The spatial structure information may be determined according to the attributes and arrangement state of the respective sub images, and may be determined according to information obtained from the preprocessing apparatus 10. [

The spatial structure information signaling unit 130 performs information processing for signaling the spatial structure information to the decoding apparatus 200. For example, the spatial structure information signaling unit 130 may perform one or more processes to be included in the image data encoded by the image encoding unit, to construct a separate data format, or to include the metadata in the metadata of the encoded image .

Then, the image encoding unit encodes the synchronized multi-view image according to the time flow. In addition, the image encoding unit can use the spatial structure information generated by the spatial structure information generation unit 120 as reference information to determine an image scanning order, a reference image, and the like.

Accordingly, the image encoding unit can perform encoding using HEVC (High Efficiency Video Coding) as described above, but it can be improved in a more efficient manner for the synchronized multi-view image according to the spatial structure information.

The transmission processing unit 150 performs one or more conversion and transmission processing for combining the encoded video data and the spatial structure information inserted from the spatial structure information signaling unit 130 and transmitting the combined information to the decoding apparatus 200 .

3 to 6 are views showing an example of a spatial structure and an image configuration of a synchronized multi-view image according to an embodiment of the present invention.

Referring to FIG. 3, a multi-view image according to an exemplary embodiment of the present invention may include a plurality of temporally synchronized and spatially synchronized image frames.

Each frame can be synchronized according to its own spatial layout and can form a layout of sub-images corresponding to one or more scenes, perspectives or views to be displayed at the same time.

Accordingly, the spatial layout information may be configured such that each sub-image constituting the synchronized multi-view image is composed of one input image through merging, stitching or the like, Point images or sub-images, positional information and angle information of the captured camera, and position information of the captured camera when the input image is composed of a plurality of images synchronized at the same time and corresponding to various views corresponding to the same POC) Image information, merging information, number of sub images, scanning order information, acquisition time information, camera parameter information, reference dependency information between sub images, and the like.

For example, as shown in FIG. 4, image information can be photographed through a divergent camera array, and a 360-degree viewable spatial image can be formed through stiching of an array image can do.

As shown in FIG. 4, the images A ', B', C ', ... photographed corresponding to the respective camera arrays A, B, C ... can be arranged in a one- And left and right and top and bottom area relationship information for stitch processing between arranged images can be exemplified as spatial structure information.

Accordingly, the spatial structure information generation unit 120 can extract the spatial structure information including the various attributes from the input image, and the spatial structure information signaling unit 130 may convert the spatial structure information into an optimized Signaling method.

The spatial structure information generated and signaled as described above can be utilized as useful reference information as described above.

For example, if the content photographed through each camera is a pre-stitched image, each of the pre-stitched images is overlapped before encoding to form one scene. On the other hand, the scene can be divided according to each view, and mutual compensation can be performed between the separated images according to the type.

Accordingly, in the case of a pre-stitched image in which one or more images photographed at a plurality of points are merged and stitched into one image in the preprocessing process and transmitted to the input of the encoder, the scene information of the merged and stitched input image, Etc. may be transferred to the encoding step and the decoding step through separate spatial structure information signaling.

Also, even in the case of a non-stitched image image type in which the images obtained from the multi-viewpoints are transmitted to one or more input images at a time synchronized with time and are encoded and decoded, reference and compensation are performed according to the spatial structure information in the encoding and decoding step To this end, various spatial layout information and corresponding data fields may be needed. The data field may be encoded together with the compression information of the input image, or may be included in separate metadata and transmitted.

Further, the data field including the spatial layout information can be utilized in the rendering process of the image post-processing apparatus 20 and the display.

To this end, the data field including the spatial layout information may include position coordinate information and chrominance information obtained at the time of capturing an image from each camera.

For example, when three-dimensional coordinate information and chrominance information (X, Y, Z), (R, G, B) of an image acquired at the time of acquiring image information from each camera are added to each sub- Information, and this information can be utilized in the post-processing and rendering process of the image after performing decoding.

Further, the data field including the spatial layout information may include camera information of each camera.

As shown in FIGS. 5 to 6, one or more cameras for photographing a three-dimensional space and providing a spatial image may be disposed.

For example, as shown in Fig. 5, the position of one or more cameras may be fixed at a central position and each direction may be set in such a manner as to acquire surrounding objects at one point in a three-dimensional space at the time of image acquisition .

Further, as shown in FIG. 6, one or more cameras may be arranged in such a manner as to photograph one object at various angles. At this time, in the VR display device that reproduces the 3D image based on the coordinate information (X, Y, Z) and the distance information at the time of image acquisition, the user's motion information (Up / Down, Left / Right, Zoom in / Zoom Out ), And the like, and decodes or post-processes a part of the corresponding image, thereby enabling a user to restore a desired point-in-time or partial image. As described above, the synchronized multi- In a system such as compression, transmission, reproduction, etc., a separate image conversion tool module or the like may be added to a portion required depending on the type and characteristics of an image, characteristics of a decoding apparatus, and the like.

For example, when the image obtained from the camera is Equirectangular type, the image encoding unit 140 converts the image type into a video type such as Icosahedron / Cube Map through a conversion tool module according to the compression performance and encoding efficiency of the image, So that the encoding can be performed through this. The conversion tool module may be used in the preprocessing apparatus 10 and the post-processing apparatus 20. The conversion information according to the conversion may be included in the spatial structure information, (20) or a VR display device.

Meanwhile, in order to transfer the synchronized multi-view image according to the embodiment of the present invention, a separate VR image compression scheme for supporting the scalability between the encoding apparatus 100 and the decoding apparatus 200 may be required.

Accordingly, in order to compress the VR image in a scalable manner, the encoding apparatus 100 can compression-encode the image in a manner that distinguishes between the base layer and the enhancement layer.

In this method, in compressing a high-resolution VR image obtained by a single input image from various cameras, the base layer compresses the original image, and in the enhancement layer, a single picture is divided into regions such as Slice / Tile Thereby performing encoding for each sub-image.

At this time, the encoding apparatus 100 can perform compression encoding through inter layer prediction, which improves the coding efficiency by using the restored image of the base layer as a reference image.

On the other hand, when the base layer is decoded and the specific image is to be decoded rapidly according to the motion of the user, the decryption unit 200 can decode a part of the enhancement layer and perform partial image decryption according to the user's motion .

In the scalable compression method, the encoding apparatus 100 encodes the base layer. In the base layer, the original image is scaled down or down-sampled at an arbitrary ratio . At this time, in the enhancement layer, the size of the image is adjusted with the same resolution through Scale Up or Upsampling of the restored image of the base layer, and the restored image of the base layer corresponding thereto is utilized as a reference picture Decoding / decoding can be performed.

According to the processing structure supporting such scalability, the decoding apparatus 200 decodes the entire bitstream of the base layer compressed with a low bit or a low resolution, and only enhances some of the entire bitstream according to the motion of the user It can be decoded into a layer. In addition, since decoding of the entire image is not performed, it is possible to restore the VR image with low complexity.

In addition, according to an image compression method that supports a separate scalability with different resolutions, the encoding device 100 can compress an original image or an image according to the intention of the image creator in the base layer, Layer prediction method for performing coding by referring to a reconstructed image of a base layer in a layer.

In this case, the input image of the enhancement layer may be an image obtained by dividing a single input image by an image segmentation method and encoding the input image into a plurality of regions. One divided area may include at most one subimage, and a plurality of divided areas may be composed of one subimage. The compressed bitstream encoded through this division method can process two or more outputs in the service and application stages. For example, in the service, the entire image is restored and output through decryption of the base layer. In the enhancement layer, only a part of the sub-image and a sub-image It can be decoded.

7 to 9 are diagrams for explaining a signaling method of spatial structure information according to various embodiments of the present invention.

As shown in FIGS. 7 to 9, the spatial structure information includes a NAL (NETWORK ABSTRACTION LAYER) on HLS such as a SPS (Sequence Parameter Set) or VPS (VIDEO PARAMETER SET) It can be signaled as one class type of UNIT type.

FIG. 7 illustrates a NAL UNIT type in which a synchronized image encoding flag is inserted according to an embodiment of the present invention. For example, in FIG. 7, a synchronized image encoding method according to an embodiment of the present invention, such as a VIDEO PARAMETER SET A flag can be inserted.

Accordingly, FIG. 8 illustrates an embodiment of inserting a spatial structure information flag into a VPS (VIDEO PARAMETER SET) according to an embodiment of the present invention.

As shown in FIG. 8, the spatial structure information signaling unit 130 according to the embodiment of the present invention may insert a flag for identifying the type of a separate input image on the VPS. The encoding apparatus 100 may insert a flag indicating that the synchronized multi-view image encoding such as the VR content is performed using the vps_other_type_coding_flag through the spatial structure information signaling unit 130 and the spatial structure information is signaled.

In addition, as shown in FIG. 9, the spatial structure information signaling unit 130 according to an embodiment of the present invention can signal that the image is a multi-view synchronized image on an SPS (Sequence Parameter Set).

For example, as shown in FIG. 9, the spatial structure information signaling unit 130 inserts the input image type (INPUT_IMAGE_TYPE) so that the index information of the synchronized multi-view image is included in the SPS and can be transmitted.

Here, if the INPUT_IMAGE_TYPE_INDEX on the SPS is not -1, or the INDEX value is -1, or if its value is designated as 0 and corresponds semantically to -1, then INPUT_IMAGE_TYPE is set to be synchronized again according to the embodiment of the present invention Point image.

In addition, when the type of the input image is a synchronized multi-view image, the spatial structure information signaling unit 130 signals the SPS by including the view information (PERSPECTIVE INFORMATION) in the SPS, thereby generating a part of the spatial structure information of the synchronized multi- May be inserted into the SPS and transmitted. The viewpoint information is information on which the image layout is signaled for each time frame according to the 3D rendering process of the 2D image, and may include order information such as an upper end, a lower end, and a side view.

Accordingly, the decoding apparatus 200 can decode the flag of the VPS or the SPS to identify whether the corresponding image has been encoded using the spatial structure information according to the embodiment of the present invention. For example, in the case of the VPS of FIG. 5, VPS_OTHER_TYPE_CODING_FLAG may be extracted to check whether the corresponding image is a synchronized multi-view image encoded using spatial structure information.

In the SPS of FIG. 9, the PERSPECTIVE_INFORMATION_INDEX information is decoded to identify actual spatial structure information such as layout.

In this case, the spatial structure information may be configured in the form of a parameter. For example, the spatial structure parameter information may be differently included in the HLS such as SPS, VPS, or the syntax may be configured as a separate function, SEI message.

Also, according to one embodiment, spatial structure information may be included in a PICTURE PARAMETER SET (PPS) and transmitted. In this case, attribute information for each sub-image may be included. For example, the independence of sub-images can be signaled. Independence may indicate that the image can be encoded and decoded without reference to another image, and the sub-images of the synchronized multi-view image may include an INDEPENDENT sub-image and a DEPENDENT sub-image. The dependent sub-images can be decoded with reference to the independent sub-images. The spatial structure information signaling unit 130 may signal independent sub-images in the form of an independent sub-image list on the PPS.

In addition, the spatial structure information may be defined and signaled as an SEI message. FIG. 10 illustrates an SEI message as spatial structure information, and parameterized spatial structure information can be inserted using a spatial layout information descriptor.

As shown in FIG. 10, the spatial structure information includes type index information (INPUT IMAGE TYPE INDEX), PERSPECTIVE INFORMATION, camera parameter information (CAMERA PARAMETER), and the like which can represent a spatial layout of an input image. May include at least one of scene angle information (SCEN ANGLE), scene dynamic range information (SCENE DYNAMIC RANGE), independent sub image information (INDEPENDENT SUB IMAGE), and scene time information (SCENE TIME INFORMATION) A variety of information necessary for efficient coding of an image can be further added. Such parameters may be defined as one descriptor type SEI message format, and the decoding apparatus 200 may parse it and utilize the spatial structure information efficiently in decoding, post-processing, and rendering.

As described above, the spatial structure information may be transmitted to the decoding apparatus 200 in the form of SEI or metadata.

Also, for example, the spatial structure information may be signaled by a selection option such as configuration in the encoding step.

As a first option, the spatial structure information may be included in the VPS / SPS / PPS or Coding unit syntax on the HLS according to the coding efficiency on the syntax.

As a second option, the spatial structure information can be signaled at once into metadata in the form of SEIs in syntax.

Hereinafter, with reference to FIG. 11 to FIG. 19, an efficient video encoding and decoding method according to the synchronized multi-view image format according to an embodiment of the present invention will be described in more detail.

As described above, a plurality of view-by-view images generated in the preprocessing step may be synthesized into one input image and then encoded. In this case, one input image may include a plurality of sub images. Each subimage may be synchronized at the same time instant and may correspond to a different view, a visual point of view (PERSPECTIVE), or a scene, respectively. This has the effect of supporting various views in the same POC (PICTURE ORDER COUNT) without using the existing depth information, and the overlapping area between each sub-image is limited to the boundary area.

In particular, the spatial structure information of the input image may be signaled as described above, and the encoding apparatus 100 and the decoding apparatus 200 may be used to perform efficient encoding and decoding by parsing the spatial structure information. That is, the encoding apparatus 100 may process multi-view image encoding using the spatial structure information in the encoding step, and the decoding apparatus 200 may process the decoding using the spatial structure information in the decoding, preprocessing, and rendering steps .

11 to 12 are views for explaining a type index table of spatial structure information according to an embodiment of the present invention.

As described above, the sub images of the input image can be arranged in various ways. Accordingly, the spatial structure information may separately include a table index for signaling the placement information. For example, as shown in FIG. 8, the synchronized multi-view image may include layouts such as CUBIC LAYOUT 4 X 3, CUBIC LAYOUT 3 X 2, ICOSAHEDRON, and EQUIRECTANGULAR, and the spatial structure information may correspond to each layout The table index shown in FIG. 9 may be inserted.

However, the table shown in FIG. 12 is arbitrarily arranged according to the input image, and can be changed according to the coding efficiency, the content distribution of the market, and the like.

Accordingly, the decoding apparatus 200 can parse the separately signaled table index and use it for decoding processing.

Particularly, in the embodiment of the present invention, each of the layout information can be usefully used for partial decoding of an image. That is, sub-image layout information such as CUBIC LAYOUT can be used to distinguish between independent sub-images and dependent sub-images, thereby determining an efficient encoding and decoding scanning sequence or performing partial decoding for a specific time.

13 is a flowchart illustrating a decoding method according to an embodiment of the present invention.

Referring to FIG. 13, the decoding apparatus 200 receives an image bitstream (S101).

Then, the decoding apparatus 200 confirms whether the multi-view image is a synchronized multi-view image (S103).

Here, the decoding apparatus 200 can identify the synchronized multi-view image from the flag signaled from the spatial structure information signaling unit 130 from the image bit stream. For example, the decoding apparatus 200 can identify in advance whether the multi-view image is a synchronized image from the VPS, SPS, or the like as described above.

If it is not a synchronized multi-view image, general full image decoding is performed (S113).

If the decoded image is a synchronized multi-view image, the decoding apparatus 200 decodes the table index from the spatial structure information (S105).

Here, the decoding apparatus 200 can discriminate whether or not it is an EQUIRECTANGULAR image from the table index (S107).

This is because, among the synchronized multi-view images, the EQUIRECTANGULAR image may not be classified as a separate sub-image, and the decoding apparatus 200 decodes the entire image for the EQUIRECTANGULAR image (S113).

If the image is not an EQUIRECTANGULAR image, the decoding apparatus 200 decodes the remaining SPATIAL LAYOUT INFORMATION (S109) and performs image decoding processing based on the spatial structure information (S111).

14 is a view for explaining a view information table of spatial structure information according to an embodiment of the present invention.

The spatial structure information according to an embodiment of the present invention may include a table for PERSPECTIVE INFORMATION.

The encoding apparatus 100 and the decoding apparatus 200 may use the viewpoint information table as information for discriminating reference, encoding, decoding order, and independent sub-images between images.

Also, at the time of rendering in the multi-view display device, the view information table may be transmitted to the system layer of the device together with the decoded image, and the user may use the information to adjust the phase according to the intention of the contents provider So that the user can view the image.

More specifically, the spatial structure information signaling unit 130 can signal a time point PERSPECTIVE of each subimage. In particular, the spatial structure information signaling unit 130 signals only the information on the top, bottom, and forward images according to the type of the image, and information on the remaining sides of the image is transmitted from the decoding apparatus 200 to the top scene, , And Bottom Scene information. Therefore, only a minimum amount of information can be signaled.

FIGS. 15 and 16 are diagrams illustrating the determination of the scanning order at the decoding end according to the signaling of the spatial structure information according to the embodiment of the present invention.

As shown in FIGS. 15 and 16, according to the type index of the spatial structure information, the scanning order of sub images can be transmitted together, and efficient decoding and rendering can be performed through the transmitted scanning order information.

15 and 16 show that the original image is scanned in the order of A-> B-> C-> D-> E-> F.

For signaling, the spatial structure information signaling unit 130 may signal only the order information of some sub-images such as Top view, Bottom view, and Forward View among the scanning sequences. The decoding apparatus 200 may derive the entire sequence using the order information of the sub-images.

In addition, the scanning order may be changed as shown in FIG. 15 or FIG. 16 according to the type, parallelism, and reference structure of the transmitted image.

The scanning order of FIG. 15 may be A-> F-> B-> C-> D-> E, and the scanning order of FIG. 16 is A-> B -> C-> D-> E-> F. This can be useful when determining the scanning order differently considering the coding efficiency.

17 is a diagram for explaining independent sub images and dependent sub images that are classified according to signaling of spatial structure information.

As shown in FIG. 17, each sub-image arranged in the spatial structure can be divided into a dependent sub-image and an independent sub-image in consideration of referenceability and parallelism. Independent sub-images have the property of being decoded without referring to other sub-images, and dependent sub-images can be restored by referring to adjacent independent sub-images or adjacent dependent sub-images.

Thus, independent sub-images may have characteristics that must be encoded or decoded prior to the dependent sub-images.

In one embodiment, the independent sub-images can be encoded or decoded with reference to the same encoded or decoded independent sub-images on the time axis, and the dependent sub-images can be encoded or decoded with reference to independent sub- Or decoded. Also, whether or not it is independent can be signaled as a separate index according to the spatial structure information.

FIGS. 18 to 19 show that boundary regions between sub-images are decoded by referring to independent sub-images according to spatial structure information.

As described above, the encoding apparatus 100 or the decoding apparatus 200 can process encoding and decoding by simultaneously considering the scanning order and the dependency information of the sub-image.

If A and F are independent sub-images as shown in FIG. 18, the scanning order can be transmitted or derived as A -> F -> B -> C -> D -> E, A and F should be preferentially decoded relative to other dependent subimages. When the remaining B, C, D, and E are decoded, neighboring boundary regions of each sub-image can be decoded by referring to independent sub-images. Thus, the boundary region of the previously decoded independent sub-image or the previously decoded dependent sub-image may be referred to in decoding the remaining sub-images.

In addition, the above-described independent sub-images can be used not only for one image frame but also for performing intra / inter coding and decoding in a boundary region of an adjacent picture.

However, as shown in FIG. 19, 1: 1 mapping may not be performed due to different resolutions (in the case of general image, the ratio of the width is wider than the height).

In this case, in order to refer to the adjacent surface, a scale corresponding to the resolution is adjusted through the image processing technique such as rotation and up sampling for the target subimage to refer to coding or decoding of the boundary region .

For example, in the coding / decoding of the upper boundary boundary of C in Fig. 19, the side value of A can be referred to. To this end, the encoder 100 or the decoder 200 upsamples a side value (Height) of A to a ratio corresponding to the width of C to generate a reference block value according to the position, and encodes and decodes Can be performed.

20 to 24 illustrate a decoding system and its operation according to an embodiment of the present invention.

Referring to FIG. 20, the decoding system 300 according to the embodiment of the present invention receives the entire synchronized multi-view image bit stream and the spatial structure information received from the encoding apparatus 100 or an external server, , And configure the client system to provide one or more decoded pictures to the user's virtual reality display device (400).

For this, the decoding system 300 includes a decoding processing unit 310, a user operation analysis unit 320, and an interface unit 330. Although the decryption system 300 is described as a separate system in the present specification, the decryption system 300 and the post-processing apparatus 20 described above for performing the necessary decryption processing and post-processing, A combination of some modules, or may be configured by extending the decoding apparatus 200. Therefore, it is not limited to the name.

Accordingly, the decoding system 300 according to the embodiment of the present invention can selectively decode a part of the entire bitstream based on the spatial structure information received from the encoding apparatus 100 and the user viewpoint information according to the user's motion analysis Can be performed. In particular, according to the selective decoding described with reference to FIG. 20, the decoding system 300 uses the spatial structure information to generate input images having a plurality of viewpoints at the same time (POC, Picture of Count) (PERSPECTIVE). Also, it is possible to perform partial decoding on ROI (Region Of Interest) pictures determined by the user's point of view based on the ROI.

For this purpose, the decoding system 300 may include an interface layer for receiving and analyzing user information, and may include a point of time at which the currently decoded image is supported and a point-of-view mapping, post-processing and rendering of the VR display device 400, Can be performed. More specifically, the interface layer may include one or more processing modules for post-processing and rendering, and an interface 330 and a user motion analyzer 320.

The interface unit 330 may receive motion information from the VR display device 400 worn by the user.

The interface unit 330 may wirelessly or wirelessly transmit at least one of an environmental sensor, a proximity sensor, a motion sensor, a position sensor, a gyroscope sensor, an acceleration sensor, and a geomagnetic sensor of the user's VR display apparatus 400 One or more data communication modules for receiving data.

The user motion analyzer 320 analyzes the user motion information received from the interface 330 to determine the user's viewpoint (PERSPECTIVE), and selects the decoded picture group corresponding to the selection information To the decryption processing unit 310.

Accordingly, the decoding processing unit 310 can set an ROI mask for selecting an ROI (Region Of Intrest) picture based on the selection information transmitted from the user motion analyzing unit 320, and the ROI mask corresponding to the set ROI mask Only the picture area can be decoded. For example, a picture group may correspond to at least one of a plurality of sub-images, or reference images, in the image frame described above.

For example, as shown in FIG. 20, when there are 1 to 8 sub-images of a specific POC decoded by the decoding processing unit 310, the decoding processing unit 310 decodes the sub-image corresponding to the time point (PERSPECTIVE) By processing only the sub image areas 6 and 7, the processing speed and efficiency can be improved in real time.

FIG. 21 illustrates the entire encoded bitstream and the GOP (GROUP OF PICTURES) according to the embodiment of the present invention, and FIG. 22 shows that the picture group to be decoded among the entire bitstream is selectively changed according to the user's viewpoint .

As described above, the decoding system 300 according to the embodiment of the present invention receives the entire encoded bit stream of the received synchronized multi-view image, and outputs the bit stream corresponding to the user's visual view (perspective view) Decoding can be performed.

At this time, the spatial structure information of the synchronized multi-view image can be signaled in the form of SEI or HLS as described above. In particular, according to one embodiment, the decoding system 300 may include a reference structure using an independent sub-image, a dependent sub-image, and a spatial random access picture (SRAP) identified from the signaling information Generation and construction. In addition, some bitstreams can be selected and decoded according to the change of the visual point (PERSPECTIVE) using the reference structure.

For this purpose, the encoding apparatus 100 can select a picture as a spatial random access picture (SRAP) by the NAL type or PPS, and the SRAP picture is used for intraprediction encoding of sub-images of another unselected picture And can be used as a reference picture.

Therefore, the coding apparatus 100 and the decoding apparatus 200 select one or more of the SRAP pictures and perform coding and decoding processing for a picture not selected as SPAP by a specific GOP (Group Of Picture) or NAL type or PPS .

More specifically, for decoding of some bitstream among the entire bitstream, a single input image composed of multi-points for the same time is divided into sub-frames in the SRAP picture, irrespective of decoding of images for different time periods. A picture that can be independently decoded using data redundancy between images, or the like.

In addition, each of the sub-images selected by the SRAP can be encoded / decoded through Intra coding or the like, and at least one SRAP picture in a certain GOP is included for decoding some bitstreams of the entire bitstream .

In addition, the pictures not selected as SRAP pictures can be used for decoding according to the user's point of view through the inter-picture prediction method using the pictures selected by the SRAP picture as reference pictures.

Accordingly, the decoding system 300 can decode one or more sub-images selected by the decoding processing unit 310 corresponding to the user's visual point (PERSPECTIVE) using the reference picture.

On the other hand, the picture selected by the SRAP can be entirely intra-predictively encoded in a single image according to a general encoding and decoding method without reference.

Accordingly, the decoding system 300 or the decoding apparatus 200 needs to store at least one SRAP picture in a DPB (DECODED PICTURE BUFFER). In addition, the sub-image between the first decoded picture or the SRAP picture in one GOP can be decoded using the picture stored in the SRAP as a reference picture.

Accordingly, FIG. 22 shows the change of the picture specified by the SRAP and the selective decoding area according to the change of the user view point. As shown in FIG. 22, the period in which the SRAP picture is decoded may be a predetermined constant time period.

Then, referring to the SRAP picture that is periodically decoded, the sub-image group decoded according to the occurrence time of the USER PERSPECTIVE OF CHANGE may be changed. For example, as shown in FIG. 22, the decoded ROI region may be changed from the first subimage group to the second image subimage group at the time of occurrence of the user viewpoint change.

23 is a diagram for explaining a sub-image decoding method according to an embodiment of the present invention.

As described above, in the method of decoding some pictures according to the user perspective, the dependency of each sub-images can be considered as shown in FIG.

As described above, the encoding apparatus 100 and the decrypting apparatus 200 can give independence to some pictures (sub images) according to the coding efficiency or the scanning order of sub images. The independent sub-images to which independence is imparted can be encoded and decoded in-picture, thereby eliminating dependence on other sub-images.

Meanwhile, the encoding apparatus 100 and the decrypting apparatus 200 may designate the remaining sub-images as a dependent sub-image. Independent sub-images having the same POC as the dependent sub-images may be added to a reference picture list of the dependent sub-images, and the dependent sub-images may be added to the in- (Inter coding) using an independent sub-image having the same POC as a reference picture, or an inter-picture prediction method for independent sub-images having different POCs Lt; RTI ID = 0.0 > and / or < / RTI >

More specifically, corresponding to the dependent sub-images, independent sub-images having different POCs may be added to the reference picture list. The dependent sub-image may be used as a reference picture in inter-picture predictive coding or inter-picture predictive coding referring to independent sub-images added to the list.

Accordingly, decoding using the intra-picture prediction method can be performed using independent sub-images as reference pictures for the dependent sub-images. In addition, the same POC as the current sub-image to be decoded can be designated as a reference picture, and decoding through the inter-picture prediction method can be performed.

For example, since the dependent sub-image has a high similarity in the boundary region with the independent sub-images having the same POC, the encoding apparatus 100 can calculate the boundary area of the dependent sub-image with respect to the boundary sub- It is possible to perform encoding using a prediction method.

At this time, independent sub-images to be referred to may be changed according to a unit such as PPS or SPS on the HLS, and the information about them may be separately signaled. The decoding apparatus 200 may then derive dependent sub-images based on the independent sub-images. In addition, the decoding apparatus 200 may also receive the independence of the entire sub-images in a separate list form .

In addition, according to the SRAP and independence-based encoding and decoding processes as described above, the decoding system 300 can easily select some bitstreams according to a user's perspective among all the bitstreams, and perform adaptive and selective decoding So that it can be efficiently performed.

 In addition, the independent sub-images or dependent sub-images may be signaled or indexed separately and a signal capable of identifying them may be delivered to the decoding apparatus 200. At this time, the decoding apparatus 200 refers to the neighboring sub-images or the previously decoded sub-images in a part or all of the sub-images to be decoded according to the viewpoint of the user, It is possible to guess the mode and perform restoration for the area.

On the other hand, as shown in FIG. 24, in the intra-picture prediction and the inter picture prediction coding as described above, a coding method considering the coding efficiency can be exemplified.

Referring to FIG. 24, in order to measure a distortion such as a PSNR of an image or to measure an error value or the like of an RDO process (encoding process), an encoding device 100 according to an embodiment of the present invention includes an area And perform a metric that applies a different weighting factor.

As shown in FIG. 24, a user viewing a synchronized multi-view image using the VR display can generally have a wide viewing range, but a narrow viewing range. Therefore, the encoding apparatus 100 can enhance the encoding efficiency by converting the multi-view image into the two-dimensional image and setting the low-view region and the low-end region to the low WEIGHT FACTOR REGION.

FIG. 24 shows a user viewpoint region of various synchronized multi-view images, and a low weighting factor application region in the case of a double-viewpoint image of Equirectangular type.

In particular, in order to apply a weight, the encoding apparatus 100 may first perform transcoding of a synchronized multi-view image into a two-dimensional image, and determine a weighting region according to the type of the multi-view image . In addition, the weighted region information may be included in the above-described spatial structure information or may be separately signaled through the HLS. The weighting factor may also be signaled on a separate HLS or transmitted in a list form.

The weight area according to the embodiment of the present invention can be set on the assumption that the area where the human can concentrate is limited on the characteristics of the natural image and the refracted 3D image.

Accordingly, the encoding apparatus 100 can perform encoding by applying a relatively low QP as a base QP to a relatively low weighted region. Also, when the PSNR is measured, the encoding apparatus 100 can measure the PSNR for a relatively low weighted region to a low value. On the other hand, the encoding apparatus 100 can perform a relatively high QP as a Base QP for an area having a relatively high weight or an operation for measuring a PSNR relatively high.

In addition, according to the embodiment of the present invention, for intra-block copy prediction as described above, decoding can be performed with reference to an image having a different viewpoint even in the same POC, and the referenced image can be independently decoded Sub-images can be illustrated.

In particular, the intra prediction for the boundary region can be efficiently applied, and the weighting factor can be used to maximize the coding efficiency for the boundary region. For example, the encoding apparatus 100 designates the weighting factor for allowing a region (PATCH) to be referenced from an independent subimage region to be specified according to the distance, and applying a scaling value to the weighting factor, It can be determined efficiently.

In constructing the MPM (MOST PROBABLE MODE) for intra-picture coding, the encoding apparatus 100 may apply a weight to a boundary region. Therefore, the encoding apparatus 100 can derive the prediction direction of the MPM around the boundary between the sub-images in the image even if the encoding apparatus 100 is not structurally neighboring.

Meanwhile, the encoding apparatus 100 may consider independent sub-images in inter-picture coding (INTER CODING). For example, the encoding apparatus 100 may refer to a co-lacated sub-image corresponding to an independent sub-image. Accordingly, when constructing a reference picture for inter-picture coding, only a sub-image of the image can be added to the reference picture list.

25 to 30 illustrate a server system for multi-view image processing according to another embodiment of the present invention.

25, a server system according to an embodiment of the present invention may include a server 500 and a client system. The client system may include a client decryption system 600 according to an embodiment of the present invention. Can be illustrated.

As described above, the synchronized multi-view image may be composed of images at each view point through processes such as merging, compositing, or stiching among a plurality of sub images, and the encoding process and the signaling process in the encoding device 100 , Spatial structure information can be generated and signaled.

Accordingly, the server 500 according to an exemplary embodiment of the present invention may be a content server that stores and manages synchronized multi-view images, and selectively transmits the synchronized multi-view images according to a request from the client decryption system 600.

Accordingly, the server 500 can store and manage the media bitstream of the synchronized multi-view image, and can provide an optional part of the bitstream according to the request of the client decryption system 600. [

For this purpose, the server 500 can manage the synchronized multi-view images according to spatial structure information at each viewpoint, and extracts an encoded bit stream for each viewpoint according to a request from the client decryption system 300 To the decoding system 300.

More specifically, the server 500 may include a bitstream providing unit 510 and a spatial structure information managing unit 520.

The spatial structure information management unit 520 stores the media bitstreams stored and managed in the server 500 into a perspective N media bitstream according to the extracted spatial layout information data Can be classified.

In addition, the spatial structure information management unit 520 can transmit the spatial structure information to the client decoding system 600 through the bit stream providing unit 510.

The client decryption system 600 is similar to the decryption system 300 described above and includes a user operation analysis unit 630 and a decryption processing unit 610 and a communication unit 640 for receiving the spatial structure information and the bit stream ).

The communication unit 640 may include one or more network modules for receiving the selective bit stream by wire or wirelessly.

When the spatial structure information is received, the client decryption system 600 can process the compatibility of the multi-view image with the perspective of the multi-view image through the user operation analysis unit 630.

The client decryption system 600 may request a bitstream for an arbitrary visual perspective through the communication unit 640 and the server 500 may transmit the bitstream to the bitstream provider 510 ) To provide the requested encoded bit stream at any time. In particular, the viewpoint may vary according to the perspective change of the user analyzed by the user behavior analyzer 630, and accordingly, the server 500 may transmit the bitstream corresponding to the user perspective .

In addition, the decoding processing unit 610 may process the decoding of the bitstream and transmit the decoded bitstream to a rendering system or a multi-view display device. In the server system embodiment as shown in FIG. 22, the decoding processing unit 610 does not need to selectively decode the bitstream, and decodes the bitstream selectively transmitted from the server 500. FIG.

When a perspective change occurs, the server 500 refers to a perspective table included in the spatial structure information according to the request of the client decryption system 600 and generates a perspective corresponding to the perspective table, To the client decryption system 600 through the bitstream providing unit 510. [

In addition, as shown in FIG. 26, in decrypting a part of an image according to a user's viewpoint, different pieces of sound information according to the movement of the user may be decoded in the transmitted data packet.

At this time, an audio signal corresponding to video compression information is mapped to 1: 1 according to each scene, and is packetized or mapped to a video and audio signal according to separately designated information .

As shown in FIG. 26, when packetizing the data of the virtual reality space image (VR Media), the packetized data may include video, audio, and additional information. At this time, The audio signal corresponding to the position and the scene may include the audio signal acquired at the corresponding position and scene. To this end, the VR media data may include a mapping table between video and audio signals, and the media corresponding to the user's motion can be restored through the mapping table. ■

At this time, the additional information metadata transmitted to the decoding system 600 may be generated based on Mapping table information between the compressed data of Video and Audio, and the metadata may be generated in the compressed bitstream of the image, And may be used for final image restoration including a scene of each image or position information of the camera.

27 to 28 are flowcharts for explaining a server operation according to an embodiment of the present invention.

Referring to FIG. 27, the server 500 can identify and manage a video stream for each viewpoint (PERSPECTIVE) by referring to a viewpoint table (PERSPECTIVE TABLE) through the spatial structure information manager 520. FIG.

The server 500 receives a bitstream request including PERSPECTIVE index information from the client system (S201), and searches for a bitstream of one or more time points (PERSPECTIVE) corresponding thereto according to the received index information (S203). Then, the retrieved bit stream can be selected and transmitted to the client system.

As described above with reference to FIG. 11, the spatial structure information for each viewpoint according to the embodiment of the present invention may be configured in the form of a table including scenes and corresponding perspective information.

As shown in FIG. 14, a total of N encoded video streams can be constructed by performing encoding on each time-synchronized multi-view image from Scene A to Scene F, and the server 500 can acquire spatial structure information By using the management unit 520, the N encoded video streams can be divided into separate bit streams. The table of FIG. 14 illustrates a case where the scenes are classified into six scenes, and N is composed of six scenes.

According to the embodiment of the present invention as shown in FIG. 27, a server 500 according to an embodiment of the present invention may include a plurality of viewpoint images (N viewpoints) viewed from various angles by one or more cameras, It is possible to store the compressed bitstreams by encoding N viewpoint images based on the spatial layout information, thereby performing selective transmission.

Here, the range of the N viewpoints can be varied, for example, a viewpoint according to various viewing angles can be determined within a range of a maximum horizontal 360 degrees vertical 180 degrees.

The preprocessing apparatus 10 according to the embodiment of the present invention can perform all or a part of the functions of the encoding apparatus 100 although the encoding apparatus 100 can perform the same.

Accordingly, the encoding apparatus 100 or the preprocessing apparatus 10 performs a point-in-time determination and spatial structure information generation for the input multi-view image (S301), and configures a perspective table (PERSPECTIVE TABLE) based thereon (S303).

Then, the encoding apparatus 100 performs encoding corresponding to N videos based on the spatial structure information, and generates a bit stream corresponding to the N view points (S305).

Thereafter, the server 500 performs classification and management of an image bitstream using the spatial structure information through the spatial structure information management unit 520, and decodes and manages an image bitstream using the decoding system 600 (a client system) The spatial structure information and the metadata of the bit stream are transmitted (S307).

Thereafter, the server 500 receives a PERSPECTIVE TABLE INDEX from the decoding system 300 through the bitstream provider 510 (S309), and searches for a bitstream at a time point corresponding to the viewpoint table index (S311). The searched bitstream may be delivered to the client decryption system 600.

Then, the server 500 determines whether a new viewpoint table index is received through the spatial structure information management unit 520 (S313).

When a new index is received, the server 500 may again provide a bitstream corresponding to the new view index through the bitstream provider 510 (S315).

As described above, the encoding apparatus 100 or the preprocessing apparatus 10 can construct the spatial structure information corresponding to the input image in the pre-coding step. Accordingly, in the encoding step, The image encoding corresponding to the N viewpoints can be processed using the structure information.

When encoding according to the N viewpoints, the encoding apparatus 100 performs encoding using N encoders or N repetitive encodings, such as 0, 1, 2, ... N perspective bitstreams can be generated. Accordingly, the server 500 receives the first, second, ..., N point of view request message (Perspective msg) received from the client through the bitstream providing unit 510 and transmits the first, 2, ... N view bit streams.

FIG. 29 illustrates a client system according to an embodiment of the present invention. The client decryption system 600 according to an embodiment of the present invention includes an interface layer 680, a rendering and post-processing layer 670, a video decoder 610 and a metadata processing unit 650. Each of the processing units may be implemented by a combination of all or some of the functions of the decryption apparatus 200 and the post-processing apparatus 20. [

The metadata processing unit 650 receives the bit stream, and can parse and manage the high level syntax (HLS) of the bit stream, spatial structure information, and other media metadata.

In addition, when the received bitstream includes image information corresponding to one or more viewpoints (PERSPECTIVE), the video decoder 610 transmits a PERSPECTIVE VIDEO DECODING request received from the rendering and post-processing layer 670 And performs image decoding of the bitstream. Accordingly, the decoded picture or group of pictures (set) can be transferred to the rendering and post-processing layer 670.

Here, the video decoder 610 may decode the bit stream selectively transmitted according to the time index transmitted to the server 500. [ However, in the case of the entire bitstream in which the viewpoint index is not specified, the same selective decoding operation as that of the decoding processing unit 310 of the decoding system 300 described above may be performed. This can be determined according to the system administrator setting.

The rendering and post-processing layer 670 receives the view index from the interface layer 680 and transmits a PERSPECTIVE VIDEO DECODING request based on the view index to the video decoder 610, Or a view group-based restored image based on the picture group (set), and transmits the restored image to the interface layer 680.

Meanwhile, the interface layer 680 receives the user motion information from the VR display device 400, analyzes the viewpoint change, determines the viewpoint index according to the viewpoint change, and transmits the determined viewpoint index to the rendering and postprocessing layer 670. The interface layer 680 may provide the restored image received from the rendering and post-processing layer 670 to the VR display device 400 of the user.

30 is a flowchart illustrating an operation of the client decryption system 600 according to an embodiment of the present invention.

Referring to FIG. 30, a client decryption system 600 according to an embodiment of the present invention firstly receives spatial structure information and media data (e.g., basic information such as metadata) from a server 500 through a metadata processing unit 650 (S401).

Then, the client decryption system 600 configures a perspective information table (PERSPECTIVE TABLE) based on the spatial structure information through the metadata processing unit 650 (S403).

Thereafter, the client decryption system 600 detects a change in the viewpoint of the user through the intrusion layer 680 (S405).

Then, the client decryption system 600 transmits the table index detected by the communication unit 640 to the server (S407).

In response to the table index, the client decoding system 600 receives a bitstream at a time point corresponding to the index from the server 500 via the communication unit 640 and decodes it through the video decoder 610 (S409).

Then, the client decryption system 600 performs post-processing and rendering processes such as point-by-point separation of the decoded image through the post-processing and rendering layer 670 (S411).

The client decryption system 600 may be capable of transmitting one or more indices to a server, thereby receiving and decoding one or more perspective bitstreams, and decoding and rendering the perspective bitstreams.

Accordingly, the viewpoint index may include a plurality of indexes corresponding to the user viewpoint, and the client decoding system 600 may include a plurality of viewpoint media bitstreams (PERSPECTIVE MEDIA BITSTREAM) corresponding to the plurality of viewpoint indices in parallel As shown in FIG.

31 to 32 are diagrams for explaining encoding and decoding processing according to an embodiment of the present invention.

31 is a block diagram illustrating a configuration of a moving image encoding apparatus according to an embodiment of the present invention. The sub-image or the entire frame of the synchronized multi-view image according to the embodiment of the present invention is input as an input video signal Can be processed.

31, a moving picture encoding apparatus 100 according to the present invention includes a picture division unit 160, a transform unit, a quantization unit, a scanning unit, an entropy encoding unit, an intra prediction unit 169, An inverse quantization unit, an inverse transform unit, a post-processing unit 171, a picture storage unit 172, a subtraction unit, and an adder unit 168.

The picture division unit 160 analyzes a video signal to determine a prediction mode by dividing a picture into a coding unit of a predetermined size for each largest coding unit (LCU: Largest Coding Unit), and calculates a prediction unit size .

The picture division unit 160 sends the prediction unit to be encoded to the intra prediction unit 169 or the inter prediction unit 170 according to a prediction mode (or a prediction method). Further, the picture division unit 160 sends the prediction unit to be encoded to the subtraction unit.

The picture may be composed of a plurality of slices, and the slice may be composed of a plurality of maximum coding units (LCU).

The LCU can be divided into a plurality of coding units (CUs), and the encoder can add information indicating whether or not to be divided to a bit stream. The decoder can recognize the position of the LCU by using the address (LcuAddr).

The coding unit CU in the case where division is not allowed is regarded as a prediction unit (PU), and the decoder can recognize the position of the PU using the PU index.

The prediction unit PU may be divided into a plurality of partitions. Also, the prediction unit PU may be composed of a plurality of conversion units (TUs).

In this case, the picture dividing unit 160 may send the image data to the subtracting unit in a block unit (for example, PU unit or TU unit) of a predetermined size according to the determined coding mode.

CTB (Coding Tree Block) is used as a unit of video encoding, and the CTB is defined as various square shapes. The CTB is called the coding unit CU (coding unit).

The coding unit (CU) may have a form of a quad tree according to the division. In the case of quadtree plus binary tree (QTBT) partitioning, the coding unit may have a form of a binary tree divided by the quad tree or the terminal node, and the maximum size according to the standard of the encoder is 64 x 256 64 < / RTI >

For example, when the maximum size is 64 × 64, the picture division unit 160 sets the depth to 0 when the depth is 3 when the maximum coding unit is a LCU (Largest Coding Unit) And performs encoding by searching for an optimal prediction unit recursively up to the coding unit (CU). Further, for a coding unit of a terminal node divided into QTBT, for example, a PU (Prediction Unit) and a TU (Transform Unit) may have the same form as the divided coding unit or may have a further divided form.

A prediction unit for performing prediction is defined as a PU (Prediction Unit). Each coding unit (CU) is predicted by a unit divided into a plurality of blocks, and is divided into a square and a rectangle to perform prediction.

The transform unit transforms the original block of the input prediction unit and the residual block, which is the residual signal of the intra prediction unit 169 or the prediction block generated by the inter prediction unit 170. [ The residual block is composed of a coding unit or a prediction unit. A residual block composed of a coding unit or a prediction unit is divided into optimal transform units and transformed. Different transformation matrices may be determined depending on the prediction mode (intra or inter). Also, since the residual signal of the intra prediction has directionality according to the intra prediction mode, the transformation matrix can be adaptively determined according to the intra prediction mode.

The transformation unit can be transformed by two (horizontal, vertical) one-dimensional transformation matrices. For example, in the case of inter prediction, a predetermined conversion matrix is determined.

On the other hand, in case of the intra prediction, when the intra prediction mode is horizontal, the probability that the residual block has the direction in the vertical direction becomes high. Therefore, the DCT-based integer matrix is applied in the vertical direction, Or a KLT-based integer matrix. When the intra prediction mode is vertical, a DST-based or KLT-based integer matrix is applied in the vertical direction and a DCT-based integer matrix is applied in the horizontal direction.

In case of DC mode, DCT-based integer matrix is applied in both directions. Further, in the case of intra prediction, the transformation matrix may be adaptively determined depending on the size of the conversion unit.

The quantization unit determines a quantization step size for quantizing the coefficients of the residual block transformed by the transform matrix. The quantization step size is determined for each coding unit of a predetermined size or larger (hereinafter referred to as a quantization unit).

The predetermined size may be 8x8 or 16x16. And quantizes the coefficients of the transform block using a quantization matrix determined according to the determined quantization step size and the prediction mode.

The quantization unit uses the quantization step size of the quantization unit adjacent to the current quantization unit as a quantization step size predictor of the current quantization unit.

The quantization unit may search the left quantization unit, the upper quantization unit, and the upper left quantization unit of the quantization unit in order, and generate a quantization step size predictor of the current quantization unit using one or two effective quantization step sizes.

For example, the effective first quantization step size searched in the above order can be determined as a quantization step size predictor. In addition, the average value of the two effective quantization step sizes searched in the above order may be determined as a quantization step size predictor, or when only one is effective, it may be determined as a quantization step size predictor.

When the quantization step size predictor is determined, a difference value between the quantization step size of the current encoding unit and the quantization step size predictor is transmitted to the entropy encoding unit.

On the other hand, there is a possibility that the left coding unit, the upper coding unit, and the upper left coding unit of the current coding unit do not exist. On the other hand, there may be coding units that were previously present on the coding order in the maximum coding unit.

Therefore, the quantization step sizes of the quantization units adjacent to the current coding unit and the quantization unit immediately before the coding order in the maximum coding unit can be candidates.

In this case, 1) the left quantization unit of the current coding unit, 2) the upper quantization unit of the current coding unit, 3) the upper left side quantization unit of the current coding unit, 4) . The order may be changed, and the upper left side quantization unit may be omitted.

The quantized transform block is provided as an inverse quantization unit and a scanning unit.

The scanning unit scans the coefficients of the quantized transform block and converts them into one-dimensional quantization coefficients. Since the coefficient distribution of the transform block after quantization may be dependent on the intra prediction mode, the scanning scheme is determined according to the intra prediction mode.

The coefficient scanning method may be determined depending on the size of the conversion unit. The scan pattern may vary according to the directional intra prediction mode. The scan order of the quantization coefficients is scanned in the reverse direction.

When the quantized coefficients are divided into a plurality of subsets, the same scan pattern is applied to the quantization coefficients in each subset. The scan pattern between subset applies zigzag scan or diagonal scan. The scan pattern is preferably scanned to the remaining subsets in the forward direction from the main subset containing the DC, but vice versa.

In addition, a scan pattern between subsets can be set in the same manner as a scan pattern of quantized coefficients in a subset. In this case, the scan pattern between the sub-sets is determined according to the intra-prediction mode. On the other hand, the encoder transmits to the decoder information indicating the position of the last non-zero quantization coefficient in the transform unit.

Information that can indicate the position of the last non-zero quantization coefficient in each subset can also be transmitted to the decoder.

The inverse quantization unit 135 dequantizes the quantized quantized coefficients. The inverse transform unit restores the inversely quantized transform coefficients into residual blocks in the spatial domain. The adder combines the residual block reconstructed by the inverse transform unit with the intra prediction unit 169 or the received prediction block from the inter prediction unit 170 to generate a reconstruction block.

The post-processing unit 171 performs a deblocking filtering process for eliminating the blocking effect generated in the reconstructed picture, an adaptive offset application process for compensating the difference value from the original image on a pixel-by-pixel basis, and a coding unit And performs an adaptive loop filtering process to compensate the value.

The deblocking filtering process is preferably applied to the boundary of a prediction unit and a conversion unit having a size larger than a predetermined size. The size may be 8x8. The deblocking filtering process may include determining a boundary to be filtered, determining a bounary filtering strength to be applied to the boundary, determining whether to apply a deblocking filter, And selecting a filter to be applied to the boundary if it is determined to apply the boundary.

Whether or not the deblocking filter is applied is determined based on i) whether the boundary filtering strength is greater than 0 and ii) whether a pixel value at a boundary between two blocks adjacent to the boundary to be filtered (P block, Q block) Is smaller than a first reference value determined by the quantization parameter.

The filter is preferably at least two or more. If the absolute value of the difference between two pixels located at the block boundary is greater than or equal to the second reference value, a filter that performs relatively weak filtering is selected.

And the second reference value is determined by the quantization parameter and the boundary filtering strength.

The adaptive offset application process is to reduce a distortion between a pixel in the image to which the deblocking filter is applied and the original pixel. It may be determined whether to perform the adaptive offset applying process in units of pictures or slices.

The picture or slice may be divided into a plurality of offset regions, and an offset type may be determined for each offset region. The offset type may include a predetermined number (e.g., four) of edge offset types and two band offset types.

If the offset type is an edge offset type, the edge type to which each pixel belongs is determined and the corresponding offset is applied. The edge type is determined based on the distribution of two pixel values adjacent to the current pixel.

The adaptive loop filtering process can perform filtering based on a value obtained by comparing a reconstructed image and an original image through a deblocking filtering process or an adaptive offset applying process. The adaptive loop filtering can be applied to the entire pixels included in the 4x4 block or the 8x8 block.

Whether or not the adaptive loop filter is applied can be determined for each coding unit. The size and the coefficient of the loop filter to be applied may vary depending on each coding unit. Information indicating whether or not the adaptive loop filter is applied to each coding unit may be included in each slice header.

In the case of the color difference signal, it is possible to determine whether or not the adaptive loop filter is applied in units of pictures. The shape of the loop filter may have a rectangular shape unlike the luminance.

Adaptive loop filtering can be applied on a slice-by-slice basis. Therefore, information indicating whether or not adaptive loop filtering is applied to the current slice is included in the slice header or the picture header.

If the current slice indicates that adaptive loop filtering is applied, the slice header or picture header additionally includes information indicating the horizontal and / or vertical direction filter length of the luminance component used in the adaptive loop filtering process.

The slice header or picture header may include information indicating the number of filter sets. At this time, if the number of filter sets is two or more, the filter coefficients can be encoded using the prediction method. Accordingly, the slice header or the picture header may include information indicating whether or not the filter coefficients are encoded in the prediction method, and may include predicted filter coefficients when the prediction method is used.

On the other hand, not only luminance but also chrominance components can be adaptively filtered. Accordingly, the slice header or the picture header may include information indicating whether or not each of the color difference components is filtered. In this case, in order to reduce the number of bits, information indicating whether or not to filter Cr and Cb can be joint-coded (i.e., multiplexed coding).

At this time, in the case of chrominance components, since Cr and Cb are not all filtered in order to reduce the complexity, it is most likely to be the most frequent. Therefore, if Cr and Cb are not all filtered, the smallest index is allocated and entropy encoding is performed .

When both Cr and Cb are filtered, the largest index is allocated and entropy encoding is performed.

The picture storage unit 172 receives the post-processed image data from the post-processing unit 171, and restores and restores the pictures in units of pictures. The picture may be a frame-based image or a field-based image. The picture storage unit 172 has a buffer (not shown) capable of storing a plurality of pictures.

The inter-prediction unit 170 performs motion estimation using at least one reference picture stored in the picture storage unit 172, and determines a reference picture index and a motion vector indicating a reference picture.

Then, based on the determined reference picture index and motion vector, a prediction block corresponding to the prediction unit to be coded is extracted from the reference pictures used for motion estimation among a plurality of reference pictures stored in the picture storage unit 172 and output .

The intra prediction unit 169 performs intraprediction encoding using the reconstructed pixel values in the picture including the current prediction unit.

The intra prediction unit 169 receives the current prediction unit to be predictively encoded and selects one of a predetermined number of intra prediction modes according to the size of the current block to perform intra prediction.

The intraprediction unit 169 adaptively filters the reference pixels to generate intra prediction blocks. If reference pixels are not available, reference pixels may be generated using available reference pixels.

The entropy encoding unit entropy-codes quantization coefficients quantized by the quantization unit, intra prediction information received from the intra prediction unit 169, motion information received from the inter prediction unit 170, and the like.

Although not shown, the inter prediction coding apparatus may include a motion information determination unit, a motion information coding mode determination unit, a motion information coding unit, a prediction block generation unit, a residual block generation unit, a residual block coding unit, and a multiplexer .

The motion information determination unit determines the motion information of the current block. The motion information includes a reference picture index and a motion vector. The reference picture index indicates any one of the previously coded and reconstructed pictures.

And indicates one of the reference pictures belonging to the list 0 (L0) when the current block is unidirectionally inter-predictive-coded. On the other hand, when the current block is bi-directionally predictive-coded, a reference picture index indicating one of the reference pictures of the list 0 (L0) and a reference picture index indicating one of the reference pictures of the list 1 (L1) .

In addition, when the current block is bi-directionally predictive-coded, it may include an index indicating one or two pictures among the reference pictures of the composite list LC generated by combining the list 0 and the list 1.

The motion vector indicates the position of the prediction block in the picture indicated by each reference picture index. The motion vector may be a pixel unit (integer unit) or a sub-pixel unit.

For example, it may have a resolution of 1/2, 1/4, 1/8 or 1/16 pixels. When the motion vector is not an integer unit, the prediction block is generated from the pixels of the integer unit.

The motion information encoding mode determination unit determines whether motion information of the current block is coded in a skip mode, a merge mode, or an AMVP mode.

The skip mode is applied when there is a skip candidate having the same motion information as the current block motion information, and the residual signal is zero. The skip mode is also applied when the current block is the same size as the coding unit. The current block can be viewed as a prediction unit.

The merge mode is applied when there is a merge candidate having the same motion information as the current block motion information. The merge mode is applied when there is a residual signal when the current block is different in size from the coding unit or the size is the same. The merge candidate and the skip candidate can be the same.

AMVP mode is applied when skip mode and merge mode are not applied. The AMVP candidate having the motion vector most similar to the motion vector of the current block is selected as the AMVP predictor.

The motion information encoding unit encodes motion information according to a method determined by the motion information encoding mode determining unit. When the motion information encoding mode is a skip mode or a merge mode, a merge motion vector encoding process is performed. When the motion information encoding mode is AMVP, the AMVP encoding process is performed.

The prediction block generation unit generates a prediction block using the motion information of the current block. If the motion vector is an integer unit, the block corresponding to the position indicated by the motion vector in the picture indicated by the reference picture index is copied to generate a prediction block of the current block.

However, when the motion vector is not an integer unit, the pixels of the prediction block are generated from the pixels in the integer unit in the picture indicated by the reference picture index.

In this case, in the case of a luminance pixel, a prediction pixel can be generated using an 8-tap interpolation filter. In the case of a chrominance pixel, a 4-tap interpolation filter can be used to generate a predictive pixel.

The residual block generator uses residual blocks of the current block and the current block to generate residual blocks. If the current block size is 2Nx2N, a residual block is generated using a 2Nx2N prediction block corresponding to the current block and the current block.

However, if the current block size used for prediction is 2NxN or Nx2N, a prediction block for each of the 2NxN blocks constituting 2Nx2N is obtained, and the 2Nx2N final prediction block using the 2NxN prediction blocks is calculated Can be generated.

The 2Nx2N residual block may be generated using the 2Nx2N prediction block. It is possible to overlap-smoothing the pixels of the boundary portion to solve the discontinuity of the boundary portion of 2NxN-sized two prediction blocks.

The residual block coding unit divides the generated residual block into one or more conversion units. Then, each conversion unit is transcoded, quantized, and entropy encoded. At this time, the size of the conversion unit may be determined according to the size of the residual block in a quadtree manner.

The residual block coding unit transforms the residual block generated by the inter prediction method using an integer-based transform matrix. The transform matrix is an integer-based DCT matrix.

The residual block coding unit uses a quantization matrix to quantize the coefficients of the residual block transformed by the transform matrix. The quantization matrix is determined by a quantization parameter.

The quantization parameter is determined for each coding unit equal to or larger than a predetermined size. The predetermined size may be 8x8 or 16x16. Therefore, when the current coding unit is smaller than the predetermined size, only the quantization parameters of the first coding unit are encoded in the coding order among the plurality of coding units within the predetermined size, and the quantization parameters of the remaining coding units are the same as the parameters. You do not have to.

The coefficients of the transform block are quantized using a quantization matrix determined according to the determined quantization parameter and the prediction mode.

The quantization parameter determined for each coding unit equal to or larger than the predetermined size is predictively encoded using a quantization parameter of a coding unit adjacent to the current coding unit. A quantization parameter predictor of the current coding unit can be generated by searching the left coding unit of the current coding unit, the upper coding unit order, and using one or two valid quantization parameters available.

For example, a valid first quantization parameter retrieved in the above order may be determined as a quantization parameter predictor. In addition, the first coding unit may be searched in order of the coding unit immediately before in the coding order, and the first validation parameter may be determined as a quantization parameter predictor.

The coefficients of the quantized transform block are scanned and converted into one-dimensional quantization coefficients. The scanning scheme can be set differently according to the entropy encoding mode. For example, in the case of CABAC encoding, the inter prediction encoded quantized coefficients can be scanned in a predetermined manner (zigzag or raster scan in the diagonal direction). On the other hand, when encoded by CAVLC, it can be scanned in a different manner from the above method.

For example, the scanning method may be determined according to the intra-prediction mode in the case of interlacing, or the intra-prediction mode in the case of intra. The coefficient scanning method may be determined depending on the size of the conversion unit.

The scan pattern may vary according to the directional intra prediction mode. The scan order of the quantization coefficients is scanned in the reverse direction.

The multiplexer multiplexes the motion information encoded by the motion information encoding unit and the residual signals encoded by the residual block encoding unit. The motion information may vary depending on the encoding mode.

That is, in the case of skipping or merge, only the index indicating the predictor is included. However, in the case of AMVP, the reference picture index, the difference motion vector, and the AMVP index of the current block are included.

Hereinafter, one embodiment of the operation of the intra predictor 169 will be described in detail.

First, the picture dividing unit 160 receives the prediction mode information and the size of the prediction block, and the prediction mode information indicates the intra mode. The size of the prediction block may be a square of 64x64, 32x32, 16x16, 8x8, 4x4, or the like, but is not limited thereto. That is, the size of the prediction block may be non-square instead of square.

Next, the reference pixels are read from the picture storage unit 172 to determine the intra-prediction mode of the prediction block.

It is determined whether or not the reference pixel is generated by examining whether or not the unavailable reference pixel exists. The reference pixels are used to determine the intra prediction mode of the current block.

If the current block is located at the upper boundary of the current picture, pixels adjacent to the upper side of the current block are not defined. In addition, when the current block is located at the left boundary of the current picture, pixels adjacent to the left side of the current block are not defined.

It is determined that these pixels are not usable pixels. In addition, it is determined that the pixels are not usable even if the current block is located at the slice boundary and pixels adjacent to the upper or left side of the slice are not encoded and reconstructed.

As described above, if there are no pixels adjacent to the left or upper side of the current block, or if there are no pixels that have been previously coded and reconstructed, the intra prediction mode of the current block may be determined using only available pixels.

However, it is also possible to use the available reference pixels of the current block to generate reference pixels of unusable positions. For example, if the pixels of the upper block are not available, the upper pixels may be created using some or all of the left pixels, or vice versa.

That is, available reference pixels at positions closest to the predetermined direction from the reference pixels at unavailable positions can be copied and generated as reference pixels. When there is no usable reference pixel in a predetermined direction, the usable reference pixel at the closest position in the opposite direction can be copied and generated as a reference pixel.

On the other hand, even if the upper or left pixels of the current block exist, the reference pixel may be determined as an unavailable reference pixel according to the encoding mode of the block to which the pixels belong.

For example, if the block to which the reference pixel adjacent to the upper side of the current block belongs is inter-coded and the reconstructed block, the pixels can be determined as unavailable pixels.

In this case, it is possible to generate usable reference pixels by using pixels belonging to the restored block by intra-coded blocks adjacent to the current block. In this case, information indicating that the encoder determines available reference pixels according to the encoding mode must be transmitted to the decoder.

Next, an intra prediction mode of the current block is determined using the reference pixels. The number of intra prediction modes that can be allowed in the current block may vary depending on the size of the block. For example, if the current block size is 8x8, 16x16, or 32x32, there may be 34 intra prediction modes. If the current block size is 4x4, 17 intra prediction modes may exist.

The 34 or 17 intra prediction modes may include at least one non-directional mode and a plurality of directional modes.

The one or more non-directional modes may be a DC mode and / or a planar mode. When the DC mode and the planar mode are included in the non-directional mode, there may be 35 intra-prediction modes regardless of the size of the current block.

At this time, it may include two non-directional modes (DC mode and planar mode) and 33 directional modes.

The planner mode generates a prediction block of the current block using at least one pixel value (or a predicted value of the pixel value, hereinafter referred to as a first reference value) located at the bottom-right of the current block and the reference pixels .

As described above, the configuration of the moving picture decoding apparatus according to an embodiment of the present invention can be derived from the configuration of the moving picture coding apparatus described with reference to FIGS. 1, 2, and 31. For example, The image can be decoded by performing an inverse process of the encoding process as described with reference to FIG.

32 is a block diagram illustrating a configuration of a moving picture decoding apparatus according to an embodiment of the present invention.

32, the moving picture decoding apparatus according to the present invention includes an entropy decoding unit 210, an inverse quantization / inverse transform unit 220, an adder 270, a deblocking filter 250, a picture storage unit 260, An intra prediction unit 230, a motion compensation prediction unit 240, and an intra / inter changeover switch 280.

The entropy decoding unit 210 decodes the encoded bit stream transmitted from the moving picture encoding apparatus into an intra prediction mode index, motion information, a quantized coefficient sequence, and the like. The entropy decoding unit 210 supplies the decoded motion information to the motion compensation prediction unit 240. [

The entropy decoding unit 210 supplies the intra prediction mode index to the intraprediction unit 230 and the inverse quantization / inverse transformation unit 220. Also, the entropy decoding unit 210 supplies the inverse quantization coefficient sequence to the inverse quantization / inverse transformation unit 220.

The inverse quantization / inverse transform unit 220 transforms the quantized coefficient sequence into an inverse quantization coefficient of the two-dimensional array. One of a plurality of scanning patterns is selected for the conversion. One of a plurality of scanning patterns is selected based on at least one of a prediction mode of the current block (i.e., one of intra prediction and inter prediction) and the intra prediction mode.

The intraprediction mode is received from an intraprediction unit or an entropy decoding unit.

The inverse quantization / inverse transform unit 220 restores the quantization coefficients using the selected quantization matrix among the plurality of quantization matrices to the inverse quantization coefficients of the two-dimensional array. A different quantization matrix is applied according to the size of the current block to be restored and a quantization matrix is selected based on at least one of a prediction mode and an intra prediction mode of the current block with respect to the same size block.

Then, the reconstructed quantized coefficient is inversely transformed to reconstruct the residual block.

The adder 270 reconstructs the image block by adding the residual block reconstructed by the inverse quantization / inverse transforming unit 220 to the intra prediction unit 230 or the prediction block generated by the motion compensation prediction unit 240.

The deblocking filter 250 performs deblocking filter processing on the reconstructed image generated by the adder 270. Accordingly, the deblocking artifact due to the video loss due to the quantization process can be reduced.

The picture storage unit 260 is a frame memory for holding a local decoded picture in which the deblocking filter process is performed by the deblocking filter 250.

The intraprediction unit 230 restores the intra prediction mode of the current block based on the intra prediction mode index received from the entropy decoding unit 210. A prediction block is generated according to the restored intra prediction mode.

The motion compensation prediction unit 240 generates a prediction block for the current block from the picture stored in the picture storage unit 260 based on the motion vector information. When motion compensation with a decimal precision is applied, a prediction block is generated by applying a selected interpolation filter.

The intra / inter selector switch 280 provides the adder 270 with a prediction block generated in either the intra prediction unit 230 or the motion compensation prediction unit 240 based on the coding mode.

The current block is reconstructed using the predicted block of the current block restored in this manner and the residual block of the decoded current block.

The moving picture bitstream according to an embodiment of the present invention may include PS (parameter sets) and slice data as a unit used to store coded data in one picture.

A PS (parameter set) is divided into a picture parameter set (hereinafter, simply referred to as PPS) and a sequence parameter set (hereinafter simply referred to as SPS) which are data corresponding to the heads of each picture. The PPS and the SPS may include initialization information required to initialize each encoding, and may include SPATALAYOUT INFORMATION according to an embodiment of the present invention.

The SPS is common reference information for decoding all pictures coded in the random access unit (RAU), and may include a profile, a maximum number of pictures usable for reference, a picture size, and the like.

For each picture coded by the random access unit (RAU), the PPS may include a variable length coding method as reference information for decoding the picture, an initial value of the quantization step, and a plurality of reference pictures.

On the other hand, the slice header SH includes information on the corresponding slice when coding is performed on a slice basis.

The method according to the present invention may be implemented as a program for execution on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include a ROM, a RAM, a CD- , A floppy disk, an optical data storage device, and the like.

The computer readable recording medium may be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner. And, functional programs, codes and code segments for implementing the above method can be easily inferred by programmers of the technical field to which the present invention belongs.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It should be understood that various modifications may be made by those skilled in the art without departing from the spirit and scope of the present invention.

Claims (18)

A method of operating a media stream providing apparatus,
Acquiring SPATALAYOUT INFORMATION of the synchronized multi-view image bitstream;
Receiving a bitstream request including a PERSPECTIVE index from a client system; And
And transmitting to the client system a PERSPECTIVE MEDIA BITSTREAM corresponding to the viewpoint index based on the spatial structure information
A method of operating a media stream providing apparatus. The method according to claim 1,
Further comprising the step of classifying the synchronized multi-view image bitstream by viewpoint based on the spatial structure information
A method of operating a media stream providing apparatus. 3. The method of claim 2,
Wherein the spatial structure information includes a PERSPECTIVE TABLE including scene information corresponding to the viewpoint index,
Wherein said classifying comprises:
And classifying the synchronized multi-view image bitstream into image bitstreams for each viewpoint based on the viewpoint table
A method of operating a media stream providing apparatus. The method according to claim 1,
The PERSPECTIVE index
(PERSPECTIVE) change processed in the client system
A method of operating a media stream providing apparatus. 5. The method of claim 4,
The client system analyzes the motion information of the user device to determine the PERSPECTIVE index
A method of operating a media stream providing apparatus. 5. The method of claim 4,
When the time index received from the client system is changed according to the change of the user view, providing the media bit stream of the second time point corresponding to the changed index to the client system
A method of operating a media stream providing apparatus. The method according to claim 1,
The synchronized multi-
And a plurality of multi-view sub-images in a temporally synchronized video frame are arranged in accordance with a spatial layout.
A method of operating a media stream providing apparatus. 1. A media stream providing apparatus,
A spatial structure information management unit for obtaining SPATIAL LAYOUT INFORMATION of the synchronized multi-view image bit stream; And
The method of claim 1, further comprising: receiving a bitstream request including a PERSPECTIVE index from a client system, the method comprising: generating a bitstream for transmitting a point-in-time media bitstream (PERSPECTIVE MEDIA BITSTREAM) Comprising
A media stream delivery device. 9. The method of claim 8,
Wherein the bitstream-
Based on the spatial structure information, the synchronized multi-view image bitstream is classified into viewpoints
A media stream delivery device. 10. The method of claim 9,
Wherein the spatial structure information includes a PERSPECTIVE TABLE including scene information corresponding to the viewpoint index,
The bitstream provider
The synchronized multi-view image bitstream is classified into image bitstreams at each viewpoint based on the viewpoint table
A media stream delivery device. 9. The method of claim 8,
The PERSPECTIVE index
(PERSPECTIVE) change processed in the client system
A media stream delivery device. 12. The method of claim 11,
The client system analyzes the motion information of the user device to determine the PERSPECTIVE index
A media stream delivery device. 12. The method of claim 11,
The bitstream provider
When the time index received from the client system is changed according to the change of the user view, a media bit stream of the second time point corresponding to the changed index is provided to the client system
A media stream delivery device. 9. The method of claim 8,
The synchronized multi-
And a plurality of multi-view sub-images in a temporally synchronized video frame are arranged in accordance with a spatial layout.
A media stream delivery device. A method of decoding a media stream,
Receiving spatial structure information (SPATIAL LAYOUT INFORMATION) of a synchronized multi-view image bit stream from a server;
Acquiring a viewpoint index corresponding to a user's viewpoint (PERSPECTIVE) and transmitting the acquired viewpoint index to the server;
Receiving a first point-of-view media bit stream (PERSPECTIVE MEDIA BITSTREAM) corresponding to the viewpoint index from the server; And
And decoding the received first view media bit stream
A method of decoding a media stream. 16. The method of claim 15,
Wherein the viewpoint index includes a plurality of indexes corresponding to a user viewpoint,
Wherein the receiving comprises receiving a plurality of point-of-view media bit streams (PERSPECTIVE MEDIA BITSTREAM) corresponding in parallel to the plurality of viewpoint indices
A method of decoding a media stream. 16. The method of claim 15,
Detecting a change in the user viewpoint;
Transmitting a second viewpoint index to the server, the second viewpoint index being determined according to the change of the user viewpoint; And
Receiving a second point-of-view media bit stream (PERSPECTIVE MEDIA BITSTREAM) corresponding to the second point-in-time index from the server; And
And decrypting the received second view media bit stream
A method of decoding a media stream. 16. The method of claim 15,
Processing the decoded first-view media bitstream, and rendering the post-processed video to the user's VR display device
A method of decoding a media stream.

Images