KR20070106318A - Decoder, processing system and processing method for multi-view frame data, and recording medium having program performing this - Google Patents

Abstract

A multi-view image decoder, a system and a method for processing multi-view image data, and a recording medium storing a program for performing the method are provided to make it possible to process a view image using one encoder and one decoder, thereby reducing the complexity of the system. K view buffers(420a-420k) store data corresponding to K views, respectively. A bit sequence distributor(410) rearranges K view image sequences having the same GOP(Group Of Picture) structure to receive a single encoded bit sequence. The bit sequence distributor(410) divides the single bit sequence into view image data units corresponding to respective views to store the view image data units respectively in corresponding view buffers. A scheduler outputs a control signal for processing the view image data stored in the view buffers. An image decoding unit(430) decodes and outputs the view image data stored in the view buffers in response to the control signal.

Description

Multi-view image decoder, multi-view image data processing system, multi-view image data processing method and a recording medium recording a program for performing the same {Decoder, processing system and processing method for multi-view frame data, and recording medium having program performing this }

1A is a diagram illustrating a method of arranging a plurality of cameras for obtaining multiview image data, which is parallel data.

1B is a diagram illustrating a plurality of camera arrangement methods for obtaining multi-view image data, which is arc data.

2 is a simplified block diagram of a multi-view picture encoder in accordance with one preferred embodiment of the present invention.

3 is a diagram illustrating a method of generating a single bit string rearranged of viewpoint image data.

4 is a simplified block diagram of a multi-view picture decoder in accordance with one preferred embodiment of the present invention.

5 is an example of a decoding time of each GOP of a GoGOP.

FIG. 6 is a schematic block diagram of a picture skip part of the multi-view picture decoder of FIG. 4; FIG.

7 is a method for selecting a decoded picture within a GOP according to a preferred embodiment of the present invention.

8 is a method for selecting a decoded picture within a GOP according to another preferred embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

200: multiview image encoder

210: multiview image rearrangement unit

220: image encoder

400: multiview image decoder

410: Bit Train Splitter

420a, 420b,... , 420K: Viewpoint buffer

430: picture decoding unit

440: scheduler

The present invention relates to a multiview image system, and more particularly, to a multiview image data processing system using one encoder and a decoder.

In general, digital data is transmitted from a certain type of transmitting device to a certain type of receiving device. In addition, the transmission apparatus typically includes an encoder for encoding data for transmission, and the reception apparatus includes a decoder for decoding the received data. Various types of digital data such as video data, audio data, audio / video data, etc. may be transmitted from the transmitting device to the receiving device and output through the receiving device.

The dominant video compression and transmission formats come from a family called hybrid block-based motion-compensated transform video coder, an example of which is ITU-T VCEG video coding. Standards, and video coding standards include H.261, MPEG-1, H.262 / MPEG-2 video, H.263, MPEG from the Video Coding Experts Group (VCEG) and ISO / IEC Moving Picture Experts Group (MPEG). -4 visual and in-process draft standard H.264 / AVC. In addition, for various other types of media including still pictures, audio, documents, web pages, etc., encoding and compression standards are also specified to synchronize and multiplex these signals together.

In general, a video stream is composed of three types of frames or pictures (hereinafter, collectively described as the term 'picture'). The three types of pictures are intra picture (I picture: intra frame), predictive picture (P picture: predictive frame), and bidirectional predictive picture (B picture: bi-directionally predictive frame).

I pictures are DCT-encoded only without using motion vector estimation / compensation, P pictures are motion-tagged / compensated with reference to I-picture or other P-picture, and then DCT-coded with the remaining difference data. The B picture uses motion compensation like the P picture, but performs motion estimation / compensation from two pictures on the time axis.

A video stream or sequence is defined by a segment called a group of pictures (GOP), where I, B, B, P, B, B, P,... In the structure formed as described above, the I picture to the next I picture are referred to as a GOP (Group of Picture). Typically, a GOP is composed of a set of pictures with a duration of a preset time (eg, 0.5 seconds) when displayed at the intended speed.

As described above, a medium for transmitting image information such as a video stream has been developed from a two-dimensional terminal technology such as a television. That is, the data amount of image information is increasing as the color image, i.e., the development of a high definition television (e.g., HDTV) from a black and white image to a high definition television (e.g., HDTV). In addition, various contents such as entertainment, games, stereoscopic movies, etc. are being developed according to the development of information and communication technology and user's request. Therefore, current image information is not a planar two-dimensional world but three-dimensional realistic content that includes space, and technology related to three-dimensional image information that can be reproduced as it is to deliver multimedia information of reality and nature itself. Is being developed. One of the technologies related to such three-dimensional image information is a multi-view video.

The multi-view image device is a next-generation image device that delivers images obtained from several cameras to a user so that the user can view the image at a desired view. That is, the user may receive a plurality of viewpoint images rather than one static viewpoint and change the image to an arbitrary viewpoint, thereby viewing the image contents of the viewpoint. The multi-view image system scheme for supplying a multi-view image is generally a simulcast method in which a multi-view input video signal is input into individual compressed bitstreams using the same number of encoders as the number of cameras. Each bit string is transmitted through a MUX (multiplexer), which typically creates a data structure in accordance with the transmission specification and transmits it to the channel. The compressed bit string thus transmitted, that is, the encoded image, is decoded through each decoder at the receiving side and finally delivered to the user. Therefore, as the number of viewpoint cameras increases, the number of encoders and decoders required increases. As the system becomes complicated, it is difficult to implement a practical system for transmitting a multiview image.

Accordingly, the present invention provides a multiview image data processing system capable of transmitting and processing a multiview image using one encoder and a decoder regardless of the number of viewpoint cameras.

The present invention also provides a multiview image encoder and a method for multiview image processing for encoding a single compressed bit string by rearranging the multiview images.

In addition, the present invention provides a multi-view image decoder and method capable of decoding a multi-view image encoded with one single compressed bit string to provide a user with an image corresponding to each viewpoint.

In addition, the present invention provides a multi-view image data processing system using a single encoder and a decoder to reduce the complexity of the overall system by processing a multi-view image, while not affecting the channel bandwidth and the number of macro blocks to be encoded. to provide.

Other objects of the present invention will be readily understood through the following description.

In order to achieve the above objects, according to an aspect of the present invention, the K view buffer for storing data corresponding to the K (random natural number) view (view), respectively; A single bit string obtained by rearranging and encoding K viewpoint image sequences having the same group of picture (GOP) structure is received, and the single bit sequence is divided into viewpoint image data units corresponding to the respective viewpoints to correspond. A bit string divider for storing in each of the time buffers; A scheduler for outputting a control signal for processing the viewpoint image data stored in the viewpoint buffer; And a picture decoder to decode and output the view image data stored in the view buffer, respectively, according to the control signal.

In order to achieve the above objects, according to another aspect of the present invention, each viewpoint image sequence having the same GOP structure is input from K (random natural numbers) cameras, and the respective viewpoint image sequences are divided into GOP units and sequentially A multi-view image rearrangement unit for generating a single bit sequence rearranged into An image encoder which encodes and transmits the single bit string; K view buffers respectively storing data corresponding to the viewpoints of the cameras; A bit string divider configured to receive the single bit string and to separate the single bit string into view image data units corresponding to the respective view points and to store the single bit stream in the corresponding view buffer; A scheduler for outputting a control signal for processing the viewpoint image data stored in the viewpoint buffer; And an image decoder which decodes and outputs the viewpoint image data stored in the viewpoint buffer according to the control signal, respectively.

Preferably, the image encoder may transmit the encoded single bit string through a single channel. The multi-viewpoint image rearranger divides each of the viewpoint image sequences into GOP units, and may rearrange them sequentially by grouping the GOP units in the same time zone, and the viewpoint image data corresponds to the viewpoints of the single bit stream. One of the encoded view image sequences may include pictures belonging to at least one GOP, or may include all of the maximum view image sequences.

The multi-view image decoder or the multi-view image data processing system may further include the following features.

Preferably, the viewpoint image data may include images belonging to at least one GOP among the encoded viewpoint image sequences corresponding to the respective viewpoints of the single bit string, and may include all of the maximum viewpoint image sequences.

Here, the scheduler may include: a utilization calculator configured to receive a decoding time of each viewpoint image data in a GOP of a time zone most recently decoded from the image decoding unit, and calculate a usage rate of the image decoding unit according to the decoding time; And a picture skip unit for determining a picture skip method in a GOP of a time zone to be decoded immediately after the picture decoder from the picture decoder utilization rate. The picture skip unit may increase the number of picture skips in the GOP of the time zone to be decoded immediately after the time as the use rate of the picture decoder in the GOP of the most recently decoded time zone increases for each time point.

The picture skip unit may include: a skip picture determination module configured to determine a number of picture skips in a GOP in a time zone to be immediately decoded immediately after the picture decoding unit in accordance with a utilization rate of the picture decoder in the GOP in a most recently decoded time zone; And a decoded picture selecting module for selecting a picture to be skipped according to the picture skip number.

The decoded picture selecting module may continuously skip a skip start picture and pictures after the skip start picture in a GOP, and the view point image data may be any one of a baseband profile sequence and a main profile sequence.

Alternatively, the decoded picture selection module may discontinuously skip pictures in a GOP, and the view picture data may be a main profile sequence.

Here, the scheduler may further include a viewpoint priority determiner that determines a viewpoint priority in a GOP of a time zone to be decoded immediately after the picture decoder utilization. The viewpoint priority determining unit may make the viewpoint priority in the GOP of the time zone to be decoded immediately after the smaller the utilization rate of the picture decoder in the GOP of the time zone most recently decoded for each viewpoint.

According to another aspect of the present invention, (a) receiving a single bit string obtained by rearranging and encoding a K (arbitrary natural number) view image sequence having the same GOP structure; (b) dividing the single bit stream into view image data units corresponding to each view and storing the single bit string in a corresponding view buffer; (c) decoding the viewpoint image data stored in the viewpoint buffer for each GOP unit of the same time zone; And (d) determining a method of skipping pictures in the GOP of the time zone to be decoded immediately after the decoding time of the GOP of the time zone decoded most recently for each view image data, and sequentially repeating step (c). A multi-viewpoint image data processing method may be provided.

Preferably, the step (d) may increase the number of picture skips in the GOP of the time zone to be decoded immediately after the decoding time of the GOP of the most recently decoded time zone is longer for each view image data.

The step (d) may include: (d1) determining the number of image skips of the time zone to be decoded immediately after the time according to the decoding time of the most recently decoded time zone for each of the viewpoint image data; (d2) may include selecting an image to be skipped according to the number of image skips. Here, the step (d2) may skip the skip start picture and the pictures after the skip start picture in the GOP continuously or discontinuously skip pictures in the GOP.

In addition, in the step (d), the longer the decoding time of the GOP in the most recently decoded time zone for each viewpoint image data, the higher the viewpoint priority in the GOP of the time zone to be decoded immediately after.

Here, the viewpoint image data may include images belonging to at least one GOP of the encoded viewpoint image sequences corresponding to the respective viewpoints of the single bit string, or may include all of the maximum viewpoint image sequences.

In order to achieve the above objects, according to another aspect of the present invention, a viewpoint image data unit corresponding to each viewpoint includes a single bit string obtained by rearranging and encoding K (arbitrary natural numbers) viewpoint image sequences having the same GOP structure. 10. A method of separating and sequentially decoding each GOP unit of a same time zone, the method comprising: (a) receiving decoding time information of a GOP of a most recently decoded time zone; (b) determining a method of skipping pictures in a GOP in a time zone to be decoded immediately after the decoding time; And (c) outputting a signal for controlling the decoding of the GOP in the time zone to be decoded immediately after the image according to the picture skipping method.

Preferably, the step (b) may increase the number of picture skips in the GOP of the time zone to be decoded immediately after the decoding time of the GOP of the most recently decoded time zone increases for each view image data.

Further, the step (b) may include: (b1) determining the number of image skips of the time zone to be decoded immediately after the time according to the decoding time of the most recently decoded time zone for each of the viewpoint image data; (b2) selecting an image to be skipped according to the number of skipped images.

In the step (b2), the skip start picture and the pictures after the skip start picture in the GOP may be continuously skipped or the pictures may be discontinuously skipped in the GOP.

In the step (b), the longer the decoding time of the GOP of the most recently decoded time zone for each of the viewpoint image data, the higher the viewpoint priority in the GOP of the time zone to be decoded immediately afterwards. The single bit string may include pictures belonging to at least one GOP among the encoded view picture sequences corresponding to the respective view points, or may include all of the view picture sequences at a maximum.

In order to achieve the above objects, according to another aspect of the present invention, there is provided a recording medium in which a program of instructions executable by a digital processing apparatus is tangibly embodied and can be read by the digital processing apparatus. A recording medium on which a program for performing the multi-viewpoint image data processing method according to any one of claims 38 to 38 can be provided.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the drawings, similar reference numerals are used for similar elements.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is said to be "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but other components may be present in the middle. It should be understood. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. Hereinafter, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted. In addition, in describing the present invention, when it is determined that the detailed description of the related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

FIG. 1A is a diagram illustrating a plurality of camera arrangement methods for acquiring multiview image data, which is parallel data, and FIG. 1B is a diagram of a plurality of camera arrangement methods for acquiring multiview image data, which is arc data. The figure shown.

A plurality of camera arrangement methods for acquiring multi-view image data, which is parallel data shown in FIG. 1A, may include K (random natural numbers) cameras 120a and 120b based on an arbitrary camera disposed perpendicular to the subject 110. , ..., 120K-1, 120K) are arrange | positioned in a line. In the case of arranging K cameras 120a, 120b, ..., 120K-1, 120K according to the method shown in FIG. 1A, the distance between each camera and the subject is generally not the same. Of course, the distance between the two cameras disposed on both sides and the subject 110 based on the camera disposed at the center may coincide.

A plurality of camera arrangement methods for acquiring multi-view image data, which is arc data shown in FIG. 1B, may include K cameras 120a, 120b,..., 120K on a circumference spaced a certain distance d from the subject 110. -1, 120K). In the case of arranging K cameras 120a, 120b, ..., 120K-1, 120K according to the method shown in FIG. 1B, the distance between each camera and the subject 110 is equal to d.

Each camera disposed in the method illustrated in FIGS. 1A to 1B receives image of a subject input through a lens angle installed as time passes, and generates image information. B, B, P, B, B, P,... It consists of segments called GOP (Group Of Picture).

Each image information generated by the plurality of cameras shown in Figs. 1A to 1B described above is synchronized with each other and has the same GOP structure. Hereinafter, the image information generated by each camera is called a viewpoint image sequence.

Variables to be used below are defined as follows.

N: number of pictures in a GOP. In other words, intra period

M: I / P section

K: number of views. That is, the number of multiview cameras

V _k : View sequence at k-th time point, k ∈ {1, 2,... , K}

GoGOP (Group of GOPs): set of GOPs in the time domain [t1, t1 + N]

D _k : Sum of decoding times of V _k pictures in GoGOP

U _k : Utilization of the picture decoding unit (to be described later) of the V _k pictures in the GoGOP

S: Number of pictures to be decoded in GoGOP

Commonly used moving picture compression techniques include moving picture experts from H.261, H.262, H.263, and the International Organization for Standardization (ISO) recommended by the International Telecommunication Union (ITU). MPEG-1, MPEG-2, MPEG-4, etc. proposed by the Motion Picture Experts Group (MPEG). In recent years, H.264 or MPEG-4 part10 AVC, a next-generation moving picture compression technology, has been enacted jointly by the International Telecommunication Union and the International Organization for Standardization. The H.264 standard has the disadvantage of being difficult to implement because it is more complicated than the existing compression technologies, but it has the advantage of being able to implement high compression rate and easily adapt to various network environments. Based on this progress, it will be described below.

Meanwhile, the H.264 standard defines standard profiles in the form of a baseline profile and a main profile. A profile is a tool that can be used, and the dual baseline profile is mainly used for A / V communication of mobile communication and videophone video conferencing terminals, and the main profile includes broadcasting services such as set-top, TV broadcasting, and DVD storage covering up to HD level Is a set of defining characteristics of terminals using high quality moving picture data.

In general, when the image is classified into I, P, and B pictures as described above, in the present invention, if there is no B picture, the baseline profile is defined, and if there is a B picture, the main profile is defined. Of course, this structure can be applied to MPEG-1, MPEG-2 and MPEG-4.

2 is a schematic block diagram of a multi-view picture encoder according to an exemplary embodiment of the present invention, and FIG.

Referring to FIG. 2, the multi-view frame encoder 200 includes a multi-view image rearranger 210 and an image encoder 220.

The multi-view image rearranger 210 receives a viewpoint image sequence photographed by K cameras 120a, 120b, ..., 120K-1, and 120K, respectively. The viewpoint image sequence is that each camera is synchronized, so that the viewpoint image sequences all have the same GOP structure (eg, IPPPI… if the number of frames in one GOP is 4).

Referring to FIG. 3, cameras corresponding to respective viewpoints have viewpoint image sequences having a plurality of GOPs (for example, when four cameras are present, four viewpoints are V ₁ , V ₂ , V ₃ , and V ₄ ). Create Each viewpoint image sequence is divided into GOP ₁ , GOP ₂ ,... This means that GOPs with the same number are generated in the same time zone. GOP _i (V _k ) indicates the i-th GOP of the k-th view picture sequence V _k . Hereinafter, description will be made based on the fact that K = 4 and N = 4.

The multi-view image rearranger 210 generates one single bit string in which four viewpoint image sequences V ₁ , V ₂ , V ₃ , and V ₄ are rearranged. As shown in Figure 3 (b), the method of rearrangement is GOP ₁ (V ₁ ), GOP ₁ (V ₂ ), GOP ₁ (V ₃ ), GOP ₁ (V ₄ ), GOP ₂ (V ₁ ) , Sort by GOP ₂ (V ₂ ). GoGOP _i (Group of GOPs, 310), which is a group of GOPs in the same time zone, is composed of {GOP _i (V ₁ ), GOP _i (V ₂ ), GOP _i (V ₃ ), GOP _i (V ₄ )}. Each GoGOP is rearranged by time zone order. That is, GoGOP ₁ , GoGOP ₂ , and GoGOP ₃ are arranged in order to generate a single bit string.

One or more pieces of information of the number of cameras (ie, the number of viewpoints), the size of the MPEG GOP, the decoder profile, etc. may be included in a header of a single bit string or generated as a separate bit string and transmitted to a multi-view image decoder to be described later. Do.

The image encoder 220 receives a single bit string in which GoGOP is rearranged by time zone order from the multi-view image rearranger 210 and encodes the H.264 / AVC. Since the encoded single bit string is one bit string, unlike the conventional number of encoders equal to the number of viewpoints for encoding the multi-view image data in a simultaneous manner, a single multi-view image data, that is, K It is possible to encode a viewpoint image sequence. The encoded single bit string is transmitted through the single transport channel 300.

Hereinafter, an apparatus and method for generating a multi-view image by simultaneously decoding a coded single bit string transmitted through a single transmission channel 300 and maintaining the same image quality between respective viewpoints will be described in detail.

4 is a simplified block diagram of a multi-view picture decoder according to an exemplary embodiment of the present invention, and FIG. 5 is an example of a decoding time of each GOP of a GoGOP. FIG. 6 is a schematic block diagram of a picture skip part of the multi-view picture decoder of FIG. 4. FIG. 7 is a method of selecting a decoded picture in a GOP according to a preferred embodiment of the present invention. Another preferred embodiment is a method for selecting a decoded picture in another GOP.

Referring to FIG. 4, the multiview image decoder 400 includes a bit stream divider 410, K view buffers 420a, 420b,..., 420K, an image decoder 430, and a scheduler 440. . The scheduler 440 includes a utilization calculator 441, a viewpoint priority determiner 443, and an image skip unit 445.

The bit string divider 410 receives a single bit string transmitted through a single transmission channel 300. As described above, the single bit string refers to data in which K view image sequences are rearranged and coded by the multi-view image encoder 200.

A separate bit string transmitted along with a single bit string header or a single bit string includes information such as the number of cameras and the size of the MPEG GOP. The bit stream divider 410 determines the number of view buffers to be used based on this information, and divides GOPs sequentially arranged in a single bit stream into view image data units, and corresponds to K corresponding to each camera (ie, view). Are stored in the four viewpoint buffers 420a, 420b, ..., 420K. The viewpoint image data may include all of the viewpoint image sequences by the GOP size when the viewpoint image sequence is the minimum size, and when the maximum size. The viewpoint buffer may store some of the viewpoint image sequences corresponding to one GOP size according to the size of the viewpoint image data, empty the buffer after decoding for each GOP, newly store the next GOP, or store a plurality of GOPs.

Each viewpoint buffer 420a, 420b, ..., 420K stores encoded GOPs corresponding to a sequence of viewpoint images photographed by K cameras. The viewpoint image data is stored in the viewpoint buffer. The viewpoint image data has a minimum size in units of GOP (since the decoding processing reference in the image decoding unit 430 is a GOP), and the maximum size may be the entire viewpoint image sequence of each viewpoint.

When the viewpoint image data has the GOP size that is the minimum unit, the viewpoint image data stored in the viewpoint buffer is decoded according to the priority to be described later in the viewpoint buffer after the decoding is performed in the other viewpoint buffer. Can be immediately deleted, and the viewpoint image data of the GOP of the next time zone can be recorded. Alternatively, each of the viewpoint buffers may be collectively updated with GOPs of the next time zone after decoding of viewpoint image data, that is, GOPs, of each viewpoint in a specific time zone is completed. In addition, various methods of storing and reading the viewpoint image data in the viewpoint buffer may be modified by those skilled in the art.

The image decoding unit 430 reads the GOPs of respective viewpoints stored in the viewpoint buffers 420a, 420b,..., 420K as necessary.

The image decoding unit 430 decodes GOPs of each viewpoint corresponding to the same time zone, that is, data encoded in one GoGOP unit, among viewpoint image data stored in each viewpoint buffer 420a, 420b, ..., 420K. In the decoding process, the decoding time, decoding target picture, skip picture, skip picture number, and the like of the GOP at each time point in the GoGOP may be changed according to the control signal output from the scheduler 440. The decoding time, the decoding target image, the skipped image, and the number of skipped images of the GOP can be adaptively changed by the scheduler 440 which will be described later according to the time zone of the GOP to be decoded.

The scheduler 440 decodes each of the viewpoint image data in the GOP to be decoded immediately after adaptively changing from the decoding time of the viewpoint image data in the most recently decoded GOP in the decoding by the image decoding unit 430, A control signal for controlling the decoding target image, the skipped image, the number of skipped images, and the like is transmitted to the image decoder 430.

The scheduler 440 processes the viewpoint image data of each viewpoint in GOP units.

If there are K viewpoints, it is assumed that the GOPs of each viewpoint are composed of N pictures, and the number of pictures of one GoGOP is K ㅧ N. Therefore, it is difficult to decode all K ㅧ N pictures with a predetermined frame rate (for example, 24 fps, 30 fps) due to problems such as system performance. Therefore, some of the K ㅧ N pictures are dropped / skip (hereinafter referred to as 'skip'). That is, a method of determining an image to skip since only remaining pictures except the skipped picture is decoded will be described later.

As shown in FIG. 5, the utilization calculator 441 uses the GOPs of the same time zone, that is, the GOPs GOP (V ₁ ) and the GOP (V) at the same time within the same GoGOP 310 from the image decoder 430. ₂ ), the most recent respective decoding times D ₁ , D ₂ , D ₃ , ..., D _{K of} GOP (V ₃ ), ..., GOP (V _K )) are input. When the picture decoding unit utilization rate U _k of the viewpoint V _k is _calculated from the most recent decoding time D _k of GOP (V _k ), the following equation (1) is given.

이다. here,

to be.

Art from the sum of a decoding time to determine the decoding time D _k is the proportion of time occupied by V _k to the picture decoding unit usage rate U _k.

The viewpoint priority determiner 443 uses the image decoder portion utilization rate of the GOP of each viewpoint in the most recently decoded time zone calculated by the utilization rate calculator 441 to determine the time zone to be decoded immediately after the image decoder 430. The priority for decoding the GOP at each time point is determined. For example, during the control signal generation in the scheduler 440 after the nth GoGOP processing, the n + 1th GoGOP may already be decoded, and thus the priority calculated using the decoding time in the nth GoGOP processing or described later. The number of skip pictures and the skip pictures may be n + 2th and n + 3th GoGOPs instead of the n + 1th.

The priority of the time point to be decoded in the GOP to be decoded immediately after the decoded part in the most recently decoded GOP is dynamically changed for each GOP. The point at which the picture decoding unit utilization rate of the most recently decoded time zone is low means that a small number of pictures are decoded compared to the time point where the value is relatively large. The reproduction frame rate is kept equal, and each viewpoint has an equal decoding time on the time axis, and each viewpoint has an equal reproduction opportunity. For example, calculating a picture decoding unit usage rate of each point within a decoded GOP to the last as shown in Table 1 below, and of which to be decoded immediately after with respect to time of V ₄ with the smallest picture decoding unit utilization GOP To assign the highest priority when decoding.

Viewpoint Utilization rate of picture decoder of most recently decoded GOP Priority in GOP to be Decrypted Immediately V ₁ 0.35 4 V ₂ 0.24 3 V ₃ 0.21 2 V ₄ 0.20 One

The determination result of the viewpoint priority determiner 443 is input to the image decoder 430 to determine the decoding order of each viewpoint in decoding by the image decoder 430.

The picture skip unit 445 determines a picture skip method when decoding the GOP in the next time zone by utilizing the picture decoder utilization rate of the GOP in the current time zone at each time point. Before the decoding of the GOP of the next time zone starts, all the viewpoints are decoded fairly evenly by dropping a relatively large number of images having a high rate of use of the image decoding unit. In other words, when the decoding of the pictures of all the time points of the buffer is finished, the picture decoding unit utilization rate is recalculated to determine the skip of the picture of the GOP of the next time zone.

The image skip part 445 is demonstrated with reference to FIG. The image skip unit 445 may include a skipped image determination module 510 that determines the number of skipped images at each time point, and a decoded image selection module 520 that selects the decoded images.

The skipped picture determination module 510 determines the number of skipped pictures in the GOP in the next time zone from the picture decoder utilization rate of the GOP in the current time zone. The method of determining the number of image skips is as follows.

There is one GOP (V _k ) composed of N pictures, and skipping F _l , the l-th picture, causes subsequent F _{L + 1} , F _{L + 2} , F _{L + 3,.} Assume that F _{N is} also skipped. In this case, the number of pictures actually decoded is (l-1). Therefore, the total number of pictures to be totally decoded in GoGOP is expressed by Equation 2 below.

Here, l _k is the first picture number skipped in the GOP (V _k ). For example, l ₁ is the first picture number skipped in the GOP (V ₁ ) at the first time point, and l ₂ is the first picture number skipped in the GOP (V ₂ ) at the second time point.

The optimum performance in GoGOP is when K ㅧ N pictures, i.e., all pictures are decoded. However, since a system that is capable of decoding all pictures in an actual multi-view picture is impossible, it is assumed that the number of pictures to be decoded in GoGOP is S. S is any value included in [0, K ㅧ N], and can be increased or decreased depending on system performance. Therefore, Equation 2 above obtains the same result as Equation 3 and 4 below.

= [S, 1]로 역매핑된다. 즉, 서로 수직하는 가로축과 세로축을 가정할 때, 가로축은 화상 복호화부 이용률 U 을, 세로축은

를 의미한다. U가 0일 때

는 S에, U가 1일 때

는 1에 대응하는 일차함수를 생각하면, 화상 복호화부 이용률 U에 따른

이 결정될 수 있다. Equation 2 above is used again to obtain a value of l _k that satisfies the same result as Equation 4 above. L _k is obtained using inverse linear mapping according to the calculated picture decoding unit utilization rate U _k . That is, U = [0, 1] is

= Reverse mapped to [S, 1]. That is, assuming that the horizontal axis and the vertical axis perpendicular to each other, the horizontal axis is the image decoding unit utilization rate U,

Means. When U is 0

Is in S, when U is 1

Considering a linear function corresponding to 1,

This can be determined.

This reduces the number of pictures to be decoded in the GOP in the time zone to be decoded immediately after the use of the picture decoding unit in the GOP in the most recently decoded time zone is high. In this case, the small number of pictures to be decoded means that the first picture number to be skipped, l _k, is small. 1 _k , which is the first picture number to be skipped at that time, is in an inverse linear relationship in which the first picture number to be skipped becomes smaller as the picture decoding unit utilization increases.

When expressed as an equation, Equation 5 below.

If the first skipped picture number calculated for all K viewpoints is summed, Equation 6 below is obtained.

는 하기의 수학식 7과 같이 계산된다. First picture number to be skipped in GOP (V _k ) from Equations 4 and 6

Is calculated as in Equation 7 below.

는 플로어(floor) 함수이다. 플로어 함수는

안의 값보다 작거나 같은 최대의 정수를 출력값으로 한다. here,

Is a floor function. Floor function is

The maximum integer less than or equal to the value inside is used as the output value.

값을 이용하여 해당 시점 이후에 복호화될 시간대의 GOP 복호화시 스킵해야 하는 화상의 개수를 결정하는 것이 가능하게 된다. So obtained

It is possible to determine the number of pictures to be skipped during GOP decoding of the time zone to be decoded after the corresponding time point using the value.

The decoded picture selection module 520 determines a picture skip method from the number of skip pictures determined by the skip picture determination module 510. The picture skip method may be continuous skip or discontinuous skip.

값을 스킵 시작 화상 번호로 결정한다. 도 7에 도시된 바와 같이, 하나의 GOP 내에 10개의 화상이 있는 경우, 스킵 시작 화상은 6번째 화상으로 계산되었으면, 스킵 시작 화상 및 스킵 시작 화상 이후의 화상들(7, 8, 9, 10번째 화상들)은 모두 스킵되도록 한다. 불연속 스킵부(523)는 구해진

값으로부터 하나의 GOP 내에서 스킵되는 화상의 수를 결정하게 된다. 그리고 스킵 화상 수에 따라, 그리고 시각 화상 화질을 이용하여 스킵되는 화상의 간격을 조절하여 불연속으로 스킵 화상을 결정하여 스킵되도록 한다. Continuous skip portion 521 is obtained

The value is determined by the skip start picture number. As shown in Fig. 7, when there are 10 pictures in one GOP, the skip start picture and the pictures after the skip start picture (7, 8, 9, 10th) have been calculated as the 6th picture. Pictures) are all skipped. The discontinuous skip portion 523 is obtained

The value determines the number of pictures that are skipped in one GOP. The skipped image is determined to be skipped by adjusting the interval of the skipped image according to the number of skipped images and the visual image quality.

The decoded picture selection module 520 is influenced by the output values from the profile module 610 and / or the cost function module 620 in determining the picture skipping method. In this case, the profile module 610 and / or the cost function module 620 may be further included in the image skip unit 445.

The profile module 610 determines whether the decoded picture is a base fine profile sequence or a main profile sequence. In general, since the baseline profile of H.264 / AVC uses only I and P pictures, decoding is relatively simple. Since the P picture is encoded by forward perdiction, the previous reference picture must exist to be reconstructed. The following shows the GOP structure used in the baseline profile. N = 6 and M = 1.

I ₁ P ₁ P ₂ P ₃ P ₄ P ₅ I ₂ P ₆ P ₇ P ₈ .

Image skip is performed with the following characteristics.

1) If I ₁ image is skipped, then P ₁ ,. , P ₅ pictures are all skipped.

2) If P ₁ image is skipped, P ₂ ,. , P ₅ pictures are all skipped. Therefore, if the P _i picture is skipped, all P pictures of the corresponding GOP from P _{i + 1} are skipped.

3) Therefore, when three pictures in the current GOP and two pictures in the next GOP are decoded, I ₁ P ₁ P ₂ and I ₂ P ₆ are reproduced.

Here, only the continuous skip method may be used as the baseline profile in the above-described image skip method. The pictures to be decoded occur in succession, and likewise in succession to the skipped pictures.

가 정해지면, 즉 복호화되는 화상의 수가 결정되면, 화상 간의 예측 구조에 따라 연속적 또는 불연속적으로 복호화하고자 하는 화상 또는 스킵하고자 하는 화상을 선택하게 된다. The order of selection in the main profile depends on the picture type. The number of the first frame to be skipped

If is determined, that is, the number of pictures to be decoded is selected, the picture to be decoded continuously or discontinuously or the picture to be skipped is selected according to the prediction structure between pictures.

The cost function module 620 determines the visual image quality through a cost function that objectively quantifies the visual image quality, and determines the image skip method.

The cost function is calculated as

1) If the image i belongs to a drop image that occurs continuously and at the same time a k _i -th consecutive image, the cost c _i of the image is k _i .

이다. 2) If the distance between two non-contiguous skip pictures is d _i , the cost c _i is

to be.

이다. 여기서, F는 시퀀스에서 스킵되는 화상들의 집합이고, G는 고려하는 GOP 내의 전체 화상의 집합이다. 3) total cost function

to be. Here, F is a set of pictures skipped in a sequence, and G is a set of all pictures in the GOP under consideration.

The value of the cost function shown in FIGS. 7 and 8 is calculated as in Equations 8 and 9 below.

가

보다 작으며, 비용 함수가 작은 도 8의 방법이 우수한 시각 화상 화질을 가지게 된다. 동일한 개수의 화상을 스킵하지만, 실제 계산되는 비용은 다른 결과를 가져오게 된다. 따라서, 시스템 성능 및 화질에 따라 적응적으로 화상 스킵 방법을 결정하는 것이 가능하다. therefore,

end

The smaller, smaller cost function of FIG. 8 results in superior visual image quality. Skipping the same number of images, but the actual cost of the calculation will result in different results. Therefore, it is possible to adaptively determine the image skip method according to the system performance and the image quality.

The image skip unit 445 determines an image skip method by referring to output values of the profile module 610 and / or the cost function module 620 according to the number of skipped images determined by the skip image determination module 510. Accordingly, the image number 530 to be decoded is determined except for the skipped image, and the image decoding unit 430 is informed of this as a control signal. The image decoding unit 430 decodes pictures corresponding to the decoded picture number 530, which is a form of a control signal, during GOP decoding of a time zone to be immediately decoded.

Hereinafter, the multi-view image data processing method in the multi-view image decoder 400 will be described.

First, the multi-view image decoder 400 rearranges and encodes a K (natural number) viewpoint image sequence having the same GOP structure from the multi-view image encoder 200 through a single transmission channel 300. Receive.

The bit stream divider 410 separates the single bit stream into view image data units and stores the single bit stream in the corresponding view buffers 420a, 420b, ..., 420K.

The image decoding unit 430 decodes the viewpoint image data stored in the viewpoint buffers 420a, 420b,..., 420K for each GOP unit of the same time zone.

In decoding, according to the decoding time of the GOP of the current time zone for each time point, that is, the viewpoint image data, the scheduler 440 determines a method of skipping the picture in the GOP of the next time zone, and repeats decoding of the GOP according to the time sequence. do. Since the image skip method has been described in detail with reference to FIGS. 5 to 8, a detailed description thereof will be omitted.

According to another preferred embodiment of the present invention, a multi-view image data processing system including a multi-view image encoder 200 and a multi-view image decoder 400 may be implemented. The operation of the multiview image encoder 200 and the operation of the multiview image decoder 400 are the same as described above.

A multi-view image data processing method according to another preferred embodiment of the present invention is as follows. A single bit string obtained by rearranging and encoding a sequence of K (arbitrary natural numbers) view images having the same GOP structure at each viewpoint In the method of dividing into view image data units corresponding to and sequentially decoding the GOP units of the same time zone, first, decoding time information of the GOP of the most recently decoded time zone is received.

A picture skip method in the GOP in the time zone to be decoded immediately after the decoding time is determined. Since the image skip method has been described in detail with reference to FIGS. 5 to 8, a detailed description thereof will be omitted.

Subsequently, a signal for controlling decoding of the GOP in the time zone to be decoded immediately after the picture skip method is output.

A program capable of performing the above-described multiview image data processing method is stored in an external server, and multiview image decoding may be performed by downloading a program through a wired or wireless network and embedding the program in a multiview image decoder.

As described above, the multiview image system according to the present invention can transmit and process a multiview image using one encoder and a decoder regardless of the number of viewpoint cameras.

In addition, by processing a multiview image using one encoder and one decoder, the channel bandwidth and the number of macro blocks to be encoded are not affected while reducing the complexity of the entire system.

In addition, by rearranging the multi-view images, a single compressed bit string may be encoded, decoded, and the user may be provided an image according to each viewpoint. A single compressed bit string can be encoded using MPEG video compression including existing H.264 / AVC.

In addition, an increase in the number of Decoded Picture Buffers (DPBs) is unnecessary as an efficient multi-view image is encoded based on a simultaneous scheme. In the decoding process, it is possible to maintain the same image quality between viewpoints by adaptively changing the decoding priority, the number of skipped images, the skipped image method, etc. in the next time zone according to the utilization rate of the image decoding unit while using a single decoder.

Although the above has been described with reference to a preferred embodiment of the present invention, those skilled in the art to which the present invention pertains without departing from the spirit and scope of the present invention as set forth in the claims below It will be appreciated that modifications and variations can be made.

Claims (39)

K view buffers each storing data corresponding to K (random natural numbers) views; A single bit string obtained by rearranging and encoding K viewpoint image sequences having the same group of picture (GOP) structure is received, and the single bit sequence is divided into viewpoint image data units corresponding to the respective viewpoints to correspond. A bit string divider for storing in each of the time buffers; A scheduler for outputting a control signal for processing the viewpoint image data stored in the viewpoint buffer; And An image decoding unit for decoding and outputting the viewpoint image data stored in the viewpoint buffer according to the control signal, respectively. Multi-view image decoder comprising a. The method of claim 1, The viewpoint image data includes multi-viewpoint image decoding, wherein the viewpoint image data includes pictures belonging to at least one GOP of the encoded viewpoint sequence corresponding to the respective viewpoints of the single bit string, or includes all of the maximum viewpoint sequence. group. The method of claim 1, wherein the scheduler, A utilization rate calculation unit for receiving a decoding time of each of the viewpoint image data in the GOP of the most recently decoded time zone from the image decoding unit and calculating a utilization rate of the image decoding unit according to the decoding time; And And a picture skip section for determining a picture skip method in a GOP in a time zone to be decoded immediately after the picture decoder from the picture decoder utilization rate. The method of claim 3, The picture skipping unit increases the number of picture skips in the GOP of the time zone to be immediately decoded as the picture decoding unit usage rate in the GOP of the most recently decoded time zone increases for each time point. . The method of claim 3, wherein the image skip unit, A skip picture determination module for determining a number of pictures skipped in a GOP in a time zone to be immediately decoded immediately after the picture decoding unit in a GOP in the most recently decoded time zone; And And a decoded picture selecting module for selecting a picture to be skipped according to the picture skip number. The method of claim 5, And the decoded picture selecting module continuously skips a skip start picture and pictures after the skip start picture in a GOP. The method of claim 6, And the view point image data is any one of a baseband profile sequence and a main profile sequence. The method of claim 5, And the decoded picture selection module skips pictures discontinuously in a GOP. The method of claim 8, And said viewpoint image data is a main profile sequence. The method of claim 3, wherein the scheduler, And a viewpoint priority determiner for determining a viewpoint priority in a GOP of a time zone to be decoded immediately after the picture decoder utilization rate. The method of claim 10, The viewpoint priority determiner makes the viewpoint priority in the GOP of the time zone to be decoded immediately after the smaller as the utilization rate of the picture decoder in the GOP of the time zone decoded most recently for each viewpoint. Picture decoder. A multi-view image rearrangement unit which receives respective viewpoint image sequences having the same GOP structure from K (random natural numbers) cameras, and generates a single bit sequence sequentially rearranged by dividing each viewpoint image sequence into GOP units; An image encoder which encodes and transmits the single bit string; K view buffers respectively storing data corresponding to the viewpoints of the cameras; A bit string divider configured to receive the single bit string and to separate the single bit string into view image data units corresponding to the respective view points and to store the single bit stream in the corresponding view buffer; A scheduler for outputting a control signal for processing the viewpoint image data stored in the viewpoint buffer; And An image decoding unit for decoding and outputting the viewpoint image data stored in the viewpoint buffer according to the control signal, respectively. Multi-view image data processing system comprising a. The method of claim 12, And the image encoder transmits the encoded single bit string through a single channel. The method of claim 12, The multi-viewpoint image rearranging unit divides each of the viewpoint image sequences into GOP units, and rearranges them sequentially by grouping them by GOP units in the same time zone. The method of claim 12, The viewpoint image data includes pictures belonging to at least one GOP of the encoded viewpoint image sequences corresponding to the respective viewpoints of the single bit string, or the maximum viewpoint image sequence includes all of the viewpoint image sequences. Processing system. The method of claim 12, wherein the scheduler, A utilization rate calculation unit for receiving a decoding time of each of the viewpoint image data in the GOP of the most recently decoded time zone from the image decoding unit and calculating a utilization rate of the image decoding unit according to the decoding time; And And a picture skip section for determining a picture skip method in a GOP in a time zone to be decoded immediately after the picture decoding section from the picture decoding section utilization rate. The method of claim 16, Multi-view image data processing, characterized in that the picture skip unit increases the number of picture skips in the GOP of the time zone to be decoded immediately after the larger the utilization rate of the picture decoding unit in the GOP of the most recently decoded time zone for each time point. system. The method of claim 16, wherein the image skip unit, A skip picture determination module for determining a number of pictures skipped in a GOP in a time zone to be immediately decoded immediately after the picture decoding unit in a GOP in the most recently decoded time zone; And And a decoded picture selection module for selecting a picture to be skipped according to the picture skip number. The method of claim 18, And the decoded picture selection module continuously skips a skip start picture and pictures after the skip start picture in a GOP. The method of claim 19, And said viewpoint image data is any one of a baseband profile sequence and a main profile sequence. The method of claim 18, Wherein said decoded picture selection module discontinuously skips pictures in a GOP. The method of claim 21, And said viewpoint image data is a main profile sequence. The method of claim 16, wherein the scheduler, And a viewpoint priority determiner for determining a viewpoint priority in a GOP of a time zone to be decoded immediately after the picture decoder utilization rate. The method of claim 23, wherein Multiview image data, characterized in that the viewpoint priority in the GOP of the time zone to be decoded immediately after the time is smaller as the utilization rate of the picture decoder in the GOP of the time zone most recently decoded for each time point decreases Processing system. (a) receiving a single bit string obtained by rearranging and encoding K (random natural numbers) viewpoint image sequences having the same GOP structure; (b) dividing the single bit stream into view image data units corresponding to each view and storing the single bit string in a corresponding view buffer; (c) decoding the viewpoint image data stored in the viewpoint buffer for each GOP unit of the same time zone; And (d) determining a method of skipping pictures in the GOP of the time zone to be decoded immediately after the decoding time of the GOP of the time zone decoded most recently for each view image data, and sequentially repeating step (c) Multi-view image data processing method comprising a. The method of claim 25, In step (d), the number of image skips in the GOP in the time zone to be immediately decoded increases as the decoding time of the GOP in the most recently decoded time zone increases for each view image data. Way. The method of claim 25, wherein step (d) (d1) determining the number of image skips of a time zone to be decoded immediately after the time according to the decoding time of the most recently decoded time zone for each view image data; (d2) multi-view image data processing method comprising selecting an image to be skipped according to the image skip number. The method of claim 27, And said step (d2) continuously skips the skip start picture and the pictures after the skip start picture in a GOP. The method of claim 27, And said step (d2) discontinuously skips pictures in a GOP. The method of claim 25, In the step (d), the viewpoint priority of the GOP of the time zone to be decoded immediately after the decoded time of the GOP of the most recently decoded time zone becomes higher according to the viewpoint image data. Treatment method. The method of claim 25, The viewpoint image data includes pictures belonging to at least one GOP of the encoded viewpoint image sequences corresponding to the respective viewpoints of the single bit string, or the maximum viewpoint image sequence includes all of the viewpoint image sequences. Treatment method. In a method of separating a single bit string obtained by rearranging and encoding K (arbitrary natural number) view image sequences having the same GOP structure into view image data units corresponding to each view, and sequentially decoding each GOP unit in the same time zone. , (a) receiving decoding time information of a GOP of a most recently decoded time zone; (b) determining a method of skipping pictures in a GOP in a time zone to be decoded immediately after the decoding time; And and (c) outputting a signal for controlling the decoding of the GOP in the time zone to be decoded immediately after the image according to the picture skipping method. 33. The method of claim 32, In step (b), the number of image skips in the GOP in the time zone to be decoded immediately after the decoding is increased as the decoding time of the GOP in the most recently decoded time zone increases for each view image data. Way. 33. The method of claim 32, wherein step (b) comprises: (b1) determining, according to the decoding time of the most recently decoded time zone, the number of image skips of the time zone to be decoded immediately after each view image data; (b2) multi-view image data processing method comprising selecting an image to be skipped according to the number of image skips. The method of claim 34, wherein And said step (b2) continuously skips a skip start picture and pictures after the skip start picture in a GOP. The method of claim 34, wherein And said step (b2) discontinuously skips pictures in a GOP. 33. The method of claim 32, In step (b), the viewpoint priority of the GOP of the time zone to be decoded immediately after the decoding is increased as the decoding time of the GOP of the most recently decoded time zone becomes longer according to the viewpoint image data. Treatment method. 33. The method of claim 32, The viewpoint image data includes pictures belonging to at least one GOP of the encoded viewpoint image sequences corresponding to the respective viewpoints of the single bit string, or the maximum viewpoint image sequence includes all of the viewpoint image sequences. Treatment method. As a recording medium in which a program of instructions executable by a digital processing apparatus is tangibly embodied and can be read by a digital processing apparatus, A recording medium on which a program for performing the multi-view image data processing method according to any one of claims 32 to 38 is recorded.

Images