JP4594432B1 - Movie playback method, movie playback system, and program - Google Patents

Abstract

A plurality of temporally synchronized image streams are multiplexed and encoded, and decoded as temporally synchronized image streams.
A moving image reproduction system extracts images having a common reproduction timing of a plurality of different image streams from a plurality of image streams, arranges the images in the order of extraction, and sets a time-convolved encoding sequence. The interface units 632 and 634 to read out, the decoding unit 638 that decodes the encoded sequence to generate a time-convolved integrated stream, and acquires the integrated stream and separates the images of each image stream constituting the integrated stream A frame synchronization processing unit 640 that writes out in time synchronization, and a display device 650 that receives a video signal from the graphic accelerator 646 and displays a moving image.
[Selection] Figure 6

Description

  The present invention relates to an image encoding technique, and more particularly to a technique for multiplexing and encoding a plurality of temporally synchronized image streams and decoding the temporally synchronized image streams.

  In recent years, with the improvement in performance of information processing devices and network technologies, digital contents to be processed by information processing devices have also been diversified. Examples of digital content include documents, still images, audio, moving images, and multimedia content in which moving images and audio are synchronized.

  These digital contents are edited and provided to the user so as to suit the user's specific preference and purpose, such as a literary work, a photo book, a movie / video, and a game. When the digital content described above is provided to the user, MPEG, MPEG-2, MPEG-4 (hereinafter referred to as MPEG, MPEG-2, MPEG-4, etc.) is referred to with a compression format referred to by MPEG. Refer to it as a series format.) Compressed to a format such as MP3, H.264, etc. and recorded on an optical recording medium such as a CD-ROM or DVD. In addition, when the above-described content is transmitted as digital data by an information processing device or digital broadcast, the content is compressed into a format such as MPEG-2 or H.264 and distributed as streaming distribution or terrestrial digital broadcast.

  Conventionally, the above-mentioned digital contents often provide a two-dimensional (hereinafter referred to as 2D) image, and a 2D image can be fully enjoyed by speeding up information processing equipment and transmission infrastructure. Digital content that can be used is provided.

  However, due to advances in information processing devices and transmission technologies, attempts have been made to provide users with 3D digital content instead of 2D images. In order to visually recognize digital content as a 3D image, a lenticular lens such as the IP (Integral Photography) method is used to give different images to the left and right eyes of the user, and both left and right such as a parallax barrier method are used. There is known a technique for visually providing three-dimensional (hereinafter, referred to as 3D) recognition by arranging two liquid crystal shutters crossed so as to transmit an image to be recognized by an eye.

  In order to provide 3D video, an image reproduction system that generates 3D recognition referred to as a multi-view method has been known. In the multi-viewpoint method, images acquired at different shooting angles for each viewpoint cycle are synchronized and played back using a playback device such as a liquid crystal display or a liquid crystal projector. A viewer such as a user or a spectator recognizes a plurality of 2D images having different parallax angles at a position where the lenticular lens focuses an image from the playback device. If the viewer changes the viewpoint while watching the video, images close to each viewpoint will be recognized, so the viewer will be able to 3D images based on the spatial convolution of 2D images from multiple shooting positions. Can be recognized.

  That is, for example, when 3D video is to be provided, it is necessary to reproduce a plurality of moving picture streams in synchronization with each other in both the IP system and the parallax barrier system. Up to now, a technique for arranging a plurality of playback devices and generating stream images acquired from different shooting angles is known. For example, in Japanese Patent Application Laid-Open No. 11-38954 (Patent Document 1), a plurality of display programs for reproducing video data by different methods and displaying them on a display screen, and display of video data according to instructions from a user. An image display device is described that includes an image data integration program for extracting conditions and selectively sending them to any one of a plurality of display programs.

  In Patent Document 1, an MPEG stream or the like is divided into a plurality of reproduction units, and each reproduction unit is selected and reproduced and displayed. In Patent Document 1, the point of switching to a 3D image is described. It does not describe at all how to generate compressed data that provides a 3D image.

  In Japanese Patent Laid-Open No. 2006-140618 (Patent Document 2), if the differential pack is used by packing and multiplexing the depth information data called differential pack (D_PACK) in a format compliant with the DVD video standard. Disclosed is a 3D video information recording device and program that can output 3D video, which is a format that can output standard 2D video as a DVD video standard if it is not used. It describes that by adding information, 3D video information can be recorded to support switching between 2D and 3D, and a 3D compressed image independent of the 3D image playback method is provided.

  Furthermore, in JP 2009-513074 A (Patent Document 3), a specific viewpoint image of at least two viewpoint images corresponding to multi-view video content is encoded as a base layer, and the specific viewpoint image and at least one of the viewpoint images are encoded. By encoding each of at least one other viewpoint image of at least two viewpoint images as an enhancement layer using prediction from a lower layer corresponding to at least one of the other viewpoint images, at least 2 An apparatus including an encoder for encoding one viewpoint image is disclosed.

  Japanese Patent Laid-Open No. 2006-54500 (Patent Document 4) encodes a left viewpoint image and a right viewpoint image into two frames using an interlace technique such as MPEG-2, and temporally converts the left and right line-of-sight images. A moving image encoding technique for displaying in synchronization with the video is disclosed. Even with the technique of Patent Document 4, it is possible to temporally multiplex and display a plurality of video streams, but there is a problem in that it is limited to left and right viewpoint images.

JP 11-38954 A JP 2006-140618 A JP 2009-513074 A JP 2006-54500 A

  Japanese Patent Application Laid-Open No. 2004-228561 describes a point of reproduction for each reproduction unit, and describes that it is possible to select 3D display at the time of switching display. Patent Document 2 describes that depth information for generating 3D is generated from encoded information and registered in MPEG data. Patent Document 3 discloses a device that encodes a multi-viewpoint image by encoding one image and predicting the other image. In Patent Document 4, it is possible to encode two video streams using an interlace method and synchronize them in time to form a video stream.

  Although it is possible to provide a video stream for displaying, for example, 3D video by using the techniques of Patent Literature 1 to Patent Literature 4, in recent years, high-definition display devices and high-performance information processing devices, In addition, it has been necessary to more efficiently encode a plurality of time-synchronized video streams and to reproduce the time-synchronized video streams in accordance with the complexity and high definition of contents.

  In addition, as content becomes more complex and more detailed, more efficient image compression technology is required, and multiple video streams can be played back while guaranteeing time synchronization from the encoded stream encoded by the image compression technology. It was necessary to do.

  That is, until now, there has been a need for a technique for efficiently compressing and decoding a plurality of image streams having common image characteristics while guaranteeing time synchronization.

  The present invention has been made in view of the above-described problems of the prior art. In the present invention, a plurality of image streams are integrated to generate an integrated stream, which is encoded by a conventional encoding method. The encoded stream is decoded by a decoder that employs a decoding method corresponding to the encoding method, and becomes an integrated stream. The integrated stream is time synchronized and reproduced as a plurality of moving images.

In the generation of the integrated stream, images constituting the image stream are extracted in the specified order for each specified time slice, and an image stack is generated by arranging the images from the beginning of the integrated stream in the extraction order. When there are time slices to be further processed, the integrated stream extracts the images belonging to the time slices in the same extraction order, places them immediately after the last image in the previous image stack, and extracts them in the following order of extraction. By arranging the processed images, the T _n-1 stack and the T _n stack are generated.

  In the present invention, the above-described integration of a plurality of image streams is performed as a temporal convolution of images. The integrated stream generated by the time convolution includes MPEG, MPEG-2, MPEG-4, H.264, and so on. It is encoded using an encoding method such as H.264. Encoding is performed using inter-frame correlation of successive images into an encoded stream. The encoded stream is decoded by a decoder that employs a decoding method corresponding to the encoding method, and an integrated stream is reproduced. The reproduced integrated stream is separated into images that provide individual image streams, time-synchronized, and then converted into an analog signal. For example, a moving image is reproduced by a display device or a liquid crystal projector of a personal computer.

  Certain embodiments of the present invention provide temporal convolution of multiple image streams that have common image characteristics and that can be expected to be compressed by inter-frame correlation prediction for multiple images belonging to a given time slice of the image stream. More specifically, it can be applied to an image stream providing a plurality of game scenes having a common story development or an image stream having different viewpoints for providing 3D video.

  According to the present invention, encoding is performed using inter-frame correlation prediction between images of the same time slice that has been time-convolved from a plurality of image sequences, thereby enabling efficient encoding and achieving a high compression rate. it can. In addition, since the image corresponding to each image sequence can be separated from the decoded integrated stream and the reproduced image can be passed to the reproduction processing unit in time synchronization, the time synchronism between pictures can be ensured, and the Enables quality video stream playback.

  Furthermore, according to the present embodiment, the image reproduction system can efficiently switch the reproduction stream according to the user's desire and operation, and information provision by digital contents can be diversified.

The functional block diagram of the encoder 100 of this embodiment. The figure which showed the detailed functional block of the encoder 160 shown in FIG. The figure which showed the functional block in the case of encoding the encoder 160 of this embodiment in a H.264 format. The figure explaining the data structure 400 which comprises the some image stream, the image which forms an image stream, a time slice, and the integrated stream produced | generated by this embodiment. The figure which showed the functional block of the decoder 500 of this embodiment. The functional block diagram of the operation | movement reproduction | regeneration system 600 of this embodiment. FIG. 7 is a detailed functional block diagram of the frame synchronization processing unit 640 described in FIG. 6. The flowchart of the encoding process which the encoder 160 of this embodiment performs. The flowchart of the image decoding method of this embodiment. To display 3D images, showing the head of the image in the image stack 1000 time slice T ₁ of the 9 image stream corresponding to the nine viewpoints FIG. It illustrates an embodiment of the image stack 1100 VIEW1~VIEW9 in time slice T = T ₂ immediately after the time slice T ₁ shown in FIG. 10. The figure explaining the inter-frame correlation prediction 1200 employ | adopted by this embodiment. The figure which showed the background which removed the flying object which is an object with the largest moving amount as a wire frame about the correlation prediction method of this embodiment. The figure which showed embodiment of the switching process of the moving image reproduced by the encoding / decoding system and moving image reproduction system of this embodiment.

  Hereinafter, the present invention will be described with embodiments, but the present invention is not limited to the embodiments described below. FIG. 1 is a functional block diagram of the encoder 100 of the present embodiment. As shown in FIG. 1, the encoder includes an A image stream supply unit 110 to an N image stream supply unit 140 (N is a positive integer of 1 or more). The A image stream supply unit 110 may be a time-series stream of still images acquired by a digital camera, for example, and images generated by CG (Computer Graphics) are arranged in a specific sequence, It is also possible to use an animation that is configured to be played by designating an image feed speed or the like.

  Each of the image stream supply units 110 to 140 can be configured using a device that generates on-the-fly using a digital camera or the like, or can be connected to a recording medium such as an HDD, DVD, or CD-ROM via an appropriate interface. The stored still image can be configured as an information processing apparatus capable of reading out as an image stream.

  The image streams supplied from the image stream supply units 110 to 140 are input to the mixer 150. The mixer 150 extracts images from each image stream in the set order for each set time slice, and arranges the images extracted earlier in order from the head of the stream, thereby arranging the plurality of image streams 110a to 140n. Time convolution.

When the generation of the image stack for the first time slice T ₁ ends, the mixer 150 extracts the images corresponding to the next time slice in the order set for the image stream, and the image stack of the time slice T ₁ that has already been generated. of go inserted into the extraction order immediately after the last image, to generate an image stack corresponding to the time slice T _2.

The time interval of the time slice T can be set to a time interval necessary for smoothly reproducing the image stream. For example, T = 1 · P ⁻¹ (P is a frame feed rate, and frame · s ^{− 1} dimension, typically about 16 to 30). The mixer 150 can be configured by installing appropriate image processing software in the information processing apparatus to execute image processing, and for example, ASIC (Application for executing time convolution).
It can also be implemented as a Specified Integrated Circuit) and is not limited at times.

  In a specific embodiment of the present embodiment, an image acquired by an analog camera can also be used. In this case, the analog image is input to the encoder 160 by A / D conversion, format conversion to BMP, or the like. A power image stream can be generated. In still another embodiment, an image stream in a format referred to as a so-called Moving JPEG that is normally implemented by a digital camera can be used. Each of the image streams 110a to 140n is time-convolved by the mixer 150 to be an integrated stream. The generated integrated stream is sent to the encoder 160, subjected to MPEG series or H.264 encoding processing to be an encoded stream, and is output from the encoder 160. The output of the encoded stream can be provided as a packet to the Internet or digital broadcasting, or can be stored as digital content recorded on a recording medium such as a DVD.

  FIG. 2 shows a detailed functional block 200 of the encoder 160 shown in FIG. As shown in FIG. 2, the encoder 160 is an embodiment that generates an encoded stream of an MPEG series format such as MPEG, MPEG-2, or MPEG-4. The encoder 160 includes an input buffer 210, an adder / subtractor 212, and a DCT unit 214. The input buffer 210 is configured as a FIFO (First in First Out) buffer, and stores an integrated stream generated from the image streams 110a to 140n to be processed in a first-in first-out manner. Note that the image input to the input buffer 210 corresponds to the format of inter-frame correlation prediction processing used by the encoder 160, for example, forward prediction, backward prediction, bidirectional prediction, prediction image non-use, etc. Image sequence rearrangement processing may be performed.

  The integrated stream image stored in the input buffer 210 is sent to an adder / subtractor 212. Typically, after information based on inter-frame correlation prediction is calculated, it is sent to a DCT (Discrete Cosine Transformation) unit 214, DCT calculation is performed. The result of the DCT calculation is sent to the quantizer 216 and quantized. After the encoding process such as Huffman encoding by the variable length encoder 218, the result is sent to the output buffer 232, and the code amount controller 230 An encoded stream (MPEG sequence) is generated under the feedback control of the code amount, and used as an output stream of the encoder 160.

The output of the quantizer 216 is sent to an inverse quantizer 220, an inverse DCT device 222, an adder 224, an image buffer 226, and an inter-frame correlation prediction 228. Used in the calculation of In the present embodiment, the encoder 160, time series not only continuous image also executes the compression by inter-frame correlation prediction between the images of the different image streams 110a~110n attributed to a particular time slice T _j.

For example, when generating an encoded stream for playing back 3D video, the calculation of inter-frame correlation prediction will give a difference image corresponding to the angle change of the viewpoint, and when playing a game story etc. The difference image corresponding to the story difference in the same time slice T _j is given. For this reason, in this embodiment, the encoder 160 uses the same encoding method as the conventional one.

  FIG. 3 shows a functional block 300 when the encoder 160 of the present embodiment is encoded in the H.264 format. As described with reference to FIG. 2, the input buffer 310 is configured as a FIFO (First in First Out) buffer, and stores an integrated stream generated from the processing target image streams 110 a to 140 n in a first-in first-out manner. Thereafter, the integrated stream of the input buffer 310 is supplied to the adder / subtractor 312. In the embodiment of FIG. 3, the image stream to be processed is inter-coded, and the adder / subtractor 312 performs the inter-frame correlation prediction from the picture of the integrated stream, and thus the prediction image supplied from the inter-frame correlation predictor 324 is used. The difference image data is generated by subtraction and supplied to the orthogonal transformation device 314.

  The inter-frame correlation predictor 324 calculates a prediction vector between images constituting different image streams, as described with reference to FIG. More specifically, the inter-frame correlation predictor 324 reads, from the image buffer 322, an image to be a reference image for an image that is currently processed, and outputs a predicted image based on a prediction vector for the reference image. Generated and supplied to the adder / subtractor 312. The adder / subtractor 312 generates a difference image by subtracting the prediction image supplied from the inter-frame correlation predictor 324 from the current processing target picture, and then supplies the difference image to the orthogonal transformation device 314.

  The orthogonal transform device 314 acquires the picture or difference image supplied from the adder / subtractor 312, applies orthogonal transform such as DCT transform, and sends the transform coefficient to the quantizer 316. The quantizer 316 quantizes the transform coefficient from the orthogonal transform device 314 under feedback control by a code amount controller 328 described later, and supplies the resulting quantized coefficient to the encoder 326. The quantized coefficients, prediction vectors, and the like from the quantizer 316 are sent to the output buffer 330 after being encoded by the encoder 326 such as variable-length encoding and arithmetic encoding, and are stored in a first-in first-out manner. The code amount controller 328 feedback-controls the processing of the quantizer 316 so that the output buffer 330 does not overflow or underflow based on the storage amount of the column reserved for the image set of the predetermined image stream of the output buffer. To do.

  The inverse quantizer 318 inversely quantizes the transform coefficient supplied from the quantizer 316 by applying the same quantization process as that of the quantizer 316, and inversely transforms the resulting transform coefficient. This is supplied to the orthogonal transformer 320. The inverse orthogonal transformer 320 performs inverse orthogonal transform processing on the transform coefficient from the inverse quantizer 318 and decodes a difference image obtained by subtracting a predicted image from an intra-coded picture that is currently being processed or an original inter-coded picture. Then, it is sent to the image buffer 322.

  In the output buffer 330, an encoded stream obtained by encoding the integrated stream is accumulated in the H.264 encoding format in the embodiment to be described, and various control data and the like are added as header information and then encoded. Output as a stream.

  FIG. 4 is a diagram illustrating a data structure 400 constituting a plurality of image streams, images forming the image stream, time slices, and an integrated stream generated in the present embodiment. In FIG. 4, N separate image streams from A image stream 410 and B image stream 420 to N image stream 430 can be used as the image stream. The image stream, for example, the A image stream 410 is composed of image sequences of image 1, image 2,..., Image 9,..., And the image sequence includes images up to the end of the A image stream. .

On the other hand, the B image stream 420 and the N image stream 430 are similarly composed of image sequences of image 1, image 2,..., Image 9,. The image corresponding to is included. Further, the images indicated by the image _i of each of the image streams 410 to 430 are sampled by the mixer 150 in order to form the same time slice T _i and placed in the integrated stream 440.

For example, the image 1 of the A image stream 410, the image 1 of the B image stream 420, and the image 1 of the N image stream are extracted in the set order and shown as the time slice T ₁ as shown in FIG. The stack is configured in the integration stream. In the embodiment described in FIG. 4, the image stack specified by the time slice T _i includes N images corresponding to N streams, and i = 1 to last as illustrated in FIG. 4. Up to (last) image stacks are formed.

Similarly, the time slice T _2, each image 2 of the A image stream 410~N image stream 430 is extracted, immediately after the last image in the image stack specified by the time slice T _1, image 2 is the extracting order When the image 2 of the N image stream 430 is arranged, the image stack specified by the time slice T ₂ is generated. The mixer 150 continues the same processing until the time slice specified by T _last , and finally forms an integrated stream in which all the images are time-convolved.

  The integrated stream 440 illustrated in FIG. 4 is input to the encoder 160 including the functional blocks described with reference to FIG. 2 or 3, and the integrated stream 440 is encoded according to the setting of the encoder 160. At this time, encoding is performed by applying the same processing as that for encoding a conventional image stream in the GOP set in the encoder 160 according to the number of pictures, the interval of I and P pictures.

FIG. 5 shows functional blocks of the decoder 500 of this embodiment. The embodiment described in FIG. 5 will be described on the assumption that the H.264 system is used as the encoding / decoding system. The input buffer (not shown) includes a FIFO buffer, and the integrated stream 440 having the data structure shown in FIG. 4 is decoded from the input encoded stream 510 by the decoding unit 520, and the integrated stream 540 is decoded. Output stream. In the embodiment shown in FIG. 5, the encoded stream 510 is sent from the time slice T ₁ to the last T _last and is sent to the decoding unit 520 in the order of input to the input buffer, and the decoding process is performed.

  The decoding unit 520 in the embodiment shown in FIG. 5 includes a variable length decoder 522, an inverse quantizer 524, and an inverse DCT device 526. The variable length decoder 522 decodes encoded data encoded by Huffman encoding or the like, and the inverse quantizer 524 converts the quantized DCT coefficient into the quantization process of the encoder 160 shown in FIG. Inverse transformation is performed to generate DCT coefficients, and data is supplied to the inverse DCT unit 526. The inverse DCT unit 526 reproduces a picture of the differential image that is currently processed using the acquired DCT coefficient and sends it to the adder 528.

  On the other hand, the output of the variable length decoder 522 is sent to the inter-frame correlation predictor 530, and a predicted picture is generated from the reference image stored in the image buffer 532 using the decoded prediction vector value. . The generated prediction picture is sent to the adder 528, synthesized with the difference image picture, stored in an output buffer (not shown) as an output stream, and then output from the decoding unit 520 as an output stream in a first-in first-out manner. Sent out. The output stream is separated while being time-synchronized with the individual stream of the image stream included in the integrated stream before the moving image is reproduced. Although the inter-frame correlation prediction has been described as a picture for convenience of explanation above, in practice, a part of the image is acquired as a difference image, or the inter-frame correlation prediction process is performed using image data in which a plurality of difference images are combined. It can be carried out.

  After each image stream is separated, each image stream is subjected to format conversion and D / A conversion to be converted into analog data, and converted into a video signal via a graphic accelerator such as a personal computer. The video is sent to a playback device such as

  FIG. 6 is a functional block diagram of the operation reproduction system 600 of this embodiment. A moving image playback system 600 shown in FIG. 6 can be configured as a controller of a personal computer, workstation, game device, or liquid crystal projector, and includes the decoder 500 shown in FIG. 5 as its functional module. .

  As shown in FIG. 6, the moving image reproduction system 600 includes a main control device 630, a display device 650, and input / output peripheral devices 660 such as a mouse, a keyboard, and a joystick for issuing various commands to the main control device 630. It is comprised including. The main control device 630 can be implemented as a personal computer according to a specific application, or as a control unit for a game device, a digital television, a liquid crystal projector, or the like.

  The main controller 630 includes a central controller (CPU) 636 for controlling various functional units, a RAM 642 as a storage device that provides an execution space for application programs, a terrestrial digital infrastructure, an Internet / local area network ( A network interface 632 for transmitting and receiving data via a public network 610 such as a LAN) and an IDE, for reading and writing data via an optical recording device such as a hard disk drive (HDD), MO, CD, DVD, etc. And a storage interface 634 of a standard such as ATA, SERIAL-ATA, or ULTRA-ATA.

  The CPU 636 implemented by the main controller 630 includes a CISC architecture microprocessor such as PENTIUM (registered trademark), XEON (registered trademark), and PENTIUM (registered trademark) compatible chip, or a RISC architecture such as POWER PC (registered trademark). A microprocessor may be mentioned, and the CPU 636 may be implemented in a single core or multi-core form. The main control device 630 includes an operating system such as WINDOWS (registered trademark), UNIX (registered trademark), LINUX (registered trademark), etc., and under the control of the OS described above, various exception handling, management of external devices, Communication session management, and execution and execution management of programs written in C, C ++, Java (registered trademark), JavaScript (registered trademark).

  Further, the main controller 630 may be controlled by any operating system such as WINDOWS (registered trademark), UNIX (registered trademark), LINUX (registered trademark), or MACOS. In addition, the main control device 630 can be implemented with browser software such as Internet Explorer (trademark), Mozilla (trademark), Opera (trademark), and Firefox (registered trademark). When the main control device 630 is mounted as a control unit for a game device or a liquid crystal projector, an OS dedicated to an embedded device other than Windows (registered trademark) or LINUX may be mounted.

  Further, main controller 630 includes a decoding unit 638, a frame synchronization processing unit 640, and a graphics accelerator (hereinafter simply referred to as GA) 646. The decoding unit 638 and the GA 646 can be configured by application programs. Further, in order to enable high-speed image processing, it is configured as a dedicated ASIC and can be configured as an expansion board, an expansion card, or an on-board chip.

On the other hand, the frame synchronization processing unit 640 is a processing module that executes processing for separating each image stream constituting the integrated stream from the integrated stream output from the decoding unit 638 in time synchronization, and includes a DLL and other runtime libraries, Or it is preferable to implement as a Plug-in program. In addition, when the main control device 630 is less expandable than a personal computer such as a game device, it can be mounted as a chip for performing dedicated processing. Frame synchronization processing unit 640, upon receiving the integration stream, the time convolved image integration stream, separated in time slice T _i units, and outputs the GA646 while time synchronization with the N image stream.

  The GA 646 receives the image stream, performs format conversion, D / A conversion, and video modulation such as VGA and XGA, and sends a video signal to the display device 650 via the graphics interface 644. Play a video up. The video signal to be sent may be an entire image stream according to a specific application, or only a specific image stream set by the image reproduction system 600 or a user command. Note that the audio data is sent to an audio control unit (not shown) as a single stream in synchronization with the image flash from the frame synchronization processing unit 640 so that the moving image and audio can be synchronized and reproduced.

FIG. 7 is a detailed functional block diagram of the frame synchronization processing unit 640 described with reference to FIG. FIG. 7 shows an embodiment of two frame synchronization processing units in relation to the output mode of the image stream. The first embodiment of the frame synchronization processing unit 640-A, a FIFO buffer 720 which functions as a frame buffer, a line buffer for synchronizing and outputting the image of the same time chunks T _i of different image stream shown in FIG. 7 Alternatively, it includes a synchronization buffer 730 formed of a ring buffer. The integrated stream 710 from the decoding unit 638 is input to the frame synchronization processing unit 640-A, and the integrated stream is sent to the synchronization buffer 730 by a first-in first-out method.

In the synchronization buffer 730, the same storage area as the number of streams of the integrated image stream is designated by a pointer. In the embodiment to be described, the synchronization buffer 730 is a time at which the reproduction timings of nine image streams are common. Control is performed so that the image becomes full when nine images of the slice T _i are stored. The synchronization stream 724 stores the integrated stream from the FIFO buffer 720 while counting the number of images.

At this time, images belonging to the same time slice T _i are accumulated in the address areas designated by the pointers 1 to N of the synchronization buffer 730. In the first embodiment, when the synchronization buffer 730 instructs the FIFO buffer 720 to stop writing, it simultaneously instructs the GA 646 to read data from the address area specified by the pointers 1 to N. The GA 646 Then, an image stored corresponding to each address area is acquired, a video signal corresponding to each image stream is generated, and sent to the display device 650 to display, for example, 3D video.

  After the contents of the synchronization buffer 730 are flushed, the frame synchronization processing unit 640 instructs the FIFO buffer 720 to write data, and sends the data again until the synchronization buffer 730 is full. The same processing is repeated until there is no image in the integrated stream, and the moving image reproduction is executed. Note that the GA 646 at this time can be mounted with a plurality of graphic chips in order to display 3D video. In another embodiment, the image stream output from the frame synchronization processing unit 640 can be sent to an independent liquid crystal projector GA for projecting each viewpoint image. A video signal to be projected may be generated and projected from each liquid crystal projector. Note that the GA 646 has an appropriate number of frame buffers so that the delay of the time synchronization processing by the frame synchronization processing unit 640 does not affect the moving image reproduction.

  In addition, the second embodiment 640-B of the frame synchronization processing unit 640 is an embodiment for reproducing only a selected image stream from among a plurality of image streams, and the configuration of the FIFO buffer 740 and the synchronization buffer 750. Is the same as in the first embodiment. On the other hand, the output of the synchronization buffer 750 is input directly to the selector 760 instead of directly to the GA 646. In the embodiment to be described, an output from the synchronization buffer 750 is selected by the selector 760, and an image corresponding to the A image stream is output to the GA 646 and used for moving image reproduction. The selector 760 receives an image switching command from the input / output peripheral device 660, and receives a select signal generated by the main control device 630, and images from the B image stream to the N image stream are discarded.

Further, when receiving a command for switching to the B image stream from the user, for example, the main control device 630 generates a select signal for outputting the B image stream and sends it to the selector 760. The selector 760 receives the select signal and switches the output image stream from the A image stream to the B image stream. In this embodiment, stored in the synchronizing buffer 750 images, since belonging to the same time slice T _i, after which the select signal is input, even when switched to another image stream, such as skipping playback movies Therefore, high-quality image switching is possible.

  FIG. 8 shows a flowchart of the encoding process executed by the encoder 160 of the present embodiment. The processing of FIG. 8 starts from step S800, and in step S801, the encoder 160 acquires the integrated stream generated by the mixer 150 from the image streams from the image stream supply units 110 to 140, and accumulates them in the input buffer. In step S802, the integrated stream stored in the input buffer is encoded using inter-frame correlation prediction, and the encoded integrated stream encoded in step S803 is generated as an output stream.

  The processing up to S801 can be performed separately from the processing from S802 via processing stored in the storage device, which is typically performed.

Incidentally, if the image more than the threshold is different than between the image attributable to the time slice T _i, it can be prevented by using the inter-frame correlation prediction. In the present embodiment, the time slice means a temporal section of an image stream that gives an image set to be reproduced in time synchronization among images constituting a plurality of image streams. In step S804, the output stream is stored in a recording medium such as an HDD device or a DVD via an appropriate storage interface, and can be distributed. In step S805, the encoding process ends.

  FIG. 9 shows a flowchart of the image decoding method of the present embodiment. The image decoding method of this embodiment starts from step S900, and reads the encoded integrated stream into the input buffer of the decoding unit 638. In step S902, decoding is performed using inter-frame correlation prediction to generate an integrated stream (decoded).

  In step S903, after decoding, the integrated stream (decoding) is transferred to the frame synchronization processing unit 640, and the synchronization buffer 730 or the synchronization buffer 750 separates the images of the plurality of image stream images while synchronizing them in time. In step S904, it is determined whether image stream selection has been commanded. If there is a user command to select an image stream (yes), in step S905, the selected image stream is sent to the GA 646 to play a moving image. .

  On the other hand, if image stream selection is not instructed (no), each image stream is sent to the GA in parallel in step S907, and moving image reproduction is performed. The image stream selection command is issued by the user from the input / output peripheral device 660 when a specific image stream is switched and displayed. In another embodiment, the image playback system 600 uses, for example, the A image stream as a default setting for moving image playback unless an explicit command is received from the user, and the other is set according to the user command or the progress of the game story. It is also possible to switch to the image stream and play the video.

  In step S906, it is determined whether or not decoding has been completed to the end of the integrated stream (decoding). If the decoding has not been completed to the end (no), the process returns to step S904 and the process is repeated. On the other hand, when the decoding of the integrated stream (decoding) is completed (yes), the moving image is reproduced to the end of the integrated stream (decoding), and the process ends in step S908.

The encoding process and moving image reproduction process of the time-convolved integrated stream according to the present embodiment will be specifically described with reference to FIGS. FIG. 10 shows images of the image stack 1000 of the _first time slice T1 of the nine image streams corresponding to nine viewpoints for displaying 3D video. A plurality of VIEWs indicated by time slices T = T ₁ form an image stack 1010, and inter-frame correlation prediction is performed on VIEW ₁ to VIEW 9. More specifically, an image stack for reproducing a 3D moving image includes images having different viewpoint angles as shown in FIG. In the integrated stream, the images shown in plan as VIEW1 to VIEW9 in FIG. 10 belong to the image stacks of VIEW1 to VIEW9 in the image stack 1010, respectively.

Further, in FIG. 11 shows an embodiment of the image stack 1100 VIEW1~VIEW9 in time slice T = T ₂ immediately after the time slice T ₁ shown in FIG. 10. In the images indicated by VIEW1 to VIEW9, images having the same VIEW number form the same original image stream. Also in FIG. 11, VIEW1 to VIEW9 form an image stack 1110. Like FIG. 10, each VIEW image is composed of images with different viewpoint angles. In the illustrated embodiment, efficient compression processing is possible. More specifically: With reference to FIGS. 10 and 11, for example, as will be understood by comparing VIEW4 to VIEW5 in FIG. 10, the movement of the object is small and the viewpoint angle is slightly different. It is possible to perform moving image compression based on efficient inter-frame correlation prediction for an image stack belonging to a time slice.

  That is, a high compression effect is provided by performing an encoding process while predicting the inter-frame correlation of an integrated stream generated by temporally convolving an image so as to cross the image stream according to the present embodiment. Can be compressed more efficiently than encoding individual image streams according to conventional methods.

  FIG. 12 is a diagram for explaining the inter-frame correlation prediction 1200 employed in the present embodiment. The image stack 1210 belonging to a specific time slice Ti has nine images in the embodiment to be described, and these nine images are substantially the same images only with different viewpoint angles. As a result, the difference images from VIEW1 to VIEW9 have only different viewpoint angles, and extremely efficient compression is possible. On the other hand, the prediction of inter-frame correlation prediction according to this embodiment can also be used to generate an encoded stream with sufficient accuracy by applying a calculation method using an existing encoder such as bidirectional prediction, forward prediction, and backward prediction. I understand that I can do it.

  Furthermore, in the embodiment in which the image stream for providing 3D video is temporally convolved to generate an integrated image in the present embodiment, compression by inter-frame correlation prediction is performed between images corresponding to the difference in start point angle with scalability in advance. Therefore, it is easy to guarantee the scalability of the compression process.

  FIG. 13 is a diagram showing the inter-frame correlation prediction method of the present embodiment with the flying object being the object with the largest movement amount as a wire frame, with the background removed. In FIG. 13, an image 1300 and an image 1310 are images included in the same time slice with different viewpoint angles and the same reproduction timing, and the image 1300 and the image 1320 are separated in time from the same image stream. It is an image corresponding to the reproduction timing. As shown in FIG. 13, in the present embodiment, the image sets that make up the same time slice Ti or Tj are made up of images that are very close to each other only by the difference in image due to the difference in viewpoint angle. In conventional image coding, image compression is performed using the left-hand side to the right-hand side (backward prediction), the reverse (forward prediction), or one (bidirectional prediction). On the other hand, in the present embodiment, compression is performed by inter-frame correlation prediction in the direction toward the top and bottom in addition to the right and left of the page. As shown in FIG. 13, when a plurality of image streams are multiplexed in time, the reproduction timing of different image streams is different between frames with the same reproduction timing than when performing inter-frame correlation prediction in time series. It is shown that high compression can be achieved by performing compression by correlation prediction.

As shown in FIG. 12, the image stack 1210 belonging to a specific time slice T _i has nine images in the embodiment to be described, and these nine images are substantially the same images only with different viewpoint angles. . As a result, the difference images from VIEW1 to VIEW9 have only different viewpoint angles, and extremely efficient compression is possible. On the other hand, the inter-frame correlation prediction of the present embodiment can be used to generate an encoded stream with sufficient accuracy by applying a calculation method using an existing encoder such as bidirectional prediction, forward prediction, and backward prediction.

  FIG. 14 shows an embodiment of switching processing of moving images reproduced by the encoding / decoding method and moving image reproduction system of the present embodiment. In the embodiment illustrated in FIG. 14, the image stream 1410 and the image stream 1420 are described as being integrated.

  It is possible to temporally synchronize time slices of multiple image sequences by extracting and consolidating images in the same time slice of multiple image streams and integrating them in time synchronization and decoding. Become. For this reason, when the user issues an image switching command from the input / output peripheral device 660 or when the image reproduction system 600 is implemented as a game device or the like, a process of switching to a bonus item according to the story development is executed. For example, moving images of different image streams synchronized in time can be displayed simply by selecting an image stream to be reproduced.

  For example, it is assumed that the user is currently playing a moving image of the image stream 1410. Assuming that the user issues a command to switch the image stream to 1430 and play a video at time slice 1420, the video stream switching command is issued to selector 760, and the playback video corresponding to image stream 1430 with the minimum time lag. It has been switched to.

  Further, the embodiment of FIG. 14 is applied not only to playback video switching but also to playback of left-eye images and right-eye images with high synchronization in 3D video playback using two viewpoints such as a parallax barrier method. can do.

  The above functions of the present embodiment can be realized by a device executable program or an equivalent integrated circuit described in a legacy programming language such as assembler, C, C ++, or Java (registered trademark) or an object-oriented programming language. The program of the embodiment can be stored and distributed in a device-readable recording medium such as a hard disk device, CD-ROM, MO, flexible disk, EEPROM, EPROM, etc., and in a format that other devices can use via a network. Can be transmitted.

Although the present embodiment has been described so far, the present invention is not limited to the above-described embodiment, and has been described as performing processing in units of pictures, that is, one frame. Can be performed on a part of an image according to a specific purpose, and other embodiments such as addition, change, deletion, etc., such as encoding and decoding processing by superimposing a plurality of images, etc. These modifications can be made within the range that can be conceived by those skilled in the art, and any embodiment is included in the scope of the present invention as long as the effects and advantages of the present invention are exhibited.

DESCRIPTION OF SYMBOLS 100 ... Encoder 110-140 Image stream supply part, 110a-140n ... Image stream, 150 ... Mixer, 160 ... Encoder, 210 ... Input buffer, 212 ... Adder / subtractor, 214 ... DCT device, 216 ... Quantizer, 218 DESCRIPTION OF SYMBOLS ... Variable length encoder, 220 ... Inverse quantizer, 222 ... Inverse DCT device, 224 ... Adder, 226 ... Image buffer, 228 ... Inter-frame correlation predictor, 230 ... Code amount controller, 232 ... Output buffer, DESCRIPTION OF SYMBOLS 310 ... Input buffer, 312 ... Addition subtractor, 314 ... Orthogonal transformation device, 316 ... Quantizer, 318 ... Inverse quantizer, 320 ... Inverse orthogonal transformer, 322 ... Image buffer, 324 ... Inter-frame correlation predictor, 326 ... Encoder, 328 ... Code amount controller, 330 ... Output buffer, 332 ... Adder,

Claims (11)

A moving image reproduction method executed by a main control device that reproduces an encoded stream obtained by multiplexing a plurality of image streams in time synchronization, wherein the main control device includes a decoding unit that decodes the encoded stream, and the decoding A frame synchronization processing unit that outputs the plurality of image streams in time synchronization with the output of the unit, and the main control unit includes:
An encoded stream obtained by encoding an integrated stream generated by transversely extracting images having a common reproduction timing of a plurality of different image streams from the plurality of image streams and temporally convolving the images in the extraction order, A decoder unit obtaining from a persistent recording medium or a public network;
Decoding the encoded stream in the decoder unit to generate the time-convolved integrated stream;
The frame synchronization processing unit receives the integrated stream from the decoder, and the frame synchronization processing unit stores the images constituting the integrated stream in a first-in first-out manner, corresponding to the number of image streams. Stopping the writing of the images constituting the integrated stream at the time of storing the number of images, and writing all the stored images to the graphic accelerator in time synchronization;
Creating the respective image streams by the graphic accelerator and reproducing the moving image, wherein the frame synchronization processing unit buffers a FIFO buffer for buffering the integrated stream decoded in the extraction order, and the FIFO buffer. A synchronization buffer for designating an image having a common reproduction timing with the pointer in the extraction order in the same address area as the number of the plurality of image streams, and the frame synchronization processing unit further includes:
When the synchronization buffer designates an image corresponding to the number of image streams, a step of instructing the graphic accelerator to read image data from an address area designated by the pointer is executed, and the frame synchronization is performed. The video playback method, wherein the processing unit is configured as a plug-in, DLL, or runtime library of the main control device.   The moving image reproducing method according to claim 1, wherein the step of generating the integrated stream includes a step of decoding the encoded sequence according to an MPEG series format or an H.264 format.   The moving image reproduction method according to claim 1, further comprising a step of switching during reproduction of an image stream to be sent to the graphic accelerator.   The moving image reproduction method according to claim 1, further comprising a step of starting to write an image constituting the integrated stream when writing to the graphic accelerator is completed.   The moving image reproduction method according to claim 1, wherein the image stream includes different viewpoint images for reproducing 3D video. A video playback system that plays back an encoded stream in which a plurality of image streams are multiplexed in time synchronization, wherein the video playback system includes:
An image having a common playback timing of a plurality of different image streams is extracted from the plurality of image streams, and an encoded stream obtained by encoding an integrated stream generated by temporal convolution of the images in the extraction order is read. An interface part;
A decoding unit that decodes the encoded stream to generate the time-convolved integrated stream;
A frame synchronization processing unit that acquires the integrated stream from the decoding unit and separates the images of the respective image streams constituting the integrated stream, and writes the image in time synchronization;
A graphics accelerator that receives the output of the frame synchronization processor and generates at least one video signal of the plurality of image streams;
A display device that receives the video signal from the graphic accelerator and displays a moving image, wherein the frame synchronization processing unit buffers a FIFO buffer that decodes the integrated stream decoded in the extraction order; and the FIFO buffer A synchronization buffer for designating images having the same reproduction timing from the pointer in the order of extraction with the same address area as the number of the plurality of image streams, and the frame synchronization processing unit further includes the synchronization buffer As a plug-in, DLL, or runtime library that executes the step of instructing the graphic accelerator to read the image data from the address area specified by the pointer when the image corresponding to the number of image streams is specified Implemented,
Video playback system.   The moving image reproduction system according to claim 6, wherein the interface unit is a network interface that acquires the encoded stream via a public network or a storage interface that performs writing to or reading from a persistent storage device.   The moving image reproduction system according to claim 6 or 7, wherein the decoding unit decodes the encoded stream in an MPEG series format or an H.264 format.   The frame synchronization processing unit according to any one of claims 6 to 8, wherein when the synchronization buffer is full of the image, the frame synchronization processing unit stops writing the FIFO buffer and writes the image to the graphic accelerator. Video playback system.   The moving image reproducing system according to any one of claims 6 to 9, wherein the moving image reproducing system reproduces a 3D image or reproduces a game story.   An apparatus-executable program for executing a moving image reproduction method according to any one of claims 1 to 5, wherein a main control apparatus that reproduces an encoded stream obtained by multiplexing a plurality of image streams in time synchronization .