US20160088294A1 - Encoding device, decoding device, streaming system, and streaming method - Google Patents
Encoding device, decoding device, streaming system, and streaming method Download PDFInfo
- Publication number
- US20160088294A1 US20160088294A1 US14/842,214 US201514842214A US2016088294A1 US 20160088294 A1 US20160088294 A1 US 20160088294A1 US 201514842214 A US201514842214 A US 201514842214A US 2016088294 A1 US2016088294 A1 US 2016088294A1
- Authority
- US
- United States
- Prior art keywords
- image
- video stream
- bit rate
- images
- coded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/154—Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/23602—Multiplexing isochronously with the video sync, e.g. according to bit-parallel or bit-serial interface formats, as SDI
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/2365—Multiplexing of several video streams
Definitions
- Embodiments described herein relate generally to an encoding device, a decoding device, a streaming system, and a streaming method.
- adaptive streaming is known as a technology for transmitting a video stream through a network.
- the adaptive streaming is a video transmission technology without interruption in video reproduction, changing a bit rate or the like of a video stream to be transmitted, according to change in network communication bandwidth.
- dynamic encoding and “multiple encoding” are known to change the bit rate or the like of the video stream to be transmitted.
- bandwidth prediction or the like is used in transmission processing to dynamically adjust the bit rate of the video stream generated by an encoder.
- video streams having a plurality of bit rates are generated by the encoder, and a transmission/reception processing side dynamically selects, from the generated video streams having the bit rates, a video stream having a bit rate corresponding to bandwidth prediction or the like.
- the video source is converted by the encoder into the video streams having the plurality of different bit rates.
- all of the video streams are independent of each other.
- the switching of the video streams is limited to a random access time between the video streams (usually, I-picture or the like at the head of the GOP or the like).
- the GOP is short for “group of pictures”.
- the I-picture is an intra-coded image.
- FIG. 1 is a configuration diagram of a streaming system according to a first embodiment
- FIG. 2 is a block diagram of a server device provided in the streaming system according to the first embodiment
- FIG. 3 is a block diagram of a client device provided in the streaming system according to the first embodiment
- FIG. 4 is a flowchart illustrating an encoding operation in the server device of the streaming system according to the first embodiment
- FIG. 5 is a schematic view illustrating an encoding operation in the server device of the streaming system according to the first embodiment
- FIG. 6 is another schematic view illustrating an encoding operation in the server device of the streaming system according to the first embodiment
- FIG. 7 is a flowchart illustrating a decoding operation in the client device of the streaming system according to the first embodiment
- FIG. 8 is a schematic view illustrating an encoding operation of a server device provided in a streaming system according to a second embodiment
- FIG. 9 is a schematic view illustrating an encoding operation of a server device provided in a streaming system according to a third embodiment
- FIG. 10 is a schematic view illustrating an operation of a streaming system according to a fourth embodiment
- FIG. 11 is a schematic view illustrating an operation of a streaming system according to a fifth embodiment.
- FIG. 12 is a schematic view illustrating an operation of the streaming system according to the fifth embodiment.
- an encoding device includes an encoder.
- the encoder is configured to generate a video stream.
- the video stream includes at least a first image group including a plurality of images encoded at a first bit rate and a second image group including a plurality of images.
- the first image group includes an intra-coded image.
- the plurality of images of the second image group are associated with the intra-coded image serving as a reference image to decode an inter-coded image, and are encoded at a second bit rate different from the first bit rate and having reproduction timing in synchronization with the plurality of images of the first image group.
- the streaming system according to a first embodiment employs an adaptive streaming system using multiple encoding.
- the streaming system according to the first embodiment when an encoder converts a video source (master data) into streams having different bit rates, the streams are generated as video streams having correlation with one another (generating a meta file, in some cases), and when the video streams are switched from one to another, a different video stream is acquired in consideration of the correlation.
- the streaming system according to the first embodiment includes a server device 1 and a client device 2 .
- the server device 1 and the client device 2 are connected to each other through a network 3 such as the Internet.
- the server device 1 encodes, as described below, a predetermined video stream to be distributed through the network 3 .
- the client device 2 receives the video stream from the server device 1 , and decodes the video stream to be displayed on a monitor device or to be stored in a recorder device or the like.
- the server device 1 has a hardware configuration similar to a normal personal computer device. That is, the server device 1 includes a CPU 11 , a ROM 12 , a RAM 13 , a hard disk drive (HDD) 14 , and a communication unit 15 .
- the CPU 11 , the ROM 12 , the RAM 13 , the HDD 14 , and the communication unit 15 are connected to each other through a bus line 16 .
- the CPU is short for “central processing unit”.
- the ROM is short for “read only memory”.
- the RAM is short for “random access memory”.
- the HDD 14 stores an encoding program for encoding the video stream.
- the CPU 11 is operated according to the encoding program to function as the encoder 17 as an example of an encoding unit, and encodes and distributes the video stream.
- the encoding program may be stored in the ROM 12 or the RAM 13 .
- the client device 2 has a hardware configuration similar to a normal personal computer device. That is, the client device 2 includes a CPU 21 , a ROM 22 , a RAM 23 , a hard disk drive (HDD) 24 , and a communication unit 25 .
- the CPU 21 , the ROM 22 , the RAM 23 , the HDD 24 , and the communication unit 25 are connected to each other through a bus line 26 .
- the communication unit 25 is one example of a receiver.
- the HDD 24 stores a decoding program for decoding the video stream received from the server device 1 .
- the CPU 21 is operated according to the decoding program to function as a decoder 27 as an example of a decoding unit, and decodes the video stream received from the server device 1 .
- the decoding program may be stored in the ROM 22 or the RAM 23 .
- the encoding program and the decoding program may be provided to be stored in a computer-readable storage medium which may be provided as a computer program product, such as a CD-ROM or a flexible disk (FD) as a file in installable or executable format.
- the encoding program and the decoding program may be provided to be recorded in a computer-readable recording medium such as a CD-R, a DVD, a Blu-ray Disc (registered trademark), or a semiconductor memory.
- the DVD is short for “digital versatile disk”.
- the encoding program and the decoding program may be provided to be installed on a network such as the Internet. Further additionally, the encoding program and the decoding program may be provided to be previously incorporated into a ROM or the like in a device.
- the encoder 17 of the server device 1 encodes the video source divided into predetermined time slots to have different bit rates, and generates the GOP having each bit rate.
- the encoder 17 generates the GOPs each having a higher bit rate, a lower bit rate, and a bit rate between the higher bit rate and the lower bit rate (medium bit rate).
- the encoder 17 generates one video stream by assembling the GOP having each bit rate in the same time slot, and stores the one video stream in the HDD 14 .
- the GOP having a higher bit rate is one example of a first image group having a first bit rate.
- the GOPs having the intermediate bit rate and the lower bit rate are one example of a second image group having a second bit rate.
- Each GOP of each video stream is configured to be independently reproduced.
- the generated video stream is configured to be stored in the HDD 14 , but when the video stream is distributed in real time, for example as a live video, the video stream is encoded in real time, and distributed to the client device 2 .
- the encoder 17 is configured to assemble the GOP having each bit rate in the same time slot and generate one video stream.
- the encoder 17 may encode the video source so that one video stream only includes the GOP having one bit rate.
- the encoder 17 encodes the GOP having a higher bit rate, the GOP having a medium bit rate, and the GOP having a lower bit rate from the video source in the same time slot.
- the encoder 17 generates a first video stream including only the GOP having a higher bit rate, a second video stream including only the GOP having a medium bit rate, and a third video stream including only the GOP having a lower bit rate.
- the encoder 17 inserts, into a file of each video stream, reference information indicating the video stream including a reference image to be referenced upon decoding a P-picture.
- the encoder 17 adds, to each video stream, a reproduction control file indicating the video stream including the reference image to be referenced upon decoding the P-picture.
- the decoder 27 references the video stream designated by the reference information or the reproduction control file, and decodes the P-picture.
- the decoder 27 of the client device 2 measures a communication bandwidth of the video stream.
- the decoder 27 selects one of the bit rates of the GOP to be received according to the measured communication bandwidth.
- the decoder 27 makes, to the server device 1 , a request for distribution of the video stream having the GOP having the selected bit rate.
- the decoder 27 makes a request for distribution of the video stream having the GOP having a higher bit rate.
- the decoder 27 makes a request for distribution of the video stream having the GOP having a lower bit rate.
- the decoder 27 makes a request for distribution of the video stream having the GOP having a medium bit rate.
- the decoder 27 receives the video stream having the GOP having the selected bit rate from the server device 1 .
- the decoder 27 decodes the received video stream, and displays the decoded video stream on the monitor device or the like.
- the decoder 27 outputs the received video stream to a recording device, and records the video stream in the recording medium.
- the client device 2 is configured to calculate the network communication bandwidth, and makes, to the server device 1 , the request for distribution of the GOP having a bit rate corresponding to the calculated communication bandwidth.
- the server device 1 may calculate the network bandwidth, and switch, for distribution, the bit rate of the GOP to be transmitted.
- a stream designation method can be extended and achieved by defining reference to each image.
- any device other than the client device 2 or server device 1 may measure the network communication bandwidth, and the client device 2 or the server device 1 may receive a result of the measurement for use as information for switching the bit rate of the GOP. Further, the result of the measurement of the network communication bandwidth measured by the client device 2 may be received by the server device 1 , and the server device 1 may switch the bit rate of the GOP. Alternatively, the result of the measurement about the communication bandwidth measured by the server device 1 may be received by the client device 2 , and the client device 2 may switch the bit rate of the GOP.
- step S 1 the encoder 17 performs predetermined encoding processing by inserting reference information of the I-picture into each video stream so that other inter-coded images (P-picture) make reference to an intra-coded image (I-picture) of the video stream having a predetermined bit rate.
- the I-picture is an image having intra-coded data decodable independently of other images.
- the P-picture is an image only having inter-coded data representing a difference relative to a preceding frame.
- a B-picture may be employed with or in place of the P-picture.
- the B-picture is an image only having inter-coded data relative representing a difference relative to preceding and succeeding images.
- FIGS. 5 and 6 schematic views of encoding operations of the encoder 17 is illustrated.
- the encoder 17 encodes the video source according to the MPEG standard. Specifically, the encoder 17 encodes the video source each predetermined time, for example, encodes the video source for the first few seconds to generate the first video stream, and encodes the video source for the next few seconds following the video source for the first few seconds to generate the second video stream, and generates a sequential video stream.
- the encoder 17 generates each video stream to include the GOP having a higher bit rate, the GOP having a medium bit rate, and the GOP having a lower bit rate.
- the GOP is short for “group of pictures”.
- a reference sign “H 1 ” denotes the GOP having a higher bit rate, included in the first video stream.
- a reference sign “H 2 ” denotes the GOP having a higher bit rate, included in the second video stream.
- a reference sign “M 1 ” denotes the GOP having a medium bit rate, included in the first video stream.
- a reference sign “M 2 ” denotes the GOP having a medium bit rate, included in the second video stream.
- a reference sign “L 1 ” denotes the GOP having a lower bit rate, included in the first video stream.
- a reference sign “L 2 ” denotes the GOP having a lower bit rate, included in the second video stream.
- the encoder 17 performs encoding processing by designating the I-picture (I-pic) of the GOP having a higher bit rate, as an image to be referenced upon decoding the P-pictures (P-pic) of the GOP having each bit rate. Specifically, the encoder 17 inserts reference information for designating the I-picture to be referenced upon decoding the P-picture into a header or the like of each GOP. It is noted that, the encoder 17 may generate the reproduction control file indicating the I-picture to be referenced, and add the reproduction control file to each GOP or a corresponding video stream.
- step S 2 of a flowchart of FIG. 4 each video stream generated by the encoder 17 is distributed to each client device 2 while communication of the communication unit 15 is controlled by the CPU 11 . Therefore, the process of the flowchart of FIG. 4 is finished.
- the CPU 21 of the client device 2 functions as the decoder 27 according to the decoding program stored in the HDD 24 , and decodes the video stream received from the server device 1 .
- step S 11 of the flowchart of FIG. 7 when the video stream is received from the server device 1 , the decoder 27 decodes the I-picture designated by the reference information or the reproduction control file. Further, the decoder 27 decodes the P-pictures of the GOP having the bit rate selected by a user, using the decoded I-picture.
- FIGS. 5 and 6 illustrate decoding of the P-pictures of the GOP having each bit rate, using the decoded I-picture.
- FIGS. 5 and 6 illustrates an example of the I-picture of the GOP having a higher bit rate, designated by the reference information.
- the I-picture of the GOP having a higher bit rate is used to decode other P-pictures.
- the I-picture of the GOP having a medium bit rate may be designated by the reference information.
- the I-picture of the GOP having a medium bit rate is used to decode other P-pictures.
- the I-picture of the GOP having a lower bit rate may be designated by the reference information. In this configuration, the I-picture of the GOP having a lower bit rate is used to decode other P-pictures.
- step S 12 the video stream decoded as described above is output to and displayed on the monitor device. Therefore, decoding of the flowchart of FIG. 7 is finished.
- the video stream may be output to the recording device, and recorded in the recording medium. In this configuration, the video stream may be recorded without being decoded. Further, data received at each bit rate according to a communication rate may be directly recorded, or may be recorded after reconstruction of each video stream having been integrated and combined.
- the streaming system performs encoding processing by designating the I-picture having either bit rate, as an image used for decoding the P-picture, for each video stream.
- the designated I-picture is decoded, and the decoded I-picture is used to decode the P-pictures of the GOP designated by the user.
- the video streams are generated upon conversion of the video source by the encoder to the video streams having a plurality of different bit rates, and the video streams are all independent of each other. Therefore, timing allowing switching of the video stream is limited to timing allowing random access between the video streams (timing of the I-picture at the head of the GOP).
- the P-pictures of each GOP are decoded using the I-picture previously designated. Therefore, as indicated by KT 1 to KT 4 of FIG. 5 , the video stream can be switched in timing of the I-picture and in timing of all P-pictures of each GOP. Therefore, in the adaptive streaming system using the multiple encoding, the timing allowing switching of the video stream can be increased.
- the streaming system according to the first embodiment readily acquires the video stream having a lower bit rate, and deterioration in quality of the video can be prevented. Further, a delay can be prevented which is caused by the longer buffering time required to accomplish the acquisition of the whole video stream within the reproduction time of the video stream.
- the above streaming system according to the first embodiment is configured to reference the I-picture of any of the GOPs, upon decoding other P-pictures.
- the streaming system according to the second embodiment is configured so that the P-picture at the head is decoded with reference to the I-picture of any of the GOPs, and succeeding P-pictures are decoded with reference to a preceding P-picture having been decoded, respectively.
- FIG. 8 is a schematic view illustrating an encoding operation of such a streaming system according to the second embodiment.
- the encoder 17 encodes the video source by designating the I-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the P-picture at the head of the GOP having each bit rate, by the reference information or the reproduction control file.
- the encoder 17 encodes the video source by designating a preceding P-picture, as the image to be referenced upon decoding the second and subsequent P-pictures of the GOP having each bit rate, by the reference information or the reproduction control file.
- the I-picture is generated in the GOP having a higher bit rate. Therefore, the decoder 27 references the I-picture of the GOP having a higher bit rate, and decodes the P-picture at the head of the GOP having each bit rate. Further, the decoder 27 references the P-picture at the head decoded using the I-picture, and decodes the second P-picture. Further, the decoder 27 references the second P-picture decoded with reference to the P-picture at the head, and decodes the third P-picture. That is, the decoder 27 uses the P-picture having been decoded, positioned immediately before the P-picture to be decoded, and decodes the second and subsequent P-pictures.
- the streaming system according to the second embodiment can switch the video stream, in timing of the I-picture, and in timing of the P-pictures having been decoded of each GOP. Therefore, the streaming system according to the second embodiment can increase the timing allowing switching of the video stream, and an effect similar to that obtained in the above first embodiment can be obtained.
- the streaming system according to a third embodiment will be described.
- the above streaming system according to the second embodiment is configured so that the P-picture at the head is decoded with reference to the I-picture of any of the GOPs, and the succeeding P-pictures are decoded with reference to the preceding P-picture having been decoded, respectively.
- the streaming system according to the third embodiment includes a P-picture decoded with reference to a P-picture of the GOP including the I-picture, in order to decode the succeeding P-pictures with reference to the preceding P-picture having been decoded.
- FIG. 9 is a schematic view illustrating an encoding operation of such a streaming system according to the third embodiment.
- the encoder 17 encodes the video source by designating the I-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the P-picture at the head of the GOP having each bit rate, by the reference information or the reproduction control file. Further, the encoder 17 encodes the video source by designating a preceding P-picture, as the image to be referenced upon decoding the second and subsequent P-pictures of the GOP including the I-picture, by the reference information or the reproduction control file.
- the encoder 17 encodes the video source by designating a preceding P-picture, as the image to be referenced upon decoding the second P-picture of the GOP having a bit rate not including the I-picture, by the reference information or the reproduction control file. Further, the encoder 17 encodes the video source by designating the second P-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the third P-picture of the GOP having a bit rate not including the I-picture, by the reference information or the reproduction control file.
- the encoder 17 encodes the video source by designating the preceding third P-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the fourth P-picture of the GOP having a bit rate not including the I-picture, by the reference information or the reproduction control file.
- the I-picture is generated in the GOP having a higher bit rate. Therefore, the decoder 27 references the I-picture of the GOP having a higher bit rate, and decodes the P-picture at the head of the GOP having each bit rate. Additionally, the decoder 27 decodes each P-picture of the GOP including the I-picture using a preceding P-picture.
- the decoder 27 decodes the second P-picture of a GOP not including the I-picture, using the P-picture at the head decoded using the I-picture.
- the decoder 27 decodes the third P-picture of the GOP not including the I-picture, with reference to the second P-picture of the GOP having a higher bit rate, including the I-picture.
- the decoder 27 decodes the fourth P-picture of the GOP not including the I-picture, with reference to the preceding third P-picture. That is, the decoder 27 decodes a predetermined number of P-pictures, with reference to the I-picture or the P-picture of the GOP including the I-picture.
- the streaming system according to the third embodiment can provide an effect similar to that obtained in the above second embodiment, for example, switching of the video stream in timing of the I-picture and in timing of the P-picture at the head of each GOP, as illustrated in FIG. 9 .
- the switching is allowed only in one direction, i.e., from a video stream including a reference source I-picture or P-picture to a reference destination video stream.
- the streaming system according to the third embodiment references a preceding frame, so that the reference destination frame can have a reduced volume of data.
- each video stream includes the GOPs having the plurality of different bit rates, and a GOP including an image partially having a high quality image.
- the encoder 17 of the server device 1 encodes the video source so that each video stream includes the GOP having a partial high-quality bit rate, in addition to the GOPs having the plurality of different bit rates. That is, “H”, “M”, “L”, and “P” of FIG. 10 denote the GOP having a higher bit rate, the GOP having a medium bit rate, the GOP having a lower bit rate, and the GOP having a partial high-quality lower bit rate, respectively. That is, the GOP having a partial high-quality lower bit rate is a GOP having a low quality image. A part or parts of the low quality image include a high quality image.
- H 1 ”, “M 1 ”, “L 1 ”, and “P 1 ” denote the GOPs included in the first video stream.
- H 2 ”, “M 2 ”, “L 2 ”, and “P 2 ” denote the GOPs included in the second video stream
- H 3 ”, “M 3 ”, “L 3 ”, and “P 3 ” denote the GOPs included in the third video stream.
- each GOP of each video stream has configurations as described in the first to third embodiments, respectively. That is, each GOP of each video stream is configured so that the P-pictures are decoded with reference to the I-picture having either bit rate. Alternatively, each GOP of each video stream is configured so that the P-pictures are decoded with reference to a P-picture having been decoded with reference to the I-picture having either bit rate.
- the GOP having a lower bit rate includes a partial high-quality image
- the GOP having a medium bit rate or the GOP having a higher bit rate may include the partial high-quality image.
- the decoder 27 measures the communication bandwidth during reception of the video stream.
- the decoder 27 selects a bit rate to be received next and receives the video stream, according to the measured communication bandwidth.
- the decoder 27 selects, receives, and decodes the GOP having a higher bit rate. By contrast, when there is no available communication bandwidth, the decoder 27 selects and decodes the GOP having a lower bit rate. Further, when there is no available communication bandwidth, the decoder 27 selects, receives, and decodes the GOP having a partial high-quality lower bit rate in order to obtain a high-quality image. When the GOP having a partial high-quality lower bit rate is selected, an image having a partial high-quality image at a desired part can be obtained while the GOP having a lower bit rate is smoothly received.
- Such a streaming system according to the fourth embodiment is configured so that, when the streaming system handling the GOP having a partial high-quality image is achieved by switching the video stream according to the adaptive streaming, the number of combinations of video data to be generated can be reduced to reduce response delay, and an effect similar to that obtained in the above embodiments can be obtained.
- the streaming system according to a fifth embodiment will be described. Even if an image having a higher bit rate is employed, partial zoom up makes the image blurred. Therefore, the streaming system according to the fifth embodiment is configured so that a partial high-quality image selected from the entire image is distributed, and the user can see a desired part of the entire image having a high-quality.
- FIG. 11 is a schematic view illustrating generation of an image in the streaming system according to the fifth embodiment.
- the encoder 17 of the server device 1 employs a wide-angle, high-resolution video source of, for example, 3840 pixels by 2160 pixels.
- the encoder 17 generates a GOP of a reduced image obtained by reducing the wide-angle, high-resolution video source into 1280 pixels by 720 pixels.
- the encoder 17 divides the wide-angle, high-resolution video source into trimming areas having a predetermined number of pixels.
- the encoder 17 generates a GOP of each trimming area.
- First to n-th trimming areas illustrated in FIG. 11 represent trimming areas each having the predetermined number of pixels, obtained by dividing the wide-angle, high-resolution video source, respectively. In the n-th trimming area, “n” is a natural number.
- the streaming system according to the fifth embodiment is configured so that, in the GOP of each trimming area, the P-pictures are decoded with reference to the I-picture of the GOP, for example, of the reduced image, as described in the first embodiment (see FIGS. 5 and 6 ).
- the streaming system according to the fifth embodiment is configured so that the P-picture at the head of the GOP of each trimming area is decoded with reference to the I-picture of the GOP of the reduced image, and subsequently, the succeeding P-pictures are decoded with reference to a preceding P-picture having been decoded, as described in the second embodiment (see FIG. 8 ).
- the streaming system according to the fifth embodiment is configured so that, when the P-picture is decoded in the GOP of each trimming area, the P-pictures of the GOP of each trimming area are decoded using the I-picture of the GOP of the reduced image or the P-picture having been decoded of the GOP of the reduced image, as described in the third embodiment (see FIG. 9 ).
- FIG. 12 is a diagram illustrating a distribution mode of the video stream in the streaming system according to the fifth embodiment.
- the user receives a reduced entire image in the client device 2 .
- the decoder 27 of the client device 2 decodes the received entire image, and displays the decoded image on the monitor device.
- the client device 2 when the user designates a desired area in the entire image, the decoder 27 makes a request for transmission of an image of a designated area designated by the user, to the server device 1 .
- the server device 1 encodes a high-resolution zoomed image of the designated area corresponding to the request for transmission of the image, and transmit the encoded image to the client device 2 .
- the first to n-th trimming areas illustrated in FIG. 11 each represent a high-resolution zoomed image (GOP of trimming area) of each designated area designated by the user.
- the decoder 27 receives and decodes the transmitted high-resolution zoomed image, and displays the decoded image on the monitor device. Therefore, the partial high-quality image corresponding to the desired part of the entire image can be seen without blur.
- the user When a high-quality zoomed image of another part is required, the user further designates another desired part from the reduced entire image. Thereby, a partial high-quality zoomed image designated by the user is distributed from the server device 1 , similarly to the above description.
- the zoomed image distributed from the server device 1 is decoded, and the zoomed image can be seen through the monitor device.
- such a streaming system according to the fifth embodiment is also configured so that, when the streaming system handling the partial high-quality zoomed image is achieved by switching the video stream using the adaptive streaming, the number of combinations of the video data to be generated can be reduced to reduce response delay, and an effect similar to that obtained in the above embodiments can be obtained.
- the GOP for display of the entire image is generated by reducing the wide-angle, high-resolution video source of, for example, 3840 pixels by 2160 pixels into 1280 pixels by 720 pixels.
- the I-picture (P-picture in some cases) of the GOP for display of the entire image reduced into 1280 pixels by 720 pixels is used as a reference image to decode the P-picture of each trimming area.
- each GOP succeeding the GOP including the I-picture may be a GOP generated using the reduced image.
- the GOP including the I-picture is generated from the wide-angle, high-resolution video source of 3840 pixels by 2160 pixels.
- the GOP of each trimming area succeeding the GOP including the I-picture is generated based on the video source of 1280 pixels by 720 pixels obtained by reducing the video source of 3840 pixels by 2160 pixels.
- each P-picture of the GOP of each trimming area is decoded with reference to the I-picture (P-picture in some cases) generated based on the wide-angle, high-resolution video source of 3840 pixels by 2160 pixels.
- the client device 2 may make a transmission request or the server device 1 may make a distribution so that the first GOP including the I-picture corresponding to only the same part as the trimming area is transferred.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Description
- This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2014-191604, filed on Sep. 19, 2014; the entire contents of which are incorporated herein by reference.
- Embodiments described herein relate generally to an encoding device, a decoding device, a streaming system, and a streaming method.
- In this day, adaptive streaming is known as a technology for transmitting a video stream through a network. The adaptive streaming is a video transmission technology without interruption in video reproduction, changing a bit rate or the like of a video stream to be transmitted, according to change in network communication bandwidth.
- In such an adaptive streaming, “dynamic encoding” and “multiple encoding” are known to change the bit rate or the like of the video stream to be transmitted. In the dynamic encoding, bandwidth prediction or the like is used in transmission processing to dynamically adjust the bit rate of the video stream generated by an encoder. Further, in the multiple encoding, video streams having a plurality of bit rates are generated by the encoder, and a transmission/reception processing side dynamically selects, from the generated video streams having the bit rates, a video stream having a bit rate corresponding to bandwidth prediction or the like.
- In the adaptive streaming using the multiple encoding, when a video source is converted by the encoder into video streams having a plurality of different bit rates, all of the video streams are generated to be synchronized in reproduction time. Therefore, it can be achieved that, for example, a video stream having a higher bit rate is reproduced for the first few seconds, a video stream having a lower bit rate is reproduced for the next few seconds, and a video stream having a medium bit rate is reproduced for an additional few seconds. This configuration allows that even if the video streams having the different bit rates are switched from one to another during reproduction, the video can be reproduced without interruption.
- In the adaptive streaming using the multiple encoding, the video source is converted by the encoder into the video streams having the plurality of different bit rates. However, all of the video streams are independent of each other.
- Therefore, there has been a problem that the switching of the video streams is limited to a random access time between the video streams (usually, I-picture or the like at the head of the GOP or the like). The GOP is short for “group of pictures”. The I-picture is an intra-coded image.
- Further, there has been a problem that when acquisition of the video stream cannot be completed during a reproduction time of the video stream although the acquisition of the video stream is started, the video reproduction is interrupted. Additionally, there has been a problem that when the video is transmitted particularly on a network having a bad condition, the video stream having a higher bit rate cannot be acquired easily. Further, there has been a problem that a longer buffering time is required to accomplish the acquisition of the video stream within the reproduction time of the video stream. Still further, there has been a problem that the video to be reproduced disadvantageously brings about deterioration in quality, increase in delay, or the like.
- As described above, in the adaptive streaming using the multiple encoding, there has been problems in reproduction efficiency and reproduction quality.
-
FIG. 1 is a configuration diagram of a streaming system according to a first embodiment; -
FIG. 2 is a block diagram of a server device provided in the streaming system according to the first embodiment; -
FIG. 3 is a block diagram of a client device provided in the streaming system according to the first embodiment; -
FIG. 4 is a flowchart illustrating an encoding operation in the server device of the streaming system according to the first embodiment; -
FIG. 5 is a schematic view illustrating an encoding operation in the server device of the streaming system according to the first embodiment; -
FIG. 6 is another schematic view illustrating an encoding operation in the server device of the streaming system according to the first embodiment; -
FIG. 7 is a flowchart illustrating a decoding operation in the client device of the streaming system according to the first embodiment; -
FIG. 8 is a schematic view illustrating an encoding operation of a server device provided in a streaming system according to a second embodiment; -
FIG. 9 is a schematic view illustrating an encoding operation of a server device provided in a streaming system according to a third embodiment; -
FIG. 10 is a schematic view illustrating an operation of a streaming system according to a fourth embodiment; -
FIG. 11 is a schematic view illustrating an operation of a streaming system according to a fifth embodiment; and -
FIG. 12 is a schematic view illustrating an operation of the streaming system according to the fifth embodiment. - According to an embodiment, an encoding device includes an encoder. The encoder is configured to generate a video stream. The video stream includes at least a first image group including a plurality of images encoded at a first bit rate and a second image group including a plurality of images. The first image group includes an intra-coded image. The plurality of images of the second image group are associated with the intra-coded image serving as a reference image to decode an inter-coded image, and are encoded at a second bit rate different from the first bit rate and having reproduction timing in synchronization with the plurality of images of the first image group.
- While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel devices, systems, and methods described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the devices, systems, and methods described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirits of the inventions.
- As one example, a streaming system to which an encoding device, a decoding device, a streaming system, and a streaming method are applied will be described below according to embodiments. First, as one example, the streaming system according to a first embodiment employs an adaptive streaming system using multiple encoding. As described below, in the streaming system according to the first embodiment, when an encoder converts a video source (master data) into streams having different bit rates, the streams are generated as video streams having correlation with one another (generating a meta file, in some cases), and when the video streams are switched from one to another, a different video stream is acquired in consideration of the correlation.
- In
FIG. 1 , a configuration diagram of the streaming system according to the first embodiment is illustrated. As illustrated inFIG. 1 , the streaming system according to the first embodiment includes aserver device 1 and aclient device 2. Theserver device 1 and theclient device 2 are connected to each other through a network 3 such as the Internet. Theserver device 1 encodes, as described below, a predetermined video stream to be distributed through the network 3. Theclient device 2 receives the video stream from theserver device 1, and decodes the video stream to be displayed on a monitor device or to be stored in a recorder device or the like. - In
FIG. 2 , a block diagram of theserver device 1 is illustrated. As illustrated inFIG. 2 , theserver device 1 has a hardware configuration similar to a normal personal computer device. That is, theserver device 1 includes aCPU 11, aROM 12, aRAM 13, a hard disk drive (HDD) 14, and acommunication unit 15. TheCPU 11, theROM 12, theRAM 13, theHDD 14, and thecommunication unit 15 are connected to each other through abus line 16. The CPU is short for “central processing unit”. The ROM is short for “read only memory”. The RAM is short for “random access memory”. - The
HDD 14 stores an encoding program for encoding the video stream. TheCPU 11 is operated according to the encoding program to function as theencoder 17 as an example of an encoding unit, and encodes and distributes the video stream. The encoding program may be stored in theROM 12 or theRAM 13. - In
FIG. 3 , a block diagram of theclient device 2 is illustrated. As illustrated inFIG. 3 , theclient device 2 has a hardware configuration similar to a normal personal computer device. That is, theclient device 2 includes aCPU 21, aROM 22, aRAM 23, a hard disk drive (HDD) 24, and acommunication unit 25. TheCPU 21, theROM 22, theRAM 23, theHDD 24, and thecommunication unit 25 are connected to each other through abus line 26. Thecommunication unit 25 is one example of a receiver. - The
HDD 24 stores a decoding program for decoding the video stream received from theserver device 1. TheCPU 21 is operated according to the decoding program to function as adecoder 27 as an example of a decoding unit, and decodes the video stream received from theserver device 1. The decoding program may be stored in theROM 22 or theRAM 23. - In this example, a description will be made on the assumption that the
encoder 17 and thedecoder 27 are achieved by software. However, all or any of theencoder 17 and thedecoder 27 may be achieved by hardware. - Further, the encoding program and the decoding program may be provided to be stored in a computer-readable storage medium which may be provided as a computer program product, such as a CD-ROM or a flexible disk (FD) as a file in installable or executable format. Alternatively, the encoding program and the decoding program may be provided to be recorded in a computer-readable recording medium such as a CD-R, a DVD, a Blu-ray Disc (registered trademark), or a semiconductor memory. The DVD is short for “digital versatile disk”. Additionally, the encoding program and the decoding program may be provided to be installed on a network such as the Internet. Further additionally, the encoding program and the decoding program may be provided to be previously incorporated into a ROM or the like in a device.
- Next, a description will be made of operation of transmission reception of the video stream in the streaming system according to the first embodiment. First, the
encoder 17 of theserver device 1 encodes the video source divided into predetermined time slots to have different bit rates, and generates the GOP having each bit rate. As an example, theencoder 17 generates the GOPs each having a higher bit rate, a lower bit rate, and a bit rate between the higher bit rate and the lower bit rate (medium bit rate). Theencoder 17 generates one video stream by assembling the GOP having each bit rate in the same time slot, and stores the one video stream in theHDD 14. The GOP having a higher bit rate is one example of a first image group having a first bit rate. The GOPs having the intermediate bit rate and the lower bit rate are one example of a second image group having a second bit rate. Each GOP of each video stream is configured to be independently reproduced. - It is noted that, in this example, the generated video stream is configured to be stored in the
HDD 14, but when the video stream is distributed in real time, for example as a live video, the video stream is encoded in real time, and distributed to theclient device 2. - Further, in this example, the
encoder 17 is configured to assemble the GOP having each bit rate in the same time slot and generate one video stream. However, theencoder 17 may encode the video source so that one video stream only includes the GOP having one bit rate. In this configuration, theencoder 17 encodes the GOP having a higher bit rate, the GOP having a medium bit rate, and the GOP having a lower bit rate from the video source in the same time slot. Theencoder 17 generates a first video stream including only the GOP having a higher bit rate, a second video stream including only the GOP having a medium bit rate, and a third video stream including only the GOP having a lower bit rate. - Further, the
encoder 17 inserts, into a file of each video stream, reference information indicating the video stream including a reference image to be referenced upon decoding a P-picture. Alternatively, theencoder 17 adds, to each video stream, a reproduction control file indicating the video stream including the reference image to be referenced upon decoding the P-picture. Thedecoder 27 references the video stream designated by the reference information or the reproduction control file, and decodes the P-picture. - Next, the
decoder 27 of theclient device 2 measures a communication bandwidth of the video stream. Thedecoder 27 selects one of the bit rates of the GOP to be received according to the measured communication bandwidth. Thedecoder 27 makes, to theserver device 1, a request for distribution of the video stream having the GOP having the selected bit rate. - Specifically, when there is enough available communication bandwidth, the
decoder 27 makes a request for distribution of the video stream having the GOP having a higher bit rate. By contrast, when there is no available communication bandwidth, thedecoder 27 makes a request for distribution of the video stream having the GOP having a lower bit rate. Further, when there is some available communication bandwidth, thedecoder 27 makes a request for distribution of the video stream having the GOP having a medium bit rate. - The
decoder 27 receives the video stream having the GOP having the selected bit rate from theserver device 1. Thedecoder 27 decodes the received video stream, and displays the decoded video stream on the monitor device or the like. Alternatively, thedecoder 27 outputs the received video stream to a recording device, and records the video stream in the recording medium. - It is noted that, in this example, the
client device 2 is configured to calculate the network communication bandwidth, and makes, to theserver device 1, the request for distribution of the GOP having a bit rate corresponding to the calculated communication bandwidth. However, theserver device 1 may calculate the network bandwidth, and switch, for distribution, the bit rate of the GOP to be transmitted. Specifically, in a process for switching streaming in the adaptive streaming, a stream designation method can be extended and achieved by defining reference to each image. - Further, any device other than the
client device 2 orserver device 1, a router device, a dedicated measurement device, or the like may measure the network communication bandwidth, and theclient device 2 or theserver device 1 may receive a result of the measurement for use as information for switching the bit rate of the GOP. Further, the result of the measurement of the network communication bandwidth measured by theclient device 2 may be received by theserver device 1, and theserver device 1 may switch the bit rate of the GOP. Alternatively, the result of the measurement about the communication bandwidth measured by theserver device 1 may be received by theclient device 2, and theclient device 2 may switch the bit rate of the GOP. - Next, a detailed description will be made of encoding and decoding operations of the video stream in the streaming system according to the first embodiment. First, in a flowchart of
FIG. 4 , the flow of encoding operation of theserver device 1 will be illustrated. Upon encoding the video stream, theCPU 11 of theserver device 1 functions as theencoder 17. In step S1, theencoder 17 performs predetermined encoding processing by inserting reference information of the I-picture into each video stream so that other inter-coded images (P-picture) make reference to an intra-coded image (I-picture) of the video stream having a predetermined bit rate. - It is noted that, the I-picture is an image having intra-coded data decodable independently of other images. Further, the P-picture is an image only having inter-coded data representing a difference relative to a preceding frame. A B-picture may be employed with or in place of the P-picture. The B-picture is an image only having inter-coded data relative representing a difference relative to preceding and succeeding images.
- In
FIGS. 5 and 6 , schematic views of encoding operations of theencoder 17 is illustrated. Theencoder 17 encodes the video source according to the MPEG standard. Specifically, theencoder 17 encodes the video source each predetermined time, for example, encodes the video source for the first few seconds to generate the first video stream, and encodes the video source for the next few seconds following the video source for the first few seconds to generate the second video stream, and generates a sequential video stream. - As illustrated in
FIGS. 5 and 6 , theencoder 17 generates each video stream to include the GOP having a higher bit rate, the GOP having a medium bit rate, and the GOP having a lower bit rate. The GOP is short for “group of pictures”. InFIG. 5 , a reference sign “H1” denotes the GOP having a higher bit rate, included in the first video stream. Similarly, a reference sign “H2” denotes the GOP having a higher bit rate, included in the second video stream. Further, a reference sign “M1” denotes the GOP having a medium bit rate, included in the first video stream. Similarly, a reference sign “M2” denotes the GOP having a medium bit rate, included in the second video stream. Further, a reference sign “L1” denotes the GOP having a lower bit rate, included in the first video stream. Similarly, a reference sign “L2” denotes the GOP having a lower bit rate, included in the second video stream. - In examples illustrated in
FIGS. 5 and 6 , theencoder 17 performs encoding processing by designating the I-picture (I-pic) of the GOP having a higher bit rate, as an image to be referenced upon decoding the P-pictures (P-pic) of the GOP having each bit rate. Specifically, theencoder 17 inserts reference information for designating the I-picture to be referenced upon decoding the P-picture into a header or the like of each GOP. It is noted that, theencoder 17 may generate the reproduction control file indicating the I-picture to be referenced, and add the reproduction control file to each GOP or a corresponding video stream. - In step S2 of a flowchart of
FIG. 4 , each video stream generated by theencoder 17 is distributed to eachclient device 2 while communication of thecommunication unit 15 is controlled by theCPU 11. Therefore, the process of the flowchart ofFIG. 4 is finished. - Next, in a flowchart of
FIG. 7 , the flow of decoding the received video stream in theclient device 2 is illustrated. TheCPU 21 of theclient device 2 functions as thedecoder 27 according to the decoding program stored in theHDD 24, and decodes the video stream received from theserver device 1. - Specifically, in step S11 of the flowchart of
FIG. 7 , when the video stream is received from theserver device 1, thedecoder 27 decodes the I-picture designated by the reference information or the reproduction control file. Further, thedecoder 27 decodes the P-pictures of the GOP having the bit rate selected by a user, using the decoded I-picture. -
FIGS. 5 and 6 illustrate decoding of the P-pictures of the GOP having each bit rate, using the decoded I-picture. It is noted that,FIGS. 5 and 6 illustrates an example of the I-picture of the GOP having a higher bit rate, designated by the reference information. In this configuration, the I-picture of the GOP having a higher bit rate is used to decode other P-pictures. However, for example, the I-picture of the GOP having a medium bit rate may be designated by the reference information. In this configuration, the I-picture of the GOP having a medium bit rate is used to decode other P-pictures. Further, the I-picture of the GOP having a lower bit rate may be designated by the reference information. In this configuration, the I-picture of the GOP having a lower bit rate is used to decode other P-pictures. - In step S12, the video stream decoded as described above is output to and displayed on the monitor device. Therefore, decoding of the flowchart of
FIG. 7 is finished. It is noted that the video stream may be output to the recording device, and recorded in the recording medium. In this configuration, the video stream may be recorded without being decoded. Further, data received at each bit rate according to a communication rate may be directly recorded, or may be recorded after reconstruction of each video stream having been integrated and combined. - As apparent from the above description, the streaming system according to the first embodiment performs encoding processing by designating the I-picture having either bit rate, as an image used for decoding the P-picture, for each video stream. Upon decoding, the designated I-picture is decoded, and the decoded I-picture is used to decode the P-pictures of the GOP designated by the user.
- Normally, in the adaptive streaming system using the multiple encoding, the video streams are generated upon conversion of the video source by the encoder to the video streams having a plurality of different bit rates, and the video streams are all independent of each other. Therefore, timing allowing switching of the video stream is limited to timing allowing random access between the video streams (timing of the I-picture at the head of the GOP).
- However, in the streaming system according to the first embodiment, the P-pictures of each GOP are decoded using the I-picture previously designated. Therefore, as indicated by KT1 to KT4 of
FIG. 5 , the video stream can be switched in timing of the I-picture and in timing of all P-pictures of each GOP. Therefore, in the adaptive streaming system using the multiple encoding, the timing allowing switching of the video stream can be increased. - Further, even if a video is transmitted on a network having a bad condition, the streaming system according to the first embodiment readily acquires the video stream having a lower bit rate, and deterioration in quality of the video can be prevented. Further, a delay can be prevented which is caused by the longer buffering time required to accomplish the acquisition of the whole video stream within the reproduction time of the video stream.
- Next, the streaming system according to a second embodiment will be described. The above streaming system according to the first embodiment is configured to reference the I-picture of any of the GOPs, upon decoding other P-pictures. By contrast, the streaming system according to the second embodiment is configured so that the P-picture at the head is decoded with reference to the I-picture of any of the GOPs, and succeeding P-pictures are decoded with reference to a preceding P-picture having been decoded, respectively.
-
FIG. 8 is a schematic view illustrating an encoding operation of such a streaming system according to the second embodiment. In this example, theencoder 17 encodes the video source by designating the I-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the P-picture at the head of the GOP having each bit rate, by the reference information or the reproduction control file. Theencoder 17 encodes the video source by designating a preceding P-picture, as the image to be referenced upon decoding the second and subsequent P-pictures of the GOP having each bit rate, by the reference information or the reproduction control file. - In an example illustrated in
FIG. 8 , the I-picture is generated in the GOP having a higher bit rate. Therefore, thedecoder 27 references the I-picture of the GOP having a higher bit rate, and decodes the P-picture at the head of the GOP having each bit rate. Further, thedecoder 27 references the P-picture at the head decoded using the I-picture, and decodes the second P-picture. Further, thedecoder 27 references the second P-picture decoded with reference to the P-picture at the head, and decodes the third P-picture. That is, thedecoder 27 uses the P-picture having been decoded, positioned immediately before the P-picture to be decoded, and decodes the second and subsequent P-pictures. - Accordingly, as illustrated in
FIG. 8 , the streaming system according to the second embodiment can switch the video stream, in timing of the I-picture, and in timing of the P-pictures having been decoded of each GOP. Therefore, the streaming system according to the second embodiment can increase the timing allowing switching of the video stream, and an effect similar to that obtained in the above first embodiment can be obtained. - Next, the streaming system according to a third embodiment will be described. The above streaming system according to the second embodiment is configured so that the P-picture at the head is decoded with reference to the I-picture of any of the GOPs, and the succeeding P-pictures are decoded with reference to the preceding P-picture having been decoded, respectively. By contrast, the streaming system according to the third embodiment includes a P-picture decoded with reference to a P-picture of the GOP including the I-picture, in order to decode the succeeding P-pictures with reference to the preceding P-picture having been decoded.
-
FIG. 9 is a schematic view illustrating an encoding operation of such a streaming system according to the third embodiment. In this example, theencoder 17 encodes the video source by designating the I-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the P-picture at the head of the GOP having each bit rate, by the reference information or the reproduction control file. Further, theencoder 17 encodes the video source by designating a preceding P-picture, as the image to be referenced upon decoding the second and subsequent P-pictures of the GOP including the I-picture, by the reference information or the reproduction control file. - Further, the
encoder 17 encodes the video source by designating a preceding P-picture, as the image to be referenced upon decoding the second P-picture of the GOP having a bit rate not including the I-picture, by the reference information or the reproduction control file. Further, theencoder 17 encodes the video source by designating the second P-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the third P-picture of the GOP having a bit rate not including the I-picture, by the reference information or the reproduction control file. Further, theencoder 17 encodes the video source by designating the preceding third P-picture of the GOP having a higher bit rate, as the image to be referenced upon decoding the fourth P-picture of the GOP having a bit rate not including the I-picture, by the reference information or the reproduction control file. - In an example illustrated in
FIG. 9 , the I-picture is generated in the GOP having a higher bit rate. Therefore, thedecoder 27 references the I-picture of the GOP having a higher bit rate, and decodes the P-picture at the head of the GOP having each bit rate. Additionally, thedecoder 27 decodes each P-picture of the GOP including the I-picture using a preceding P-picture. - Further, the
decoder 27 decodes the second P-picture of a GOP not including the I-picture, using the P-picture at the head decoded using the I-picture. Thedecoder 27 decodes the third P-picture of the GOP not including the I-picture, with reference to the second P-picture of the GOP having a higher bit rate, including the I-picture. Further, thedecoder 27 decodes the fourth P-picture of the GOP not including the I-picture, with reference to the preceding third P-picture. That is, thedecoder 27 decodes a predetermined number of P-pictures, with reference to the I-picture or the P-picture of the GOP including the I-picture. - Therefore, the streaming system according to the third embodiment can provide an effect similar to that obtained in the above second embodiment, for example, switching of the video stream in timing of the I-picture and in timing of the P-picture at the head of each GOP, as illustrated in
FIG. 9 . In the streaming system according to the third embodiment, the switching is allowed only in one direction, i.e., from a video stream including a reference source I-picture or P-picture to a reference destination video stream. However, the streaming system according to the third embodiment references a preceding frame, so that the reference destination frame can have a reduced volume of data. - Next, the streaming system according to a fourth embodiment will be described. In the streaming system according to the fourth embodiment, each video stream includes the GOPs having the plurality of different bit rates, and a GOP including an image partially having a high quality image.
- In the streaming system according to the fourth embodiment, as illustrated in
FIG. 10 , theencoder 17 of theserver device 1 encodes the video source so that each video stream includes the GOP having a partial high-quality bit rate, in addition to the GOPs having the plurality of different bit rates. That is, “H”, “M”, “L”, and “P” ofFIG. 10 denote the GOP having a higher bit rate, the GOP having a medium bit rate, the GOP having a lower bit rate, and the GOP having a partial high-quality lower bit rate, respectively. That is, the GOP having a partial high-quality lower bit rate is a GOP having a low quality image. A part or parts of the low quality image include a high quality image. Further, “H1”, “M1”, “L1”, and “P1” denote the GOPs included in the first video stream. Similarly, “H2”, “M2”, “L2”, and “P2” denote the GOPs included in the second video stream, and “H3”, “M3”, “L3”, and “P3” denote the GOPs included in the third video stream. - Further, the GOPs of the respective video streams have configurations as described in the first to third embodiments, respectively. That is, each GOP of each video stream is configured so that the P-pictures are decoded with reference to the I-picture having either bit rate. Alternatively, each GOP of each video stream is configured so that the P-pictures are decoded with reference to a P-picture having been decoded with reference to the I-picture having either bit rate.
- It is noted that, in this example, the GOP having a lower bit rate includes a partial high-quality image, but the GOP having a medium bit rate or the GOP having a higher bit rate may include the partial high-quality image.
- In the
client device 2, thedecoder 27 measures the communication bandwidth during reception of the video stream. Thedecoder 27 selects a bit rate to be received next and receives the video stream, according to the measured communication bandwidth. - Specifically, when there is available communication bandwidth, the
decoder 27 selects, receives, and decodes the GOP having a higher bit rate. By contrast, when there is no available communication bandwidth, thedecoder 27 selects and decodes the GOP having a lower bit rate. Further, when there is no available communication bandwidth, thedecoder 27 selects, receives, and decodes the GOP having a partial high-quality lower bit rate in order to obtain a high-quality image. When the GOP having a partial high-quality lower bit rate is selected, an image having a partial high-quality image at a desired part can be obtained while the GOP having a lower bit rate is smoothly received. - Such a streaming system according to the fourth embodiment is configured so that, when the streaming system handling the GOP having a partial high-quality image is achieved by switching the video stream according to the adaptive streaming, the number of combinations of video data to be generated can be reduced to reduce response delay, and an effect similar to that obtained in the above embodiments can be obtained.
- Next, the streaming system according to a fifth embodiment will be described. Even if an image having a higher bit rate is employed, partial zoom up makes the image blurred. Therefore, the streaming system according to the fifth embodiment is configured so that a partial high-quality image selected from the entire image is distributed, and the user can see a desired part of the entire image having a high-quality.
-
FIG. 11 is a schematic view illustrating generation of an image in the streaming system according to the fifth embodiment. In the streaming system according to the fifth embodiment, as illustrated inFIG. 11 , theencoder 17 of theserver device 1 employs a wide-angle, high-resolution video source of, for example, 3840 pixels by 2160 pixels. Theencoder 17 generates a GOP of a reduced image obtained by reducing the wide-angle, high-resolution video source into 1280 pixels by 720 pixels. Further, theencoder 17 divides the wide-angle, high-resolution video source into trimming areas having a predetermined number of pixels. Theencoder 17 generates a GOP of each trimming area. First to n-th trimming areas illustrated inFIG. 11 represent trimming areas each having the predetermined number of pixels, obtained by dividing the wide-angle, high-resolution video source, respectively. In the n-th trimming area, “n” is a natural number. - Further, the streaming system according to the fifth embodiment is configured so that, in the GOP of each trimming area, the P-pictures are decoded with reference to the I-picture of the GOP, for example, of the reduced image, as described in the first embodiment (see
FIGS. 5 and 6 ). Alternatively, the streaming system according to the fifth embodiment is configured so that the P-picture at the head of the GOP of each trimming area is decoded with reference to the I-picture of the GOP of the reduced image, and subsequently, the succeeding P-pictures are decoded with reference to a preceding P-picture having been decoded, as described in the second embodiment (seeFIG. 8 ). Alternatively, the streaming system according to the fifth embodiment is configured so that, when the P-picture is decoded in the GOP of each trimming area, the P-pictures of the GOP of each trimming area are decoded using the I-picture of the GOP of the reduced image or the P-picture having been decoded of the GOP of the reduced image, as described in the third embodiment (seeFIG. 9 ). - Next,
FIG. 12 is a diagram illustrating a distribution mode of the video stream in the streaming system according to the fifth embodiment. In the streaming system according to the fifth embodiment, the user receives a reduced entire image in theclient device 2. Thedecoder 27 of theclient device 2 decodes the received entire image, and displays the decoded image on the monitor device. Next, in theclient device 2, when the user designates a desired area in the entire image, thedecoder 27 makes a request for transmission of an image of a designated area designated by the user, to theserver device 1. Theserver device 1 encodes a high-resolution zoomed image of the designated area corresponding to the request for transmission of the image, and transmit the encoded image to theclient device 2. The first to n-th trimming areas illustrated inFIG. 11 each represent a high-resolution zoomed image (GOP of trimming area) of each designated area designated by the user. Thedecoder 27 receives and decodes the transmitted high-resolution zoomed image, and displays the decoded image on the monitor device. Therefore, the partial high-quality image corresponding to the desired part of the entire image can be seen without blur. - When a high-quality zoomed image of another part is required, the user further designates another desired part from the reduced entire image. Thereby, a partial high-quality zoomed image designated by the user is distributed from the
server device 1, similarly to the above description. The zoomed image distributed from theserver device 1 is decoded, and the zoomed image can be seen through the monitor device. - As in the fourth embodiment, such a streaming system according to the fifth embodiment is also configured so that, when the streaming system handling the partial high-quality zoomed image is achieved by switching the video stream using the adaptive streaming, the number of combinations of the video data to be generated can be reduced to reduce response delay, and an effect similar to that obtained in the above embodiments can be obtained.
- It is noted that, in the fifth embodiment, the GOP for display of the entire image is generated by reducing the wide-angle, high-resolution video source of, for example, 3840 pixels by 2160 pixels into 1280 pixels by 720 pixels. In the fifth embodiment, the I-picture (P-picture in some cases) of the GOP for display of the entire image reduced into 1280 pixels by 720 pixels is used as a reference image to decode the P-picture of each trimming area.
- However, only an original resolution of the GOP including the I-picture referenced as the reference image upon decoding may be kept intact, and each GOP succeeding the GOP including the I-picture may be a GOP generated using the reduced image. In this configuration, the GOP including the I-picture is generated from the wide-angle, high-resolution video source of 3840 pixels by 2160 pixels. Further, the GOP of each trimming area succeeding the GOP including the I-picture is generated based on the video source of 1280 pixels by 720 pixels obtained by reducing the video source of 3840 pixels by 2160 pixels.
- Therefore, each P-picture of the GOP of each trimming area is decoded with reference to the I-picture (P-picture in some cases) generated based on the wide-angle, high-resolution video source of 3840 pixels by 2160 pixels. Thus, a high-quality image (image without blur) of the trimming area or the like can be obtained. It is noted that, when it is apparent that there is no available network communication bandwidth, the
client device 2 may make a transmission request or theserver device 1 may make a distribution so that the first GOP including the I-picture corresponding to only the same part as the trimming area is transferred. - While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
Claims (11)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/591,127 US11997302B2 (en) | 2014-09-19 | 2022-02-02 | Encoding device, decoding device, streaming system, and streaming method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014-191604 | 2014-09-19 | ||
JP2014191604A JP6239472B2 (en) | 2014-09-19 | 2014-09-19 | Encoding device, decoding device, streaming system, and streaming method |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/591,127 Continuation US11997302B2 (en) | 2014-09-19 | 2022-02-02 | Encoding device, decoding device, streaming system, and streaming method |
Publications (1)
Publication Number | Publication Date |
---|---|
US20160088294A1 true US20160088294A1 (en) | 2016-03-24 |
Family
ID=55526999
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/842,214 Abandoned US20160088294A1 (en) | 2014-09-19 | 2015-09-01 | Encoding device, decoding device, streaming system, and streaming method |
US17/591,127 Active US11997302B2 (en) | 2014-09-19 | 2022-02-02 | Encoding device, decoding device, streaming system, and streaming method |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/591,127 Active US11997302B2 (en) | 2014-09-19 | 2022-02-02 | Encoding device, decoding device, streaming system, and streaming method |
Country Status (2)
Country | Link |
---|---|
US (2) | US20160088294A1 (en) |
JP (1) | JP6239472B2 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200145685A1 (en) * | 2018-11-02 | 2020-05-07 | Kabushiki Kaisha Toshiba | Transmission device, communication system, transmission method, and computer program product |
WO2022120828A1 (en) * | 2020-12-11 | 2022-06-16 | 深圳市大疆创新科技有限公司 | Video frame extraction method, device, and storage medium |
US11895332B2 (en) | 2020-04-21 | 2024-02-06 | Kabushiki Kaisha Toshiba | Server device, communication system, and computer-readable medium |
US11936925B2 (en) | 2020-04-21 | 2024-03-19 | Kabushiki Kaisha Toshiba | Server device, information processing method, and computer program product |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6769231B2 (en) * | 2016-10-17 | 2020-10-14 | 富士通株式会社 | Moving image coding device, moving image coding method, moving image decoding device, moving image decoding method, and moving image coding computer program and moving image decoding computer program |
JP7302076B2 (en) * | 2018-11-02 | 2023-07-03 | 株式会社東芝 | Transmission device, server device, transmission method and program |
JP2022179992A (en) * | 2021-05-24 | 2022-12-06 | ソニーグループ株式会社 | Information processing system, information processing device, and information processing method |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7492819B2 (en) * | 2001-12-25 | 2009-02-17 | Panasonic Corporation | Video coding apparatus |
US8340183B2 (en) * | 2007-05-04 | 2012-12-25 | Qualcomm Incorporated | Digital multimedia channel switching |
US20130202034A1 (en) * | 2011-11-18 | 2013-08-08 | General Instrument Corporation | Explicit way for signaling a collocated reference picture for video coding |
US8773584B2 (en) * | 2011-08-11 | 2014-07-08 | Panasonic Corporation | Playback apparatus, playback method, integrated circuit, broadcast system, and broadcast method using a broadcast video and additional video |
US20150156511A1 (en) * | 2012-06-28 | 2015-06-04 | Axis Ab | System and method for encoding video content using virtual intra-frames |
US9060201B2 (en) * | 2008-10-28 | 2015-06-16 | Cisco Technology, Inc. | Stream synchronization for live video encoding |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6795501B1 (en) * | 1997-11-05 | 2004-09-21 | Intel Corporation | Multi-layer coder/decoder for producing quantization error signal samples |
JPH11341488A (en) * | 1998-05-21 | 1999-12-10 | Nec Corp | Digitizing system and method for moving image data |
JP2002064817A (en) * | 2000-08-21 | 2002-02-28 | Kddi Research & Development Laboratories Inc | Object scalably encoding device |
AU2002351218A1 (en) * | 2001-12-04 | 2003-06-17 | Polycom, Inc. | Method and an apparatus for mixing compressed video |
FI114527B (en) * | 2002-01-23 | 2004-10-29 | Nokia Corp | Grouping of picture frames in video encoding |
EP1670260A3 (en) * | 2002-01-23 | 2010-03-03 | Nokia Corporation | Grouping of image frames in video coding |
KR100608715B1 (en) | 2003-09-27 | 2006-08-04 | 엘지전자 주식회사 | SYSTEM AND METHOD FOR QoS-QUARANTED MULTIMEDIA STREAMING SERVICE |
JP5065051B2 (en) * | 2005-02-18 | 2012-10-31 | トムソン ライセンシング | Method for deriving encoding information of high-resolution image from low-resolution image, and encoding and decoding apparatus for realizing the method |
JP4703522B2 (en) * | 2006-09-08 | 2011-06-15 | 株式会社東芝 | Video decoding device |
EP1933563A1 (en) * | 2006-12-14 | 2008-06-18 | Thomson Licensing | Method and apparatus for encoding and/or decoding bit depth scalable video data using adaptive enhancement layer residual prediction |
JP5390017B2 (en) * | 2010-03-24 | 2014-01-15 | パナソニック株式会社 | Video processing device |
JP5085808B2 (en) * | 2011-01-21 | 2012-11-28 | パナソニック株式会社 | Moving picture coding apparatus and moving picture coding method |
US9953436B2 (en) * | 2012-06-26 | 2018-04-24 | BTS Software Solutions, LLC | Low delay low complexity lossless compression system |
JP5743968B2 (en) * | 2012-07-02 | 2015-07-01 | 株式会社東芝 | Video decoding method and video encoding method |
US9485526B2 (en) | 2012-07-16 | 2016-11-01 | Time Warner Cable Enterprises Llc | Multi-stream shared communication channels |
CN104396241B (en) * | 2012-09-28 | 2016-08-24 | 索尼公司 | Image processing apparatus and method |
WO2014106692A1 (en) * | 2013-01-07 | 2014-07-10 | Nokia Corporation | Method and apparatus for video coding and decoding |
US9654804B2 (en) * | 2014-09-03 | 2017-05-16 | Vigor Systems Inc. | Replacing video frames in a transport stream |
-
2014
- 2014-09-19 JP JP2014191604A patent/JP6239472B2/en active Active
-
2015
- 2015-09-01 US US14/842,214 patent/US20160088294A1/en not_active Abandoned
-
2022
- 2022-02-02 US US17/591,127 patent/US11997302B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7492819B2 (en) * | 2001-12-25 | 2009-02-17 | Panasonic Corporation | Video coding apparatus |
US8340183B2 (en) * | 2007-05-04 | 2012-12-25 | Qualcomm Incorporated | Digital multimedia channel switching |
US9060201B2 (en) * | 2008-10-28 | 2015-06-16 | Cisco Technology, Inc. | Stream synchronization for live video encoding |
US8773584B2 (en) * | 2011-08-11 | 2014-07-08 | Panasonic Corporation | Playback apparatus, playback method, integrated circuit, broadcast system, and broadcast method using a broadcast video and additional video |
US20130202034A1 (en) * | 2011-11-18 | 2013-08-08 | General Instrument Corporation | Explicit way for signaling a collocated reference picture for video coding |
US20150156511A1 (en) * | 2012-06-28 | 2015-06-04 | Axis Ab | System and method for encoding video content using virtual intra-frames |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200145685A1 (en) * | 2018-11-02 | 2020-05-07 | Kabushiki Kaisha Toshiba | Transmission device, communication system, transmission method, and computer program product |
US11647217B2 (en) | 2018-11-02 | 2023-05-09 | Kabushiki Kaisha Toshiba | Transmission device, communication system, transmission method, and computer program product |
US11895332B2 (en) | 2020-04-21 | 2024-02-06 | Kabushiki Kaisha Toshiba | Server device, communication system, and computer-readable medium |
US11936925B2 (en) | 2020-04-21 | 2024-03-19 | Kabushiki Kaisha Toshiba | Server device, information processing method, and computer program product |
WO2022120828A1 (en) * | 2020-12-11 | 2022-06-16 | 深圳市大疆创新科技有限公司 | Video frame extraction method, device, and storage medium |
Also Published As
Publication number | Publication date |
---|---|
US11997302B2 (en) | 2024-05-28 |
US20220159288A1 (en) | 2022-05-19 |
JP2016063481A (en) | 2016-04-25 |
JP6239472B2 (en) | 2017-11-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11997302B2 (en) | Encoding device, decoding device, streaming system, and streaming method | |
JP6841896B2 (en) | Transmission method and transmitter | |
KR102324326B1 (en) | Streaming multiple encodings encoded using different encoding parameters | |
US8914835B2 (en) | Streaming encoded video data | |
EP3445053B1 (en) | Transmission method and reception method | |
KR101510822B1 (en) | Adaptive trick play streaming | |
JP6225446B2 (en) | Moving image data distribution apparatus, method, program, and system | |
JP6184408B2 (en) | Receiving apparatus and receiving method thereof | |
US9392276B2 (en) | Video encoding apparatus, video decoding apparatus, video encoding method, and video decoding method | |
ES2894240T3 (en) | Method for initiating a transmission of a supplied streaming content to a client device and access point to implement this method | |
EP2642764B1 (en) | Transcoding a video stream to facilitate accurate display | |
EP2629476A1 (en) | Adaptive display streams | |
JP4715306B2 (en) | STREAM CONTROL DEVICE, STREAM REPRODUCTION METHOD, VIDEO RECORDING / REPRODUCTION SYSTEM | |
CN112672163B (en) | Transcoder adjustment for segment mobility | |
JP6193569B2 (en) | RECEPTION DEVICE, RECEPTION METHOD, AND PROGRAM, IMAGING DEVICE, IMAGING METHOD, AND PROGRAM, TRANSMISSION DEVICE, TRANSMISSION METHOD, AND PROGRAM | |
JP6258168B2 (en) | Distribution device, playback device, and distribution system | |
US10298651B2 (en) | Encoding device, decoding device, computer program product, and streaming system | |
JP2022031267A (en) | Streaming system and encoding program | |
JP6400163B2 (en) | Reception device, reception method, transmission device, transmission method, and program | |
JP7499583B2 (en) | Video distribution device, video transmission device, video distribution system, video distribution method, and video distribution program | |
US20210289238A1 (en) | Information processing apparatus and method | |
EP3026907B1 (en) | Encoding device, encoding method, and encoding program | |
US20210168472A1 (en) | Audio visual time base correction in adaptive bit rate applications | |
KR101336820B1 (en) | Apparatus and method for decoding specialized multi-channel trick mode | |
JP2005176068A (en) | Motion image distribution system and method therefor |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: KABUSHIKI KAISHA TOSHIBA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GONDO, SHUNICHI;KODAMA, TOMOYA;CHUJOH, TAKESHI;AND OTHERS;SIGNING DATES FROM 20151027 TO 20151130;REEL/FRAME:037284/0413 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |