JP4343913B2 - Image communication system, image distribution system, image communication method, image distribution system control program, and program recording medium - Google Patents

Description

  The present invention relates to image communication technology using hierarchically encoded data, and in particular, in an image distribution system that distributes video to a large number of client terminals, the bit rate of an image can be changed in multiple stages for each client. The present invention relates to an image distribution technology that can improve the distribution performance.

  As a hierarchical encoding method in image communication, for example, an MPEG-4 FGS (Fine Granularity Scalable) encoding method as described in Non-Patent Document 1 below is known. In the hierarchical coding system based on FGS coding, two encoded data of a base layer bit stream and an enhancement layer bit stream are generated, and an enhancement layer bit stream is adaptively cut, and an arbitrary bit rate can be set. It is possible to retrieve the stream.

  The purpose of such hierarchical encoding is to realize a function for eliminating image breaks by changing the bit rate in accordance with the change in the transmission band when the transmission band is changed. It also realizes the so-called One Source Multi Use, in which multiple bit rates can be extracted from a single source.

Further, as an image communication system to which a hierarchical coding system is applied, for example, the one described in Patent Document 1 below is known. In the image communication system described in Patent Document 1, a function that can freely change the bit rate of image data for each client terminal is realized by taking advantage of the characteristics of the hierarchical coding system.
JP-A-2005-341075 “AMENDMENT 4: Streaming video profile”, ISO / IEC 14496-2: 1999 / FDAM4

  In the conventional hierarchically coded image communication system, when the bit rate is adjusted by adaptively cutting the enhancement layer bitstream, the amount of data per frame of the enhancement layer is first calculated from the bit rate specified by the user. , Extract the encoded enhancement layer stream for each transmission unit, count up the number of bits, determine whether the data amount per frame is exceeded, and determine the maximum value that does not exceed the data amount per frame. I was sending. In other words, the bit rate extraction accuracy could be specified in detail up to the transmission unit, but on the other hand, the processing load related to counting and determination of the number of bits is large, which is one of the bottlenecks in improving the transmission performance. In an image distribution system in which a single server device is required to distribute images to many client terminals simultaneously, the processing load has become a big problem.

  The present invention solves the above-described problems, reduces the processing load on the server device that distributes the image encoded by the hierarchical encoding method in an image distribution system or the like, and reduces the number of bit streams that the server device can simultaneously distribute. The purpose is to increase.

In order to solve the above problems, the present invention provides an input video as encoded data of a base layer and an enhancement layer in an image communication system that performs communication of hierarchically encoded data composed of a base layer bit stream and an enhancement layer bit stream. Hierarchical encoding means for performing hierarchical encoding, and the bit rate range of the hierarchically encoded base layer and enhancement layer encoded data are further divided into a plurality of ranges for each hierarchy, and each divided bit rate range is divided into a plurality of ranges. A priority table creating means for creating a priority table for associating a priority number; a priority number giving means for assigning the priority number for each transmission unit of the encoded data based on the priority table; and indicating a bit rate of the data Bit rate instruction means , On the basis of the priority table, a priority number calculating means for calculating a priority number corresponding the bit rate designated by the bit rate instruction means, among the hierarchically encoded data to be transmitted, a priority number that is the calculated characterized in that it comprises a bit rate transcoding means to adjust the coding bit rate by performing code conversion to reduce the amount of data with reference to the priority number assigned to the encoded data.

  FIG. 1 is a diagram for explaining the outline of the present invention. In the hierarchical encoding method, input video is hierarchically encoded and output as a base layer bit stream 21 and an enhancement layer bit stream 22.

  According to the present invention, at the time of hierarchical encoding, the bit rate range of hierarchically encoded data is divided into a plurality of smaller ranges, and a priority table 24 that associates priority numbers with the divided ranges is created in advance. The priority number assigning unit 26 extracts encoded data for each transmission unit from the base layer bit stream 21 and the enhancement layer bit stream 22 and performs the following processing based on the created priority table 24. That is, the bit amount is counted up for each transmission unit, the priority table 24 is referenced, the corresponding priority number is obtained by comparing with each bit rate value corresponding to each priority number, and the priority number is added to the extracted encoded data. Is granted.

  The bit rate transcoding unit 14 calculates the priority number Pr corresponding to the instructed bit rate based on the priority table 24, and compares the calculated priority number Pr with the priority number assigned in the encoded data. , The following processing to reduce the data amount is performed. That is, the encoded data having the priority number assigned to the encoded data greater than the calculated priority number Pr is discarded, and only the encoded data equal to or less than Pr is transmitted.

  In this way, a bit rate that fits within the specified bit rate by performing processing of converting the instructed bit rate to a priority number and determining transmittable encoded data using the priority number in the encoded data. Can be realized.

  According to the present invention, in the image distribution system encoded by the hierarchical encoding method, it is possible to reduce the processing load related to the bit rate adjustment of the server device that distributes the image, as compared with the conventional method. Therefore, it is possible to increase the number of bit streams with which one server device can communicate simultaneously.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 2 shows the overall configuration of an image distribution system to which the present invention is applied. Each encoder terminal 2A, 2B encodes video input from a video source such as a camera or VTR by a hierarchical encoding method to generate a bit stream, and transmits the bit stream to the server device 1 via the network. The server device 1 cuts out only the amount of data requested from each of the client terminals 3A to 3C, and distributes the hierarchically encoded data including the base layer bit stream and the extended bit layer stream to the individual client terminals 3A to 3C. To do.

  FIG. 3 shows a configuration example of the server device, the encoder terminal, and the client terminal. The server device 1 includes an encoded data receiving unit 10 that receives a scalable bit stream from the encoder terminal 2, a priority table receiving unit 11 that receives a priority table from the encoder terminal 2, and stream identification information from the client terminal 3. A stream information receiving unit 12 that receives bit rate information, a priority number calculating unit 13 that calculates a priority number by collating the received bit rate information with a priority table, and a scalable bit stream received by the encoded data receiving unit 10 A bit rate transcoding unit 14 for converting the bit rate and an encoded data transmission unit 15 for transmitting the scalable bit stream generated by the bit rate transcoding unit 14 to each client terminal 3 are provided.

  The encoder terminal 2 hierarchically encodes video input from a video source such as a camera to generate a scalable bitstream, and divides the encoding bit rate range during hierarchical encoding into a plurality of stages, A priority table creation unit 23 that creates a priority table 24 that associates each range with a priority number; a priority table transmission unit 25 that sends the priority table 24 created by the priority table creation unit 23 to the server device 1; A priority number giving unit 26 for taking out a video packet from the scalable bitstream encoded by the hierarchical encoding unit 20 and giving a priority number while referring to the priority table created by the table creating unit 23; And a coded data transmission unit 27 for transmitting a scalable bitstream priority is granted by the priority number giving unit 26 to the server device 1.

  The client terminal 3 includes a stream information transmission unit 30 that transmits to the server device 1 identification information of a stream received by the device itself and bit rate information of a desired data amount, and encoded data that receives a scalable bit stream from the server device 1 The receiving unit 31 includes a hierarchical decoding unit 32 that decodes the received scalable bitstream, and a display unit 33 that displays the decoded image on a display or the like.

  The desired bit rate at the time of receiving data at the client terminal 3 may be displayed on the display unit 33 by, for example, a slider bar for inputting the bit rate, and may be designated by the user. You may determine automatically according to a communication condition. In addition, the client terminal 3 may specify the bit rate at the time of receiving data, and the device on the network or the server device 1 may specify it according to the network status.

  The operation of the present embodiment will be described below with reference to FIG. The encoder terminal 2 generates a scalable bit stream composed of the base layer bit stream 21 and the enhancement layer bit stream 22 at the hierarchical encoding unit 20 from the input video. The priority table creation unit 23 creates a priority table based on the coding parameters in the hierarchical coding unit 20.

FIG. 4 shows a configuration example of the priority table. The priority table 24 includes a priority number and a bit rate value indicating the maximum value of the bit rate range corresponding to the priority number. In the priority table creation unit 23, the encoding parameters for encoding in the hierarchical encoding unit 20, that is, the encoding bit rate of the base layer is Bb [bps], and the encoding bit rate of the enhancement layer is Be [bps]. , The frame rate of the base layer is f [fps], the ratio of the data amount of the I picture and P picture is IPr, and the GOV (data from the I picture to the next I picture) interval is T _GOV [second (sec)] When the bit rate of the base layer bit stream is divided into two stages and the bit rate of the enhancement layer bit stream is divided into N-2 stages, that is, the bit rate of the scalable bit stream combining the base layer bit stream and the enhancement layer bit stream is When dividing into N stages, the bit rate corresponding to each priority number Value is calculated as follows.

B ₁ = Bb / (1+ (T _GOV * f−1) * IPr)
B ₂ = Bb
Bn = Bb + (n-2) * Be / (N-2) n = 3,4, ..., N
The priority number assigning unit 26 receives the base layer bit stream 21 and the enhancement layer bit stream 22 from the hierarchical encoding unit 20 and refers to the priority table 24 created by the priority table creating unit 23 while referring to the video packet in the bit stream. Is given a priority number.

  FIG. 5 is a flowchart for explaining priority number assignment processing for the base layer bitstream. In general, a bit stream encoded by the MPEG-4 encoding method is composed of video packets as shown in FIG. 5B in order to cope with transmission errors such as packet loss on the network. The priority number assignment process is performed for each video packet.

  In step S1, one video packet is extracted from the base layer bitstream. In step S2, it is determined whether or not the extracted video packet is encoded data of an I picture. If the encoded data is an I picture, an extension header with the priority number set to “1” is attached to the video packet. On the other hand, if it is not encoded data of an I picture, an extension header with a priority number set to “2” is added to the video packet. The above process is repeated for a series of base layer bitstreams. FIG. 5C shows a bit stream after the priority number is assigned.

  FIG. 6 is a flowchart for explaining the priority number assigning process for the enhancement layer bitstream.

In step S11, the bit count value for counting the bit amount is initialized to "0", and the priority number Pn of the current video packet is initialized to 3. Further, since the bit rate corresponding to the priority number 3 is B ₃ in the example of FIG. 4 from the priority table 24, if the frame rate is f, by dividing B ₃ by f, the per frame per priority number 3 The bit amount Bv is calculated.

  Next, in step S12, one video packet is extracted from the enhancement layer bitstream. In step S13, the packet size of the extracted video packet is acquired, and in step S14, the bit count value is counted up by the packet size. If the bit count value is equal to or less than the bit amount Bv per frame in the priority number 3, the process proceeds to step S18, and an extension header in which the priority number Pn (3 in this case) is set is added to the video packet.

  On the other hand, if the bit count value exceeds Bv, the process proceeds to step S16, and the priority number Pn is incremented by 1 (in this case, 4). Subsequently, the process proceeds to step S17, where the bit amount Bv per frame at the priority number Pn (4 in this case) is calculated, and then the process proceeds to step S18, where the priority number Pn (4 in this case) is set in the video packet. Add an extension header.

  Next, in step S19, it is determined whether or not the video packet to which the extension header is added is the final video packet of the frame data. If the video packet is not the final of the frame data, the process returns to step S13. In this way, the processing from step S13 to step S18 is repeated until the final video packet of the frame.

  In the above example, the bit rate range of the base layer bit stream is divided into two stages. However, the video packet of the I picture encoded data is added by determining whether the video packet is the encoded data of the P picture. The priority number 1 may be assigned, the priority number 2 may be assigned to the video packet of the encoded data of the P picture, and the priority number 3 may be assigned to the video packet of the encoded data of the B picture.

Further, although the bit rate range of the enhancement layer bit stream is divided at equal intervals, the following formulas Bn = Bb + .alpha.n * (n-2) * Be / (N-2) n = 3,4,.
(Where αn is a positive coefficient)
The bit rate range may be determined by weighted division by calculating Bn using.

  In addition, although priority numbers are assigned in units of video packets, a plurality of video packets are grouped into one RTP (Real-time Transport Protocol) packet, and priority numbers are assigned to RTP extension headers in units of RTP packets. May be.

  The client terminal 3 selects one specific stream among the streams from the plurality of encoder terminals 2 received by the server apparatus 1, and the stream information transmitting unit 30 determines an identifier and reception bit rate for designating the stream. The server device 1 is notified. The server device 1 receives the stream identification information and the reception bit rate from the client terminal 3 at the stream information receiving unit 12.

  In the priority number calculation unit 13, the priority of the corresponding encoder terminal from the stream identification information received by the stream information receiving unit 12 in the priority table 24 from the encoder terminals 2 A and 2 B received in advance by the priority table receiving unit 11. The table 24 is selected, and the priority number is calculated from the reception bit rate based on the selected priority table 24. That is, if the reception bit rate is Br, the bit rate value corresponding to the priority number n of the priority table 24 is Bn, and Bn ≦ Br <Bn + 1, then n is the priority number Pr corresponding to the reception bit rate Br and To do.

  Based on the calculated priority number, the bit rate transcoding unit 14 adjusts the transmission bit rate.

  FIG. 7 is a flowchart showing the processing of the bit rate transcoding unit 14. First, in step S21, one video packet with a priority number received by the encoded data receiving unit 10 is taken out and copied.

  In step S22, the priority number Pcur assigned to the video packet is obtained by referring to the extension header of the copied video packet.

  In step S23, it is determined whether or not the priority number Pcur acquired in step S22 is larger than Pr calculated by the priority number calculation unit 13. If Pcur is less than or equal to Pr, the process proceeds to step S24, and the corresponding copied video packet is output to the encoded data transmission unit 15.

  On the other hand, if Pcur is larger than Pr, the process proceeds to step S25 and the corresponding copied video packet is discarded. The above process is repeated for a series of base layer bitstreams and enhancement layer bitstreams.

  The processes performed by the server device 1, the encoder terminal 2, and the client terminal 3 can be realized not only by hardware and firmware, but also by a computer and a software program, and the program can be read by the computer. It can be provided by being recorded on a recording medium or via a network.

It is a figure for demonstrating the outline | summary of this invention. It is a figure which shows the example of the image delivery system to which this invention is applied. It is a figure which shows the structural example of the encoder terminal which concerns on embodiment of this invention, a server apparatus, and a client terminal. It is a figure which shows the structural example of a priority table. It is a flowchart of the priority number provision process in a base layer bit stream. It is a flowchart of the priority number provision process in an enhancement layer bitstream. It is a flowchart of a bit rate transcoding part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Server apparatus 2 Encoder terminal 3 Client terminal 10 Encoded data receiving part 11 Priority table receiving part 12 Stream information receiving part 13 Priority number calculation part 14 Bit rate transcoding part 15 Encoded data transmission part 20 Hierarchical encoding part 21 Base layer Bit stream 22 Enhancement layer bit stream 23 Priority table creation unit 24 Priority table 25 Priority table transmission unit 26 Priority number assignment unit 27 Encoded data transmission unit 30 Stream information transmission unit 31 Encoded data reception unit 32 Hierarchical decoding unit 33 Display unit

Claims (6)

In an image communication system that performs communication of hierarchically encoded data composed of a base layer bit stream and an enhancement layer bit stream,
Hierarchical encoding means for hierarchically encoding input video into encoded data of a base layer and an enhancement layer;
The bit rate range of the hierarchically encoded base layer and enhancement layer encoded data is further divided into a plurality of ranges for each layer, and a priority table is created that associates a priority number with each divided bit rate range. Priority table creation means;
Priority number assigning means for assigning the priority number for each transmission unit of the encoded data based on the priority table;
Bit rate indicating means for indicating the bit rate of the data;
Priority number calculating means for calculating a corresponding priority number from the bit rate indicated by the bit rate indicating means based on the priority table;
The encoding bit rate is adjusted by performing code conversion to reduce the data amount while referring to the calculated priority number and the priority number given in the encoded data among the hierarchical encoded data to be transmitted. An image communication system comprising: a bit rate transcoding means for performing the operation. One or a plurality of encoder terminals, a plurality of client terminals, and a server device connected to the encoder terminals and the client terminals via a network, and a hierarchically encoded layer consisting of a base layer bit stream and an enhancement layer bit stream Image distribution system that distributes data,
The encoder terminal is
Means for hierarchically encoding input video in the terminal into encoded data of a base layer and an enhancement layer ;
The bit rate range of the hierarchically coded base layer and enhancement layer encoded data is further divided into a plurality of ranges for each layer, and a priority number is assigned to each divided bit rate range. Means to create a priority table
Means for transmitting the priority table to the server device;
Priority number assigning means for assigning the priority number for each transmission unit of the encoded data based on the priority table;
Means for transmitting the encoded data to which the priority number is assigned to the server device;
The client terminal is
Means for notifying the server device of the bit rate of data received by the terminal;
Means for receiving hierarchically encoded data from the server device;
Means for hierarchically decoding received hierarchically encoded data;
Means for displaying the hierarchically decoded image,
The server device
Priority number calculating means for calculating a corresponding priority number from the bit rate notified from the client terminal based on the priority table received from the encoder terminal;
The encoding bit rate is adjusted by performing code conversion to reduce the data amount while referring to the calculated priority number and the priority number given in the encoded data among the hierarchical encoded data to be transmitted. An image distribution system comprising: a bit rate transcoding means for performing the operation. The image delivery system according to claim 2,
Instead of having a means for notifying the server device of the bit rate of data received by the client terminal, the device on the network or the server device determines the bit rate of data distributed to each client terminal, and An image distribution system comprising means for notifying priority number calculation means. An image communication method for communicating hierarchically encoded data composed of a base layer bit stream and an enhancement layer bit stream,
A hierarchical encoding process in which input video is hierarchically encoded into base layer and enhancement layer encoded data;
The bit rate range of the hierarchically encoded base layer and enhancement layer encoded data is further divided into a plurality of ranges for each layer, and a priority table is created that associates a priority number with each divided bit rate range. Priority table creation process,
A priority number assigning step for assigning the priority number for each transmission unit of the encoded data based on the priority table;
A bit rate indicating process for indicating the bit rate of the data;
A priority number calculating step of calculating a corresponding priority number from the bit rate indicated by the bit rate indicating step based on the priority table;
The encoding bit rate is adjusted by performing code conversion to reduce the data amount while referring to the calculated priority number and the priority number given in the encoded data among the hierarchical encoded data to be transmitted. An image communication method comprising: performing a bit rate transcoding process. A program for causing a computer to execute processing used to realize the image distribution system according to claim 2,
As each means of each unit and the encoder terminal provided in the server device of an image distribution system according to claim 2 or claim 3 comprises, an image distribution system control program for causing the computer to function. A computer-readable recording medium recording a program for causing a computer to execute processing used to realize the image delivery system according to claim 2 or claim 3,
Each unit and as each unit in which the encoder terminal comprises a program storage medium storing an image distribution system control program for causing the computer to function included in the server device of an image distribution system according to claim 2 or claim 3.

Images