JP4295079B2 - Special video data processing method, special video data processing apparatus and special video data processing system - Google Patents

Description

  The present invention relates to a video distribution system that records and stores video in a server and distributes a video stream stored in response to a request from a terminal to a requested terminal. The present invention relates to a modified data processing method and data processing apparatus for realizing special reproduction such as fast forward, fast rewind, and pause.

Conventionally, particularly in a video playback system using a network, a playback speed and / or a playback direction of a video stream compressed by MPEG-2 or the like (hereinafter referred to as a normal playback stream to be distinguished from a special playback stream) is set. When the modified special playback is to be executed, only the intra-frame coded pictures (I frames) that are intermittent in time are extracted from the normal playback stream, and the extracted I frames are continuously delivered to the terminal. In general, it is configured to reproduce after reception. However, in this method, only the I frame is extracted from the normal reproduction stream, so that the stream may deviate from the original MPEG-2 structure. For this reason, there has been a problem that some mechanism has to be prepared in order to enable the reproduction decoder to reproduce a special reproduction stream composed of only I frames. In addition, since only I frames having a large code amount are extracted at high speed and continuously distributed, the transfer code amount (transfer rate) per unit time becomes very large. There has been a problem that the buffer of the device or the separation device has overflowed, exceeded the bandwidth limit of the network, or has overflowed the decoder buffer on the reproduction side. Therefore, it is necessary to take measures such as expanding the bandwidth of the network and increasing the size of each buffer memory in the transmission path in order to enable transfer of the special reproduction stream having an increased transfer rate. In particular, the problem regarding the network bandwidth is serious, and the special reproduction method using only the I frame generally requires about three times the bandwidth compared with the normal reproduction although it depends on the code amount of the I frame. However, it is difficult to actually increase the bandwidth for special reproduction.
Therefore, as one countermeasure, a method of separately preparing a special reproduction stream is used. However, with this method, although the system configuration is simple, since a special playback stream is prepared separately in addition to the normal playback stream, the amount of data to be stored increases, and the normal playback stream and the special playback stream are associated and managed. There was a problem that it was necessary to do.

  With respect to this problem, Patent Document 1 inserts a repeat picture encoded with a motion vector of a macroblock as 0 and a prediction error as 0 after the data encoded as a special reproduction stream in a frame or in a field, and distributes the data. A way to do it was proposed. According to this method of inserting a repeat picture, the transfer rate can be greatly reduced by adding a repeat picture with a very small code amount following an I frame with a large code amount in execution of special reproduction. The transfer rate is prevented from increasing, and there is no need to secure a large memory for special playback. Furthermore, the special playback stream that is generated can be syntactically configured in the same way as a normal playback stream, so there is no need to add additional logic or circuitry to handle the special stream, and it is also necessary to switch to dedicated logic during special playback. Absent.

  However, with this method, even when converting multiple streams at the same time or converting a stream with a high rate, the I frame is selectively extracted from the normal playback stream at a fixed rate according to the speed of special playback. There is a problem in that a special reproduction stream cannot be generated because the load on the storage device to be stored and the input transmission device is increased.

As another prior art, as described in Patent Document 2, a request is issued from the terminal side, and a special playback video is prepared and distributed on the server side based on this request, and the terminal simply receives it as it is. The system is shown. In this case, an I picture or the like is selected in the order in which the data conversion unit is designated for special reproduction, is output to the scheduling unit, forms a stream, and is sent to the terminal. However, in this method, since the order and number of frames are determined depending on the type of special playback, when playback requests from a plurality of terminals are concentrated, the playback content and start time generally differ. May not be able to be read out in an attempt to retrieve data from the data storage unit.
Japanese Patent Laid-Open No. 08-098136 JP 2001-359072 A

  In the conventional method of performing special reproduction, even if the improved method disclosed in Patent Document 2 is used, even when a plurality of streams are converted at the same time or when a stream with a high reproduction rate is converted, an intraframe code at a fixed rate is used. Since extracted pictures are extracted, there is a problem that a read load from a storage device or a transmission device becomes high, and a special playback stream may not be generated.

  The present invention has been made to solve the above-described problems. When a special playback stream is generated from an accumulated normal playback stream, or when a normal playback stream is input from a transmission device, the special playback stream is generated. In this case, the present invention provides a video data processing method and a video data processing apparatus that can generate data at a high speed with little load on reading.

A special video data processing device according to the present invention is a server device that transmits or relays a special playback video stream according to a request from a stream of stored video encoded using inter-frame prediction.
An analysis unit that analyzes a picture by determining a transmission interval of an intra-frame coded picture extracted according to a load for reading a predetermined video from the stored video, and an intra-frame coded picture based on an analysis result of the analysis unit A special reproduction processing unit provided with an intra-frame coded picture conversion processing unit for changing parameters and a repeat picture addition unit showing the same display as the extracted intra-frame coded picture is provided.
A special video data processing method according to the present invention is a video data processing method for transmitting or relaying a special playback video stream according to a request from a stored video stream encoded using inter-frame prediction.
A transmission interval setting / picture analysis step of performing picture analysis by determining a transmission interval of an intra-frame encoded picture according to a load for reading a predetermined video from the stored video;
An intra-frame coded picture data conversion step for changing parameters of the intra-frame coded picture based on the analysis result of the picture analysis;
A repeat picture adding step for adding a repeat picture showing the same display as the extracted intra-frame encoded picture.

  According to the present invention, since the I frame read interval is changed according to the read load amount per predetermined time, there is an effect that the special reproduction stream can be generated and distributed without changing the transmission rate to the terminal.

Embodiment 1 FIG.
FIG. 1 is a video storage / distribution comprising a server that receives and accumulates and redistributes video from a network according to an embodiment of the present invention, and a terminal that receives and reproduces video distributed from the server. Shows the system. In this video storage / delivery system, normal playback at the same playback speed as the speed at which the video was recorded, and special playback with the speed changed from normal playback, for example, twice or five times the normal playback speed. Fast-forward playback with a speed of 15 times, etc., fast-rewind playback with the playback direction reversed with respect to fast-forward playback, and pause for continuously displaying the same video are realized. Here, it is assumed that the video stream used in the present embodiment is recorded as an ISO / IEC13818-1, so-called MPEG-2 program stream.

  The server 1 has the following elements as a hardware configuration. That is, first, the storage unit 2 for storing the video stream is provided, and a plurality of normal reproduction streams are stored. When storing the video stream in the server 1, the video normal video stream received via the network receiving unit 21 is divided into GOP (Group of Picture) units by the GOP extracting unit 22, and the writing unit 3 stores the storing unit. 2 is written. The GOP extraction unit 22 records the address of the GOP unit data written in the storage unit 2 and the reproduction time stamp in the address map 23. The reading unit 4 reads the GOP data recorded at the corresponding address of the corresponding stream from the storage unit 2 based on the time stamp and address information recorded in the address map 23 according to an instruction from the reproduction control unit 6. In addition, the read load detection unit 19 detects a load amount when reading from the storage unit 2 and notifies the reproduction control unit 6 of the load amount. The playback control unit 6 controls the flow of the stream from the storage unit 2 by controlling the reading unit 4, the switch 5, and the switch 7 depending on whether a playback request from the terminals 51 to 53 described later is normal playback or special playback. . Further, the reproduction control unit 6 controls the number of I pictures read by the I picture extraction unit 12 according to the read load amount from the read load detection unit 19. The I picture extraction unit 12 extracts from the normal playback stream the sequence header indicating the head of the MPEG-2 video unit to the intra-frame encoded picture data which is the first picture data. In this embodiment, since the normal reproduction stream is stored in the program stream format, demultiplexing that separates the program stream into elementary streams is also performed in parallel with the extraction operation.

  The special reproduction processing unit 11 includes an analysis unit 13 that analyzes an input intra-frame coded picture, a decoder buffer calculation unit 14 that calculates a decoder buffer of a special reproduction stream to be generated, and an input intra-frame coded picture. An intra-frame encoded picture data conversion processing unit 15 that changes the encoding parameter, a repeat picture adding unit 16 that generates a repeat picture that displays the same video as the intra-frame encoded picture, and a generated code amount of the special reproduction stream to be generated is controlled And a PS conversion unit 18 for multiplexing the program stream in the same format as the special playback stream, and generates the special playback stream based on the intra-frame coded picture of the normal playback stream. The analysis unit 13 analyzes VBV Delay, Temporal Reference, Picture Coding Type, code amount, and the like of the input intra-frame coded picture, and acquires necessary parameters in the special reproduction processing unit 11. The decoder buffer calculation unit 14 calculates the VBV delay of the special reproduction stream to be generated based on the parameters acquired from the analysis unit 13. The intra-frame coded picture data conversion processing unit 15 is adapted to, for example, the Temporal Reference, VBV Delay, and sequence header to the intra-frame coded picture after the sequence header so as to be suitable for the special reproduction stream generated by the input intra-frame coded picture. Change VBV Buffer Size Value. The repeat picture adding unit 16 encodes the motion vector to 0 and the prediction error to 0 so that the display content is the same as the preceding intra-frame encoded picture, and uses a skipped macroblock to greatly increase the code amount. Generate reduced data. The code amount control unit 17 predicts the generated code amount with respect to the target transfer rate of the special reproduction stream to be generated. If the predicted code amount is smaller than the target transfer rate, the code amount control unit 17 approaches the target transfer rate by stuffing, while predicting If the code amount is likely to exceed the target transfer rate, temporarily increase the target code amount and readjust the related Bit Rate Value and SCR, for example, and perform processing that does not contradict the MPEG standard. To control the amount of generated code. The PS conversion unit 18 divides the special reproduction stream generated in the video elementary stream format into units of packs, and converts them into a program stream format. The distribution unit 8 distributes the normal reproduction stream and the special reproduction stream to the terminal 51 requested to distribute via the network 31 according to, for example, RTP (Real-time Transport Protocol) which is a transmission protocol for streaming reproduction of video. I do.

  Next, the configuration of the terminal will be described. Since the terminals 51 to 53 all have the same configuration, the terminal 51 will be described here. The terminal 51 requests the server 1 connected via the network 31 to store or distribute, and specifies the desired video stream network address, video type, storage time, etc. at the time of storage.

  Also, at the time of distribution, the type of desired video stream, normal playback or special playback, and in the case of special playback, the playback direction and playback speed are specified. The terminal 51 receives the video stream distributed from the server 1 and performs normal reproduction and special reproduction by further decoding. The terminal 51 receives a playback input unit 43 that receives an operation from the user, a playback instruction unit 44 that converts the received user request into an internal command and transmits the command to the server 1 via the network, and receives a video stream from the server 1. The receiving unit 45 includes a decoder 46 that decodes the received video stream, and a display unit 47 that displays the decoded video.

  The reproduction input unit 43 accepts the type of stream that the user desires to reproduce and the reproduction state such as normal reproduction, fast forward, fast rewind, and pause. The playback instruction unit 44 is logically connected to the server 1 via the network 31. The user request received by the reproduction input unit 43 is converted into an internal command by the reproduction instruction unit 44 and transmitted to the connected server 1. The receiving unit 45 receives a stream to be received by the terminal 51 and extracts a necessary video stream from the received RTP packet. The decoder 46 decodes the MPEG-2 compressed stream and displays it on the display unit 47.

Next, the operation in this embodiment will be described. First, a case where data that the user wants to view on the terminal 51 is normally reproduced will be described. In the terminal 51, the type of stream requested by the user and the playback state (normal playback) are input to the playback input unit 43, and the playback instruction unit 44 converts the user's request content into an internal command and sends the server via the network 31. A command is transmitted to one reproduction control unit 6. The reproduction control unit 6 receives a command from the reproduction instruction unit 44, and controls the reading unit 4, the switch 5, and the switch 7 according to the content of the command. In this case, since normal reproduction is instructed, the switch 5 is set to the a side and the switch 7 is set to the c side. The reproduction control unit 6 obtains a storage address from the time stamp to be reproduced and reads out the stream to be reproduced based on a table in which the time stamp held in the address map 23 is associated with the storage address of the storage unit 2. The unit 4 is instructed to read one GOP data recorded at the approximate address. The reading unit 4 reads the GOP data recorded at the designated address from the storage unit 2 and sends the read GOP data to the distribution unit 8 via the switches 5 and 7. The distribution unit 8 converts the normal reproduction stream in the program stream format into an RTP packet, and distributes it to the network 31 in consideration of the network bandwidth, the transfer rate, and the reproduction time stamp. At this time, the RTP packet is transmitted by designating the IP address of the requested terminal 51 so that another terminal does not receive it by mistake.
As described above, when the distribution of one GOP is completed, the reproduction control unit 6 obtains the time stamp and storage address to be distributed next from the address map 23 and reads out the storage address of the GOP to be reproduced next. Give instructions. The read GOP data is distributed by the distribution unit 8 following the previous GOP data. By repeating this operation, the normal reproduction stream is sequentially read out and sent to the network as RTP packetized data.

  Next, the normal playback stream reception operation in the terminal 51 will be described. In the terminal 51, the receiving unit 45 receives only the RTP packet in which the IP address of the terminal 51 is designated as the transmission destination among the RTP packets transmitted from the server 1 via the network 31. An RTP packet header unnecessary for reproduction is removed from the received RTP packet and converted to a program stream. Further, MPEG-2 decoding is performed by the decoder 46 and display is performed by the display unit 47. RTP packets are transmitted one after another from the server 1 regardless of the reception and playback state of the terminal 51. Therefore, the receiving unit 45 receives all RTP packets in which the IP address of the terminal 51 is described as the transmission destination in the terminal 51, and transmits the received data to the decoder 46 one after another. By such an operation, normal reproduction by RTP communication between the server 1 and the terminal 51 is possible.

  Next, the special playback operation will be described. When the user instructs the playback input unit 43 to perform special playback, for example, fast-forwarding, at the terminal 51, the playback instruction unit 44 transmits a corresponding command to the playback control unit 6 of the server 1 via the network 31. When performing the special reproduction, the reproduction control unit 6 sets the switch 5 to the b side and the switch 7 to the d side so that the video stream flows to the special reproduction processing unit 11. When fast-forwarding is performed in the present embodiment, not all GOPs are reproduced, but some GOPs are skipped according to the fast-forwarding speed. The number of skipped GOPs also depends on the number of repeat pictures added to the special playback stream. For example, when two special pictures are used and a special playback stream consisting of 3 pictures per GOP is configured, all GOPs of the normal playback stream are read out at 5 × speed, and every 3 GOPs at 15 × speed. The configuration is such that the corresponding GOP is read out. The reproduction control unit 6 obtains an accumulation address corresponding to the time stamp to be reproduced as fast forward from the correspondence table of the time stamp and the accumulation address held in the address map 23 in accordance with the fast-forward speed, and obtains the accumulation address from the accumulation unit 2. An instruction is sent to the reading unit 4 so as to read.

  The reading unit 4 reads the GOP data of the stream for special reproduction recorded at the storage address from the storage unit 2 and sends it to the extraction unit 12 via the switch 5. The extraction unit 12 demultiplexes the normal playback stream, which is a program stream format, into video elementary, and extracts only the intra-frame coded picture. The extracted intra-frame coded picture is sent to the special reproduction processing unit 11, and processing for generating a special reproduction stream is performed. The special reproduction processing unit 11 first extracts and analyzes parameters such as VBV Delay, Temporal Reference, Picture Coding Type, and code amount from the intra-frame coded picture input in the analysis unit 13, and generates a frame of the special reproduction stream to be generated. Assemble the configuration.

FIG. 2 shows the frame structure of the fast-forward special reproduction stream generated by the special reproduction processing unit 11 of the present embodiment. The upper side of FIG. 2 is a normal playback stream from which a special playback stream is generated, and has a GOP structure with M = 3 and N = 15. A trick play stream lower side is generated from the normal play stream for fast forward, it consists repeat pictures B _R of two frames following the intra-frame coded picture I ₀ and I ₀ of the normal play stream repeats the display contents of the I ₀ ing. Then the next GOP to have continued I ₁₅ and two repeat pictures B _R. Thereafter, a minimum stuffing bit is inserted in S76 in FIG. 7 until the predetermined transmission rate to the terminal is met. As can be seen by comparing the upper side and the lower side in FIG. 2, the number of frames in the GOP is reduced to 3 in the special playback stream compared to the normal playback stream that originally has 15 frames in the GOP. As a result, fast-forward video display at 5 × speed can be performed. Similarly intra-frame coded picture I ₀ and after the two repeat pictures B _R, I ₃₀ and two repeats are intra-frame coded picture of the GOP is not shown further in the following without continuing I ₁₅ It can be realized 10 times speed by continuing picture B _R. In this way, the fast-forward speed can be adjusted by selectively extracting the I frame from the normal reproduction stream in accordance with the fast-forward speed.

On the other hand, a repeat picture is encoded so that the display content of an intra-frame encoded picture is repeatedly displayed. For example, encoding is performed so that a motion vector is 0 and a prediction error is 0. In order to reduce the amount, a skipped macroblock in which encoding of the macroblock is omitted is used. Here, a bidirectional prediction picture is used as a coding scheme for a repeat picture, but a forward prediction picture may be used. By using this repeat picture, as shown in FIG. 2, the special reproduction stream shown in the present embodiment does not use any picture data from B ₁ to B ₁₄ in one GOP in the normal reproduction stream. GOP with the code amount is very small since replaced by repeat pictures B _R and some stuffing of intra-frame coded picture I ₀ constituting the generated trick play stream from two repeat pictures B _R constituted The transfer rate can also be reduced. For this reason, the transfer rate becomes very large in transfer of only I frames performed as general special playback, whereas in the special playback stream by the special playback method shown in this embodiment, an increase in transfer rate can be prevented. it can. For example, when 15-times special playback is performed by transferring only I frames for a normal playback stream having a transfer rate of 6 Mbps and 1 GOP of 15 frames, a bandwidth of 18 Mbps is generally required. However, as shown in the present embodiment, in special playback using two repeat pictures, when all GOPs are played back, 5 Mbps playback speed is 6 Mbps, and if playback is performed every 3 GOPs, 15 Gbps speed is also in the 6 Mbps band. Special playback can be performed. When playback is performed on the special playback stream having this configuration, the same video is played back for three frames, but there is no visual problem at all. The picture type is a bidirectional prediction picture in the present embodiment, but may of course be a forward prediction picture.

The operation of changing the I-frame reading interval according to the load amount, which is the gist of the present invention, will be described.
FIG. 3 is a diagram for explaining this, and the analysis unit 13 performs the processing shown in FIG. 4 on the normal reproduction stream of FIG. In a system in which the server 1 assembles and transmits a special reproduction frame based on a request from a terminal, if read requests from a plurality of terminals overlap, the read load becomes heavy and the requested speed cannot be followed. In step S61 in FIG. 4, the analysis unit 13 knows this load amount, sets an appropriate I-frame transmission interval in step S62, and transmits it to the subsequent special reproduction processing unit 11. Furthermore, VBV Delay and code amount are analyzed. When the reading load of the storage unit 2 is light, that is, when the number of read streams is small, the I frame in the normal reproduction stream is set for every three GOPs as shown in “low load fast forward stream” in FIG. the read, add two repeat pictures B _R, delivers a smooth trick play stream. However, when the reading load of the storage unit 2 is heavy, that is, when there are a plurality of read requests and distribution is performed, the interval set in S62 is thinned out, and is indicated by “fast load stream at high load” in FIG. An I picture is read every 10 GOPs. While maintaining the transmission rate distributed in this way, the load can be reduced and the operation of the calling unit 4 can be maintained.
In this case, the case of fast-forward playback has been described, but this function can be similarly applied to other special playback.

  The decoder buffer calculation unit 14 calculates the VBV delay of the special reproduction stream based on the VBV delay obtained by the analysis unit 13 and the code amount of the normal reproduction stream. Normally, VBV Delay indicates transition of picture data in the VBV buffer of a continuous stream, and VBV Delay must be a value that does not cause overflow and underflow in the VBV buffer. Therefore, when determining VBV Delay, it is necessary to control the exact value by simulating the transition of VBV. In the present embodiment, it is assumed that VBV Delay is derived in the decoder buffer calculation unit 14. In the intra-frame coded picture data conversion processing unit 15, the bit rate value from the temporal reference, VBV delay, and sequence header to the intra-frame coded picture so as to become an intra-frame coded picture corresponding to the generated special reproduction stream. Or VBV Buffer Size Value is changed. For example, when the Temporal Reference of the intra-frame coded picture is 2 in the normal playback stream and one repeat picture is added in the special playback stream, the Temporal Reference is changed to 1. The VBV Delay is also replaced with the VBV Delay for special playback stream calculated by the decoder buffer calculation unit 14. The repeat picture adding unit 16 generates repeat picture data that displays the same display content as the intra-frame coded picture, and performs coding after the intra-frame coded picture. Further, the code amount control unit 17 performs stuffing if the predicted code amount is smaller than the target transfer rate, and if it exceeds the target transfer rate, temporarily increases the target transfer rate to generate the special reproduction stream to be generated. Make adjustments so that data can be transferred. Finally, the special reproduction stream generated by the PS unit 18 is multiplexed into the program stream. The special reproduction stream generated by the special reproduction processing unit 11 as described above is sent to the distribution unit 8 via the switch 7. The distribution unit 8 performs RTP packetization using the terminal that has made the request as the transmission destination IP address in the same manner as the normal reproduction stream, and performs distribution. By repeating this operation for each GOP used as the special reproduction stream, the special reproduction stream can be distributed.

  Next, the reception operation of the special reproduction stream in the terminal 51 will be described. In the terminal 51, only the RTP packet in which the IP address of the terminal 51 is specified as the transmission destination among the RTP packets transmitted from the server 1 via the network 31 by the reception unit 45 as in the case of receiving the normal reproduction stream. Receive. An RTP packet header unnecessary for reproduction is removed from the received RTP packet and converted to a program stream. The special playback stream returned to the program stream is a fast-forward video in which the content of the video stream is special playback, but there is no difference in syntax from the normal program stream. Therefore, similarly to the normal reproduction, the decoder 46 can decode the MPEG-2 and display the reproduced image on the display unit 47.

FIG. 5 shows a special reproduction stream in the case of pause. The configuration of the fast-forward reproduction shown in FIG. 2, take the same construction except for using always the same I ₀ is I frame used. In pause, using intra-frame coded picture I ₀ of the GOP to be displayed as a pause in the normal play stream is continued two repeat pictures B _R Following I _0. Further repeated two continue configuring the I ₀ and repeat pictures B _R repeatedly to configure the same picture. With this configuration, it is possible to configure a special reproduction stream for pause that shows the same video continuously. Since in the case of pause can I view continuously same image, by increasing the number of repeat pictures B _R, it is also possible to lower the load on the server and network.

  Next, an operation for performing a temporary stop will be described with reference to FIG. In FIG. 1, the reproduction control unit 6 instructs the reading unit 4 to always read out the storage address of the same time stamp data from the storage unit 2 while referring to the address map 23. The read GOP data is converted into a special reproduction stream as shown in FIG. 3 by the special reproduction processing unit 11, and distributed from the distribution unit 8 to the terminal 51. The generated special playback stream for pause has a data structure in which the same playback image is continuously displayed, and the terminal 51 temporarily stops decoding and playback processing in the same manner as in normal playback or fast-forwarding. A stop playback image can be displayed.

FIG. 6 shows a special reproduction stream in the case of fast rewinding. (A) on the upper side of the figure is a normal reproduction stream from which a special reproduction stream is generated, and has a GOP structure of M = 3 and N = 15. The lower side is a fast rewind special reproduction stream generated from the normal reproduction stream. In fast rewinding, it is necessary to be configured so that a video later in time is displayed first. Temporally intra-frame coded picture I ₁₅ after the order fast rewind trick play stream normal play stream earlier, followed two repeat pictures B _R continues, further followed inherently temporally previously displayed the intra-frame coded picture I ₀ and repeat pictures B _R to be taken two continue configuration. With this configuration, it is possible to generate a special playback stream for fast rewind playback in which the order of the displayed video is opposite to that of the normal playback stream. As can be seen from the comparison between the upper side and the lower side in FIG. 4, the normal playback stream originally having 15 frames in the GOP is reduced to 3 in the special playback stream, and the result is as follows. As a result, it is possible to display a fast-rewind video display at 5 × speed. Further, if the intra-frame coded picture used for the special reproduction stream to be created is set every 2 GOPs, fast rewinding at 10 times speed can be achieved if every 3 GOPs.

  Next, the operation for fast rewinding will be described with reference to FIG. In FIG. 1, the playback control unit 6 determines the time stamp of the normal playback stream to be read in accordance with the fast rewind speed. Further, the storage address of the time stamp data to be read is determined with reference to the address map 23, and the corresponding GOP data is read from the storage unit 2. The read GOP data is converted by the special reproduction processing unit 11 into a special rewind stream for fast rewinding. Similarly, the next read time stamp is determined in accordance with the fast rewind speed, the storage address is obtained from the address map 23, and the corresponding GOP data is read from the storage unit 2 and sent to the special playback processing unit 11 to continue the operation shown in FIG. A special playback stream as shown in FIG. The generated fast-rewind special reproduction stream is distributed from the distribution unit 8 to the terminal 51. The generated fast-rewind special playback stream has a data structure in which the playback image is reversed in time, and the terminal 51 performs fast-rewinding simply by continuing sequential decoding and playback processing as in normal playback. Can be displayed.

  As described above, according to the special reproduction apparatus shown in the present embodiment, the special reproduction stream is generated in real time, so that it is necessary to prepare the special reproduction stream on the recording medium in advance or the special reproduction stream prepared in advance. There is no need for special management. The special playback stream to be generated can be transferred without increasing the load on the network because a repeat picture with a small code amount is added to the I frame extracted from the normal playback stream to reduce the code amount per second. is there. Also, since the generated special playback stream is generated at the playback side at a normal speed so as to present the video as if it had been specially played back, a special playback stream is played back on the playback side. There is no need to set up a mechanism. Furthermore, since the encoding parameter value of the I frame that is extracted from the normal playback stream and used for the special generation stream is reset so as to comply with the rules of the special playback stream and MPEG-2 stream to be generated, the encoding parameter is invalid. The disturbance of the reproduced video due to the value and the failure of multiplexing in the multiplexing device are eliminated. Also, since the value is reset for the VBV Delay that controls the decoder buffer, it is possible to prevent the decoder buffer from failing. Further, since VBV Delay is an appropriate value, no multiplex error occurs even if there is a multiplex device on the route to which the special playback stream is distributed. Furthermore, since the adjustment of the transfer rate can be controlled for the special reproduction stream to be generated, it is possible to make the transfer rate the same as the normal reproduction or to change the transfer rate only during the special reproduction. As a result, there is no video disturbance during special playback. Further, since the GOP or I frame address is recorded at the time of accumulation, a special reproduction stream can be generated at high speed. Further, since the amount of I-frame reading is adjusted according to the load on the storage device or the transmission device, the special reproduction stream can be generated reliably even when the load increases.

In the above description, the operation has been described on the assumption that the server 1 is composed of hardware elements. In particular, the special reproduction processing unit 11 is equivalent to the hardware stream memory and the picture extraction unit 12 to the PS conversion unit 18. You may comprise with software with a function. That is, FIG. 7 shows a flow in which the function performed by the special reproduction processing unit 11 is configured by a program.
The special reproduction processing unit by the program that has received the special reproduction transmission instruction performs an operation corresponding to the analysis unit 13 in steps S71 and S72. That is, load analysis is performed to calculate the I frame transmission interval, and further analysis of VBV Delay, Temporal Reference, code amount, and the like is performed. In S73, an operation corresponding to the decoder buffer calculation unit 14 is performed, and in S74, an operation corresponding to the hardware intra-frame coded picture data conversion processing unit 15 is performed. Necessary I pictures are stored in a hardware stream memory (not shown), and in S75, necessary repeat pictures are added following the I pictures in accordance with the type of special reproduction. In S76, necessary stuffing bits are inserted in consideration of the transmission rate to the terminal, program stream conversion is performed, and the completed special reproduction stream is sent to the distribution unit 8 via the switch 7.

Embodiment 2. FIG.
FIG. 8 shows an application example in a wide area monitoring system. In the present embodiment, a system is shown in which videos captured by the cameras 101 to 103 are accumulated and stored in the server 120, and videos accumulated from a plurality of terminals 111 to 113 can be viewed. In addition to being able to view any video at normal playback speed, the terminal can also perform special playback such as fast forward, fast rewind, and pause.

The system is configured as follows. The monitoring cameras 101 to 103 compress the captured video into an MPEG-2 program stream, further perform RTP / UDP packetization corresponding to the RTP protocol, and transmit the compressed video to the server 120 via the network 100. The server 120 has three storage methods for the video streams photographed by the cameras 101 to 103, and can be selected by initial setting. The first is a primary storage server function that always receives and stores multicast streams from cameras 101 to 103 endlessly, and the second is an alarm storage server that receives and stores multicast streams from cameras 101 to 103 triggered by an alarm. This function is an image storage server function for saving a video stream stored in another server, and these three functions can be switched and used by default. Here, the description will be continued assuming that the server 120 is set to the primary storage server function. The server 120 is provided with a request reception module 121 that responds to external storage and distribution requests. The request reception module 121 receives video distribution and storage requests from the terminals 111 to 113 and another server not shown in the figure, and uses an internal I / F to other modules in the server 120. Take control. It also sends a response message to the requesting terminal or server.
The request reception module 121 corresponds to RTSP (Real Time Streaming Protocol), and can perform normal reproduction and special reproduction of the video stream stored in the server 120. The request reception module 121 also has various search functions using the video database 123. For example, the request reception module 121 can search for a target video stream using a time or a photographed camera as a search key.

  The reception module 122 performs reception processing of a stream distributed by multicast from the cameras 101 to 103. The advantages that the cameras 101 to 103 transmit by multicast are that one video stream can be stored simultaneously by another server and that the video of the camera can be viewed directly from the terminals 111 to 113. If you want to send only for, you may send by unicast. The reception module 122 performs reception only when the transmission IP address of the RTP / UDP packet is the IP address or multicast address of the server 120 and the port numbers match. Also, the time when the leading data of the GOP sent by the RTP / UDP packet is received by the receiving module 122 is acquired as a time stamp. Since the received RTP / UDP packet has the RTP / UDP header added, it is removed, and the payload part of the RTP / UDP packet is combined into a program stream. The storage module 125 stores the received video stream, which is a program stream, in the HDD 127 in GOP units, and manages the data stored in the HDD 127 in GOP units. The video database 123 provides a recording / management and search function for the video stream managed by the storage module 125 and the meta information of the GOP. Examples of the type of meta information of the video stream include a camera number, a video compression method and compression level, a time stamp indicating the time acquired by the receiving module 122, and a shooting time. The distribution module 124 distributes the video stream managed by the storage module 125 to the network 100 in accordance with a request from the request reception module 121. When a video stream is distributed as a normal playback stream to the terminal 111 via the network 100, for example, the program stream as transmission data is converted into an RTP / UDP packet, and the IP address of the terminal 111 is set as a transmission destination IP address. The transmission timing for each GOP is controlled based on the reception time stamp for each GOP recorded in the video database 123.

  On the other hand, when the special reproduction stream is distributed, the normal reproduction stream read from the storage module 125 is once transmitted to the special reproduction module 126, and the special reproduction stream generated by the special reproduction module 126 is distributed. As for the delivery timing of the special reproduction stream, the delivery is performed so that the reception time stamp interval of the GOP to be delivered is 1/15 if it is 15 times speed for fast forward and fast rewind. In order to use the special reproduction module 126, an intra-frame encoded picture (I frame) is extracted from the normal reproduction stream and passed to the special reproduction module 126, and parameters necessary for generating the special reproduction stream, for example, Parameters such as the transfer rate of the normal playback stream and the transfer rate of the desired special playback stream are also passed to the special playback module 126 to generate the special playback stream. The terminals 111 to 113 can view the video stream stored in the server 120, and can be viewed by sending a reproduction request to the request reception module 121 of the server 120. When the terminal 111 requests the server 120 for a video stream desired to be viewed, the corresponding video stream is distributed from the server 120 in units of RTP / UDP packets. The terminal 111 can reproduce the requested video stream by receiving the packet having the same transmission IP address and port number in the RTP / UDP packet on the network.

  Next, the special playback module 126 will be described with reference to FIG. The special playback module 126 includes an I frame buffer 131 for receiving an I frame transferred from the distribution module 124, an analysis unit 132 for analyzing the data structure and encoding parameters of the received I frame data, and a special playback stream to be generated. Decoder buffer calculation unit 133 that calculates the VBV delay of the I frame data, an I frame data conversion processing unit 134 that changes the parameter for the I frame data used for the special playback stream, a repeat indicating the same display contents as the I frame following the I frame Repeat picture adding unit 135 for generating / adding pictures, code amount control unit 136 for controlling the code amount of the generated special playback stream to match the target transfer rate, and converting the generated special playback stream into a program stream PS conversion unit 137, and Composed of trick play stream buffer 138 for transmitting trick play streams form the distribution module 124.

  Next, the operation of the present embodiment will be described. Assume that the server 120 is set as a primary storage server that always receives and stores multicast streams from the cameras 101 to 103 endlessly as a storage method. When the primary storage server is set, the receiving module 122 always monitors the RTP / UDP packet of the network 100, the destination IP address of the RTP / UDP packet is its own IP address or multicast address, and the port number is If they match, the corresponding RTP / UDP packet is received. The received video stream in packet units is converted into a program stream, and then recorded in the HDD 127 by the storage module 125 in GOP units. In addition, the video database 123 records meta information such as the time stamp when the GOP is received, camera information, video compression method and compression level. By repeating this operation for each packet, the video stream is accumulated in the server 120.

Next, a reproduction method will be described. First, for example, a case where an arbitrary video is selected from the terminal 111 and reproduced will be described. First, the terminal 111 acquires a list of video streams stored in the server 120 from the request reception module 121 of the server 120. In order to request a list of video streams, the search function of the video database 123 is used, and if a search is executed without specifying anything as a search key, a list of accumulated video streams can be acquired. The terminal 111 can select any desired video stream from the list of video streams acquired in this way. A video ID that does not overlap is assigned to each video stream, and the video ID is used to specify the video stream. Next, in order to play back the selected video stream, a playback request command for the selected video ID is issued to the request receiving module 121. Upon receiving the reproduction request, the request reception module 121 sends a command to the distribution module 124 to start reproduction of the designated video ID. The distribution module 124 instructs the storage module 125 to acquire a video stream that matches the video ID and matches the playback time. The storage module 125 reads the GOP of the video stream that matches the corresponding video ID and playback time from the HDD 127 and sends it to the distribution module 124. The distribution module 124 distributes the GOP acquired from the storage module 125. Distribution by the distribution module 124 is performed by converting the GOP of the video stream into an RTP / UDP packet, specifying the IP address of the terminal 111 and the same port number as the receiving terminal as the destination IP address, and further recording in the video database 123 The distribution timing is controlled based on the reception time stamp which is one of the meta information of the data. That is, the distribution module 124 distributes the GOP so as to reproduce the reception time stamp that is the time at which the reception module 122 receives the GOP as the distribution timing of the GOP. In other words, as normal image distribution starting from a specified time stamp, the distribution timing from the camera of video data that has been captured and encoded by the camera is reproduced, and overflow and underflow in various buffers in the network and playback environment. Real-time decoding can be performed without generating a flow. The RTP / UDP packet distributed from the distribution module 125 is received by the terminal 111 that is the transmission destination. The terminal 111 can obtain a normal reproduction stream in the form of a program stream by removing the RTP / UDP header from continuously delivered RTP / UDP packets and connecting the payloads of a plurality of RTP / UDP packets.
From the above, the terminal 111 can reproduce the normal reproduction stream acquired by the internal decoder (not shown).

Next, special reproduction will be described. Here, a fast-forward method, which is one of typical special reproductions, will be described. For simplicity of explanation, it is assumed that the normal reproduction data stored is an MPEG-2 program stream and N = 15. In the case of fast-forwarding, if only I frames in the GOP are reproduced sequentially, the speed is 15 times faster. However, assuming that two repeat pictures are used, the speed is 5 times faster. In order to further increase the speed of fast-forwarding, GOPs may be read discretely. For example, 15-times fast reading is possible every 3 GOPs, and 30-times fast forwarding is possible every 6 GOPs.
Now, it is assumed that 15 × fast-forward special playback is requested from an arbitrary video from the terminal 111. As in the case of normal playback, the terminal 111 issues a video ID for performing special playback and a command for performing special playback to the request receiving module 121 of the server 120. Upon receiving the reproduction request, the request reception module 121 issues a selected video ID and a fast-forward command to the distribution module 124. The distribution module 124 acquires the meta information of the stream of the selected video ID using the video database 123, and determines the GOP to be read for realizing the 15-times fast-forwarding.

  FIG. 10 shows the relationship between the GOP and the reception time stamp when the load is low and when the load is high. In FIG. 10, when the system load is low, in order to realize 15 times speed when N = 15, it is sufficient to read out every 3 GOPs with two repeat pictures. Further, when the system load is high, nine repeat pictures may be used and readout may be performed every 10 GOPs. In this way, the read GOP is determined according to the fast-forward speed and the system load, and the storage module 125 reads the determined GOP from the HDD 127. The distribution module 124 extracts the I frame from the GOP data of the acquired normal reproduction stream and sends it to the special reproduction module 126. In FIG. 9, I frame data sent from the distribution module 124 is stored in the I frame buffer 131 of the special playback module. The analysis unit 132 analyzes VBV Delay, Temporal Reference, Picture Coding Type, and code amount of the I frame data stored in the I frame buffer 131 and acquires parameters necessary for each process in the special reproduction module 126. The decoder buffer calculation unit 133 calculates the VBV delay of the special reproduction stream to be generated based on the parameters acquired from the analysis unit 132. As a calculation method of VBV Delay, the method described in the first embodiment is used. The I frame data conversion processing unit 134 changes, for example, the Temporal Reference, VBV Delay, and the Bit Rate Value and VBV Buffer Size Value from the sequence header to the I frame so as to conform to the special playback stream in which the input I frame picture is generated. To do. The repeat picture adding unit 135 greatly reduces the amount of data by using the skipped macroblock so that the motion vector is 0 and the prediction error is 0 so that the display content is the same as the preceding I frame. Create data encoded in. The code amount control unit 136 predicts the generated code amount with respect to the target transfer rate of the special reproduction stream to be generated. If the predicted code amount is smaller than the target transfer rate, the code amount control unit 136 approaches the target transfer rate by stuffing and If the amount of encoding is likely to exceed the target transfer rate, temporarily increase the target transfer rate and readjust the related Bit Rate Value and SCR, for example, and perform processing that does not contradict the MPEG standard. Thus, the generated code amount is controlled. The PS conversion unit 137 divides the special playback stream generated in the video elementary stream format into pack units, and converts it into the program stream format. The generated program stream is stored in the special playback stream buffer, and the data is passed to the distribution module 124.

The distribution module 124 converts the received special playback stream into an RTP / UDP packet in the same manner as the normal playback stream, specifies the IP address and port number of the terminal 111 as the transmission destination IP address, and further stores the distribution recorded in the video database 123. Considering the fast-forwarding speed based on the reception time stamp for each GOP, which is one of the meta information of data, that is, if it is 15 times speed, the interval of the reception time stamp between GOPs is 1/15. Control delivery timing. The RTP / UDP packet distributed from the distribution module 124 is received by the terminal 111 that is the transmission destination. The terminal 111 can acquire the special reproduction stream by removing the RTP / UDP header from the continuously distributed RTP / UDP packets to form a program stream packet and connecting a plurality of payload packets. By playing the acquired special playback stream in the same manner as the normal playback stream, it is possible to realize 15 × fast-forward special playback. Since this special reproduction stream is not different from the normal reproduction stream in terms of syntax, it can be handled in the same manner as the normal reproduction stream.
As described above, the terminal 111 can reproduce the acquired special reproduction stream.

  In this example, the fast-forward 15 times speed is illustrated, but if the fast-forward 30 times speed, one I frame is read every 6 GOPs at low load and every 20 GOPs at high load, and the time interval of the time table to be distributed is 1/15 If it is fast-rewinding, it may be read in the reverse direction and sent to the special playback module 126. If it is paused, the same GOP data may be repeatedly sent to the special playback module 126.

  Regarding the delivery timing of fast-forward and fast-rewind special playback streams, real-time performance could be secured by delivering based on the time stamp at the time of reception, but the timing of delivery in pause special playback is the same as the reception time Therefore, it is meaningless to use the reception time table. Therefore, a method of distributing based on the time information of the generated special reproduction stream is adopted.

  Now, the special playback stream shown in the present embodiment is generated at the playback side at a normal speed so as to present the video as if it had been specially played back. It is not necessary to provide a special mechanism for reproduction, and it can be handled in the same manner as a normal reproduction stream. However, in order to continue playback seamlessly when transitioning from normal playback to special playback or from special playback to normal playback, the SCR (System Clock Reference), PTS (Presentation Time Stamp) and DTS (Decoding Time Stamp) of the program stream ) To ensure the continuity of time information. If these pieces of time information are not continuous, depending on the decoder, the decoder buffer may overflow or underflow, or the decoding may be interrupted or stopped. Therefore, when switching between normal reproduction and special reproduction, it is desirable to offset the SCR of the subsequent stream so that the SCR of the subsequent stream is continuous based on the final SCR of the preceding stream. Furthermore, it is also necessary to perform the offset equivalent to the SCR for the subsequent stream PTS and DTS.

  As described above, according to the special reproduction method shown in the present embodiment, the special reproduction stream is generated in real time. Therefore, it is necessary to prepare the special reproduction stream on the recording medium in advance, or the special reproduction stream prepared in advance. There is no need for special management. The special playback stream to be generated can be transferred without increasing the load on the network because a repeat picture with a small code amount is added to the I frame extracted from the normal playback stream to reduce the code amount per second. is there. Also, since the generated special playback stream is generated at the playback side at a normal speed so as to present the video as if it had been specially played back, a special playback stream is played back on the playback side. There is no need to set up a mechanism. Furthermore, since the encoding parameter value of the I frame that is extracted from the normal reproduction stream and used for the special generation stream is reset so as to comply with the rules of the special reproduction stream and MPEG-2 stream to be generated, the encoding parameter is invalid. The disturbance of the reproduced video due to the value and the failure of multiplexing in the multiplexing device are eliminated. Further, since the transfer rate adjustment can be controlled for the special reproduction stream to be generated, the transfer rate can be set to the same as the normal reproduction, or the transfer rate can be changed only during the special reproduction. Further, when the special reproduction stream is generated, the reading load from the storage unit can be adjusted dynamically or statically according to the load of the system, so that the stream can be reliably generated while suppressing the load. As a result, there is no video disturbance during special playback.

  As described above, according to the present invention, even when the load on the storage device increases, the interval for reading intra-frame coded pictures from the normal playback stream can be changed, so that a special playback stream can be generated reliably. Can do.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the internal structure of the system and each apparatus of the server, network, and terminal in Embodiment 1 of this invention. 6 is a diagram for explaining a stream structure for fast-forward playback in Embodiment 1. FIG. 6 is a diagram illustrating a stream structure for fast-forward playback at low load and high load in Embodiment 1. FIG. FIG. 6 is a flowchart showing an operation performed by an analysis unit in the first embodiment. 6 is a diagram illustrating a stream structure of pause reproduction in Embodiment 1. FIG. 6 is a diagram for explaining a stream structure of fast-rewind playback in Embodiment 1. FIG. FIG. 10 is a flowchart showing an operation performed by another special reproduction processing unit in the first embodiment. It is a figure which shows the wide area monitoring system in Embodiment 2, and the detailed structure of a server. 6 is a diagram illustrating a configuration of a special reproduction module according to Embodiment 2. FIG. It is a figure explaining the GOP time stamp at the time of low load and high load in Embodiment 2.

Explanation of symbols

  1 server (device), 2 storage unit, 3 writing unit, 4 reading unit, 5 switch, 6 playback control unit, 7 switch, 8 distribution unit, 11 special playback processing unit, 12 I picture extraction unit, 13 analysis unit, 14 Decoder buffer calculation unit, 15 intra-frame coded picture data conversion processing unit, 16 repeat picture addition unit, 17 code amount control unit, 18 PS conversion unit, 21 reception unit, 22 GOP extraction unit, 23 address map, 31 network, 41 Accumulation input unit, 42 Accumulation instruction unit, 43 Reproduction input unit, 44 Reproduction instruction unit, 45 Receiving unit, 46 Decoder, 47 Display unit, 51 terminal, 52 terminal, 53 terminal, 100 network, 101 camera 1, 102 camera 2, 103 Camera 3, 111 Terminal 1, 112 Terminal 2, 113 Terminal 3, 120 Server, 121 request reception module, 122 reception module, 123 video database, 124 distribution module, 125 storage module, 126 special playback module, 127 HDD, 131 I frame buffer, 132 analysis unit, 133 decoder buffer calculation unit, 134 I frame data conversion process , 135 repeat picture addition unit, 136 code amount control unit, 137 PS conversion unit, 138 special playback stream buffer.

Claims (5)

In response to a request from the terminal, the storage movies generates Zosu stream or al special playback video stream, generated trick play video streams you send to the terminal a special video data encoded using inter-frame prediction In the processing device,
A special playback processing unit that inputs the stored video stream and generates a special playback video stream is provided.
The special reproduction processing unit
The transmission interval of the intra-frame encoded picture is determined according to the load for reading the stored video stream, and the intra- frame encoded picture is extracted from the stored video stream based on the predetermined transmission interval and included in a part of the special playback video stream In addition, an analysis unit that performs picture analysis of the extracted intra-frame encoded picture ;
Based on the analysis result of the analysis unit, the intra-frame coded picture conversion processing unit for changing a coding parameter of the intra-frame coded picture in the coding parameter corresponding to the trick play video streams,
A repeat picture showing the same display as the intra-frame encoded picture extracted by the analysis unit is generated, and the generated repeat picture is added to the intra-frame encoded picture to make the repeat picture a part of the special playback video stream. Repeat picture addition part to include
Special video data processing apparatus characterized by comprising a. Wherein the special reproduction processing unit further to the stuffing bits to satisfy the transmission interval and the predetermined amount of transmission intra-frame coded picture with a code amount control unit to be inserted into the trick play video streams The special video data processing apparatus according to claim 1. A special video data processing apparatus according to claim 1;
Sending the request to the specialized video data processing apparatus, a special, characterized in that a structure having a <br/> with one or more terminals to receive the special playback video stream from the special video data processing apparatus Video data processing system. Special video data that includes a special playback processing unit that generates a special playback video stream from an accumulated video stream encoded using inter-frame prediction in response to a request from the terminal, and transmits the generated special playback video stream to the terminal In the video data processing method of the processing device ,
The analysis unit of the special reproduction processing unit determines the transmission interval of the intra-frame encoded picture according to the load for reading the stored video stream, and extracts the intra-frame encoded picture from the stored video stream based on the determined transmission interval. An analysis step that includes a part of the specially reproduced video stream and analyzes the extracted intra-frame encoded picture ;
The intra-frame coded picture conversion processing unit of the special reproduction processing unit encodes the coding parameter of the intra-frame coded picture corresponding to the special reproduction video stream based on the analysis result of the analysis unit in the analysis step. An intra-frame coded picture data conversion step to change to a parameter ;
The repeat picture adding unit of the special reproduction processing unit generates a repeat picture showing the same display as the intra-frame encoded picture extracted by the analysis unit in the analysis step, and the generated repeat picture is converted into an intra-frame encoded picture. special video data processing method of the repeat pictures, characterized by comprising <br/> and repeat pictures added steps included as part of the trick play video streams by adding. After said repeat pictures adding step, the code amount control unit of the special reproduction processing unit, inserts stuffing bits in order to satisfy the transmission interval and the predetermined amount of transmission intra-frame coded picture in the trick play video streams 5. The special video data processing method according to claim 4, further comprising a stuffing bit insertion step.

Images