JP2007324722A - Moving picture data distribution apparatus and moving picture data communication system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To attain distribution, reception and reproduction of live video and audio, and reuse of received data as contents even when the specifications defined by the HTTP/1.1 are not mounted in a transmitter side, or in a receiver side, or neither of the transmitter and receiver sides. <P>SOLUTION: A moving picture distribution apparatus is disclosed which calculates a size (Content-length) of distribution data on the basis of request information from a client (S503), notifies a reproduction apparatus about the size to generate blocks each comprising a set of metadata and coded data required for reproduction processing (S505, S509), and sequentially transmits the blocks. The moving picture distribution apparatus transmits negligible data in place of the coded data so that the total transmission size is equal to the Content-length. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a moving image data distribution apparatus and a moving image data communication system, for example, a technique suitable for use in data transmission / reception.

  In recent years, so-called surveillance cameras for the purpose of monitoring crimes and confirming remote locations are becoming popular. A format called RTP (A Transport Protocol for Real-Time Applications) is generally used to realize the distribution of moving image video and audio (hereinafter referred to as live video distribution) subjected to such real-time encoding. .

  However, in live video distribution using RTP, there are problems that a firewall is set from the viewpoint of network security, and that data packets for storing live video are likely to be lost. For this reason, live video distribution using a file transfer format based on a simpler protocol such as HTTP (Hyper Text Transfer Protocol) has been studied.

  Conventionally, as a file recording format suitable for performing such real-time processing, there is a fragmented video format (Fragmented Movie) in the ISO Base Media File format. In the fragment video format, metadata related to the entire video data is described at the beginning of the file, and after that, a portion of the actual video data for a certain period of time is recorded, and then the metadata of the video data that follows is recorded. A combination of moving image data is continuously recorded. In other words, it is possible to handle a certain combination of moving image data and its metadata as one data block. By recording in such a format, real-time encoded moving image data can be continuously recorded with a smaller load.

  In addition, a protocol suitable for file transfer such as HTTP needs to notify the receiving side in advance of the size of data to be transferred because of its nature, and the mechanism corresponding to this has been expanded.

  In live video distribution, the end cannot be known before distribution starts. For this reason, for example, in HTTP, the transmission data is divided into small blocks, and the transfer size is notified from the distribution side to the reception side for each block, so that the entire size is not required before distribution starts. Is defined.

  In addition, it has been proposed that a transmission request size is notified from the reception side to the transmission side, thereby realizing a similar mechanism (see, for example, Patent Document 1). In this method, division reception is performed by sequentially acquiring a part of data from the reception side while maintaining communication connection using the Range function defined in HTTP.

JP 2005-27010 A

  However, for example, when trying to distribute a live video in which the overall distribution file size is not fixed using a method such as the above-mentioned Chunked Transfer Encoding, the specifications defined in HTTP / 1.1 are both the transmission side and the reception side. Must be implemented. When either of them implements HTTP / 1.0, which is the previous specification of HTTP, there is a problem that this cannot be used. In the case of a device with restrictions on CPU resources, memory, etc., there are many cases where it is necessary to implement HTTP / 1.0, and in such a case, it cannot be realized.

  On the other hand, since the delivery file size cannot be determined, delivery can be performed without notifying the file size, and communication can be terminated by disconnecting the communication connection. In this case, such communication disconnection is possible. Is not preferred because it is exceptional. In addition, because of the disconnection in the state where the transfer is not completed, especially from the protocol viewpoint, since the received data is not complete, the received file becomes incomplete, There is also a problem that reuse becomes difficult.

  In addition, when the Range function defined in HTTP is used, this function is primarily intended to acquire a file for each convenient data size at the judgment of the receiving side. For this reason, there is a possibility that the problem of the disconnection of the communication connection may occur, and the processing becomes difficult unless both the transmission side and the reception side have mutually understood such a mechanism. There was a point.

  In view of the above-described problems, the present invention can perform live video and audio distribution and reception / reproduction even if the specifications defined in HTTP / 1.1 are not implemented at least on the transmission side or reception side. And it aims at making received data reusable as contents.

  The present invention has been made in view of the above problems, and includes at least one of encoded data of moving images and management information of the encoded data according to the generation order of moving image data including at least one of video and audio. Moving image data generating means for generating the data block, moving image data size estimating means for estimating the size of the moving image data in advance, and moving image data estimated by the moving image data size estimating means First communication data generating means for generating the size information and the first data block in a format suitable for the communication procedure, and the moving image data generated by the moving image data generating means are sequentially suitable for the communication procedure. Sequentially generated by the second communication data generating means for generating the format, the first communication data generating means and the second communication data generating means. Ended from communication data integration means for integrating the size of the communication data received, the size information of the moving image data estimated by the moving image data size estimation means, and the size information of the communication data integrated by the communication data integration means When it is determined that the end is determined by the end determination unit, the size information of the moving image data estimated by the moving image data size estimation unit is integrated by the moving image data size estimation unit. Third communication data generating means for generating ignored data having a size equal to the difference in size information of communication data, and communication means for sequentially transmitting the communication data generated by the third communication data generating means. A moving image data distribution device and the like characterized by the above are provided.

  According to the present invention, it is possible to transmit and receive moving image data in which live video and audio are encoded. On the receiving side, live image and audio can be reproduced, and the received data can be reused. Can be accumulated.

(First embodiment)
Hereinafter, a first embodiment when applied to a network camera having a communication function of the present invention will be described with reference to the drawings.
FIG. 1 is a conceptual diagram showing a system configuration example suitable for implementing this embodiment.
As shown in FIG. 1, this system includes a network camera 101 that generates a moving image and distributes it on a network, and a playback device 102 that receives and displays the distributed moving image.

  The network camera 101 is an imaging device having a communication function, and has recently become widespread as a camera used for monitoring purposes. On the other hand, the playback device 102 is a device that plays back a moving image having a communication function corresponding to the network camera 101. For example, the playback device 102 may be realized as an application program that operates on a PC (personal computer), or may be realized as a part of a monitoring device including a dedicated display.

  In terms of conceptual functions, the network camera 101 includes a moving image generation module 103 that captures a moving image and generates encoded data, and a communication data generation module that converts the encoded moving image into data suitable for communication. 104 and a communication server 105 that actually transmits a moving image in response to a request.

  In terms of conceptual functions, the playback device 102 requests a moving image from the network camera 101, obtains a moving image, and decodes the encoded data of the received moving image to visualize (rendering, etc.). And a moving image display module 106.

Next, a general processing flow will be described.
First, the user operates the playback device 102 to instruct to acquire an image from the network camera 101. Such an operation may be an operation of inputting a URL (Uniform Resource Locator) of the network camera 101 using, for example, a GUI (Graphical User Interface). Further, the operation may be an operation of pressing a switch using information that is defined in advance for communication connection in the playback apparatus 102.

  The image acquisition instruction is notified from the communication client 106 to the communication server 105 via a network by a communication procedure (protocol) such as HTTP (Hyper Text Transfer Protocol).

  In the present embodiment, specifically, the communication procedure used between the communication client 106 and the communication server 105 is HTTP, and the communication server 105 is an HTTP server itself called HTTPd. The message used for the request is performed by designating a CGI (Common Gateway Interface) using a URL, for example.

  In this case, CGI is a program for acquiring a moving image. When the moving image is acquired by the URL, the communication server 105 calls the corresponding resource to perform the transmission process, and the process is the process of the communication data generation module 104. As described above, in the case of a CGI program, the processing itself of the communication data generation module 104 is configured by CGI.

  Here, the communication data generation module 104 acquires the moving image generated by the moving image generation module 103 and shapes it into a format that is optimal for transmission to the playback apparatus 102. Further, according to the present embodiment, processing such as generation of an HTTP header is performed, and data for transmission is output. The communication server 105 sends the data for transmission to the communication client 106.

  The communication client 106 receives the moving image requested by the URL designation as a result of the request. If the reception is successful, the received data is transferred to the moving image display module 107, where reproduction display is performed.

  Here, the outline of the processing on the system configuration described with reference to FIG. 1 is equivalent to the HTTP processing generally used in so-called WWW (World Wide Web), and is not a special usage method. However, the point that the moving image generation module 103 generates a moving image instead of the recorded static resource is excluded.

  Before describing the features of this embodiment in detail, the details of the system configuration of the system configuration suitable for this embodiment configured by the network camera 101 and the playback device 102 will be further described with reference to FIG. FIG. 2 is a block diagram illustrating an example of the internal configuration of the network camera 101 and the internal configuration of the playback apparatus 102 according to the present embodiment.

  The internal configuration of the network camera 101 is as follows. First, an image of a subject that is a source of a moving image to be sent is formed by an optical sensor 205 through a photographing lens unit 201 that is an optical system.

  The taking lens unit 201 has a mechanism for moving a lens group by a motor or the like for focusing, for example, and the direct control is performed by the taking lens unit drive circuit 202. At this time, the aperture unit 203 including the aperture is operated by the aperture unit drive circuit 204 so that the amount of light for image formation is appropriately adjusted.

  The optical sensor 205 is constituted by a solid-state imaging device (CCD, CMOS, etc.), and has a property of converting incident light into electric charge according to an optical quantity and storing it. This electric charge is read out and digitized by the A / D converter 207 to generate an uncompressed digital image.

  The optical sensor 205 is appropriately controlled by a pulse signal or the like output from the optical sensor driving circuit 206, and continues a series of operations for reading out charges accumulated during a designated time at a designated timing. Thus, a continuous digital image can be obtained. Needless to say, the continuous images acquired in this way are moving images.

  Next, continuously acquired digital images are transferred to the image signal processing circuit 208 for compression encoding, and after performing image correction such as white balance correction and gamma correction, the code compression circuit To 209. Transfer of digital images between such processes is performed by accessing the memory 210 at high speed using, for example, a DMA (Direct Memory Access) circuit. The code compression circuit 209 performs compression coding processing on the image data in the memory 210.

  For example, in a so-called motion JPEG image by continuous JPEG (ISO / IEC 10918) encoding, when a digital image is an RGB signal at the input of the image signal processing circuit 208, a signal is generated as a YC signal composed of a luminance signal Y and a chroma signal CbCr. It becomes. Then, after dividing these into blocks of 8 × 8 pixels, processes such as discrete cosine transform, quantization, and Huffman coding are performed, and a final compressed image is output.

  Alternatively, when the compression encoding method is a format that performs inter-frame prediction such as MPEG-2 (ISO / IEC 13818) or MPEG-4 (ISO / IEC 14496), a specific image (frame) to be compressed is used. ) Refer to the previous and next frames. Then, while referring to the previous and subsequent frames, motion compensation prediction, macroblock processing, and the like are performed, and compressed images (bitstreams) that are dependent on each other are output.

  The compressed moving image output in this manner is temporarily stored in the memory 210, shaped into a recording file format, and sent to the network via the network controller 211 and the communication circuit 212.

  The series of operations is controlled as the processing described with reference to FIG. That is, it is a program operation by a control program (firmware) that is generally called.

  The control program (firmware) is normally stored in the ROM 213 as static information. The network camera 101 reads a control program (firmware) from this ROM as necessary. Then, the entire operation is performed by processing by the CPU 214.

  The control program (firmware) processed by the CPU 214 has several other processing functions. For example, the control program (firmware) of the photographing lens unit 201 described above is instructed to perform a zoom operation (of the focal length). Move) and auto focus. Here, description of processing such as a moving image request from the playback device 102 is omitted. This is because, in the description of the apparatus configuration, such a processing flow becomes complicated. Needless to say, for example, such a moving image request message is also communicated via the communication circuit 212.

Next, the playback device 102 will be described. Here, similarly, a case where a moving image is received will be described as an example. The internal configuration of the playback device 102 is as follows.
The moving image data is taken into the playback device from the communication circuit 221 via the network controller 222. Such processing is performed by a control program (firmware) on the CPU 229 by a program stored in the ROM 228 as in the network camera 101, for example.

  The received data is temporarily stored in the memory 230. If necessary, it may be stored in the recording device 228 by the I / O controller 229. The stored received data is converted into a format that can be processed by the code decompression circuit 223, decoded by the code decompression circuit 223, and then subjected to image processing by the image signal processing circuit 224 so as to be suitable for display. And displayed on the display device 226.

  The display device 226 is converted from digital to analog by the A / D converter 225 if the refresh timing is adjusted by the display device driving circuit 227 or if the display device is analog-driven.

  In the description of FIG. 2, the control by the control program (firmware) has been described. Of course, part or all of the firmware processing can be processed by hardware for speeding up. For example, a part of protocol processing is generally easy to be hardwareized, and there are many examples where hardware is implemented as a so-called offloader device configuration. In the present embodiment, the most general configuration is merely described as processing by a control program.

  Next, in order to facilitate the subsequent description, an MP4 file as fragment video data applied to the present embodiment will be described. In the present embodiment, for example, MPEG-4 (ISO / IEC 14496) is used as the encoding format processed by the code compression circuit 209 and the code expansion circuit 223.

  The MP4 file is a file format internationally standardized as a standard file format of MPEG-4, and fragment video data is one of the formats defined in this MP4 file format as Fragmented Movie. In the present embodiment, the MP4 file format using MPEG-4 is shown, but the same operation is performed even in the ISO Base Media File Format file format using Motion-JPEG2000, for example. Alternatively, the encoding format is H.264. H.264 may be used.

  These are different encoding formats, but as will become apparent later, even if the encoding formats are different, they can be applied in the present embodiment as long as they are fragment video data.

Here, the structure of the MP4 file format and fragment video data (Fragmented Movie) will be described. FIG. 3 is a conceptual diagram showing an example of a data structure for an MP4 file format including a Fragmented Movie.
In the MP4 file format, the presentation of the entire content is called “movie”, and the presentation of the media stream constituting the content is called “track”.

  The first header, Movie_BOX ('moov') 302, typically includes a video track 303 that logically handles the entire moving image data and an audio track 307 that logically handles the entire audio data. ing. The basic configuration contents of the video track 303 and the audio track 307 are almost the same. That is, each track records various attribute information of actual media data, and the contents thereof are only slightly different depending on the characteristics of the media data.

  The data included in the video track 303 includes, for example, information such as so-called decoder configuration information for decoding encoded data and a rectangular size of a moving image. Further, an offset 304 indicating the position on the file where the media data is actually recorded, a sample size 305 indicating the size of each frame data (also called a picture) of the media data, and decoding of each frame data A time stamp 306 indicating the time and the presentation time is recorded.

  The entire configuration of the MP4 file 301 includes a header information (metadata) portion indicating the physical position, temporal position, characteristic information, and the like of video and audio data, and a medium that is an entity of encoded video / audio data. It consists of a data part.

  Figure 3 includes a Fragmented Movie original BOX. However, in an MP4 file with a simple structure that does not have a Fragmented Movie structure, Movie_BOX ('moov') 302 excluding Movie_Extends_BOX ('mvex') 308 that indicates information about the extended portion of the Fragmented Movie and media data that forms a pair with it. Only the Media_Data_BOX ('mdat') 311 that is a part.

  On the other hand, in an MP4 file having a Fragmented Movie structure, content header information and media data can be divided in arbitrary time units, and the divided “fragments” are recorded in chronological order from the beginning of the file.

  At this time, in the first Movie_BOX ('moov') 302 including the attribute information of the entire content, as shown in FIG. 3, Movie_Extends_BOX ('mvex') storing information such as the entire playback time (duration) including the fragment portion ') 308 is arranged. Then, information regarding data included in Media_Data_BOX ('mdat') 311 that follows is held.

  Movie_Fragment_BOX ('moof') 312 that appears next is header information of the fragment portion. Information on data included in Media_Data_BOX ('mdat') 313 is held. As described above, a combination of Movie_Fragment_BOX ('moof') and Media_Data_BOX ('mdat') is added in the same manner.

  In an MP4 file having a Fragmented Movie structure, Movie_Extends_BOX ('mvex') 308 storing extension information by Fragmented Movie exists in Movie_BOX ('moov') 302. The data included in Movie_Extends_BOX ('mvex') 308 includes the playback time (duration) 309 of the entire movie including the fragment portion, the default value such as the sample size of media data included in the fragment portion, and the duration for each sample. The information 310 can be set.

  By setting a default value here, when using the default value in the sample information in the Movie_Fragment_BOX ('moof') 312 that follows, the setting of the value for each sample can be omitted.

  As described above, in the MP4 file format file, various attribute information related to the media data is stored as a metadata area separately from the media data. This makes it possible to easily access the desired sample data regardless of how the media data is physically stored.

  Further, according to the Fragmented Movie structure, a file structure in which a plurality of blocks are connected in time-series order with metadata and media data as one block can be achieved. Next, with reference to FIG. 1 and FIG. 2, the parts that characterize this embodiment will be described more specifically and in detail.

FIG. 4 is a diagram illustrating an example of a URL used when a request for a moving image is made from the playback apparatus 102 to the network camera 101.
The communication client 106 and the communication server 105 are adapted to HTTP processing, and are connected from the communication client 106 to the communication server 105 with a URL as shown in the URL example 401 of FIG.

  The first half of the five examples of the URL example 401 are the same. This means that access is made to the CGI that is the relative path (cgi-bin / getmp4.cgi) of the address (// hostname /) given by the hostname by HTTP (http :). As a result, the HTTP processing server included in the communication server 105 starts the CGI program. The parameters in the latter half are values delivered to the CGI program.

  FIG. 4 shows the entire block 402 described by disassembling the last of the five examples. The protocol part 403, the address part 404, and the relative path part 405 are the parts described above. Further, the separator portion 406 is a separator from the subsequent parameter portion. The parameter part can be divided into five parts by a parameter separator part 408.

  The first parameter 407 (duration = 180) expresses that the requested moving image time is 180. Since this unit is specified to be a unit obtained by dividing 1 second into two in the subsequent second parameter 409 (scale = 1), a moving image of 180 seconds is specified. .

  The third parameter 410 (bitrate = 512000) specifies that the bit rate is 512000 bps. Then, with the fourth parameter 411 (top = 6) and the fifth parameter 412 (frg = 4), the first data block at the time of moving image transmission is 6 seconds, and thereafter, the data block is transmitted every 4 seconds. Specifies what to expect. That is, in this example, it means that a 3-minute moving image is requested at 512 kbps so that a data block of 6 seconds is first and then a data block of 4 seconds is obtained.

  The data block includes a combination of the first header ('moov') and data ('mdat') in the MP4 file as the fragment video data described above, and a fragment header ('moof') and data ('mdat'). ). It should be noted that the URL example described here is an implementation example, and the types, number, order, format, and the like of parameters may be different specifications. These are purely application dependent.

  Further, although the parameters have been described as if they are always delivered by URL, some or all of these parameters may be defined, stored, and recorded on the communication server 105 in advance. By doing so, even if necessary parameters are not notified from the communication client 106, they can be used.

Next, the operation of the CGI program started by the communication server 105 according to the above example will be described.
The CGI program needs to acquire the encoded data of the moving image output from the moving image generation module 103 for transmission.

  As described above, the encoded moving image data is generated by driving the compression encoding circuit 209. However, in order to simplify the operation description, the function of the CGI program is included in the communication data generation module 104. Shall be. Then, it is assumed that the moving image generation module 103 performs up to encoding of moving image data.

  When the moving image generation module 103 is stopped, the communication data generation module 104 needs to start it. This may correspond to the control program (firmware) starting to drive the code compression circuit 209 or the like, or simply to initialize the interface to obtain the encoded data. unknown.

FIG. 5 is a flowchart showing an example of the operation of the CGI program.
First, the activated CGI program initializes variables and the like used in the subsequent steps (step S501), and obtains parameters 531 to be transferred (step S502).

  Here, the parameter 531 corresponds to the parameter portion 402 described with reference to FIG. Next, the communication data length is calculated using the parameter 531 (step S503). The specific calculation method in step S503 for calculating the communication data length will be described in more detail later. Specifically, the communication data length is the file length of the MP4 sent from the communication server 105 to the communication client 106. It is.

  What should be noted here is that if the parameters described using FIG. 4 are the same as the example used in this description, the data does not include the data length, and only the CGI program of 512 kbps data for 3 minutes. Is a parameter that is understood. That is, in step S503 for calculating the communication data length, the data length of the MP4 file that is actually transmitted according to the parameter is calculated.

After the communication data length is calculated, an HTTP header is generated (step S504).
Details of the generated HTTP header will also be described later, but a particularly important point is that the data length of the MP4 file to be transmitted is described in the HTTP header. The data length of the MP4 file has already been calculated in step S503 for calculating the communication data length, and this value is used.

  Next, the encoded data is read from the encode buffer 532 as moving image data stored in the transmitted MP4 file, and a first data block is generated (step S505). The first data block is a combination part of the first header ('moov') and data ('mdat') in the MP4 file described above. The HTTP header and the first data block generated in this way are output as communication data 534 by data output 1 which is the next step (step S506).

  In a general HTTP server, communication data output from a CGI program is transmitted to a communication client to which the HTTP server is already connected via a standard input / output of the operating system. Here, it is assumed that output is performed according to some implementation specifications of such an HTTP server. In addition, since moving image data generally has a large data length, it may be output using a primary file or the like.

  Next, the end condition is estimated (step S507). Then, it is determined whether or not the end condition is satisfied (step S508). If the end condition is satisfied, the process proceeds to step S511 for generating a free data block, and if not, the process proceeds to step S509 for generating a subsequent data block. move on.

  Here, the estimation of the end condition is determined from the communication data length calculated in advance in step S503 for calculating the communication data length and the integrated value of the data length of the communication data 534 actually output, This is a process for determining whether the integrated value of the data length of the communication data 534 to be finally output does not exceed the calculated communication data length. If it is determined that this is exceeded, the data length is inconsistent with the data length notified to the communication client 106 according to the HTTP protocol. Termination conditions are estimated so that such contradiction does not occur.

  If the termination condition is not satisfied, step S509 for generating a subsequent data block is executed. The subsequent data block corresponds to a combination of the fragment header ('moof') and data ('mdat') in the MP4 file described above.

  Specifically, as in the first data block generation process (step S505), the encoded data is read from the encode buffer 532, and this subsequent data block is generated. The generated subsequent data block is output as communication data 534 in the data output 2 process (step S510), similarly to the data output 1 process (step S506) that is the output of the first data block. After the subsequent data block is output in this way, the end condition estimation step S507 is performed again. As long as the termination condition is not satisfied, this series of processing continues.

  The end condition estimation step S507 immediately after the data output 1 process (step S506) is performed and the end condition estimation process (step S507) after the data output 2 process (step S510) are basically performed. Is the same process. However, the integrated value of the data length of the communication data 534 actually output is different in that it clearly increases every time the data output 2 process (step S510) is passed. Details of the end condition estimation step S507 will also be described later.

  On the other hand, if the end condition is satisfied as a result of the determination in step S508, free data block generation is performed (step S511). The free data block generation is a process for generating a data area ('free') which is effective as communication data but does not include moving image data or data related to the moving image data itself ('free'). This kind of meaningless area is defined as an international standard such as an MP4 file so that it can be used for the convenience of the application because it is not necessary for implementation or editing of moving images. ing.

  The data generated by the free data block generation process (step S511) is the same as the data output 1 process (step S506) and the data output 2 process (step S510), which are outputs of the first data block. That is, it is output as communication data 534 in the data output 3 process (step S512).

  The communication data 534 output in this manner is a correct MP4 file by itself, and is finally completed by performing a close process as a final process of the CGI program (step S513).

  Here, the details of a specific calculation method of the communication data length calculation process (step S503) will be described. As described above, the communication data length is specifically the MP4 file length transmitted from the communication server 105 to the communication client 106.

  The MP4 file as fragment video data includes a combination of the first header ('moov') and data ('mdat') and zero or more fragment headers ('moof') and data ('mdat') in the MP4 file. Composed of a combination. Accordingly, the communication data length calculated here, more specifically, the file length of MP4 is a size obtained by adding these data lengths.

  However, it is difficult to accurately calculate these individual data lengths. This is because at least the data length of the compressed and encoded moving image data depends on the complexity of the moving image before encoding and the parameters at the time of compression encoding. Many encoders are designed so that a target bit rate is set and compression encoding is performed at a bit rate close to the target bit rate, but it is quite difficult to determine the data length in advance with accuracy.

  In addition, it can be said that it is easier to estimate the size of the first header ('moov') etc. in advance compared to video data, but there is still a part due to continuous length compression etc. in the data storage format It may be difficult to do so. That is, the communication data length calculation process (step S503) is nothing but exact, and it is nothing more than estimating the expected more accurate data length.

For most parts of the first header ('moov'), it is possible to estimate the data size of most items included in the header in advance.
Since it is redundant to explain how to estimate all the items included in these headers, detailed description thereof is omitted here. For example, Movie Header Box ('mvhd') is one of typical examples of items whose size can be determined in advance.

  On the other hand, for example, consider a case where the frame rate is fixed and the time stamp increases at a constant value in relation to the time scale. In this case, since the time-to-sample box ('stts') is recorded by continuous length compression, the recording size can be made constant.

  However, if the recorded time stamp difference is not constant, the recording size is not constant. In such a case, it is necessary to estimate the recording size in relation to the number of moving image data samples (number of frames) recorded in pairs with the first header. However, the recording size still varies up to the maximum number of samples (the number of frames), and the data length of this portion can be estimated relatively easily as an approximate value. .

More specifically, in the case of this Time-to-sample Box, ISO / IEC 14496-12, which is an international standard,
aligned (8) class TimeToSampleBox
extends FullBox ('stts', version = 0, 0) {
unsigned int (32) entry_count;
int i;
for (i = 0; i <entry_count; i ++) {
unsigned int (32) sample_count;
unsigned int (32) sample_delta;
}
}
The minimum size is 24 bytes, and the maximum size is 16 bytes and 4 bytes multiplied by the number of samples.

  Further, for example, Sample Size Box ('stsz') is defined so that the size increases in proportion to the number of samples. In this case, this can be estimated from the number of samples of data ('mdat') paired with the first header ('moov'). Thus, the size of the first header ('moov') in the communication data length calculation process (step S503) can be estimated almost accurately from the number of samples of the corresponding data ('mdat').

  Next, the size estimation of the data ('mdat') portion corresponding to the first header ('moov') will be described. The size of this portion can be estimated from, for example, the bit rate and the time (Duration) of this data block. If the bit rate is 8 Mbps and the time of the data block is 5 seconds, it is apparent from these values that 8 Mbps × 5 seconds ÷ 8 bits to 5 Mbytes can be calculated. Of course, a method may be used in which the number of samples (the number of frames) and the frame rate are determined in advance, and the time of the data block is calculated from these values.

  In general, such a value is defined in some form, and the encoding is controlled accordingly. The problem here is that it is difficult to accurately estimate the data size of the encoded data. As described above, what is important is not an accurate estimation but an estimation of a more accurate data length than expected.

  Similar to the combination of the first header ('moov') and data ('mdat') in the MP4 file, the data length of the combination of the fragment header ('moof') and data ('mdat') is estimated. be able to.

  The header ('moof') of a fragment is also defined based on almost the same concept, although its storage format and stored data are different from the header ('moov'). The difference is that the combination of the fragment header ('moof') and data ('mdat') occurs multiple times. For example, the entire MP4 file to be sent is 180 seconds, the combination of the first header ('moov') and data ('mdat') is 5 seconds, and the header ('moof') and data of one fragment after that Consider the case where the combination of ('mdat') is 4 seconds. In this case, fragments are generated a plurality of times in 175 seconds obtained by subtracting the first 5 seconds from 180. Therefore, it can be calculated that 44 fragments are generated by dividing 175 seconds by 4 seconds.

  First, one fragment will be described. Each movie_Fragment_BOX ('moof') 312 which is a fragment header has one Movie_Fragment_Header_BOX ('mfhd') and Track_Fragment_BOX ('traf') for each video or audio track. Furthermore, Track_Fragment_BOX ('traf') has one Track_Fragment_Header_BOX ('tfhd') for each track. As a result, the structure has one or more Track_Fragment_Run_BOX ('trun').

  Here, since Track_Fragment_BOX ('traf') only serves as a container for the included BOX similarly to Movie_Fragment_BOX ('moof') 312, its size is constant (8 bytes). . Movie_Fragment_Header_BOX ('mfhd') holds a sequence number added every time Movie_Fragment_BOX ('moof') 312 appears, and the size of the BOX is always constant (16 bytes).

  Track_Fragment_Header_BOX ('tfhd') holds a track ID. Optionally, default values such as the reference offset value of the fragment and the size and duration of the sample stored in the data ('mdat') can be set. For this reason, the BOX size differs depending on whether or not the default value is set.

  However, in general, whether or not to set default values depending on the encoding method and parameters related to encoding is determined at the time of file generation, and these settings are dynamically changed for tracks with the same ID. There is no. For this reason, the size of Track_Fragment_Header_BOX ('tfhd') having the same track ID in the same file can be regarded as constant.

For example, when a reference offset value (8 bytes) is set as an option, Track_Fragment_Header_BOX ('tfhd') is defined as follows according to ISO / IEC 14496-12. For this reason, the size of this BOX is 24 bytes. In addition, it is possible to determine which option field is valid by setting the following tf_flags.
aligned (8) class TrackFragmentHeaderBox
extends FullBox ('tfhd', 0, tf_flags) {
unsigned int (32) track_ID;
// all the following are optional fields
unsigned int (64) base_data_offset;
unsigned int (32) sample_description_index;
unsigned int (32) default_sample_duration;
unsigned int (32) default_sample_size;
unsigned int (32) default_sample_flags
}

  By the way, when there are multiple tracks, the method of storing the sample data in the data ('mdat') is to record (interleave) different track data alternately for each continuous 'cluster of samples' for a certain time. The method is general. This 'sample lump' is called 'chunk' in the MP4 file format.

  Here, one or more Track_Fragment_Run_BOX ('trun') existing in Track_Fragment_BOX ('traf') stores data information for each 'chunk'. For this reason, for example, when video and audio tracks exist, the video and audio chunks are stored in an interleaved manner in the data ('mdat'), and there is a Track_Fragment_Run_BOX ('trun') for each chunk. become.

  Even if there is only one track, there may be a plurality of Track_Fragment_Run_BOX ('trun'). For example, when the video data is in the encoding format of the type that performs inter-frame prediction such as MPEG-4, chunks are generally set for each GOV (Group of VOP, VOP: Video Object Planes) that is a group of encoded data. Form. Therefore, how many Track_Fragment_Run_BOX ('trun') exist in one Movie_Fragment_BOX ('moof') 312 is determined by the time of one data block and the time per chunk.

  Track_Fragment_Run_BOX ('trun') holds the number of samples stored in the chunk, and can optionally set the duration and size of each sample. Regarding the setting of this option, the BOX size is variable depending on the number of samples of the corresponding chunk in the same file, but the time for each chunk is usually constant. Therefore, if the data rate such as the frame rate of the video data and the sampling rate of the audio data is constant, the number of samples in the chunk is also constant between the same track IDs. The number of samples in one data block is determined by the time per data block and the data rate for each media track.

  Therefore, the number of Track_Fragment_Run_BOX ('trun') per data block can be obtained by “time per fragment / time per chunk” for each track. The number of samples per chunk can be obtained by “time of one data block / data rate per media track”.

  According to ISO / IEC 14496-12, Track_Fragment_Run_BOX ('trun') is defined as follows. For this reason, for example, it is assumed that data_offset, first_sample_flags, and sample_duration and sample_size are set as parameter options set for each sample as options of this BOX. Then, the size of the BOX can be obtained by “number of BOX × 24 bytes + number of samples per chunk × 8 bytes”.

Similarly to Track_Fragment_Header_BOX ('tfhd'), it is possible to determine which option field is valid by setting the following tr_flags.
aligned (8) class TrackRunBox
extends FullBox ('trun', 0, tr_flags) {
unsigned int (32) sample_count;
// the following are optional fields
signed int (32) data_offset;
unsigned int (32) first_sample_flags;
// all fields in the following array are optional
{
unsigned int (32) sample_duration;
unsigned int (32) sample_size;
unsigned int (32) sample_flags
unsigned int (32) sample_composition_time_offset;
} [sample_count]
}

  From the above, the approximate value of the size of the Movie_Fragment_BOX ('moof') 312 can be obtained by summing the sizes of the BOXes included in the Movie_Fragment_BOX ('moof') 312 described above. In addition, the estimated value of the fragment data ('mdat') part is estimated from the bit rate and the time (Duration) of this data block in the same way as the data ('mdat') corresponding to the first header ('moov'). can do.

  From the calculated values of the fragment header ('moof') and data ('mdat') and the number of fragments described above, the data lengths of all fragment portions can be estimated. Of course, the target size of each data block is determined in advance, not the calculation from the bit rate and time as described above. Then, the bit rate may be determined from the target size, and the encoding parameter may be set for the code compression circuit 209 that performs encoding.

  Next, details of the end condition estimation process (step S507) will be described. As described above, the communication data length calculated in advance by the communication data length calculation process (step S503) is the same as the data length of the communication data 543 actually transmitted in the end condition estimation process (step S507). To be compared.

  In the end condition estimation process (step S507), the integrated value of the data length of the communication data 543 actually output in the data output 1 process (step S506) and the data output 2 process (step S510), and the next output fragment It is determined whether the sum of the data block length and the expected size is larger than the communication data length calculated in advance. If it is determined to be large, the end condition is satisfied, and if it is determined to be small, the end condition is not satisfied.

  Here, the expected size of the data block length of the next fragment to be output is nothing but the data block length of one fragment calculated in the communication data length calculation process (step S503).

  Here, since the size of the data that is actually output may exceed the predicted size calculated in advance, the predicted size calculated in advance may be corrected. That is, comparison may be made by adding the maximum difference between the data size of the fragment actually output so far and the expected size, or by adding a preset error.

  When adding the set error, the integrated value of the data length of the communication data 543 actually output in the data output 1 process (step S506) and the data output 2 process (step S510) and the fragment to be output next. The set error value is added to the sum of the expected size of the data block length. And it is judged whether this is larger than the communication data length calculated beforehand.

Next, the free data block generation process (step S511) will be described.
The data block generated by the free data block generation process (step S511) is output immediately after the last fragment (the output data of the data output 1 process (step S506) depending on conditions).

  The generated data is data that is not particularly meaningful in itself, which is called a free box or a skip box in MP4. This detail is not described in detail because it is defined in the MP4 standard, but a description of the data length is added.

  The size of the data block generated in the free data block generation process (step S511) is obtained by subtracting the data length actually output so far from the data length calculated in the communication data length calculation process (step S503). Is. If a free box is used, according to the definition, the size including the data length of the free box and the identifier indicating the 4-byte free box is made the size of this data block. In this way, the data length calculated in the communication data length calculation process (step S503) can be completely matched with the actually output data length.

At the end of the description of the flowchart using FIG. 5, the generated HTTP header will be described. FIG. 6 is a schematic diagram showing the overall image of the communication data 543 described above.
In FIG. 6, communication data 543 includes an HTTP header 601, a first data block composed of a combination of the first header ('moov') and data ('mdat') in the subsequent MP4 file, and a fragment header in the MP4 file ( It consists of two subsequent data blocks consisting of a combination of 'moof') and data ('mdat'), and a free data block.

  In the schematic diagram shown in FIG. 6, the combination of the fragment header ('moof') and data ('mdat') occurs twice, but as explained, this is an arbitrary number of zero or more times. Needless to say, it occurs several times. In addition, in order to clearly show that the data ('mdat') is a part for storing moving image data, this is indicated by an arrow from the encode buffer 532.

  The HTTP header 601 is an example of a status line and a response header in the HTTP protocol, and does not necessarily describe all standard messages, but includes at least important items in the present embodiment. The first line is a status line, which in this example indicates that HTTP / 1.0 was successful for an HTTP request. As a result, the communication client 106 can know that the process is being performed correctly.

  The next line is a part of the response header and notifies the size of data transmitted according to HTTP. The communication client 106 can accurately know the size of data transmitted by this line, that is, received. Here, the size of the data to be transmitted is nothing but the MP4 file length calculated in the communication data length calculation process (step S503) in the flowchart described with reference to FIG. The communication client 106 acquires the file size estimated by the CGI program of the communication server 105 based on this information.

  On the other hand, by appropriately adjusting the file size according to the file size, the CGI program can process the moving image data recorded and stored as a file accurately in advance as if it was downloaded and acquired.

  The third line shows the format of the moving image data. If the communication server 105 and the communication client 106 do not understand the format of the moving image data in advance, the communication client 106 can know the format of the received data by this line.

  As described above, according to this method, even if the specification defined in HTTP / 1.1 is not implemented on the transmission side, the reception side, or both, if the HTTP / 1.0 is supported, the live Video can be distributed and received / played. Further, since the received data becomes correct data as an MP4 file in the fragment video format, it can be reused as content.

(Second Embodiment)
In the first embodiment, the embodiment using HTTP as the communication protocol has been described. However, the present embodiment is applicable to communication formats other than HTTP. For example, the communication server 105 and the communication client 106 shown in the first embodiment have a mechanism for calling a function between terminals such as SOAP (Simple Object Access Protocol). A technique for acquiring moving image data by remote function call from the communication client 106 has been technically established.

  SOAP itself is a lightweight protocol for performing communication between objects via a network, and HTTP or the like can be used as means for realizing it. However, since SOAP itself does not have a description standard for downloading moving image data, an application that uses this actually defines and uses a description method.

Next, an example of a message using SOAP is shown below.
<SOAP-ENV: Envelope>
<SOAP-ENV: Body>
<ns: getMP4>
<ns: duration> 180 </ ns: duration>
</ ns: getMP4>
</ SOAP-ENV: Body>
</ SOAP-ENV: Envelope>

  In this example, data for 180 seconds is requested with the function name “getMP4”. When this SOAP message is transmitted from the communication client 106 to the communication server 105, the communication server 105 performs processing of this function in the same manner as the CGI. The detailed method of such processing depends on the implementation, and may be processed in a separate process as is common in CGI programs, or may be processed in the program that received the message.

The function called in this way calculates the communication data length in the same manner as the communication data length calculation process (step S503) shown in FIG. 5 in the first embodiment. An example of the result of processing by SOAP is shown below.
<SOAP-ENV: Envelope>
<SOAP-ENV: Body>
<ns: getMP4Response>
<ns: length> 45000000 </ ns: length>
<ns: source> (data) </ ns: source>
</ ns1: getMP4Response>
</ SOAP-ENV: Body>
</ SOAP-ENV: Envelope>

  In this example, the result returned to the communication client 106 is the data length given by “ns: length” and the actual moving picture data given by “ns: source”. What should be noted here is that the data length and the actual moving image data are handled in exactly the same manner as the flowchart shown in FIG. 5 in the first embodiment.

  Thus, this embodiment is applicable also to communication formats other than HTTP. In this example, the video data is processed as a part of the SOAP message. However, due to restrictions on encoding of transmission data, instead of returning the protocol and access method for acquiring video data, There is no difference in the technical background of the present invention.

(Other embodiments according to the present invention)
The present invention is applicable as long as the management information is interleaved along with time in addition to the ISO Base Media File format or similar media data such as moving images. As such a file format, it can be applied to an AVC file (MPEG-4 Advanced Video Coding file), a Motion-JPEG 2000 file, and the like. All of these file formats have the same basic format as the ISO Base Media File format and can be considered to be substantially equivalent.

  Each means constituting the moving image data distribution apparatus and each process of the control method of the moving image data distribution apparatus in the embodiment of the present invention described above is realized by the operation of a program stored in a RAM or ROM of a computer. it can. This program and a computer-readable recording medium recording the program are included in the present invention.

  Further, the present invention can be implemented as, for example, a system, apparatus, method, program, or recording medium. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowchart shown in FIG. 5) for realizing the functions of the above-described embodiments is directly or remotely supplied to the system or apparatus. This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a floppy (registered trademark) disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  As another method, the program of the present invention is encrypted, stored in a recording medium such as a CD-ROM, distributed to users, and encrypted from a homepage via the Internet to users who have cleared predetermined conditions. Download the key information to be solved. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  As another method, the program read from the recording medium is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

It is a conceptual diagram which shows the system configuration example in the 1st Embodiment of this invention. 2 is a block diagram illustrating an example of an internal configuration of a network camera 101 and an internal configuration of a playback apparatus 102 according to the first embodiment of the present invention. FIG. In the 1st Embodiment of this invention, it is a conceptual diagram which shows an example of the data structure about the MP4 file format containing a fragment image | video format. In the 1st Embodiment of this invention, it is a figure which shows an example of URL. 4 is a flowchart illustrating an example of an operation of a CGI program in the first embodiment of the present invention. It is a figure which shows the whole image of communication data in the 1st Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Network camera 102 Playback apparatus 103 Moving image generation module 104 Communication data generation module 105 Communication server 106 Communication client 107 Moving image display module 201 Shooting lens unit 202 Shooting lens unit drive circuit 203 Aperture unit 204 Aperture unit drive circuit 205 Optical sensor 206 Optical Sensor drive circuit 207 A / D converter 208 Image signal processing circuit 209 Code compression circuit 210 Memory 211 Network controller 212 Communication circuit 213 ROM
214 CPU
221 Communication circuit 222 Network controller 223 Sign expansion circuit 224 Image signal processing circuit 225 A / D converter 226 Display device 227 Display device drive circuit 228 Recording device 229 I / O controller

Claims (6)

Video data generated so as to be composed of one or more data blocks consisting of encoded data of the video and management information of the encoded data according to the generation order of the video data consisting of at least one of video and audio Generating means;
Moving image data size approximating means for estimating the size of the moving image data in advance;
First communication data generation means for generating the size information of the moving image data estimated by the moving image data size estimation means and the first data block in a format suitable for a communication procedure;
Second communication data generating means for sequentially generating moving image data generated by the moving image data generating means in a format suitable for a communication procedure;
Communication data integrating means for integrating the sizes of communication data sequentially generated by the first communication data generating means and the second communication data generating means;
End determination means for determining end from the size information of the moving image data estimated by the moving image data size estimation means and the size information of the communication data accumulated by the communication data accumulation means;
When it is determined that the end is determined by the end determination unit, the difference between the size information of the moving image data estimated by the moving image data size estimation unit and the size information of the communication data integrated by the moving image data size estimation unit Third communication data generating means for generating ignored data of equal size;
A moving image data distribution apparatus comprising: communication means for sequentially transmitting communication data generated by the third communication data generation means.   The moving image data size estimation means is a moving image that approximates the sum of the encoded data size calculated from the encoding rate of the moving image data and the distribution time of the moving image data and the management information of the encoded data. 2. The moving image data distribution apparatus according to claim 1, wherein the moving image data distribution device has a size of image data.   The termination determining means includes the size information of the communication data, the size information of the communication data accumulated by the moving image data size estimating means, the encoding rate of the moving image data, and the next of the moving image data. This is performed by determining whether or not it is smaller than the sum of the size of the encoded data calculated from the time of the data block, the management information of the encoded data, and an error value of 0 or more given in advance. The moving image data distribution apparatus according to claim 1, wherein:   The moving image data includes MPEG, JPEG, or encoded data comparable to the MPEG and JPEG, and the main body of communication data distributed by the communication means is in ISO Base Media File format or a derivative format thereof. The moving image data distribution apparatus according to any one of claims 1 to 3.   The moving image data distribution apparatus according to claim 1, wherein the communication unit uses HTTP or SOAP as a communication procedure. The moving image data distribution device according to any one of claims 1 to 5,
A moving image data communication comprising: a client communication unit that communicates with the moving image data distribution device; and a reproduction device that includes a moving image reproduction unit that reproduces the moving image data received by the client communication unit. system.

Images