JP4433287B2 - Receiving apparatus and method, and program - Google Patents

Description

The present invention relates to a receiving apparatus, method, and program, and more particularly, to a receiving apparatus, method, and program that can notify a bandwidth of a communication network required for transmission of hierarchically encoded data.

  Recently, high-resolution displays from mobile phones with low-resolution displays and CPUs (Central Processing Units) with low processing power in streaming distribution such as video-on-demand and live video, or real-time communications such as video conferencing and videophones. And multiple receiving terminals with different resolutions and processing capacities from the same source to personal computers (PCs) with high processing capacities (PC (Personal Computer)) matched the resolution and processing capacities of each receiving terminal. There has been proposed a communication system using hierarchical coding that can use services.

  There are MPEG (Moving Picture Experts Group) 4 and Motion JPEG (Joint Photographic Experts Group) 2000 as compression / decompression methods capable of such hierarchical encoding.

  In MPEG4, as a hierarchical coding technique, FGS (Fine Granuality Scalability) coding method will be incorporated into the standard, and it will be profiled. From low-quality data to high-quality data can be distributed in a scalable manner. It is said that it is possible.

  In JPEG2000, wavelet conversion is used as a method for converting data into frequency components. In JPEG2000, it is possible to packetize by giving priority to the spatial resolution at the time of decoding and to give priority to the image quality at the time of decoding by making use of the characteristics of the wavelet transform.

  Furthermore, JPEG2000 can store hierarchized data in a file format according to the standard of Motion JPEG2000 (Part 3) that can handle not only still images but also moving images.

  Conventionally, as data on the transmission side, it has been necessary to prepare data of different formats according to the data processing capability of the receiving terminal and data of a size corresponding to the transmission band of the communication network. As a result, it becomes possible to simultaneously distribute data corresponding to the processing capability of the receiving terminal and the transmission band of the communication network to a receiving terminal (receiving device) having different processing capabilities from one file data. .

  In addition, since distribution of such image data normally requires real-time properties, when communicating on a communication network, UDP, which is a protocol defined in IETF RFC (Internet Engineering Task Force Request For Comments) 768, is used. Image data is transmitted according to (User Datagram Protocol). Further, in a layer above UDP, a format defined for each application program, that is, for each encoding method, is used in accordance with RTP (Realtime Transport Protocol), which is a protocol defined by IETF RFC1889.

  A conventional transmission apparatus stores hierarchically encoded data as a payload in a packet to be transmitted (see, for example, Patent Document 1). In this transmission apparatus, priority information indicating priority for each layer is added to the header of the packet, and packets corresponding to layers up to a predetermined priority are received in order from the first layer with the highest priority. Sent to the device.

JP 2003-152544 A

  However, in the technique disclosed in Patent Document 1 described above, even if the transmission band of the communication network is known and the hierarchy of the data to be transmitted can be changed, the receiving side does not have the transmission device. The number of layers of data that are sent, and how much transmission bandwidth is needed to receive each layer of data, There was a problem that the hierarchy could not be selected properly.

  The present invention has been made in view of such a situation, and enables a receiving side to grasp the bandwidth of a communication network required for transmission of hierarchically encoded data.

The receiving apparatus according to the present invention transmits, from the transmitting apparatus, hierarchy table information indicating a transmission layer indicating the highest layer of transmission data to be transmitted and a communication network band necessary for transmitting transmission data up to the transmission layer. Receiving means for receiving, from a transmitting device, a packet storing transmission data, transmission layer information indicating a transmission layer of the transmitted transmission data, and maximum layer information indicating the highest layer among a plurality of layers Storage means for storing the received hierarchy table information, calculation means for calculating a reception band that is a bandwidth of the communication network at the time of packet reception, based on the reception time of the packet and the packet size of the packet, and the received hierarchy In addition to displaying table information, transmission data, transmission layer information, maximum layer information, and reception bandwidth stored in the received packet Display means for displaying a display screen displayed, generated according to a user instruction, and transmitting means for transmitting a change request to the transmitting device requesting a change in the transmit hierarchy of transmission data transmitted from the transmitting device It is characterized by providing.

The receiving means further receives a bandwidth measurement packet for measuring the bandwidth of the communication network, which is transmitted from the transmission device before transmission of the packet by the transmission device, and from the transmission device based on the received bandwidth measurement packet Bandwidth measuring means for calculating the bandwidth of the communication network when receiving the bandwidth measurement packet, which is used to determine the transmission layer of the transmitted data to be transmitted, is further provided, and the transmission means transmits the measurement result by the bandwidth measuring means to the transmission device Further, the display means can display the measurement result on the display screen .
The calculating means can further calculate the packet loss rate based on the received packet, and the display means can display the loss rate on the display screen.

The reception method or program according to the present invention transmits layer table information indicating a transmission layer indicating the highest layer of transmission data to be transmitted and a bandwidth of a communication network necessary for transmitting transmission data up to the transmission layer. A first reception control step for controlling reception from the apparatus; a storage control step for controlling storage of received hierarchical table information; a first display control step for controlling display of received hierarchical table information; and transmission data , Second reception control for controlling reception of a packet storing transmission layer information indicating a transmission layer of transmitted data and maximum layer information indicating a highest layer among a plurality of layers from a transmission device Based on the step, the packet reception time, and the packet packet size, the reception bandwidth that is the bandwidth of the communication network at the time of packet reception is calculated An output step, a second display control step for controlling display of a display screen on which transmission data, transmission layer information and maximum layer information stored in the received packet, and a reception band are displayed, and a user instruction And a transmission control step for controlling transmission to the transmission device of a change request for requesting a change in the transmission hierarchy of transmission data transmitted from the transmission device .

  The reception device may be an independent device or a block that performs reception processing of the communication device.

In the receiving apparatus, method, and program of the present invention, a hierarchy indicating a transmission hierarchy indicating the highest hierarchy of transmission data to be transmitted and a bandwidth of a communication network necessary for transmitting transmission data up to the transmission hierarchy The table information is received from the transmitting device, the received hierarchy table information is stored, and the received hierarchy table information is displayed. A packet storing transmission data, transmission layer information indicating the transmission layer of the transmitted transmission data, and maximum layer information indicating the highest layer among the plurality of layers is received from the transmission device, and the packet is received. Based on the time and the packet size of the packet, the reception bandwidth, which is the bandwidth of the communication network at the time of packet reception, is calculated, the transmission data, transmission layer information, maximum layer information stored in the received packet, and the reception bandwidth Is displayed, and a change request for requesting a change in the transmission hierarchy of transmission data transmitted from the transmission device, generated in response to a user instruction, is transmitted to the transmission device.

  The network is a mechanism in which at least two devices are connected and information can be transmitted from one device to another device. The devices that communicate via the network may be independent devices, or may be internal blocks that constitute one device.

  The communication is not only wireless communication and wired communication, but also communication in which wireless communication and wired communication are mixed, that is, wireless communication is performed in a certain section and wired communication is performed in another section. May be. Further, communication from one device to another device may be performed by wired communication, and communication from another device to one device may be performed by wireless communication.

  According to the present invention, it is possible to grasp the bandwidth of a communication network required for transmission of hierarchically encoded data on the data receiving side.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, although there is an embodiment which is described in the embodiment of the invention but is not described here as corresponding to the invention, it means that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

According to the present invention, a receiving device is provided. This receiving apparatus (for example, the client 14 in FIG. 1) indicates a transmission layer indicating the highest layer of transmission data to be transmitted, and a communication network band necessary for transmitting transmission data up to the transmission layer. The hierarchy table information (for example, the content hierarchy table in FIG. 8) is received from the transmission apparatus, and the transmission data, the transmission hierarchy information indicating the transmission hierarchy of the transmitted transmission data, and the highest hierarchy among the plurality of hierarchies are displayed. Receiving means (for example, UDP communication unit 151 in FIG. 6) for receiving a packet storing the maximum hierarchy information shown from the transmission device, and storage means (for example, content hierarchy holding unit in FIG. 6) for storing the received hierarchy table information 159) and the reception bandwidth, which is the bandwidth of the communication network at the time of packet reception, based on the packet reception time and the packet packet size Display means (for example, the stream information calculation unit 155 in FIG. 6), received hierarchy table information, transmission data stored in the received packet, transmission hierarchy information, maximum hierarchy information, reception bandwidth, A display means for displaying a display screen on which the message is displayed (for example, the output unit 87 in FIG. 6), and a change requesting a change in the transmission hierarchy of transmission data generated in response to a user instruction and transmitted from the transmission device Transmission means (for example, TCP communication unit 158 in FIG. 6) for transmitting the request to the transmission device .

According to the present invention, a receiving method is provided. This reception method uses hierarchical table information (for example, FIG. 8) indicating a transmission layer indicating the highest layer of transmission data to be transmitted and a bandwidth of a communication network necessary for transmitting transmission data up to the transmission layer. A first reception control step (for example, step S101 in FIG. 13) for controlling reception of the content hierarchy table from the transmission device, and a storage control step (for example, step in FIG. 13) for controlling storage of the received hierarchy table information. S102), a first display control step (for example, step S104 in FIG. 13) for controlling the display of the received hierarchy table information, transmission data, transmission hierarchy information indicating the transmission hierarchy of the transmitted transmission data, and a plurality of The second reception control step for controlling the reception from the transmitting device of the packet storing the maximum layer information indicating the highest layer among the layers. 15 (for example, step S121 in FIG. 15) and a calculation step (for example, step in FIG. 15) for calculating a reception band that is a bandwidth of the communication network at the time of packet reception based on the packet reception time and the packet size of the packet. S128) and a second display control step (for example, FIG. 15) for controlling display of a display screen on which transmission data, transmission layer information, maximum layer information, and reception band stored in the received packet are displayed. Step S132 and Step S133), and a transmission control step for controlling transmission to the transmission device of a change request for requesting a change in the transmission hierarchy of transmission data transmitted from the transmission device, generated in accordance with a user instruction (for example, , And step S127) of FIG.

  This program can be recorded on a recording medium.

  The present invention can be applied to a communication system that transmits streaming data in real time, such as an Internet phone, a remote video conference system, a live video streaming distribution system, or a video phone.

  FIG. 1 is a diagram showing an embodiment of a streaming distribution system to which the present invention is applied. In this streaming distribution system, a server 12 and a client 14 are connected to a communication network 13 such as the Internet.

  The video camera 11 captures an image and supplies image data corresponding to the captured image to the server 12. For example, the video camera 11 captures a moving image and supplies image data corresponding to the moving image to the server 12.

  Image data is an example of streaming data. The streaming data may be data that is requested to be transmitted or received sequentially as time passes, such as audio data or real-time control data.

  The server 12 transmits a bandwidth measurement packet for measuring the bandwidth of the communication network 13 to the client 14 via the communication network 13. Further, the server 12 receives the band measurement result transmitted from the client 14. Here, the band refers to the transmission rate in the communication network 13 connecting the server 12 and the client 14, that is, the amount of data that can be transmitted per unit time.

  The server 12 hierarchically encodes the image data supplied from the video camera 11, and stores the hierarchically encoded image data and hierarchical information in each packet corresponding to each hierarchical layer. . Here, the hierarchy information is maximum hierarchy information indicating the number of hierarchized image data hierarchies and transmission indicating the highest hierarchy (hereinafter referred to as a transmission hierarchy) among the hierarchy of image data to be transmitted. It consists of hierarchical information. In the following, the highest hierarchy among the hierarchized image data hierarchy is referred to as the maximum hierarchy.

  Note that the image data supplied from the video camera 11 is hierarchized into a plurality of hierarchies from the lowest hierarchy indispensable for decoding to the highest hierarchy for higher quality decoding. Further, it has been described that the hierarchy of image data indispensable for decoding is the lowest hierarchy, and the hierarchy of image data for decoding with higher quality is the highest hierarchy. The hierarchy of image data that cannot be used may be the highest hierarchy, and the hierarchy of image data for higher quality decoding may be the lowest hierarchy.

  The server 12 selects a transmission layer based on the received bandwidth measurement result, and based on the selected transmission layer, the packet corresponding to each of the layers from the first layer, which is the lowest layer, to the transmission layer Are transmitted to the client 14 via the communication network 13.

  The communication network 13 is a transmission path including a wired or wireless communication line, a network, or the Internet, and transmits a packet transmitted from the server 12 to the client 14.

  The server 12 receives the change request transmitted from the client 14, and changes the transmission hierarchy based on the received change request.

  The client 14 receives various packets transmitted from the server 12 via the communication network 13.

  The client 14 receives the bandwidth measurement packet transmitted from the server 12. The client 14 calculates the bandwidth of the communication network 13 based on the received bandwidth measurement packet, and transmits the calculated bandwidth as a bandwidth measurement result to the server 12 via the communication network 13.

  The client 14 receives a packet storing image data transmitted from the server 12. The client 14 extracts layer information from the received packet, and displays the extracted layer information on an output unit (not shown). The client 14 generates a change request for requesting a change in the transmission hierarchy, and transmits the generated change request to the server 12 via the communication network 13.

  Next, an example of bandwidth measurement processing and transmission layer change processing in the server 12 will be described with reference to the time chart of FIG.

  In FIG. 2, the horizontal axis indicates time, and the bandwidth measurement packet 31 includes, for example, a packet 41-1 and a packet 41-2 having the same packet size for measuring the bandwidth of the communication network 13. Further, for example, streaming data that is image data for reproducing the frame 32 is stored and transmitted in the packets 42-1 to 42-3, and streaming data that is image data for reproducing the frame 33 is: Streaming data, which is image data for reproducing the frame 34, is stored and transmitted in the packets 43-1 to 43-3 and transmitted in the packets 44-1 to 44-4.

  At time t1, the server 12 transmits a packet 41-1 for measuring the bandwidth of the communication network 13 to the client 14 via the communication network 13. Similarly, at time t2, the server 12 transmits a packet 41-2 for measuring the bandwidth of the communication network 13 to the client 14 via the communication network 13. The packet 41-1 and the packet 41-2 are transmitted to the client 14 continuously (continuously) without a time interval.

  At time t11, the client 14 receives the packet 41-1 transmitted from the server 12, and at time t12, the client 14 receives the packet 41-2 transmitted from the server 12. Upon receiving the packet 41-1 and the packet 41-2, the client 14 calculates the bandwidth of the communication network 13 based on the received packet 41-1 and the packet 41-2. Details of the band measurement will be described later.

  At time t <b> 13, the client 14 transmits a band measurement result that is a calculated band measurement result to the server 12 via the communication network 13.

  At time t3, the server 12 receives the bandwidth measurement result transmitted from the client 14. When the server 12 receives the band measurement result, the server 12 selects a transmission layer of the packet storing the image data to be transmitted to the client 14 based on the received band measurement result.

  The server 12 hierarchically encodes the image data into a predetermined number of hierarchies, and stores the hierarchized image data of each hierarchy in each of the packets corresponding to each hierarchy. Based on the transmission layer, the server 12 transmits each packet corresponding to each of the layers from the first layer to the transmission layer to the client 14 via the communication network 13.

  For example, the server 12 stratifies image data for reproducing one frame into four hierarchies from the first hierarchy that is the lowest hierarchy to the fourth hierarchy that is the highest hierarchy. The image data of the first hierarchy is data for decoding low-resolution image data. When the image data of the second hierarchy is decoded together with the image data of the first hierarchy, the image of the first hierarchy This is image data from which an image having a higher resolution than the data can be obtained.

  Here, when the image data of the first hierarchy, which is the image data of the lowest hierarchy, is decoded, it is possible to obtain image data for displaying a low-resolution image. Even if only the image data of the second hierarchy is decoded, image data for displaying an image cannot be obtained. That is, the image data of the lowest hierarchy is image data indispensable for decoding, and the image data of the hierarchy higher than the lowest hierarchy is only the image data (the image data of the second hierarchy or higher). Even if decoding, it is impossible to obtain image data for displaying an image, and it is possible to obtain image data for displaying an image when both the image data of itself and the image data of a layer lower than itself are decoded.

  Further, when the image data of the third hierarchy is decoded together with the image data of the first hierarchy and the image data of the second hierarchy, the image data of the first hierarchy and the image data of the second hierarchy are decoded. This is image data from which an image having a higher resolution than the obtained image is obtained. The fourth layer image data is the first layer image data, the second layer image data, and the third layer image data. When decoded together with this, the image data is the image having the highest resolution and the resolution close to the image before hierarchical encoding.

  That is, when the image data is hierarchized into four hierarchies from the first hierarchy to the fourth hierarchy by hierarchical coding, the first hierarchy is the lowest level that is indispensable for decoding of the image data. The second layer is one layer higher than the first layer, the third layer is one layer higher than the second layer, and the fourth layer is the highest quality. This is the highest layer for high-decoding.

  Therefore, for example, when the third hierarchy is selected as the transmission hierarchy, the server 12 sends a packet corresponding to the first hierarchy, a packet corresponding to the second hierarchy, and a packet corresponding to the third hierarchy. The data is transmitted to the client 14 via the communication network 13.

  Here, an example has been described in which hierarchical encoding is performed with priority on the resolution at the time of decoding, but it is also possible to perform hierarchical encoding with priority on the spatial position and color components of the image. .

  At time t4, the server 12 transmits the packets 42-1 to 42-3 to the client 14 via the communication network 13 based on the selected transmission layer. Similarly, the server 12 transmits the packets 43-1 to 43-3 to the client 14 via the communication network 13.

  The client 14 receives the packets 42-1 to 42-3 and the packets 43-1 to 43-3 transmitted from the server 12.

  At time t <b> 14, the client 14 transmits a change request for requesting a change in the transmission hierarchy to the server 12 via the communication network 13.

  At time t5, the server 12 receives the change request transmitted from the client 14. The server 12 changes the transmission layer based on the received change request, and transmits the packets 44-1 to 44-4 to the client 14 via the communication network 13 based on the changed transmission layer. To do. The client 14 receives the packets 44-1 to 44-4 transmitted from the server 12.

  In this way, the server 12 selects a transmission layer according to the bandwidth, and transmits a packet storing image data to the client 14 based on the selected transmission layer.

  FIG. 3 is a diagram for explaining band measurement.

  As a method for measuring the bandwidth, there is a method called a packet pair. In this method, the server 12 transmits packets of the same packet size to the client 14 via the communication network 13 in a so-called back-to-back manner without leaving a transmission interval. Thus, the bandwidth of the bottleneck link of the communication network 13 between the server 12 and the client 14 is measured based on the transmission delay of the communication network 13. The bandwidth of this bottleneck link is a bandwidth that is the transmission speed of the communication network 13.

  Here, transmitting packets of equal packet size by back-to-back refers to transmitting two packets continuously without leaving a time interval. In addition, when packets are transmitted continuously in this way, it is also referred to as transmission with tail-to-nose.

  In FIG. 3, the vertical axis represents time. At time t31, the server 12 transmits the packet 41-1 to the client 14 via the communication network 13. At time t32, the server 12 transmits the packet 41-2 to the client 14 via the communication network 13.

  The packet 41-1 and the packet 41-2 transmitted from the server 12 are received by the client 14 via the node 61-1 and the node 61-2 in the communication network 13.

  At time t33, the client 14 receives the packet 41-1 transmitted from the server 12. At time t34, the client 14 receives the packet 41-2 transmitted from the server 12.

  When the transmission speed between the node 61-1 and the node 61-2 is low, the packet 41-1 and the packet 41-2 transmitted from the server 12 without leaving a transmission interval are transferred from the node 61-1 to the node 61-. It can be seen that it is extended in the time axis direction in the interval of 2. Therefore, in the communication network 13, it can be seen that the section from the node 61-1 to the node 61-2 is the bottleneck link having the lowest packet transmission band in the section from the server 12 to the client 14.

  The client 14 calculates the bandwidth of the bottleneck link, that is, the bandwidth based on the reception times of the received packets 41-1 and 41-2. The bandwidth is calculated from equation (1).

  (Bandwidth) = S1 / (t34-t33) (1)

  Here, S1 represents the packet size (packet size) of 41-2.

  In this way, the client 14 calculates the bandwidth based on the bandwidth measurement packet transmitted from the server 12.

  FIG. 4 is a block diagram illustrating an example of the configuration of the server 12. A CPU (Central Processing Unit) 81 executes various processes according to a program recorded in a ROM (Read Only Memory) 82 or a recording unit 88. A RAM (Random Access Memory) 83 appropriately stores programs executed by the CPU 81 and data. The CPU 81, ROM 82, and RAM 83 are connected to each other by a bus 84.

  An input / output interface 85 is also connected to the CPU 81 via the bus 84. The input / output interface 85 is connected to an input unit 86 including a keyboard, a mouse, and a switch, and an output unit 87 including a display, a speaker, and a lamp. The CPU 81 executes various processes in response to commands input from the input unit 86.

  The recording unit 88 connected to the input / output interface 85 is composed of, for example, a hard disk and records programs executed by the CPU 81 and various data. The communication unit 89 communicates with an external device such as the client 14 via the communication network 13 such as the Internet or other networks.

  Alternatively, the program may be acquired via the communication unit 89 and recorded in the recording unit 88.

  The drive 90 connected to the input / output interface 85 drives the magnetic disk 111, the optical disk 112, the magneto-optical disk 113, or the semiconductor memory 114 when they are mounted, and programs and data recorded there. Get etc. The acquired program and data are transferred to the recording unit 88 and recorded as necessary.

  Since the client 14 is configured in the same manner as the server 12, the description thereof is omitted.

  FIG. 5 is a block diagram illustrating a functional configuration of the server 12.

  The server 12 includes an encoder 131, a packetizer 132, a buffer 133, a UDP communication unit 134, a TCP communication unit 135, a transmission layer selection unit 136, a content layer holding unit 137, a transmission layer information holding unit 138, and a layer processing unit 139. Configured.

  The encoder 131 refers to the content hierarchy table held (recorded) in the content hierarchy holding unit 137 for image data, which is an example of streaming data supplied from the video camera 11, for each frequency band by a predetermined method. Hierarchical coding. The encoder 131 supplies the hierarchically encoded image data to the packetizer 132.

  Here, the content hierarchy table includes the number of hierarchies for hierarchizing image data and the necessary bandwidth, which is a transmission bandwidth necessary for transmitting image data corresponding to each layer via the communication network 13. Corresponding (related) is recorded. Details of the content hierarchy table will be described later.

  Note that the encoder 131 may perform hierarchical encoding without referring to the content hierarchy table. In this case, the encoder 131 supplies information indicating the number of hierarchized image data layers and the required bandwidth of each layer to the content layer holding unit 137.

  The packetizer 132 acquires the hierarchy information from the hierarchy processing unit 139. The packetizer 132 generates an RTP packet by storing the hierarchical information acquired from the hierarchical processing unit 139 and the image data supplied from the encoder 131 in an RTP packet. The packetizer 132 supplies the generated RTP packet to the buffer 133.

  More specifically, the packetizer 132 converts the hierarchically encoded image data corresponding to one layer, the information indicating the layer of the image data stored in the RTP packet, the maximum layer information, and the transmission layer information into one RTP. Each of the RTP packets corresponding to each layer is generated by storing in the packet. The packetizer 132 supplies each of the generated RTP packets corresponding to each layer to the buffer 133.

  Note that the packetizer 132 may store each of the image data of the layers from the first layer to the transmission layer in one RTP packet. In this case, the packetizer 132 stores the maximum layer information and the transmission layer information of the layer-encoded image data from the first layer to the transmission layer in one RTP packet, so that the RTP packet Is generated. The packetizer 132 supplies the generated RTP packet corresponding to each layer to the buffer 133.

  Under the control of the layer processing unit 139, the buffer 133 supplies each of RTP packets corresponding to layers from the first layer to the transmission layer among the RTP packets supplied from the packetizer 132 to the UDP communication unit 134. . The buffer 133 discards RTP packets that are not supplied to the UDP communication unit 134.

  The UDP communication unit 134 transmits various data via the communication network 13. The UDP communication unit 134 transmits and receives data according to UDP.

  The UDP communication unit 134 transmits the RTP packet supplied from the buffer 133 and the bandwidth measurement packet supplied from the transmission layer selection unit 136 to the client 14 via the communication network 13.

  The TCP communication unit 135 transmits and receives various data via the communication network 13. The TCP communication unit 135 transmits and receives data in accordance with TCP (Transmission Control Protocol), which is a protocol defined in IETF RFC793.

  The TCP communication unit 135 transmits the content layer table supplied from the content layer holding unit 137 to the client 14 via the communication network 13. Further, the TCP communication unit 135 receives the change request transmitted from the client 14 and supplies the received change request to the hierarchy processing unit 139. The TCP communication unit 135 receives the band measurement result transmitted from the client 14 and supplies the received band measurement result to the transmission layer selection unit 136.

  Note that the UDP communication unit 134 and the TCP communication unit 135 control the communication unit 89 to transmit and receive data via the communication unit 89.

  The transmission layer selection unit 136 generates a band measurement packet for measuring the band of the communication network 13 and supplies the generated band measurement packet to the UDP communication unit 134.

  The transmission layer selection unit 136 selects a transmission layer based on the band measurement result supplied from the TCP communication unit 135 and the content layer table recorded in the content layer holding unit 137, and generates transmission layer information. The transmission layer selection unit 136 supplies the generated transmission layer information to the transmission layer information holding unit 138.

  The content hierarchy holding unit 137 holds (records) a content hierarchy table. The content layer holding unit 137 supplies the content layer table to the encoder 131, the TCP communication unit 135, the transmission layer selection unit 136, and the layer processing unit 139.

  Further, when information indicating the number of hierarchized image data hierarchies and the required bandwidth of each hierarchy is supplied from the encoder 131, the content hierarchy holding unit 137 supplies the hierarchized image data hierarchies. The content hierarchy table is generated based on the number of and the information indicating the required bandwidth of each hierarchy, and the generated content hierarchy table is recorded.

  The transmission layer information holding unit 138 holds (stores) the transmission layer information supplied from the transmission layer selection unit 136 or the layer processing unit 139. The transmission layer information holding unit 138 supplies the stored transmission layer information to the layer processing unit 139.

  The hierarchy processing unit 139 generates hierarchy information based on the transmission hierarchy information supplied from the transmission hierarchy information holding unit 138 and the content hierarchy table supplied from the content hierarchy holding unit 137, and the generated hierarchy information is stored in the packetizer 132. Supply.

  The layer processing unit 139 controls the buffer 133 to supply the UDP communication unit 134 with RTP packets corresponding to the layers from the first layer to the transmission layer based on the generated layer information.

  The layer processing unit 139 generates transmission layer information based on the change request supplied from the TCP communication unit 135, and supplies the generated transmission layer information to the transmission layer information holding unit 138.

  FIG. 6 is a block diagram illustrating a functional configuration of the client 14.

  The client 14 includes an input unit 86, an output unit 87, a UDP communication unit 151, an RTP processing unit 152, a decoder 153, a synthesis unit 154, a stream information calculation unit 155, a hierarchy information holding unit 156, a bandwidth calculation unit 157, and a TCP communication unit 158. , A content layer holding unit 159, and a layer change request unit 160.

  The UDP communication unit 151 corresponds to the UDP communication unit 134 of the server 12 and receives various data via the communication network 13. The UDP communication unit 151 transmits and receives data according to UDP.

  The UDP communication unit 151 receives the bandwidth measurement packet transmitted from the server 12 and supplies the received bandwidth measurement packet to the bandwidth calculation unit 157.

  Further, the UDP communication unit 151 receives the RTP packet transmitted from the server 12 and supplies the received RTP packet to the RTP processing unit 152.

  The RTP processing unit 152 extracts image data from the RTP packet supplied from the UDP communication unit 151 and supplies the extracted image data to the decoder 153. Also, the RTP processing unit 152 extracts layer information from the RTP packet supplied from the UDP communication unit 151 and supplies the extracted layer information to the layer information holding unit 156.

  The RTP processing unit 152 extracts the sequence number from the RTP packet supplied from the UDP communication unit 151, and the extracted sequence number, the reception time of the RTP packet, and the packet size of the RTP packet (the size of the RTP packet) are stream information. It supplies to the calculation part 155.

  The decoder 153 decodes the image data supplied from the RTP processing unit 152 by a decoding method corresponding to the encoder 131, and supplies the decoded image data to the synthesis unit 154.

  More specifically, the decoder 153 decodes each image data of the layers from the first layer to the transmission layer supplied from the RTP processing unit 152 into one image data by a decoding method corresponding to the encoder 131, The decoded image data is supplied to the synthesis unit 154.

  The combining unit 154 combines (superimposes) the image data supplied from the decoder 153 and the image data for displaying the hierarchical information supplied from the hierarchical information holding unit 156, and supplies the superimposed image data to the output unit 87.

  The synthesizing unit 154 acquires image data for displaying the loss rate of the RTP packet and image data for displaying the reception band from the stream information calculation unit 155, and acquires image data for displaying the band measurement result from the band calculation unit 157. You may make it do.

  In this case, the synthesis unit 154 displays the image data supplied from the decoder 153, the image data for displaying the hierarchy information supplied from the hierarchy information holding unit 156, and the loss rate of the RTP packet acquired from the stream information calculation unit 155. The image data for displaying the image data and the reception band and the image data for displaying the band measurement result acquired from the band calculation unit 157 are superimposed, and the superimposed image data is supplied to the output unit 87.

  Further, the synthesis unit 154 may supply only the image data supplied from the decoder 153 to the output unit 87 without acquiring the image data for displaying the hierarchy information from the hierarchy information holding unit 156.

  The stream information calculation unit 155 calculates the loss rate of the RTP packet received by the UDP communication unit 151 based on the sequence number supplied from the RTP processing unit 152. Further, the stream information calculation unit 155 calculates the reception band of the communication network 13 based on the reception time of the RTP packet supplied from the RTP processing unit 152 and the packet size of the RTP packet.

  Here, the reception band refers to the band of the communication network 13 at the time when the UDP communication unit 151 receives the RTP packet. Details of the calculation of the reception band of the communication network 13 will be described later.

  The stream information calculation unit 155 generates image data for displaying the RTP packet loss rate based on the calculated RTP packet loss rate, and supplies the generated image data to the output unit 87. Further, the stream information calculation unit 155 generates image data for displaying the reception band of the communication network 13 based on the calculated reception band, and supplies the generated image data to the output unit 87. The stream information calculation unit 155 supplies the image data for displaying the loss rate of the generated RTP packet and the image data for displaying the reception band of the communication network 13 to the combining unit 154 without supplying to the output unit 87. You may do it.

  The hierarchy information holding unit 156 holds (stores) the hierarchy information supplied from the RTP processing unit 152. The hierarchy information holding unit 156 generates image data for displaying the hierarchy information based on the hierarchy information supplied from the RTP processing unit 152, and supplies the generated image data to the synthesis unit 154. The hierarchy information holding unit 156 supplies the hierarchy information supplied from the RTP processing unit 152 to the hierarchy change request unit 160.

  Note that the hierarchy information holding unit 156 may supply image data for displaying the generated hierarchy information to the output unit 87 without supplying the image data to the synthesis unit 154.

  The hierarchy information holding unit 156 is configured to include a transmission hierarchy information holding unit 181 and a maximum hierarchy information holding unit 182.

  The transmission layer information holding unit 181 of the layer information holding unit 156 stores the transmission layer information of the layer information supplied from the RTP processing unit 152. The maximum hierarchy information holding unit 182 of the hierarchy information holding unit 156 stores the maximum hierarchy information of the hierarchy information supplied from the RTP processing unit 152.

  The bandwidth calculation unit 157 calculates the bandwidth of the communication network 13 based on the bandwidth measurement packet supplied from the UDP communication unit 151.

  The band calculation unit 157 generates a band measurement result based on the calculated band, and supplies the generated band measurement result to the TCP communication unit 158 and the output unit 87.

  More specifically, the bandwidth calculation unit 157 generates a bandwidth measurement result based on the calculated bandwidth, and supplies the generated bandwidth measurement result to the TCP communication unit 158. Then, the band calculating unit 157 generates image data for displaying the generated band result, and supplies the generated image data to the output unit 87.

  Note that the band calculating unit 157 may supply the image data for displaying the generated band result to the synthesizing unit 154 without supplying the image data to the output unit 87.

  The TCP communication unit 158 corresponds to the TCP communication unit 135 of the server 12, and transmits and receives various data via the communication network 13. The TCP communication unit 158 transmits and receives data according to TCP.

  The TCP communication unit 158 receives the content layer table transmitted from the server 12 and supplies the received content layer table to the content layer holding unit 159.

  The TCP communication unit 158 transmits the band measurement result supplied from the band calculation unit 157 to the server 12 via the communication network 13. The TCP communication unit 158 transmits the change request supplied from the hierarchy change request unit 160 to the server 12 via the communication network 13.

  Note that the UDP communication unit 151 and the TCP communication unit 158 control a communication unit (not shown) and transmit / receive data via the communication unit.

  The content hierarchy holding unit 159 holds (stores) the content hierarchy table supplied from the TCP communication unit 158. The content hierarchy holding unit 159 generates image data for displaying the content hierarchy table based on the content hierarchy table supplied from the TCP communication unit 158, and supplies the generated image data to the output unit 87.

  The output unit 87 is supplied from the image data supplied from the combining unit 154, the image data that displays the loss rate of the RTP packet supplied from the stream information calculation unit 155, the image data that displays the reception band, and the band calculation unit 157. The image data for displaying the band measurement result and the image data for displaying the content layer table supplied from the content layer holding unit 159 are output to the output unit 87 which is a display, and the image, loss rate, reception band, and band measurement are output. Display results and content hierarchy table.

  When the user instructs to change the transmission hierarchy by operating the input unit 86, the input unit 86 generates a change signal and supplies the generated change signal to the hierarchy change request unit 160.

  More specifically, when the user instructs to change the transmission layer to a layer one level higher than the set layer by operating the input unit 86, the input unit 86 changes the transmission layer. A change signal for changing to the next higher hierarchy is generated, and the generated change signal is supplied to the hierarchy change request unit 160. In addition, when the user operates the input unit 86 to instruct the user to change the transmission layer to a layer one level lower than the set layer, the input unit 86 is one level lower than the transmission layer. A change signal for changing to the next layer is generated, and the generated change signal is supplied to the layer change request unit 160.

  Here, the hierarchy one level above the set hierarchy is, for example, the fourth hierarchy when the set hierarchy is the third hierarchy. Similarly, the hierarchy of the set hierarchy is For example, when the set hierarchy is the third hierarchy, the next lower hierarchy means the second hierarchy.

  Note that the user may instruct to change the transmission layer by inputting a desired transmission layer to the input unit 86. In this case, the input unit 86 generates information indicating the transmission layer input by the user, and supplies the generated information indicating the transmission layer to the layer change request unit 160.

  The hierarchy change request unit 160 generates a change request based on the hierarchy information supplied from the hierarchy information holding unit 156 and the change signal supplied from the input unit 86, and supplies the generated change request to the TCP communication unit 158. .

  More specifically, when a change signal for changing the transmission hierarchy to the next higher hierarchy is supplied from the input unit 86, the hierarchy change request unit 160 receives the hierarchy information supplied from the hierarchy information holding unit 156. A change request indicating that the layer one level higher than the transmission layer shown is to be a transmission layer is generated, and the generated change request is supplied to the TCP communication unit 158.

  Further, when a change signal for changing the transmission hierarchy to the next lower hierarchy is supplied from the input unit 86, the hierarchy change request unit 160 indicates the transmission hierarchy indicated by the hierarchy information supplied from the hierarchy information holding unit 156. A change request to make the layer one lower level of the transmission layer a transmission layer is generated, and the generated change request is supplied to the TCP communication unit 158.

  In addition, when the information indicating the transmission layer is supplied from the input unit 86, the layer change request unit 160 generates a change request indicating that the layer specified by the information indicating the transmission layer is the transmission layer, and the generated change The request is supplied to the TCP communication unit 158.

  Next, transmission layer change processing by the server 12 that executes the server program will be described with reference to the flowchart of FIG.

  In step S11, the transmission layer selection unit 136 performs band measurement processing. Although details of the bandwidth measurement processing will be described later, in the bandwidth measurement processing, the bandwidth of the communication network 13 between the server 12 and the client 14 is measured, and the transmission layer is selected based on the measured bandwidth. Is done.

  In step S <b> 12, the content layer holding unit 137 supplies the recorded content layer table to the TCP communication unit 135. More specifically, in step S12, the content layer holding unit 137 stores the recorded content layer table in a packet according to TCP, and supplies the packet storing the content layer table to the TCP communication unit 135.

  Here, the content hierarchy table is, for example, a table representing a hierarchy and a necessary bandwidth shown in FIG. The content hierarchy table includes a hierarchy in which the image data supplied from the video camera 11 is hierarchized, and a necessary band that is a band of the communication network 13 necessary for transmitting the hierarchized image data of each hierarchy. Is recorded.

  When encoding is performed based on the content hierarchy table shown in FIG. 8, the image data supplied from the video camera 11 is the encoded image data of the first hierarchy, which is the lowest hierarchy, and the first hierarchy. Encoded image data of the second hierarchy, which is the next higher hierarchy, encoded image data of the third hierarchy, which is the uppermost hierarchy of the second hierarchy, one of the third hierarchy It is hierarchized into encoded image data of the fourth hierarchy that is the upper hierarchy and encoded image data of the fifth hierarchy that is the highest hierarchy.

  In addition, in order to transmit the RTP packet in which the first layer image data is stored, the bandwidth of the communication network 13 is required to be 0.1 Mbps or more, and the RTP packet in which the second layer image data is stored. In order to transmit, the bandwidth of the communication network 13 is required to be 0.2 Mbps or more, and in order to transmit the RTP packet storing the image data of the third layer, the bandwidth of the communication network 13 is 0.3 Mbps or more. is necessary. Further, in order to transmit the RTP packet storing the image data of the fourth layer, the bandwidth of the communication network 13 is required to be 0.4 Mbps or more, and the RTP packet storing the image data of the fifth layer is required. In order to transmit, the bandwidth of the communication network 13 is required to be 0.5 Mbps or more.

  Returning to the description of FIG. 7, in step S <b> 13, the TCP communication unit 135 transmits the content layer table supplied from the content layer holding unit 137 to the client 14 via the communication network 13.

  In step S <b> 14, the hierarchy processing unit 139 determines whether the TCP communication unit 135 has received the change request transmitted from the client 14. If it is determined in step S14 that the change request has been received, the process proceeds to step S15, where the hierarchy processing unit 139 transmits the transmission hierarchy based on the change request received by the TCP communication unit 135 and supplied from the TCP communication unit 135. To change.

  More specifically, in step S15, the layer processing unit 139 generates transmission layer information based on the change request received by the TCP communication unit 135 and supplied from the TCP communication unit 135, and the generated transmission layer information is displayed. The data is supplied to the transmission layer information holding unit 138. Then, the transmission layer information holding unit 138 changes the transmission layer by storing the transmission layer information supplied from the layer processing unit 139.

  For example, when the change request supplied from the TCP communication unit 135 is a change request for changing the transmission layer to the fourth layer in step S15, the layer processing unit 139 determines that the transmission layer is the fourth layer. Is generated, and the generated transmission layer information is supplied to the transmission layer information holding unit 138. The transmission layer information holding unit 138 has the fourth transmission layer supplied from the layer processing unit 139. Transmission hierarchy information indicating the hierarchy is stored.

  On the other hand, if it is determined in step S14 that a change request has not been received, there is no need to change the transmission hierarchy, so the process of step S15 is skipped and the procedure proceeds to step S16.

  In step S16, each unit of the server 12 performs a data transmission process. Although details of the data transmission process will be described later, in the data transmission process, the image data supplied from the video camera 11 is hierarchically encoded, and each of the image data of each layer corresponds to each layer. Stored in each RTP packet. Each RTP packet corresponding to each layer is transmitted to the client 14 via the communication network 13.

  When the data transmission process ends, the procedure returns to step S14 and repeats the above-described process.

  In this way, the server 12 changes the transmission hierarchy based on the change request transmitted from the client 14.

  With reference to the flowchart of FIG. 9, the bandwidth measurement process corresponding to the process of step S11 of FIG. 7 will be described.

  In step S <b> 31, the transmission layer selection unit 136 generates a bandwidth measurement packet and supplies the generated bandwidth measurement packet to the UDP communication unit 134. For example, in step S <b> 31, the transmission layer selection unit 136 generates a packet pair that is a bandwidth measurement packet, and supplies the generated packet pair to the UDP communication unit 134.

  In step S 32, the UDP communication unit 134 transmits the bandwidth measurement packet supplied from the transmission layer selection unit 136 to the client 14 via the communication network 13.

  In step S32, the bandwidth measurement packet transmitted to the client 14 is received by the client 14. The client 14 that has received the bandwidth measurement packet generates a bandwidth measurement result based on the received bandwidth measurement packet, and transmits the generated bandwidth measurement result to the server 12 via the communication network 13.

  In step S33, the TCP communication unit 135 receives the band measurement result transmitted from the client 14, and supplies the received band measurement result to the transmission layer selection unit 136.

  In step S <b> 34, the transmission layer selection unit 136 selects a transmission layer based on the band measurement result supplied from the TCP communication unit 135 and the content layer table recorded in the content layer holding unit 137. Then, the transmission layer selection unit 136 generates transmission layer information based on the selected transmission layer, and supplies the generated transmission layer information to the transmission layer information holding unit 138.

  For example, in step S34, the transmission layer selection unit 136 records the content layer holding unit 137 when the bandwidth measurement result indicated by the bandwidth measurement result supplied from the TCP communication unit 135 is 0.35 Mbps. Referring to the content hierarchy table shown in FIG. 8, since 0.35 Mbps is larger than 0.3 Mbps and smaller than 0.4 Mbps, the third hierarchy is selected as the transmission hierarchy. Then, the transmission layer selection unit 136 generates transmission layer information indicating that the transmission layer is the third layer based on the selected transmission layer, and sends the generated transmission layer information to the transmission layer information holding unit 138. Supply.

  In step S35, the transmission layer information holding unit 138 holds (stores) the transmission layer information supplied from the transmission layer selection unit 136, and the process ends.

  In this way, the transmission layer selection unit 136 measures the bandwidth of the communication network 13.

  Next, a bandwidth calculation process by the client 14 that executes the client program will be described with reference to the flowchart of FIG.

  In step S <b> 51, the UDP communication unit 151 receives the bandwidth measurement packet transmitted from the server 12 and supplies the received bandwidth measurement packet to the bandwidth calculation unit 157.

  In step S52, the bandwidth calculation unit 157 calculates the bandwidth of the communication network 13 based on the bandwidth measurement packet supplied from the UDP communication unit 151. For example, in step S52, the bandwidth calculation unit 157 substitutes the reception time of the packet pair, which is the bandwidth measurement packet supplied from the UDP communication unit 151, and the packet size of one packet of the packet pair into the equation (1). Thus, the bandwidth of the communication network 13 is calculated.

  In step S53, the bandwidth calculation unit 157 generates a bandwidth measurement result indicating the bandwidth of the communication network 13 based on the calculated bandwidth of the communication network 13, and the generated bandwidth measurement result is transmitted to the TCP communication unit 158 and the output unit. 87.

  More specifically, in step S53, the bandwidth calculation unit 157 generates a bandwidth measurement result based on the calculated bandwidth of the communication network 13, and stores the generated bandwidth measurement result in a packet according to TCP. Then, the bandwidth calculation unit 157 supplies the packet storing the bandwidth measurement result to the TCP communication unit 158. Further, the band calculation unit 157 generates image data for displaying the band measurement result based on the generated band measurement result, and supplies the generated image data to the output unit 87.

  In step S54, the output unit 87 supplies the image data for displaying the band measurement result supplied from the band calculation unit 157 to the output unit 87 which is a display, and displays an image on the display based on the image data.

  In step S55, the TCP communication unit 158 transmits the band measurement result supplied from the band calculation unit 157 to the server 12 via the communication network 13, and the band calculation process ends.

  In this way, the client 14 receives the bandwidth measurement packet transmitted from the server 12, and calculates the bandwidth of the communication network 13 based on the received bandwidth measurement packet.

  A data transmission process corresponding to the process of step S16 of FIG. 7 will be described with reference to the flowchart of FIG.

  In step S71, the server 12 initializes data necessary for data transmission processing. For example, in step S71, the encoder 131 sets the value of the built-in timer to 0 ms, and the packetizer 132 initializes the time stamp and the sequence number.

  In step S72, the encoder 131 determines whether or not the timer has ended based on the value of the built-in timer. If it is determined that the timer has not ended, the encoder 131 returns to step S72 and the timer ends. The determination process is repeated until it is determined that the determination has been made.

  If it is determined in step S72 that the timer has expired, the process proceeds to step S73.

  For example, when the number of frames of image data is 30 per second, when the timer increases the value corresponding to the passage of time, in step S72, the encoder 131 sets a predetermined value such as 33 ms and the timer. By comparing with the value, it is determined whether or not the timer has expired.

  For example, in step S71, when the timer value is set to 33 ms and the timer value decreases with the passage of time, the encoder 131 sets the timer value and a predetermined value such as 0 ms. By comparing, it is determined whether or not the timer has expired. In this case, 0 ms or 33 ms compared with the timer value is an example in the case where the number of frames is 30 per second, and does not limit the present invention.

  In step S73, the encoder 131 captures one frame of image data supplied from the video camera 11. For example, in step S73, the encoder 131 sequentially acquires image data supplied from the video camera 11, and captures one frame of the acquired image data.

  In step S74, the encoder 131 refers to the content hierarchy table recorded in the content hierarchy holding unit 137 and hierarchically encodes the image data captured by a predetermined method for each frequency band. The encoder 131 supplies the hierarchically encoded image data to the packetizer 132.

  For example, in step S74, the encoder 131 performs hierarchical encoding by a method such as Motion JEPEG2000. In this case, the encoder 131 refers to the content layer table recorded in the content layer holding unit 137, and the sum of the bit rates of the image data from the first layer to each layer in the layer encoding is the content layer table. The layer coding is performed so as not to exceed the values of the necessary bandwidth of each layer recorded in (1). For example, the encoder 131 applies the wavelet transform to the captured image data, thereby hierarchically encoding the image data for each frequency band.

  Therefore, for example, when performing hierarchical encoding with reference to the content hierarchy table shown in FIG. 8 in step S74, the encoder 131 sets the bit rate of image data of the first hierarchy to 0.1 Mbps. Encode. The bit rate of the image data of the second layer is the same as the required bandwidth value of the second layer, with the sum of the bit rate of the image data of the first layer and the bit rate of the image data of the second layer. Is encoded so that That is, since the bit rate of the image data of the first layer is 0.1 Mbps and the necessary bandwidth of the second layer is 0.2 Mbps, the bit rate of the image data of the second layer is 0.1 Mbps. Encode.

  Similarly, the encoder 131 sets the third layer image data from the bit rate of the first layer image data to the bit rate of the third layer image data of the required bandwidth of the third layer. The fourth layer image data is encoded at 0.1 Mbps so as to be the same as the value, and the total of the bit rates of the first layer image data to the fourth layer image data is the fourth layer image data. The encoding is performed at 0.1 Mbps so as to be the same as the required bandwidth value of the first layer, and the fifth layer image data is converted from the first layer image data bit rate to the fifth layer image data bit rate. Is encoded at 0.1 Mbps so that the total of the required bandwidth is the same as the required bandwidth value of the fifth layer.

  The encoder 131 is not limited to Motion JEPEG2000, and may be hierarchically encoded by a scheme such as MPEG4. In this case, the encoder 131 hierarchically encodes the image data for each frequency band by applying discrete cosine transform to the captured image data. Then, the encoder 131 supplies information indicating the number of image data layers in encoding and the required bandwidth of each layer to the content layer holding unit 137. The content hierarchy holding unit 137 generates a content hierarchy table based on the supplied information indicating the number of hierarchized image data hierarchies and the required bandwidth of each hierarchy, and records the generated content hierarchy table.

  In step S75, the hierarchy processing unit 139 generates hierarchy information based on the content hierarchy table recorded in the content hierarchy holding unit 137 and the transmission hierarchy information supplied from the transmission hierarchy information holding unit 138, and the generated hierarchy Information is supplied to the packetizer 132.

  In step S76, the packetizer 132 generates an RTP packet corresponding to each layer based on the image data supplied from the encoder 131 and the layer information supplied from the layer processing unit 139.

  More specifically, in step S76, the packetizer 132 receives the image data of each layer, the transmission layer information, the maximum layer information, and information indicating the layer image data stored in the RTP packet supplied from the encoder 131. Each RTP packet corresponding to each layer is generated by storing in each RTP packet corresponding to the layer.

  Therefore, for example, when the transmission layer is the third layer and the maximum layer is the fifth layer, the packetizer 132 has the first layer image data, the transmission layer information, the maximum layer information, and the first layer. By storing information indicating that the image data of the layer is stored in the RTP packet in the RTP packet corresponding to the first layer, an RTP packet corresponding to the first layer is generated, and the second layer By storing information indicating that image data, transmission layer information, maximum layer information, and image data of the second layer are stored in the RTP packet in the RTP packet corresponding to the second layer, 2 An RTP packet corresponding to the third layer is generated, and the third layer image data, transmission layer information, maximum layer information, and third layer image data are stored in the RTP packet. It is by storing the RTP packet corresponding to the third hierarchical information showing that are to generate an RTP packet corresponding to the third hierarchy. Similarly, information indicating that the image data of the fourth layer, the transmission layer information, the maximum layer information, and the image data of the fourth layer are stored in the RTP packet is the RTP packet corresponding to the fourth layer. Is stored in the RTP packet corresponding to the fourth layer, and the image data of the fifth layer, the transmission layer information, the maximum layer information, and the image data of the fifth layer are stored in the RTP packet. Is stored in an RTP packet corresponding to the fifth layer, thereby generating an RTP packet corresponding to the fifth layer.

  In step S76, the packetizer 132 may store each of the image data of the layers from the first layer to the transmission layer in one RTP packet. In this case, the packetizer 132 generates an RTP packet by storing the maximum layer information and the transmission layer information of the image data of the layers from the first layer to the transmission layer in one RTP packet.

  The RTP packet is an example of a packet that stores image data to be transmitted to the client 14.

  FIG. 12 is a diagram for explaining an RTP packet. At the head of the RTP packet, 2-bit version information represented by “V” in FIG. 12 is arranged. The version information indicates the version of the RTP packet.

  Next to the version information, 1-bit padding represented by “P” in FIG. 12 is arranged, and following the padding, 1-bit extension information is arranged in the RTP packet. The extended information is represented by “X” in FIG. The extension information indicates the presence or absence of the extension header, and is set to a predetermined value when the extension header is arranged in the RTP packet.

  Following the extended information, a CSRC (Contributing Source) count is arranged in the RTP packet. The CSRC count is represented by “CC” in FIG. The CSRC count represents the number of CSRC identifiers.

  One-bit manufacturer information arranged following the CSRC count is defined by a profile. The manufacturer information is represented by “m” in FIG.

  The 7-bit payload type arranged after the manufacturer information is information for defining the format of the RTP packet. The payload type is represented by “PT” in FIG. In the RTP packet, the payload type is 33.

  The sequence number is 16-bit information arranged next to the payload type. The sequence number is a number indicating the order of reproduction of the RTP packets, and is incremented by 1 for each transmission. The sequence number is used to detect packet loss and repair the order of RTP packets.

  The 32-bit time stamp placed next to the sequence number is information indicating the time when the first octet of the streaming data stored in the RTP packet is sampled. Based on the time stamp, the RTP packet Control of processing time is executed at the time of development, and real-time image or audio reproduction control is performed. A common time stamp is set for RTP packets storing streaming data of the same frame.

  The SSRC (Synchronization Source) identifier is 32-bit information arranged next to the time stamp and indicates the source of streaming data stored in the RTP packet.

  In the RTP packet, next to the SSRC identifier, transmission layer information representing the transmission layer of streaming data stored in the RTP packet is arranged. The transmission layer information is represented by “transmission layer” in FIG. Next to the transmission layer information, the maximum layer information indicating the maximum layer of streaming data stored in the RTP packet is arranged. The maximum layer information is represented by “maximum layer” in FIG.

  Further, in the RTP packet, next to the maximum layer information, a layer of original data representing a layer of streaming data stored in the RTP packet is arranged. The hierarchy of original data is represented by “hierarchy of original data” in FIG.

  In the RTP packet, next to the original data layer, streaming data is stored. In FIG. 12, “Original Data” indicates streaming data.

  Returning to the description of the flowchart of FIG. 11, in step S77, the packetizer 132 supplies each of the generated RTP packets corresponding to each layer to the buffer 133. In step S78, the buffer 133 controls the layer processing unit 139. The RTP packet corresponding to each of the layers from the first layer to the transmission layer among the RTP packets corresponding to each layer supplied from the packetizer 132 is supplied to the UDP communication unit 134. The RTP packet not supplied to the UDP communication unit 134, that is, the RTP packet corresponding to the layer higher than the transmission layer is discarded.

  In step S79, the UDP communication unit 134 transmits the RTP packet supplied from the buffer 133 to the client 14 via the communication network 13, and in step S80, the packetizer 132 updates the time stamp added to the RTP packet. To do.

  In step S81, the encoder 131 sets a built-in timer, and the process ends. For example, in step S81, the encoder 131 sets the value of the built-in timer to 0 ms.

  In this way, the server 12 stores the image data in the RTP packet, and transmits the RTP packet in which the image data is stored to the client 14 via the communication network 13.

  With reference to the flowchart of FIG. 13, the content hierarchy table reception processing by the client 14 will be described.

  In step S101, the TCP communication unit 158 receives the content layer table transmitted from the server 12, and supplies the received content layer table to the content layer holding unit 159.

  In step S102, the content hierarchy holding unit 159 stores the content hierarchy table supplied from the TCP communication unit 158.

  In step S103, based on the content hierarchy table supplied from the TCP communication unit 158, image data for displaying the content hierarchy table is generated, and the image data for displaying the generated content hierarchy table is supplied to the output unit 87.

  In step S104, the output unit 87 displays the content layer table based on the image data for displaying the content layer table supplied from the content layer holding unit 159, and the content layer table reception process ends.

  For example, in step S104, the output unit 87 displays the content hierarchy table shown in FIG. 14 on the output unit 87, which is a display, based on the image data for displaying the content hierarchy table supplied from the content hierarchy holding unit 159. Let

  A window 201 is displayed on the output unit 87 which is a display. The window 201 is a content that represents a hierarchy in which the image data transmitted from the server 12 is hierarchized and a necessary bandwidth of the communication network 13 that is required to transmit the image data of each hierarchy. A hierarchy table is displayed.

  Further, when the user operates the input unit 86 to move a pointer (not shown) displayed on the output unit 87 that is a display onto the button 202, when the button 202 is clicked, the window 201 is clicked. Is closed. Here, clicking means an operation of pressing and releasing the left button of the mouse.

  In this way, the client 14 receives the content hierarchy table transmitted from the server 12 and displays the received content hierarchy table.

  Next, with reference to FIG. 15, processing for transmitting a change request by the client 14 will be described.

  In step S 121, the UDP communication unit 151 receives the RTP packet transmitted from the server 12 and supplies the received RTP packet to the RTP processing unit 152.

  In step S122, the RTP processing unit 152 extracts layer information from the RTP packet supplied from the UDP communication unit 151, and supplies the extracted layer information to the layer information holding unit 156.

  More specifically, in step S122, the RTP processing unit 152 extracts transmission layer information and maximum layer information from the RTP packet supplied from the UDP communication unit 151, and extracts the extracted transmission layer information and maximum layer information from the layer. The information is supplied to the hierarchy information holding unit 156 as information. Then, the RTP processing unit 152 extracts image data from the RTP packet supplied from the UDP communication unit 151, and supplies the extracted image data to the decoder 153. Further, the RTP processing unit 152 extracts the sequence number from the RTP packet supplied from the UDP communication unit 151, and displays the extracted sequence number, the reception time of the RTP packet, and the packet size of the received RTP packet as a stream. It supplies to the information calculation part 155.

  In step S123, the hierarchy information holding unit 156 stores the hierarchy information supplied from the RTP processing unit 152. More specifically, in step S123, the transmission layer information holding unit 181 of the layer information holding unit 156 stores the transmission layer information that is the layer information supplied from the RTP processing unit 152, and the maximum of the layer information holding unit 156 is stored. The hierarchy information holding unit 182 stores the maximum hierarchy information that is the hierarchy information supplied from the RTP processing unit.

  In step S124, the hierarchy change request unit 160 determines whether or not a change signal is supplied from the input unit 86.

  The change signal is generated by the input unit 86 when the user instructs the change of the transmission hierarchy by operating the input unit 86, and the generated change signal is supplied to the hierarchy change request unit 160. . The user refers to the received image, content layer table, reception band, RTP packet loss rate, band, and transmission layer displayed on the output unit 87, which is a display, and instructs transmission layer change as necessary. To do.

  If it is determined in step S124 that a change signal has been supplied from the input unit 86, the process advances to step S125, and the hierarchy change request unit 160 acquires the hierarchy information from the hierarchy information holding unit 156.

  In step S126, the hierarchy change request unit 160 generates a change request based on the change signal supplied from the input unit 86 and the hierarchy information acquired from the hierarchy information holding unit 156, and the generated change request is transmitted to the TCP communication unit 158. To supply.

  More specifically, in step S126, the hierarchy change request unit 160 obtains from the hierarchy information holding unit 156 when a change signal for changing the transmission hierarchy to the next higher hierarchy is supplied from the input unit 86. Based on the transmission layer information that is the layer information, a change request for changing the transmission layer to a layer one level higher than the transmission layer indicated by the transmission layer information is generated, and the generated change request is stored in a packet according to TCP. To do. Then, the hierarchy change request unit 160 supplies the packet storing the change request to the TCP communication unit 158.

  Therefore, for example, in step S126, the hierarchy change request unit 160 is supplied with a change signal indicating that the transmission hierarchy is changed to the next higher hierarchy from the input unit 86, and the hierarchy information acquired from the hierarchy information holding unit 156 is used. If certain transmission layer information is transmission layer information indicating that the transmission layer is the fourth layer, a change request for changing the transmission layer to the fifth layer is generated, and the generated change request is sent to the TCP. Stored in a packet conforming to Then, the hierarchy change request unit 160 supplies the packet storing the change request to the TCP communication unit 158.

  In step S126, the hierarchy change request unit 160 receives the hierarchy information acquired from the hierarchy information holding unit 156 when the change signal indicating that the transmission hierarchy is changed to the next lower hierarchy is supplied from the input unit 86. Based on the transmission layer information, a change request for changing the transmission layer to a layer one level lower than the transmission layer indicated by the transmission layer information is generated, and the generated change request is stored in a packet according to TCP. Then, the hierarchy change request unit 160 supplies the packet storing the change request to the TCP communication unit 158.

  Therefore, for example, in step S126, the hierarchy change request unit 160 is the hierarchy information acquired from the hierarchy information holding unit 156 when the change signal indicating that the transmission hierarchy is changed to the next lower hierarchy is supplied from the input unit 86. If the transmission layer information is transmission layer information indicating that the transmission layer is the fourth layer, a change request for changing the transmission layer to the third layer is generated, and the generated change request conforms to TCP. Store in packet. Then, the hierarchy change request unit 160 supplies the packet storing the change request to the TCP communication unit 158.

  In step S127, the TCP communication unit 158 transmits the change request supplied from the hierarchy change request unit 160 to the server 12 via the communication network 13, and proceeds to step S128.

  On the other hand, if it is determined in step S124 that the change signal has not been supplied from the input unit 86, it is not necessary to send a change request to the server 12, so the processing from step S125 to step S127 is skipped, and the procedure Proceed to S128.

  In step S128, the stream information calculation unit 155 calculates a reception band based on the reception time and packet size of the RTP packet supplied from the RTP processing unit 152, and displays the reception band based on the calculated reception band. Generate image data. Then, the stream information calculation unit 155 supplies image data for displaying the generated reception band to the output unit 87.

  For example, in step S128, the stream information calculation unit 155 substitutes the sum of the packet sizes of the RTP packets constituting one frame supplied from the RTP processing unit 152 and the time when the RTP packet is received into Expression (2). Thus, the reception band is calculated.

  (Reception band) = S2 / (t42-t41) (2)

  Here, S2 is the total packet size of the RTP packets constituting one received frame. Also, t41 is the reception time of the first RTP packet received among the RTP packets constituting one frame, and t42 is the first among the RTP packets constituting the next one frame. This is the reception time of the RTP packet received at.

  Therefore, for example, in the example of FIG. 2, when the reception bandwidth is calculated based on the packets 42-1 to 42-3 constituting the received frame 32, the total packet size S2 of the RTP packet is the packet 42. -1 to packet 42-3, and the reception time t41 of the first packet received among the packets constituting the frame 32 is the time when the client 14 received the packet 42-1. Of the packets constituting the next frame 33, the reception time t42 of the first packet received is the time when the client 14 received the packet 43-1.

  In step S129, the stream information calculation unit 155 calculates the loss rate of the RTP packet based on the sequence number supplied from the RTP processing unit 152, and generates image data for displaying the loss rate based on the calculated loss rate. Generate. Then, the stream information calculation unit 155 supplies image data for displaying the generated loss rate to the output unit 87.

  In step S <b> 130, the decoder 153 decodes the image data supplied from the RTP processing unit 153 by a method corresponding to the encoding method of the encoder 131. The decoder 153 supplies the decoded image data to the synthesis unit 154.

  For example, in step S130, when each of the image data of the third hierarchy from the first hierarchy is supplied from the RTP processing unit 152, the decoder 153 receives the image of the third hierarchy from the supplied first hierarchy. Each of the data is decoded into one image data by a method corresponding to the encoding method of the encoder 131, and the decoded image data is supplied to the synthesis unit 154.

  In step S131, the synthesis unit 154 superimposes the image data supplied from the decoder 153 and the data for displaying the hierarchy information supplied from the hierarchy information holding unit 156, and supplies the superimposed image data to the output unit 87.

  In step S132, the output unit 87 causes the output unit 87, which is a display, to display an image based on the image data supplied from the combining unit 154. For example, in step S132, the output unit 87 causes the output unit 87, which is a display, to display the image shown in FIG. 16 based on the image data supplied from the combining unit 154.

  A window 231 is displayed on the output unit 87 which is a display, and the received image 241 is displayed on the window 231. The window 231 displays the characters “reception band”, “loss rate”, “band”, and “transmission layer” representing the reception band, RTP packet loss rate, band, and transmission layer, respectively. Each value is displayed to the right of each displayed character.

  For example, the reception band value “0.4 Mbps” is displayed on the right side of the character “reception band” indicating the reception band, and the loss rate value “1%” is displayed on the right side of the character “loss rate” indicating the loss rate of the RTP packet. ”Is displayed, the band value“ 0.3 Mbps ”is displayed to the right of the character“ band ”representing the band, the maximum layer is the fifth layer to the right of the character“ transmission layer ”representing the transmission layer, “3/5” is displayed indicating that the transmission layer is the third layer.

  In addition, a number line representing the maximum hierarchy and the transmission hierarchy is displayed under the characters “transmission hierarchy” representing the transmission hierarchy. In the example of FIG. 16, the square displayed on the scale represented by “3” on the number line indicates that the transmission layer is the third layer.

  Furthermore, when the user moves a pointer (not shown) by operating the input unit 86 and the button 232 is clicked, the window 231 is closed.

  More specifically, in step S132, the output unit 87 outputs the image data supplied from the synthesizing unit 154, the image data supplied from the stream information calculating unit 155, the image data for displaying the loss rate, and the image data for displaying the reception band, In addition, the image data for displaying the band measurement result supplied from the band calculating unit 157 is superimposed, and the image is displayed on the output unit 87 which is a display based on the superimposed image data. In addition, when new image data is supplied from the synthesis unit 154, the stream information calculation unit 155, or the band calculation unit 157, the output unit 87 displays the image data of the displayed image and the newly supplied image data. And the image is displayed on the output unit 87 which is a display based on the superimposed image data.

  Therefore, for example, when new image data for displaying the band measurement result is supplied from the band calculation unit 157, the output unit 87 discards the image data for displaying the band measurement result that has been displayed so far, and performs synthesis. The image data supplied from the unit 154, the image data supplied from the stream information calculation unit 155 to display the loss rate and the image data to display the reception band, and the band measurement result newly supplied from the band calculation unit 157 Is superimposed on the image data, and an image is displayed on the output unit 87, which is a display, based on the superimposed image data.

  In step S133, the output unit 87 causes the output unit 87, which is a display, to display the reception band of the communication network 13 based on the image data that displays the reception band of the communication network 13 supplied from the stream information calculation unit 155. .

  In step S134, the output unit 87 causes the output unit 87, which is a display, to display the loss rate of the RTP packet based on the image data that displays the loss rate of the RTP packet supplied from the stream information calculation unit 155. Returning to S121, the above-described processing is repeated.

  In this way, the client 14 generates a change request and transmits the generated change request to the server 12 via the communication network 13. Further, since the client 14 displays the loss rate, the hierarchy information, and the reception band, the user can know whether or not the set transmission hierarchy is appropriate for the band of the communication network 13. it can.

  As described above, the client 14 can display the content hierarchy table transmitted from the server 12.

  Next, a second embodiment of the present invention will be described.

  An example of bandwidth measurement processing and transmission layer change processing in the server 12 will be described with reference to the time chart of FIG.

  In FIG. 17, the horizontal axis indicates time, and the bandwidth measurement packet 261 includes, for example, a packet 271-1 and a packet 271-2 having the same packet size for measuring the bandwidth of the communication network 13. Further, for example, streaming data that is image data for reproducing the frame 262 is stored and transmitted in the packets 272-1 to 272-3, and streaming data that is image data for reproducing the frame 263 is: Streaming data, which is image data for reproducing the frame 264, is stored and transmitted in the packets 273-1 to 273-3, and is stored and transmitted in the packets 274-1 to 274-4.

  At time t51, the server 12 transmits a packet 271-1 for measuring the bandwidth of the communication network 13 to the client 14 via the communication network 13. Similarly, at time t <b> 52, the server 12 transmits a packet 271-2 for measuring the bandwidth of the communication network 13 to the client 14 via the communication network 13. The packet 271-1 and the packet 271-2 are transmitted to the client 14 continuously (continuously) without a time interval.

  At time t61, the client 14 receives the packet 271-1 transmitted from the server 12, and at time t62, the client 14 receives the packet 271-2 transmitted from the server 12. Upon receiving the packet 271-1 and the packet 271-2, the client 14 calculates a bandwidth in the communication network 13 based on the received packet 271-1 and the packet 271-2.

  At time t <b> 63, the client 14 transmits a band measurement result that is a calculated band measurement result to the server 12 via the communication network 13. For example, the client 14 stores the bandwidth measurement result in a packet, and transmits the packet storing the bandwidth measurement result to the server 12 via the communication network 13.

  At time t <b> 53, the server 12 receives the bandwidth measurement result transmitted from the client 14. Upon receiving the band measurement result, the server 12 selects a transmission layer based on the received band measurement result, and at time t54, the layer information including the transmission layer information and the maximum layer information is transmitted via the communication network 13. It is sent to the client 14. For example, the server 12 stores the hierarchical information in a packet, and transmits the packet storing the hierarchical information to the client 14 via the communication network 13.

  After transmitting the hierarchy information to the client 14, the server 12 transmits the packets 272-1 to 272-3 to the client 14 via the communication network 13 based on the selected transmission hierarchy. Similarly, the server 12 transmits the packets 273-1 to 273-3 to the client 14 via the communication network 13.

  The client 14 receives the hierarchy information, the packets 272-1 to 272-3, and the packets 273-1 to 273-3 transmitted from the server 12.

  At time t <b> 64, the client 14 transmits a change request to the server 12 via the communication network 13. For example, the client 14 stores the change request in a packet, and transmits the packet storing the change request to the server 12 via the communication network 13.

  The server 12 receives the change request transmitted from the client 14. The server 12 changes the transmission layer based on the received change request, and generates layer information based on the changed transmission layer.

  At time t55, the server 12 transmits the generated hierarchy information to the client 14 via the communication network 13. After transmitting the hierarchy information, the server 12 transmits packets 274-1 to 274-4 to the client 14 via the communication network 13 based on the changed transmission hierarchy.

  The client 14 receives the hierarchy information and the packets 274-1 to 274-4 transmitted from the server 12.

  In this way, the server 12 selects a transmission layer according to the band, and transmits a packet storing image data to the client 14 based on the selected transmission layer.

  FIG. 18 is a block diagram illustrating a functional configuration of the server 12.

  The server 12 includes an encoder 291, a packetizer 292, a buffer 293, a UDP communication unit 294, a TCP communication unit 295, a transmission layer selection unit 296, a content layer holding unit 297, a transmission layer information holding unit 298, and a layer processing unit 299. Configured.

  Note that the encoder 291, the buffer 293, the UDP communication unit 294, the transmission layer selection unit 296, the content layer holding unit 297, and the transmission layer information holding unit 298 are respectively the encoder 131, the buffer 133, the UDP communication unit 134 in FIG. Since it is the same as each of the transmission hierarchy selection part 136, the content hierarchy holding | maintenance part 137, and the transmission hierarchy information holding part 138, the description is abbreviate | omitted.

  The packetizer 292 generates an RTP packet by storing the image data supplied from the encoder 291 in an RTP packet. The packetizer 292 supplies the generated RTP packet to the buffer 293.

  More specifically, the packetizer 292 generates each RTP packet corresponding to each layer by storing the image data corresponding to one layer, which is layer-encoded, in one RTP packet. The packetizer 292 supplies each of the generated RTP packets corresponding to each layer to the buffer 293.

  The TCP communication unit 295 transmits and receives various data via the communication network 13. The TCP communication unit 295 transmits and receives data according to TCP.

  The TCP communication unit 295 transmits the content layer table supplied from the content layer holding unit 297 to the client 14 via the communication network 13. The TCP communication unit 295 transmits the hierarchy information supplied from the hierarchy processing unit 299 to the client 14 via the communication network 13. Also, the TCP communication unit 295 receives the change request transmitted from the client 14 and supplies the received change request to the hierarchy processing unit 299. The TCP communication unit 295 receives the band measurement result transmitted from the client 14 and supplies the received band measurement result to the transmission layer selection unit 296.

  Note that the UDP communication unit 294 and the TCP communication unit 295 control the communication unit (not shown), and transmit and receive data via the communication unit.

  The hierarchy processing unit 299 generates hierarchy information based on the transmission hierarchy information supplied from the transmission hierarchy information holding unit 298 and the content hierarchy table supplied from the content hierarchy holding unit 297, and the generated hierarchy information is transmitted to the TCP communication unit. 295.

  The layer processing unit 299 controls the buffer 293 to supply the UDP communication unit 294 with each of the RTP packets corresponding to each layer from the first layer to the transmission layer based on the generated layer information.

  The layer processing unit 299 generates transmission layer information based on the change request supplied from the TCP communication unit 295, and supplies the generated transmission layer information to the transmission layer information holding unit 298.

  FIG. 19 is a block diagram illustrating a functional configuration of the client 14.

  The client 14 includes a UDP communication unit 311, an RTP processing unit 312, a decoder 313, a synthesis unit 314, a stream information calculation unit 315, a hierarchy information holding unit 316, a bandwidth calculation unit 317, a TCP communication unit 318, a content hierarchy holding unit 319, an output A unit 320, an input unit 321, and a hierarchy change request unit 322 are configured.

  Note that the UDP communication unit 311, decoder 313, synthesis unit 314, stream information calculation unit 315, band calculation unit 317, content layer holding unit 319, output unit 320, input unit 321, and layer change request unit 322 are shown in FIG. 6 is the same as the UDP communication unit 151, decoder 153, synthesis unit 154, stream information calculation unit 155, bandwidth calculation unit 157, content layer holding unit 159, output unit 87, input unit 86, and layer change request unit 160 in FIG. The description is omitted.

  The RTP processing unit 312 extracts image data from the RTP packet supplied from the UDP communication unit 311 and supplies the extracted image data to the decoder 313. Further, the RTP processing unit 312 extracts a sequence number from the RTP packet supplied from the UDP communication unit 311, and supplies the extracted sequence number, RTP packet reception time, and packet size of the RTP packet to the stream information calculation unit 315. To do.

  The hierarchy information holding unit 316 holds (stores) the hierarchy information supplied from the TCP communication unit 318. The hierarchy information holding unit 316 generates image data for displaying the hierarchy information based on the hierarchy information supplied from the TCP communication unit 318, and supplies the generated image data to the synthesis unit 314. The hierarchy information holding unit 316 supplies the hierarchy information supplied from the TCP communication unit 318 to the hierarchy change request unit 322.

  Note that the hierarchy information holding unit 316 may supply the image data for displaying the generated hierarchy information to the output unit 320 without supplying the image data to the synthesis unit 314.

  The hierarchy information holding unit 316 is configured to include a transmission hierarchy information holding unit 331 and a maximum hierarchy information holding unit 332.

  The transmission layer information holding unit 331 of the layer information holding unit 316 stores the transmission layer information of the layer information supplied from the TCP communication unit 318. The maximum hierarchy information holding unit 332 of the hierarchy information holding unit 316 stores the maximum hierarchy information of the hierarchy information supplied from the TCP communication unit 318.

  The TCP communication unit 318 corresponds to the TCP communication unit 295 of the server 12 and transmits / receives various data via the communication network 13. The TCP communication unit 318 transmits and receives data according to TCP.

  The TCP communication unit 318 receives the content layer table transmitted from the server 12 and supplies the received content layer table to the content layer holding unit 319. In addition, the TCP communication unit 318 receives the hierarchy information transmitted from the server 12 and supplies the received hierarchy information to the hierarchy information holding unit 316.

  The TCP communication unit 318 transmits the band measurement result supplied from the band calculation unit 317 to the server 12 via the communication network 13. The TCP communication unit 318 transmits the change request supplied from the hierarchy change request unit 322 to the server 12 via the communication network 13.

  Note that the UDP communication unit 311 and the TCP communication unit 318 control a communication unit (not shown) and perform data transmission / reception via the communication unit.

  Next, transmission layer change processing by the server 12 that executes the server program will be described with reference to the flowchart of FIG.

  In step S151, the transmission layer selection unit 296 executes bandwidth measurement processing. The bandwidth measurement process in step S151 is the same as the process described with reference to the flowchart of FIG. In the band measurement process, a transmission layer is selected.

  In step S152, the hierarchy processing unit 299 generates hierarchy information based on the content hierarchy table supplied from the content hierarchy holding unit 297 and the transmission hierarchy information supplied from the transmission hierarchy information holding unit 298, and the generated hierarchy information Is supplied to the TCP communication unit 295.

  More specifically, in step S152, the hierarchy processing unit 299 generates hierarchy information based on the content hierarchy table supplied from the content hierarchy holding unit 297 and the transmission hierarchy information supplied from the transmission hierarchy information holding unit 298. The generated hierarchy information is stored in a packet according to TCP. Then, the layer processing unit 299 supplies the packet storing the layer information to the TCP communication unit 295.

  In step S153, the TCP communication unit 295 transmits the hierarchy information supplied from the hierarchy processing unit 299 to the client 14 via the communication network 13, and proceeds to step S154.

  Since the processes of steps S154 to S157 are the same as the processes of steps S12 to S15 in FIG. 7, description thereof is omitted. If it is determined in step S156 that the TCP communication unit 295 has received a change request from the client 14, the process proceeds to step S157, and if it is determined that a change request has not been received, the transmission hierarchy needs to be changed. Therefore, the process from step S157 to step S159 is skipped, and the procedure proceeds to step S160.

  In step S158, the hierarchy processing unit 299 generates hierarchy information based on the content hierarchy table supplied from the content hierarchy holding unit 297 and the transmission hierarchy information supplied from the transmission hierarchy information holding unit 298, and the generated hierarchy information Is supplied to the TCP communication unit 295.

  More specifically, in step S158, the hierarchy processing unit 299 generates hierarchy information based on the content hierarchy table supplied from the content hierarchy holding unit 297 and the transmission hierarchy information supplied from the transmission hierarchy information holding unit 298. The generated hierarchy information is stored in a packet according to TCP. Then, the layer processing unit 299 supplies the packet storing the layer information to the TCP communication unit 295.

  In step S159, the TCP communication unit 295 transmits the hierarchy information supplied from the hierarchy processing unit 299 to the client 14 via the communication network 13, and proceeds to step S160.

  In step S160, each unit of the server 12 performs data transmission processing. Although details of the data transmission processing will be described later, in the data transmission processing, the image data supplied from the video camera 11 is hierarchically encoded, and the image data of each layer is an RTP packet corresponding to each layer. Stored in Each RTP packet corresponding to each layer is transmitted to the client 14 via the communication network 13.

  When the data transmission process ends, the procedure returns to step S156 to repeat the above-described process.

  In this way, the server 12 changes the transmission hierarchy based on the change request transmitted from the client 14.

  Next, a data transmission process corresponding to the process of step S160 of FIG. 20 will be described with reference to the flowchart of FIG.

  Since each of the processing of step S181 to step S184 is the same as each of the processing of step S71 to step S74 in FIG. 11, the description thereof is omitted.

  In step S185, the packetizer 293 generates an RTP packet corresponding to each layer shown in FIG. 22 based on the image data supplied from the encoder 291 and proceeds to step S186.

  FIG. 22 is a diagram for explaining an RTP packet. Since version information, padding, extended information, CSRC count, manufacturer information, payload type, sequence number, time stamp, SSRC identifier, and original data are the same as in the case of the RTP packet shown in FIG. Omitted.

  More specifically, in step S185, the packetizer 292 stores each of the image data corresponding to each layer supplied from the encoder 291 in each RTP packet corresponding to each layer, thereby corresponding to each layer. Generate each RTP packet.

  Therefore, for example, when the maximum hierarchy is the fifth hierarchy, the packetizer 292 generates an RTP packet corresponding to the first hierarchy by storing the image data of the first hierarchy in the RTP packet, and 2 By storing the image data of the third layer in the RTP packet, the RTP packet corresponding to the second layer is generated, and by storing the image data of the third layer in the RTP packet, it corresponds to the third layer. RTP packet to be generated. Similarly, the fourth layer image data is stored in the RTP packet to generate an RTP packet corresponding to the fourth layer, and the fifth layer image data is stored in the RTP packet. RTP packet corresponding to the hierarchy of is generated.

  The RTP packet is an example of a packet that stores image data to be transmitted to the client 14.

  Since each of the processing from step S186 to step S190 is the same as each of the processing from step S77 to step S81 in FIG. 11, the description thereof is omitted.

  In this way, the server 12 stores the image data in the RTP packet, and transmits the RTP packet in which the image data is stored to the client 14 via the communication network 13.

  Next, with reference to FIG. 23, processing for transmitting a change request by the client 14 will be described.

  In step S211, the UDP communication unit 311 receives the RTP packet transmitted from the server 12, and supplies the received RTP packet to the RTP processing unit 312.

  In step S212, the hierarchy information holding unit 316 performs a hierarchy information reception process, and proceeds to step S213. Although details of the hierarchy information reception process will be described later, the received hierarchy information is stored in the hierarchy information holding unit 316 in the hierarchy information reception process.

  Since each of the processing from step S213 to step S216 is the same as each of the processing from step S124 to step S127 in FIG. 15, the description thereof is omitted.

  In step S217, the hierarchy information holding unit 316 performs hierarchy information reception processing, and proceeds to step S218.

  Since the processes in steps S218 to S224 are the same as those in steps S128 to S134 in FIG. 15, the description thereof is omitted.

  In step S222, the output unit 320 may display the image shown in FIG. 24 on the output unit 320 that is a display.

  In this case, a window 231 is displayed on the output unit 320 that is a display, and the received image 241 is displayed on the window 231. In another window 351 displayed at the same time as the window 231, the characters “reception band”, “loss ratio”, “band”, which indicate the reception band, the loss rate of the RTP packet, the band, and the transmission layer, And “transmission hierarchy” are displayed, and the respective values are displayed to the right of the displayed characters.

  For example, the reception band value “0.4 Mbps” is displayed on the right side of the character “reception band” indicating the reception band, and the loss rate value “1%” is displayed on the right side of the character “loss rate” indicating the loss rate of the RTP packet. ”Is displayed, the band value“ 0.3 Mbps ”is displayed to the right of the character“ band ”representing the band, the maximum layer is the fifth layer to the right of the character“ transmission layer ”representing the transmission layer, “3/5” is displayed indicating that the transmission layer is the third layer.

  In addition, a number line representing the maximum hierarchy and the transmission hierarchy is displayed under the characters “transmission hierarchy” representing the transmission hierarchy. In the example of FIG. 24, the square displayed on the scale represented by “3” on the number line indicates that the transmission layer is the third layer.

  Furthermore, when the user moves a pointer (not shown) by operating the input unit 321 and the button 232 is clicked, the window 231 is closed. Similarly, when the user moves the pointer (not shown) by operating the input unit 321 and the button 352 is clicked, the window 351 is closed.

  In this way, the client 14 generates a change request and transmits the generated change request to the server 12 via the communication network 13.

  Next, with reference to the flowchart of FIG. 25, the hierarchy information reception process corresponding to the processes of steps S212 and S217 of FIG. 23 will be described.

  In step S <b> 241, the hierarchy information holding unit 316 determines whether the TCP communication unit 318 has received the hierarchy information transmitted from the server 12.

  If it is determined in step S241 that the TCP communication unit 318 has received the hierarchy information, the process proceeds to step S242, and the hierarchy information holding unit 316 acquires the hierarchy information from the TCP communication unit 318 and stores the acquired hierarchy information. To do.

  More specifically, in step S242, the layer information holding unit 316 acquires layer information from the TCP communication unit 318, and the transmission layer information holding unit 331 of the layer information holding unit 316 is a transmission layer that is the acquired layer information. Store information. Further, the maximum hierarchy information holding unit 332 of the hierarchy information holding unit 316 stores the maximum hierarchy information that is the acquired hierarchy information, and the process ends.

  On the other hand, if it is determined in step S241 that the TCP communication unit 318 has not received the hierarchy information, the process in step S242 is skipped, and the process ends.

  In this way, the hierarchy information holding unit 316 stores the received hierarchy information.

  As described above, the client 14 can display the content hierarchy table transmitted from the server 12.

  According to the first aspect of the present invention, since the data transmitted from the transmission device is received, the received data can be output (displayed). In addition, according to the first aspect of the present invention, the content hierarchy table transmitted from the transmission device is displayed. Therefore, on the data receiving side, the bandwidth of the communication network required for transmission of the hierarchically encoded data is reduced. I can grasp it.

  According to the second invention, since data is stored in a packet, data can be transmitted. According to the second invention, since the content layer table is transmitted, it is possible to notify the data receiving side of the bandwidth of the communication network required for transmitting the data to be hierarchically encoded.

  Although it has been explained that the bandwidth is measured by sending a packet pair, it may be measured by the packet train method, etc. In addition, Diffserv (Differentiated Services) and SNMP (Simple Network Management Protocol) are used. You may make it measure by doing.

  Further, although it has been described that an image is displayed on the output unit 87 of the client 14, an output device that displays an image may be provided separately from the client 14.

  The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer.

  As shown in FIG. 4, the recording medium is distributed to provide a program to the user separately from the computer, and includes a magnetic disk 111 (including a flexible disk) on which the program is recorded, an optical disk 112 (CD- ROM (Compact Disc-Read Only Memory), DVD (including Digital Versatile Disc)), magneto-optical disk 113 (including MD (Mini-Disc) (trademark)), or a package medium composed of semiconductor memory 114, etc. In addition, it is configured by a ROM 82 on which a program is recorded and a hard disk included in the recording unit 88, which are provided to the user in a state of being incorporated in advance in the computer.

  The program for executing the series of processes described above is installed in a computer via a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via an interface such as a router or a modem as necessary. You may be made to do.

  Further, in the present specification, the step of describing the program stored in the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

  In the present specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

It is a figure which shows one Embodiment of the streaming delivery system to which this invention is applied. It is a time chart explaining a streaming data delivery system. It is a figure explaining the measurement of a zone | band. It is a block diagram which shows the example of a structure of a server. It is a block diagram which shows the structure of the function of a server. It is a block diagram which shows the structure of the function of a client. It is a flowchart explaining the process of a change of a transmission hierarchy. It is a figure explaining a content hierarchy table. It is a flowchart explaining the process of a band measurement. It is a flowchart explaining the process of a zone | band calculation. It is a flowchart explaining the process of data transmission. It is a figure explaining an RTP packet. It is a flowchart explaining a content hierarchy table reception process. It is a figure explaining the content hierarchy table displayed on an output part. It is a flowchart explaining the process of transmission of a change request. It is a figure explaining the image displayed on an output part. It is a time chart explaining a streaming data delivery system. It is a block diagram which shows the structure of the function of a server. It is a block diagram which shows the structure of the function of a client. It is a flowchart explaining the process of a change of a transmission hierarchy. It is a flowchart explaining the process of data transmission. It is a figure explaining an RTP packet. It is a flowchart explaining the process of transmission of a change request. It is a figure explaining the image displayed on an output part. It is a flowchart explaining the process of hierarchy information reception.

Explanation of symbols

  12 server, 13 communication network, 14 client, 81 CPU, 82 ROM, 83 RAM, 86 input unit, 87 output unit, 88 recording unit, 111 magnetic disk, 112 optical disk, 113 magneto-optical disk, 114 semiconductor memory 131 encoder, 132 Packetizer, 136 transmission layer selection unit, 137 content layer holding unit, 138 transmission layer holding unit, 139 layer processing unit, 152 RTP processing unit, 153 decoder, 155 stream information calculation unit, 156 layer information holding unit, 157 band calculation unit, 159 content layer holding unit, 160 layer change request unit, 181 transmission layer information holding unit, 182 maximum layer information holding unit, 296 transmission layer selection unit, 298 transmission layer information holding unit, 299 layer processing unit 312 RTP processor, 316 hierarchy information holding unit, 317 bandwidth calculation unit, 320 hierarchical change request unit

Claims (5)

Hierarchically encoded transmissions from the lowest layer, which is indispensable for decoding, to a higher layer for higher quality decoding, transmitted from the transmitting device via the communication network In a receiving device that receives data,
Hierarchical table information indicating a transmission hierarchy indicating the highest hierarchy of the transmission data to be transmitted and a bandwidth of the communication network necessary for transmitting the transmission data up to the transmission hierarchy is received from the transmission apparatus. In addition, a packet storing the transmission data, transmission layer information indicating the transmission layer of the transmitted transmission data, and maximum layer information indicating the highest layer among the plurality of layers is received from the transmission device. a receiving unit that,
Storage means for storing the received hierarchy table information ;
Calculation means for calculating a reception band which is a band of the communication network at the time of reception of the packet, based on a reception time of the packet and a packet size of the packet;
Display means for displaying the received hierarchy table information and displaying a display screen on which the transmission data, the transmission hierarchy information, the maximum hierarchy information, and the reception band stored in the received packet are displayed. When,
And a transmission unit configured to transmit to the transmission device a change request for requesting a change in the transmission layer of the transmission data transmitted from the transmission device, generated according to a user instruction. . The receiving means further receives a bandwidth measurement packet for measuring a bandwidth of the communication network, transmitted from the transmission device before transmission of the packet by the transmission device,
Band measurement means for calculating the bandwidth of the communication network at the time of reception of the band measurement packet, which is used to determine the transmission layer of the transmission data transmitted from the transmission device based on the received band measurement packet Further comprising
The transmission means further transmits the measurement result by the band measurement means to the transmission device,
The receiving apparatus according to claim 1 , wherein the display unit displays the measurement result on the display screen . The calculation means further calculates a loss rate of the packet based on the received packet,
The display means displays the loss rate on the display screen.
The receiving device according to claim 2. Hierarchically encoded transmissions from the lowest layer, which is indispensable for decoding, to a higher layer for higher quality decoding, transmitted from the transmitting device via the communication network In a receiving method of a receiving device that receives data,
Receiving from the transmitting device, layer table information indicating a transmission layer indicating the highest layer of the transmission data to be transmitted and a bandwidth of the communication network required for transmitting the transmission data up to the transmission layer A first reception control step for controlling
A storage control step for controlling storage of the received hierarchy table information ;
A first display control step for controlling display of the received hierarchy table information;
Reception of a packet storing the transmission data, transmission layer information indicating the transmission layer of the transmitted transmission data, and maximum layer information indicating the highest layer among the plurality of layers from the transmission device. A second reception control step for controlling;
A calculation step of calculating a reception bandwidth that is a bandwidth of the communication network at the time of reception of the packet based on the reception time of the packet and the packet size of the packet;
A second display control step for controlling display of a display screen on which the transmission data stored in the received packet, the transmission layer information, the maximum layer information, and the reception band are displayed;
A transmission control step for controlling transmission to the transmission device of a change request for requesting a change in the transmission hierarchy of the transmission data transmitted from the transmission device, generated according to a user instruction. And the receiving method. Hierarchically encoded transmissions from the lowest layer, which is indispensable for decoding, to a higher layer for higher quality decoding, transmitted from the transmitting device via the communication network To the receiving device that receives data ,
Receiving from the transmitting device, layer table information indicating a transmission layer indicating the highest layer of the transmission data to be transmitted and a bandwidth of the communication network required for transmitting the transmission data up to the transmission layer A first reception control step for controlling
A storage control step for controlling storage of the received hierarchy table information ;
A first display control step for controlling display of the received hierarchy table information;
Reception of a packet storing the transmission data, transmission layer information indicating the transmission layer of the transmitted transmission data, and maximum layer information indicating the highest layer among the plurality of layers from the transmission device. A second reception control step for controlling;
A calculation step of calculating a reception bandwidth that is a bandwidth of the communication network at the time of reception of the packet based on the reception time of the packet and the packet size of the packet;
A second display control step for controlling display of a display screen on which the transmission data stored in the received packet, the transmission layer information, the maximum layer information, and the reception band are displayed;
A process including a transmission control step for controlling transmission to the transmission device of a change request for requesting a change in the transmission hierarchy of the transmission data transmitted from the transmission device, generated in response to a user instruction A program characterized by letting

Images