JP6083964B2 - Transmission device, transmission method, and program - Google Patents

Description

  The present invention relates to a transmission device, a transmission method, and a program for transmitting data.

  Conventionally, RTP (A Transport Protocol for Real-Time Applications, RFC3550, IETF) is known as a communication protocol for transmitting data in real time via a network.

  In addition, as a technique for encoding data, H.264 is used. H.264 / AVC (ISO / IEC 14496) and JPEG (ISO / IEC 10918) are known.

  In real-time data transmission, the transmission device needs to transmit transmission data without delay in order to ensure real-time performance in the reception device.

  However, when the amount of transmission data is large or the bandwidth of the network is insufficient, the transmission processing of the data transmission device may be delayed. In the configuration in which the transmission data is buffered in the transmission buffer of the transmission device, the transmission data stays in the transmission buffer when the transmission processing of the transmission device is delayed. If transmission data stays in the transmission buffer, there is a possibility that the real-time property of data reception in the data reception device cannot be secured.

  Therefore, as a method to ensure real-time transmission even when transmission data stays in the transmission buffer, give up sending transmission data staying in the transmission buffer and change the transmission data in the buffer. A method of discarding and not transmitting is known.

  For example, a technique is known in which transmission data up to the next output video data that is output within the screen is discarded and not transmitted among the video data in the transmission buffer (for example, Patent Document 1). Here, the intra-coded video data is video data encoded without referring to video data of other video frames, and the receiving device does not refer to video data of other video frames. This is video data that can be decrypted.

JP 2010-245822 A

If transmission data stays in the transmission buffer, there is a possibility that the real-time property of data reception in the receiving device cannot be secured.

  As a means for solving the above-described problems, the transmission apparatus of the present invention has the following configuration.

That is, the buffer means for holding data according to the Film images, and transmitting means for pre-sending a Kide over data to the receiving device, the data of the frame to be decoded without reference to other frames the in response to being held in the buffer means, without transmitting at least a portion of the data of Lud over data held in it said buffer means, frame to be decoded without reference to another frame the beam data have a transmission control means for performing transmission control for transmitting from said transmitting means to said receiving apparatus, said transmission control means, the data is the buffer means of the frame to be decoded without reference to another frame When a part of data constituting the first frame is transmitted to the receiving device by the transmitting unit and the remaining data is stored in the buffer unit when held, the remaining data Is transmitted from the transmitting means to the receiving device.
Further, as one means for solving the above problems, another transmission apparatus of the present invention has the following configuration.
Buffer means for holding data relating to video, transmission means for transmitting the data to a receiving device, and data of a frame to be decoded without referring to other frames being held in the buffer means, Transmission control is performed to transmit data of a frame to be decoded without referring to other frames without transmitting at least a part of the data held in the buffer means to the receiving device. Transmission control means, wherein the transmission control means sends data from the transmission means to the receiving device when the amount of data held in the buffer means is greater than a first threshold value and less than or equal to a second threshold value. The data amount of data held in the buffer means is larger than the second threshold value than the data amount of data to be controlled not to transmit. It performs the transmission control so that the data amount increases data to control not to transmit to the receiving apparatus from the transmitting unit in the case.

ADVANTAGE OF THE INVENTION According to this invention, the real time property of the data delivered to a receiver can be improved.

Configuration example of transmitting apparatus in embodiment 1 Functional block diagram of a transmission apparatus according to the first embodiment Flowchart showing packet transmission processing in the first embodiment Flowchart illustrating transmission processing of packets held in the queue according to the first embodiment. Flowchart showing packet discard processing in the first embodiment The figure which shows the example of the packet discard process in Example 1 The figure which shows the structure of the packet in Example 1. FIG. Flowchart showing packet transmission processing in the second embodiment Flowchart showing packet discard processing in the second embodiment Flowchart showing packet transmission processing in Embodiment 3 The flowchart which shows the packet 1st discard process in Example 3. H. The figure which shows the example of the inter-frame reference in H.264 / AVC encoding

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

<Example 1>
In this embodiment, a transmission device that transmits frame data of a video captured by a camera or the like to a reception device in real time will be described. Here, the frame data may include video data and accompanying metadata and audio data. Each of the transmission device and the reception device may be realized by a single computer device, or may be realized by distributing each function to a plurality of computer devices as necessary. When configured by a plurality of computer devices, they may be connected via a LAN (Local Area Network) or the like so that they can communicate with each other.

  A configuration example of the transmission apparatus in this embodiment is shown in FIG. The control unit 101 controls the operation of each component of the transmission apparatus illustrated in FIG. The control unit 101 is configured by a processor such as a CPU (Central Processing Unit), for example. When the control unit 101 is configured as a processor, the control unit 101 reads and executes a program stored in the storage unit 102, thereby controlling the operation of each component of the transmission apparatus illustrated in FIG.

  The storage unit 102 stores a program executed by the control unit 101. The storage unit 102 is configured by, for example, a ROM (Read Only Memory).

  The holding unit 103 functions as a main memory, a work area, and the like of the control unit 101. The holding unit 103 is configured by, for example, a RAM (Random Access Memory). The holding unit 103 serves as a transmission buffer (queue) that holds data that has not been transmitted to the receiving apparatus. In this embodiment, the holding unit 103 outputs the held packet by FIFO (First-In First-Out) control. That is, the holding unit 103 holds packet data obtained by dividing frame data.

  The communication unit 104 controls input / output of information with the receiving apparatus via the network. As the network, for example, the Internet, public radio, LAN, or the like can be used. There are no restrictions on the network communication standard, scale, wired or wireless.

  The imaging unit 105 images a subject. The imaging unit 105 is configured by an imaging element such as a lens or a CMOS (Complementary Metal Oxide Semiconductor). The image sensor converts an image of a subject formed by the lens into an image signal.

  The imaging control unit 106 controls the shooting direction and angle of view of the imaging unit 105. The imaging control unit 106 performs control such as panning / tilting / zooming of the imaging unit 105, for example.

  The image processing unit 107 performs image processing on the image signal captured and generated by the imaging unit 105. For example, the image processing unit 107 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the data generated by the imaging unit 105.

  The encoding unit 108 performs processing for converting the data subjected to the image processing by the image processing unit 107 according to a predetermined encoding method. In this embodiment, H. An example of encoding with H.264 / AVC will be described, but the encoding method is not particularly limited to this. A bus 110 connects the components shown in FIG. 1 and serves as a transfer path for various data.

  The generation unit 109 generates a data segment obtained by dividing the encoded data into a plurality of pieces. For example, the generation unit 109 generates a packet according to the transmission protocol from the data encoded by the encoding unit 108. In this embodiment, a case where RTP (Real-time Transport Protocol) is used as a transmission protocol will be described. However, the transmission protocol is not limited to this. For example, a configuration using TCP / IP (Transmission Control Protocol / Internet Protocol) or the like may be used.

  Next, a functional configuration diagram of the control unit 101 will be described with reference to FIG. When the control unit 101 includes a processor, each function shown in each block in FIG. 2 is realized by the processor executing a program stored in the storage unit 102. Alternatively, some or all of the functions shown in each block in FIG. 2 may be realized by individual hardware.

  The input control unit 201 performs control to input data such as video data, audio data, and metadata to the encoding unit 108. The encoding control unit 202 controls the encoding unit 108 to encode the input data. The generation control unit 203 controls the generation unit 109 to packetize the encoded data according to the transmission protocol.

  The transmission control unit 204 causes the generated packet to be transmitted from the communication unit 104 to the receiving device. Further, the transmission control unit 204 performs control to transmit the packet data held in the holding unit 103 to the receiving device in accordance with the control of the management unit 206. The holding control unit 205 performs control for causing the holding unit 103 to hold an untransmitted packet among the generated packets.

  The management unit 206 manages the packets held in the holding unit 103. Specifically, an untransmitted packet is enqueued to the holding unit 103 as a queue, or the head packet of the queue is dequeued and passed to the communication unit 104. Further, the management unit 206 performs control so that the packet in the holding unit 103 is discarded and not transmitted as necessary.

  Next, the operation of the transmission apparatus according to the present embodiment will be described with reference to FIGS. In the form in which the control unit 101 includes a processor, the processing flows in FIGS. 3 to 5 show programs for causing the processor to execute the procedures shown in FIGS. 3 to 5. The processor built in the control unit 101 is a computer, and executes a program read from the storage unit 102.

  First, a processing flow until the packet (this) generated by the generation unit 109 is input to the communication unit 104 will be described with reference to FIG. A packet (this) is input to the communication unit 104 under the control of the transmission control unit 204 (S301). When a packet is input to the communication unit 104 (Yes in S301), the management unit 206 confirms whether the packet is buffered in the holding unit 103 (S302). On the other hand, when there is no packet input in the communication unit 104 (No in S301), the process ends.

  When the packet (this) is input to the communication unit 104 and the packet is not held in the holding unit 103 (No in S302), the management unit 206 determines whether or not the packet is transmitted to the receiving device ( S303). The determination method is not particularly limited.

  Examples of the case where the packet is not transmitted to the receiving device include a case where the network is in a congested state and a state where the receiving device cannot receive the packet.

  When RTP is used as a protocol used for communication between the transmission device and the reception device, it can be determined whether the network is in a congestion state based on the round trip time. Here, the round trip time refers to the time from when one of the transmission device and the reception device transmits a packet to the other until the response is returned from the other device. As a method for determining whether or not the network is congested based on the round trip time, for example, control such as TFRC (TCP Friendly Rate Control) is known. Depending on the result of such determination, the determination in step S303 may be performed. For example, when it is determined that the network is in a congestion state, it can be determined that the packet is not transmitted to the receiving device.

  When TCP is used as a protocol used for communication between the transmission device and the reception device, an acknowledgment (Ack) for one or more segments (data per transmission unit) transmitted from the transmission device is received from the reception device. In response, the next segment is transmitted. Therefore, when an acknowledgment is not received from the receiving device, a state in which no packet is transmitted from the transmitting device occurs. Therefore, the determination in step S303 may be performed according to whether a predetermined confirmation response for the data transmitted by the transmission device is received from the reception device. For example, when a predetermined confirmation response is not received from the receiving device, it can be determined that the packet is not transmitted to the receiving device.

  In addition, when TCP is used, data having a data amount exceeding the size of the reception buffer (window size) of the reception device cannot be transmitted to the reception device at a time. Therefore, when the size of data to be transmitted exceeds the window size, packet data exceeding the window size is not transmitted to the receiving device. Therefore, the determination in step S303 may be performed according to the data size transmitted by the transmission device and the data size receivable by the reception device. For example, when the size of data transmitted by the transmission device exceeds the window size, it can be determined that the packet is not transmitted to the reception device.

  When the management unit 206 determines that the packet is transmitted to the receiving device (Yes in S303), the transmission control unit 204 controls the communication unit 104 to transmit the input packet (this) to the receiving device. Processing is performed (S304).

  On the other hand, when the management unit 206 determines that the packet is not transmitted to the receiving device (No in S303), the management unit 206 performs a process of enqueuing the packet (this) to the holding unit 103 as a queue (S305).

  In the process of step S302, if the packet is held in the holding unit 103 as a queue (No in S302), the transmission control unit 204 performs transmission processing of the packet held in the holding unit 103 (S400). . That is, while enqueuing the packet (this) in the holding unit 103 as a queue, a process of transmitting the packets already held in the queue to the receiving device in order from the top is performed.

  Here, the transmission process in step S400 will be described with reference to FIG. First, the management unit 206 determines whether or not the packet (this) is the first packet among the packets that are stored in the storage unit 103 and are encoded without referring to other frames. (S401). The frame encoded without referring to other frames is a frame encoded by intra prediction encoding, for example. This corresponds to an IDR (Instantaneous Decoder Refresh) frame in H.264 / AVC encoding. For example, an I-frame (Intra-coded Frame) is applicable. Packets constituting a frame to be encoded without referring to another frame can be decoded without referring to data of the other frame in the receiving apparatus. In this embodiment, an example will be described in which it is determined whether or not the packet (this) is the first I frame among the packets held in the holding unit 103 in step S401.

  When the packet (this) is held in the holding unit 103 and is the first packet among the packets constituting the frame encoded without referring to other frames (Yes in S401), FIG. 5 is used. Then, a packet discarding process described later is performed (S500). The packet discarding process according to the present embodiment is a process for performing control so that a part of the packet data held in the holding unit 103 is not transmitted to the receiving apparatus. In this manner, the management unit 206 causes the packet discarding process to be performed when the holding unit 103 holds the packet data that forms the head of the intra-frame encoded frame data.

  When it is determined that the packet (this) is not the first I frame among the packets held in the holding unit 103 (No in S401), or when the process of step S500 is performed, the management unit 206 The process of step S402 is performed. In step S402, the management unit 206 enqueues the packet (this) in the holding unit 103 as a queue.

  Next, the management unit 206 determines whether or not packet transmission is performed. The determination method is the same as that in step S303 in FIG. 3, and is not particularly limited.

  When it is determined that packet transmission is performed (Yes in S403), the management unit 206 reads (dequeues) the first packet from the holding unit 103 (queue) (S404). Then, the transmission control unit 204 transmits the read packet to the receiving device (S405).

  Subsequently, the management unit 206 determines whether or not the holding unit 103 serving as a queue becomes empty by reading the packet (S406). When packets remain in the queue (No in S406), the management unit 206 repeats the processing from step S403 to step S406 until it determines that no packet is transmitted in step S403. On the other hand, if no packet remains in the queue (Yes in S406), the management unit 206 ends the packet transmission process.

  If it is determined in step S403 that packet transmission is not performed, the management unit 206 compares the size of data held in the queue (accumulated data size) with a predetermined threshold (S407).

  When the accumulated data size exceeds the threshold value (Yes in step S407), a packet discard process (S500) described later with reference to FIG. 5 is performed, and the in-queue packet transmission process is terminated after the discard process. In this way, the management unit 206 causes the packet discarding process to be performed when the size of the packet data held in the holding unit 103 exceeds a predetermined threshold.

  In step S403, instead of the process of comparing the accumulated data size with a predetermined threshold value, the time during which one packet data is held in the queue (holding time) may be compared with the threshold value. If the holding time exceeds the threshold value, the packet discarding process (S500) is performed. If the holding time is equal to or less than the threshold value, the process of step S402 is performed. In this way, the management unit 206 can cause the packet discarding process to be performed when the time during which one packet data is held in the holding unit 103 exceeds a predetermined threshold.

  Next, the packet discarding process in S500 described above will be described with reference to FIG. First, the management unit 206 acquires the first packet in the queue as a packet (next) (S501). Here, the head packet in the queue is the first packet dequeued among the packets held in the queue.

  Next, the management unit 206 determines whether or not the packet (next) has been acquired (S502). When the first packet (next) is acquired, the management unit 206 determines whether the packet (next) is the first packet of the frame (S503). Here, the head packet of a frame is a packet that is first dequeued from a queue among a plurality of packets that constitute one packet.

  When the packet (next) is not the first packet of the frame (No in S503), the management unit 206 acquires a packet to be dequeued next to the packet (next) in the queue, and acquires the acquired packet as a new packet ( next) (S504). Then, the process returns to step S502.

  When the packet (next) is the head packet (Yes in S503), the management unit 206 discards all packets after the packet (next) including the packet (next) in the queue (S505). Then, the management unit 206 ends the packet discard process.

  In step S502, when the packet (next) cannot be acquired in the queue (Yes in S502), the data size of the packet held in the queue is compared with a predetermined threshold (S506). Here, the case where the packet (next) cannot be acquired is the case where the last packet in the queue is the packet (next). That is, the determination process of step S503 is performed for all the packets in the queue, but none of the packets in the queue is the first packet of the frame. Here, the last packet in the queue is a packet dequeued last among the packets in the queue.

  If the data size of the packet held in the queue exceeds a predetermined threshold (Yes in step S506), all the packets in the queue are discarded (S507), and the in-queue packet discarding process is terminated. If the data size of the packet held in the queue is equal to or smaller than the threshold value (No in S506), the key packet discarding process is terminated as it is.

  In this way, in the packet discarding process, the management unit 206 transmits the second packet, which is packet data constituting the frame data composed of the first packet data transmitted to the receiving device and held in the holding unit 103. Data is transmitted to the receiving device.

  Also, by performing the above-described packet discarding process in step S401 in FIG. 4, the packet data from the first packet data of the first frame data to before the first packet data of the second frame data is discarded. Can do. In other words, it is possible to control so as not to transmit from the third packet data constituting the head of the first frame data to the packet data before the fourth packet data constituting the head of the second frame data. . In the present embodiment, the fourth packet data is a packet (this) determined to be the leading packet of the I frame in step S401.

  In the example shown in FIG. 2 shows a packet obtained by dividing transmission data encoded using the H.264 / AVC encoding method.

  The configuration of each packet held in the queue 600 is shown in FIG. The packet data 700 corresponds to each of the packet data 601 to 609 shown in FIG. Packet data 700 includes an RTP packet 701 and an element header 704.

  The RTP packet 701 further includes an RTP payload 702 and an RTP header 703. The RTP payload 702 is an H.264 file. Video data encoded using an H.264 / AVC encoding method is stored. The RTP header 703 stores marker bits, sequence numbers, time stamps, and the like for video data stored in the RTP payload 702.

  The element header 704 includes a head flag 705 and an I frame flag 706. The frame head flag 705 is a flag indicating whether or not it is the head packet of the frame. The I frame flag 706 is a flag indicating whether or not the packet constitutes an I frame. These flags are used in the process of S401 in the packet transmission process described with reference to FIG. 4 and the process of step S503 in the packet discard process described with reference to FIG. The element header 704 is discarded when the packet data 700 is dequeued and sent to the communication unit 104.

  Processing when the packet data 609 that is the first packet of the I frame is enqueued in the queue 600 will be described with reference to FIG. Here, in the example shown in FIG. 6A, it is assumed that packets constituting the P frame (1), the P frame (2), and the P frame (3) have already been enqueued in the queue 600. Each P frame (1) to (3) is one frame. Further, in the example shown in FIG. 6A, it is assumed that the packet data 601 out of the three pieces of packet data 601, 602, and 603 constituting the P frame (1) has been dequeued and transmitted to the receiving apparatus.

  In the example of FIG. 6A, when the packet data 609 is to be enqueued, the presence / absence of a packet in the queue 600 is determined (S302 in FIG. 3). If it is determined that a packet is stored in the queue 600 (No in S302), the packet transmission process described with reference to FIG. 4 is executed (S400 in FIG. 3).

  In the packet transmission process, as described with reference to FIG. 4, it is first determined whether or not the packet data 609 to be enqueued is the first packet of the I frame (S401 in FIG. 4). In the example of FIG. 6A, it is determined that the input packet is the head packet of the I frame (Yes in S401), and the process proceeds to the packet discarding process (S500 in FIG. 4) described with reference to FIG. The packet discarding process in the present embodiment is a process for performing control so that a part of the packet data held in the holding unit 103 is not transmitted to the receiving apparatus.

  In the packet discarding process described with reference to FIG. 5, packets are acquired in order from the first packet in the queue, and the first packet in the frame is searched. In the example shown in FIG. 6A, since the packet data 602 is not the first frame of the P (1) frame, the process proceeds to No in step S503 shown in FIG. 5, and the next packet data 603 is the first packet of the frame. Judgment is made. Similarly, since the packet data 603 is not the first packet of the P (1) frame, it is determined whether the next packet data 604 is the first packet of the frame.

  The packet data 604 is the head packet of the P (2) frame as shown in FIG. 6A (Yes in S503 in FIG. 5). Therefore, the process proceeds to step S505 shown in FIG. 5, the packet data after the packet data 604 is discarded, and the packet discard process is terminated.

  Here, when the packet discarding process is performed, the sequence number of the RTP header 703 of the packet remaining in the queue 600 may be advanced by the number of discarded packets. When the packet discarding process (S500 in FIG. 4) ends, the packet data 609 is enqueued (S402 in FIG. 4).

  FIG. 6B shows a situation where the packet data 609 is enqueued. In the example of FIG. 6B, the packets (packet data 602 and packet data 603) constituting P (1) that have already transmitted a part of the frame (packet data 601) remain in the queue 600 without being discarded. Accordingly, the packet data 602 and the packet data 603 are transmitted to the receiving device. In this way, when packet discard processing is performed, second packet data which is packet data constituting frame data composed of the first packet data transmitted to the receiving apparatus and held in the holding unit 103 Is transmitted to the receiving apparatus.

  According to the transmission apparatus shown in the present embodiment, when a part of a frame has been transmitted, the packet constituting the frame can be transmitted to the reception apparatus without being discarded. Therefore, it is possible to prevent a part of the packets constituting the frame from being transmitted and disturbing the video of the frame transmitted to the receiving apparatus.

Further, according to the transmission apparatus shown in the present embodiment, when the first packet of a frame (for example, I frame) that can be decoded without referring to another frame is enqueued, the packet in the queue A part or all of the data is discarded (not transmitted).
In the example of FIG. 6, packet data from the first packet data 604 of the first frame data P (2) to the front of the first packet data 609 of the second frame data can be discarded. In this way, newly enqueued packet data can be promptly transmitted to the receiving device. Therefore, the real-time property of data distributed to the receiving device can be improved.

  In this embodiment, following the packets (602, 603) constituting a frame in which some data has already been transmitted, the first packet (609) of a frame that can be decoded without referring to other frames is transmitted. . Therefore, it is possible to prevent packet data that cannot be decoded by the receiving apparatus from being transmitted to the receiving apparatus along with the discarding process. Here, when packet data that cannot be decoded by the receiving device is transmitted along with the discarding process, for example, packet data that is referred to when decoding packet data transmitted to the receiving device is a packet In some cases, it is discarded by the discarding process.

  In this embodiment, the case where the first packet of the I frame is enqueued and the case where the packet discarding process is performed when the accumulated data size of the queue exceeds the threshold has been described, but the present invention is not limited to this. The packet discarding process may be performed when either the first packet of the I frame is enqueued or when the accumulated data size of the queue exceeds the threshold value. The packet discarding process may be performed at a predetermined timing without being limited to the above case.

  Further, in the present embodiment, in the packet discarding process, when the first packet of the first frame in the queue is detected, an example of discarding subsequent packets is described with reference to FIG. 5, but the present invention is not limited to this. For example, when the head packet of a frame after an arbitrary number of frames from the head of the queue is detected, the subsequent packets may be discarded.

<Example 2>
In this embodiment, a case will be described in which data is transmitted with no distinction as to whether or not other frames need to be referred to when data is decoded. For example, the case where the data encoded using the encoding format by JPEG (Joint Photographic Experts Group) is transmitted is included. Alternatively, the case where all the frames to be transmitted are frames (for example, I frames) encoded without referring to other frames is included.

  Since the configuration of the transmission apparatus according to the present embodiment is the same as that described with reference to FIGS. 1 and 2 in the first embodiment, the description thereof is omitted.

  Next, the operation of the transmission apparatus in the present embodiment will be described with reference to FIGS. In the form in which the control unit 101 includes a processor, the processing flows in FIGS. 8 and 9 show programs for causing the processor to execute the procedures shown in FIGS. 8 and 9. The processor built in the control unit 101 is a computer, and executes a program read from the storage unit 102.

  First, packet transmission processing in the present embodiment will be described with reference to FIG. First, the management unit 206 enqueues the packet (this) in the holding unit 103 as a queue (S801).

  Subsequently, the management unit 206 determines whether or not a packet is transmitted (S802). The determination process in step S802 is the same as the process in step S303 described with reference to FIG. When a packet is transmitted (Yes in S802), the management unit 206 dequeues the head packet in the queue (S803).

  In step S803, when the first packet in the queue is dequeued, the management unit 206 determines whether the dequeued packet is the first packet in the frame (S804).

  When the dequeued packet is the first packet of the frame (Yes in S804), the management unit 206 compares the detention time of the dequeued packet in the queue with a predetermined threshold (S805).

  Here, the packet retention time in the queue can be calculated from the difference between the current time and the enqueued time by recording the information of the enqueued time in the element header 704 described with reference to FIG. 7 in the first embodiment. However, the method for calculating the residence time is not limited to this.

  If the dequeuing time of the dequeued packet in the queue exceeds the threshold (Yes in S805), the dequeued packet is discarded (S808). Then, after discarding the dequeued packet, the packet discarding process is performed and the packet transmission process is terminated (S900). The packet discarding process in step S900 is a process for performing control so that a part of the packet data held in the holding unit 103 is not transmitted to the receiving device. Details of the packet discarding process in this embodiment will be described later with reference to FIG.

  If the dequeued packet is not the first packet of the frame (No in S804), or if the residence time is equal to or less than the threshold (No in S805), the transmission control unit 204 performs control to transmit the dequeued packet. (S806). After transmitting the packet, the management unit 206 determines whether the queue is empty (S807).

  When the queue is empty (Yes in S807), the management unit 206 ends the packet transmission process. On the other hand, when data is held in the queue (No in S807), the management unit 206 returns to the process of step S802.

  If it is determined in step S802 that no packet is transmitted, the management unit 206 compares the size of data held in the queue (accumulated data size) with a predetermined threshold (S809). If the accumulated data size in the queue exceeds the threshold value (Yes in S809), the packet transmission process is terminated after performing the packet discard process (S900). On the other hand, when the accumulated data size in the queue is equal to or smaller than the threshold (No in S809), the packet transmission process is terminated.

  According to the transmission processing in this embodiment, when the first packet of a frame is dequeued from the queue and the retention time in the queue is equal to or greater than the threshold, the dequeued packet is discarded, and further The packet discard process is performed. According to such a configuration, the packet discarding process is started from the top packet of the frame. That is, when a part of the data of the frame has been transmitted, the packet discarding process is started after all the other data constituting the frame has been transmitted. In this way, when a part of data of a frame has been transmitted, all the packets constituting the frame can be transmitted. Therefore, it is possible to prevent the frame image from being disturbed by the packet discarding process.

  Next, the packet discarding process (S900) in the present embodiment will be described with reference to FIG. First, the management unit 206 determines whether or not the queue is empty (S901). If the queue is empty (Yes in S901), the management unit 206 ends the packet discard process. On the other hand, when transmission data is held in the queue (No in S901), the management unit 206 acquires the first packet in the queue (S902).

  When the leading packet of the queue is acquired, the management unit 206 determines whether the acquired packet is the leading packet of the frame (S903). If it is determined that the acquired packet is not the first packet of the frame (No in S903), the packet is discarded (S906). On the other hand, when the acquired packet is the first packet (Yes in S903), the management unit 206 compares the retention time in the queue of the packet with a predetermined threshold (S904). Here, the packet residence time can be calculated by the method described in step S805, for example.

  When the retention time in the queue exceeds the threshold (Yes in S904), the management unit 206 discards the packet (S906). On the other hand, when the packet retention time in the queue is within the threshold (No in S904), the management unit 206 compares the accumulated data size in the queue with a predetermined threshold (S905).

  When the stored data size in the queue exceeds the threshold (Yes in S906), the management unit 206 discards the packet (S906). On the other hand, when the accumulated data size in the queue is within the threshold (NO in S905), the management unit 206 ends the packet discarding process.

  After discarding the packet in step S906, the management unit 206 returns to the process of step S901. Then, the packet discarding process is continued.

  In the present embodiment, the case where the staying time in the queue of the packet and the data size held in the queue are determined has been described, but the present invention is not limited to this. For example, it is possible to determine only one of the residence time in the queue of the packet or the data size held in the queue.

  As described above, the packet discarding process is ended when there is a head packet of the frame at the head of the queue and the stay time and the accumulated data size in the queue are equal to or smaller than the threshold value. That is, after the packet is discarded, packet transmission is started from the top packet of the frame. That is, the entire data constituting the frame can be transmitted to the receiving device. Therefore, it is possible to prevent only a part of data constituting the frame from being transmitted to the receiving device. In this way, it is possible to prevent the frame image from being disturbed along with the packet discarding process.

  According to the transmitting apparatus according to the present embodiment, when the packet discarding process is started, the packet discarding process is started after transmitting all the data constituting the frame for the frame configured by the transmitted data. can do. Also, according to the transmission apparatus of the present embodiment, when performing packet discard processing, transmission can be started from the top data of the frame after discarding the packet. In this way, it is possible to prevent the frame image from being disturbed along with the packet discarding process.

<Example 3>
In this embodiment, an example will be described in which control is performed so that packets in a queue are discarded and not transmitted step by step. Since the configuration of the transmission apparatus according to the present embodiment is the same as that described with reference to FIGS. 1 and 2 in the first embodiment, the description thereof is omitted.

  Next, the operation of the transmission apparatus according to the present embodiment will be described with reference to FIGS. In the form in which the control unit 101 incorporates a processor, the processing flows in FIGS. 10 and 11 show programs for causing the processor to execute the procedures shown in FIGS. 10 and 11. The processor built in the control unit 101 is a computer, and executes a program read from the storage unit 102.

  First, packet transmission processing according to the present embodiment will be described with reference to FIG. First, it is determined whether or not the packet (this) is the head packet of a frame (for example, an I frame) encoded without referring to another frame (S1001).

  When the packet (this) is the head packet of the I frame (Yes in S1001), the second discarding process (S500) is executed. Since the content of the second discarding process is the same as that described with reference to FIG. 5 in the first embodiment, the description thereof is omitted.

  When the packet (this) is not the first packet of the I frame (No in S1001) or when the process of step S500 is terminated, the management unit 206 enqueues the packet (this) in the queue (S1002).

  When the packet (this) is enqueued, the management unit 206 determines whether or not packet transmission is performed (S1003). The determination in step S1003 is the same as that described in step S303 in FIG.

  When the packet is transmitted (Yes in S1003), the management unit 206 dequeues the packet from the queue (S1004). Then, the transmission control unit 204 performs control to transmit the dequeued packet (S1005).

  When the packet is transmitted in step S1005, the management unit 206 determines whether or not the queue is empty (S1006). When data is held in the queue (No in S1006), the management unit 206 returns to the process of step S1003. On the other hand, if the data is empty in the queue (Yes in S1006), the management unit 206 ends the packet transmission process.

  If it is determined in step S1003 that no packet is transmitted, the management unit 206 compares the size of data held in the queue (accumulated data size) with a predetermined first threshold value (S1007). When the accumulated data size is equal to or smaller than the first threshold (No in S1007), the management unit 206 ends the packet transmission process.

  When the accumulated data size is larger than the first threshold value (Yes in S1007), the management unit 206 compares the accumulated data size with the second threshold value (where the first threshold value <the second threshold value) ( S1008). When the accumulated data size is equal to or smaller than the second threshold (No in S1008), the management unit 206 performs a first packet discarding process (S1100). Details of the first packet discarding process will be described later with reference to FIG. When the accumulated data size is larger than the second threshold (Yes in S1008), the management unit 206 performs a second packet discarding process (S500). When the management unit ends the first packet discard process or the second packet discard process, the management unit ends the packet transmission process. Here, it is assumed that the data amount of the packet discarded in the second packet discarding process is larger than the data amount of the packet discarded in the first packet discarding process.

  In step S1007 and step S1008, instead of the process of comparing the accumulated data size with a predetermined threshold, the time during which one packet data is held in the queue (holding time) may be compared with the threshold. When the holding time is longer than the first threshold and not more than the second threshold (first threshold <second threshold), the first packet discarding process is performed. When the holding time is longer than the second threshold, the second packet discarding process is performed.

  Next, the first packet discarding process in the present embodiment will be described with reference to FIG. First, the management unit 206 acquires the first packet in the queue and sets it as a packet (next) (S1101).

  Next, the management unit 206 determines whether or not the packet (next) is the head packet of the frame (S1102). When the packet (next) is not the first packet of the frame (No in S1102), the management unit 206 acquires a packet next to the packet (next) and sets it as a new packet (next) (S1103).

  When the processing of step S1103 is performed, the management unit 206 determines whether or not the next packet can be acquired (S1104). That is, the management unit 206 determines whether the processing has been completed up to the last packet in the queue.

  When the packet (next) cannot be acquired (No in S1104), the management unit 206 ends the first discarding process. In other words, the management unit 206 ends the first discarding process when all the packets in the queue have been scanned. On the other hand, when the packet (next) can be acquired, the management unit 206 returns to the process of step S1102 and continues the process.

  If it is determined in step S1102 that the packet (next) is the head packet of the frame, the management unit 206 determines whether the frame is a frame that is not referenced by another frame (S1105). . In the determination process, a flag indicating that there is a reference from another frame is added to the element header 704 described with reference to FIG. 7, and the presence or absence of the reference is determined based on the presence or absence of the flag. it can. However, the determination method is not limited to this.

Here, frames that are not referred to by other frames will be described with reference to FIG.
In the example of FIG. Reference between frames in encoding using H.264 / AVC will be described.

  Frames 1201 to 1208 are respectively H.264. This is a frame encoded using H.264 / AVC. In the example of FIG. 12, it is assumed that packets are transmitted from left to right. Frames from the I frame 1201 to the P frame 1207 constitute a set of GOPs (Group Of Pictures).

  The arrow in FIG. 12 indicates the reference direction at the time of encoding, and the characters written inside the frame indicate the type of the encoded frame. Here, the I frame is a frame that is encoded without using inter-frame prediction. P frames are frames that are encoded using only forward prediction. The B frame is a frame that is encoded by selecting one of forward prediction, backward prediction, and bidirectional prediction.

  In the example of FIG. 12, the I frame 1201 is referred to by the B frame 1202 and the P frame 1203. The P frame 1203 is referred to as the B frame 1202 and the B frame 1204. In the example of FIG. 12, the B frames 1202, 1204, and 1206 are frames that are not referred to by other frames.

  Returning to the description of the processing in FIG. When the frame including the packet (next) is a frame that is not referred to by another frame (No in S1105), the management unit 206 proceeds to the process of step S1103. On the other hand, when the frame including the packet (next) is a frame referenced from another frame (Yes in S1105), the management unit 206 discards the packet (next) from the queue (S1106). When the packet (next) is discarded in step S1106, the management unit 206 acquires a packet next to the packet (next) in the queue, and sets the acquired packet as a new packet (next) (S1107).

  When a new packet (next) is set, the management unit 206 determines whether a new packet (next) can be acquired from the queue (S1108). When a new packet (next) cannot be acquired, the management unit 206 ends the first discard process. That is, when the process is performed up to the last packet held in the queue, the first discard process is terminated.

  On the other hand, if a new packet (next) can be acquired (No in step S1108), it is determined whether the new packet (next) is the head packet of the frame (S1109). When the new packet (next) is not the first packet of the frame (No in S1109), the management unit 206 returns to the process of step S1106. On the other hand, when it is determined that the new packet (next) is the top packet of the frame (Yes in S1109), the process returns to step S1105.

  In the first discard process in the present embodiment, the packets constituting the frame that is not referred to by other frames are discarded from the packets in the queue. However, in the first discard process, the packet discard process may be performed by other methods. For example, an arbitrary frame may be discarded from the back in the queue. For example, when encoding is performed using an encoding method that does not refer to other frames, such as JPEG, discard processing is performed by a method such as discarding one frame from a plurality of frames in the queue. It is good as well.

  In this embodiment, when the size of data held in the queue is larger than the first threshold, it is further determined whether the data size in the queue is larger than the second threshold larger than the first threshold. When the data size in the queue is larger than the first threshold value and smaller than or equal to the second threshold value, the packets constituting the frame that is not referenced from other frames among the packets held in the queue are discarded. When the data size in the queue is larger than the second threshold value, the data constituting the frame including the transmitted data remains in the queue, and the data in the queue is discarded.

  In this way, according to the present embodiment, the data held in the queue can be discarded step by step according to the size of the data held in the queue.

<Other examples>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

103 Holding Unit 104 Communication Unit 204 Transmission Control Unit 205 Holding Control Unit 206 Management Unit

Claims (15)

A buffer means for holding data according to the Film image,
Transmission means for pre-sending a Kide over data to the receiving device,
In response to the data of the frame to be decoded without reference to another frame is held in said buffer means, at least a portion of the de chromatography of Lud over data held in said buffer means without sending the data, have a transmission control means for performing transmission control to transmit the data frames to be decoded without reference to other frames from said transmitting means to the receiving device,
The transmission control means is configured such that when data of a frame to be decoded without referring to another frame is held in the buffer means, a part of data constituting the first frame is transferred to the reception apparatus by the transmission means. When the remaining data has been transmitted and held in the buffer means, the remaining data is transmitted from the transmitting means to the receiving device.
A transmission apparatus characterized by the above. Buffer means for holding video data;
Transmitting means for transmitting the data to a receiving device;
In response to the fact that the data of the frame to be decoded without referring to other frames is held in the buffer means, at least a part of the data held in the buffer means is not transmitted, and the other Transmission control means for performing transmission control for transmitting data of a frame to be decoded without referring to the frame from the transmission means to the reception device;
Have
The transmission control unit is configured to control data not to be transmitted from the transmission unit to the receiving device when a data amount of data held in the buffer unit is greater than a first threshold value and less than or equal to a second threshold value. The amount of data to be controlled so as not to be transmitted from the transmitting unit to the receiving device is larger than the amount of data when the data amount of the data held in the buffer unit is larger than the second threshold value. Perform the transmission control
A transmission apparatus characterized by the above. When the transmission control unit performs the transmission control when the data amount of the data held in the buffer unit is larger than the second threshold, the data of the frame to be decoded without referring to another frame All data held in the buffer means before being held in the buffer means is not transmitted from the transmitting means to the receiving device.
The transmission apparatus according to claim 2, wherein: The transmission control means performs the transmission control when data constituting the head of the second frame is held in the buffer means without being transmitted from the transmission means to the receiving device. The transmission device according to any one of claims 1 to 3 . The at least part of the data is data of the second frame.
The transmission apparatus according to claim 4, wherein: Wherein at least a portion of said data, the transmitting device according to any one of claims 1 to 5, characterized in that a data of frames to be decoded with reference to another frame. Said transmission control means, a state in which the data amount of Lud over data be held by the buffer means exceeds a predetermined threshold value, the state of network congestion, the state in which the receiving device can not receive data, and, one When the time of data is held in said buffer means is in one of the states of the state exceeds a predetermined threshold value, any one of claims 1-6, characterized in that to perform the transmission control Item 1. The transmitter according to Item 1. A transmit how to transmit the data according to the image on the receiving device,
A holding step of holding front Kide over data in the buffer means,
In response to the data of the frame to be decoded without reference to another frame is held in said buffer means, at least a portion of the de chromatography of Lud over data held in said buffer means without sending the data, and a transmission control step which performs transmission control for transmitting the signal unit sending the data of the frame to be decoded without reference to other frames to the receiving device,
In the transmission control step, when data of a frame to be decoded without referring to another frame is held in the buffer unit, a part of data constituting the first frame is transferred to the receiving device by the transmission unit. When the remaining data has been transmitted and held in the buffer means, the remaining data is transmitted from the transmitting means to the receiving device.
A transmission method characterized by the above. A transmission method for transmitting data relating to a video to a receiving device,
A holding step for holding the data in a buffer means;
In response to the fact that the data of the frame to be decoded without referring to other frames is held in the buffer means, at least a part of the data held in the buffer means is not transmitted, and the other A transmission control step for performing transmission control for transmitting data of a frame to be decoded without referring to the frame from the transmission means to the reception device;
Have
In the transmission control step, when the data amount of the data held in the buffer unit is larger than the first threshold value and less than or equal to the second threshold value, the data to be controlled so as not to be transmitted from the transmission unit to the receiving device The amount of data to be controlled so as not to be transmitted from the transmitting unit to the receiving device is larger than the amount of data when the data amount of the data held in the buffer unit is larger than the second threshold value. Perform the transmission control
A transmission method characterized by the above. In the transmission control step, when the transmission control is performed when the data amount of the data held in the buffer means is larger than the second threshold, the frame data decoded without referring to other frames All data held in the buffer means before being held in the buffer means is not transmitted from the transmitting means to the receiving device.
The transmission method according to claim 9. In the transmission control step, the transmission control is performed when data constituting the head of the second frame is held in the buffer means without being transmitted from the transmission means to the receiving device. The transmission method according to any one of claims 8 to 10 . The at least part of the data is data of the second frame.
The transmission method according to claim 11. Wherein at least a portion of said data is a method of transmission according to any one of claims 8 to 1 2, characterized in that the data of the frame to be decoded with reference to another frame . Wherein in the transmission control step, a state in which the data amount of Lud over data be held by the buffer means exceeds a predetermined threshold value, the state of network congestion, the state in which the receiving device can not receive data, and, one When the time data is held in said buffer means is in one of the states of the state exceeds a predetermined threshold value, any one of claims 8-1 3, characterized in that said transmission control The transmission method described in.   The program for functioning a computer as a transmission control means of any one of Claims 1-7.

Images