JP2009094847A - Method and program for processing animation communication - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the following problem: a client device to which an image frame where constituent packets are lost is transmitted from a server device can not reproduce the image frame at all. <P>SOLUTION: The animation communication processing method includes: a step (S202) of calculating transmission band on the basis of an error notification packet received from the client device and a packet amount transmitted until receiving the error notification packet, calculating a transmission interval of a packet so as to fall within the transmission band and storing the transmission interval into a storing part; checking whether the transmission timing of a subsequent image frame to be transmitted overlaps the transmission timing of the preceding image frame, before transmitting an image frame (S301); transmitting a packet of the subsequent image frame according to the transmission interval of the packet, when the transmission timing of the image frames does not overlap (S302); and discarding the subsequent image frame without transmitting the subsequent image frame, when the transmission timing of the image frames overlaps (S303). <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a moving image communication technique.

  A streaming server device (hereinafter simply referred to as a server) and a streaming client device (hereinafter simply referred to as a client) are connected by a LAN (Local Area Network), and image data (for example, simply referred to as a moving image) from the server (for example, , Image frames) are transmitted via a LAN, and an image service system for reproducing a received image by a client (for example, see Patent Document 1) is known.

  Here, an example of the configuration of the image service system as described above will be described with reference to FIG. An imaging apparatus (for example, a video camera) 2 that captures still images and moving images is connected to the server 1 in the image service system in FIG. The server 1 transmits image data captured by the imaging device 2 to the client 3 via the network 100. Then, the client 3 reproduces the transmitted image data and displays it using a display device (for example, a display device) 4.

  The server 1 and the client 3 include a control unit (not shown) such as a CPU (Central Processing Unit), a storage unit (not shown) including a memory and a hard disk drive device, and a communication unit including a network interface device (not shown). It shall be provided. The server 1 and the client 3 may be considered as computers, for example. The server 1 also has an input unit (for example, an input unit for inputting a moving image once stored in the storage unit or an input unit for inputting from a photographing apparatus; not shown) for inputting captured moving image or still image data. Prepare.

  The streaming method in the server 1 is motion JPEG that appears as a moving image (or quasi-moving image) by transmitting 30 still images (JPEG (Joint Photographic Experts Group) image frames) per second.

Generally, since the data size of one JPEG image frame (hereinafter referred to as an image frame) is larger than one packet of the LAN, it is necessary to divide it into a plurality of packets for transmission. The manner of dividing the image frame into a plurality of packets will be described with reference to FIG. For example, the image frames F _A about 50KByte (kilobytes) in Figure 4, it is transmitted by dividing into packets P _A of 1500Byte per packet (byte). Similarly, the image frame F _B is divided into packet groups P _B and the image frame F _C is divided into packet groups P _C. Further, the image frame generation interval by the photographing apparatus 2 is, for example, 33 ms (milliseconds). Further, the packet transmission interval follows a preset value.

In addition, the packet is divided by assigning (or embedding in the packet) a serial number for identifying the packet obtained by dividing the image frame in the header portion of the packet according to the order in which the image frames were shot (shooting order). Yes. The serial number includes, for example, a number indicating the shooting order for each image frame and a number for each packet obtained by dividing the image frame. More specifically, will be described with reference to FIG. 4, the image frame F _A is divided into packets allocated to the serial number A ₁ to A _n. In the serial numbers A _{1 to An} , “A” is a number for each image frame, and subscripts “1” to “n” are numbers for each divided packet.

Therefore, the client 3, when all the packets that are divided (e.g., packets P _A) arrives, the image frame (e.g., image frames F _A) to reproduce an image constitute accordance serial number. That is, one frame of image data is composed of packets (or LAN frames) in a plurality of LANs.
Japanese Patent Laid-Open No. 10-42185 (paragraphs [0014] to [0050] and the like).

  In the image service system as described above, even in the LAN environment where the transmission band is not sufficient and the packet is lost (that is, the packet is lost), even one packet constituting one image frame is lost. The client cannot play the image frame.

  Furthermore, even if a slight packet loss occurs in an image frame, if it occurs uniformly in each image frame, the moving image cannot be reproduced at all by the client.

  The present invention has been made based on the above-described problem, and a moving image that transmits a minimum image in a moving image so that the client can play it even if a packet loss occurs in the server in the system as described above. To provide a communication processing method and a moving image communication processing program.

  In order to solve the above problem, the invention according to claim 1 is provided with a storage unit, a communication unit, and an input unit for inputting captured moving image data, and an image frame constituting the moving image data input by the input unit. A server that allocates a serial number for identifying a packet obtained by dividing the image frame according to the shooting order of the image frame, divides the packet into packets, stores the amount of transmitted packets in the storage unit, and transmits the packets to the client device via the communication line A video communication processing method used for a device, wherein an error notification packet having a packet loss notification including a lost packet sequence number is received from the client device using the communication unit, the error notification packet, and the error The transmission bandwidth is calculated based on the amount of packets transmitted by the server device itself until the notification packet is received, and it falls within that transmission bandwidth. Calculating the packet transmission interval, storing the calculated packet transmission interval in the storage unit, and transmitting the preceding image frame to be transmitted before transmitting the image frame. The step of checking whether or not the transmission timing of the image frame overlaps, and if the transmission timing of the subsequent image frame does not overlap the transmission timing of the preceding image frame, the transmission is performed according to the packet transmission interval stored in the storage unit Transmit the subsequent image frame if the transmission timing of the subsequent image frame overlaps the transmission timing of the preceding image frame with the step of transmitting the packet of the subsequent image frame using the communication unit at an interval. And without discarding the subsequent image frame And wherein the door.

  The invention described in claim 2 is a moving image communication processing program, wherein the moving image communication processing method according to claim 1 is described as a computer program executable by a computer.

  According to the first aspect of the present invention, it is possible to transmit by thinning out image frames in a moving image based on packet loss.

  According to the invention of claim 2, the moving image communication processing method of claim 1 can be described as a computer program.

  As described above, according to the first aspect of the present invention, even if a packet loss occurs on a transmission path with a narrow transmission band, a minimum image in a moving image can be transmitted to the client device. Further, it is possible to avoid a situation in which the client device cannot reproduce the moving image at all.

  According to invention of Claim 2, the computer program which mounted the moving image communication processing method of Claim 1 can be provided.

  These can contribute to the video communication technology field.

  Hereinafter, a moving image communication processing method of an image service system according to an embodiment of the present invention will be described in detail with reference to the drawings. Note that the apparatus configuration of the present embodiment is the same as that shown in FIG.

  In the image service system according to the present embodiment, when an image frame is generated periodically (for example, at an interval of 33 ms (milliseconds)) according to the specifications of a photographing apparatus (for example, a video camera) and the transmission band is sufficiently wide, the preceding image It is assumed that the frame is ideally transmitted until the subsequent image frame is generated. However, in practice, since the transmission interval of the image frame packet is extended, the server in the image service system in the present embodiment extends the transmission interval of the preceding image frame packet, and A moving image communication processing method is adopted in which, when the generation is not in time, the subsequent image frame is discarded and transmission is not performed.

  A moving image communication processing method according to this embodiment will be described with reference to FIGS. In the following description, the description of the same reference numerals as in FIG. 3 is omitted. 1 is a flowchart showing a video communication processing method in the server, FIG. 1A is a flowchart showing a main processing loop in the server, FIG. 1B is a flowchart showing error notification packet processing in the server, 1 (C) is a flowchart showing image frame transmission processing in the server, and FIG. 2 is a conceptual diagram of a moving image transmission processing method in the server.

  First, when the client 3 checks the received packet and detects a missing packet (missing serial number), the client 3 immediately sends an error notification packet including a missing packet notification (for example, a notification including the serial number of the missing packet) to the server 1. Send.

When the server 1 receives the error notification packet including the packet loss notification using the communication unit (S101, S102, S103, S201), the server 1 transmits the packet in accordance with the allocated transmission band as shown in FIG. The interval _Tb is appropriately extended to reliably transmit one image frame to the client 3 (S202). Note that step S104 in FIG. 1 is a process related to a packet that is not an error notification packet, and thus description thereof is omitted.

More specifically, the server 1 receives the error notification packet notified from the client 3 (for example, the serial number of the missing packet included in the error notification packet) and the server 1 itself until the error notification packet is received. A transmission band is calculated based on the amount of transmitted packets, a packet transmission interval T _b that fits within the transmission band is calculated, and the transmission interval T _b is stored in a predetermined area of the storage unit. Note that the amount of packets transmitted by the server 1 itself is updated every packet transmission and stored in the storage unit.

  Note that step S203 in FIG. 1 is a process related to another error notification packet that does not include a packet loss notification, and thus description thereof is omitted.

  Further, for example, the server 1 generates an event of transmitting an image frame by inputting a new image frame (that is, a subsequent image frame) (S105, S106), and before the image frame is transmitted, It is checked whether or not the transmission timing overlaps the transmission timing (or transmission interval) of the preceding image frame (that is, the preceding image frame is being transmitted) (S301). If the transmission timing of the subsequent image frame overlaps with the transmission timing of the preceding image frame, the process proceeds to step S303. If the transmission timing of the subsequent image frame does not overlap with the transmission timing of the preceding image frame, the process proceeds to step S302.

In step S302, according to the transmission interval T _b of the packet stored in the predetermined area, to open the transmission interval, a packet of the image frame to be transmitted and transmits via the communication unit.

In step S303, the subsequent image frame is discarded without being transmitted. For example, as in FIG. 2, when extending the transmission interval T _b of the packet of the preceding image frame F _A, since overlaps the transmission timing of the subsequent image frame F _B, without sending the image frame F _B to, discard the image frame F _B. Therefore, in FIG. 2, the image frame to be transmitted next to the image frame F _A is the image frame F _c .

  Needless to say, the procedure in the moving image communication processing method of the present embodiment can be configured by a computer program, and the program can be executed by the computer.

  Furthermore, a computer program describing the above-described moving image communication processing method may be mounted so as to access a memory or an external storage device that stores input / output data required for the moving image communication processing method.

  As described above, according to the present embodiment, image frames in streaming are thinned out, and reproduction of 30 image frames per second cannot be achieved. The situation can be avoided.

  Although the present invention has been described in detail only for the specific examples described above, it is obvious to those skilled in the art that various changes and modifications are possible within the scope of the technical idea of the present invention. Such variations and modifications are naturally within the scope of the claims.

  For example, in the present embodiment, communication is performed via a network, but a general communication line (for example, a telephone line) may be used.

  In addition, this embodiment can be applied not only to motion JPEG but also to any form that transmits every image frame.

FIG. 1A is a flowchart illustrating a moving image communication processing method in the present embodiment, FIG. 1A is a flowchart illustrating a main processing loop in a streaming server device, and FIG. 1B is a flowchart illustrating error notification packet processing in the streaming server device; FIG. 1C is a flowchart showing image frame transmission processing in the streaming server device. The conceptual diagram of the moving image transmission processing method in a streaming server apparatus. The figure which shows an example of a general image service system. The conceptual diagram which shows a mode that an image frame is divided | segmented into several packets.

Explanation of symbols

1 ... streaming server 2 ... imaging device 3 ... streaming client 4 ... display device 100 ... network _{F A, F B, F C} ... image frame _{P A, P B, P C} ... packets A ₁ ~A _3, A _n, B _{1 to} B ₃ , B _n , C _{1 to} C ₃ , C _n ... packet T _b ... packet transmission interval

Claims (2)

A storage unit, a communication unit, and an input unit for inputting captured video data,
Assigning a serial number for identifying a packet obtained by dividing the image frame according to the shooting order of the image frame constituting the moving image data input by the input unit, and dividing the packet into packets;
Store the amount of transmitted packets in the storage unit,
A video communication processing method used for a server device that transmits those packets to a client device via a communication line,
An error notification packet having a packet loss notification including a lost packet sequence number is received from the client device using the communication unit,
The transmission bandwidth is calculated based on the error notification packet and the amount of packets transmitted by the server device itself until the error notification packet is received, and the transmission interval of packets that fit within the transmission bandwidth is calculated, and the calculated Storing packet transmission intervals in the storage unit;
Before transmitting an image frame, checking whether the transmission timing of the subsequent image frame to be transmitted overlaps the transmission timing of the preceding image frame;
When the transmission timing of the subsequent image frame does not overlap with the transmission timing of the preceding image frame, the transmission interval is opened according to the transmission interval of the packet stored in the storage unit, and the packet of the subsequent image frame is passed through the communication unit. Using the send step,
If the transmission timing of the subsequent image frame overlaps with the transmission timing of the preceding image frame, discarding the subsequent image frame without transmitting the subsequent image frame;
A video communication processing method characterized by comprising:   A moving image communication processing program according to claim 1, wherein the moving image communication processing method is described as a computer program executable by a computer.

Images