JP2007166023A - Information processing apparatus and control method thereof, imaging apparatus, program, and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for compensating lost frames without impairing requirement of real time performance in case of a burst error during transmission of a moving picture. <P>SOLUTION: Ordinarily, frame information acquired at a first frame rate are sequentially transmitted to an external device. When occurrence of the burst error is detected, the information processing apparatus allows a holding means to sequentially hold frame information acquired at a second frame rate lower than the first frame rate. When no burst error is detected, the information processing apparatus allows the holding means to sequentially hold the frame information acquired at the second frame rate and transmits the frame information held in the holding means to the external device at the first frame rate in the ascending order. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a moving image transmission technique, and more particularly to a moving image transmission technique that requires real-time transmission.

  2. Description of the Related Art Conventionally, systems such as a video phone, a video conference system, and a surveillance camera system have been put to practical use for transmitting and viewing and storing real-time moving images captured by a camera to a remote place.

  In moving picture transmission of such a system, the real-time property of moving picture frame transmission is more important than the reliability of each moving picture frame. For this reason, for example, when the data transfer capability of the communication path is low, it is necessary to transmit the moving image frame in real time even if the frame rate of the moving image is lowered or the image quality itself is lowered. In addition, when an error occurs on the communication path, re-transmission is performed within a range that does not impair the real-time property, and when the real-time property is impaired, a configuration that discards the moving image frame is required.

  However, when a moving image frame is discarded, in a system in which the moving image is encoded by an image compression method that performs forward reference like MPEG, when the discarded moving image frame is a frame that is referred to later, Therefore, it becomes impossible to decode subsequent frames. In order to avoid this problem, a configuration is known in which an intra frame is generated and transmitted without forward reference when an error occurs (for example, Patent Document 1). According to this configuration, it is possible to minimize the influence on the moving image frame that is temporally after the transmission error has occurred.

  In addition, a protocol for transmitting such a real-time moving image and selecting an optimal communication method according to the performance of the communication path is known. For example, RTP / RTCP (Non-Patent Document 1), H.P. The RTP payload format of the H.261 video stream (Non-Patent Document 2) and RTSP (Non-Patent Document 3), which is a protocol for controlling the moving image stream, have been standardized and put into practical use.

  RTP (Realtime Transport Protocol) is a standard related to transmission of moving image data itself such as frame data. On the other hand, RTCP (Realtime Control Protocol) is a standard related to transmission of additional information that is useful in transmission of RTP packets. RTCP defines, for example, standards related to periodic detection and notification of quality related to communication paths such as packet loss rate and delay. RTP is generally implemented on UDP (User Datagram Protocol) instead of TCP (Transmission Control Protocol), and is used together with RTCP.

  Furthermore, with the development and diversification of communication channels, the opportunity to transmit moving images over the air has increased. For this reason, for example, it is considered that a situation in which a wireless LAN typified by 802.11x is present in the middle of a communication path such as a videophone is generalized. However, such wireless communication does not always guarantee a constant communication band, and is vulnerable to disturbances. For this reason, in wireless moving image data transmission, a so-called burst error, in which communication errors occur continuously, is more likely to occur than in wired moving image data transmission.

In order to minimize image degradation due to communication errors such as burst errors on the communication path, currently, communication packets with errors are retransmitted as necessary, intra-frame interpolation and inter-frame interpolation for moving images. Countermeasures such as performing interpolation as much as possible are taken.
Japanese Patent Laid-Open No. 7-193821 RFC1889: RTP RFC 2032: RTP Payload Format for H.261 Video Streams RFC2326: RTSP

  However, when transmitting a moving image, if a burst error occurs on the communication path, if the retransmission is performed for all of the frames in which the error has occurred, the real-time property of the moving image is greatly impaired.

  On the other hand, if transmission of a frame in which a burst error has occurred is not performed at all, it is not possible to satisfy a request for an application in which surrounding changes need to be monitored and accumulated one by one, such as moving image transmission in a monitoring system.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for compensating for missing frames within a range that does not impair the real-time requirement when a burst error occurs during moving image transmission. .

In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is,
An information processing apparatus for sending video information including a plurality of frame information to an external device,
Acquisition means for acquiring frame information;
Sending means for sequentially sending the frame information obtained at the first frame rate in the obtaining means to the external device;
Detecting means for detecting a burst error in communication with the external device;
Holding means capable of holding frame information;
When the detection unit detects the occurrence of a burst error, control is performed so that the acquisition unit sequentially holds the frame information acquired at the second frame rate lower than the first frame rate in the acquisition unit. First control means;
If no burst error is detected by the detecting means, the holding means sequentially holds the frame information acquired at the second frame rate in the acquiring means, and the frame information held in the holding means is And second control means for controlling the sending means to send at the first frame rate in chronological order.

  ADVANTAGE OF THE INVENTION According to this invention, when the burst error generate | occur | produces during moving image transmission, the technique which compensates the missing | missing of a frame within the range which does not impair the request | requirement of real time property can be provided.

  Embodiments according to the present invention will be described below in detail with reference to the accompanying drawings. However, the constituent elements described in this embodiment are merely examples, and are not intended to limit the scope of the present invention only to them.

<< First Embodiment >>
(System configuration)
First, a system configuration incorporating the information processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 12 is a diagram showing a system configuration in which the information processing apparatus according to this embodiment is incorporated.

  In FIG. 12, reference numeral 1201 denotes a transmission apparatus that transmits a moving image to a reception apparatus 1202 described later. The transmission device 1201 is realized by, for example, a monitoring camera, a personal computer (PC), a workstation (WS), a mobile phone, a PHS, a personal digital assistant (PDA), or the like. Hereinafter, the transmission apparatus 1201 will be described assuming a situation in which a captured moving image is transmitted by a surveillance camera.

  Reference numeral 1202 denotes a receiving device that receives the moving image transmitted from the transmitting device 1201 and displays the moving image on the display unit. The receiving device 1202 is realized by, for example, a PC, WS, a mobile phone, a PHS, a personal digital assistant (PDA), or the like.

  Reference numeral 1203 denotes a network in which the transmission device 1201 and the reception device 1202 are connected. The network 1203 is typically the Internet, but it may be any line that can transmit and receive data, such as a public line (analog line, ISDN, etc.), LAN, WAN, or wireless LAN, regardless of whether it is wired or wireless. It may be a simple configuration. As a communication protocol using a network, for example, TCP / IP or the like can be adopted. Hereinafter, it is assumed that the transmission device 1201 and the reception device 1202 are connected to the network 1203 via a wireless medium such as a wireless LAN.

(Hardware configuration)
Next, the hardware configuration of the transmission device 1201 and the reception device 1202 in this embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a hardware configuration in this embodiment. FIG. 1A shows a transmission side (transmission device) 1201, and FIG. 1B shows a reception side (reception device) 1202.

  In FIG. 1A, reference numeral 101 denotes an imaging unit that converts incident light into an electrical signal to acquire frame data (image data). A photoelectric sensor that converts a lens and light passing through the lens into an electrical signal. It is composed of a conversion element or the like.

  Reference numeral 102 denotes a CPU (Central Processing Unit) that controls and performs operations of the entire transmission apparatus 1201. The CPU 102 executes an application program, an operating system (OS), a control program, and the like, and performs control to temporarily store information, files, and the like necessary for executing the program in the RAM 103.

  Reference numeral 103 denotes a RAM (Random Access Memory), which is a random access memory for use as a work area or buffer for computation. Reference numeral 104 denotes a ROM (Read Only Memory), which is a read only memory storing a basic program, basic data, and the like.

  Reference numeral 105 denotes an encoding unit that performs an encoding (image compression) process on the image data converted into an electric signal in the imaging unit 101 based on a predetermined encoding method. Data transmission between the transmission device 1201 and the reception device 1202 is performed by communicating encoded image data.

  A communication unit 106 performs communication including transmission / reception of frame data with an external apparatus, and includes a network card, a wireless communication antenna, and the like.

  Reference numeral 107 denotes a real time clock (RTC) that measures time. The RTC 107 is used in processing such as synchronization of image-compressed images and sound, and addition of a time stamp to a packet transmitted on a communication path.

  Reference numeral 108 denotes a bus for transmitting and receiving data between the blocks 101 to 107.

  On the other hand, in FIG. 1B, reference numeral 111 denotes a CPU that controls and calculates the operation of the entire receiving apparatus 1202. The CPU 111 executes an application program, an OS, a control program, and the like, and performs control for temporarily storing information, files, and the like necessary for executing the program in the RAM 112.

  Reference numeral 112 denotes a RAM, which is a random access memory for use as a work area or buffer for computation. Reference numeral 113 denotes a ROM, which is a read-only memory that stores basic programs, basic data, and the like.

  Reference numeral 114 denotes a decoding unit that performs a process of decoding the encoded data received from the transmission device 1201.

  A communication unit 115 performs communication with the outside, and includes a network card, a wireless communication antenna, and the like. Reception of image data is performed via the communication unit 115.

  Reference numeral 116 denotes an RTC that measures time. The RTC 107 is used as an index or the like for synchronizing the received moving image and sound.

  Reference numeral 117 denotes a display unit for displaying a received moving image or the like.

  Reference numeral 118 denotes a bus for transmitting and receiving data between the blocks 111 to 117.

(State transition)
With reference to FIG. 2, a description will be given of communication-related states that can be taken by the transmission apparatus 1201 and the reception apparatus 1202 in this embodiment. FIG. 2 is a state transition diagram relating to the state of communication between the transmission device 1201 and the reception device 1202.

  In FIG. 2, reference numeral 201 denotes a “communication disconnection state” in which the transmission device 1201 and the reception device 1202 disconnect communication. Reference numeral 202 denotes a “transmission preparation state”, which is a state in which connection processing is being performed between the transmission device 1201 and the reception device 1202. Reference numeral 203 denotes a “moving image transmission state” in which a moving image is actually transmitted.

  The moving image transmission state 203 further includes a “normal transmission state” 203a, a “burst error state” 203b, and an “error recovering state” 203c. The normal transmission state 203a is a state in which normal image transmission is performed between the transmission device 1201 and the reception device 1202. The burst error state 203b is a state in which a transition occurs when a burst error occurs in communication between the transmission device 1201 and the reception device 1202. The error recovering state 203c is a state in which processing for recovering from a burst error is being performed.

  The state transition is performed as follows. That is, in the communication disconnection state 201, when a communication request is issued from the moving image receiving side (receiving device 1202) to the moving image transmitting side (transmitting device 1201), the state transits to the transmission preparing state 202. In addition, when communication between the transmission device 1201 and the reception device 1202 is established and preparation for transmission of a moving image is made, the state transitions to a normal transmission state 203a, which is an initial state of the moving image transmission state 203. When a burst error is detected in a state in which video transmission is normally performed (normal transmission state 203a), the state transits to the burst error state 203b. Then, in the burst error state 203b, when it is detected that the burst error is recovered, the state transits to the error recovering state 203c. Error recovery processing is performed in the error recovering state 203c, and when the processing is completed, the normal transmission state 203a is restored. Further, when the transfer of the moving image is completed, the communication is disconnected and the state transits to the communication disconnected state 201.

  The transmission device 1201 and the reception device 1202 manage the state transition corresponding to the state transition diagram as described above based on the status of communication with the external device.

(State transition in the video transmission state)
Next, state transition in the moving image transmission state 203 will be described with reference to FIG. FIG. 3 is a diagram illustrating a communication sequence in the moving image transmission state 203.

[Normal transmission status]
Consider a situation in which the transmission side (transmission device 1201) transfers the nth moving image frame to the reception side (reception device 1202) at a certain moment in the normal transmission state 203 as in step S301. In this case, the transmission device 1201 transmits the data constituting the packet by adding not only the image of the moving image frame but also the frame number. In FIG. 3, it is represented by one arrow, but it may actually be composed of a plurality of communication packets.

  Immediately after transferring the nth frame, the transmission side 1201 buffers the transmitted frame, that is, the nth frame (step S302). If the frame is encoded by an encoding method that references forward or backward, it is re-encoded into an intra frame and buffered. Hereinafter, for simplification of description, it is assumed that frame buffering is performed by re-encoding into an intra frame. Note that frame buffering is performed, for example, by storing frame data in the RAM 103 in the configuration shown in FIG.

  Upon receiving the nth frame, the receiving side 1202 returns an Ack indicating that it has been received to the transmitting device 1201 (step S303). However, Ack includes information for identifying the frame number. Upon receiving Ack, the moving image transmission side 1201 deletes the buffered nth frame from the buffer in response to the reception of Ack (step S304). Such a process is repeated while the moving image is normally transmitted.

[Burst error status]
Next, the operation when a burst error occurs will be described. Consider a situation in which, in step S321, an attempt is made to transfer the m-th moving image frame from the transmission side 1201 to the reception side 1202, but packet loss has occurred and the reception side 1202 has not been reached. In this case, at the time of step S321, the transmission side 1201 and the reception side 1202 have not yet detected the presence or absence of a transmission error. Similar to the processing in the normal distribution state 203a, the transmission device 1201 re-encodes the moving image frame into an intra frame in response to the transmission of the m-th moving image frame and performs buffering (step S322).

  Similarly, the transmission device 1201 tries to transfer the next m + 1-th moving image frame in step S323, re-encodes the frame into an intra frame, and performs buffering (step S324). In this case, since it has not recovered from the burst error state, the (m + 1) th frame loses a packet and cannot reach the receiving side 1202.

  On the other hand, the receiving side 1202 detects that the expected m-th frame data does not arrive even after a predetermined time has elapsed, and issues a retransmission request for the frame number m (step S325). Information relating to the retransmission request includes information for identifying the frame number. However, the retransmission request (step S325) also does not reach the image transmission side 1201 due to a burst error.

  Next, the moving image transmission side 1201 attempts to transfer the m + 2th moving image frame, re-encodes the frame into an intra frame, and performs buffering (step S327). This frame also does not reach the receiving side 1202 due to a burst (packet) error (step S326).

  The receiving side 1202 waits for the transfer of the moving image frame in response to the retransmission request for the mth frame, but if the moving image data of the mth frame does not arrive for a certain period, it repeatedly performs the retransmission request (steps S328, S331, S334). If the Ack for the transmitted moving image does not return for a certain period, the transmitting side 1201 also interrupts transmission of the next moving image frame to be sent in real time, and performs only the process of re-encoding into an intra frame and buffering (step S329, S330, S333). At this point, the transmission device 1201 and the reception device 1202 each transition to the burst error state 203b.

  When the frame rate of the moving image frame that the transmission side 1201 should originally transmit in real time is high at the time of transition to the burst error state 203b, when all the frames are buffered, the buffer area of the transmission device 1201 becomes full immediately. End up. For this reason, in the burst error state 203b, the transmission device 1201 performs buffering by reducing the frame rate of moving image data from the normal time. The frame rate is the number of frames acquired or transmitted per unit time.

  Further, it is also possible to monitor the buffer empty state and perform a process of thinning out the image data stored in the buffer when the buffer free space is reduced. The fact that the buffer free space has decreased can be detected based on, for example, that the buffer free space has fallen below a predetermined value. The example of FIG. 3 shows a state where the frame data of frame numbers n + 1 and n + 3 are deleted (thinned out) in step S332.

  If the video frame being buffered is an image that is forward-referenced or backward-referenced from another frame, deleting this frame makes it impossible to decode other frames that refer to this frame. . For this reason, when performing the process of thinning out the frame data, it is essential to convert the frame data into an intra frame and store it in the buffer.

[Error recovery status]
When the communication environment returns from the burst error state to the normal state, the retransmission request from the reception side 1202 to the transmission side 1201 succeeds. That is, the retransmission request arrives at the transmission side 1201 (step S341). When the retransmission request reaches the transmission apparatus 1201, the state transits to the error recovery state 203c.

  In response to the retransmission request S341 with the frame number m, the transmission side 1201 transmits the frame with the frame number m in the buffer to the reception side 1202 (step S342). Further, if it is originally (in the normal transmission state 203a), the frame number r to be transmitted in real time at this time is encoded into an intra frame without being transmitted to the receiving side 1202, and buffered (step S343).

  In step S344, the image receiving side 1202 returns Ack for the frame having the frame number m in response to the reception of the image frame having the frame number m. On the other hand, the image transmission side 1201 deletes the buffered frame number m data from the buffer in response to reception of Ack.

  As described above, the transmission side 1201 accumulates the image frames to be transmitted in real time in the buffer while sequentially transmitting from the already buffered old moving image frames to the reception side 1202. What is important here is that in the configuration according to the present embodiment, image frames that should be transmitted in real time are stored in the buffer at a frame rate sufficiently lower (slower) than the rate related to transmission / deletion of old moving image frames. Is to do. As a result, the receiving side 1202 can perform chasing playback moving image display.

  In this way, video data that has been buffered after a certain amount of time has elapsed by transmitting the already stored video frames while storing video frames that should be transmitted in real time at a sufficiently slow frame rate. Disappears. The data to be transmitted in real time can be transferred to the receiving side 1202 as it is in real time. When there is no buffered video data, the normal transmission state 203a is restored.

(Transmission preparation processing)
Next, processing executed by the transmission side 1201 and the reception side 1202 in order to realize the processing described above with reference to FIG. 3 will be described with reference to FIGS.

  FIG. 4 is a flowchart showing the flow of processing from the communication disconnection state 201 to the transmission preparation state 202 and the moving image transmission state 203. 4A is a flowchart of the moving picture transmission side (transmission apparatus) 1201, and FIG. 4B is a flowchart of the moving picture reception side (reception apparatus) 1202.

  First, each of the moving image transmission side 1201 and the moving image reception side 1202 is initialized (steps S401 and S411), and the transmission side 1201 waits until a communication request is received from the reception side 1202 (step S402).

  When the moving image receiving side 1202 starts moving image reception, it makes a moving image transmission request to the moving image transmitting side 1201 (step S412). The moving image transmission request can be configured to be triggered by, for example, an instruction input by the user of the reception device 1202 or occurrence of a timer event. Then, the process proceeds to step S413, and waits until a transmission parameter inquiry to be described later is received from the transmission side 1201.

  When the moving image transmission side 1201 receives this moving image transmission request S412 (YES in step S402), the moving image transmission side 1201 proceeds to step S403, and makes an inquiry about transmission parameters to the moving image reception side 1202. In the case of the present embodiment, the transmission parameter is a parameter relating to a moving image suitable for the moving image receiving side 1202 to perform reproduction processing, such as moving image resolution, frame rate, and moving image compression format. After inquiring the transmission parameter to the moving image receiving side 1202, the moving image transmitting side 1201 waits until receiving the transmission parameter (step S404).

  When the moving image receiving side 1202 receives the inquiry about the moving image transmission parameter (YES in step S413), the moving image receiving side 1202 returns a parameter relating to the moving image suitable for the reproduction processing on the moving image receiving side 1202 to the moving image transmitting side 1201 (step S414). Then, the transmission preparation process on the moving image receiving side 1202 is ended, and the state transits to the moving image transmission state 203.

  When receiving this parameter (YES in step S404), the moving image transmission side 1201 sets it as a transmission parameter in step S405, and transitions to a moving image transmission state 203.

(Video transmission processing)
Next, processing executed by the moving image transmission side 1201 and the moving image reception side 1202 in the moving image transmission state 203 will be described with reference to FIG. FIG. 5 is a flowchart showing a flow of processing executed by the moving image transmission side 1201 and the moving image reception side 1202 in the moving image transmission state 203. FIG. 5A shows the moving image transmission side 1201, and FIG. The flow of operations on the side 1202 is shown.

  The moving image transmission side 1201 performs the processing of steps S501 to S513 in order from the top frame for all frames constituting the moving image to be transmitted.

  The moving image transmitting side 1201 captures a moving image frame of a certain number n in the imaging unit 101 (step S501), and then compresses the moving image in the encoding unit 105 according to a preset encoding method (step S502). . Then, the encoded frame n is transmitted to the moving image receiving side 1202 (step S503).

  When the moving image receiving side 1202 receives information from the moving image transmitting side 1201 (YES in step S521), the moving image receiving side 1202 determines whether the received content is a forced communication end notification (step S522) or a moving image frame (step S523). If the received information is a moving image frame (YES in step S523), Ack corresponding to the received frame is returned to the moving image transmission side 1201 (step S524). In the case of a forced communication end notification (YES in step S522), the moving image reception process ends.

  On the other hand, if the moving image transmission side 1201 transmits the moving image frame n in step S503, and the frame n is an inter frame that refers to another frame, it is converted into an intra frame that does not refer to the frame in step S504. Here, it goes without saying that if the conversion is unnecessary, the process of step S504 is skipped. Next, the frame converted as necessary is stored in the buffer (step S505). In the normal transmission state 203a, it is necessary to secure an available buffer capacity. For this reason, when the buffer is full, that is, when there is no free space in the buffer, or when the free space of the buffer falls below a predetermined value (YES in step S506), the frames accumulated in the buffer are thinned out (step S511). . The frame thinning method is as described above.

  Next, it is determined in step S507 whether the communication error is within an allowable range. Specifically, for example, the reception status of the Ack for the previously sent frame is monitored, and it is monitored whether the Ack for the transmitted frame is returned within a certain time. Alternatively, whether or not a communication error is permitted may be determined based on the ratio of the number of Ack that has been returned to the number of transmitted frames. If the communication error exceeds the allowable range (NO in step S507), a burst error process described later is performed (step S512), and an error recovery process (chase transmission process) described later is performed when the communication path is restored (step S512). S513).

  If the communication error is within the allowable range (YES in step S507), in steps S508 and S509, processing is performed to delete the frame corresponding to Ack received from the moving image receiving side 1202 from the buffer. That is, in step S508, it is confirmed whether or not the Ack of the already sent moving image frame has been received (step S508). If there is an Ack (YES in step S508), the frame having the frame number corresponding to Ack is deleted from the buffer.

  When all the frames have been transmitted and the transmission of the moving image is finished (YES in step S510), the process ends. When all the frames have not been transmitted (NO in step S510), the process returns to step S501. .

(Burst error processing)
Next, processing in the burst error state 203b will be described with reference to FIG. FIG. 6A is a flowchart showing the flow of processing executed on the moving image transmission side 1201 and FIG. 6B is the flow of processing executed on the moving image reception side 1202. Further, the process of FIG. 6A corresponds to the burst error process (step S512) of FIG.

  As shown in FIG. 6A, the moving image transmission side 1201 performs a loop process that repeatedly executes the processes of steps S601 to S606 at a predetermined cycle. First, in step S601, a frame of a moving image is captured, that is, image data is acquired using the imaging unit 101. Next, the captured frame is converted into an intra frame (step S602) and stored in the buffer (step S603). If the buffer is full at this time, or if the free space is less than the predetermined value (YES in step S604), the frames accumulated in the buffer are thinned out in step S605, and the process proceeds to step S606. If the buffer is not full (NO in step S604), the process proceeds to step S606 without doing anything.

  In step S606, it is determined whether or not a retransmission request has been received from the previous execution of step S606 to the present time. If a retransmission request has been received (YES in step S606), the burst error state 203b is transited to the error recovering state 203c, and the burst error processing is terminated. If the retransmission request has not been received (NO in step S606), the burst error state 203b is continued, and the loop processing from step S601 to step S606 is continued. Note that the loop processing including capture is executed such that the cycle is larger than the cycle of the loop processing (steps S501 to S510 in FIG. 5) in the normal transmission state 203a. When the burst error process is completed, a follow-up transmission process in step S513, that is, an error recovery process described later is performed.

  On the other hand, as illustrated in FIG. 6B, the moving image receiving side 1202 performs a loop process that repeats the processes in steps S611 to S613 at a predetermined cycle. That is, a retransmission request is made to the moving image transmission side 1201 in step S611. Next, in step S612, it is determined whether or not information has been received from the moving picture transmission side 1201. If there is reception (YES in step S612), the process returns to step S611. If there is no reception (NO in step S612), the process proceeds to step S613. In step S613, it is determined whether the received information is a moving image frame. In the case of a moving image frame (YES in step S613), it is determined that the communication path has been restored, the burst error state 203b is transited to the error restoring state 203c, and the burst error processing is terminated. If it is not a moving image frame, the process returns to step S611 to continue the loop processing. Note that the loop processing on the moving image receiving side 1202 is executed such that the cycle is larger than the cycle of the loop processing (steps S521 to S524 in FIG. 5) in the normal transmission state 203a. As a result, the acquired frame information can be sequentially held in the buffer at a frame rate lower than normal.

(Error recovery processing)
Next, the flow of processing in the error recovery in progress state 203c will be described with reference to FIG. FIG. 7 is a flowchart showing the flow of error recovery processing. 7A shows the flow of error recovery processing on the moving image transmission side 1201, and FIG. 7B shows the flow of error recovery processing on the moving image reception side 1202. FIG. 7A corresponds to the error recovery process (chase transmission process) in step S513 in FIG.

  In step S701, the moving picture transmission side 1201 refers to the frame number m included in the retransmission request received in step S606, and extracts the image of the frame number m from the buffer. Then, the extracted frame (number m) is transmitted to the moving image receiving side 1202 (step S702).

  Next, in step S <b> 703, it is determined whether or not Ack of number m corresponding to the transmitted frame has returned from the moving image receiving side 1202. If it has returned (YES in step S703), the process proceeds to step S704. If it has not returned (NO in step S703), the process returns to step S702 to transmit the frame of number m again. In addition, when repeating the process of step S702 and step S703, it controls to perform for every predetermined period.

  In step S704, a frame corresponding to the frame number (m) included in Ack and a frame older than that are deleted from the frames accumulated in the buffer. Then, the next (frame number m + 1) frame is transmitted (step S705). Next, in step S706, it is determined whether or not it is time to perform video frame capture. If it is time to capture a moving image frame (YES in step S706), the process proceeds to step S707. If not (NO in step S706), the process proceeds to step S709. In step S707, capture, that is, acquisition of image data using the imaging unit 101 is performed. In step S708, the captured image data is stored in a buffer. Then, the process proceeds to step S709.

  It should be noted that the timing of capturing the moving image frame is such that in the error recovery state 203c, the moving image frame accumulated in the burst error state 203b is transmitted at once, and the events occurring during the transmission are captured little by little to catch up with the real time. Set to. For this reason, the capture needs to be performed at a relatively low frame rate. For example, the moving picture frame capture timing can be determined based on a period equal to or greater than the period of steps S501 to S510 in FIG.

  In step S709, it is determined whether or not the Ack of the already transmitted moving image frame has been returned. If Ack has returned (YES in step S709), the process proceeds to step S710. If Ack has not returned (NO in step S709), the process returns to step S705 and the process is repeated. In step S710, the frame corresponding to the returned Ack is deleted from the buffer. Proceed to step S711.

  If the buffer becomes empty by the above processing, the chasing process (error recovery process) in step S513 ends. That is, in step S711, it is determined whether or not the buffer is empty. If it is empty (YES in step S711), the error recovery process is terminated. If it is not empty (NO in step S711), the process returns to step S705 to end the process. In step S711, it is determined whether or not the amount of data held in the buffer is below a predetermined value. If it is below, it is considered that the buffer is empty (YES in step S711), and the error recovery process is terminated. You may do it. At this time, the state transits from the error recovery state 203c to the normal transmission state 203a.

  In the middle of the above processing, if a retransmission request is further received, the process may return to step S701, and the frame corresponding to the retransmission request and the previous frame may be deleted (step S704).

  On the other hand, the moving image receiving side 1202 executes processing equivalent to the normal transmission state 203a of FIG. That is, in step S721, the process waits until information is received from the moving picture transmission side 1201, and if received (YES in step S721), the process proceeds to step S722. In step S722, it is determined whether the received information is a moving image frame. If the received information is a moving image frame (YES in step S722), the process proceeds to step S723. If not (NO in step S722), the process returns to step S721. In step S723, Ack corresponding to the received moving image frame is returned to the moving image transmission side 1201.

  As described above, in the configuration according to the present embodiment, when a burst error occurs, an event that occurred during the burst error is captured and buffered at a frame rate lower than normal, and buffered at the time of error recovery. Transfer video data at normal frame rate. As a result, it is possible to realize chasing playback. Particularly in monitoring applications, events that occur during burst errors can be recorded and played back even though they are broken up. In addition, when no burst error has occurred, it is possible to transmit a good real-time monitoring image as before.

  In the present embodiment, a configuration has been described in which a captured moving image frame is periodically buffered during a burst error. However, the present invention is not limited to this. That is, for example, the captured moving image frame and the previous frame may be compared to detect whether or not there is motion, and buffering may be performed only when motion is detected. For example, the motion can be detected based on the fact that the difference between the captured frame and the previous frame is obtained and the difference is equal to or less than a predetermined threshold. Note that the buffering performed only when motion is detected can be configured to be executed in at least one of the burst error state 203b and the error recovering state 203c.

  In the present embodiment, when a moving image frame is buffered on the transmission side 1201, it is converted into an intra frame. However, it is configured not only to be converted into an intra frame but also to be reduced and buffered. Also good. Thereby, the data to be buffered can be reduced.

  In this embodiment, Ack is returned for each moving image frame. However, Ack corresponding to a plurality of moving image frames may be returned collectively. Thereby, the transmission amount of a communication path can be reduced.

<< Second Embodiment >>
The second embodiment according to the present invention will be described below with reference to FIGS. In the present embodiment, a configuration example using RTP / RTCP recommended as an international standard will be described. Note that the description of the same components as in the first embodiment, such as the system configuration, hardware configuration, and possible states, will be omitted.

(State transition in the video transmission state)
FIG. 8 is a diagram illustrating a communication sequence in moving image transmission using RTP / RTCP. In the present embodiment, communication using RTP / RTCP is implemented by data transmission using UDP instead of TCP. In transmission of data by UDP, unlike TCP, it is not defined that an Ack is returned or an automatic retransmission is performed for a received packet. For this reason, UDP communication is suitable for real-time video transmission.

  The transmission side 1201 stores the moving image frame in the RTP packet together with the sequence number corresponding to the moving image frame, and transmits it to the reception side 1202. Further, information such as a time stamp for detecting a transmission delay and the number of transmitted RTP packets is stored in the RTCP packet and transmitted to the receiving side 1202. The receiving side 1202 stores information such as the loss rate of the received RTP packet and the number of lost packets in the RTCP and sends it to the transmitting side 1201.

[Normal transmission status]
Consider a situation where the transmission side 1201 is transmitting moving image data to the reception side 1202 in the normal transmission state 203a (steps S801 to S809). The transmission side 1201 periodically transmits RTCP packets to the reception side 1202 (step S801). Packets that are RTCP transmitted periodically from the transmission side 1201 to the reception side 1202 are called sender reports (hereinafter referred to as SR). RTCP (SR) stores information such as a time stamp at the time of transmission and the number of RTP packets transmitted after transmitting the previous RTCP (SR). On the other hand, the receiving side 1202 calculates a delay time, a packet loss rate, and the like according to the reception of RTCP (SR), and sends an RTCP packet including these pieces of information to the transmitting side 1201 as a receiver report (hereinafter referred to as RR). To do. Details of SR and RR are disclosed in Non-Patent Document 1.

  Also, the transmission side 1201 stores a frame captured and acquired at a predetermined timing in an RTP packet and transmits it to the reception side 1202 (S802, S804, S806). Then, the transmitted image frame is converted into an intra frame and buffered (S803, S805, S807). In the first embodiment, the receiving side 1202 returns an Ack every time a frame is received. However, in the present embodiment, an Ack is not returned because the embodiment conforms to the RTP standard.

  Processing on the transmission side 1201 that performs such an operation will be described with reference to FIG. FIG. 9 is a flowchart showing a flow of operations on the transmission side 1201. As illustrated in FIG. 9, the transmission side 1201 executes the processing of steps S <b> 901 to S <b> 915 for all frames to be transmitted.

  First, one frame of a moving image is captured at a predetermined timing in step S901, and compressed in a predetermined moving image compression format (for example, MPEG2) (step S902). Next, an RTP packet is generated based on the information including the compressed frame data, and an RTP header is added and transmitted to the receiving side 1202 (step S903).

  If the transmitted frame is not an intra frame, it is converted to an intra frame in step S904 and stored in a buffer in step S905. If the transmitted frame is an intra frame, the process of step S904 is not necessary. It is determined whether or not the buffer is full during buffering (including the case where the free space is lower than a predetermined value) (step S906). If it is full (YES in step S906), the frames accumulated in the buffer are thinned out (step S907), and the process proceeds to step S908. If not full (NO in step S906), the process proceeds to step S908.

  In step S908, it is determined whether it is time to transmit RTSP (SR). When it is time to send RTSP (SR) (YES in step S908), a sender report packet is generated and transmitted to the receiving side 1202 (step S909), and the process proceeds to step S910. If it is not time to send RTSP (SR) (NO in step S908), the process proceeds to step 910.

  In step S910, it is determined whether RTSP (RR) has been received. If received (YES in step S910), the process advances to step S912. In step S912, all frame data before the frame data corresponding to the received packet is deleted from the moving image frames stored in the buffer. For example, in the example of FIG. 8, in step S809, the moving image frame data stored in the buffer is deleted up to the moving image frame corresponding to RTCP (RR) received in step S808 just before. Then, the process proceeds to step S915.

  The transmission side 1201 executes the above process for all frames that transmit. In step S915, it is determined whether or not the transmission process for all frames has been completed. If the transmission process has been completed (YES in step S915), the moving image transmission process is terminated. If not completed (NO in step S915), the process returns to step S901 to repeat the process.

[Burst error status]
Next, processing when a burst error occurs immediately after RTSP (SR) is transferred from the transmission side 1201 to the reception side 1202 in step S821 in FIG. 8 will be described.

  Initially, the transmission side 1201 transfers the moving image frame to the receiving side 1202 by RTP as in the normal transmission state 203a (steps S822, S824, S827,...), And continues buffering of the moving image frame (steps S823, S825). , S828 ..). After this, when a burst error occurs, the RTCP packets do not reach each other. Each of the transmission side 1201 and the reception side 1202 can estimate the arrival time of the packets, but if the packet arrival time does not reach the packet arrival time, the transmission side 1201 and the reception side 1202 respectively determine if a transmission error has occurred. To do.

  In this case, on the transmission side 1201, even if a new video frame is captured, it is converted into an intra image and buffered without being transmitted (steps S834 and S837). In the burst error state 203b, if the moving image is captured and buffered at the same frame rate as that in the normal transmission state 203a, the buffer becomes full immediately, so the buffering is performed at a reduced frame rate. Further, it is monitored whether or not the buffer is full, and when the buffer is full (or when the free space falls below a predetermined value), a process of thinning out already accumulated video frames is performed (step S836). .

  In addition, although the RTCP (RR) is periodically transmitted to the receiving side 1202, the RTCP (RR) corresponding to the RTCP (SR) that has finally arrived from the transmitting 1201 side is continuously transmitted (steps S826, S832, and S835). ).

  The above processing is performed by executing the burst error processing in step S913 in FIG. 9, but this processing is the same as the processing in the first embodiment, and thus the description thereof is omitted.

[Error recovery status]
Here, when the RTCP (RR) transmission is successful in step S841, the state transits to the error recovery state 203c. In addition, the transmission side 1201 starts transfer sequentially from the next buffered moving image frame of the transmission side 1201 packet corresponding to the received RTCP packet. In the example of FIG. 8, RTCP (RR) in step S841 is a packet corresponding to RTCP (RR) in step S821, and there is a high possibility that the moving image frames after S821 cannot be transmitted. For this reason, the transmission side 1201 transmits the actually buffered moving image frame (frame number m) to be transmitted in step S822 in step S842. Thereafter, the transmission side 1201 sequentially sends the buffered moving image frames (steps S844, S845, S847,...).

  Also during this time, the transmission side 1201 continues buffering while capturing moving images at a low frame rate (S843, S848...). While moving image frames to be transmitted remain in the buffer, only buffering is performed without transmission when captured. Also, after returning from the burst error in step S841, RTCP is transmitted and received in the same way as during normal transmission. Then, the moving image transmission side 1201 deletes the buffered transmitted moving image frame corresponding to the RTCP (RR) that has received the RTCP (RR) (steps S851 and S858). When there is no moving image frame to be transmitted immediately by RTP among the images already buffered, the normal transmission state 203a is restored.

  The above processing is performed by executing the error recovery processing in step S914. Since this processing is the same as the processing in the first embodiment, description thereof is omitted.

  As described above, in the present embodiment, processing similar to that of the first embodiment is performed using RTP / RTCP, which is a standard protocol for moving image transmission. That is, when a burst error occurs for transmission, the captured images are transmitted in the oldest order when the burst error is recovered, and the moving image transmission is controlled so as to catch up in real time.

  Thus, as in the first embodiment, when the communication state is good, moving image transmission near real time can be performed, and at the same time, even when a burst error occurs, image omission can be minimized. That is, even when a burst error occurs, it is possible to reduce the loss of the image as much as possible by buffering the moving image and performing transfer at the time of the burst error when returning from the burst error.

<< Third Embodiment >>
Next, a third embodiment according to the present invention will be described with reference to FIG. In the second embodiment, a configuration example using RTP / RTCP recommended as an international standard has been described, but in this embodiment, a configuration example using ARQ (Automatic Repeat reQuest) will be described.

  Since the system configuration, device configuration, possible states, and the like are the same as those described in the first embodiment, the description thereof is omitted.

  FIG. 10 is a diagram illustrating a communication flow in moving image transmission using ARQ in addition to RTP / RTCP. ARQ is related to a protocol that automatically designates a frame number and makes a retransmission request separately from RTP / RTCP. The communication flow shown in FIG. 10 is similar to FIG. 8 referred to in the second embodiment, but the present embodiment is different in that a request for retransmission of a moving image frame is clearly made using ARQ. .

  In FIG. 10, for example, consider a situation in which an RTP packet loss happens to occur for a frame with the frame number n + 2 in the normal transmission state 203a (step S1006). However, this packet loss is not due to a burst error.

  In this case, the receiving side 1202 issues a retransmission request (NACK) for the (n + 2) th frame in response to the occurrence of a predetermined timeout event, and sends it to the transmitting side 1201 (step S1008). However, this retransmission request is based on ARQ. The transmission side 1201 retransmits the (n + 2) th frame to the reception side 1202 in response to receiving this retransmission request. The receiving side 1202 may issue a retransmission request every time a packet error is found, or may issue a retransmission request as necessary, not for all errors.

  Next, consider a situation where a burst error has occurred. In this case, all of RTP and RTCP (SR) from the transmission side 1201 and RTCP (RR) and NACK from the reception side 1202 do not reach the communication partner. In such a situation, the receiving side 1202 continues to transmit a retransmission request periodically (steps S1029 and S1035). On the other hand, the transmission side 1201 lowers the capture frame rate and buffers the data of the moving image frame as in the first and second embodiments (steps S1023, S1025, S1028...). The processing for thinning out frames when the buffer free space becomes small is the same as in the first and second embodiments.

  When returning from the burst error and the retransmission request reaches the transmission side 1201 in step S1041, the transmission side 1201 retransmits the moving picture frame of the frame number m related to the retransmission request to the reception side 1202. Thereafter, the buffered moving image frames are sequentially transmitted to the receiving side 1202, and at the same time, real-time moving image frames are periodically captured. The captured image is not transmitted immediately until the buffer becomes empty, but only buffering is performed (steps S1043, S1048...).

  Similar to the first and second embodiments, if the frequency of capturing and buffering is lower than the case of transferring the moving image frames accumulated in the buffer at the time of error recovery, there is no buffered moving image frame at a certain time. . At this point, the normal transmission state 203a is restored.

  As described above, in the configuration according to the present embodiment, it is possible to identify what should be directly retransmitted by the retransmission request (NACK) without waiting for the arrival of the RTCP packet and further interpreting the contents of the RTCP. For this reason, a processing load is reduced as compared with the configuration according to the second embodiment, and a system with a better response to a change in state is constructed than RTCP in which the frequency of arrival of packets is low.

<< Fourth Embodiment >>
Next, a fourth embodiment according to the present invention will be described with reference to FIG. In the first to third embodiments, at the time of recovery from a burst error, all the images previously captured at the time of the burst error are transferred, and then the captured image is transmitted after the recovery from the burst error. Then, when catching up in real time, that is, when the buffer becomes empty, processing for returning to the normal transmission state 203a is performed. On the other hand, in the present embodiment, both the transmission of the image accumulated at the time of the burst error and the transmission of the image captured in the error recovery state 203c are performed in the error recovery state 203c.

  FIG. 11 is a sequence diagram showing a communication flow in the present embodiment. Since the communication flow in the normal transmission state 203a and the burst error state 203b is the same as that in the third embodiment, description thereof is omitted. In the present embodiment, as in the third embodiment, the configuration using RTP / RTCP and ARQ is exemplarily described, but the present invention is not limited to this. For example, as in the first embodiment, the configuration according to this embodiment may be applied to a configuration that does not use these protocols.

  Consider a situation (step S1141) in which the transmission side 1201 receives a retransmission request sent from the reception side 1202 after returning from a burst error. In this case, the transmission side 1201 recognizes that it has recovered from the burst error, and resumes the transmission of the moving image.

  When transmitting a moving image, if it happens to be the timing when the image is captured, the captured moving image frame (frame number r) is transmitted on the RTP packet. (Step S1142). Next, the video frame buffered at the time of the burst error is transmitted to the receiving side 1202 at the timing permitted by the communication path or the receiving side 1202 (steps S1144 and S1145). Thereafter, this is repeated until all the moving image frames buffered at the time of the burst error are transmitted.

  When the reception side 1202 receives a real-time moving image frame (frame number r + i) on the RTP packet, it displays it on the entire screen. On the other hand, when a video frame at the time of a burst error is received, a real-time video may be displayed in a small picture-in-picture manner, or may not be displayed, and a large-capacity storage device (such as a hard disk device) ( (Not shown) may be configured to accumulate.

  The receiving side 1202 can be configured to distinguish between a real-time moving image frame and a moving image frame at the time of a burst error based on the frame number of the received frame. For example, refer to the frame number acquired after error recovery, and if this number is consecutive to the frame number acquired immediately before the burst error occurred, it is determined as a video frame at the time of burst error, otherwise it is real time It can be determined that the video frame is a correct one. Note that the frame number obtained after the error is restored is referred to, and the determination of whether or not this number is continuous with the frame number obtained immediately before the occurrence of the burst error is that the difference between these numbers is equal to or smaller than a predetermined value. Can be determined based on whether or not.

  In this way, a system that monitors real-time moving images as much as possible, and displays a large-capacity storage device that cannot be displayed in real time due to a burst error, and displays it according to a user's instruction input, etc. Depending on the application, the requirements can be met. For example, the requirements can be satisfied in the usage of the monitoring system.

  Further, in the present embodiment, in the process for recovering from a burst error, a real-time moving image frame and a previously buffered moving image frame are separately transmitted for each RTP packet. Absent. For example, a real-time moving image frame and a past moving image frame may be divided and combined, or a picture-in-picture image may be generated and transmitted. Thereby, the receiving side 1202 can display a real-time moving image and a moving image at the time of a burst error.

<< Other Embodiments >>
The exemplary embodiments of the present invention have been described in detail above. However, the present invention can take embodiments as, for example, a system, apparatus, method, program, or storage medium. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  The present invention can also be achieved by supplying a program that realizes the functions of the above-described embodiment directly or remotely to a system or apparatus, and the computer of the system or apparatus reads and executes the supplied program code. Including the case where it is achieved.

  Therefore, since the functions of the present invention are implemented by a computer, the program code installed in the computer is also included in the technical scope of the present invention. That is, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include the following. Namely, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-) R) and the like are included.

  In addition, the following types of programs may be considered. That is, it is also possible to connect to a homepage on the Internet using a browser of a client device and download a computer program according to the present invention or a compressed file including an automatic installation function from the homepage to a recording medium such as an HD. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  The following supply forms are also conceivable. That is, first, the program according to the present invention is encrypted, stored in a storage medium such as a CD-ROM, and distributed to users. Further, the present invention allows a user who has cleared a predetermined condition to download key information to be decrypted from a homepage via the Internet, execute a program encrypted by using the key information, and install the program on a computer. The structure which concerns on is implement | achieved. Such a supply form is also possible.

  In addition, the following realization modes in which the functions of the above-described embodiments are realized by the computer executing the read program are also assumed. In other words, based on the instructions of the program, the OS running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  Further, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the above-described embodiment is also based on the instructions of the program. The function is realized. That is, a CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a figure showing the hardware constitutions in this embodiment. It is a figure showing the state transition of communication in this embodiment. It is a figure showing the sequence of communication in 1st Embodiment. It is a figure showing the flow of a process from the start in 1st Embodiment until it will be in the state in communication transmission. It is a figure showing the flow of a process in the state in communication transmission in 1st Embodiment. It is a figure showing the flow of a process in the burst error state in 1st Embodiment. It is a figure showing the flow of a process in the error recovery state in 1st Embodiment. It is a figure showing the sequence of communication in 2nd Embodiment. It is a figure showing the flow of a process in the state in communication transmission in 2nd Embodiment. It is a figure showing the sequence of communication in 3rd Embodiment. It is a figure showing the sequence of communication in 4th Embodiment. It is a figure showing the system configuration | structure in this embodiment.

Claims (18)

An information processing apparatus for sending video information including a plurality of frame information to an external device,
Acquisition means for acquiring frame information;
Sending means for sequentially sending the frame information obtained at the first frame rate in the obtaining means to the external device;
Detecting means for detecting a burst error in communication with the external device;
Holding means capable of holding frame information;
When the detection unit detects the occurrence of a burst error, control is performed so that the acquisition unit sequentially holds the frame information acquired at the second frame rate lower than the first frame rate in the acquisition unit. First control means;
If no burst error is detected by the detecting means, the holding means sequentially holds the frame information acquired at the second frame rate in the acquiring means, and the frame information held in the holding means is An information processing apparatus comprising: a second control unit configured to control the transmission unit to transmit the data at the first frame rate in chronological order.   The information processing apparatus according to claim 1, further comprising: a thinning unit that thins out the frame information held in the holding unit.   The information processing apparatus according to claim 2, wherein the thinning unit performs processing when the free space of the holding unit falls below a predetermined value.   When the amount of data held by the holding unit falls below a predetermined value due to the sending under the control of the second control unit, the second control unit sends the frame information acquired by the acquiring unit to the frame information acquired by the acquiring unit. The information processing apparatus according to any one of claims 1 to 3, wherein the information is transmitted to the external device in order by the sending means without being held by the holding means.   The information processing apparatus according to claim 1, wherein the acquisition unit acquires the plurality of pieces of frame information by imaging. At least one of the first control means and the second control means is:
In the plurality of pieces of frame information acquired by the acquisition unit, when the difference between the continuous frame information is equal to or less than a predetermined value, the holding unit holds one of the continuous frame information. The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. Further comprising a conversion means for converting the resolution of the frame information;
7. At least one of the first control means and the second control means causes the holding means to hold the acquired frame information after reducing the acquired frame information by the conversion means. The information processing apparatus according to claim 1. An information processing apparatus for sending video information including a plurality of frame information to an external device,
Acquisition means for acquiring frame information;
Sending means for sequentially sending the frame information acquired by the acquiring means to the external device;
Detecting means for detecting a burst error in communication with the external device;
Holding means capable of holding frame information;
A first control unit configured to control the holding unit to sequentially hold the frame information acquired by the acquiring unit when occurrence of a burst error is detected by the detecting unit;
If no burst error is detected in the detection means, the frame information held in the holding means is read out in the oldest order and synthesized based on the read frame information and the frame information acquired in the acquisition means An information processing apparatus comprising: second control means for generating frame information and controlling the sending means to send the combined frame information. An information processing apparatus for sending video information including a plurality of frame information to an external device,
Acquisition means for acquiring frame information;
Sending means for sequentially sending the frame information acquired by the acquiring means to the external device;
Detecting means for detecting a burst error in communication with the external device;
Holding means capable of holding frame information;
A first control unit that controls the holding unit to sequentially hold the frame information acquired by the acquiring unit when occurrence of a burst error is detected by the detecting unit;
When no burst error is detected in the detection means, the frame information held in the holding means is read in chronological order, and the read frame information and the frame information acquired in the acquisition means are mixed. And second control means for controlling the sending means to send the information.   The information processing apparatus according to claim 9, wherein the second control unit causes the sending unit to alternately send the read frame information and the acquired frame information.   The information processing apparatus according to claim 1, wherein the transmission of the frame information is performed based on RTP.   The information processing apparatus according to claim 1, wherein the detection unit performs the detection based on presence or absence of a communication response by the external apparatus.   The information processing apparatus according to claim 1, wherein the detection unit performs the detection based on presence / absence of communication using the RTCP with the external apparatus. An information processing apparatus that receives frame information from an external device and displays the frame information,
Receiving means for receiving the frame information and a serial number of the frame information from the external device;
Display means for displaying frame information;
Determining means for determining whether the received frame information corresponds to either the first frame information or the second frame information based on the serial number;
If the received frame information is determined to be the first frame information by the determining means, the received frame information is converted to the first resolution. If the received frame information is determined to be the second frame information, the received frame information is converted to the second resolution. An information processing apparatus comprising: display control means for displaying on the display means. An imaging device for sending moving image information including a plurality of frame information to an external device,
Acquisition means for imaging and acquiring frame information;
Sending means for sequentially sending the frame information obtained at the first frame rate in the obtaining means to the external device;
Detecting means for detecting a burst error in communication with the external device;
Holding means capable of holding frame information;
When the detection unit detects the occurrence of a burst error, control is performed so that the acquisition unit sequentially holds the frame information acquired at the second frame rate lower than the first frame rate in the acquisition unit. First control means;
If no burst error is detected by the detecting means, the holding means sequentially holds the frame information acquired at the second frame rate in the acquiring means, and the frame information held in the holding means is An image pickup apparatus comprising: a second control unit configured to control the transmission unit to transmit at the first frame rate in the order of oldness. A control method of an information processing apparatus that includes a holding unit capable of holding frame information and sends moving image information including a plurality of frame information to an external device,
An acquisition process for acquiring frame information;
A sending step of sequentially sending the frame information acquired at the first frame rate to the external device in the acquiring step;
A detection step of detecting a burst error in communication with the external device;
When occurrence of a burst error is detected in the detection step, control is performed so that the holding unit sequentially holds the frame information acquired at the second frame rate lower than the first frame rate in the acquisition step. A first control step;
If no burst error is detected in the detecting step, the holding unit sequentially holds the frame information acquired at the second frame rate in the acquiring step, and the frame information held in the holding unit is A control method for controlling the information processing apparatus, comprising: a second control step of controlling transmission at the first frame rate in chronological order.   A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 14.   A computer-readable storage medium storing the program according to claim 17.

Images