JP6119765B2 - Information processing apparatus, information processing system, information processing program, and moving image data transmission / reception method - Google Patents

Description

  The present invention relates to moving image data transmission / reception technology.

  Currently, mobile terminal devices such as smartphones (hereinafter referred to as mobile terminals) that can be taken out and driven by a battery can shoot videos with a built-in camera and view them on the mobile terminal. Some have graphic performance that can be done. Such a mobile terminal has a video playback function and an Internet connection function, and can view the content of a video site on the Internet. In addition, mobile terminals that can view personal photos and videos stored in an auxiliary storage device of a personal computer (hereinafter referred to as a personal computer) in the home via the Internet are provided.

  By the way, at the time of transmission, a moving image is transmitted as a different stream by separately encoding a video frame and an audio frame. In this case, the audio and the video may be reproduced with a shift when the moving image is viewed. The adjustment of the video frame and the audio frame so that the viewer can feel as if he / she is speaking without feeling uncomfortable when he / she sees the person displayed on the screen is called lip sync adjustment.

  As examples of techniques in which video frames and audio frames are individually encoded, transmitted as different streams, and reproduced at a transmission destination, there are first to fifth techniques. As a first technique, there is a receiving apparatus that absorbs a difference in clock frequency between an encoder side and a decoder side and lip-syncs the audio frame output timing with the video frame output timing. The receiving apparatus receives a plurality of encoded video frames sequentially attached with video time stamps based on a reference clock on an encoder side and a plurality of encoded audio frames sequentially attached with audio time stamps based on a reference clock. Decoding means for decoding is included. Further, the receiving apparatus has storage means for storing a plurality of video frames and a plurality of audio frames obtained as a result of decoding the encoded video frame and the encoded audio frame by the decoding means. The receiving device also has a calculation means for calculating a time difference caused by a difference between the clock frequency of the reference clock on the encoder side and the clock frequency of the system time clock on the decoder side. Further, the receiving apparatus adjusts video frame output timing when sequentially outputting a plurality of video frames in units of frames based on an audio frame output timing when sequentially outputting a plurality of audio frames in units of frames according to a time difference. Have

  As a second technique, the lip sync control apparatus includes first input means for inputting an encoded audio signal that includes an audio reference signal input at a predetermined timing. Furthermore, the lip sync control device includes a second input means for inputting an encoded video signal including a video reference signal input at the same timing as the audio reference signal, and an audio signal input by the first input means. First decoding means for decoding. The lip sync control device further includes second decoding means for decoding the video signal input by the second input means, and an audio reference signal included in the audio signal decoded by the first decoding means. . The lip sync control device further includes a time shift detection unit that detects a time shift amount between the video reference signal included in the video signal decoded by the second decoding unit. Further, the lip sync control device controls the audio signal and the video signal, which are earlier in time relative to each other, to be output after being delayed by the amount of time shift based on the detection result of the time shift detection means. And a control means.

  A third technique is an image communication apparatus that communicates video and audio via a network, and includes a control unit that controls a transmission rate from an application layer according to a reception state on the video and audio reception side. There is an image communication device. The image communication apparatus further includes transmission means for transmitting video and audio at a transmission rate controlled by the control means.

  As a fourth technique, there is a method in which many packets that have a small effect on quality due to discard are discarded when a packet is discarded, and packets that have a large effect on quality due to discard are not discarded as much as possible.

  As a fifth technique, the same time information is inserted as a digital watermark into each of the synchronized digital video and digital audio, and the time information is extracted from each of the digital video and digital audio after the digital watermark is out of synchronization. Then, the time information of the digital video and the time information of the digital audio are compared, and at least one of the digital video and the digital audio is delayed so that they match.

JP 2005-102192 A JP 2008-131591 A JP-A-10-164533 JP-A-4-362932 JP 2003-259314 A

  Each of the first to fifth techniques is a technique in which video frames and audio frames are individually encoded and transmitted as different streams and reproduced at a transmission destination.

  However, even when the first to fifth technologies are used, the image quality at the time of reproduction is deteriorated in the network environment where the transmission rate is lower than the reproduction rate of the moving image as compared with the original image quality of the data before transmission.

  Accordingly, in one aspect, an object of the present invention is to improve the reproduction quality of moving image data transmitted from a storage destination via a network having a narrow bandwidth.

  The first information processing apparatus according to one aspect includes a storage unit, a monitoring unit, a deletion unit, a frame information generation unit, and a transmission unit. The storage unit stores first video data and audio data including synchronization information associated with the first video data. The monitoring unit monitors the state of the communication network. The deletion unit determines, from the first frame rate indicating the first number of frames per unit time of the first video data, the audio level of the corresponding audio data in the first video data according to the monitoring result. The second video data having a second frame rate lower than the first frame rate is generated by deleting consecutive video frames having a value equal to or less than a predetermined threshold. The frame information generation unit generates frame information regarding the deleted frame. The transmission unit transmits the second video data and frame information.

  The second information processing apparatus includes a reception unit, a complementary image generation unit, and a video data generation unit. From the first frame rate indicating the first number of frames per unit time of the first video data, the receiving unit has the audio level of the corresponding audio data of the first video data equal to or less than a predetermined threshold. Second video data in which consecutive video frames are deleted to a second frame rate smaller than the first frame number and frame information related to the deleted frames are received. The complementary image generation unit generates a complementary image that complements the deleted frame image using the frame information. The video data generation unit generates video data of the first frame rate using the complementary image and the second video data.

  According to the information processing system according to the present embodiment, it is possible to improve the reproduction quality of moving image data transmitted from a storage destination via a network.

An example of a block diagram of an information processing system concerning this embodiment is shown. 1 shows an example of the configuration of an information processing system according to the present embodiment. An example of the structure of the meta-information of video data is shown. The figure for demonstrating the mode of the deletion and restoration | reconstruction of a frame is shown. The figure which shows an example of a complement frame production | generation process is shown. An example of a method for creating a complementary frame by discriminating an object having a large movement amount will be described. The figure for demonstrating adjustment of the frame rate in the transmission terminal of moving image data is shown. 6 shows a flowchart of a frame reduction process. Details of an operation flow for determining a frame to be deleted based on the audio level of each frame in the moving image data stored in the work buffer are shown. The figure for demonstrating the decoding process of a receiving terminal is shown. It is a flowchart of frame reconstruction by a receiving terminal. The operation | movement flowchart of the production | generation process of the complementary frame with respect to the deletion frame in a receiving terminal is shown. An example of the sequence diagram (the 1) of the information processing system concerning this embodiment is shown. An example of the sequence diagram (the 2) of the information processing system concerning this embodiment is shown. An example of the structure of the server in this embodiment is shown. 2 shows an example of a hardware configuration of a server or a personal computer according to the present embodiment. An example of the hardware constitutions of the mobile terminal which concerns on this embodiment is shown. An example of the structure of the information processing system in this embodiment (modification) is shown.

  FIG. 1 is an example of a functional block diagram of the information processing system according to the present embodiment. The first information processing apparatus 1 includes a storage unit 2, a monitoring unit 3, a deletion unit 4, a frame information generation unit 5, and a transmission unit 6.

  The storage unit 2 stores first video data and audio data including synchronization information associated with the first video data. The monitoring unit 3 monitors the state of the communication network.

  The deletion unit 4 determines the audio of the corresponding audio data from the first frame rate based on the first frame rate indicating the first number of frames per unit time of the first video data according to the monitoring result. Second video data having a second frame rate lower than the first frame rate is generated by deleting consecutive video frames having a level equal to or lower than a predetermined threshold. In addition, the deletion unit 4 deletes any of the consecutive frames whose similarity is equal to or greater than a predetermined threshold between consecutive frames.

The frame information generation unit 5 generates frame information regarding the deleted frame.
The transmission unit 6 transmits the second video data, audio data, and frame information.

  The second information processing device 7 includes a reception unit 8, a complementary image generation unit 9, and a video data generation unit 10.

  From the first frame rate indicating the first number of frames per unit time of the first video data, the receiving unit 8 determines that the audio level of the corresponding audio data of the first video data is equal to or lower than a predetermined threshold. The second video data in which a certain continuous video frame is deleted and the second frame rate is smaller than the first frame number and the frame information related to the deleted frame are received. The receiving unit 8 further receives audio data including synchronization information associated with the first video data.

  The complementary image generation unit 9 generates a complementary image that complements the deleted frame image using the frame information. Further, the complementary image generation unit 9 generates a complementary image by duplicating the frame immediately before the deleted frame. In addition, the complementary image generation unit 9 uses the frames before and after the deleted frame to determine an object whose movement amount included in the deleted frame is equal to or greater than a predetermined threshold, and an area for displaying the object is The complementary image is generated by duplicating the area indicating the object of the non-deleted frame immediately after the deleted frame.

  The video data generation unit 10 generates video data of the first frame rate using the complementary image and the second video data. Further, the video data generation unit 10 uses the frame information to insert a complementary image at the position of the deleted frame of the second video data to generate video data of the first frame rate. In addition, the video data generation unit 10 uses the synchronization information to synchronize the complementary image corresponding to the deleted frame and the audio data corresponding to the deleted frame to generate video data at the first frame rate. To do.

  With such a configuration, the transmission rate or frame rate of moving image data can be changed according to the bandwidth of the network. Therefore, the data amount per unit time of streaming data flowing through the network can be reduced. Further, it is possible to prevent a delay of the video frame with respect to the audio frame. In addition, since the transmission amount of the video frame is larger than that of the audio frame, it is possible to prevent the loss of the video frame that occurs when the video frame does not reach within a certain time during the video decoding process of the mobile terminal.

  Furthermore, even if video data is received via a network environment where the transmission rate is lower than the video playback rate, and the receiver does not have enough buffer for video playback, the original frame of video data before transmission It becomes possible to reproduce moving image data at a rate.

  Also, in order to determine the video frame to be deleted based on the audio level, even when the deleted video frame is restored by copying the undeleted video frame before and after the deleted frame, the viewer's A sense of incongruity can be suppressed. Compared to generating complementary frames by synthesizing previous and next video frames, the amount of computation for generating complementary frames can be reduced, and the difference between video and audio depending on the creation time of complementary frames can be reduced. Can be prevented.

  In addition, when a complementary frame corresponding to the deleted video frame is generated, the time stamp of the generated complementary frame is synchronized with the time stamp of the audio frame corresponding to the video frame, so that the video data and Audio data can be synchronized. As a result, it is possible to minimize the difference between video reproduction and audio reproduction in moving image data transmitted from a storage destination via a network with a narrow bandwidth. Furthermore, the memory area for lip sync adjustment can be reduced, and the calculation load can be reduced. For example, in the video memory or the audio memory, the lip sync adjustment may not be performed by delaying the video data or the audio data. Therefore, a memory area for lip sync and a GPU (Graphics Processing Unit) that can withstand the video processing can be reduced.

  An example of the system configuration of this embodiment will be described. FIG. 2 shows an example of the configuration of the information processing system according to the present embodiment.

  As shown in FIG. 2, the information processing system includes a personal computer 33, a server 31, and a mobile terminal 35. The personal computer 33 and the server 31, and the server 31 and the mobile terminal 35 are connected via a communication carrier 34 via a network. The server 31 is an example of the first information processing apparatus 1. The mobile terminal 35 is an example of the second information processing device 7.

  The information processing system includes a personal computer 33, a server 31, and a mobile terminal 35. The personal computer 33 stores moving image data. The server 31 receives the moving image data from the personal computer 33, and deletes and transfers a plurality of video frames from the received moving image data. The mobile terminal 35 can reproduce the moving image data received from the server 31.

  The personal computer 33 stores moving image data accessed from the mobile terminal 35 via the server 31. The personal computer 33 is disposed in the home, for example, and is connected to the server 31 via the Internet. Access from the server 31 to the personal computer 33 is restricted by the authentication function. Further, the personal computer 33 holds a file including a list of moving image files that can be provided to the server 31 for each server 31. Or the personal computer 33 may hold | maintain the file containing the list information of the moving image file which can be provided with respect to the mobile terminal 35 for every mobile terminal 35. FIG. The personal computer 33 may be a host computer having the function of the server 31, or may be a storage device that stores a moving image file and is connected to a network.

  The server 31 receives a viewing request for a moving image file stored in the personal computer 33 from the mobile terminal 35. The server 31 also monitors the bandwidth status of the server 31 and the mobile terminal 35. Then, the server 31 acquires the video data to be viewed from the personal computer 33, deletes a plurality of video frames from the acquired video data according to the network bandwidth condition, and transfers the video data to the mobile terminal 35. . Details of network bandwidth monitoring and video frame deletion will be described later. The server 31 has an authentication function for establishing a connection with the mobile terminal 35.

  The mobile terminal 35 receives a request for a moving image to be reproduced from the user. Then, it instructs the personal computer 33 to transfer the moving image. The mobile terminal 35 receives the moving image data from the server 31 in a streaming format. If there is a video frame deleted at the time of transmission in the received video data, the mobile terminal 35 generates a complementary frame corresponding to the video frame and inserts it into the video data. Thereby, the personal computer 33 restores the video data before deleting the video frame. Then, the mobile terminal 35 reproduces the restored moving image data. By performing the restoration process, the mobile terminal 35 reproduces the moving image data at the same frame rate as the moving image data stored in the personal computer 33.

  In the following description, transmitting data from the personal computer 33 to the server 31 may be referred to as “upload”, and transmitting data from the server 31 to the mobile terminal 35 may be referred to as “downlink” (download). . Further, the personal computer 33 or the server 31 that is the transmission source of the moving image data may be referred to as a transmission terminal, and the server 31 or the mobile terminal 35 that is the reception side of the moving image data may be referred to as a reception terminal. Further, a video frame deleted in the personal computer 33 may be referred to as a deleted frame.

  Here, the structure of meta information of video data will be described. The meta information of the video data is sent from the transmitting terminal to the receiving terminal together with the video data. The transmitting terminal adds the information of the deleted video frame to the meta information, and the receiving terminal restores the moving image data by generating a complementary frame using the meta information. FIG. 3 shows an example of the structure of meta information of video data.

  The meta information is information related to the moving image data and is associated with the moving image data. The meta information includes a format size (content resolution) 43, a video title 44, a video time 45, a creation date 46, contents 47, a deletion frame start number 41, and a deletion frame period (number of frames) 42.

  The format size (content resolution) 43, the video title 44, the video time 45, the creation date 46, and the content 47 are included in the moving image data stored in the personal computer 33. The format size (content resolution) 43, video title 44, video time 45, creation date 46, and content 47 are respectively the format size (content resolution), video title, video time, creation date, and content of the corresponding video data. is there.

  The deletion frame start number 41 and the deletion frame period (number of frames) 42 are data items added to the metadata when the video frame is deleted at the transmission terminal. The deletion frame start number 41 is a frame number (frame identification number) of a video frame deleted by the transmission terminal. The deletion frame period (the number of frames) 42 indicates a period when the video frames to be deleted are continuous, and is represented by, for example, the continuous number of deletion frames.

  The deletion frame start number 41 and the deletion frame period (the number of frames) 42 are information on the deletion frame every predetermined period (for example, every 1 second). Therefore, when there are a plurality of deletion frames within a predetermined period, data items of the deletion frame start number 41 and the deletion frame period 42 associated with each deletion frame are added to the meta information.

  Here, the deletion frame start number 41 and the deletion frame period (number of frames) 42 may be the frame number of the restart frame and the period (number of frames) of the restart frame. In this embodiment, since the deleted frames are continuous, if there is information on the restart frame number and the frame period of the video positioned next to the deleted and missing video frame, the video at the mobile terminal 35 can be obtained. It is possible to recognize in advance the start number of the deleted frame and its duration during playback. Thereby, the mobile terminal 35 may complement the start number of the deleted frame and its period.

  Next, a description will be given of an operation in which a video frame is deleted from a moving image data and transmitted at a transmitting terminal, a complementary frame is generated at a receiving terminal, and the moving image data is restored.

  The determination of which video frame is deleted at the transmission terminal is made based on the audio level of the audio frame corresponding to each video frame.

  In video playback, the lip sync deviation appears most prominently in the movement of the person's mouth and the voice in the scene where the person speaks. When the volume level is low, it is considered that the mouth is not moving even if a person's face is displayed in the video. Therefore, it is possible to suppress a viewer's uncomfortable feeling when reproducing a moving image by deleting consecutive video frames having a low audio level and restoring by copying a video frame immediately before the deleted frame.

  In the video frame deletion process, only video data is deleted, and audio data is not deleted. This is because it is not necessary to delete the audio data because the mobile terminal reproduces the same number of frames as the original video.

  Next, a description will be given of how a video frame is deleted from video data at the transmitting terminal and transmitted, and a complementary frame is generated at the receiving terminal to restore the video data. FIG. 4 is a diagram for explaining how video frames are deleted and restored. In FIG. 4, the video frame deletion process is performed by the server 31, but may be performed by the personal computer 33.

  Consider a case in which frames A to K are transmitted from the personal computer 33 to the mobile terminal 35. First, the personal computer 33 transmits frames A to K to the server 31 together with meta information. When the server 31 receives the frames A to K together with the meta information, the server 31 recognizes the audio level corresponding to each video frame. The sound level may be, for example, a numerical value of a sound volume, a sound volume belonging to a specific frequency band belonging to a human audible range, or the like, but is not limited thereto.

  In the example of FIG. 4, as indicated by the audio level 50, the audio level of frame A is “60”, the audio level of frame B is “70”, the audio level of frame C is “90”, and so on. The voice level is as shown in FIG.

  The server 31 determines a frame whose sound level is equal to or lower than a predetermined threshold among the frames. Of the frames whose audio level is equal to or lower than a predetermined threshold, a video frame whose audio level of the immediately preceding frame is equal to or lower than the predetermined threshold is recognized as a deletion target frame.

  In the case of FIG. 4, it is assumed that the predetermined threshold value is 20. Then, the frames whose audio level is 20 or less are D to G. In the frame D, the audio level of the immediately preceding frame is equal to or higher than the threshold (frame C, audio level 90), so the video frame D is not recognized as a deletion target frame. On the other hand, the frames E to G are recognized as frames to be deleted because the audio level of the immediately preceding frame is equal to or less than the threshold value. Note that video data and audio data are associated with each frame, and the audio level of the video frame is recognized using information of the audio frame corresponding to the video frame.

  Next, the server 31 deletes the recognized deletion target frame from the moving image data. Then, the server 31 adds information regarding the deleted frame to the meta information. In the case of the example in FIG. 4, information on the video frames E to G is added to the meta information. Specifically, the server 31 stores the frame number of the video frame E in the deletion frame start number 41 and stores the integer value “3” in the deletion frame period (number of frames) 42.

  Here, the number of video frames to be deleted is determined such that the transmission rate of the moving image data to be transmitted is smaller than the bandwidth of the network between the server 31 and the mobile terminal 35.

  There may be a case where there is no frame having a sound level equal to or lower than a threshold value within a predetermined period of the moving image data, that is, a predetermined number of consecutive frames. For example, in the case of FIG. 4, when the threshold is set to 3, there is no frame having an audio level equal to or lower than the threshold among the frames A to K. In this case, there is no frame to be deleted, and a moving image is distributed at a transmission rate that exceeds the bandwidth of the network. This case can be dealt with by increasing the period of the moving image data in which the deletion target is simultaneously determined. Details of this operation will be described later.

  The server 31 transmits the meta information in which the information of the deleted video frame is added to the mobile terminal 35 along with the meta information. Then, the mobile terminal 35 receives video data in a state where the video frames E to G are deleted. The mobile terminal 35 receives the meta information simultaneously with the video data, and recognizes that the deleted video frames are the video frames E to G. Specifically, the mobile terminal 35 recognizes information on the deletion frame start number 41 and the deletion frame period (number of frames) 42 of the meta information, and specifies the deletion frame.

  Then, the mobile terminal 35 generates a complement frame of the deleted video frames E to G, and inserts the complement frame at the position of the video frame from which the video data is deleted, thereby performing the restoration process of the video data. Specifically, the mobile terminal 35 inserts a complementary frame at the position of the deleted frame specified from the deleted frame start number 41 and the deleted frame period (number of frames) 42 of the corresponding meta information.

  The complementary frame generation process is executed by duplicating the video frames before and after the deletion frame.

  FIG. 5 is a diagram illustrating an example of a complementary frame generation process. The example of FIG. 5 shows an example of complementation when the video frames D to F are deleted at the transmission terminal based on the audio level of the frame. The frame corresponding to the video frame D is generated by duplicating the video frame C, which is the previous video frame not deleted. On the other hand, the video frames E and F are generated by duplicating the video frame G, which is a video frame that has not been deleted immediately after.

  The generation of the complementary frame may be configured to change the copy source video frame in consideration of the amount of movement of the object in the video frame. Specifically, the mobile terminal 35 compares the video frames that have not been deleted before and after the deletion frame, and discriminates an object having a large movement amount and an object that is not. Then, the mobile terminal 35 sets the pixel value at the position corresponding to the object whose movement amount is not large as a value obtained by duplicating the pixel value of the video frame not deleted immediately before. A pixel value at a position corresponding to an object having a large movement amount changes a copy source video frame before or after a predetermined time among a plurality of consecutive deletion frames. That is, the pixel value of the deleted frame up to a predetermined time is a value obtained by duplicating the pixel value of the video frame not deleted immediately before, and the pixel value of the deleted frame after the predetermined time is not deleted immediately after The pixel value of the video frame is a duplicated value. Note that the amount of movement can be measured by various methods such as a method using a motion vector search for estimating the amount of motion of an image as used in motion compensation of MPEG (Moving Picture Experts Group), for example.

  FIG. 6 shows an example of a method in which an object having a large movement amount and an object that is not so are discriminated and a complementary frame is generated. The example of FIG. 6 shows an example of complementation when the video frames D to F are deleted at the transmission terminal based on the audio level of the frame. As shown in FIG. 6, the pixel value at the position indicating the object with the large moving amount of the video frames D and E is the pixel value at the corresponding position of the video frame C that is the previous video frame that has not been deleted. Also, the pixel value at the position indicating the object with the large moving amount of the video frame F is set as the pixel value at the corresponding position of the video frame G which is the video frame not deleted immediately after. On the other hand, a pixel value at a position indicating an object with a small moving amount of the video frames D, E, and F is a pixel value at a corresponding position in the video frame C.

  Here, at the time of moving image reproduction, since the audio output does not change with respect to the original moving image, no sound skip occurs. In addition, since the number of video frames is complemented and the number of apparent video frames is ensured, it is possible to suppress the degradation of the visual quality of the reproduced video. Furthermore, where the audio level changes significantly, it is considered that the video also changes greatly. In this embodiment, video frames whose audio level is lower than a predetermined threshold are deleted, so that it is possible to suppress the deterioration in the visual image quality of the reproduced video. .

  As described above, the method of determining the deletion target frame based on the audio level of the frame has been described. For the determination of the deletion target frame, in addition to the audio level, the closeness with the video frames before and after the time series of the video data, That is, a method that further considers the similarity may be used.

  Various methods may be used to calculate the similarity. For example, the similarity calculated by comparing the video frame with the immediately previous video frame may be used as the similarity, or the sum of squares of differences in pixel values (SSD). : Similarity may be calculated by calculating using Sum of Squared Differences).

  In this case, the transmitting terminal determines a video frame having a similarity greater than or equal to a predetermined threshold among the video frames, and an audio level of the video frame determined to have a similarity greater than or equal to a predetermined threshold is equal to or lower than the predetermined threshold. Are consecutive video frames as deletion target frames.

  Next, the distribution quality monitoring of the moving image data of the server 31 will be described. Distribution quality monitoring is an adjustment amount for adjusting the video frame rate of moving image data to be transmitted so that the bandwidth used for moving image transfer is equal to or less than the network bandwidth between the server 31 and the mobile terminal 35. Is to decide.

  The server 31 dynamically monitors the status of the network bandwidth between the server 31 and the mobile terminal 35, and adjusts the transmission rate by deleting video frames from the moving image data according to the monitoring result. Here, the server 31 deletes the video frame so that the transmission rate of the moving image data does not exceed the network bandwidth.

  In the distribution quality monitoring, the server 31 compares the data amount of the moving image to be transmitted with the data amount that can be stably transmitted / received, and the data amount of the moving image to be transmitted falls within the range of the data amount that can be stably transmitted / received. Next, the video frame rate of the moving image data is determined. The video frame rate of the moving image data determined here is referred to as a transmission frame rate in the following description.

  First, the server 31 determines the usable bandwidth between the server 31 and the mobile terminal 35 from the result of bandwidth monitoring between the server 31 and the mobile terminal 35. Alternatively, the server 31 determines an available bandwidth from the result of bandwidth monitoring and the transmission path capacity of the line. Then, the server 31 calculates the transmission bit rate of the moving image data from the resolution, bit color, compression rate, activity level, and frame rate of the moving image data. Then, the server 31 derives the deletion amount of the video frame rate of the moving image data so that the calculated transmission bit rate is within the usable bandwidth.

  The activity level is the frequency of packet transmission with respect to the stream, and varies depending on, for example, the content of the moving image data. For example, in the case of silent video data, there is no need to transmit an audio stream, and the activity level of the audio stream is zero.

  Next, the delivery quality monitoring operation will be described with a specific example. It is assumed that the communication standard between the server 31 and the mobile terminal 35 is LTE (Long Term Evolution). The bandwidth in this case is assumed to be 75 Mbps at the maximum. In the following example, it is assumed that moving image data is encoded at a fixed bit rate in the codec.

  The bandwidth that can be used for video transmission depends on traffic from other applications in the network, and is not always 100% usable. Therefore, the server 31 performs bandwidth monitoring, that is, calculates the time required for a certain amount of packets to reach the server 31 from the personal computer 33, and grasps the traffic status of the network. Based on this traffic situation, the ratio of the bandwidth guaranteed for moving picture distribution to the maximum transmission speed of the line is defined as Z%. Here, it is assumed that Z = 1.

  The moving image data has, for example, a resolution of 1280 [dot per inch (dpi)] × 720 [dpi] full HD (full high definition), a bit color of 8, and a video frame rate of 24 fps (frames per second). The bit rate of the video data is (1280 x 720 (resolution)) x (3 (RGB) x 256 (8-bit color)) x 24 (frame) = 17 Gbps This value is one frame uncompressed Is the case value.

  Here, for example, a compression method such as MPEG includes a full I picture (Intra picture), a 1/2 size P picture (Predictive picture), and a 1/4 size B picture (Bi-directional predictive picture). The I picture, P picture, and B picture are configured in a ratio of 1: 4: 10. Therefore, the compression rate of moving image data is about 11/30. Furthermore, for example, since MPEG-AVC is difference data of only a moving part, it is further compressed to approximately 1/24. Therefore, it is assumed that the moving image data is compressed to 11/30 × 1/24 = 11/720.

  Considering this compression, the transmission bit rate of moving image data is 17 Gbps × 11/720 = 259 Mbps. Here, the moving image data can be compressed in accordance with the resolution of the mobile terminal 35 where the reproduction is performed. For example, consider a case where the resolution of the mobile terminal 35 recognized by the server 31 is 800 × 480. In this case, the moving image data can be compressed to 800 × 480/1280 × 720 = 0.42 times before transmission.

  Here, the original video data is content to be displayed on a device with a high resolution, and when video data is played back on the mobile terminal 35 with a low resolution, the appearance changes even if the resolution of the video data is reduced to the resolution of the mobile terminal 35. Never do. In addition, the performance of the video playback chip of the mobile terminal 35 often cannot use all the amount of information included in the moving image data, and cannot reproduce fine changes in video.

  Therefore, the resolution based on the resolution of the mobile terminal 35 recognized by the server 31 is A, and the number of video frames is B. When A = 800 × 480 and B = 24, the transmission bit rate of video data is A × (RGB color frames: 3 × 256) × B × 11/720 = (800 × 480) × (3 × 256) × 24 × 11/720 ≒ 108Mbps

  However, this value is not less than the network bandwidth of 75 Mbps. Therefore, the transmission terminal deletes and transmits the video frame so that the transmission bit rate of the moving image data is equal to or less than the network bandwidth. If the ratio of the post-deletion video frame to the pre-deletion video frame is E, 108M × E ≦ 77M may be satisfied, and E ≦ 0.71. Since 24 × E, that is, 24 × 0.71 = 17.04, the video frame rate of the moving image data to be transmitted may be changed to 17 fps. In this case, since the video frame rate of the original moving image is 24 fps, the number of video frames to be deleted is 7 in the moving image data for 1 second.

  Next, an operation for deleting the video frame of the moving image data to be distributed so as to be equal to or less than the video frame rate of the moving image data determined in the distribution quality monitoring will be described.

  FIG. 7 is a diagram for explaining adjustment of the video frame rate of moving image data in the transmission terminal.

  The file decoder 51 separates the streaming data into video data and audio data. Then, the file decoder 51 outputs the separated video data to the video encoder 53 and outputs the separated audio data to the audio encoder 55. Further, the file decoder 51 transmits the meta information of the streaming data to the frame control unit 52.

  Here, the separated video data and audio data are associated with each frame. For example, video frames and audio frames having the same frame number are associated with each other. The same value is set for the time stamp of the corresponding video frame and audio frame. The time stamp is defined by adding the number of seconds from the start of playback to each frame, and the frame is played back at the time of the time stamp. Each video frame and each audio frame holds time stamp information.

  The frame control unit 52 determines a video frame to be deleted so that the frame rate of the video data is equal to or less than the transmission frame rate. Then, the frame control unit 52 adds information on the deleted video frame to the meta information. That is, the frame control unit 52 stores the information of the deleted video frame in the deleted frame start number 41 and the deleted frame period (number of frames) 42 of the meta information.

  The video data output from the file decoder 51 is input to the video encoder 53. The video encoder 53 deletes the deleted frame determined by the frame control unit 52 from the video data, and constructs the video frame again. Then, the video encoder 53 encodes the reconstructed video frame into a transmission format. The encoded video data is divided or aggregated into, for example, RTMP (Real Time Messaging Protocol) packets. The video encoder 53 outputs the encoded video data to the video memory 54, and the encoded video data is transmitted from the video memory 54 to the receiving terminal.

  On the other hand, the audio data separated by the file decoder 51 is input to the audio encoder 55. The audio encoder 55 converts the received audio data into a sending format and outputs it to the audio memory 56. The encoded audio data is transmitted from the audio memory 56 to the receiving terminal. Here, the audio frame corresponding to the deleted video frame is not deleted, and the meta information of the audio data is transmitted without being changed.

  Next, an operation flow of video frame deletion processing in the transmission terminal will be described. FIG. 8 shows a flowchart of video frame reduction processing. Here, the transmission terminal determines a frame to be deleted for each video frame of a predetermined period stored in the work buffer, and by deleting the frame, the video frame rate of the moving image data stored in the work buffer is set for transmission. It is adjusted so that it is below the frame rate. The flow in FIG. 8 is periodically executed for each video frame of a predetermined period stored in the work buffer.

  First, the transmitting terminal reads (buffers) moving image data for a predetermined period into a work buffer (S61). Next, the transmission terminal confirms the result of the distribution quality monitoring and recognizes the transmission frame rate (S62). Then, the transmitting terminal deletes the video frames currently stored in the work buffer so that the video frame rate of the moving image data currently stored in the work buffer becomes the transmission frame rate recognized in S62. The number of frames is confirmed (S63).

  Next, the transmission terminal determines a deletion target frame to be deleted based on the audio level corresponding to each frame among the video frames stored in the work buffer (S64). Then, the transmitting terminal determines whether or not the number of deletion target frames determined in S64 is equal to or greater than the number of deleted frames confirmed in S63 (S65).

  When the number of frames to be deleted determined in S64 is smaller than the number of deleted frames confirmed in S63 (No in S65), the transmitting terminal increases the period of frames to be stored in the work buffer (S66), and deletes the deleted frames again. A determination is made (S64). When the number of deletion target frames determined in S64 is equal to or greater than the number of deletion frames confirmed in S63 (Yes in S65), the transmitting terminal deletes the deletion target frame determined in S64 and uses the information on the deletion frames as meta information. (S66). Then, the transmission terminal encodes the video data for delivery and distributes the video data (S67).

  FIG. 9 shows the details of the flow of the operation (S64) in which the transmitting terminal discriminates the deletion target frame based on the audio level of each frame in the moving image data stored in the work buffer. In FIG. 9, for the sake of explanation, the frame number of the discrimination target video frame is assumed to be n (hereinafter, the video frame having the frame number n is referred to as n frame).

  The transmitting terminal determines whether or not the audio level of the video frame (n-1 frame) with the frame number n-1 is equal to or greater than a predetermined threshold (S71). When the audio level of the n-1 frame is less than the predetermined threshold (No in S71), the transmitting terminal determines that the n frame is not a deletion target frame (S74). Then, the transmitting terminal increments the value of n (S75).

  When the sound level of the n-1 frame is equal to or higher than the predetermined threshold value in S71 (Yes in S71), the transmitting terminal determines whether or not the sound level of the n frame is equal to or higher than the predetermined threshold value (S72).

  When the audio level of n frames is less than the predetermined threshold (No in S72), the transmitting terminal determines that the n frames are not frames to be deleted (S74). Then, the transmitting terminal increments the value of n (S75).

  If the sound level of the n frame is equal to or greater than the predetermined threshold value in S72 (Yes in S72), the transmitting terminal determines that the n frame is a frame to be deleted (S73). Then, the transmitting terminal increments the value of n (S75).

  After incrementing the value of n in S75, the transmitting terminal determines whether or not the discrimination process has been performed on the video frames in all the work buffers (S76). If there is a video frame in the work buffer that has not been subjected to the discrimination process (No in S76), the process returns to S71. If discrimination processing has been performed on all the video frames in the work buffer (Yes in S76), the processing ends.

  Note that the work buffer does not have to store all the video frames of the moving image data, and there is an area in which the video frame can be deleted and the video frame rate can be made equal to or lower than the transmission frame rate determined by the distribution quality monitoring. I just need it.

  Next, an operation when the video frame determined to be deleted is deleted from the video data for a predetermined period and the video frame rate of the video data for the predetermined period does not become lower than the transmission frame rate will be described. Here, the period of moving image data that is compared with the transmission frame rate at a time in the transmitting terminal is referred to as a moving image data discrimination target period.

  As described above, the transmission terminal determines the deletion target by dividing the moving image data every predetermined period, and determines whether or not the video frame rate of the moving image in this predetermined period is smaller than the transmission frame rate. ing.

  If the video frame rate of the moving image data in the predetermined period does not become lower than the transmission frame rate as a result of deleting the video frame determined to be deleted, the transmitting terminal increases the determination target period of the moving image data. In the present embodiment, the transmission terminal increases the period of the moving image data stored in the work buffer in which the video frames that are simultaneously determined whether or not to be deleted are stored. The transmission terminal deletes the video frame so that the video frame rate of the moving image data during the period of the video frame that can be stored in the work buffer is equal to or less than the transmission frame rate determined by the distribution quality monitoring.

  Specifically, for example, in the example of FIG. 4, when there are no more than a predetermined number of video frames having an audio level equal to or less than a threshold among the video frames A to K, the video frames L to V are further buffered in the work buffer. Among them, the video frame whose similarity is equal to or less than the threshold is deleted. Then, it is confirmed whether or not the video frame rate is equal to or lower than the transmission frame rate determined by the distribution quality monitoring in the period of the video frames A to V.

  FIG. 10 is a diagram for explaining the decoding process of the receiving terminal. The decoding processing unit includes a file decoder 91, a frame control unit 92, a video decoder 93, a video memory 94, an audio decoder 95, and an audio memory 96.

  The file decoder 91 separates the streaming data received from the transmission terminal into video data and audio data. Then, the file decoder 91 outputs the separated video data to the video decoder 93 and outputs the audio data to the audio decoder 95. In addition, the file decoder 91 extracts meta information from the received streaming data and transmits it to the frame control unit 92.

  The frame control unit 92 recognizes the deleted frame from the meta information received from the file decoder 91, and outputs the deleted frame information to the video decoder 93. The deleted frame information includes, for example, a deleted frame start number 41 and a deleted frame period (number of frames) 42. Further, the frame control unit 92 recognizes the time stamp of the audio frame corresponding to the deleted frame using the meta information, and outputs the time stamp information of the audio frame to the video decoder 93 as a control signal.

  The video decoder 93 receives video data from the file decoder 91 and receives deletion frame information from the frame control unit 92. The video decoder 93 decodes the video data. Then, the video decoder 93 generates a complementary frame of the deleted frame from the video data and the deleted frame information, and reconstructs the video frame. The time stamp of the complementary frame generated here is set to the same value as the time stamp of the audio frame corresponding to the deleted frame by the control signal received from the frame control unit 92. Thereby, lip sync adjustment is performed. Then, the video decoder 93 outputs the reconstructed video data to the video memory 94.

  The audio decoder 95 receives audio data from the file decoder 91 and performs decoding processing. Then, the audio data is output to the audio memory 96.

  Here, the lip sync adjustment is not performed when the data is stored in the video memory 94 and the audio memory 96 but is completed in the video decoder 93. Thereby, the memory capacity for lip sync adjustment at the receiving terminal can be reduced.

  Next, the operation of frame reconstruction by the receiving terminal will be described. FIG. 11 is a flowchart of frame reconstruction by the receiving terminal.

  First, the video decoder 93 of the receiving terminal executes a deletion frame complementary frame generation process (S101).

  The video decoder 93 receives information regarding the deleted frame included in the meta information from the frame control unit 92. The information regarding the deleted frame includes, for example, a deleted frame start number 41 and a deleted frame period (number of frames) 42. Then, the video decoder 93 recognizes the deletion frame number and the period of the deletion frame from the information regarding the deletion frame (S102). Note that the restart frame number may be recognized instead of the deletion frame number.

  Next, the video decoder 93 assigns a frame number to the generated complementary frame based on the recognized deletion frame number or restart frame number (S103). This frame number assignment can be said to be an operation of inserting a complementary frame at the position of the deleted frame.

  Next, when the video decoder 93 detects a frame of a predetermined period (S104), the generated complementary frame and the time stamp value of the audio frame corresponding to the frame number assigned in S103 are matched (S105). As a result, the lip sync is adjusted, and the video and audio can be synchronized.

  Then, the video decoder 93 outputs the video data for which the lip sync adjustment has been completed to the video memory 94, and outputs the audio data to the audio memory 96. Then, the moving image reproduction application is notified that the video data and the audio data have been output to the video memory 94 and the audio memory 96, respectively (S106). Using this data, the mobile terminal 35 reproduces the moving image. Note that the flow of FIG. 11 is periodically executed for each frame of a predetermined period.

  FIG. 12 is an operation flowchart of a complementary frame generation process for a deleted frame in the receiving terminal. In FIG. 12, an example in which the value of the deletion frame start number 41 of the meta information is n + 1 and the value of the deletion frame period (number of frames) 42 is T will be described for the sake of explanation. Further, when there is an object having a large movement amount in the video frame, a threshold value used for changing whether the value of the pixel indicating the object is duplicated from the immediately preceding video frame or the immediately following video frame Let x be x. In this example, the threshold value x is an integer representing the number of frames, but is not limited thereto.

  The receiving terminal confirms from the metadata information that there is a video frame deleted at the transmitting terminal (S121). That is, the receiving terminal recognizes the deletion frame start number 41 and the deletion frame period (number of frames) 42 included in the metadata.

  Then, the receiving terminal buffers the video frame (n frame) having the frame number immediately before the deleted frame start number 41 among the frames not deleted (S122).

  Next, the receiving terminal buffers a frame ((n + 1 + T) frame) having a frame number immediately after the deleted frame start number 41 among frames that have not been deleted (S123).

  Then, the receiving terminal uses the n frame and the (n + 1 + T) frame to determine an object having a large movement amount in the frame (S124).

  Next, the frame to be complemented is set to n + 1 frame (S125). Then, the receiving terminal determines whether or not the frame number of the complementary processing target frame is n + x or more (S126). When the frame number of the complement processing target frame is less than n + x (No in S126), the complement processing target frame is generated by duplicating n frames (S127).

  The receiving terminal increments the frame number to be complemented (S129), and determines whether the frame number is greater than n + T (S130). When the complement processing target frame number is n + T or less (No in S130), the process returns to S126. If the frame number to be complemented is greater than n + T (Yes in S130), the process ends.

  In S126, when the frame number of the complement processing target frame is n + x or more (Yes in S126), the object with the large amount of movement determined in S124 is duplicated from the (n + 1 + T) frame, and the other object is duplicated from the n frame. (S128).

  Then, the frame number to be complemented is incremented (S129), and it is determined whether or not the frame number is greater than n + T (S130). When the complement processing target frame number is n + T or less (No in S130), the process returns to S126. If the frame number to be complemented is greater than n + T (Yes in S130), the process ends.

  13A and 13B are sequence diagrams of the information processing system according to the present embodiment. FIG. 13 shows an operational relationship among the personal computer 33, the server 31, and the mobile terminal 35.

  The mobile terminal 35 connects to the server 31 that relays between the mobile terminal 35 and the personal computer 33 in order to acquire the moving image data stored in the personal computer 33 (S401). Before the connection is established, the server 31 performs authentication in order to confirm the validity of the mobile terminal 35 that requested the connection. Further, the server 31 recognizes the resolution at the time of moving image reproduction on the mobile terminal 35 simultaneously with the authentication of the mobile terminal 35 (S402). If the authentication is successful, the server 31 sends a device approval response to the mobile terminal 35 (S403).

  Next, the server 31 connects to the personal computer 33 storing the moving image data requested by the mobile terminal 35 (S404). Even when the connection between the personal computer 33 and the server 31 is established, authentication for confirming the validity of the connection is performed. If the authentication is successful, the personal computer 33 transmits a connection response to the server 31 (S405).

  Next, the personal computer 33 notifies the mobile terminal 35 of list information of videos that can be provided via the server 31 (S406). A list of videos that can be provided may be preset for each mobile terminal 35 or for each server 31 and stored as a file in the personal computer 33. Further, the resolution information of the mobile terminal 35 may be received from the server 31, and the personal computer 33 may determine a video that the mobile terminal 35 can reproduce from the video data according to the information.

  When the notified list is displayed on the screen of the mobile terminal 35, the user operates the mobile terminal 35 to select a moving image to be reproduced. The mobile terminal 35 makes a delivery request for the selected playback video (hereinafter referred to as a playback video request) to the server 31 (S407).

  Next, the server 31 performs information collection work for monitoring the network bandwidth between the mobile terminal 35 and the server 31 (S408).

  Next, the server 31 transmits the reproduction moving image request received from the mobile terminal 35 to the personal computer 33 (S409). When receiving the playback video request, the personal computer 33 notifies the server 31 of meta information related to the video specified by the playback video request (S410).

  When the server 31 receives the meta information from the personal computer 33, the server 31 monitors the distribution quality of the moving image transmission between the server 31 and the mobile terminal 35 (S411). Specifically, the server 31 adjusts the video frame rate of the moving image data to be transmitted so that the bandwidth used for moving image transfer is equal to or less than the network bandwidth between the server 31 and the mobile terminal 35. Determine the amount. At this time, the server 31 notifies the mobile terminal 35 of the video data change information as a reproduction information change request so that the mobile terminal 35 can recognize the video data change information (S412).

  When receiving the reproduction information change request from the server 31, the mobile terminal 35 recognizes the change of the distribution video and sends a reproduction setting response to the server 31 (S413).

  Next, the server 31 notifies the personal computer 33 of a streaming start request (S414). When receiving the streaming start request from the server 31, the personal computer 33 sends a streaming start response to the mobile terminal 35 via the server 31 (S415). The personal computer 33 distributes the streaming data to the server 31 (S416).

  When receiving the streaming data, the server 31 recognizes the audio level of the moving image data (S417). Then, the server 31 determines and selects the video frame to be deleted so that the video frame rate becomes the transmission frame rate derived by the distribution quality monitoring based on the recognized audio level. Here, the server 31 may change the resolution of the moving image data to the resolution recognized in S402. Then, the server 31 deletes the selected deletion target frame from the moving image data (S418).

  Next, the server 31 adds the deleted video frame information to the meta information (S419). Then, the server 31 encodes the deleted moving image data into a format for sending (S420), and streams the encoded data to the mobile terminal 35 (S421).

  When receiving the streaming data, the mobile terminal 35 decodes the received data (S422). Then, the mobile terminal 35 restores the moving image data by generating a complementary frame corresponding to the video frame deleted in S418 and inserting it into the moving image data (S423). Then, the mobile terminal 35 reproduces the restored moving image data (S424).

  Then, the operations of S408 to S424 are repeated for each predetermined amount of data, and streaming distribution of moving image data by the personal computer 33 and reproduction by the mobile terminal 35 are performed. Note that the operations in S408 to S424 for each predetermined data amount may be performed in parallel for each data amount.

  Next, the connection operation (S401, S404) between the mobile terminal 35 and the server 31, and between the server 31 and the personal computer 33 will be described in detail.

  An application program (hereinafter referred to as an application) for reproducing a moving image is installed in the mobile terminal 35 and is connected to the target server 31 by the application. The designation of the target server 31 may be selected by the user, or may be set in advance in the application. For example, when the mobile terminal 35 is connected like a 3G (3rd Generation) line, the mobile terminal 35 can specify the line and connect to the Internet. In addition, the mobile terminal 35 and the personal computer 33 enable P2P (Peer to Peer) connection via the Internet.

  Authentication is performed when a connection is established between the server 31 and the mobile terminal 35 and between the server 31 and the personal computer 33. Authentication information used there is, for example, specific device information such as an IP address and a MAC address of each device. . The authentication information is managed on the server 31, and the IP address and unique device information are stored grouped for each mobile terminal 35 in association with the mobile terminal 35 and the personal computer 33 to which the mobile terminal 35 can be connected. One personal computer 33 may be associated with one mobile terminal 35 or may be associated with a plurality.

  An operation when establishing a connection between the mobile terminal 35 and the server 31 will be described. When receiving the connection request from the mobile terminal 35, the server 31 collates the IP address or unique device information of the mobile terminal 35 that has made the connection request with the IP address or unique device information stored in the server 31. If the collation results match, the server 31 establishes a connection assuming that the authentication is successful. Various authentication techniques may be used for authentication, and a password authentication method, an electronic certificate authentication method, or the like may be used.

  Next, an operation when establishing a connection between the server 31 and the personal computer 33 will be described. After establishing the connection with the mobile terminal 35, the server 31 connects to the personal computer 33 corresponding to the mobile terminal 35. Specifically, when one personal computer 33 is associated with one mobile terminal 35 in the authentication information stored in the server 31, the server 31 confirms the authentication information and corresponds to the mobile terminal 35. Connect to PC 33. Alternatively, when the mobile terminal 35 establishes a connection with the server 31, the mobile terminal 35 may designate a personal computer 33 as a connection destination, and the server 31 may be connected to the designated personal computer 33.

  The personal computer 33 that has received the connection request from the server 31 authenticates whether or not the server 31 that has requested access is legitimate. Various authentication techniques are used for the authentication, as in the authentication performed by the server 31 for the mobile terminal 35. Alternatively, access control may be performed by granting access authority to a moving image file held by the personal computer 33 or a directory (folder) of the personal computer 33.

  The personal computer 33 and the server 31, and the server 31 and the mobile terminal 35 may be connected using a high security network technology such as VPN (Virtual Private Network). Furthermore, in data transmission between the personal computer 33 and the server 31, and between the server 31 and the mobile terminal 35, the transmitted data may be encrypted by various encryption techniques. The server 31 and the personal computer 33 may be arranged in the same intranet.

  Next, the information collection work (S408) for monitoring the network bandwidth by the server 31 will be described in detail.

  The server 31 monitors the bandwidth between the mobile terminal 35 and the server 31. The server 31 transmits a band detection packet to the mobile terminal 35 in order to detect the bandwidth of the network connected to the mobile terminal 35. In the transmission packet, the time when the server 31 transmits is recorded. When the mobile terminal 35 receives the packet, it measures the reception time and compares it with the transmission time recorded in the packet. Thereby, the mobile terminal 35 can calculate the time required for a certain amount of packets to reach the mobile terminal 35 from the server 31. The mobile terminal 35 transfers the bandwidth monitoring information obtained here to the server 31. The server 31 divides a predetermined amount of data into a plurality of packets, and measures the time required for the mobile terminal 35 to receive the last packet after the server 31 transmits the first packet. Bandwidth monitoring information may be acquired.

  Note that transmission of packets for bandwidth monitoring is performed within a range where there is no problem in reproduction of actual moving image data. Here, the bandwidth may be monitored by sending a command for monitoring the bandwidth, or by sending a command such as ping and measuring the response time. Alternatively, the bandwidth between the personal computer 33 and the server 31 and between the server 31 and the mobile terminal 35 may be monitored at a time by sending a packet from the personal computer 33 to the mobile terminal 35 via the server 31. Note that not only bandwidth monitoring but also line traffic monitoring may be performed based on transmission delay.

  On the other hand, the amount of information that enables stable transmission (or reception) of data per second for each type of line connected to the Internet is defined. The server 31 determines the data amount (usage amount) that can be stably transmitted / received per unit time based on the prescribed information amount and the bandwidth monitoring information.

  Here, after the bandwidth monitoring operation (after S408), the server 31 transmits the monitoring result to the mobile terminal 35, and the user confirms the bandwidth monitoring result by the mobile terminal 35 and designates the resolution of the moving image to be reproduced. Also good. In that case, the designated resolution information is used in distribution quality monitoring.

  FIG. 14 shows an example of the configuration of the server 31 in the present embodiment. The server 31 includes a decoding processing unit 131, an arithmetic processing unit 132, a storage 133, a content server 134, and a streaming server 135.

  The decode processing unit 131 decodes moving image data uploaded from a terminal device such as the personal computer 33. Here, since the moving image data to be uploaded is divided, integrated, or compressed into a sending format, decoding processing is performed to restore the moving image data.

  The arithmetic processing unit 132 performs bandwidth monitoring and distribution quality management, and performs a video frame deletion process and a meta information change process according to the result.

  The storage 133 stores an operating system, middleware, and applications, which are read out to the memory by the arithmetic processing unit 132 and executed.

  The content server 134 manages content prepared for streaming playback, and the mobile terminal 35 can select a moving image to be played back from the managed content.

  The streaming server 135 distributes moving image data to the mobile terminal 35. As the moving image data, the moving image data from which the video frame is deleted is received from the arithmetic processing unit 132. The streaming server 135 is divided according to protocols used for distribution, such as HTTP (HyperText Transfer Protocol) and HTTP / RTMP.

  FIG. 15 shows an example of the hardware configuration of the server 31 or the personal computer 33 according to this embodiment. The server 31 or the personal computer 33 includes a CPU (Central Processing Unit) 161, an SDRAM (Synchronous Dynamic Random Access Memory) 162, a serial port 163, a flash memory 164, a digital I / O, and an analog I / O 165. The server 31 or the personal computer 33 includes a storage 166, a chip set 167, a communication card 168, a CF (Compact Flash (registered trademark)) interface card 169, and a real time clock 170.

  The CPU 161 uses the SDRAM 162 or the flash memory 164 to execute a program describing the procedure of the above-described flowchart stored in the storage 166. Further, the CPU 161 exchanges data with the communication card 168, the CF interface card 169, and the real time clock 170 via the chip set 167. The server 31 or the personal computer 33 inputs and outputs moving image data from the serial port 163, digital I / O, or analog I / O 165 via a communication card. The CPU 161 provides some or all of the functions of the monitoring unit 3, the deletion unit 4, the frame information generation unit 5, the transmission unit 6, the reception unit 8, the complementary image generation unit 9, and the video data generation unit 10. The CPU 161 performs encoding / decoding of the moving image data, lip sync, authentication operation for establishing a connection, and reproduction of the moving image data. The storage 166 provides a part or all of the functions of the storage unit 2. The CPU 161 can use the SDRAM 162 as a temporary data storage area (working buffer) for performing frame deletion processing of moving image data and restoration processing of moving image data. The SDRAM 162 is not limited to this and can be various RAMs (Random Access Memory).

  The flash memory 164 stores a kernel, an application in the server 31, a setting file, and the like. The flash memory has an expansion area and can be used as a temporary data storage area (working buffer) for performing frame deletion processing of moving image data and restoration processing of moving image data.

  The CF interface card 169 is used as an auxiliary function for use in maintenance of the server 31. Since the storage is built in, this is used for data processing between many personal computers 33 and mobile terminals 35.

  The real-time clock 170 is a dedicated chip having a function as a computer clock. The time stamp of the frame is set according to the clock of the real time clock 170.

  A part of the first information processing apparatus 1 and the second information processing apparatus 7 of the embodiment may be realized by hardware. Alternatively, the first information processing device 1 and the second information processing device 7 of the embodiment may be realized by a combination of software and hardware.

  FIG. 16 shows an example of the hardware configuration of the mobile terminal 35 according to the present embodiment. As shown in FIG. 16, the mobile terminal 35 includes a CPU 201, a memory 202, a storage unit 203, a reading unit 204, a communication interface 206, an input / output unit 207, and a display unit 208. Note that the CPU 201, the memory 202, the storage unit 203, the reading unit 204, the communication interface 206, the input / output unit 207, and the display unit 208 are connected to each other via a bus 209, for example.

  The CPU 201 uses the memory 202 to execute a program describing the above-described flowchart procedure. The CPU 201 provides some or all of the functions of the reception unit 8, the complementary image generation unit 9, and the video data generation unit 10. The CPU 201 also decodes moving image data, lip sync, and reproduces moving image data. The memory 202 is, for example, a semiconductor memory, and includes a RAM area and a ROM area. The storage unit 203 is a hard disk, for example. Note that the storage unit 203 may be a semiconductor memory such as a flash memory. An application for reproducing moving image data is stored in the memory 202 or the storage unit 203 and executed by the CPU 201. The mobile terminal 35 may be configured without the storage unit 203.

  The reading unit 204 accesses the removable recording medium 205 according to an instruction from the CPU 201. The detachable recording medium 205 is, for example, a semiconductor device (USB memory or the like), a medium to / from which information is input / output by a magnetic action (magnetic disk or the like), a medium to / from which information is input / output by an optical action (CD-ROM, For example, a DVD).

  The communication interface 206 transmits / receives data via a network according to instructions from the CPU 201. The communication interface 206 receives moving image data. The input / output unit 207 corresponds to, for example, a device that receives an instruction from the user. The user can use the input / output unit 207 to specify the moving image data to be reproduced and the resolution of the moving image data. The display unit 208 displays the reproduced moving image data.

An information processing program for realizing the embodiment is provided to the mobile terminal 35 in the following form, for example.
(1) Installed in advance in the storage unit 203.
(2) Provided by the removable recording medium 205.
(3) Provided via a network.

(Modification)
FIG. 17 shows an example of the configuration of the information processing system in the present embodiment (modification). The difference between this modification and the embodiment is that the receiving terminal is in the form of a terminal device 37 such as a personal computer 33, for example. Even in the same personal computer 33, image quality deterioration may be confirmed depending on the bandwidth of the Internet. In such a case, the configuration described in the embodiment is applied. The modification is an example in which the personal computer 33 performs the frame deletion process.

  The personal computer 33 may perform the network bandwidth monitoring, the distribution quality monitoring, the frame deletion processing and the processing corresponding thereto performed by the server 31 in this embodiment. Network bandwidth monitoring and distribution quality monitoring may also be performed between the server 31 and the personal computer 33. In the present embodiment, the moving image data is stored in the personal computer 33, but the moving image data may be stored in the server 31 and provided from the server 31 to the mobile terminal 35. The mobile terminal 35 may be a thin client terminal. Further, the decoding in the present embodiment may be performed in accordance with a standard such as MPEG2.

  In the present embodiment, the moving image is distributed from the server 31 to the mobile terminal 35 in the streaming format. However, the distribution method is not limited to the streaming format.

  In addition, this embodiment is not limited to embodiment described above, A various structure or embodiment can be taken in the range which does not deviate from the summary of this embodiment.

DESCRIPTION OF SYMBOLS 1 1st information processing apparatus 2 Storage part 3 Monitoring part 4 Deletion part 5 Frame information generation part 6 Transmission part 7 2nd information processing apparatus 8 Reception part 9 Complementary image generation part 10 Video data generation part 11 Communication network

Claims (8)

A storage unit for storing first video data and audio data including synchronization information associated with the first video data;
A monitoring unit for monitoring the state of the communication network;
In accordance with the result of the monitoring , the similarity between consecutive video frames in the first video data is determined from a first frame rate indicating the first number of frames per unit time of the first video data. Is equal to or higher than a predetermined threshold, and the corresponding audio data has an audio level equal to or lower than a predetermined threshold, and any one of the continuous video frames is deleted to a second frame rate lower than the first frame rate. A deletion unit for generating the second video data,
A frame information generation unit that generates frame information about the deleted frame;
A transmission unit for transmitting the second video data and the frame information;
An information processing apparatus comprising: The information processing apparatus according to claim 1, wherein the transmission unit further transmits the audio data. From the first frame rate indicating the first number of frames per unit time of the first video data , the similarity between consecutive video frames of the first video data is greater than or equal to a predetermined threshold, and and second image data one of the video frame that is being removed, the lower than the first frame rate second frame rate for audio level of the audio data is the continuous is below a predetermined threshold for the A receiving unit that receives frame information related to the deleted frame and audio data including synchronization information associated with the first video data ;
A complementary image generation unit that generates a complementary image that complements the image of the deleted frame using the frame information;
Using the complementary image, the second video data, and the synchronization information, the video data of the first frame rate, the complementary image corresponding to the deleted frame, and the deleted image A video data generation unit that generates the video data synchronized with the audio data corresponding to the frame ;
An information processing apparatus comprising: The video data generation unit inserts the complementary image at the position of the deleted frame of the second video data using the frame information, and generates video data of the first frame rate. 3. The information processing apparatus according to 3 . The information processing apparatus according to claim 3, wherein the complementary image generation unit generates the complementary image by duplicating a frame immediately before the deleted frame. The first information processing apparatus
Monitor the status of the communication network
According to the result of the monitoring , the similarity between consecutive video frames in the first video data is determined from the first frame rate indicating the first number of frames per unit time of the first video data. not less than a predetermined threshold value, the speech level of the corresponding audio data by deleting one of the video frames the continuous is below a predetermined threshold value, and the lower second frame rate than the first frame rate 2 video data,
Generating frame information about the deleted frame;
Executing a process of transmitting the second video data and the frame information, and audio data including synchronization information associated with the first video data ;
The second information processing apparatus
Receiving the transmitted second video data and the frame information and the audio data including the synchronization information ;
Generating a complementary image that complements the image of the deleted frame using the frame information;
Using the complementary image, the second video data, and the synchronization information, the video data of the first frame rate, the complementary image corresponding to the deleted frame, and the deleted image A video data transmission / reception method for executing processing for generating the video data synchronized with the audio data corresponding to the frame . From the first frame rate indicating the first number of frames per unit time of the first video data, continuous video in which the audio level of the corresponding audio data is equal to or less than a predetermined threshold among the first video data A receiving unit for receiving second video data in which a frame is deleted and having a second frame rate smaller than the first frame number, and frame information regarding the deleted frame;
A complementary image generation unit that generates a complementary image that complements the image of the deleted frame using the frame information, and is included in the deleted frame using frames before and after the deleted frame An object whose movement amount is equal to or greater than a predetermined threshold is determined, and the area for displaying the object is complemented by duplicating an area indicating the object of the undeleted frame immediately after the deleted frame. The complementary image generation unit for generating an image;
A video data generation unit configured to generate video data of the first frame rate using the complementary image and the second video data;
An information processing apparatus comprising: The first information processing apparatus
Monitor the status of the communication network
Depending on the result of the monitoring, the audio level of the corresponding audio data of the first video data is predetermined from the first frame rate indicating the first number of frames per unit time of the first video data. To generate second video data having a second frame rate lower than the first frame rate by deleting consecutive video frames that are equal to or less than a threshold value of
Generating frame information about the deleted frame;
The second video data and the frame information are transmitted
Execute the process,
The second information processing apparatus
Receiving the transmitted second video data and the frame information;
Generating a complementary image that complements the image of the deleted frame using the frame information;
Using the complementary image and the second video data, the video data of the first frame rate is generated.
Execute the process,
In the generation of the complementary image, an area in which the movement amount included in the deleted frame is greater than or equal to a predetermined threshold using the frames before and after the deleted frame, and the object is displayed Generates the complementary image by duplicating a region indicating the object of the non-deleted frame immediately after the deleted frame
Video data transmission / reception method.

Images