JP5428702B2 - Stream communication system, server device, and client device - Google Patents

Description

  The present invention relates to a stream communication system, a server apparatus, and a client apparatus, and more particularly, to a technique that appropriately copes with network bandwidth fluctuations and client apparatus processing load fluctuations in a system that distributes video streams.

  In a server / client system in which a server apparatus and a client apparatus perform video stream communication via a network, there is a tendency for “real-time performance” to be increasingly demanded recently. For example, when considering a server client system in which a graphical user interface of an operation system operating on a server device is transmitted to a client device by video stream communication and displayed on a display device provided in the client device, This delay is a problem, and a high degree of real time is required.

  There is Patent Document 1 as a document describing a technique related to the present invention. In Patent Document 1, a mobile terminal that receives video data distribution from a video distribution server detects the CPU load and network bandwidth status of the terminal, transmits it to the server, and when necessary based on these on the server side, It describes that the distribution method is switched from the streaming method to the download method. However, the video distribution server in this related technology needs to store (buffer) the video data to be distributed so as to be compatible with the download method. For this reason, this related technique cannot meet the above-described demand for real-time property.

JP 2009-152952 A

  In order to realize the above-described advanced real-time property, an ideal network environment and sufficient performance of the client device are expected. However, if a dedicated network or dedicated hardware is used to achieve this, the degree of freedom in system design is limited. Various networks are conceivable for connecting the server device and the client device, including a best effort network such as the Internet or a wireless LAN. Also, it is conceivable that the client device uses a general-purpose personal computer from the viewpoint of versatility and cost.

  In a best-effort network, the available network bandwidth fluctuates, so if the network bandwidth becomes narrow, sufficient bandwidth cannot be secured for the bit rate of the video stream, and the client device Clicking may occur in the video to be played. Factors that cause network bandwidth fluctuation include bandwidth compression due to other communications and the influence of physical conditions such as radio wave conditions (in the case of a wireless LAN).

  For such a problem, generally, a method of buffering stream data is used in order to absorb fluctuations in the network bandwidth. However, this method cannot be used when the above-described advanced real-time property is required. This is because buffering is synonymous with delaying the video.

  The use of a general-purpose personal computer for the client device also has a problem that leads to video delay in realizing the above-described advanced real-time property. In general-purpose personal computers, system users usually construct various environments according to their own preferences and needs. Therefore, compared with the case where dedicated hardware is used, there is a relatively high possibility that a sudden shortage of hardware resources occurs and a video is delayed.

  For example, if computer anti-virus software is installed in a general-purpose personal computer and virus search processing is started in the background during playback and display of a video stream, the anti-virus software uses CPU resources. A video stream receiving application may not be able to use sufficient CPU resources, which may cause a problem that the image is crazed.

  Conventionally, these problems have been addressed by reducing the network load or CPU load by adjusting the bit rate and frame rate of the video delivered by the user. However, this method is complicated for the user. Even if the network load or CPU load is sudden, and even when sufficient resources are available, it is difficult for the user to recognize this, and the low bit rate and frame rate are used. This forces the user to be inconvenient.

  The present invention has been made in view of the above circumstances, and in a server / client system in which a server apparatus and a client apparatus perform video stream communication via a network, fluctuations in network bandwidth and stream processing capability of the client apparatus are It is an object of the present invention to provide a stream communication system, a server device, and a client device that can appropriately cope with fluctuations.

  In order to achieve the above object, according to a first aspect of the present invention, there is provided a stream communication system in which video data is encoded by a server device, stream-transmitted to the client device, and decoded by the client device. , Encoding means for encoding the video data based on predetermined parameters, and stream transmission means for transmitting the encoded video data to the client apparatus as a stream, wherein the client apparatus receives the transmitted stream, Stream receiving means for measuring a reception time which is a time taken for receiving a stream of frames; a decoding means for decoding a received stream and measuring a decoding time which is a time taken for decoding a stream of one frame; Parameters and reception time Parameter adjustment means for calculating an adjusted parameter based on the decoding time, and the encoding means of the server device replaces the video data with the predetermined parameter instead of the predetermined parameter. A stream communication system characterized by encoding is provided.

  In order to achieve the above object, according to a second aspect of the present invention, there is provided a server device that encodes video data and transmits the encoded video data to a client device, the time taken to receive a stream from the client device, and Receiving the bit rate and the frame rate used when encoding the video data determined based on the time taken for decoding, and encoding the video data based on the bit rate and the frame rate, A server device is provided.

  In order to achieve the above object, according to a third aspect of the present invention, there is provided a client device that receives and decodes a stream of encoded video data, the time taken to receive the stream, A bit rate and a frame rate used for encoding video data are determined based on a time taken for decoding, and the determined bit rate and frame rate are transmitted to a server device that transmits the video data. A client device is provided.

  According to the present invention, in a server / client system in which a server apparatus and a client apparatus perform video stream communication via a network, it is possible to appropriately cope with fluctuations in network bandwidth and fluctuations in stream processing capacity of the client apparatus. It is possible to provide a stream communication system, a server device, and a client device.

It is a block diagram which shows the whole structure of the form for implementing this invention. It is a flowchart which shows the procedure of the parameter adjustment process which the parameter adjustment part 205 of FIG. 1 performs. It is a conceptual diagram for demonstrating the statistical value of the processing time used for the parameter adjustment process of this embodiment. It is a conceptual diagram for demonstrating the calculation method of the bit rate in the parameter adjustment process of this embodiment. It is a conceptual diagram for demonstrating the judgment criteria of whether to change the frame rate in the parameter adjustment process of this embodiment.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In the present embodiment, the server apparatus and the client apparatus are connected via a network, and the server client system (hereinafter referred to as “video stream communication system”) in which the client apparatus receives a video stream output from the server apparatus is connected to the present embodiment. It is an embodiment to which the invention is applied.

  FIG. 1 shows the overall configuration of the present embodiment. As illustrated, in the video stream communication system according to the present embodiment, the server device 100 and the client device 200 are connected via a network 300. The server apparatus 100 includes a moving image input unit 101, a moving image encoding unit 102, a stream transmission unit 103, and a server control communication unit 104. The client device 200 includes a moving image playback unit 201, a moving image decoding unit 202, a stream reception unit 203, a client control communication unit 204, a parameter adjustment unit 205, and an image quality setting storage unit 206.

  Each unit constituting the server device 100 is realized by cooperation between a general-purpose computer and a software program that performs predetermined information processing using the computer. The same applies to each part constituting the client device 200.

  The moving image input unit 101 has a function of inputting a moving image for stream communication to the video stream communication system according to the present embodiment. Various moving images are conceivable. For example, a graphical user interface (desktop screen) of an operating system operating on the server device 100 or another computer, or a distribution server (not shown) on the Internet. To the server apparatus 100 may be used. The moving image input by the moving image input unit 101 is first input to the moving image encoding unit 102.

  The moving image encoding unit 102 has a function of converting (encoding) the input moving image into a format suitable for the transmission path. At the time of encoding, the moving image encoding unit 102 uses predetermined parameters. As parameters, the frame rate and the bit rate are suitable from the viewpoint of more appropriately solving the problem of dealing with fluctuations in the network bandwidth and fluctuations in the stream processing capability of the client device. In the following description, a frame rate and a bit rate are used as parameters unless otherwise specified.

  The stream transmission unit 103 has a function of transmitting the moving image data encoded by the moving image encoding unit 102 to the client device 200 by stream communication. The server control communication unit 104 has a function of receiving the control communication of the video stream communication system transmitted from the client device 200 and changing the parameter of the moving image encoding unit 102 based on the control communication.

  The network 300 can use various types of networks, and includes a best effort type network such as a wireless LAN and the Internet.

  The stream reception unit 203 of the client device 200 has a function of receiving stream communication transmitted from the server device 100. In addition, the stream reception unit 203 sends the moving image data included in the received stream communication to the moving image decoding unit 202. The stream receiving unit 203 measures the time taken to receive stream data for one frame. The time taken to receive one frame of stream data is hereinafter referred to as “reception time”. The stream receiving unit 203 calculates a reception time for each frame and sends the reception time to the parameter adjustment unit 205.

  The moving image decoding unit 202 has a function of decoding moving image data transmitted from the stream receiving unit 203. The moving image obtained by decoding is input to the moving image reproducing unit 201 and reproduced. In addition, the moving image decoding unit 202 measures the time taken to decode the stream data for one frame. The time taken to decode the stream data for one frame is hereinafter referred to as “decoding time”. The moving image decoding unit 202 calculates a decoding time for each frame and sends it to the parameter adjustment unit 205.

  The image quality setting storage unit 206 has a function of storing settings (preferences) regarding the image quality of a moving image viewed by the user of the client device 200. The image quality setting can be set according to the use scene of the user. In this embodiment, priority is given to the “motion priority” setting that prioritizes increasing the frame rate and to increasing the data amount per frame. Includes "Quality priority" setting. The parameter adjustment unit 205 refers to the image quality setting stored in the image quality setting storage unit 206 as necessary.

  The parameter adjustment unit 205 has a function of adjusting parameters based on the reception time measured by the stream reception unit 203 and the decoding time measured by the moving image decoding unit 202. The adjusted parameter is sent to the client control communication unit 204, and the client control communication unit 204 transmits the parameter to the server apparatus 100 by the control communication of this video stream communication system. Hereinafter, parameter adjustment processing performed by the parameter adjustment unit 205 will be described.

  FIG. 2 shows the procedure of parameter adjustment processing. For example, the parameter adjustment processing may be started at regular intervals of several seconds, and the reception time and decoding time are measured by a predetermined number of frames by the stream reception unit 203 and the moving image decoding unit 202. It is good also as the timing made.

  In FIG. 2, the parameter adjustment unit 205 calculates stream reception time and decoding time statistics from the measured reception time and decoding time (step S101). Next, representative values of the reception time and decoding time to be adjusted and used are determined from the statistical values (step S102). Next, based on the representative value, it is determined whether parameter adjustment is necessary (step S103). The determination in step S103 is, for example, a determination that parameter adjustment is necessary when the representative value obtained in step S102 fluctuates beyond a predetermined value.

  Here, the statistical value is, for example, the reception time and decoding time of the latest 10 to 100 frames among the frames in which the reception time and the decoding time are measured (in the following description, “processing without distinguishing the reception time and the decoding time”). It may be called “time”)). Specifically, the average value of the most recent frame, the standard deviation of the most recent frame (see FIG. 3A), and the processing of frames positioned at the top few percent when the most recent frames are arranged for each processing time length Time values (see FIG. 3B), etc. can be used. Among these statistical values, it is most preferable to use the processing time value of a frame located at a position of the top few percent when arranged for each processing time length as a representative value. This will be described with reference to FIG.

  FIG. 3 is a conceptual diagram for explaining the statistical value of the processing time used in the parameter adjustment processing of the present embodiment. FIG. 3A shows a case where a representative value is calculated using the average value and standard deviation of the most recent frame as statistical values, and the representative value in this case can be, for example, the sum μ + σ of the average value μ and the standard deviation σ. . This is effective when the processing time is normally distributed. FIG. 3B shows a case in which the value of the processing time of the frame located at the upper few percent position when the latest frames are arranged for each processing time length is used as a representative value. The frames indicated by the arrows in the figure are frames that are arranged in order from the frame with the longest processing time and are positioned at the top several percent. Here, the top percentage can be arbitrarily determined.

  As a comparative example, considering that the maximum value of the processing time in the latest 10 to 100 frames is a representative value, this comparative example has a bit rate (obtained as a result of the subsequent parameter adjustment processing) with respect to the processing capability. There is a problem that may be too low. As another comparative example, considering that the average value of the most recent frame is a statistical value, this comparative example may also be clogged in about half the time and is relatively ineffective. In an environment using a best-effort network or a general-purpose computer, the load is frequently increased rapidly. Therefore, the statistical value determination method as shown in FIG. 3B is most preferable.

  Referring to FIG. 2 again, when executing parameter processing, the parameter adjustment unit 205 compares the representative value of the reception time calculated in step S102 with the representative value of the decoding time, and adjusts the larger one. The value to be used (Tr) is set (step S104). The significance of using the larger one for adjustment is that the longer the processing time is, the more the processing capacity is reduced, so that the adjusted parameter is adjusted to the one where the processing capacity is further decreased.

Next, the parameter adjustment unit 205 uses Tr to determine the adjusted bit rate (Br) from the following equation (step S105).
Br = TsBs / Tr
Where Ts: the reciprocal of the frame rate of the stream
Bs: Bit rate of the stream

  The meaning of the above formula will be described with reference to FIG. FIG. 4 is a conceptual diagram for explaining a bit rate calculation method in the parameter adjustment processing of the present embodiment. The above formula and Ts in FIG. 4 are the reciprocals of the frame rate of the stream set at the time when the parameter adjustment processing is executed. That is, Ts means the length of time that can be allocated to processing per frame. Bs is the bit rate of the stream set at the same time when the parameter adjustment process is executed. Then, TsBs (in FIG. 4, a square area surrounded by two values of Ts and Bs) corresponds to the data amount per frame. Note that Ts and Bs receive setting values from the stream reception unit 203.

  If the Tr value determined up to step S104 in FIG. 2 is Tr1 in FIG. 4 (Ts <Tr1), the bit rate obtained by the above equation is Br1 in FIG. That is, the value falls below the bit rate Bs of the stream. On the other hand, when the value of Tr is Tr2 (Ts> Tr2) in FIG. 4, the value rises to Br2, which is higher than the bit rate Bs of the stream. In FIG. 4, the area (data amount) of a rectangle surrounded by two values of Tr1 and Br1 and the area (data amount) of a rectangle surrounded by two values of Tr2 and Br2 are two values of Ts and Bs. The area (data amount) of the quadrangle surrounded by is equal. This means that the processing time changes when the bit rate changes to process the same amount of data, and conversely indicates that the bit rate can be estimated from the processing time and the data amount.

  Referring to FIG. 2 again, after determining the adjusted bit rate, the parameter adjusting unit 205 next determines whether or not to change the frame rate, and if it determines that the frame rate is to be changed, performs the changing process (step S1). S106, S107). The determination as to whether or not to change the frame rate depends on the mode of fluctuation of the network bandwidth or the stream processing capability, but also depends on the user's preference whether to maintain the image quality at a certain level. This will be described with reference to FIG.

  FIG. 5 is a conceptual diagram for explaining a criterion for determining whether or not to change the frame rate in the parameter adjustment processing of the present embodiment. As shown in the figure, when the above-described Ts is larger than Tr (Tr <Ts, Tr2 in FIG. 4), the parameter adjustment unit 205 refers to the image quality setting stored in the image quality setting storage unit 206. If the “motion priority” setting is selected, the frame rate is changed. The frame rate after change (after adjustment) is the inverse of Tr (in the example of FIG. 4, the inverse of Tr2). On the other hand, in the case of “image quality priority” setting, the frame rate is not changed.

  Similarly, when the above-described Ts is smaller than Tr (Tr> Ts, Tr1 in FIG. 4), the parameter adjustment unit 205 refers to the image quality setting stored in the image quality setting storage unit 206 to “ If it is “motion priority”, the frame rate is not changed. If “Quality priority” is selected, the frame rate is changed. The frame rate after change (after adjustment) is the inverse of Tr (in the example of FIG. 4, the inverse of Tr1).

  If the frame rate is extremely low, the moving image is in a frame advance state and does not function as a moving image. Even if a value equal to or higher than the frame rate of the moving image input to the moving image input unit 101 is set in the moving image encoding unit 102, the frame rate of the stream after encoding is the moving image input to the moving image input unit 101. No more than the frame rate. Therefore, when changing the frame rate, it is more appropriate to set the upper limit value and the lower limit value in advance and control the changed frame rate to be within the set range (for example, the changed frame rate is If the value is below the lower limit, the lower limit is the frame rate after the change). Similarly, if the bit rate exceeds a certain bit rate, the effect of improving the image quality cannot be recognized. Therefore, the upper and lower bit rates are set so that the changed bit rate is within the set range. (For example, when the changed bit rate exceeds the upper limit value, the upper limit value is set as the changed bit rate).

This is the end of the description of the parameter adjustment processing of the present embodiment.
The parameter adjusted as described above is transmitted by the client control communication unit 204 to the server device 100, and the server device 100 receives the parameter. Then, the moving image encoding unit 102 inputs the moving image according to the adjusted parameter. Perform encoding of.

  Therefore, if there is a delay in stream reception due to fluctuations in the network bandwidth, or if there is a delay in stream playback processing of the client device 200 due to an increase in processing load by an external application, encoding is performed with a lower bit rate. Since the stream is communicated, the user of the client device 200 does not experience a problem such as a scratch on the moving image reproduced by the moving image reproducing unit 201.

  On the other hand, if the network load or processing load is sudden and sufficient resources are available, the encoded stream will be automatically increased and the encoded stream will be communicated. For the user of the apparatus 200, the usability is improved without being complicated and without causing inconvenience.

  That is, according to this embodiment, it is possible to appropriately cope with fluctuations in the network bandwidth and fluctuations in the stream processing capability of the client device.

DESCRIPTION OF SYMBOLS 100 Server apparatus 101 Moving image input part 102 Moving image encoding part 103 Stream transmission part 104 Server control communication part 200 Client apparatus 201 Moving picture reproduction | regeneration part 202 Moving image decoding part 203 Stream receiving part 204 Client control communication part 205 Parameter adjustment part 206 Image quality Setting memory

Claims (8)

A stream communication system that encodes video data on a server device, transmits the stream to a client device, and decodes the video data on the client device,
The server device
An encoding means for encoding video data based on predetermined parameters;
Stream transmitting means for streaming the encoded video data to the client device,
The client device is
Stream receiving means for receiving a transmitted stream and measuring a reception time which is a time taken to receive a stream of one frame;
Decoding means for decoding a received stream and measuring a decoding time which is a time taken to decode a one-frame stream;
Parameter adjusting means for calculating an adjusted parameter based on the parameter, the reception time, and the decoding time;
The encoding means of the server device includes:
Encode video data based on the adjusted parameters instead of the predetermined parameters ,
The parameters include a bit rate and a frame rate used when encoding video data,
A stream communication system, wherein the adjusted bit rate calculated by the parameter adjusting means is calculated by the following equation .
Br = TsBs / Tr
However, Br is the bit rate after adjustment, Ts is the reciprocal of the frame rate before adjustment, Bs is the bit rate before adjustment, and Tr is the larger of the reception time and the decoding time. The encoding means of the server device includes:
When the adjusted parameter calculated by the parameter adjusting means is a value exceeding the upper limit value of the preset parameter,
Video data, and wherein the encoding on the basis of the upper limit value of the parameter in place of the predetermined parameters, Claim 1 stream communication system according. The encoding means of the server device includes:
When the adjusted parameter calculated by the parameter adjusting means is a value lower than the preset lower limit value of the parameter,
Video data, and wherein the encoding based on the lower limit value of the parameter in place of the predetermined parameters, Claim 1 or 2 stream communication system according. Frame rate after adjustment calculated by the parameter adjusting means is characterized by a reciprocal of the larger one of the receiving time and the decoding time (Tr), any one of claims 1 to 3 of 1 The stream communication system according to item . 5. The stream communication system according to claim 4 , wherein the parameter adjusting unit determines whether or not to adjust the frame rate based on a user's preference for image quality. The parameter adjusting means determines a representative value of the reception time from a statistical value using a plurality of reception times measured by the stream receiving means, and represents a representative of the decoding time from a statistical value using a plurality of decoding times measured by the decoding means. Determine the value,
The reception time of the Tr uses a representative value of the reception time,
Wherein Tr decoding time is characterized by using a representative value of the decoding time, a stream communication system according to any one of claims 1 5. A server device that encodes video data and transmits it to a client device,
Wherein the client device is determined based on the reception time taken to receive the stream and decoding time taken to decode, after adjustment including bit rate and frame rate after adjustment to be used for encoding the video data parameters It receives,
Encode video data based on the adjusted parameters ,
The adjusted bit rate included in the received adjusted parameter is calculated by the following equation .
Br = TsBs / Tr
However, Br is the bit rate after adjustment, Ts is the reciprocal of the frame rate before adjustment, Bs is the bit rate before adjustment, and Tr is the larger of the reception time and the decoding time. A client device that receives and decodes a stream of encoded video data,
A reception time taken to receive the stream, on the basis of the decoding time required to decode the stream, the bit rate after adjustment to determine the bit rate and frame rate after adjustment to be used for encoding the video data was determined And the adjusted parameters including the frame rate are transmitted to the server device that transmits the video data ,
The adjusted bit rate included in the adjusted parameter is calculated by the following formula, and the client device:
Br = TsBs / Tr
However, Br is the bit rate after adjustment, Ts is the reciprocal of the frame rate before adjustment, Bs is the bit rate before adjustment, and Tr is the larger of the reception time and the decoding time.

Images