WO2011030680A1

WO2011030680A1 - Encoding device, encoding method, and encoding program

Info

Publication number: WO2011030680A1
Application number: PCT/JP2010/064603
Authority: WO
Inventors: 天野勝博
Original assignee: ブラザー工業株式会社
Priority date: 2009-09-08
Filing date: 2010-08-27
Publication date: 2011-03-17
Also published as: JP2011061362A

Abstract

Provided are an encoding device, an encoding method, and an encoding program, which will enable the bit rate of data output to the network to be changed in quick response to the fluctuation in the network band. A video conference device detects information about the network band and determines the generating condition of non-reference pictures on the basis of the detected information. An input image (41) is encoded with the determined generating condition to generate encoded data (48). From among the generated encoded data, the video conference device deletes non-reference pictures, which are not referred to when decoding any of the other pictures, when the network's available band drops. Only the remaining pictures that were not deleted are output to the network.

Description

Encoding apparatus, encoding method, and encoding program

The present invention relates to an encoding device, an encoding method, and an encoding program for compressing and encoding image data and outputting them to a network.

Conventionally, various techniques for outputting moving image data to a network are known. In the case of outputting data to a network, a technique for controlling an output bit rate according to a network bandwidth is known. For example, the communication service unit described in Patent Literature 1 performs data transmission according to a band by temporarily storing data in a buffer. In this case, since it is necessary to prevent the occurrence of buffer overflow, the bit rate of the encoder is also reduced.

Japanese Patent Laid-Open No. 2002-217989

However, in the conventional technique for reducing the bit rate of the encoder, since it is necessary to go through an encoding process or the like after the bit rate is reduced until data is actually output at the reduced bit rate, time is required. There is a delay. Therefore, it has been impossible to change the bit rate of the output to the network by quickly following the fluctuation of the band.

An object of the present invention is to provide an encoding device, an encoding method, and an encoding program that can change the bit rate of data output to a network by quickly following changes in the bandwidth of the network.

The encoding device according to the first aspect of the present invention encodes image data input by an image input means, and a reference picture that is a picture to be referred to when other pictures are decoded, and any other pictures A non-reference picture that is a picture that is not referred to even when decoding the video, and outputs the result to a network, the detection means for detecting information on the bandwidth of the network, and the non-reference picture A generating unit that determines the generation conditions based on the band information detected by the detecting unit, an encoding unit that encodes image data under the generation conditions determined by the determining unit, and the encoding unit Deletion to delete non-reference pictures among encoded pictures according to the band information detected by the detection means And stage, and an output means for outputting the remaining pictures in which the non-reference picture is deleted to the network by said deletion means.

The encoding apparatus according to the first aspect can determine the generation condition of the non-reference picture that has little influence on the video quality even if it is deleted according to the information of the network bandwidth. Then, the generated non-reference picture can be deleted before being output to the network according to the information of the network bandwidth. Therefore, the bit rate of data output can be changed by quickly following the fluctuation of the network bandwidth without causing a time delay due to the encoding process or the like.

The detection means may include first detection means for detecting a fluctuation amount of the network bandwidth as the bandwidth information. The determining means determines a ratio of the number of non-reference pictures to the number of pictures generated within a predetermined period of reference pictures and non-reference pictures based on the variation detected by the first detection means. What is necessary is just to determine as production | generation conditions. Since the encoding apparatus can delete many non-reference pictures if the ratio of non-reference pictures is increased, the bit rate of output to the network can be drastically reduced. On the other hand, the encoding apparatus can perform the encoding process efficiently by reducing the ratio of non-reference pictures. Therefore, the encoding apparatus can perform appropriate processing by generating the proportion of non-reference pictures that have little effect on video quality even if deleted according to the amount of bandwidth variation.

The determination unit is configured to calculate a ratio of a non-reference picture generated by encoding a prediction error with another picture by inter-frame encoding to a number of pictures generated within the predetermined period, The generation condition may be determined. In image compression coding, an I picture (Intra-coded Picture) generated by intra-frame coding becomes a non-reference picture, and a P picture (Predictive-coded Picture) and B picture generated by inter-frame coding. (Bidirectional-coded Picture) may be a non-reference picture. The data size of a picture generated by interframe coding is smaller than the data size of an I picture generated by intraframe coding. Therefore, when the determining unit determines the generation ratio of the non-reference picture generated by the interframe coding, the data size of the generated picture does not increase rapidly even when the generation ratio is changed. Therefore, the load applied to the network bandwidth is not increased.

The detection means may include second detection means for detecting the bandwidth of the network as the bandwidth information. The deletion unit may delete the non-reference picture when the band detected by the second detection unit is lowered. In this case, the encoding device can quickly reduce the bit rate of data output when the band is reduced. Therefore, it is possible to appropriately prevent the bandwidth from becoming congested.

The encoding device may include storage control means for storing the picture encoded by the encoding means in a buffer for temporarily storing data. The deletion unit may delete the non-reference picture stored in the buffer. In this case, since the encoding apparatus can appropriately delete the non-reference picture stored in the buffer, the deletion process can be easily performed.

The detection means may include third detection means for detecting a fluctuation amount of the free capacity of the buffer as the band information. The determination unit may determine the generation condition based on the fluctuation amount of the free space of the buffer detected by the third detection unit. In this case, the encoding apparatus can measure network bandwidth information including internal factors in the apparatus from the buffer, and determine the generation condition of the non-reference picture. Therefore, a non-reference picture can be generated more appropriately.

The determination means minimizes a deviation in the number of reference pictures generated between a non-reference picture and a next non-reference picture generated among a plurality of repeatedly generated pictures. It is desirable to determine the generation conditions. In this case, non-reference pictures in consecutive pictures are generated in a distributed manner without being generated in a biased manner. Therefore, even when the encoding apparatus deletes the non-reference picture, it is possible to reduce the possibility that the video is reproduced so as to be interrupted.

It is desirable that the deleting means deletes a non-reference picture that is an I picture in preference to a non-reference picture that is a P picture. By preferentially deleting an I picture whose data amount is larger than that of a P picture, it is possible to quickly follow a change in network bandwidth and reduce the number of pictures to be deleted.

The encoding method according to the second aspect of the present invention encodes the image data input by the image input means, and a reference picture that is a picture to be referred to when other pictures are decoded, and any other pictures An encoding method for generating a continuous picture composed of non-reference pictures which are pictures that are not referenced even at the time of decoding, and outputting them to a network, the step of detecting information on the bandwidth of the network, and the non-reference pictures Are determined based on the band information detected by the detection step, an encoding step for encoding image data under the generation condition determined by the determination step, and the encoding step. Among the encoded pictures, a non-reference picture is detected by the detection step. A deletion step of deleting in response to the information, the remaining pictures in which the non-reference picture is deleted by said deletion step and an output step of outputting to the network.

According to the encoding method according to the second aspect, it is possible to determine the generation condition of the non-reference picture that has little influence on the video quality even if it is deleted, according to the network bandwidth information. Then, the generated non-reference picture can be deleted before output according to network bandwidth information. Therefore, the bit rate of data output can be changed by quickly following the fluctuation of the network bandwidth without causing a time delay due to the encoding process or the like.

The encoding program according to the third aspect of the present invention encodes image data input by the image input means, and a reference picture that is a picture that is referenced when other pictures are decoded, and any other pictures An encoding program for generating a continuous picture composed of non-reference pictures which are pictures that are not referenced even at the time of decoding, and outputting them to a network, the computer detecting a network bandwidth information; and A determination step of determining a non-reference picture generation condition based on information of a band detected by the detection step; an encoding step of encoding image data using the generation condition determined by the determination step; Non-reference pictures among the pictures encoded by the encoding step are detected. A deletion step of deleting in accordance with the bandwidth of the information detected by the step, and wherein the remaining pictures in which the non-reference picture is deleted by said deletion step be executed and an output step of outputting to the network.

According to the encoding program according to the third aspect of the present invention, it is possible to determine the generation condition of the non-reference picture that has little influence on the video quality even if it is deleted according to the information of the network bandwidth. Then, the generated non-reference picture can be deleted before output according to network bandwidth information. Therefore, the bit rate of data output can be changed by quickly following the fluctuation of the network bandwidth without causing a time delay due to the encoding process or the like.

2 is a block diagram showing an electrical configuration of the video conference apparatus 1. FIG. It is a functional block diagram of the video conference apparatus 1 which concerns on 1st embodiment. It is a flowchart of the main process which the video conference apparatus 1 performs. This is an example of a generation condition determined when the rate of variation is 0% and the number of pictures in a GOP is 10. This is an example of a generation condition determined when the rate of variation is 50% and the number of pictures in a GOP is 10. This is an example of a generation condition determined when the rate of variation is 40% and the number of pictures in a GOP is 10. This is an example of a generation condition determined when the rate of variation is 90% and the number of pictures in a GOP is 10. It is a flowchart of the process for every frame performed by the main process which concerns on 1st embodiment. It is a functional block diagram of the video conference apparatus 101 which concerns on 2nd embodiment. It is a flowchart of the process for every frame performed by the main process which concerns on 2nd embodiment.

Hereinafter, a video conference apparatus 1 according to a first embodiment that embodies the encoding apparatus of the present invention will be described with reference to the drawings. The drawings to be referred to are used for explaining technical features that can be adopted by the present invention. The configuration of the apparatus, the flowcharts of various processes, and the like described in the drawings are not intended to be limited to these, but are merely illustrative examples.

The video conference apparatus 1 is connected to another video conference apparatus 1 via the network 8 (see FIG. 1). Each video conference device 1 inputs and outputs image data and audio data. As a result, users at multiple bases can share video and audio. Therefore, even when all the users are not at the same base, the users can smoothly execute the conference.

The electrical configuration of the video conference device 1 will be described with reference to FIG. The video conference apparatus 1 includes a CPU 10 that controls the video conference apparatus 1. A ROM 11, a RAM 12, a hard disk drive (hereinafter referred to as “HDD”) 13, and an input / output interface 19 are connected to the CPU 10 via a bus 18.

The ROM 11 stores a program for operating the video conference device 1 and initial values. The RAM 12 temporarily stores various information used in the control program. The HDD 13 is a non-volatile storage device that stores various types of information. Instead of the HDD 13, a storage device such as an EEPROM or a memory card may be used.

The input / output interface 19 is connected to an audio input processing unit 21, an audio output processing unit 22, a video input processing unit 23, a video output processing unit 24, an operation unit 25, and an external communication I / F 26. The voice input processing unit 21 processes input of voice data from the microphone 31 that inputs voice. The audio output processing unit 22 processes the operation of the speaker 32 that outputs audio. The video input processing unit 23 processes input of video data (moving image data) from the camera 33 that captures video. The video output processing unit 24 processes the operation of the display device 34 that displays video. The operation unit 25 is used for a user to input various instructions to the video conference apparatus 1. The external communication I / F 26 connects the video conference device 1 to the network 8.

The RAM 12 will be described in detail. The RAM 12 is provided with various storage areas such as a work area 121 and a FIFO buffer area 122 (hereinafter also referred to as “FIFO buffer 122”). The work area 121 stores various data such as flags necessary for processing. In the FIFO buffer area 122, encoded data that is encoded image data is temporarily stored before being output to the network 8. The FIFO buffer is a buffer that outputs stored data in the order in which they are stored.

Next, an outline of image data processing in the video conference apparatus 1 will be described. The video conference apparatus 1 converts the image data input from the camera 33 into H.264. Image compression encoding is performed based on the H.264 standard to generate encoded data. The video conference apparatus 1 outputs the generated encoded data to another video conference apparatus 1 via the network 8. Note that the video conference device 1 decodes the encoded data input from the other video conference device 1 via the network 8 and causes the display device 34 to display the decoded data. However, since this process is not the main part of the present invention, the following will describe the process of image compression encoding and output of encoded data. Image compression coding includes intra-frame coding and inter-frame coding.

The intra-frame coding is coding performed by intra-screen prediction within one frame of image data of a plurality of consecutive frames input by the camera. An I picture (Intra-coded Picture) that is encoded data generated by intra-frame encoding is decoded independently without referring to other pictures.

On the other hand, in interframe coding, prediction data is calculated by referring to data of a frame different from data of a frame to be encoded among continuous frame data, and the calculated prediction error is encoded. Coded data generated by interframe coding includes a P picture (Predictive-coded Picture) and a B picture (Bidirectional-coded Picture). In this embodiment, P pictures generated by referring to past pictures are mainly used. To decode a P picture, a picture referenced at the time of encoding is required. However, the data amount of the P picture is smaller than that of the I picture that is decoded alone.

A picture that is referred to when any other P picture is decoded is referred to as a “reference picture”. On the other hand, a picture that is not referred to when any other P picture is decoded is referred to as a “non-reference picture”. Generally, when an I picture and a P picture are generated in succession, the I picture is often used as a reference picture. However, when encoded data is generated in the order of I picture, I picture, and P picture, if the I picture immediately before the P picture is referenced when the P picture is decoded, the first I picture is a non-reference picture. It becomes. When decoding a P picture, it is possible to refer to either an I picture or a P picture. A P picture can be either a reference picture or a non-reference picture.

As shown in FIG. 2, in the video conference device 1, first, DCT / quantization 42 is performed on the input image 41 from the camera 33. Here, the coefficient transformed by DCT (Discrete Cosine Transform) is quantized according to the quantization parameter. Next, inverse quantization / inverse DCT 43 is performed on a part of the quantized data. The data subjected to the inverse quantization / inverse DCT 43 is subjected to the deblocking filter 44 and stored in the frame memory 45.

When intra-frame coding is performed, intra-frame prediction 46 is performed on the data stored in the frame memory 45, and further DCT / quantization 42 is performed. Entropy encoding 47 is performed on the quantized data. The encoded data 48 generated by the entropy encoding 47 is input to the FIFO buffer 122. As for the encoded data input to the FIFO buffer 122, only the encoded data that has not been deleted is subjected to the network output 50 through the non-reference picture deleting means 49 described later.

When inter-frame coding is performed, motion prediction 51 is performed using the input image 41, and motion compensation 52 based on the previous predicted image in the frame memory 45 is performed. The prediction error calculated by the motion compensation 52 is subjected to weighted prediction 53 using a weighting factor related to brightness, and further DCT / quantization 42 is performed. Entropy encoding 47 is performed on the quantized data, and encoded data 48 is generated. The subsequent flow is the same as in the case of intra-frame coding.

Here, the processing when the bandwidth of the network 8 is reduced (when narrowed) will be described. If data output at a high bit rate is performed in a state where the bandwidth is reduced, problems such as occurrence of congestion and packet loss may occur. As a result, the video quality is degraded.

Conventionally, for example, there is an apparatus that outputs encoded data corresponding to a band by providing a buffer (“FIFO buffer 122” in the present embodiment) that temporarily stores encoded data. In this apparatus, it is necessary to prevent the FIFO buffer 122 from overflowing. Therefore, the free capacity of the FIFO buffer 122 is monitored, and when the free capacity decreases, the data amount of the encoded data is adjusted by performing feedback to the quantization parameter in the DCT / quantization 42. However, a time delay occurs because it is necessary to go through an encoding process or the like after the quantization parameter is changed until the encoded data is actually generated based on the changed quantization parameter. Therefore, the bit rate cannot be changed quickly, and there is a possibility that overflow will occur. Further, in the method of buffering the encoded data by the FIFO buffer 122, a time delay occurs until the buffered encoded data is output to the network 8. Therefore, the bit rate cannot be changed by quickly following the fluctuation of the band. As a result, for example, when a video conference is being performed, reproduction of the video is delayed, and the user cannot smoothly execute the conference.

The video conference device 1 according to the first embodiment performs the available bandwidth measurement 55 and the FIFO buffer monitoring 56 of the network 8. Then, the non-reference picture deleting unit 49 deletes a part of the encoded data in the FIFO buffer 122 when the available bandwidth of the network 8 is reduced. By deleting a part of the encoded data to be output, the bit rate of output to the network 8 can be quickly reduced when the available bandwidth is reduced.

However, if the reference picture is deleted, the device that receives the encoded data cannot decode not only the deleted picture but also the P picture that needs to refer to the deleted picture at the time of decoding. Therefore, non-reference pictures should be deleted in order to prevent significant degradation of video quality. If the non-reference picture that can be deleted is not stored in the FIFO buffer 122 when the available bandwidth decreases, the bit rate cannot be changed in accordance with the decrease in the available bandwidth. Therefore, the video conference apparatus 1 measures the fluctuation amount of the usable bandwidth of the network 8 and the fluctuation amount of the free capacity of the FIFO buffer 122. The picture mode control block 58 changes the P picture generation ratio for all the pictures based on the measured variation. The reference picture control block 59 selects the reference picture of the P picture based on the measured variation, thereby changing the ratio of the non-reference picture to all the pictures. When the available bandwidth decreases, the non-reference picture is deleted. Details of the above processing will be described below.

A main process performed by the video conference apparatus 1 will be described with reference to FIGS. The main process is executed by the CPU 10 in accordance with a program stored in the ROM 11. The main process is started when an instruction to execute transmission / reception of image data is input.

As shown in FIG. 3, when the main process is started, various data are initialized (S1). The usable bandwidth (W) of the network 8 and the variation (A) of the usable bandwidth are measured (S2). For the measurement of available bandwidth, for example, a packet train transfer method is used for probe packet transfer, and pathload that estimates the available bandwidth using the increasing tendency of one-way transfer delay between probe packets, ICMP (INTERNET CONTROL MESSAGE PROTOCOL, etc.) ) ECHO REQUEST packets are continuously transmitted, and a known bandwidth measurement technique such as cprobe that obtains an available bandwidth by observing the packet interval of the response packet may be used. As the amount of change in the available bandwidth, a difference between the value of the available bandwidth (W) detected this time and the value of the available bandwidth (W) detected last time may be used. The fluctuation amount includes both an increase amount and a decrease amount.

Next, feedback to the quantization parameter is performed so that the encoded data is output in the measured available bandwidth (W) (S3). A ratio of the fluctuation amount (A) of the usable bandwidth to the usable bandwidth (W) is calculated (S4). A non-reference picture generation condition is determined from the calculated variation amount ratio (S5).

Referring to FIGS. 4 to 7, the method for determining the non-reference picture generation condition will be described in detail. The non-reference picture generation condition mainly indicates a ratio of the number of non-reference pictures to a preset number of pictures generated within a predetermined time (for example, 1 second). Specifically, the video conference apparatus 1 determines the number of non-reference pictures by determining the number of pictures in the GOP (Group of Pictures) and the number of reference pictures that are referenced when the P picture is encoded and decoded. Determine the percentage. A GOP is a set of a preset number of pictures generated within a predetermined period in order to efficiently manage a plurality of data. By reducing the number of reference pictures in the GOP, the ratio of the number of non-reference pictures to the number of pictures in the GOP can be increased. Further, as will be described below, the proportion of non-reference pictures can be determined by appropriately determining which reference picture of the P picture is used. The reference picture control block 59 (see FIG. 2) described above controls the motion compensation 52 based on the determined relationship between the P picture and the reference picture.

FIG. 4 is an example of non-reference picture generation conditions that are finally determined when the rate of change in the available bandwidth is 0% and the number of pictures in the GOP is 10. If the rate of change in the usable bandwidth is 0%, the possibility that the usable bandwidth is suddenly reduced is low. Therefore, it is rare to rapidly reduce the bit rate by deleting a large number of pictures. Therefore, it is not necessary to increase the ratio of non-reference pictures. If it is not necessary to increase the number of non-reference pictures, the reference picture of a P picture is preferably the picture immediately before the P picture. This is because the prediction error is reduced and the amount of data is reduced by using the immediately preceding picture as a reference picture. Therefore, as shown in FIG. 4, all the reference pictures of the P picture are pictures immediately before the P picture. As a result, the non-reference picture is only the last picture in the GOP.

FIG. 5 is an example of non-reference picture generation conditions determined when the rate of change in the available bandwidth is 50% and the number of pictures in the GOP is 10. The video conference apparatus 1 determines the generation condition of the non-reference picture so that the ratio of the change amount of the available band and the ratio of the non-reference picture in the picture in the GOP are closest. Therefore, in the example shown in FIG. 5, since the ratio of the amount of change in the usable bandwidth is 50%, the generation condition is determined so that 5 out of 10 pictures are non-reference pictures.

Furthermore, the video conference apparatus 1 determines the generation condition so that the reference picture and the non-reference picture are arranged equally. In other words, the generation condition is determined so that the deviation in the number of reference pictures located between the two most recent non-reference pictures is minimized. When this deviation is large, the reference picture and the non-reference picture are not evenly arranged. Therefore, there is a high possibility that problems such as video interruption occur when non-reference pictures are deleted. Therefore, in the example illustrated in FIG. 5, the video conference apparatus 1 alternately arranges reference pictures and non-reference pictures. As a result, there is no bias in the number of reference pictures located between non-reference pictures. When determining the reference picture of the P picture, the closest (newest) picture among the reference pictures before the P picture is determined as the reference picture of the P picture. That is, by making the reference pictures of a plurality of P pictures the same picture, the proportion of non-reference pictures that are P pictures can be increased.

FIG. 6 shows an example of generation conditions determined when the rate of change in the available bandwidth is 40% and the number of pictures in the GOP is 10. If the rate of change in the usable bandwidth is 40%, the video conference apparatus 1 determines the generation condition so that four out of ten pictures are non-reference pictures. Then, the arrangement of the reference picture and the non-reference picture is determined so that the number of reference pictures located between the non-reference pictures is “2”, “1”, “2”, and “1” in order.

FIG. 7 shows an example of generation conditions determined when the rate of change in the available bandwidth is 90% and the number of pictures in the GOP is 10. If the rate of change in the available bandwidth is 90%, the video conference apparatus 1 determines the generation condition so that nine out of ten pictures are non-reference pictures. In this case, the reference pictures of all P pictures are the first I picture in the GOP. In this way, by setting the reference pictures of all P pictures in the GOP as I pictures, the ratio of non-reference pictures that are P pictures can be maximized.

Note that, as shown in FIGS. 4 to 7, the video conference apparatus 1 can determine the rate at which non-reference pictures that are P pictures are generated in the process of S5. Even if the ratio of I pictures to all pictures is increased, the ratio of non-reference pictures can be increased. However, the data size of the I picture is larger than the data size of the P picture. Therefore, the load applied to the network 8 increases. On the other hand, the video conference apparatus 1 can change the generation ratio of the non-reference picture by changing the ratio of the non-reference picture that is a P picture without rapidly increasing the load applied to the network 8. .

Returning to the explanation of FIG. When the non-reference picture generation conditions are determined (S5), the process for each frame is performed (S6). In the process for each frame, the image data is encoded (encoded) according to the determined generation condition, and the non-reference picture is deleted based on the information of the network bandwidth.

As shown in FIG. 8, when the processing for each frame is started, first, according to the determined quantization parameter (S3, see FIG. 3) and the non-reference picture generation condition (S5, see FIG. 3) Data is encoded one by one in order (S11). Encoded data that is encoded data is input to the FIFO buffer area 122 (S12). It is determined whether or not there is a non-reference picture in the FIFO buffer area 122 (S13). If there is no non-reference picture (S13: NO), the process proceeds to S21. When there is a non-reference picture (S13: YES), the available bandwidth (W) of the network 8 is measured (S15). The amount of change (C) in the free capacity of the FIFO buffer area 122 is measured (S16). The fluctuation amount (C) is the difference between the previous free capacity of the FIFO buffer area 122 and the current free capacity of the FIFO buffer area 122.

It is determined whether or not the available bandwidth (W) measured this time in S15 is lower than the available bandwidth (W) measured last time in S2 (see FIG. 3) or S15 (S17). If the current time is not lower (S17: NO), the process proceeds directly to S19. If the available bandwidth (W) measured this time is lower (S17: YES), the non-reference picture in the FIFO buffer area 122 is deleted (S18).

In the video conference apparatus 1, when there are a plurality of non-reference pictures in the FIFO buffer area 122, the number of non-reference pictures corresponding to the amount of decrease in available bandwidth (W) is deleted. Further, when both a non-reference picture that is a P picture and a non-reference picture that is an I picture are stored in the FIFO buffer area 122, an I picture having a larger data amount is preferentially deleted. For example, when an I picture (I) and a P picture (P) are generated in the order of I / I / P / I / I / P and the P picture refers to the immediately preceding I picture, The I picture two frames before is preferentially deleted. In the present embodiment, H.264 is used. Since the H.264 standard is adopted, an access unit (unit of picture defined in H.264) composed only of non-reference pictures is deleted.

Next, the fluctuation amount (A) of the available bandwidth measured last time in S2 (see FIG. 3) in order to determine the generation condition of the non-reference picture, and the fluctuation amount of the free capacity of the FIFO buffer area 122 measured this time in S16. The difference from (C) is calculated (S19). The remaining encoded data that has not been deleted in the FIFO buffer area 122 is sequentially output to the network 8 so that the bit rate of data output follows the available bandwidth (W) of the network 8 (S21).

Next, it is determined whether or not the absolute value of the difference in fluctuation amount calculated in S19 is equal to or larger than the absolute value of the difference in changeable value of the output bit rate when the generation condition is updated (S22). For example, when processing is performed with 15 pictures in a GOP, each time a new non-reference picture in one GOP is deleted or the deletion is stopped by one, the output bit rate is changed. The average value can be changed by about 6.7%. Therefore, in this case, the changeable value of the output bit rate is increased by about 6.7% by updating the generation condition so that the number of non-reference pictures in the GOP is increased by one. Conversely, when the generation condition is changed so as to reduce the number of non-reference pictures in the GOP by one, the changeable value of the output bit rate is reduced by about 6.7%. The video conference apparatus 1 does not update the generation condition when the change in the amount of change is small, and updates only when the change in the amount of change is large. Therefore, in S22, the absolute value of the difference between the changeable value of the output bit rate before the generation condition is updated and the changeable value of the output bit rate after the update is calculated. If the absolute value of the difference in variation calculated in S19 is smaller than the absolute value of the difference in output bit rate changeable value (S22: NO), the process directly returns to S11. If it is equal to or greater than the absolute value of the difference in the changeable value of the output bit rate (S22: YES), the process returns to the main process (see FIG. 3) in order to update the generation condition. In the main process, when the frame-by-frame process (S6) is completed, the process returns to S2, and the non-reference picture generation conditions are updated (S2 to S5).

As described above, the video conference apparatus 1 according to the first embodiment determines the non-reference picture generation condition based on the band information of the network 8. The image data is encoded with the determined generation conditions. In accordance with the band information, the non-reference picture is deleted from the encoded picture, and the remaining picture is output to the network. Therefore, the video conference apparatus 1 determines the generation condition of the non-reference picture that has little influence on the video quality even if it is deleted according to the band information of the network 8, and outputs the generated non-reference picture to the network. Can be deleted before. Therefore, the video conference apparatus 1 can quickly follow the fluctuation of the bandwidth of the network 8 and change the output bit rate without causing a time delay due to encoding processing, buffering, or the like.

Specifically, the video conference apparatus 1 can quickly and rapidly reduce the output bit rate by increasing the ratio of non-reference pictures. On the other hand, if the proportion of non-reference pictures is reduced, the reference picture of a P picture can be made as close as possible to that P picture, and the encoding process can be performed efficiently. The video conference apparatus 1 determines the ratio of non-reference pictures to all pictures as a generation condition from the fluctuation amount of the available bandwidth of the network 8. Therefore, it is possible to appropriately generate a non-reference picture that has little influence on the video quality even if it is deleted in accordance with the amount of change in the available bandwidth. When the available bandwidth actually decreases, the output bit rate can be quickly decreased by deleting the generated non-reference picture.

In image compression coding, there are cases where an I picture generated by intra-frame coding is a non-reference picture and cases where a P picture and a B picture generated by inter-frame coding are non-reference pictures. The data size of a picture generated by interframe coding is smaller than the data size of an I picture generated by intraframe coding. The video conference apparatus 1 can appropriately determine the generation ratio of non-reference pictures (non-reference pictures that are P pictures) generated by interframe coding. As a result, even when the generation ratio is changed, the total amount of data size of the generated picture does not increase rapidly. Therefore, the load applied to the network bandwidth is not increased.

The video conference apparatus 1 temporarily stores the generated encoded data in the FIFO buffer area 122. Accordingly, the deletion process can be easily performed by appropriately deleting the non-reference pictures stored in the FIFO buffer area 122. It is also possible to delete a plurality of non-reference pictures at once. In addition, the video conference apparatus 1 acquires network bandwidth information including internal factors of the video conference apparatus 1 by measuring the amount of change in the free capacity of the FIFO buffer area 122. When the obtained amount of variation becomes large, the non-reference picture generation conditions can be updated. Therefore, a non-reference picture can be generated by appropriately reflecting internal factors of the video conference apparatus 1 itself.

The video conference apparatus 1 can generate a plurality of non-reference pictures by distributing them without bias. Therefore, even when a plurality of non-reference pictures are deleted, it is possible to reduce the possibility that the video is reproduced so as to be interrupted. Further, when deleting the non-reference picture, the video conference apparatus 1 can reduce the output bit rate more quickly and efficiently by preferentially deleting the I picture having a larger data amount than the P picture. .

In the first embodiment, the video conference device 1 corresponds to the “encoding device” of the present invention. The camera 33 corresponds to “image input means”. The usable bandwidth (W) of the network 8, the usable bandwidth variation (A), and the free capacity variation (C) of the FIFO buffer area 122 correspond to “network bandwidth information”. The CPU 10 that detects band information in S2 of FIG. 3 and S15 and S16 of FIG. 8 functions as a “detection unit”. The CPU 10 that determines the non-reference picture generation conditions in S5 of FIG. 3 functions as a “determination unit”. The CPU 10 that encodes the image data in S11 of FIG. 8 functions as an “encoding unit”. The CPU 10 that deletes the non-reference picture in S18 of FIG. 8 functions as a “deleting unit”. The CPU 10 that outputs the encoded data in S21 of FIG. 8 functions as an “output unit”.

3, the CPU 10 that measures the fluctuation amount (A) of the usable bandwidth of the network 8 in S2 of FIG. The CPU 10 that measures the available bandwidth (W) of the network 8 in S2 of FIG. 3 and S15 of FIG. 8 functions as a “second detection unit”. The FIFO buffer area 122 of the RAM 12 corresponds to a “buffer”. The CPU 10 that inputs the encoded data to the FIFO buffer area 122 in S12 of FIG. 8 functions as a “storage control unit”. The CPU 10 that measures the fluctuation amount of the free capacity of the FIFO buffer area 122 in S16 of FIG. 8 functions as a “third detection unit”.

The process of detecting band information in S2 of FIG. 3 and S15 and S16 of FIG. 8 corresponds to the “detection step” of the present invention. The process of determining the non-reference picture generation condition in S5 of FIG. 3 corresponds to the “determination step”. The process of encoding the image data in S11 of FIG. 8 corresponds to the “encoding step”. The process of deleting the non-reference picture in S18 of FIG. 8 corresponds to a “deletion step”. The process of outputting the encoded data in S21 of FIG. 8 corresponds to an “output step”.

Next, the video conference apparatus 101 according to the second embodiment of the present invention will be described with reference to FIG. 9 and FIG. The video conference apparatus 101 according to the second embodiment is different from the video conference apparatus 1 according to the first embodiment only in that the encoded data is not buffered. Therefore, the same number is attached | subjected about the same structure and process, and this description is abbreviate | omitted or simplified.

With reference to FIG. 9, an outline of image data processing in the video conference apparatus 101 will be described. As shown in FIG. 9, the video conference apparatus 101 does not include the FIFO buffer 122 (see FIG. 2). The picture mode control block 58 and the reference picture control block 59 generate a non-reference picture using information obtained by the available bandwidth measurement 55 of the network 8. Thus, the present invention can be implemented without using a buffer for buffering encoded data. Details of the processing will be described below.

The CPU 10 of the video conference apparatus 101 starts a main process when an instruction to execute transmission / reception of image data is input. The main process is the same as the main process (see FIG. 3) performed by the video conference apparatus 1 of the first embodiment except for the frame-by-frame process described below. Therefore, the description of the main process is omitted. As shown in FIG. 10, when the processing for each frame is started, the image data is converted according to the determined quantization parameter (S3, see FIG. 3) and the non-reference picture generation conditions (S5, see FIG. 3). One by one is encoded (S11). It is determined whether there is a non-reference picture in the encoded data output to the network 8 (S113). If there is no non-reference picture (S113: NO), the process proceeds to S121.

When there is a non-reference picture (S113: YES), the available bandwidth (W) of the network 8 is measured (S15). The fluctuation amount (B) of the usable bandwidth of the network 8 is measured (S116). It is determined whether or not the available bandwidth (W) measured this time in S15 is lower than the available bandwidth (W) measured last time in S2 (see FIG. 3) or S15 (S17). If the current time is not lower (S17: NO), the process proceeds to S119 as it is. If the available bandwidth (W) measured in the current processing of S15 is lower (S17: YES), the non-reference picture before output to the network is deleted (S118).

Next, the difference between the fluctuation amount (A) of the available bandwidth measured last time in S2 (see FIG. 3) and the fluctuation amount (B) of the available bandwidth measured this time in S116 is calculated (S119). The encoded data that has not been deleted is output to the network 8 (S121). It is determined whether or not the absolute value of the difference in the available bandwidth fluctuation amount calculated in S119 is equal to or greater than the absolute value of the difference in the changeable value of the output bit rate when the generation condition is updated (S22). If the absolute value of the difference in the amount of change in the available bandwidth is smaller than the absolute value of the difference in the changeable value of the output bit rate (S22: NO), the process returns to S11 as it is. If the difference is greater than or equal to the absolute value of the changeable value (S22: YES), the process returns to the main process (see S3) in order to update the generation condition.

As described above, the video conference apparatus 101 according to the second embodiment determines a non-reference picture generation condition in accordance with the amount of change in the available bandwidth of the network 8, and selects a non-reference picture when the available bandwidth decreases. Can be deleted. Accordingly, the output bit rate to the network 8 can be changed by quickly following the change in the available bandwidth. As described above, the present invention can be implemented without using a buffer for buffering encoded data.

In the second embodiment, the available bandwidth (W) of the network 8 and the fluctuation amount (A, B) of the available bandwidth correspond to “network bandwidth information” of the present invention. The CPU 10 that detects band information in S2 of FIG. 3 and S15 and S116 of FIG. 10 functions as a “detection unit”. The CPU 10 that encodes the image data in S11 of FIG. 10 functions as an “encoding unit”. The CPU 10 that deletes the non-reference picture in S118 in FIG. 10 functions as a “deleting unit”. The CPU 10 that outputs the encoded data in S121 of FIG. 10 functions as an “output unit”.

It goes without saying that the present invention is not limited to the above-described embodiment, and various modifications are possible. For example, the present invention is not limited to a video conference apparatus. In addition, the present invention can be applied to any device that outputs encoded data via a network, such as a server that distributes video. In the above embodiment, the Coding is performed based on the H.264 standard, but other standards may be adopted.

The video conference apparatuses 1 and 101 of the above embodiment perform processing such as deletion of non-reference pictures based on the value of the available bandwidth of the network 8. However, the video conference apparatuses 1 and 101 may measure the bandwidth actually used in the network 8 and perform processing.

In the determination of the non-reference picture generation condition performed in S5 of FIG. 3, a condition other than the condition determined in the above embodiment may be determined. For example, in addition to the ratio of non-reference pictures to all pictures, the frame rate, resolution, and the like may be determined as generation conditions.

In the processing of S118 of FIG. 10 in the second embodiment, it is not always necessary to delete the non-reference picture when the available bandwidth of the network 8 is reduced. For example, the video conference apparatus 101 may delete a non-reference picture when the amount of decrease in available bandwidth exceeds a threshold value.

In the above embodiment, the non-reference picture generation condition is updated when the change amount of the fluctuation amount of the free capacity of the FIFO buffer 122 or the change amount of the fluctuation amount of the usable bandwidth becomes large (FIG. 8). (See S22 and S22 in FIG. 10). However, the trigger for updating the non-reference picture generation condition can also be changed. For example, the generation condition may be repeatedly updated every predetermined time or every time a predetermined number of pictures are output.

1,101 Video conference device 8 Network 10 CPU
12 RAM
33 Camera 122 FIFO buffer area

Claims

Encoding image data input by the image input means, from a reference picture that is a picture that is referenced when other pictures are decoded, and a non-reference picture that is a picture that is not referenced when any other picture is decoded An encoding device that generates a continuous picture and outputs it to a network,
Detecting means for detecting information of a bandwidth of the network;
A determination unit that determines a generation condition of the non-reference picture based on information on a band detected by the detection unit;
Encoding means for encoding image data under the generation conditions determined by the determination means;
A deletion unit that deletes a non-reference picture among the pictures encoded by the encoding unit in accordance with band information detected by the detection unit;
An encoding device comprising: output means for outputting, to the network, a remaining picture from which the non-reference picture has been deleted by the deleting means.
The detection means includes first detection means for detecting a fluctuation amount of the bandwidth of the network as the bandwidth information,
The determining means, based on the fluctuation amount detected by the first detecting means, calculates a ratio of the number of non-reference pictures to the number of pictures generated within a predetermined time consisting of reference pictures and non-reference pictures. The encoding apparatus according to claim 1, wherein the encoding condition is determined as a generation condition.
The determination unit is configured to calculate a ratio of a non-reference picture generated by encoding a prediction error with another picture by inter-frame encoding to a number of pictures generated within the predetermined time, The encoding apparatus according to claim 2, wherein the encoding condition is determined as the generation condition.
The detection means comprises second detection means for detecting the bandwidth of the network as the bandwidth information,
The encoding apparatus according to claim 1, wherein the deleting unit deletes the non-reference picture when the band detected by the second detecting unit is lowered.
A storage control means for storing the picture encoded by the encoding means in a buffer for temporarily storing data;
The encoding apparatus according to claim 1, wherein the deletion unit deletes a non-reference picture stored in the buffer.
The detection means includes third detection means for detecting a fluctuation amount of the free capacity of the buffer as information of the band,
6. The encoding apparatus according to claim 5, wherein the determination unit determines the generation condition based on a fluctuation amount of the free space of the buffer detected by the third detection unit.
The determination means minimizes a deviation in the number of reference pictures generated between generation of a non-reference picture and generation of a non-reference picture next among the repeatedly generated pictures. The encoding apparatus according to claim 1, wherein the condition is determined.
The deletion means is an I picture generated by intraframe encoding rather than a non-reference picture that is a P picture generated by encoding a prediction error with another picture by interframe encoding. 2. The encoding apparatus according to claim 1, wherein a certain non-reference picture is deleted preferentially.
By encoding the image data input by the image input means, from a reference picture that is a picture that is referenced when other pictures are decoded and a non-reference picture that is a picture that is not referenced when any other picture is decoded An encoding method for generating a continuous picture and outputting to a network,
A detecting step for detecting information of a bandwidth of the network;
A determination step of determining a generation condition of the non-reference picture based on information of a band detected by the detection step;
An encoding step for encoding image data under the generation conditions determined by the determination step;
A deletion step of deleting a non-reference picture among the pictures encoded by the encoding step according to the band information detected by the detection step;
And an output step of outputting the remaining picture from which the non-reference picture has been deleted by the deleting step to the network.
By encoding the image data input by the image input means, from a reference picture that is a picture that is referenced when other pictures are decoded and a non-reference picture that is a picture that is not referenced when any other picture is decoded An encoding program for generating a continuous picture and outputting it to a network,
On the computer,
A detecting step for detecting information of a bandwidth of the network;
A determination step of determining a generation condition of the non-reference picture based on information of a band detected by the detection step;
An encoding step for encoding image data under the generation conditions determined by the determination step;
A deletion step of deleting a non-reference picture among the pictures encoded by the encoding step according to the band information detected by the detection step;
And a step of outputting, to the network, a remaining picture from which the non-reference picture has been deleted by the deleting step.