JP5603747B2 - Video encoding device - Google Patents

Description

  The present invention relates to a video encoding apparatus that encodes a moving image, and in particular, a video encoding apparatus that buffers a reconstructed image or a stream in a predictive encoding technique in an external memory using a DRAM (Dynamic Random Access Memory). About.

  In recent years, video coding technology has become an indispensable technology due to the increase in video distribution content due to the development of broadband networks and the use of large-capacity storage media such as DVD (Digital Versatile Disc) and large-screen video display devices. . In addition to the high resolution of the imaging device and the display device, a technology for encoding at a high resolution in the moving image encoding technology is indispensable.

  For example, as one of the moving image encoding technologies capable of encoding with high resolution and high image quality, the international standard H.264 is available. H.264 / AVC (Advanced Video Coding). The encoding process is a process of converting an original image that is an input image into a stream having a smaller data amount. H. In the H.264 / AVC encoding scheme, encoding is performed using a prediction technique such as intra-frame prediction (intra-frame prediction) and inter-frame prediction (inter-frame prediction).

  H. There are mainly two prediction methods used in H.264 / AVC encoding: intra prediction and inter prediction. Furthermore, in the intra prediction, a plurality of prediction methods are prepared according to combinations of block sizes and prediction directions serving as prediction units. Also in the inter-screen prediction, a plurality of prediction methods are prepared according to the size of a block which is a prediction unit. H. In H.264 / AVC, an encoding method with high image quality and high compression is realized by dynamically selecting these prediction methods according to the target image quality and code amount.

  Here, referring to FIG. An outline of H.264 / AVC encoding will be described. FIG. 1 is a diagram illustrating a configuration of a conventional video encoding device that performs H.264 / AVC encoding processing.

  In the encoding process using intra prediction, the mode selection unit 930 selects the intra prediction unit 910 side. Then, the stream 91 is obtained from the original image 90 through the intra-screen prediction unit 910, the orthogonal transform unit 940, the quantization unit 950, and the variable length coding unit 980. In the encoding process using inter-screen prediction, the mode selection unit 930 selects the inter-screen prediction unit 920 side. Then, the stream 91 is obtained from the original image 90 through the inter-screen prediction unit 920, the orthogonal transform unit 940, the quantization unit 950, and the variable length encoding unit 980.

  Next, H.I. The contents of each process in H.264 / AVC encoding will be described.

  The intra-screen prediction unit 910 includes a reconstructed image 92 configured by adding the original image 90, the restored difference image 97 output from the inverse orthogonal transform unit 970, and the predicted image 95 output from the mode selection unit 930. Is entered. Then, an appropriate intra-screen prediction mode is selected from the original image 90 and the reconstructed image 92 by intra-screen prediction processing, and intra-screen prediction information D81 representing the mode information of the intra-screen prediction mode, and the intra-screen prediction result A predicted image 93 and an intra-screen prediction error D82 representing the difference between the original image 90 and the intra-screen predicted image 93 are generated.

  The inter-screen prediction unit 920 receives input of the reconstructed image 92 generated from the input original image 90 and the original images 90 before and after (the past or the future), and receives the inter-screen prediction information D83 and the inter-screen prediction image 94. And an inter-screen prediction error D84 representing the difference between the original image 90 and the inter-screen prediction image 94 is generated.

  The encoding control unit 990 includes an intra-screen prediction error D82 input from the intra-screen prediction unit 910, an inter-screen prediction error D84 input from the inter-screen prediction unit 920, and a code amount input from the variable length encoding unit 980. From information D86 (described later), one of the encoding modes of intra prediction and inter prediction is determined according to the mode selection algorithm. Then, mode selection information D87 indicating the determined encoding mode is output to mode selection section 930. Further, the quantization coefficient D88 is determined according to the rate control algorithm and output to the quantization unit 950.

  Note that the mode selection algorithm and the rate control algorithm greatly affect the code amount and image quality of the stream 91, and therefore there are various methods depending on the content of the original image 90 to be encoded and the purpose of video encoding. .

  In accordance with the mode selection information D87 input from the encoding control unit 990, the mode selection unit 930 selects the intra-screen prediction image 93 when the intra-screen prediction unit 910 is selected, and selects the inter-screen prediction unit 920. In this case, the inter-screen prediction image 94 is output as the prediction image 95.

  The orthogonal transform unit 940 generates a frequency component D89 from the difference image 96 that is a difference between the original image 90 and the predicted image 95 by orthogonal transform processing.

  The quantization unit 950 performs a quantization process from the quantization coefficient D88 input from the encoding control unit 990 and the frequency component D89 input from the orthogonal transform unit 940, and generates a quantization value D90 with a reduced amount of information. Output.

  The inverse quantization unit 960 performs an inverse quantization process on the quantized value D90 to generate a restored frequency component D91.

  The inverse orthogonal transform unit 970 performs an inverse orthogonal transform process on the restored frequency component D91 to generate a restored difference image 97. Then, the generated restored difference image 97 and the predicted image 95 output by the mode selection unit 930 are added and stored as a reconstructed image 92.

  The variable length encoding unit 980 encodes the quantized value D90 and the intra-screen prediction information D81 or the inter-screen prediction information D83 into a data string having a smaller data amount, and outputs the data 91 as a stream 91. Also, the code amount information D86 is output to the encoding control unit 990. The code amount information D86 is information indicating the code amount of the stream 91 after variable length encoding.

  H. In H.264 / AVC encoding, an encoding operation is performed using a macroblock (hereinafter, referred to as “MB” as appropriate) composed of 16 × 16 pixel values as a processing unit. In the processing screen, the area to be processed by the MB moves the entire screen at regular intervals in the raster scan order from the upper left of the processing screen.

  Here, when the video encoding process is implemented, generally large-capacity data such as the reconstructed image 92 and the stream 91 is written to the external memory. As this external memory, a large-capacity and low-cost DRAM is mainly used. A DRAM is a recording-type memory that records information by adding a charge to a capacitor.

  Therefore, in the DRAM, the capacitor is lost in a certain time after the charge is added, and the recorded information may be lost. Therefore, the DRAM has an operation mode called refresh in which the value of each capacitor is read and overwritten again.

  In addition, since the charge stored in the capacitor is lost when the charge is read from the capacitor, the DRAM has a function of automatically rewriting the read information simultaneously with the reading. As a result, the information recorded on the DRAM can be prevented from disappearing by reading before the charge of the capacitor is lost or by performing a refresh operation.

  However, the DRAM cannot be accessed during the refresh operation. Therefore, there is a problem that the transfer bandwidth of the DRAM is reduced by performing the refresh. For such a problem, for example, a technique such as Patent Document 1 has been proposed. Patent Document 1 relates to an image processing apparatus that performs three-dimensional computer graphics.

  In the image processing apparatus described in Patent Document 1, the problem that the data transfer speed is lowered due to the prohibition of access to the DRAM being refreshed is caused by a method of performing refresh during a horizontal blanking period in which data transfer is not performed. It has been solved. In addition, this method prevents data loss in the DRAM without reducing the memory bandwidth during steady operation and realizes high-speed image processing.

JP-A-6-251166

  As described above, in the image processing apparatus disclosed in Patent Document 1, refreshing is performed during a period in which access to the DRAM is not performed, thereby preventing data loss on the DRAM without reducing the memory bandwidth during steady operation. Image processing is realized.

  On the other hand, there is a demand for encoding with a larger screen size that shortens the number of cycles in the horizontal blanking period and the vertical blanking period.

  However, the refresh method disclosed in Patent Document 1 is a technique that can be realized because there is a period during which no access to the DRAM is performed, which is a horizontal blanking period. Therefore, if the frame rate is constant because the screen size is large, the horizontal blanking period and the vertical blanking period become relatively small, or access to the DRAM for reasons such as transfer of audio data during the blanking period. When there is little free time during which data transfer is not performed, the time for refresh cannot be secured and the problem of data loss on the DRAM cannot be solved.

  Therefore, in view of these problems, the present invention prevents data loss on the DRAM even when encoding a video having a larger screen size that cannot secure sufficient time for refreshing in the video encoding device. An object of the present invention is to provide a video encoding device capable of preventing the failure of the video.

  In order to solve the above problems, the present invention provides a video encoding unit that encodes an input image to be input by intra-screen prediction or inter-screen prediction, and usage data used in the encoding by the video encoding unit. And a memory controller for refreshing the external memory, and the video encoding unit is connected to the external memory. A bandwidth determination unit that determines whether or not the external memory can be refreshed in a free time during which the use data is not transferred between the first and second bandwidths; and the bandwidth determination unit cannot refresh the external memory. If determined, the usage data will be used in the encoding within the refresh cycle of the external memory. And a reference screen selection means for setting a reference relationship in the intra-screen prediction or the inter-screen prediction, and the memory controller is capable of refreshing the external memory in the bandwidth determination means. If it is determined, a video encoding device is proposed in which the external memory is refreshed within the idle time.

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

  In addition, in the case of a video having a screen size that can secure a sufficient free time for refreshing in the horizontal blanking period or the vertical blanking period, the encoding can be performed by refreshing the external memory.

  In addition, the band determination unit compares a calculated value calculated based on the number of horizontal pixels and the number of vertical pixels of the input image among the pixels constituting the input image with a predetermined value, When the calculated value is less than the predetermined value, it is determined that the external memory can be refreshed. When the calculated value is equal to or larger than the predetermined value, the external memory is refreshed. You may come to judge that it is not possible.

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

  According to the present invention, refresh is not required by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

It is a block diagram which shows the structural example of the video coding apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structural example of the video encoding part 110 which concerns on 1st Embodiment. It is a block diagram which shows the structural example of the encoding control part 200 which concerns on 1st Embodiment. It is a block diagram which shows the structural example of the band determination part 310 which concerns on 1st Embodiment. It is a figure which shows an example of the reference relationship which needs refreshing. It is a figure which shows an example of the reference relationship from which a refresh is unnecessary. It is a figure explaining the concept of the rate control performed in the code amount control part. FIG. 4 is a diagram showing a command sequence that can be issued to the external memory 120 by the memory controller 290 within 1 MB processing time. It is a block diagram which shows the structural example of the band determination part 310 which concerns on 2nd Embodiment. It is a figure which shows the structure of the conventional video coding apparatus.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings referred to in the following description, the same parts as those in the other drawings are denoted by the same reference numerals.

[First Embodiment]
(Configuration of video encoding device)
The configuration of the video encoding device according to this embodiment will be described below. FIG. 1 is a block diagram illustrating a configuration example of a video encoding apparatus according to the present embodiment.

  A video encoding device 1 shown in FIG. 1 includes a video encoding unit 110, an external memory 120, and a memory bus 130 that connects the video encoding unit 110 and the external memory 120.

  The video encoding unit 110 receives an input of the original image 10 and the encoding setting information S1 from the outside of the video encoding device 1, and outputs a stream 11. The encoding setting information S <b> 1 is information for controlling the operation of the video encoding unit 110. The encoding setting information S1 includes code amount control setting information D25, mode selection setting information D24, reference screen setting information D23, and video information D21.

  The external memory 120 is a volatile memory that needs to be refreshed for holding recorded contents. In the present embodiment, the external memory 120 is implemented by a DRAM. As will be described later, the external memory 120 buffers the reconstructed image 12, the inter-screen predicted image 14, the stream 11, and the like output from the video encoding unit 110 via the memory bus 130.

  The video encoding device 1 is realized by a configuration similar to that of a general computer such as a memory (not shown) such as a CPU (Central Processing Unit) and a RAM (Random Access Memory), a storage device such as a hard disk, and a network interface. Is done. The function of each component of the video encoding device 1 is, for example, when the CPU reads and executes a program stored in a hard disk or the like, or for example, customizable sequencer logic in an FPGA (Field Programmable Gate Array) It is a function realized by designing.

(Configuration of Video Encoding Unit 110)
Hereinafter, each configuration of the video encoding unit 110 will be described. FIG. 2 is a diagram illustrating a configuration example of the video encoding unit 110.

  The video encoding unit 110 includes an encoding control unit 200, an intra-screen prediction unit 210, an inter-screen prediction unit 220, a mode selection unit 230, an orthogonal transform unit 240, a quantization unit 250, an inverse quantization unit 260, and an inverse orthogonal transform unit. 270, a variable length coding unit 280, and a memory controller 290.

  Here, the components other than the encoding control unit 200 and the memory controller 290 have the same functions as the components of the conventional video encoding device shown in FIG. In the following description, the description of the same parts as in the prior art will be simplified as appropriate.

  The intra-screen prediction unit 210 receives input of the original image 10 and the reconstructed image 12 and evaluates a plurality of intra-screen prediction modes. Then, the intra prediction code amount D2 in the intra prediction mode with the smallest prediction code amount is set to the encoding control unit 200, the intra prediction mode D1 is set to the variable length encoding unit 280, and the intra prediction image 13 is set to the mode. The data is output to the selection unit 230.

  The inter-screen prediction unit 220 receives the reconstructed image 12 from the memory controller 290 and the original image 10 from the outside of the video encoding unit 110. Then, a plurality of inter-screen prediction modes are evaluated. As a result, the inter-screen prediction code amount D4 in the inter-screen prediction mode with the smallest prediction code amount is stored in the encoding control unit 200, the inter-screen prediction mode D3 is stored in the variable length encoding unit 280, and the inter-screen prediction image 14 is displayed. Output to the memory controller 290.

  The mode selection unit 230 receives the intra-screen prediction image 13 from the intra-screen prediction unit 210, the inter-screen prediction image 14 from the memory controller 290, and the mode selection signal D7 from the encoding control unit 200. The mode selection signal D7 is the same information as the mode selection information D87 in FIG. Then, the mode selection unit 230 outputs the intra-screen prediction image 13 or the inter-screen prediction image 14 as the prediction image 15 in accordance with the mode selection signal D7.

  The orthogonal transform unit 240 performs an orthogonal transform process on the difference image 16 that is a difference between the predicted image 15 and the original image 10, and outputs a frequency component D9.

The quantization unit 250 receives the quantization parameter D8 from the encoding control unit 200 and the frequency component D9 from the orthogonal transform unit 240. The quantization parameter D8 is the same information as the quantization coefficient D88 in FIG. The quantization unit 250 quantizes the quantization parameter D8 and the frequency component D9, and outputs the quantization value D10. The variable length encoding unit 280 receives the quantization value D10 from the quantization unit 250. An intra-screen prediction mode D1 is input from the intra-prediction unit 210, and an inter-screen prediction mode D3 is input from the inter-screen prediction unit 220. Then, variable length encoding is performed, and the stream 11 is output to the memory controller 290 and the code amount information D6 is output to the encoding control unit 200. The code amount information D6 is the same information as the code amount information D86 of FIG.

  The inverse quantization unit 260 receives the quantized value D10 from the quantization unit 250, performs inverse quantization, and outputs a restored frequency component D11.

  The inverse orthogonal transform unit 270 receives the restoration frequency component D11 from the inverse quantization unit 260 and outputs the restoration difference image 17.

  In addition, the restored differential image 17 output from the inverse orthogonal transform unit 270 and the predicted image 15 output from the mode selection unit 230 are summed to form the reconstructed image 12.

  The memory controller 290 writes the reconstructed image 12 and the inter-screen prediction image 14 output from the inter-screen prediction unit 220 to the external memory 120 via the memory bus 130. Also, the stream 11 output from the variable length encoding unit 280 is output to the outside via the memory bus 130. In addition, a refresh command is issued to the external memory 120 during idle time when data transfer is not performed.

  Thereafter, the memory controller 290 receives the reference relationship setting information D12 from the encoding control unit 200. In accordance with the progress of the video encoding process, the reconstructed image 12 is displayed in the inter-screen prediction unit 220, the inter-screen prediction image 14 is displayed in the mode selection unit 230, and the stream 11 is output outside the video encoding unit 110. , Read from the external memory 120, respectively. The reference relationship setting information D12 will be described in detail later.

  The encoding control unit 200 includes encoding setting information S1 from the outside of the video encoding unit 110, an intra-screen prediction code amount D2 from the intra-screen prediction unit 210, and an inter-screen prediction code amount from the inter-screen prediction unit 220. The code amount information D6 is input from the variable length encoding unit 280 to D4. Also, the encoding control unit 200 outputs reference relationship setting information D12, a mode selection signal D7, and a quantization parameter D8. The configuration of the encoding control unit 200 will be described in detail later.

(Configuration of Encoding Control Unit 200)
Hereinafter, the encoding control unit 200 will be described in detail. FIG. 3 is a diagram illustrating a configuration example of the encoding control unit 200 according to the present embodiment.

  The encoding control unit 200 includes a band determination unit 310, a reference screen selection unit 320, a code amount control unit 330, a mode selection unit 340, and a setting register 350.

  The band determination unit 310 can refresh the external memory 120 between the external memory 120 and the encoding control unit 200 in a free time during which use data used for encoding in the video encoding unit 110 is not transferred. Determine whether it is possible.

  Specifically, for example, from the video information D21 output from the setting register 350, a band (that is, a data transfer rate) used between the memory controller 290 and the external memory 120 is used using a band determination algorithm described later. To calculate. Based on the calculated bandwidth, the bandwidth determination signal D22 is set to “1” when there is free time sufficient to refresh the external memory 120, and the bandwidth determination signal D22 is set to “1” when there is no free time sufficient to perform refresh. Output as 0 ".

  In addition, the band determination unit 310 compares the calculated value calculated based on the number of horizontal pixels and the number of vertical pixels of the original image 10 among the pixels constituting the original image 10 with a predetermined value to calculate When the value is less than the predetermined value, it is determined that the external memory 120 can be refreshed, and when the calculated value is greater than or equal to the predetermined value, it is determined that the external memory 120 cannot be refreshed. It may be like this.

  In addition, the band determination unit 310 compares the calculated value calculated based on the number of horizontal pixels and the number of vertical pixels of the original image 10 among the pixels constituting the original image 10 with a predetermined value, Whether or not the external memory 120 can be refreshed may be determined based on the pixel bit depth of the original image 10 or the luminance component and color difference component of the original image 10.

  The processing in the band determination unit 310 will be described in detail later.

  The reference screen selection unit 320 receives the band determination signal D22 from the band determination unit 310, the reference screen setting information D23 from the setting register 350, and outputs the reference relationship setting information D12.

  When the bandwidth determination unit 310 determines that the external memory 120 cannot be refreshed, the reference screen selection unit 320 uses the data used for encoding in the video encoding unit 110 as a refresh cycle of the external memory 120. The reference relationship in the intra prediction or inter prediction is set so that it is used in the encoding within a predetermined time.

  The processing in the reference screen selection unit 320 will be described in detail later.

  If the bandwidth determination unit 310 determines that the external memory 120 can be refreshed, the memory controller 290 refreshes the external memory 120 within a free time during which no data is transferred.

  The mode selection unit 340 receives the intra-screen prediction code amount D2 and the inter-screen prediction code amount D4 from the intra-screen prediction unit 210 and the inter-screen prediction unit 220, respectively, and the mode selection setting information D24 from the setting register 350. A mode selection signal D7 is output. The mode selection signal D7 is the same information as the mode selection information D87 in FIG. 9, and the processing in the mode selection unit 340 is the same as the processing in the conventional general video encoding device.

  The code amount control unit 330 receives the code amount information D6 from the variable length coding unit 280, the code amount control setting information D25 from the setting register 350, and the band determination signal D22 from the band determination unit 310, and sets the quantization parameter D8. Output. The quantization parameter D8 is the same information as the quantization coefficient D88 in FIG. 9, and the processing in the code amount control unit 330 is the same as the processing in the conventional general video encoding device.

  Specifically, the code amount control unit 330 is a quantization unit that is executed on the frequency component of the difference image 16 that is the difference between the prediction image 15 generated by intra prediction or inter prediction and the original image 10. Rate control regarding quantization processing at 250 is performed. Further, the code amount control unit 330 performs rate control based on the target bit rate and the refresh cycle of the external memory 120 when the band determination unit 310 determines that the external memory 120 cannot be refreshed. .

  The processing in the code amount control unit 330 will be described in detail later.

  The setting register 350 records encoding setting information S1 input from the outside of the video encoding unit 110, and outputs mode selection setting information D24, video information D21, reference screen setting information D23, and code amount control setting information D25. . Specifically, the setting register 350 classifies the coding setting information S1 into setting information for each type, and outputs mode selection setting information D24, video information D21, reference screen setting information D23, and code amount control setting information D25. To do.

(Bandwidth determination algorithm)
Here, the concept of the band determination algorithm used in the band determination unit 310 will be described with a specific example.

  In the present embodiment, the preconditions are as follows. That is, when the original image 10 is a progressive video having a screen width of 1280 pixels, a screen height of 720 pixels, and a frame rate of 60, the video encoding device 1 has 4000 per frame within the horizontal blanking period and the vertical blanking period. It is assumed that there is enough free time to issue one refresh command.

  On the other hand, it is assumed that the external memory 120 being used needs to issue refresh commands at least 8192 times during a refresh cycle of 64 ms (microseconds; the same applies hereinafter) in order to keep holding data. . Further, it is assumed that a refresh command can be issued a maximum of 8 times during a 1 MB (macroblock, the same applies hereinafter) processing cycle.

  A band determination method in the case where the video screen input to the video encoding device 1 is an interlaced video image having a horizontal size of 1920 pixels, a vertical length of 1088 pixels, and a frame rate of 30 under the above conditions will be described below.

  First, while maintaining the data in the external memory 120, a refresh allowable increase indicating how much the number of processing macroblocks can be increased from a progressive video with a screen width of 1280 pixels, a screen height of 720 pixels, and a frame rate of 60 Calculate the number of macroblocks.

The refresh allowable increase macroblock number is obtained by the following calculation formula.

720p free time MB processing count−required refresh count / refresh command issuable count per 1 MB (formula (1))

Here, “720p free time MB processing count” is the total number of macroblocks that can be processed during the horizontal blanking period and the vertical blanking period in which MB processing is not performed in one second in the progressive video of this example. . Accordingly, the refresh allowable increase macroblock number is 4000 × 60 / 8-8192 × 1000/64/8 = 30000-16000 = 14000 MB.

Next, the number of processing macro blocks per frame that can be allowed while holding the data in the external memory 120 (hereinafter referred to as the number of allowable processing MBs) is calculated. The allowable processing MB number is obtained by the following calculation formula. (Note that a macroblock is composed of 16 × 16 pixel values.)

720 pMB processing number + refresh allowable increase macroblock number (formula (2))

Here, “720 pMB processing count” is the total number of macroblocks processed per second in the progressive video of this example. Further, the refresh allowable increase macroblock number is 140000 MB as described above. Therefore, the upper limit value of the allowable processing MB number (hereinafter referred to as the upper limit value of the refresh allowable macro block number) is 1280/16 × 720/16 × 60 + 14000 = 2216 + 14000 = 230,000 MB (formula (3)).

  Next, in the case of interlaced video with a screen width of 1920 pixels, a screen height of 1088 pixels, and a frame rate of 30, the number of 720 iMB processes (the total number of macroblocks processed in one second in the interlaced video in this example) is 1920/16. * 1088/16 * 60/2 = 244800 is calculated.

  Here, as described above, the upper limit of the number of refresh allowable macroblocks is 230000 MB, and 244800 MB is the number of MBs exceeding this. Therefore, it is determined that the interlaced video having the horizontal 1920 pixels, the vertical 1088 pixels, and the picture rate 60 (frame rate 30) cannot be refreshed. Therefore, in this case, a value “0” is output from the band determination unit 310 as the band determination signal D22.

(Bandwidth determination unit 310)
Hereinafter, the configuration of the band determination unit 310 in which the above-described band determination algorithm is implemented will be described. FIG. 4 is a diagram illustrating a configuration example of the band determination unit 310 according to the present embodiment.

  The bandwidth determination unit 310 receives video information D21 including a screen horizontal MB number of 32, a screen vertical MB number of 33, and a field rate of 31. Then, determination according to the above-described band determination algorithm is performed, and a band determination signal D22 is output.

  Note that the screen length MB number 33 in the interlaced video is half the value of the screen length MB number 33 of the same screen size in the progressive video.

  The multiplier 3110 of FIG. 4 multiplies three values of the screen horizontal MB number 32, the screen vertical MB number 33, and the field rate 31 according to the above equation (2), and the result of multiplication is set as the allowable processing MB number D31. Output.

  The comparator 3120 compares the output allowable processing MB count D31 with the refresh allowable macroblock count upper limit 34, and if the allowable processing MB count D31 is smaller than the refresh allowable macroblock count upper limit 34, “1” is output. When the allowable processing MB number D31 is equal to or greater than the refresh allowable macroblock number upper limit 34, “0” is output to the outside of the band determination unit 310 as the band determination signal D22. The refresh allowable macroblock number upper limit 34 is assigned, for example, the constant value “230,000” obtained by the above-described equation (3).

(Reference screen selection unit 320)
The configuration of the reference screen selection unit 320 will be described below.

  The reference screen selection unit 320 receives the band determination signal D22 from the band determination unit 310, the reference screen setting information D23 from the setting register 350, and outputs the reference relationship setting information D12.

  Specifically, when the value of the band determination signal D22 is “1”, the reference relationship according to the reference screen setting information D23 input from the setting register 350 is set in the reference relationship setting information D12. When the value of the band determination signal D22 is “0”, a reference relationship that does not require refreshing is set in the reference relationship setting information D12.

FIG. 5A is a diagram illustrating an example of a reference relationship that requires refresh. In the reference relationship shown in FIG. 5A, the top field and the bottom field of the first frame of the GOP (Group Of Pictures) always refer to the top field of the first frame of the GOP, and the bottom field always refers to the bottom field of the first frame of the GOP. It becomes a relationship.
FIG. 5B is a diagram illustrating an example of a reference relationship that does not require refreshing. In the reference relationship shown in FIG. 5B, the top field always refers to the top field of the previous frame, and the bottom field always refers to the immediately preceding top field.

  When the field rate is “60”, the processing time for one field is 16.6 ms. For this reason, in the reference relationship shown in FIG. 5A, in the fields after the field number “4”, the reference destination does not exist within the refresh cycle of 64 ms, and refresh is required. On the other hand, in the reference relationship shown in FIG. 5B, the reference destination is a field that always exists within the refresh cycle of 64 ms, and refresh is not required.

  That is, by forcibly applying a reference relationship that does not require refresh as illustrated in FIG. 5-2, a sufficient external memory bandwidth is ensured for encoding processing, and encoding without data corruption is performed. Can do.

  Note that the reference relationship applied when the refresh is unnecessary is not limited to the reference relationship shown in FIG. If any reconstructed image is referenced or overwritten within the refresh cycle, or if it does not need to be refreshed because it is not referenced after the refresh cycle has passed, the appropriate reference can be made according to the application and purpose. Relationships can be set.

(Code amount control unit 330)
FIG. 6 is a diagram for explaining the concept of rate control executed in the code amount control unit 330. In the graph shown in FIG. 6, the vertical axis represents the amount of code amount control buffer (hereinafter referred to as “code amount control buffer”) used, and the horizontal axis represents time. That is, the solid line 620 in the graph of FIG. 6 is a plot of the usage amount of the code amount control buffer on the time axis. A broken line 610 is a line indicating the size of the code amount control buffer (hereinafter referred to as “code amount control size”).

  In the code amount control unit 330, as shown in the graph of FIG. 6, rate control is performed so that the buffer usage amount (solid line 620) that increases at a constant bit rate does not overflow and underflow the code amount control size (broken line 610). Is done.

  In addition, the code amount control unit 330 receives the input of the band determination signal D22 and switches the value of the code amount control size for rate control. Specifically, when the band determination signal D22 is “1”, refreshing is possible, so that the code amount control size is controlled according to the setting from the outside as usual.

On the other hand, when the band determination signal D22 is “0”, since refresh is impossible, the code amount control size is as follows so that the stream 11 does not stay in the external memory 120 in a period longer than the refresh cycle. Use the value obtained by the following formula. The “target bit rate” means a target code amount per time of a stream output from the video encoding device.

Code amount control size = target bit rate × refresh cycle (formula (4))

For example, when refresh is impossible, the target bit rate is 20 Mbps (megabits / second), and the refresh cycle is 64 ms, the code amount control size is set to 160 Kbytes (kilobytes).

  As a result, the stream 11 does not stay in the buffer beyond the refresh cycle by limiting the code amount control size even in a video having a larger screen size that cannot secure a free time for refreshing. Thus, it is possible to ensure a sufficient bandwidth of the external memory 120 for the encoding process and perform encoding without data corruption.

  Note that the rate control method applied to non-refreshable video is not limited to the rate control method described above. If the condition that the stream 11 generated by the video encoding process does not stay in the external memory 120 for a period longer than the refresh cycle is satisfied, an appropriate method can be adopted according to the application and purpose.

[Second Embodiment]
Hereinafter, as a second embodiment of the present invention, a case will be described in which the video encoding device 1 can select a pixel bit depth and a color difference format (Y, Cb, Cr).

  FIG. 7 shows that the memory within 1 MB processing time when the pixel bit depth and the color difference format are 4: 2: 2/10 bit (bit), 4: 2: 2/8 bit, and 4: 2: 0/10 bit, respectively. FIG. 5 is a diagram showing a simplified command sequence that can be issued by the controller 290 to the external memory 120.

  In this example, the following conditions were assumed. That is, the 1 MB processing time was 256 cycles (indicated as “cyc” in FIG. 7). Of the commands issued to the external memory 120 by the memory controller 290 (hereinafter referred to as “data transfer commands” as appropriate), “RefRD”, “PredRD”, and “StrRD” are data read commands related to video encoding, respectively. It is. “RefWT”, “PreWT”, and “StrWT” are data write commands related to video encoding.

  When the pixel bit depth and the color difference format are 4: 2: 2/10 bits, respectively, the consumption cycle of each data transfer command is assumed to end in 40 cycles for simplicity (FIG. 7A). In the following, “Refresh” is a refresh command, and 35 cycles are consumed.

  Here, when the pixel bit depth is 8 bits, or when the color difference format is 4: 2: 0, the data amount of the image data decreases, so the transfer amount of data related to video encoding also decreases. Also, when the pixel bit depth and color difference format are 4: 2: 2/8 bit, all data transfer commands are executed in 32 cycles, and when 4: 2: 0/10 bit, all data transfer commands are executed in 30 cycles. It is possible (FIGS. 7B and 7C).

  Therefore, as shown in FIG. 7, in the 1 MB processing time of 256 cycles, a free time during which data transfer is not performed occurs (time T1, T2). The memory controller 290 can issue a refresh command using 35 cycles by using this free time.

  Therefore, in the band determination unit 310 in the present embodiment, in addition to the screen horizontal MB number, the screen vertical MB number, and the field rate, the pixel bit depth and the color difference format are also included in the determination target.

(Bandwidth determination unit 310)
FIG. 8 is a diagram illustrating a configuration example of the band determination unit 310 according to the present embodiment.

  The band determination unit 310 receives the video information D21 including the screen horizontal MB number 32, the screen vertical MB number 33, the field rate 31, the pixel bit depth 41, and the color difference format 42. Then, determination according to the above-described band determination algorithm is performed, and a band determination signal D22 is output. Specifically, it is as follows.

  Note that the screen length MB number 33 in the interlaced video is half the value of the screen length MB number 33 of the same screen size in the progressive video.

  The multiplier 3110 multiplies the three values of the screen horizontal MB number 32, the screen vertical MB number 33, and the field rate 31 according to the above equation (2), and outputs the multiplication result as the allowable processing MB number D31.

  The comparator 3120 compares the output allowable processing MB count D31 with the refresh allowable macroblock count upper limit 34, and if the allowable processing MB count D31 is smaller than the refresh allowable macroblock count upper limit 34, “1” is output. , “0” is output as the processing MB number determination signal D41 if the refresh allowable macroblock number upper limit 34 or more.

  The comparator 3130 outputs “0” as the pixel bit depth determination signal D42 if the pixel bit depth 41 is equal to the maximum pixel bit depth (ie, 10 bits) 43, and “1” if it is smaller than this.

  The comparator 3140 outputs “0” as the color difference format determination signal D43 if the color difference format 42 matches the maximum color difference format (ie, 4: 2: 2) 44, and “1” otherwise.

  The logical adder 3150 outputs a logical sum of the processed MB number determination signal D41, the pixel bit depth determination signal D42, and the color difference format determination signal D43 as a band determination signal D22.

  As a result, a refresh command is issued when a refreshable free time occurs even when the screen size is the maximum size, such as when the pixel bit depth 41 is 8 bits or when the color difference format 42 is 4: 2: 0. You can switch the operation as you do.

(Appendix)
As is apparent from the detailed description of the embodiments according to the present invention, a part or all of the above-described embodiments can be described as the following supplementary notes. However, the following supplementary notes are merely examples of the present invention, and the present invention is not limited only to such cases.

(Appendix 1)
A video encoding unit that encodes an input image to be input by intra prediction or inter prediction;
A memory controller that records use data used in the encoding in the video encoding unit in an external memory that is a volatile memory that needs to be refreshed in order to retain the recorded content, and refreshes the external memory;
Have
The video encoding unit includes:
Bandwidth determination means for determining whether or not the external memory can be refreshed in an idle time during which the use data is not transferred to and from the external memory;
When it is determined by the bandwidth determination means that the external memory cannot be refreshed, the in-screen prediction or the screen is used so that the use data is used in the encoding within the refresh cycle of the external memory. Reference screen selection means for setting a reference relationship in inter prediction, and
The memory controller refreshes the external memory within the idle time when the bandwidth determination means determines that the external memory can be refreshed. apparatus.

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

  In addition, in the case of a video having a screen size that can secure a sufficient free time for refreshing in the horizontal blanking period or the vertical blanking period, the encoding can be performed by refreshing the external memory.

(Appendix 2)
The band determining unit compares a calculated value calculated based on the number of horizontal pixels and the number of vertical pixels of the input image among pixels constituting the input image with a predetermined value, and calculates the calculated value. Is less than the predetermined value, it is determined that the external memory can be refreshed. If the calculated value is equal to or greater than the predetermined value, the external memory cannot be refreshed. The video encoding device according to appendix 1, wherein the video encoding device is determined.

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

(Appendix 3)
The encoding control means includes
There is further provided a code amount control means for performing rate control related to a quantization process performed on a frequency component of a difference image that is a difference between a predicted image generated by the intra prediction or the inter prediction and the input image. And
The code amount control unit performs the rate control based on a target bit rate and a refresh cycle of the external memory when the band determination unit determines that the external memory cannot be refreshed. The video encoding apparatus according to appendix 2, characterized by:

  According to this configuration, refresh is not required by performing rate control relating to quantization processing. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

(Appendix 4)
The band determination unit refreshes the external memory based on the pixel bit depth of the input image or the luminance component and color difference component of the input image in addition to the comparison between the calculated value and the predetermined value. The video encoding apparatus according to appendix 2, wherein it is determined whether or not it can be performed.

  Even if the screen size is the maximum size, there may be a refreshable free time depending on the pixel bit depth and the color difference format. Therefore, when the pixel bit and the color difference format are also subject to the band determination, the operation can be switched to issue a refresh command when a refreshable free time occurs.

(Appendix 5)
A video encoding unit that encodes an input image to be input by intra-screen prediction or inter-screen prediction, and usage data used in the encoding in the video encoding unit needs to be refreshed to maintain the recorded contents A video encoding method executed in a video encoding device having a memory controller that records in an external memory that is a volatile memory and refreshes the external memory,
A bandwidth determination step for determining whether or not the external memory can be refreshed in a free time during which the use data is not transferred between the external memory and the video encoding unit;
If it is determined in the bandwidth determination step that the external memory cannot be refreshed, the in-screen prediction or the screen is used so that the use data is used in the encoding within the refresh cycle of the external memory. A reference screen selection step for setting a reference relationship in inter prediction,
If it is determined in the bandwidth determination step that the external memory can be refreshed, a memory refresh step for refreshing the external memory within the free time;
A video encoding method comprising:

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

  In addition, in the case of a video having a screen size that can secure a sufficient free time for refreshing in the horizontal blanking period or the vertical blanking period, the encoding can be performed by refreshing the external memory.

(Appendix 6)
The band determination step compares a calculated value calculated based on the number of horizontal pixels and the number of vertical pixels of the input image among the pixels constituting the input image with a predetermined value, and calculates the calculated value. Is less than the predetermined value, it is determined that the external memory can be refreshed. If the calculated value is equal to or greater than the predetermined value, the external memory cannot be refreshed. The video encoding method according to appendix 5, wherein the video encoding method is determined.

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

(Appendix 7)
There is further provided a code amount control step for performing rate control relating to a quantization process performed on a frequency component of a difference image that is a difference between a predicted image generated by the intra prediction or the inter prediction and the input image. And
The code amount control step performs the rate control based on a target bit rate and a refresh cycle of the external memory when it is determined in the band determination step that the external memory cannot be refreshed. The video encoding method according to appendix 6, characterized by:

  According to this configuration, refresh is not required by performing rate control relating to quantization processing. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

(Appendix 8)
The band determination step refreshes the external memory based on the pixel bit depth of the input image or the luminance component and color difference component of the input image in addition to the comparison between the calculated value and the predetermined value. The video encoding method according to appendix 6, wherein it is determined whether or not it can be performed.

  Even if the screen size is the maximum size, there may be a refreshable free time depending on the pixel bit depth and the color difference format. Therefore, when the pixel bit and the color difference format are also subject to the band determination, the operation can be switched to issue a refresh command when a refreshable free time occurs.

(Appendix 9)
A video encoding unit that encodes an input image to be input by intra-screen prediction or inter-screen prediction, and usage data used in the encoding in the video encoding unit needs to be refreshed to maintain the recorded contents In a computer having a memory controller that records in an external memory that is a volatile memory and refreshes the external memory,
A bandwidth determination step for determining whether or not the external memory can be refreshed in a free time during which the use data is not transferred between the external memory and the video encoding unit;
If it is determined in the bandwidth determination step that the external memory cannot be refreshed, the in-screen prediction or the screen is used so that the use data is used in the encoding within the refresh cycle of the external memory. A reference screen selection step for setting a reference relationship in inter prediction,
If it is determined in the bandwidth determination step that the external memory can be refreshed, a memory refresh step for refreshing the external memory within the free time;
A video encoding program for executing

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

  In addition, in the case of a video having a screen size that can secure a sufficient free time for refreshing in the horizontal blanking period or the vertical blanking period, the encoding can be performed by refreshing the external memory.

(Appendix 10)
The band determination step compares a calculated value calculated based on the number of horizontal pixels and the number of vertical pixels of the input image among the pixels constituting the input image with a predetermined value, and calculates the calculated value. Is less than the predetermined value, it is determined that the external memory can be refreshed. If the calculated value is equal to or greater than the predetermined value, the external memory cannot be refreshed. The video encoding program according to appendix 9, wherein the video encoding program is determined.

  According to this configuration, the refresh is unnecessary by limiting the reference relationship of intra prediction or inter prediction. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

(Appendix 11)
There is further provided a code amount control step for performing rate control relating to a quantization process performed on a frequency component of a difference image that is a difference between a predicted image generated by the intra prediction or the inter prediction and the input image. And
The code amount control step performs the rate control based on a target bit rate and a refresh cycle of the external memory when it is determined in the band determination step that the external memory cannot be refreshed. The video encoding program according to appendix 10, characterized by:

  According to this configuration, refresh is not required by performing rate control relating to quantization processing. As a result, even in the case of a video with a large screen size in which sufficient free time cannot be secured for refreshing in the horizontal blanking period or vertical blanking period, encoding can be performed while suppressing data corruption. it can.

(Appendix 12)
The band determination step refreshes the external memory based on the pixel bit depth of the input image or the luminance component and color difference component of the input image in addition to the comparison between the calculated value and the predetermined value. The video encoding program according to appendix 10, wherein it is determined whether or not it can be performed.

  Even if the screen size is the maximum size, there may be a refreshable free time depending on the pixel bit depth and the color difference format. Therefore, when the pixel bit and the color difference format are also subject to the band determination, the operation can be switched to issue a refresh command when a refreshable free time occurs.

DESCRIPTION OF SYMBOLS 1 Video coding apparatus 10 Original image 11 Stream 12 Reconstructed image 13 In-screen prediction image 14 Inter-screen prediction image 15 Prediction image 16 Difference image 17 Restoration difference image 31 Field rate 32 Screen width MB number 33 Screen length MB number 34 Refresh tolerance Maximum number of macroblocks 41 Pixel bit depth 42 Color difference format 43 Maximum pixel bit depth 44 Maximum color difference format 110 Video encoding unit 120 External memory 130 Memory bus 200 Coding control unit 210 In-screen prediction unit 220 Inter-screen prediction unit 230 Mode selection Unit 240 orthogonal transform unit 250 quantization unit 260 inverse quantization unit 270 inverse orthogonal transform unit 280 variable length coding unit 290 memory controller 310 bandwidth determination unit 320 reference screen selection unit 330 code amount control unit 340 mode selection unit 350 setting register 3110 Multiplier 3120 Comparator 3130 Comparator 3140 Comparator 3150 Logical sum S1 Encoding setting information D1 Intra-screen prediction mode D2 Intra-screen prediction code amount D3 Inter-screen prediction mode D4 Inter-screen prediction code amount D6 Code amount information D7 Mode selection signal D8 Quantization parameter D9 Frequency component D10 Quantized value D11 Restored frequency component D12 Reference relationship setting information D21 Video information D22 Band decision signal D23 Reference screen setting information D24 Mode selection setting information D25 Code amount control setting information D31 Allowable processing MB number D41 processing MB number determination signal D42 Pixel bit depth determination signal D43 Color difference format determination signal

Claims (2)

A video encoding unit that encodes an input image to be input by intra prediction or inter prediction;
A memory controller that records use data used in the encoding in the video encoding unit in an external memory that is a volatile memory that needs to be refreshed in order to retain the recorded content, and refreshes the external memory;
Have
The video encoding unit includes:
Bandwidth determination means for determining whether or not the external memory can be refreshed in an idle time during which the use data is not transferred to and from the external memory;
When it is determined by the bandwidth determination means that the external memory cannot be refreshed, the in-screen prediction or the screen is used so that the use data is used in the encoding within the refresh cycle of the external memory. Reference screen selection means for setting a reference relationship in inter prediction, and
The memory controller refreshes the external memory within the idle time when the bandwidth determination means determines that the external memory can be refreshed. apparatus.   The band determining unit compares a calculated value calculated based on the number of horizontal pixels and the number of vertical pixels of the input image among pixels constituting the input image with a predetermined value, and calculates the calculated value. Is less than the predetermined value, it is determined that the external memory can be refreshed. If the calculated value is equal to or greater than the predetermined value, the external memory cannot be refreshed. The video encoding apparatus according to claim 1, wherein the determination is made.

Images