JP4393341B2 - Moving picture coding apparatus and moving picture coding method - Google Patents

Description

  The present invention relates to a moving image encoding apparatus and a moving image encoding method, and in particular, refers to a plurality of frame data in a moving image, performs motion compensation on the frame data to be encoded, and encodes the moving image. The present invention relates to a technique suitable for use.

  Conventionally, the H.264 encoding method has attracted attention as a new moving image encoding method. This encoding method is an encoding method jointly developed by ITU-T and ISO.

  Unlike the conventional MPEG-1, 2, and 4 encoding schemes, this new encoding scheme uses 4 × 4 integer conversion and a plurality of intra predictions are prepared. In addition, an in-loop filter is used, and motion compensation is performed in seven types of sub-blocks. Also, the motion compensation pixel accuracy can be compensated for the ¼ pixel accuracy similarly to the MPEG-4 encoding method. Furthermore, universal variable length coding and context adaptive variable length coding are used as entropy coding.

  Furthermore, in the MPEG-1, 2, 4 encoding method, motion compensation is performed using two reference images (frames) before and after the frame to be encoded, but more reference images are used. It is possible. The num_ref_frames code included in the head header of the bitstream can take a maximum of 16 values.

  That is, in motion compensation, 16 frames before and after the encoding target frame can be referred to as a reference image. As described above, for the macroblock to be encoded, the prediction error is calculated with 1/4 pixel accuracy for seven types of sub-blocks for an image of up to 16 frames, and the macroblock that minimizes the prediction error is selected. By doing so, it becomes possible to greatly improve the coding efficiency.

Here, H. The configuration of a conventional video encoding apparatus using the H.264 encoding method will be described with reference to FIG.
FIG. 11 is a block diagram illustrating a configuration example of a conventional moving image encoding apparatus.
Image data is input in units of macroblocks to the moving image encoding apparatus. The selector 1000 switches whether to perform intra coding. In the case of intra coding, image data is input to the intra predictor 1001. The intra predictor 1001 performs prediction in nine modes and calculates a prediction error.

  On the other hand, in cases other than intra coding, the difference is input to the differentiator 1002, and the difference from the predicted image is calculated as a prediction error.

  The transformer / quantizer 1003 performs integer transformation of the 4 × 4 pixel block on the calculated prediction error, and quantizes the obtained coefficient. The quantization coefficient that is the quantization result is variable-length encoded by the entropy encoder 1004 and output to the output unit 1014. At the same time, the quantization result is input to the inverse quantization / inverse transformer 1005 to restore the prediction error, and the adder 1006 adds it to the predicted image. The result is appropriately stored in the frame memories 1007 to 1010 as a decoded image.

  The motion estimator 1011 compares the decoded image stored in the frame memories 1007 to 1010 with the input image, and calculates a motion vector with a 1/4 pixel accuracy for each subblock. The motion vector and the selected frame number are input to the motion compensator 1012, read the reference image from the corresponding frame memory, select the reference image with the smallest prediction error, and output it to the differencer 1002 as the predicted image.

  The motion vector and the selected frame number are input to the motion encoder 1013, encoded, and output to the output unit 1014. The output device 1014 shapes and outputs the encoded data according to the format (see, for example, Non-Patent Document 1 and Non-Patent Document 2).

  On the other hand, camcorders used for shooting moving images are equipped with an autofocus function for automatically focusing. In the “TV lens focus operating device” described in Patent Document 1, the focus position at which the evaluation value calculated by the evaluation value calculating means for determining the focus evaluation value indicating the sharpness of the subject is maximized is obtained. A television lens that controls the motor to operate the focus is disclosed.

"Overview of the H.264 / AVC video Coding Standard" (IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, JULY 2003) "H.264 is in the midst of stiff MPEG-4" (Nikkei Electronics 2003.7.7 pp. 65-74) JP-A-11-103409

  However, the above-mentioned H.P. In an encoding method that refers to a plurality of frames, such as the H.264 encoding method, a motion vector search is performed in order to perform motion compensation. The amount of calculation increases as the number of reference images increases. There was a problem that it was necessary.

  The present invention has been made in view of the above-described problems, and enables a motion vector search to be efficiently performed by suitably selecting a reference frame to be used for video encoding. With the goal.

The moving image encoding apparatus of the present invention is a moving image encoding apparatus that performs motion compensation on frame data to be encoded with reference to a plurality of frame data in a moving image, and encodes the moving image. Te, input means for inputting frame data, and the sharpness detecting means for detecting the sharpness from the frame data inputted by said input means, has been sharpness is higher the higher predetermined number of frames detected by said sharpness detecting means Referring to the frame data, as reference frame data to be referred to when coding the frame data of the encoding target, and selecting means for selecting from among the frame data inputted by said input means, which is selected by said selection means Motion compensation means for performing motion compensation based on frame data and the frame data to be encoded;
Using the result of motion compensation performed by the motion compensation means, encoding means for encoding the frame data to be encoded, and output means for outputting the frame data encoded by the encoding means It is characterized by having.

The moving image encoding method of the present invention is a moving image encoding method that performs motion compensation on frame data to be encoded with reference to a plurality of frame data in a moving image, and encodes the moving image. An input step for inputting frame data, a sharpness detection step for detecting a sharpness from the frame data input by the input step, and a predetermined number of frames having a high sharpness detected by the sharpness detection step. data, a selection step of selecting the frame data of the encoding target from among the frame data inputted by said input step as reference frame data to be referred to when coding, the reference frame data selected by said selecting step And a motion compensation step for performing motion compensation based on the frame data to be encoded, and the motion compensation step. A coding step for coding the frame data to be coded using the motion compensation result, and an output step for outputting the frame data coded in the coding step. .

The program of the present invention refers to a plurality of frame data in a moving image, executes motion compensation for the frame data to be encoded, and causes a computer to execute a moving image encoding method for encoding the moving image. An input process for inputting frame data;
A sharpness detection step of detecting a sharpness from the frame data inputted by the input step, the frame data of the sharpness detecting step number predetermined upper frame is higher sharpness detected by the encoding target frame data As a reference frame data to be referred to when encoding, a selection step for selecting from among the frame data input by the input means, reference frame data stored in the storage medium selected by the selection step, A motion compensation step for performing motion compensation based on the frame data to be encoded, and an encoding step for encoding the frame data to be encoded using a result of motion compensation performed by the motion compensation step. And causing the computer to execute an output step of outputting the frame data encoded in the encoding step It is characterized in.

The recording medium of the present invention is characterized by recording the program described above.

  According to the present invention, since the reference frame is selected using the sharpness as an index, it is possible to perform high-quality encoding only by referring to a small number of frames.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a configuration example of a moving picture coding apparatus according to the first embodiment of the present invention. In the first embodiment, H.264 is used as the encoding method used by the video encoding apparatus. The H.264 encoding method will be described as an example, but the present invention is not limited to this. Further, for ease of explanation, forward prediction with reference to a past frame is taken as an example. For ease of explanation, the maximum number of reference frames is 16, but the present invention is not limited to this.

  In FIG. 1, reference numeral 1 denotes a selector, which selects an output according to an encoding mode of intraframe encoding / interframe encoding. 2 is H. An intra predictor that performs intra prediction using the H.264 encoding method, and 3 is a subtractor that calculates a motion prediction error. Reference numeral 4 denotes a transform / quantizer that performs integer orthogonal transform and quantizes the obtained coefficient. Reference numeral 6 denotes an inverse quantization / inverse transformer that inversely quantizes the quantized coefficient and executes integer-type inverse orthogonal transform.

  Reference numeral 5 denotes an entropy encoder that encodes the quantization result of the transformer / quantizer 4. 16 is an output device, and the generated code is converted to H.264. The data is formatted according to the H.264 encoding format and output. Reference numeral 7 denotes an adder that adds the prediction error and the prediction value (prediction image) obtained from the inverse quantization / inverse converter 6.

Reference numerals 8_1 to 8_16 denote frame memories, which store locally decoded image data in units of frames. A selector 9 controls input / output.
Reference numeral 10 denotes a motion estimator, which extracts an optimal motion vector from a corresponding frame based on an input image and a decoded image. A motion compensator 13 generates a predicted image from the motion vector calculated by the motion estimator 10 and corresponding frame information. A motion encoder 14 encodes motion information based on the calculated motion vector and the corresponding frame information.

  Reference numeral 11 denotes a sharpness detector that detects the sharpness of the input image data, and reference numeral 12 denotes a reference frame determiner that determines a reference frame and outputs it as reference information. Reference numeral 15 denotes a reference information encoder that encodes reference information.

  1 further includes a CPU that controls the entire apparatus, a ROM that stores various control programs for controlling the apparatus, a work area for various data for executing various controls, and a temporary storage area. The RAM functions as a save area.

Next, the moving picture coding operation of the moving picture coding apparatus in FIG. 1 will be described below.
Prior to encoding, the entropy encoder 5 generates header information including the number of frames that can be used for reference, and outputs the header information from the output unit 16.

  The image data is input in units of frames and input to the selector 1. The selector 1 controls the selection operation so that intra-frame encoding is performed at regular intervals and inter-frame encoding is performed at other times.

First, a case where intra-frame coding is performed on the first first frame will be described.
The input frame data is input to the intra predictor 2 in units of macroblocks, and the intra predictor 2 executes intra prediction for each block. The prediction result is input to the transformer / quantizer 4, the integer / orthogonal transformation is executed in the transformer / quantizer 4, and the obtained coefficient is quantized. The quantization coefficient which is the quantization result is input to the entropy encoder 5 and the inverse quantization / inverse transformer 6.

  The entropy encoder 5 performs entropy encoding on the input quantization result, and the encoded data is input to the output unit 16. The inverse quantization / inverse transformer 6 obtains a decoded image from the inputted quantization result and inputs it to the adder 7. In intra-frame coding, since there is no predicted image, a predicted value “0” is added.

On the other hand, the sharpness detector 11 detects the sharpness from the image data. The sharpness may be the first derivative of the image with respect to the edge strength represented by the edge strength of the image. For the sake of explanation, the primary differential value Δx (i, j) for the pixel x (i, j) is
Δx (i, j) = 4x (i, j) −x (i−1, j) −x (i, j−1) −x (i + 1, j) −x (i, j + 1) (1)
The sum of the data whose absolute value of Δx (i, j) is equal to or greater than the threshold value is equal to or greater than the threshold value but is divided by the number is defined as the sharpness Ek (k is the frame number).

The calculated sharpness Ek is input to the reference frame determiner 12.
FIG. 2 shows a detailed block diagram of the reference frame determiner 12. In FIG. 2, 100 is a sharpness buffer for temporarily storing the input sharpness Ek, 101_1 to 101_16 are sharpness memories for storing the sharpness Ek, and each of them is a frame memory 8_1 in FIG. Corresponds to ~ 8_16.

  Reference numeral 102 denotes a sharpness comparator, which compares the sharpness Ek stored in the sharpness memories 101_1 to 101_16, selects the top N, determines the corresponding frame number, and has the smallest sharpness Ek. Or select. The determined frame number is stored in the sharpness selection frame information memory 105. Reference numeral 103 denotes a frame counter, which increases the content by one every time one frame is processed.

  Reference numeral 104 denotes a time selection frame information memory that stores the numbers of neighboring frames in terms of time. Reference numeral 105 denotes a sharpness selection frame information memory for storing the number of the frame selected based on the sharpness Ek. A sharpness memory controller 106 controls the sharpness memories 101_1 to 101_16.

The input sharpness Ek is stored in the sharpness buffer 100.
The sharpness memory controller 106 stores the content of the sharpness buffer 100 in the corresponding sharpness memory if there is no data stored in the sharpness memories 101_1 to 101_16. When data is stored in all of the sharpness memories 101_1 to 101_16, the sharpness memory to be stored is selected. At this time, the frame counter 103 outputs the value of k.

  The time selection frame information memory 104 selects (k−1) number frame, (k−2) number frame, and (k−3) number frame from the value k. (K-2) and (k-3) are stored. If there is no corresponding frame, only those that exist are stored.

  After that, the sharpness comparator 102 stores the kth frame and the (k-1) th frame stored in the time selection frame information memory 104 in the sharpness memories 101_1 to 101_16. The sharpness Ek of the frame, excluding the (k-3) th frame, is read from the sharpness memories 101_1 to 101_16, the smallest one is selected, and the number is input to the sharpness memory controller 106.

  The sharpness memory controller 106 determines the frame number corresponding to the selected sharpness Ek, and stores the contents of the sharpness buffer 100 in the sharpness memory of the corresponding frame among the sharpness memories 101_1 to 101_16. The output of the sharpness memory controller 106 is also input to the frame memories 8_1 to 8_16.

  Returning to FIG. 1, since the reference frame that is long in time is updated, the reference information encoder 15 converts the reference information into the H.264 format. The reference frame update information is encoded according to the H.264 encoding format. The decoded image data output from the adder 7 is stored in one of the frame memories 8_1 to 8_16 based on the output from the sharpness memory controller 106 in FIG.

Subsequently, a case where interframe coding is performed on the subsequent frames after the second frame will be described.
The sharpness detector 11 detects the sharpness Ek from the image data. The calculated sharpness Ek is input to the reference frame determiner 12. The reference frame determiner 12 takes as an example a case where five frames to be referenced are selected, but the number of frames to be selected is not limited to this.

Turning to FIG. 2, the input sharpness Ek is stored in the sharpness buffer 100.
At this time, the frame counter 103 outputs a count value of k. The time selection frame information memory 104 selects (k−1) th frame, (k−1) th frame, (k−2) th frame, and (k−3) th frame from the count value k. , (K-2), (k-3) are stored. If there is no corresponding frame, only those that exist are stored.

  After that, the sharpness comparator 102 includes the kth frame, the (k-1) th frame, and the (k-2) th frame stored in the time selection frame information memory 104 in the sharpness memories 101_1 to 101_16. , The sharpness Ek of the frames excluding the (k-3) th frame is read from the sharpness memories 101_1 to 101_16, and the smallest one with the highest sharpness Ek is selected. For those with high sharpness Ek, the corresponding number is stored in the sharpness selection frame information memory 105.

  The smallest number is input to the sharpness memory controller 106. The sharpness memory controller 106 determines a frame number corresponding to the selected sharpness Ek, and stores the contents of the sharpness buffer 100 in the sharpness memory of the corresponding frame among the sharpness memories 101_1 to 101_16. The output of the sharpness memory controller 106 is also input to the frame memories 8_1 to 8_16.

  Returning to FIG. 1, the selector 9 reads the contents of the selected frame memory corresponding to the numbers stored in the time selection frame information memory 104 and the sharpness selection frame information memory 105 of FIG. 2 and inputs them to the motion estimator 10. To do. The motion estimator 10 calculates a motion vector from the contents of these frame memories and the input image data, generates a predicted image by the motion compensator 13, and inputs it to the differencer 3. The selector 1 selects the differentiator 3 as an output.

  The subtractor 3 calculates a prediction error, and the calculation result is input to the transformer / quantizer 4 to perform integer orthogonal transformation and quantize the obtained coefficient. The quantization coefficient which is the quantization result is input to the entropy encoder 5 and the inverse quantization / inverse transformer 6. The motion vector calculated by the motion estimator 10 is encoded by the motion encoder 14, and the encoded data is input to the output unit 16.

In the entropy encoder 5, the input quantization result is entropy encoded, and the encoded data is input to the output unit 16.
The inverse quantization / inverse transformer 6 obtains a prediction error from the input quantization result and inputs it to the adder 7. The prediction image (prediction value) from the motion compensator 13 is added to the prediction error. The decoded image data output from the adder 7 is stored in the frame memory based on the output from the sharpness memory controller 106 in FIG.

Next, the processing procedure of the moving image encoding process in the moving image encoding device of the first embodiment will be described with reference to the flowchart of FIG.
First, in step S1, header information is generated and output. Next, in step S2, it is determined whether or not all frames have been encoded. If the result of this determination is that encoding is complete (YES in step S2), the processing is terminated. On the other hand, if the encoding has not ended (NO in step S2), the process proceeds to step S3.

  In step S3, frame data to be encoded is input. Next, in step S4, the sharpness Ek is calculated from the frame data. Thereafter, the process proceeds to step S5, and a temporally neighboring M (in this case, “3”) frame is selected as a reference frame.

Next, the process proceeds to step S6, and N frames with high sharpness Ek are selected as reference frames (in this case, “2”) from the frames excluding the selected M frames.
Next, in step S7, in the case of intra-frame encoding, encoding is performed using data in the frame. In the case of inter-frame encoding, frame data is encoded with reference to the reference frame.

Next, in step S8, the frame with the lowest sharpness Ek is selected.
Next, in step S9, a decoded image is reproduced from the encoded data, and the frame selected in step S8 is deleted and stored. The sharpness Ek is also stored in the same manner. Thereafter, the processing returns to step S2 in order to process the next frame.

  As described above, according to the first embodiment, since the sharpness Ek is high, a frame that seems to contain a lot of important information is referred to, so that motion compensation is efficiently performed. It can be carried out. Thereby, it is possible to improve the encoding efficiency and the image quality.

  In the first embodiment, the maximum number of reference frames is five. However, the number of reference frames is not limited to this, and it is apparent that other frames can be referred to by arranging the necessary number of frame memories. . Further, although the maximum number of frames to be stored is 16, it is not limited to this.

  Further, in the first embodiment, the number of frames that are temporally adjacent and the number of frames that are selected based on the sharpness Ek are not limited to this. Furthermore, the number of frames may be variable.

  In the first embodiment, only forward prediction has been described as an example. However, even when backward prediction or bi-directional prediction is used, it is obvious that it can be handled by arranging the necessary number of frame memories. Of course, a part or all of the functions of the various constituent elements of the moving picture encoding apparatus according to the first embodiment may be described in software, and the processing may be executed by an arithmetic unit such as a CPU.

(Second Embodiment)
FIG. 4 is a block diagram showing a configuration of a moving picture coding apparatus according to the second embodiment of the present invention. In the second embodiment, a camcorder will be described as an example of a video encoding device.

  In FIG. 4, reference numeral 401 denotes an image pickup unit, which includes an optical unit composed of a lens and the like that can be adjusted in focus, and an electronic circuit unit that generates an analog electrical signal of an image by photoelectric conversion. A sharpness detector 402 detects the sharpness Ek of the image data. Reference numeral 403 denotes an in-focus level determination unit, which determines the degree of focus from the sharpness Ek and generates a signal for controlling the focus. Reference numeral 404 denotes a lens focus adjuster for adjusting the focus based on the output of the in-focus level determination unit 403. The focus adjuster 404 includes a motor that adjusts the position of the lens.

  Reference numeral 405 denotes an A / D converter that converts an analog electric signal into a digital signal, and reference numeral 406 denotes a frame memory that stores the digital signal. Reference numeral 407 denotes a reference frame determiner that selects a frame to be referenced based on the output of the sharpness detector 402. In the present embodiment, for ease of explanation, the number of reference frames is described as a maximum of 5 frames, but the present invention is not limited to this.

  Reference numeral 409 denotes a frame memory group for storing the decoded image, and reference numeral 408 denotes a frame memory controller for controlling the frame memory group. Reference numeral 410 denotes a motion compensator that performs motion compensation by referring to the decoded image of the frame memory group 409 according to the output of the reference frame determiner 407, and reference numeral 411 denotes an encoder that performs encoding. The encoding method is H.264. The H.264 encoding method will be described as an example, but the present invention is not limited to this.

  Reference numeral 412 denotes a recorder for recording the encoded data on the recording medium 413. Reference numeral 413 denotes a recording medium for recording encoded data. Reference numeral 414 denotes a decoder that generates a decoded image for reference.

  An analog electrical signal of a shooting frame from the imaging unit 401 is input to an A / D converter 405 and a sharpness detector 402. The sharpness detector 402 applies a high-pass filter (HPF) to the analog signal, digitizes the value, and adds one frame.

  The added value is input to the focus degree determination unit 403. The focus degree determination unit 403 outputs a signal to the focus adjustment unit 404 so that the calculated sharpness Ek becomes the maximum value. Conventional techniques may be used for determining the in-focus level. For example, the determination described in Patent Document 1 may be used. The focus adjuster 404 adjusts the position of the lens by driving a motor in accordance with the input signal.

  The sharpness Ek is also input to the reference frame determiner 407. The reference frame determiner 407 determines a frame to be referred to from the sharpness Ek, similarly to the reference frame determiner 12 of FIG. The frame memory controller 408 enables the corresponding frame data to be read from the frame memory group 409.

  On the other hand, the analog electrical signal input to the A / D converter 405 is digitized and stored in the frame memory 406. The motion compensator 410 receives the frame determined by the reference frame determiner 407 from the frame memory group 409, compares the contents of the frame memory 406 with the reference frame, and performs motion compensation as in the first embodiment. , Output prediction error and input image data. The encoder 411 performs intra-frame encoding and inter-frame encoding on these data to generate encoded data. The encoded data is recorded on the recording medium 413 via the recorder 412.

  The decoder 414 decodes the generated encoded data to generate a decoded image, and the memory of the frame memory group 409 containing the frame data indicated by the reference frame determiner 407 stores the frame memory controller 408. Store according to instructions.

Next, the processing flow of the moving image encoding process of the moving image encoding device according to the second embodiment will be described with reference to FIG.
FIG. 5 is a flowchart showing a moving picture coding process in the moving picture coding apparatus according to the second embodiment of the present invention. In FIG. 5, the same step numbers are assigned to portions that perform the same functions as those in FIG. 3 of the first embodiment, and the description thereof is omitted.

  Similar to the first embodiment, after the processing of step S1 to step S2, frame data is imaged in step S500. Next, after the process of step S4, in step S501, the focus is adjusted based on the sharpness Ek.

  Thereafter, in step S7, in the case of intra-frame coding, data in the frame is used, and in the case of inter-frame coding, N frames selected based on the temporally neighboring M frames and the sharpness Ek. The frame data is encoded with reference to FIG.

In step S502, the encoded data is recorded on the recording medium 413.
Next, in step S503, if the sharpness Ek is improved as compared with the sharpness Ek of the previous frame, the process proceeds to step S506, and if not, the process proceeds to step S504. In step S504, the frame with the most elapsed time is selected from the M frames selected in terms of time. In step S505, the encoded data is decoded, and the generated frame data is stored in the frame selected in step S504.

  On the other hand, as a result of the determination in step S503, if the process proceeds to step S506, it is determined whether or not the sharpness Ek is monotonously increased by going back to “M frame”. If the result of this determination is a monotonous increase, the process proceeds to step S507; otherwise, the process proceeds to step S508.

  In step S507, the frame before “(M + 1) frames” is removed from the reference frames of N frames determined based on the sharpness Ek. This is because the sharpness Ek monotonously increases, and the frame before “(M + 1) frames” is included in the N frame, and is therefore targeted for removal. If the process proceeds to step S508, the frame with the lowest sharpness Ek is removed from the N frames of reference frames determined based on the sharpness Ek.

  Next, in step S509, it is selected to use the frame data of the decoded image of the input frame as a substitute for the frame excluded in step S507 or step S508.

  Next, in step S510, the encoded data is decoded, and the generated frame data is stored in the frame determined in step S509. Thereafter, the process returns to step S2 to process the next frame.

  As described above, according to the second embodiment, by using together with the focus adjustment function, by referring to the frame in focus, a lot of important information with high sharpness Ek is included. Therefore, motion compensation can be performed using a frame that seems to be present, and motion compensation can be performed efficiently. In particular, in the present embodiment, an image with a high sharpness Ek is used as a reference image during focus adjustment for an image that is difficult to focus during the focus adjustment. Since it is possible to refer to an image with a high sharpness Ek instead of referencing from the out-of-focus image sometimes, motion compensation can be performed from the top of the focused image.

In the second embodiment, the sharpness Ek is detected as the sum of high-frequency components obtained by HPF for an analog electric signal. However, the present invention is not limited to this, and HPF calculation or orthogonal transformation is performed for a digital signal. It is clear that high frequency components can be dealt with.
Of course, a part or all of the functions of the various components of the moving picture coding apparatus according to the second embodiment may be described in software, and the process may be executed by an arithmetic device such as a CPU.

  In the second embodiment, the reference frame is determined based on the sharpness Ek. However, as shown in FIG. 6, the degree of focus is measured by another method, and this is sharpened by the reference frame determiner. You may enter it as a degree.

  In FIG. 6, reference numeral 420 denotes a distance measuring device that measures the distance to the subject, and measures the distance using ultrasonic waves or the like. Reference numeral 421 denotes a focus adjuster that outputs a focal length. Reference numeral 422 denotes an in-focus degree determiner, which compares the distance of the distance measuring device 420 with the focal length of the focus adjuster 421 and sets 1 / D as the in-focus degree F from the difference D. When this value D is sufficiently small, it is determined that the degree of focus is large. Reference numeral 423 denotes a reference frame determiner, which determines a reference frame with the focus degree F as a change of the sharpness degree Ek.

  This makes it possible to perform motion compensation from the beginning of the in-focus image instead of referring to the out-of-focus image when the image is in focus when adjusting the focus. it can.

  Further, although the camcorder is taken as an example as the second embodiment, the present invention is not limited to this. By using a communication interface instead of the recording device 412 and a communication line instead of the recording medium 413, a moving image transmission apparatus is provided. Of course.

(Third embodiment)
FIG. 7 is a block diagram showing a configuration of a moving picture encoding apparatus according to the third embodiment of the present invention. In FIG. 7, 300 is a central processing unit (CPU) that controls the entire apparatus and performs various processes, and 301 provides an operating system (OS), software, and a storage area that are necessary for the operation. It is memory. Reference numeral 302 denotes a bus that interconnects various components constituting the moving image encoding apparatus and transfers data and control signals.

  A terminal 303 is used to start the apparatus, set various conditions, and issue a playback instruction. A storage device 304 stores software. A storage device 305 stores the encoded stream. The storage device 304 and the storage device 305 can also be configured by a portable storage medium that can be moved separately from the moving image encoding device.

  Reference numeral 306 denotes a camera (imaging unit) that can capture a moving image frame by frame. Reference numeral 307 denotes a monitor for displaying an image, and reference numeral 309 denotes a communication circuit, which includes a LAN, a public line, a wireless line, a broadcast wave, and the like. A communication interface (I / F) 308 transmits and receives a stream via the communication circuit 309.

  The memory 301 controls the entire apparatus and stores an OS for operating various software and software to be operated. In addition, an image area for storing image data, a code area for storing generated encoded data, There is a working area for storing parameters for various operations and encoding.

  In such a configuration, a process of encoding a moving image taken from the camera 306 and transmitting it to the external terminal via the communication line 309 will be described. Here, H.264 is used as the encoding method. The H.264 encoding method will be described as an example, but the present invention is not limited to this, and any encoding method may be used as long as encoding is performed by referring to a plurality of frames of two or more frames. For ease of explanation, the previous five frames are referred to as encoding target frames, but the present invention is not limited to this.

  Prior to processing, activation is instructed from the terminal 303 to the entire apparatus, and each component is initialized. As a result, the software stored in the storage device 304 is expanded in the memory 301 via the bus 302, and the software is activated.

Here, the data configuration of the memory 301 will be described with reference to FIG. FIG. 10 is a diagram showing a data configuration of a memory according to the third embodiment of the present invention.
As shown in FIG. 10, the memory 301 includes an OS 301 a, H.264, and the like for controlling the entire apparatus and operating various software. The video encoding software 301b that executes the H.264 encoding method, the reference frame determination software 301c that determines a frame to be referenced by the video encoding software 301b, and the communication software 301d that controls communication are stored. Further, an image area 301e for storing image data, a code area 301f for storing encoded data, and a working area 301g are configured.

In such a configuration, a control procedure of the moving image encoding apparatus by the CPU 300 will be described with reference to FIG.
FIG. 8 is a flowchart showing processing executed by the video encoding apparatus according to the present embodiment.

  In step S700, the communication software is activated and preparations for outputting encoded data via the communication line 309 are executed. When the encoded data is stored in the code area, the communication software transmits it to the communication line 309 via the communication interface 308, and clears the corresponding area of the code area after transmission. Hereinafter, transmission of encoded data in the code area will not be particularly mentioned.

  Next, in step S701, the reference frame determination software is activated, each unit is initialized, and moving image shooting is started. At this time, the frame counter variable G is cleared to zero.

  Next, as in the first embodiment, header information is generated and stored in the code area 301f on the memory 301 in steps S1 to S4. In step S3, the frame data to be encoded is input and stored in the image area 301e on the memory 301. In step S4, the sharpness Ek is calculated from the frame data and stored in the working area 301g on the memory 301 in units of frames. In step S702, the reference frame determination software is operated.

Next, details of the reference frame selection process in step S702 will be described with reference to FIG.
FIG. 9 is a flowchart showing details of the process in step S702 according to the third embodiment of the present invention.
First, in step S801, the sharpness Ek calculated in step S4 of FIG. 8 is compared with the first sharpness threshold T, and if greater than the first sharpness threshold T, the process proceeds to step S802, otherwise The process proceeds to step S805.

  Next, in step S802, the sharpness Ek is compared with the second sharpness threshold Z. The second sharpness threshold Z is sufficiently larger than the first sharpness threshold T, and this value is the sharpness Ek extracted from something like sandstorm noise. If it is larger than the second sharpness threshold Z, the process proceeds to step S805. Otherwise, the process proceeds to step S803.

  In step S803, a similarity Ri with a reference frame equal to or greater than the first sharpness threshold T is calculated. Here, for the purpose of explanation, motion compensation is performed between images obtained by binarizing the sharpness Ek obtained by Equation (1) for each pixel value with an average value, and the resulting prediction error is obtained. It is not limited to. Similarity Ri can also be compared with the sum of absolute differences of pixel values and the sum of prediction errors when motion compensation is performed.

In step S804, the similarity Ri is compared with the third threshold ρ. If the similarity Ri is larger than the third threshold ρ, the process proceeds to step S805, and if not, the process proceeds to step S810.
In step S805, the frame counter G is compared with the threshold value Gmax. If it is smaller than the threshold value Gmax, the process proceeds to step S806. Otherwise, the process proceeds to step S808.

  In step S806, 1 is added to the frame counter G. Thereafter, in step S807, the oldest reference frame (M frame) near the time is selected as a removal candidate, and the process proceeds to step S5 in FIG.

  Next, in step S808, the reference frame (N frame) selected with the sharpness Ek and having the lowest sharpness Ek stored in the working area 301g on the memory 301 is selected as a removal candidate.

  Next, in step S809, “N ′” obtained by subtracting “the number N of frames” selected by the sharpness Ek by one is obtained, and “M ′” obtained by increasing the “number of frames M” in the vicinity by one by one. ”, The frame counter G is reset to“ 0 ”, and the process proceeds to step S5 in FIG.

  Next, in step S810, it is determined whether or not the number N of frames selected by the sharpness Ek is smaller than the maximum value Nmax. If smaller, the process proceeds to step S811, and if not, the process proceeds to step S813. Nmax may be a value obtained by subtracting 2 from the maximum number of reference frames, for example. In the present embodiment, Nmax is 3 according to this example.

Next, in step S811, the oldest frame on the memory 301 among the reference frames (M frames) near the time is selected as a removal candidate.
Next, in step S812, “N ′” obtained by increasing the “number of frames N” selected by the sharpness Ek by one is obtained, and “M ′” obtained by subtracting the “frame number M” nearby in time from one. " In step S813, the reference frame (N frame) selected with the sharpness Ek having the lowest sharpness Ek stored in the working area 301g on the memory 301 is selected as a candidate for removal.

  Thereafter, in step S814, the frame counter G is reset to “0”, and the process proceeds to step S5 in FIG. Then, in the same manner as in the first embodiment described above, reference frames are selected and encoded in steps S5 to S7.

  Thereafter, in step S703, the decoded image is reproduced from the encoded data, and the removal candidate frame selected in step S702 is deleted and stored. The sharpness Ek is also stored in the same manner. If the values of “frame number N ′” and “frame number M ′” are obtained, the values are set as new “frame number N” and “frame number M”. Thereafter, the processing returns to step S2 in order to process the next frame.

  As described above, according to the third embodiment, motion compensation can be efficiently performed by referring to a frame that has a high sharpness Ek and is considered to contain a lot of important information. . Furthermore, by making the number of frames near the time in the reference frame and the number of frames selected by the sharpness Ek variable according to the sharpness Ek, it becomes possible to refer to the optimum frame. Furthermore, it is possible to refer to important frame data by removing frame data with high sharpness Ek but not important from the reference frame.

  In the third embodiment, the reference frames are five frames before the encoding target frame. However, the present invention is not limited to this, and by securing a necessary amount of image area for storing the reference frames. It is clear that other numbers of references can be handled. In the third embodiment, the maximum value of the number of reference frames based on the sharpness Ek is provided. However, the present invention is not limited to this and may be omitted. Moreover, although the maximum value is not provided for the number of frames in the vicinity of time, this is not limited. For example, it is possible to define Mmax so as not to exceed this.

  Furthermore, in the third embodiment, the example in which the encoded data in the code area 301f is transmitted to the communication line 309 via the communication interface 308 has been described. However, the encoded data may be stored in the storage device 305. .

  Further, the input image is not limited to a captured image from the camera, and moving image data stored in the storage device 305 can be used. Further, when a scene change or the like is detected, the reference frame numbers N and M can be reset.

(Another embodiment according to the present invention)
Each means constituting the moving picture coding apparatus and each step of the moving picture coding method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable storage medium storing the program are included in the present invention.

  In addition, the present invention can be embodied as a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowcharts shown in FIGS. 3, 5, 8, and 9) that realizes the functions of the above-described embodiments is directly transferred to the system or apparatus. Alternatively, it may be achieved by supplying from a remote location and the computer of the system or apparatus reads and executes the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of an object code, a program executed by an interpreter, script data supplied to the OS, or the like. As a recording medium for supplying the program, for example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a recording medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instructions of the program is used for the actual processing. The functions of the above-described embodiment can be realized by performing some or all of the processes.

  Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

BRIEF DESCRIPTION OF THE DRAWINGS It is the block diagram which shows the 1st Embodiment of this invention and shows the structure of the moving image encoder which concerns on this invention. FIG. 3 is a block diagram illustrating a detailed configuration example of a reference frame determiner according to the first embodiment of this invention. It is a flowchart which shows the 1st Embodiment of this invention and shows the example of a moving image encoding operation | movement. It is a block diagram which shows the 2nd Embodiment of this invention and shows the structural example of the moving image encoder concerning this invention. It is a flowchart which shows the 2nd Embodiment of this invention and shows an example of a moving image encoding operation | movement. It is the block diagram which shows the 2nd Embodiment of this invention and shows another structural example of the moving image encoder concerning this invention. It is a block diagram which shows the 3rd Embodiment of this invention and shows the structural example of the moving image encoder concerning this invention. It is a flowchart which shows the 3rd Embodiment of this invention and shows the example of a moving image encoding operation | movement. It is a flowchart which shows the 3rd Embodiment of this invention and shows the example of determination operation | movement of a reference frame. It is a figure which shows the 3rd Embodiment of this invention and shows the data structure of memory. It is a block diagram which shows a prior art example and shows the structural example of a moving image encoder.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 9 Selector 2 Intra predictor 3 Difference machine 4 Transformer / quantizer 5 Entropy encoder 6 Inverse quantizer / inverse transformer 7 Adder 8_1-8_16 Frame memory 10 Motion estimator 11 Sharpness detector 12 Reference frame Determiner 13 motion compensator 14 motion encoder 15 reference information encoder 16 output unit

Claims (8)

A moving image encoding apparatus that performs motion compensation on frame data to be encoded with reference to a plurality of frame data in a moving image, and encodes the moving image,
Input means for inputting frame data;
Sharpness detection means for detecting sharpness from the frame data input by the input means;
Frames input by the input unit as reference frame data to be referred to when the frame data of the upper predetermined number of frames with high sharpness detected by the sharpness detection unit is encoded when the frame data to be encoded is encoded A selection means for selecting from the data ;
Motion compensation means for performing motion compensation based on the reference frame data selected by the selection means and the frame data to be encoded;
Encoding means for encoding the frame data to be encoded using the result of motion compensation performed by the motion compensation means;
A moving picture coding apparatus comprising: output means for outputting the frame data coded by the coding means.   The moving image according to claim 1, wherein the input unit further includes an imaging unit that images a subject, and the sharpness detection unit includes a focus degree detection unit that detects a focus degree. Encoding device. The selection means uses the frame data of the upper predetermined number of frames with high sharpness detected by the sharpness detection means, the frame data to be encoded and the frame data of the predetermined number of frames near the time as reference frame data. selecting as a moving image encoding apparatus according to claim 1 or 2, characterized in. A moving image encoding method that performs motion compensation on frame data to be encoded with reference to a plurality of frame data in a moving image, and encodes the moving image,
An input process for inputting frame data;
A sharpness detection step of detecting the sharpness from the frame data input by the input step;
Frame data predetermined number of top high been sharpness detected by said sharpness detecting step, the frame data inputted by said input step as reference frame data to be referred to when coding the frame data of the encoding target A selection process to select from,
A motion compensation step for performing motion compensation based on the reference frame data selected by the selection step and the frame data to be encoded;
An encoding step of encoding the frame data to be encoded using a result of motion compensation performed by the motion compensation step;
A moving image encoding method comprising: an output step of outputting the frame data encoded in the encoding step.   5. The moving image according to claim 4, wherein the input step further includes an imaging step of imaging a subject, and the sharpness detection step includes a focus degree detection step of detecting a focus degree of focus. Encoding method. In the selection step, frame data of a higher predetermined number of frames having a high degree of sharpness detected by the sharpness detection step, frame data to be encoded and frame data of a predetermined number of frames in the vicinity of time are referred to as reference frame data video encoding method according to claim 4 or 5, characterized in that selected as. A program for referring to a plurality of frame data in a moving image, executing motion compensation for the frame data to be encoded, and causing a computer to execute a moving image encoding method for encoding the moving image,
An input process for inputting frame data;
A sharpness detection step of detecting a sharpness from the frame data inputted by said input step,
Frames input by the input means as reference frame data to be referred to when the frame data of the upper predetermined number of frames with high sharpness detected by the sharpness detection step is encoded when encoding the frame data to be encoded A selection process to select from the data ;
A motion compensation step for performing motion compensation based on the reference frame data stored in the storage medium selected by the selection step and the frame data to be encoded;
An encoding step of encoding the frame data to be encoded using a result of motion compensation performed by the motion compensation step;
A program causing a computer to execute an output step of outputting frame data encoded in the encoding step.   A computer-readable recording medium, wherein the program according to claim 7 is recorded.