WO2005088981A1 - Encoding apparatus, encoding method, decoding apparatus, and decoding method - Google Patents

Encoding apparatus, encoding method, decoding apparatus, and decoding method Download PDF

Info

Publication number
WO2005088981A1
WO2005088981A1 PCT/JP2005/004721 JP2005004721W WO2005088981A1 WO 2005088981 A1 WO2005088981 A1 WO 2005088981A1 JP 2005004721 W JP2005004721 W JP 2005004721W WO 2005088981 A1 WO2005088981 A1 WO 2005088981A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
image
frames
input
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2005/004721
Other languages
English (en)
French (fr)
Inventor
Mitsuru Maeda
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Priority to US10/586,352 priority Critical patent/US8064518B2/en
Publication of WO2005088981A1 publication Critical patent/WO2005088981A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/58—Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H04N19/139—Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/573—Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding

Definitions

  • the present invention relates to a technique for encoding frames that form a moving image by motion compensation, and a technique for decoding the encoded result .
  • This encoding method is jointly developed by the ITU-T and ISO.
  • 4 x 4 integer transformation is used, and a plurality of intra-prediction schemes are prepared, unlike in the conventional MPEG-1, 2, and 4 encoding methods.
  • An intra-loop filter is used, and motion compensation is made using seven different types of subblocks .
  • the pixel precision of the motion compensation is as high as 1/4 pixel precision as in the MPEG-4 encoding method.
  • universal variable-length encoding and context adaptive variable-length encoding are used as entropy encoding.
  • MPEG-1, 2, or 4 performs motion compensation using two reference images before and after the current image, but H.264 can use more reference images .
  • a num_ref_frames code included in the first header of a bitstream can assume a maximum of 16 values. That is, 16 frames before and after the current frame can be referred to.
  • prediction errors of seven different types of subblocks are calculated for images of a maximum of 16 frames, as described above, and a macroblock that can minimize the prediction errors is selected, thus greatly improving the encoding efficiency.
  • Fig. 11 shows the arrangement of an H.264 encoder. This encoder receives image data for respective macroblocks .
  • a switch 1000 switches whether or not intra-encoding is performed.
  • image data is input into an intra-predictor 1001, which predicts in nine modes in order to calculate prediction errors.
  • image data is input into a differentiator 1002, which calculates differences from predicted images to generate prediction errors .
  • a transformer/quantizer 1003 computes the integer transformation of the obtained prediction errors for 4 x 4 pixel blocks , thus quantizing respective coef icients .
  • the quantization result undergoes variable-length encoding by an entropy encoder 1004, and the encoded result is output to an output unit 1014 At the same time, the quantization result is input to an inverse quantxzer/inverse transformer 1005, which reconstructs the prediction errors.
  • the prediction errors are added to prediction images by an adder 1006.
  • a motion estimator 1011 compares decoded images stored in the frame memories 1007 to 1010 with an input image to calculate motion vectors for respective subblocks with 1/4 pixel precision.
  • the motion vectors and the numbers of the selected frames are also input to a motion compensator 1012, which loads reference images from the corresponding frame memories, selects a reference image with minimum prediction errors, and outputs it as a predicted image to the differentiator 1002.
  • the motion vectors and the numbers of the selected frames are input to a motion encoder 1013 and are encoded, so that the motion vectors are encoded to mvd_idx_IO codes and the like, and information of the reference frames is encoded to ref_idx_IO codes and the like.
  • the encoded codes are output to the output unit 1014.
  • the output unit 1014 shapes and outputs encoded data in accordance with the format .
  • Fig. 22 shows the arrangement of an H.264 decoder.
  • An entropy decoder 51021 performs variable-length decoding to obtain the quantization result of 4 x 4 transformation coefficients .
  • the quantization result is input to an inverse quantizer/inverse transformer 51022 to reconstruct prediction errors .
  • image data is input to an intra-predictor 51023 to perform prediction from surrounding pixels, thus reconstructing and outputting pixel data.
  • an adder 51024 adds the prediction errors to predicted images to reconstruct and output pixel data. At the same time, these pixel data are stored in the frame memories 51025 to 51028 accordingly.
  • a motion decoder 51029 decodes mvd_idx_IO codes and the like representing motion vectors and ref_idx_IO codes and the like representing reference, frame information, and inputs decoded data to a motion compensator 51030.
  • the motion compensator 51030 loads reference images from the corresponding frame memories and outputs them as predicted images to the adder 51024.
  • the aforementioned encoding apparatus encodes by referring to a plurality of frames, and tries to reduce prediction errors . ith reference to many frames .
  • an encoding apparatus of the present invention comprises the following arrangement .
  • an encoding apparatus for encoding images of frames , which form a moving image by motion compensation, characterized by comprising: input means for inputting images of frames; section division means for dividing the frames into a plurality of sections on the basis of the images of the frames input by the input means; representative image setting means for setting one representative image that represents the image of each frame in each of the sections divided by the section division means; and reference image selection means for selecting one representative image to be referred to so as to encode an image of a frame of interest from the representative images set for respective sections , and in that the image of the frame of interest is encoded by motion compensation using the images of the frames in the section that includes the representative image selected by the reference image selection means .
  • an encoding apparatus of the present invention comprises the following arrangement. That is, an encoding apparatus for encoding images of frames, which form a moving image by motion compensation, characterized by comprising: input means for inputting images of frames; section division means for dividing the frames into a plurality of sections on the basis of the images of the frames input by the input means; representative image setting means for setting one representative image that represents the image of each frame in each of the sections divided by the section division means; and reference image selection means for selecting one representative image to be referred to so as to encode an image of a frame of interest from the representative images set for respective sections, and in that the image of the frame of interest is encoded by motion compensation using the representative image selected by. the reference image selection means.
  • an encoding apparatus of the present invention comprises the following arrangement . That is , an encoding apparatus for encoding images of frames, which form a moving image by motion compensation, characterized by comprising: input means for inputting images of frames for every predetermined number of frames; representative image setting means for setting one representative image that represents images of the predetermined number of frames on the basis of the images of the frames for every predetermined number of frames ; and output means for outputting encoded results of the images of the frames input by the input means together with information required to specify the representative image.
  • a decoding apparatus of the present invention comprises the following arrangement.
  • a decoding apparatus for decoding an encoded result of images of frames that form a moving image by motion compensation, characterized by comprising: input means for inputting encoded results of images of frames for a predetermined number of frames, and information required to specify a representative frame which represents the frames for the predetermined number of frames, which is appended to each of the encoded results of the frames; and storage means for storing a decoded result of the representative frame specified by the information, and in that the encoded result of an image of each frame input by the input means after the representative frame stored in the storage means is decoded using the decoded result of the representative frame stored in the storage means.
  • an encoding apparatus of the present invention comprises the following arrangement .
  • an encoding apparatus for encoding images of frames, which form a moving image by motion compensation, characterized by comprising: input means for inputting images of frames; setting means for setting an image of a frame input at a predetermined cycle of the images of the frames input by the input means as an image of a representative frame; switching means for switching an encoding method depending on whether or not a frame to be encoded is a representative frame; and output means for outputting encoded results of the images of the. frames input by the input means together with information required to specify the representative frame.
  • an encoding apparatus of the present invention comprises the following arrangement.
  • an encoding apparatus for encoding images of rames , which form a moving image by motion compensation, characterized by comprising: input means for inputting images of frames for every predetermined number of frames; calculation means for calculating a difference between an image of a frame input by the input means and a decoded image obtained by decoding a result of encoding the image of the frame; determination means for determining an image of a frame having a smallest difference of the differences calculated by the calculation means for the frames for the predetermined number of frames as a representative image that represents the images of the frames for the predetermined number of frames; and output means for outputting encoded results of the images of the frames input by the input means together with information required to specify the representative frame.
  • an encoding apparatus of the present invention comprises the following arrangement . That is, an encoding apparatus for encoding images of frames, which form a moving image by motion compensation, characterized by comprising: input means for inputting images of frames; storage means for storing an image of a representative frame to be referred to when inter-frame encoding is applied to an input image input by the input means; calculation means for calculating similarity levels of images stored in the storage means with the input image; storage control means for, when a minimum similarity level of the similarity levels calculated by the calculation means for the images stored in the storage means is not less than a predetermined value, storing the input image in the storage means as a representative frame; and output means for outputting encoded results of the images of the frames input by the input means together with information required to specify the representative image.
  • an encoding apparatus of the present invention comprises the following arrangement. That is, an encoding apparatus for encoding images of frames, which form a moving image by motion compensation, characterized by comprising: input means for inputting images of frames; storage means for storing an image of a representative frame to be referred to when inter-frame encoding is applied to an input image input by the input means; motion vector calculation means for calculating motion vectors with the input image for images stored in the storage means; calculation means for calculating errors of the motion vectors calculated by the motion vector calculation means for the images stored in the storage means ; storage control means for, when a minimum error of the errors calculated by the calculation means for the images stored in the storage means is not more than a predetermined value, storing the input image in the storage means as a representative frame; and output means for outputting encoded results of the images of the frames input by the input means using the motion vectors together with information required to specify the representative image.
  • an encoding method of the present invention comprises the following arrangement . That is, an encoding method of encoding images of frames, which form a moving image by motion compensation, characterized by comprising: an input step of inputting images of frames; a section division step of dividing the frames into a plurality of sections on the basis of the images of the frames input in the input step; a representative image setting step of setting one representative image that represents the image of each frame in each of the sections divided in the section division step; and a reference image selection step of selecting one representative image to be referred to so as to encode an image of a frame of interest from the representative images set for respective sections, and in that the image of the frame of interest is encoded by motion compensation using the images of the frames in the section that includes, the representative image selected in the reference image selection step.
  • an encoding method of the present invention comprises the following arrangement . That is, an encoding method of encoding images of frames, which form a moving image by motion compensation, characterized by comprising: an input step of inputting images of frames; a section division step of dividing the frames into a plurality of sections on the basis of the images of the frames input in the input step; a representative image setting step of setting one representative image that represents the image of each frame in each of the sections divided in the section division step; and a reference image selection step of selecting one representative image to be referred to so as to encode an image of a frame of interest from the representative xmages set for respective sections , and in that the image of the frame of interest is encoded by motion compensation using the representative image selected in the reference image selection step.
  • an encoding method of the present invention comprises the following arrangement. That is , an encoding method of encoding images of frames , which form a moving image by motion compensation, characterized by comprising: an input step of inputting images of frames for every predetermined number of frames;. a representative image setting step of setting one representative image that represents images of the predetermined number of frames on the basis of the images of the frames for every predetermined number of frames ; and an output step of outputting encoded results of the images of the frames input in the input step together with information required to specify the representative image.
  • a decoding method of the present invention comprises the following arrangement.
  • a decoding method of decoding an encoded result of images of frames that form a moving image by motion compensation characterized by comprising: an input step of inputting encoded results of images of frames for a predetermined number of frames, and information required to specify a representative frame which represents the frames for the predetermined number of frames, which is appended to each of the encoded results of the frames; and a storage step of storing a decoded result of the representative frame specified by the information, and in that the encoded result of an image of each frame input in the input step after the representative frame stored in the storage step is decoded using the decoded result of the representative frame stored in the storage step.
  • an encoding method of the present invention comprises the following arrangement.
  • an encoding method of encoding images of frames which form a moving image by motion compensation, characterized by comprising: an input step of inputting images of frames; a setting step of setting an image of a frame input at a predetermined cycle of the images of the frames input in the input step as an image of a representative frame; a switching step of switching an encoding method depending on whether or not a frame to be encoded is a representative frame; and an output step of outputting encoded results of the images of the frames input in the input step together with information required to specify the representative frame.
  • an encoding method of the present invention comprises the following arrangement.
  • an encoding method of encoding images of frames which form a moving image by motion compensation, characterized by comprising: an input step of inputting images of frames for every predetermined number of frames; a calculation step of calculating a difference between an image of a frame input in the input step and a decoded image obtained by decoding a result of encoding the image of the frame; a determination step of determining an image of a frame having a smallest difference of the differences calculated in the calculation step for the frames for the predetermined number of frames as a representative image that represents the images of the frames for the predetermined number of frames; and an output step of outputting encoded results of the images of the frames input in the input step together with information required to specify the representative frame.
  • an encoding method of the present invention comprises the following arrangement. That is, an encoding method of encoding images of frames , which form a moving image by motion compensation, characterized by comprising: an input step of inputting images of frames; a storage step of storing an image of a representative frame to be referred to when inter-frame encoding is applied to an input image input in the input step; a calculation step of calculating similarity levels of images stored in the storage step with the input image; a storage control step of storing, when a minimum similarity level of the similarity levels calculated by the calculation step for the images stored in the storage step is not less than a predetermined value, the input image in the storage step as a representative frame; and an output step of outputting encoded results of the images of the frames input in the input step together with information required to specify the representative image.
  • an encoding method of the present invention comprises the following arrangement . That is, an encoding method of encoding images of frames, which form a moving image by motion compensation, characterized by comprising: an input step of inputting images of frames; a storage step of storing an image of a representative frame to be referred to when inter-frame encoding is applied to an input image input in the input step; a motion vector calculation step of calculating motion vectors with the input image for images stored in the storage step; a calculation step of calculating errors of the motion vectors calculated in the motion vector calculation step for the images stored in the storage step; a storage control step of storing, when a minimum error of the errors calculated in the calculation step for the images stored in the storage step is not more than a predetermined value, the input image in the storage step as a representative frame; and an output step of outputting encoded results of the images of the frames input in the input step using the motion vectors together with information required to specify the representative image.
  • Fig. 1 is a block diagram showing the arrangement of a moving image encoding apparatus according to the first embodiment of the present invention
  • Fig. 2 is a flowchart of an encoding process according to the first embodiment of the present invention
  • Fig. 3 is a block diagram showing the arrangement of a moving image.
  • encoding apparatus according to the second embodiment of the present invention
  • Fig. 4 is a flowchart of an encoding process according to the second embodiment of the present invention
  • Fig. 1 is a block diagram showing the arrangement of a moving image encoding apparatus according to the first embodiment of the present invention
  • Fig. 3 is a block diagram showing the arrangement of a moving image.
  • Fig. 4 is a flowchart of an encoding process according to the second embodiment of the present invention
  • FIG. 5 is a block diagram showing the basic arrangement of a moving image encoding apparatus according to the third embodiment of the present invention
  • Fig. 6 shows an example of the memory map in a memory 301
  • Fig. 7 is a flowchart of a process implemented by a CPU 300 upon executing moving image encoding software and communication software according to the third embodiment of the present invention
  • Fig. 8 is a flowchart showing, details of the process in step S205
  • Fig. 9 shows the contents of a table that represents a section
  • Fig. 10 is a flowchart of a moving image encoding process according to the fourth embodiment of the present invention
  • Fig. 11 is a block diagram showing the arrangement of an H.264 encoder
  • FIG. 12 is a block diagram showing the arrangement of a moving image encoding apparatus according to the fifth embodiment of the present invention
  • Fig. 13 is a flowchart of an encoding process according to the fifth embodiment of the present invention
  • Fig. 14 is a block diagram showing the arrangement of a decoding apparatus according to the sixth embodiment of the present invention
  • Fig. 15 is a flowchart of a decoding process according to the sixth embodiment of the present invention
  • FIG. 16 is a block diagram showing the arrangement of a moving image encoding apparatus according to the seventh embodiment of the present invention
  • Fig. 17 is a flowchart of an encoding process according to the seventh embodiment of the present invention
  • Fig. 18 shows an example of the.
  • FIG. 19 is a flowchart of a process implemented by a CPU 300 upon executing moving image encoding software and communication software according to the eighth embodiment of the present invention
  • Fig. 20 shows an example of the memory map in a memory 301 in the ninth embodiment of the present invention
  • Fig. 21 is a flowchart of a process implemented by a CPU 300 upon executing moving image decoding software and communication software according to the ninth embodiment of the present invention
  • Fig. 22 is a block diagram showing the arrangement of a conventional H.264 decoder
  • Fig. 23 is a block diagram showing the arrangement of a moving image encoding apparatus according to the 10th embodiment of the present invention
  • Fig. 24 is a flowchart of a moving image encoding process according to the 10th embodiment of the present invention
  • Fig. 25 is a block diagram showing the arrangement of a moving image encoding apparatus according to the 11th embodiment of the present invention
  • Fig. 26 is a flowchart of a moving image encoding process according to the 11th embodiment of the present invention
  • Fig. 27 is a flowchart of a process implemented by a CPU 300 upon executing moving image encoding software and communication software according to the 12th embodiment of the present invention
  • Fig. 28 is a flowchart of a process implemented by a CPU 300 upon executing moving image decoding software and communication software according to the 13th embodiment of the present invention.
  • Fig. 1 is a block diagram showing the arrangement of a moving image encoding apparatus according to this embodiment.
  • a description will be given based on the H.264 encoding method, but temporally discontinuous reference frames are to be referred to unlike in H.264.
  • the base method is not limited to H.264.
  • forward prediction that refers to previous frames will be exemplified.
  • the present invention can also be applied to two-way prediction.
  • image data of a frame will also be referred to as frame data.
  • This frame data (an image of a frame) includes data indicating the frame number of this frame.
  • reference numerals 19 to 22 denote frame memories for storing images of frames (input images) of an externally input moving image.
  • Reference numeral 1 denotes a selector which selects an output destination in accordance with an intra-frame encoding/inter-frame encoding mode.
  • Reference numeral 2 denotes an intra-predictor which performs intra-prediction based on the H.264 encoding method; and 3, a differentiator for obtaining motion prediction errors .
  • Reference numeral 4 denotes a transformer/quantizer which performs integer orthogonal transformation, and quantizes input coefficients; and 7, an inverse quantizer/inverse transformer which performs the reverse process of the transformer/quantizer 4.
  • Reference numeral 8 denotes an adder which adds motion predicted images .
  • Reference numeral 5 denotes an entropy encoder which encodes the quantization result of the transformer/quantizer 4; and 6 , a buffer which stores generated codes for a plurality of frames.
  • Reference numerals 9, 10, 11, and 12 denote frame memories each of which stores locally decoded image data and the frame number for each frame.
  • Reference numeral 13 denotes a motion estimator which extracts an optimal motion vector from the corresponding frame of an input image and decoded image data; and 14, a motion compensator which generates a predicted image on the basis of the motion vector calculated by the motion estimator 13 and information of the corresponding frame.
  • Reference numeral 15 denotes a motion encoder which encodes motion information on the basis of the calculated motion vector and information of the corresponding frame.
  • Reference numeral 17 denotes a frame memory controller which controls inputs/outputs of the frame memories 9 to 12; and 16, an output unit which outputs encoded data stored in the buffer 6.
  • Reference numeral 25 denotes a section setting unit which sets a section by calculating a similarity level upon comparison of input images.
  • Reference numeral 18 denotes a representative frame determination unit which determines a representative frame that represents the section.
  • Reference numeral 26 denotes a selector which selects a frame to be encoded.
  • Reference numeral 27 denotes a section memory which stores the set section; and 28, a representative frame memory which stores the determined representative frame.
  • Reference numeral 29 denotes a reference frame determination unit which determines a section to be referred to.
  • the section setting unit 25 "1" indicating the first frame is set as a start point value indicating the start point and an end point value indicating the end point of the section.
  • the output unit 16 generates and outputs header information which represents the maximum number of frames to be referred to.
  • Image data of frames input to the moving image encoding apparatus of this embodiment are stored in turn in the frame memories 19 to 22.
  • Each input image data is input to the section setting unit 25 together with image data of the previously input frame. That is, a set of images of the N-th and (N-l)-th frames are input to the section setting unit 25.
  • the section setting unit 25 calculates a difference R between the two input frame data (image data of frames ) .
  • a sum total value obtained by adding the absolute values of differences between pixel values of corresponding pixels in the two images for all the pixels is used as the difference R.
  • XN(1, j) (i and j indicate the position of a pixel in an image X N ) be the pixel value of the image X N of the N-th frame stored in the frame memory 19
  • X N - ⁇ (i, j) be the pixel value of an image X N _ ⁇ of the (N-l)-th frame stored in the frame memory 20
  • the section setting unit 25 adds 1 to the end point value, and repeats the same process by incrementing N by 1.
  • the section setting unit 25 stores the start and end point values of the section on a free area of the section memory 27 or overwrites them on data indicating the oldest section if no free area is available in the section memory 27. Also, these values are input to the representative frame determination unit 18. The representative frame determination unit 18 loads these values (start and end point values) indicating the section and image data. of frames in the section stored in the frame memories 19 to 22, and calculates a difference Q.
  • the representative frame determination unit 18 selects a frame with the smallest difference Q of the calculated differences QN.
  • the unit 18 inputs the number of the selected frame to the representative frame memory 28.
  • the section setting unit 25 sets a value obtained by adding 1 to the end point value in the start and end point values .
  • the selector 26 refers to the latest start and end point values stored in the section memory 27, sequentially reads out frame data between the start and end point values of those stored in the frame memories 19 to 22, and inputs them to the selector 1.
  • the selector 1 selects the intra-predictor 2 as an output so as to intra-frame encode frames at predetermined intervals (e.g., every 15 frames).
  • the selector 1 selects the differentiator 3 as an output so as to perform inter-frame encoding.
  • input frame data undergoes prediction for respective blocks .
  • each input image data is input to the motion estimator 13 for respective slices .
  • Representative frame data of respective sections are read out from the frame memories 9 to 12 with reference to "the representative frame numbers of previously set sections) held in the representative frame memory 28, and the motion estimator 13 and motion compensator 14 generate predicted images corresponding to input images of the representative frames. Then, errors (prediction errors) between the predicted images of the representative frames and input images are calculated, and a representative frame with smallest prediction errors from the input image is specified.
  • the start and end point values as information indicating the two ends of the section that includes the frame number of the specified representative frame are read out from the representative frame memory 28, and images of frames in the section specified by the readout start and end point values are read out from the frame memories 9 to 12.
  • the motion estimator 13 and motion compensator 14 execute a known encoding process based on motion compensation using the readout images and input images . Since the motion compensator 14 generates a predicted image corresponding to an input image, that image is input to the differentiator 3. The differentiator 3 calculates a prediction error from the input image and predicted image.
  • a motion vector is encoded by the motion encoder 15, and is stored in the buffer 6.
  • the result obtained by the intra-predictor 2 or differentiator 3 is input to the transformer/quantizer 4, which computes the integer orthogonal transforms and quantizes their coefficients.
  • the quantization result is input to the entropy encoder 5 and inverse quantizer/inverse transformer 7.
  • the entropy encoder 5 entropy-encodes the input quantization result and stores encoded data in the buffer 6 for respective frames .
  • the intra-predictor 2 or differentiator 3 outputs the prediction error
  • the prediction error is supplied to the adder 8 via the transformer/quantizer 4 and inverse quantizer/inverse transformer 7, and is added to a "predicted image corresponding to an input image" input from the motion compensator 14 by the adder 8.
  • the frame memories 9 to 12 store data obtained by reclaiming frame data stored in the frame memories 19 to 22.
  • the aforementioned encoding process based on motion compensation is a known one except for the process for specifying images to be referred to (in the above description, images of frames in the section including the frame number of the specified representative frame) .
  • a section of a sequence of frames whose images are not so different among continuous images (the difference Rn between neighboring frames is equal to or smaller than the threshold Th) is set. Furthermore, a frame having a smallest difference from other frames in those of this section is set as the frame of a reference image.
  • the number of images to be referred to upon encoding frames in the section is one, and the number of images to be referred to can be reduced, thus allowing a faster encoding process.
  • the inverse quantizer/inverse transformer 7 obtains the prediction error (a difference between the input image and predicted image) from the input quantization result, and adds it to a predicted image from the motion compensator 14, thus reclaiming an input image.
  • Fig. 2 is a flowchart of the aforementioned encoding process. As described above, respective units shown in Fig. 1 are initialized, and the output unit 16 generates and outputs header information indicating the maximum number of frames to be referred to (step SI).
  • step S2 It is checked if the encoding process of all frames to be encoded is complete (step S2). If frames to be encoded still remain, the flow advances to step S3 to check if the frame memories 19 to 22 become full of image data of non-encoded frames (step S3). If the frame memories still have an enough space, the flow advances to step S4; otherwise, the flow advances to step S8. If the frame memories still have an enough space, frame data to be encoded are sequentially input to the frame memories 19 to 22 (step S4).
  • the section setting unit 25 refers to a frame at a position indicated by the start point value. In this case, the section setting unit 25 calculates a difference R between the current reference frame and next frame according to equation (1) (step S5).
  • step S6 It is checked if the calculated difference R is smaller than the predetermined threshold Th (step S6). If R ⁇ Th, the section setting unit 25 determines that the difference between the current reference frame image and the next frame image is not so large, updates the end point value by adding 1 to it (step S7), and repeats the processes in step S3 and subsequent, steps. In this manner, if it is determined that the difference between the current reference frame image and the next frame image is not so large, the end point is updated one by one to update the section specified by the start and end point values.
  • the section setting unit 25 determines that the current reference frame image and the next frame image have a difference, and stores the start and end point values in the section memory 27 so as to refer to the section specified by the start and end point values in subsequent processes (step S8).
  • the representative frame determination unit 18 loads these values (start and end point values) indicating the section, and image data of respective frames in this section of the frame memories 19 to 22, and executes the process for calculating differences QN according to equation (2) for these frames (step S9). Then, the unit 18 sets a frame with a smallest difference QN of those calculated for respective frames as a representative frame.
  • the unit 18 inputs and stores the frame number of the frame with a smallest difference Q in the frame memory 28 (step S10) .
  • the subsequent processes calculate motion vectors for respective frames in this section, and encode these frames . It is checked if the encoding process of all the frames in the section is complete (step Sll). If frames to be encoded in the section still remain, the flow advances to step S12 to encode images of the frames to be encoded one by one .
  • the process for encoding one frame image will be described below.
  • One frame image is encoded for respective encoding units (to be referred to as slices hereinafter) . It is checked if the encoding process of all encoding units of one image is complete (step S12).
  • step S13 The motion compensator 14 reads out data of representative frames of respective sections held in the representative frame memory 28 on the basis of the "frame numbers for previously set sections" from the frame memories 9 to 12.
  • the motion compensator 14 generates predicted images of the representative frames corresponding to the frame image to be encoded, and calculates errors (prediction errors) between the predicted images corresponding to the representative frames , and the frame image to be encoded (step S13).
  • step S14 the representative frame with a smallest prediction error from the frame to be encoded is specified, and the start and end point values as information indicating the two ends of the section including the frame number of the specified representative frame are read out from the representative frame memory 28, thus setting the section to be referred to so as to encode the encoding unit to be encoded (step S14).
  • Frame images in the section set in step S14 are read out from the frame memories 19 to 22, and the encoding unit undergoes the encoding process using the readout images and the encoding unit to be encoded (step S15) .
  • the process in step S15 will be described in more detail below.
  • the motion estimator 13 compares the encoding unit to be encoded with the corresponding portion of each frame in the section set in step S14 to obtain a motion vector by a known technique.
  • the motion encoder 15 encodes this motion vector and outputs the encoded result to the buffer 6.
  • the motion compensator 14 generates a predicted image for this encoding unit on the basis of the calculated motion vector, and the differentiator 3 calculates the difference between this predicted image and encoding unit as a prediction error.
  • This prediction error is encoded by the transformer/quantizer 4 and entropy encoder 5, and the encoded result is output to the buffer 6.
  • the buffer 6 stores the encoded data, of the motion vector and that of the prediction error as the encoded results of this encoding unit.
  • this encoding unit is directly encoded.
  • the flow returns to step S12 to repeat the subsequent processes .
  • the encoding process in steps S12 to S15 is a known one, and this embodiment has the aforementioned feature in the selection method of the image to be referred to upon encoding.
  • step S12 determines whether the encoding process of all the encoding units that form one frame image is complete. If it is determined in step S12 that the encoding process of all the encoding units that form one frame image is complete, the flow advances to step S16 to store the frame image encoded in steps S12 to S15 in a frame memory that does not store any image data or a frame memory that stores the frame of the oldest number of the frame memories 9 to 12 (step S16) .
  • the output unit 16 externally outputs the encoded data output to the buffer 6 (step S17).
  • step Sll In order to encode the next frame in the section, the flow returns to step Sll to repeat the subsequent processes. If it is determined in step Sll that the encoding process of all the frames in the section is complete, the flow advances to step S18.
  • the section setting unit 25 sets a value obtained by adding 1 to the end point value in the start and end point values to update the start and end point values (step S18). The flow then returns to step S2 to repeat the subsequent processes.
  • the difference is the sum total of the differences for respective pixels between images.
  • the present invention is not particularly limited to this as long as the difference indicates a similarity between images .
  • differences for respective pixels are calculated for all pixels but may be calculated for subsampled pixels .
  • the moving image encoding apparatus according to this embodiment is implemented by hardware having the arrangement shown in Fig. 1.
  • Fig. 3 is a block diagram showing the arrangement of a moving image encoding apparatus according to this embodiment.
  • the same reference numerals in Fig. 3 are identical to Fig.
  • Reference numeral 117 denotes a frame memory controller which controls inputs/outputs of the frame memories 9 to 12; and 125, a section setting unit which sets a section by calculating a similarity level upon comparison of input images.
  • Reference numeral 118 denotes a representative frame determination unit which determines a representative frame that represents the section.
  • the output unit 16 generates and outputs header information which represents the maximum number of frames to be referred to.
  • Frames that form a moving image are sequentially input to the frame memory 19.
  • the selector 1 inputs frame data which is input to and stored in the frame memory to the intra-predictor 2 for respective macroblocks .
  • the intra-predictor 2 performs prediction for blocks, and a prediction error as the prediction result undergoes integer orthogonal transformation and quantization by the transformer/quantizer 4.
  • the quantization result is input to the entropy encoder 5.
  • the entropy encoder 5 entropy-encodes the input quantization result, and stores encoded data in the buffer 6 for respective frames .
  • the inverse quantizer/inverse transformer 7 obtains a decoded image from the input quantization result.
  • the decoded image data is stored in a free frame memory or a frame memory that stores image data of the oldest representative frame of the frame memories 9 to 12 in accordance with an instruction from the frame memory controller 117.
  • the encoded data stored in the buffer 6 is output from the output unit 16. In this manner, the process in the intra-frame encoding mode is the same as that in the first embodiment .
  • the process when the inter-frame encoding mode is set will be described below. Basically, the encoding process is done as in the first embodiment.
  • the section and representative frame are updated as needed parallel to encoding without determining the section and representative frame in advance.
  • the update process will be explained below.
  • the differentiator 3 outputs a difference between an input image and predicted image as a prediction error. This prediction error is added to a predicted image by the adder 8 to reclaim a decoded image.
  • the decoded image is stored in one of the frame memories 9 to 12.
  • Each of input image data stored in the frame memories 9 to 12 is input to the section setting unit 125 together with image data of the previously input frame. That is, a set of images of the N-th and (N-l)-th frames are input to the section setting unit 125.
  • the section setting unit 125 calculates a difference R between the two input frame data according to equation (1). Note that in this embodiment, the method of calculating the difference R is not limited to such specific method. If the calculated difference R is smaller than the pre-set threshold Th, the section setting unit 125 adds 1 to the end point value to update the end point value.
  • the representative frame determination unit 118 calculates differences QN according to equation (2) using images from the frame indicated by the current start point value to the (N-l)-th frame, and the image of the N-th frame (in this case, "images from the frame indicated by the current start point value to the (N-l)-th frame” correspond to X M (1, j), and the "image of the N-th frame” corresponds to XN(1, j))- Also, the section setting unit 125 calculates a difference SN between the N-th frame image and image data of a frame set as the representative frame of the current section according to equation (1).
  • the representative frame determination unit 118 adds the difference SN and a cumulative difference QD (an initial value of this difference QD is zero) of the differences from the representative frame upon encoding up to the (N-l)-th frame to calculate a new cumulative difference QD" .
  • This cumulative difference QD" is compared with the difference QN. As a result of comparison, if the difference QN is smaller than the cumulative difference QDA the N-th frame is set as a new representative frame. For this purpose, the frame number (i.e., N) of the N-th frame is set in the representative frame memory 28. After that, the value of the difference QN is substituted as the value of the difference QD.
  • Fig. 4 is a flowchart of the aforementioned encoding process.
  • the same numbers are assigned to the same processing steps as those in Fig. 2, and a description thereof will be simplified.
  • This embodiment will exemplify a case wherein the representative frame is selected for respective frames.
  • the present invention is not limited to this, and slices or macroblocks may be used as units as in the first embodiment.
  • step SI header information indicating the maximum number of frames to be referred to
  • step SI header information indicating the maximum number of frames to be referred to
  • step S2 it is checked if the encoding process of all frames to be encoded is complete. If frames to be encoded still remain, the flow advances to step S4 to sequentially input frame data to be encoded to the frame memory 19 (step S4). It is then checked which of the intra-frame encoding and inter-frame encoding modes is set (step S101). If the intra-frame encoding mode is set, the flow advances to step S102.
  • the selector 1 outputs an input image from the frame memory 19 to the intra-predictor 2, which applies known intra-encoding to the input image (step S102).
  • step S104 the flow advances to step S104.
  • the motion compensator 14 reads out data of representative frames of respective sections held in the representative frame memory 28 on the basis of the "frame numbers for previously set sections" from the frame memories 9 to 12.
  • the motion compensator 14 generates predicted images of the representative frames corresponding to the frame image to be encoded, and calculates errors (prediction errors) between the predicted images corresponding to the representative frames, and the frame image to be encoded (step S104).
  • step S105 the representative frame with a smallest prediction error from the frame to be encoded is specified, and the start and end point values as information indicating the two ends of the section including the frame number of the specified representative frame are read out from the representative frame memory 28, thus setting the section to be referred to so as to encode the frame to be encoded (step S105).
  • Frame images in the section set in step S105 are read out from the frame memories 19 to 22, and the frame to be encoded undergoes the encoding process using the readout images and the frame to be encoded (step S106) .
  • the processes in steps S104 to S106 are substantially the same as those in steps S13 to 15, except for an encoding unit.
  • the output unit 16 externally outputs this encoded data stored in the buffer 6. Furthermore, as in step S16, the reclaimed image of the encoded frame is stored in a frame memory that does not store any image data or a frame memory that stores the frame of the oldest number of the frame memories 9 to 12 (step S107) .
  • the subsequent processes set information to be referred to so as to encode the next frame, i.e., the position of the representative frame and the section.
  • the representative frame determination unit 118 compares the reclaimed image (reclaimed in step S107) of the frame encoded in step S106 with the image (which is similarly reclaimed and is stored in one of the frame memories 9 to 12) of the frame immediately before this frame to calculate a difference R on the basis of equation (1) (step S108).
  • the unit 118 compares the calculated difference R with the threshold Th (step S109). If the difference R is equal to or larger than the threshold Th, the flow advances to step S110; otherwise, the flow advances to step S112. If the difference R is equal to or larger than the threshold Th, the flow advances to step S110.
  • step S106 Since it is determined that the frame encoded in step S106 belongs to those in the immediately preceding section, i.e., that frame has large, differences from every frames in the immediately preceding section, it is determined that the frame encoded in step S106 belongs to a next new section, and a setting process of the start and end points of the next section, and a representative frame in this section is executed.
  • the section setting unit 125 sets the number of the frame encoded in step S106 as the start and end points of a new section in the section memory 27 (step S110).
  • step Sill the number of this frame is stored in the representative frame memory 28
  • step S113 In order to execute the encoding process of the next frame, the flow returns to step S2 to execute subsequent processes. On the other hand, if the difference R is smaller than the threshold Th in step S109, the flow advances to step S112. A process for adding 1 to the end point currently stored in the section memory 27 so as to include the frame encoded in step S106 in the immediately preceding section is executed (step S112). Next, the section setting unit 125 calculates differences QN and SN according to equation (1) (step S113).
  • the representative frame determination unit 116 calculates a difference QN in accordance with equation (2) using images from the frame indicated by the current start point value to the (N-l)-th frame, and the image of the N-th frame (in this case, "images from the frame indicated by the current start point value to the (N-l)-th frame” correspond to X M (i, j), and the "image of the N-th frame” corresponds to XN(1, j) in equation (2)).
  • a difference SN between the N-th frame image and image data of the frame set as the current representative frame is calculated according to equation (1) .
  • the representative frame determination unit 118 adds the difference SN and a cumulative difference QD (an initial value of this difference QD is zero) of the differences from the representative frame upon encoding up to the (N-l)-th frame to calculate a new cumulative difference QD' (step S114).
  • the updated QD is represented by QD ' .
  • the cumulative difference QD' is compared with the difference QN (step S115). As a result of comparison, if the difference QN is smaller than the cumulative difference QD', the flow advances to step S116 to set the N-th frame (i.e., the frame encoded in step S106) as a new representative frame.
  • the frame number (i.e., N) of the N-th frame is set in the representative frame memory 28 (step S116).
  • the value of the difference QN is substituted as the value of the difference QD (step S117).
  • the flow advances from step S115 to step S118.
  • the value of the difference QD' is substituted as the value of the difference QD (step S118).
  • the flow returns to step S2 to encode the next frame, thus repeating the subsequent processes.
  • Fig. 5 is a block diagram showing the basic arrangement of a moving image encoding apparatus according to this embodiment .
  • reference numeral 300 denotes a CPU which controls the overall apparatus using programs and data loaded onto a memory 301 and executes respective processes to be described later.
  • Reference numeral 301 denotes a memory which comprises an area for temporarily storing programs and data loaded from storage devices 304 and 305, and also a work area used when the CPU 300 executes respective processes.
  • Reference numeral 303 denotes a terminal which is used to input various settings, various instructions, and the like to the moving image encoding apparatus of this embodiment.
  • Reference numeral 304 denotes a storage device which stores software programs for making the CPU 300 execute various processes, and data. Some or all of these software programs are loaded onto the memory 301 under the control of the CPU 300, and are to be processed by the CPU 300.
  • Reference numeral 305 denotes a storage device for storing moving image stream data.
  • the storage devices 304 and 305 are independent devices. However, the present invention is not limited to such specific arrangement. For example, a single storage device may store programs and data stored in the storage devices 304 and 305.
  • the storage devices 304 and 305 may be removable storage media. In this case, a device for reading out programs and data recorded on these recording media must be added to the moving image encoding apparatus of this embodiment .
  • Reference numeral 306 denotes a camera for sensing a moving image. The sensed moving image data (image data of respective frames) is recorded in the storage device 305.
  • Reference numeral 307 denotes a display device which comprises a CRT, liquid crystal display, and displays various images, characters, and the like.
  • Reference numeral 308 denotes a communication I/F (interface).
  • the moving image encoding apparatus of this embodiment makes data communications with external apparatuses via this communication I/F 308 and a network 309 such as a LAN, public network, wireless network, broadcasting wave, or the like.
  • Reference numeral 302 denotes a communication path which includes a bus and network and interconnects the aforementioned units .
  • the process for encoding a moving image, which is sensed by the camera 306 and is recorded in the storage device 305, and outputting the encoded result onto the network 309 via the communication I/F 308, which process is done by the moving image encoding apparatus with the above arrangement, will be described below.
  • the H.264 encoding method will be exemplified as the encoding method.
  • the present invention is not limited to such specific method, and may be applied to any other encoding methods as long as they encode data with reference to a plurality of frames (i.e., two or more frames). For the sake of simplicity, only forward prediction will be explained, but the present invention is not limited to this .
  • intra-frame encoding is done at predetermined intervals, and inter-frame encoding is done at other timings.
  • this instruction is supplied to the CPU 300, which initializes the respective units.
  • the CPU 300 reads out software programs and data such as a software program for making the CPU 300 execute a moving image encoding process (to be described later), a software program for making a communication process (to. be described later) , and the like from the storage device 305, and loads them onto the memory 301.
  • Fig. 6 shows an example of the memory map in the memory 301.
  • Fig.. 6 areas for storing the moving image encoding software and communication software are assured on the memory 301, and an area for storing an OS (operating system) for making the CPU 300 execute processes that control these software programs is also assured.
  • the memory 301 includes image areas (1 to 9; to be described later), a code area, and a working area.
  • Fig. 7 is a flowchart of the process implemented when the CPU 300 executes these moving image encoding software and communication software.
  • the CPU 300 clears image areas and code area on the memory 301, generates header information, and stores it on the code area (step S201). At this time, the CPU 300 assures a table area indicating sections on the working area.
  • Fig. 9 shows the contents (configuration example) of this table.
  • Each record includes a valid flag, section start point value, section end value, and representative frame number.
  • the maximum number of records is equal to the number of image areas for storing reference images .
  • image area 1 since image area 1 stores an input image, reference images for eight frames are stored.
  • F(x), S(x) , E(x) , and D(x) be the. valid flag, section start point value, section end value, and representative frame number of record x. Referring back to Fig. 7, it is checked if the encoding process of all frames is complete (step S202). If frames to be encoded still remains, the flow advances to step S203.
  • step S203 Data for one frame of the moving image sensed by the camera 306 is read out from the storage device 305, and is stored in one image area on the memory 301 (step S203). If the process in step S203 is the first one, data is stored in image area 1. Note that the data of the frame to be stored includes the frame number of this frame. If the frame to be encoded is the one which is to undergo intra-frame encoding, the flow advances to step S210; otherwise, the flow advances to step S205. The frame image stored in the image area in step S203 and image data as the representative frame undergo motion compensation (step S205). If intra-frame encoding is to be made, the CPU 300 performs intra-frame encoding while executing intra-prediction (step S210) .
  • Fig. 8 is a flowchart showing details of . the process in step S205. Initially, zero is substituted in a variable x used to count the number of records , thus resetting the variable x (step S250). Next, whether or not the motion compensation process for all records is complete is checked by determining whether or not the value of the variable x has reached the maximum number of records (step S251). If the motion compensation process is complete, the flow advances to step S206 in Fig. 7; otherwise, the flow advances to step S252.
  • step S254 if F(x) ⁇ 0, i.e., if valid data are stored in the record x, the flow advances from step S252 to step S254 to substitute zero in the variable R(x) so as to reset it (step S254). It is then checked if processes in steps S257 to S259 have been done for all macroblocks that form the frame image stored in the image area in step S203 (step S2.55). If the processes in steps S257 to S259 have been done for all macroblocks that form the frame image stored in the image area in step S203, the flow advances to step S256. On the other hand, if the processes in steps S257 to S259 have not been done yet for all macroblocks that form the frame image stored in the image area in step
  • step S203 the flow advances from step S255 to step S257.
  • a known motion vector search process is executed by reading out frame data of the representative frame number D(x) with reference to the table shown in Fig. 9, and comparing it with the frame image stored in the image area in step S203 (step S257).
  • a prediction error p between the image stored in the image area in step S203 and a predicted image of this image generated based on the motion vector is calculated (step S258).
  • the process for calculating the prediction error is a known technique as in the above embodiments.
  • the calculated prediction error p is added to the variable R(x) (step S259).
  • the flow then returns to step S255 to repeat the processes in steps S257 to S259 for all the macroblocks.
  • the prediction error can be calculated on the basis of the frame image of the representative frame / number D(x) and the image stored in the image area in step S203.
  • the prediction errors from the predicted image of the image stored in the image area in step S203 can be calculated for respective representative frames .
  • the image stored in the image area in step S203 undergoes inter-frame encoding as the known encoding process based on motion compensation with frame images in the section selected in step S206 (step S207).
  • the data of the encoded result is stored in the code area on the memory 301.
  • locally decoded image data is stored in an image area with the oldest frame number.
  • prediction errors calculated upon motion vector search are calculated for respective macroblocks, and their sum total value ⁇ is calculated.
  • the CPU 300 outputs encoded data of the code area oh the memory 301 to the network 309 via the communication interface 308, and then clears the corresponding portion of the code area on the memory 301 (step S208).
  • the CPU 300 changes the end point value of the latest section (immediately before the encoded frame) to the encoded frame number.
  • the flow then returns to step S202 to process the next frame.
  • an increase in prediction errors can be suppressed while reducing the number of frames to be referred to and implementing a faster motion compensation process. Since it is checked based on the motion compensation error calculated upon encoding if the section is to be updated, no new calculations are required, thus minimizing an increase in processing scale.
  • the number of frames to be referred to is eight.
  • the present invention is not limited to such specific value, and various other numbers of frames to be referred can be supported by assuring the required number of image areas and the like.
  • the process according to this embodiment may be implemented using dedicated hardware as in the first and second embodiments.
  • the encoded result is externally output via the communication interface 308.
  • the present invention is not limited to this.
  • the encoded result may be stored in the storage device 305.
  • image data sensed by the camera 306 is temporarily recorded in the storage device 305.
  • the present invention is not limited to this .
  • the sensed image data may be directly stored in the image areas on the memory 301.
  • Fig. 10 is a flowchart showing a moving image encoding process according to this embodiment.
  • a moving image encoding apparatus comprises the same arrangement as that of the third embodiment.
  • the CPU 300 executes the process according to the flowchart shown in Fig. 10.
  • the same step numbers in Fig. 10 are assigned to the same processing steps as those in Fig. 7, and a description thereof will be omitted.
  • the CPU 300 substitutes zero in a variable T(x) used to record a reference count of a section so as to reset it (step S300). It is then checked if the encoding process of all slices is complete (step S301). If slices to be encoded still remain, the flow advances to step S302.
  • the CPU 300 executes the process according to the flowchart shown in Fig.
  • step S206 the CPU 300 sets a section (for respective slices in this embodiment) (step S303).
  • the CPU 300 increments the value of the variable T(x) indicating the reference count to the set section by 1 (step S304).
  • the value of the variable x is that of the record x which stores the section (start and end point values) set in step S303.
  • the CPU 300 executes an encoding process of the slice to be encoded by the same process as in the above embodiment (step S305).
  • step S208 the flow advances to step S208, and the CPU 300 outputs encoded data (step S208) as in the third embodiment.
  • the CPU 300 checks with reference to the values of the variable T(x) and variable x if the value of the variable x that maximizes the value of the variable T(x) is a maximum value (step S306). That is, it is checked if the section which was referred to most frequently is the latest section. If the section which was referred to most frequently is not the latest section, the flow advances to step S212 to execute the processes in steps S212 and S213 as in the third embodiment. On the other hand, if the section which was referred to most frequently is the latest section, the flow advances to step S214. The CPU 300 executes a section update process as in the third embodiment (step S214), and updates the representative frame of the updated section (step S307).
  • step S307 The process in step S307 will be described in more detail below.
  • the start point value S(x) and end point value E(x) are read out from the record of the updated section.
  • D(x) (S(x) + E(x))/2 (3)
  • Fig. 12 is a block diagram showing the arrangement of a moving image encoding apparatus according to this embodiment. In this embodiment, a description will be given based on the H.264 encoding method. However, the present invention is not limited to such specific method.
  • reference numerals 519 to 523 denote frame memories for storing images of frames (input images) which form a moving image.
  • Reference numeral 525 denotes a selector which selects an input source and outputs data.
  • Reference numeral 51 denotes a selector which selects an output destination in accordance with an intra-frame encoding/inter-frame encoding mode; 52, an intra-predictor which performs intra-prediction based on the H.264 encoding method; and 53, a differentiator for obtaining motion prediction errors .
  • Reference numeral 54 denotes a transformer/quantizer which performs integer orthogonal transformation, and quantizes input coefficients; and 57, an inverse quantizer/inverse transformer which performs the reverse process of the transformer/quantizer 54.
  • Reference numeral 58 denotes an adder which adds motion prediction errors.
  • Reference numeral 55 denotes an entropy encoder which encodes the quantization result; and 56, a buffer which stores generated codes for a plurality of frames .
  • Reference numerals 59, 510, 511, and 512 denote frame memories each of which stores locally decoded image data and the frame number for each frame.
  • Reference numeral 513 denotes a motion estimator which extracts an optimal motion vector from the corresponding frame of an input image and decoded image data; and 514, a motion compensator which generates a predicted image on the basis of the motion vector calculated by the motion estimator 513 and information of the corresponding frame.
  • Reference numeral 515 denotes a motion encoder which encodes motion information on the basis of the calculated motion vector and information of the corresponding frame.
  • Reference numeral 517 denotes a frame memory controller which controls inputs/outputs of the frame memories 59 to 512; and 516, an output unit which combines and outputs encoded data and reference information stored in the buffer 56.
  • Reference numeral 518 denotes a representative, frame determination unit which determines a representative frame by comparing input images.
  • Reference numeral 524 denotes a representative frame flag generator for outputting the frame number of the representative frame to the output unit 516 when the representative frame determination unit 518 determines the representative frame.
  • the representative frame flag generator 524 Prior to encoding, the representative frame flag generator 524 generates a code indicating that encoded data of interest adopts a method that refers to a representative frame, and outputs that code to the output unit 516.
  • This code is encoded as user data to maintain compatibility to the H.264 encoding method.
  • the present invention is not limited to this, and such information may be embedded as a digital watermark.
  • the output unit 516 generates and outputs header information which represents the maximum number of frames that can be used in reference. At this time, the number of frames that can be used in reference is 3. Also, the frame memories are reset to zero. Image data of frames input to the apparatus of this embodiment are stored in turn in the frame memories 519 to 523.
  • the representative frame determination unit 518 calculates inter-frame differences R. For the sake of simplicity, the sum total of the absolute values of pixel values between positionally corresponding pixels of respective images is used as the difference R as in the first embodiment.
  • XN(1, j) (i and j indicate the position of a pixel in an image XN) be the pixel value of the N-th frame stored in the frame memory 519
  • X N+ ⁇ (i, j) be the pixel value of the (N+l)-th frame stored in the frame memory 520
  • X N+2 (i, j) be the pixel value of the (N+2)-th frame stored in the frame memory 521
  • X N+3 (i, j) be the pixel value of the (N+3)-th frame stored in the frame memory 522
  • X N+4 (i, j) be the pixel value of the (N+4)-th frame stored in the frame memory 523.
  • R N ⁇ (
  • differences R( N + 2 ), R(N+3), and R(N + ) are calculated.
  • the representative frame determination unit 518 refers to these differences R N to R( N+4 ) and selects a frame with a smallest difference as a representative frame. For example, if R( +2 ) of the differences R N to R( N+4 ) is smallest, the (N+2)-th frame is selected as a representative frame. The representative frame determination unit 518 holds the frame number of the frame with a smallest difference.
  • the selector 525 reads out frame data in the frame memories 519 to 523 in turn, and inputs them to the selector 51.
  • the selector 51 selects the intra-predictor 52 as an output so as to intra-frame encode frames at predetermined intervals (e.g., every 15 frames).
  • the selector 1 selects the differentiator 53 as an output so as to perform inter-frame encoding.
  • input frame data undergoes prediction for respective blocks.
  • each input image data is input to the motion estimator 513 for respective macroblocks.
  • the motion estimator 513, motion compensator 514, and motion encoder 515 perform the same processes as those of the motion estimator 13, motion compensator 14, and motion encoder 15 shown in Fig. 1 to attain an encoding process based on motion compensation for these five frames . Note that an image to be referred to upon making motion compensation for these five frames is generated as follows.
  • predicted images of data of the respective representative frames held in the frame memories 59 to 512 to the frame to be encoded are generated, errors (prediction errors) between the predicted images of the representative frames and the image of the frame to be encoded are calculated, and the representative frame with a smallest prediction error from the frame to be encoded is selected as a reference image.
  • errors prediction errors
  • this process is equivalent to a process for encoding five or more frames at the same time.
  • the representative frame determination unit 518 Upon detection of a representative frame of frame data to be read out from the frame memories 519 to 523 on the basis of the held representative frame number, the representative frame determination unit 518 outputs that frame number to the representative frame flag generator 524.
  • the representative frame flag generator 524 outputs this frame number to the output unit 516. Note that the representative frame flag generator 524 holds the notified latest frame number, and outputs the currently held frame number until it receives a new number. Note that the present invention is not limited to the frame number as long as the representative frame can be specified.
  • the output unit 516 appends "the frame number of the representative frame" from the representative frame flag generator 524 to encoded data for five frames stored in the buffer 56 and externally outputs these data.
  • the present invention is not limited to the frame number as long as the. representative frame can be specified. Since the intra-predictor 52 or differentiator 53 outputs the prediction error for the frame data of interest, the prediction error is supplied to the adder 58 via the transformer/quantizer 54 and inverse quantizer/inverse transformer 57, and is added to a "predicted image corresponding to the frame data of interest" input from the motion compensator 514 (to be described later) by the adder 58.
  • the adder 58 adds this difference to "the predicted image corresponding to the frame data of interest" input from the motion compensator 514, thus reclaiming the frame data of interest.
  • the frame memory controller 517 controls the frame memories 59 to 512 to store the reclaimed frame data when the representative frame determination unit 518 determines a representative frame.
  • Fig. 13 is a flowchart of the aforementioned encoding process according to the present invention.
  • the output unit 516 generates and outputs header information including information indicating that the encoding method to be executed below uses a representative frame, and information indicating the maximum number of frames to be referred to (step S51). If the following encoding process has not been executed for all frames to be encoded (step S52), the flow advances to step S53 to execute the subsequent processes.
  • step S53 the each of frame memories 519 to 523 stores frame data for one frame (step S54). After frame data are stored in the frame memories
  • step S55 The representative frame determination unit 518 calculates differences R N for respective five frames (step S55). Of five differences, a frame with a smallest difference is determined as a representative frame (step S56). In this case, the representative frame determination unit 518 outputs the frame number of the representative frame to the representative frame flag generator 524. In the subsequent processes, these five frame data are encoded. It is checked if the encoding process for all the five frames is complete (step S57). If the encoding process is complete, the flow returns to step S52 to repeat the subsequent processes.
  • step S58 to check if the frame to be encoded is the representative frame. Whether or not the frame to be encoded is the representative frame can be determined by checking if the frame number of the frame to be encoded matches that obtained by the representative frame determination unit 518 (if these numbers match, it is determined that the frame to be encoded is the representative frame) . Hence, this determination process is executed (step S58). If the frame to be encoded is the representative frame, the flow advances to step S59, and the representative frame flag generator 524 outputs the frame number of the representative frame to the output unit 516 (step S59).
  • step S512 the encoding process based on motion compensation is applied to the frame to be encoded.
  • This encoding process is the same as that in the first embodiment, as described above(step s510).
  • the encoded image of the representative frame is decoded as in the first embodiment, and the decoded image is stored in a frame memory that stores the frame of the oldest frame number of the frame memories 59 to 512 (step S511) .
  • step S58 the flow advances from step S58 to step S512 to execute the same encoding process as in step S510 (step S512).
  • the output unit 516 appends "the frame number of the representative frame" obtained from the representative frame flag generator 524 to the encoded data of these frames from the buffer 56, and externally outputs these data (step S513) .
  • frames for a long period of time can be referred to using a fewer number of frames , and the memory size can be suppressed upon encoding. Since a frame having a higher similarity to other frames is selected as the representative frame, the encoding efficiency can be improved. Since user data or the like, which can be ignored upon decoding, is selected as the encoded data of the representative frame, the compatibility to existing H.264 can be maintained.
  • an object to be referred to is a frame.
  • units e.g., slices
  • Intra-frame encoding may be used as the encoding method of the representative frame.
  • a frame with high image quality is used as reference image data, the image quality can be improved.
  • Fig. 14 is a block diagram showing the arrangement of a decoding apparatus according to this embodiment .
  • Reference numeral 5100 denotes an input unit which receives encoded data, and separately outputs it.
  • Reference numeral 5101 denotes an entropy decoder which decodes encoded data to obtain a quantization result.
  • Reference numeral 5102 denotes an inverse quantizer/inverse transformer which inversely quantizes the quantization result and computes the inverse transforms of integer orthogonal transformation.
  • Reference numeral 5103 denotes an intra-predictor which performs intra-prediction based on the H.264 encoding method; and 5104, an adder which adds the motion prediction errors from the inverse quantizer/inverse transformer 5102 to predicted images from a motion compensator 5110.
  • Reference numeral 5113 denotes a frame memory for temporarily storing a decoded image.
  • Reference numerals 5105, 5106, 5107, and 5108 denote frame memories each of which stores decoded image data and its frame number for each frame. In this embodiment, the number of frame memories is four so as to decode the encoded result output from the moving image encoding apparatus according to the fifth embodiment. However, the present invention is not limited to such specific value.
  • Reference numeral 5109 denotes a motion decoder which decodes a motion vector and corresponding frame information; and 5110, a motion compensator which generates a predicted image on the basis of the motion vector and corresponding frame information decoded by the motion decoder 5109.
  • Reference numeral 5111 denotes a representative frame analyzer which notifies a frame memory controller 5112 of "the frame number of the representative frame” with reference to “the frame number of the representative frame” appended to the encoded data received by the input unit 5100; and 5112, a frame memory controller which controls inputs/outputs of the frame memories 5105 to 5108.
  • the input unit 5100 receives header information which includes the number of frames that can be used in code reference as the method that refers to the representatxve frame, and the like, interprets the header information, and initializes the respective units.
  • the input unit 5100 loads encoded data (appended with the frame number of the representative frame of each frame) of each frame, and outputs data of "the frame number of the representative frame” appended to this encoded data to the representative frame analyzer 5111.
  • the input unit 5100 outputs encoded data associated with a motion vector in this encoded data to the motion decoder 5109, and encoded data associated with an image to the entropy decoder 5101. If the encodxng mode of the frame is intra-frame encoding, the entropy decoder 5101 decodes input encoded data to reconstruct a quantization result.
  • the inverse quantizer/inverse transformer 5102 inversely quantizes the quantization result and computes the inverse transforms of integer orthogonal transformation, thus reconstructing image data.
  • the reconstructed image undergoes prediction for respective blocks in the intra-predictor 5103 to obtain a decoded image.
  • the decoded image is output to the frame memory 5113.
  • the intra-predictor 5103 or adder 5104 outputs decoded data (i.e., a decoded image of the representative frame) with the same frame number as that notified from the representative frame analyzer 5111, the frame memory controller 5112 stores this decoded image data in a frame memory which stores the frame of the oldest frame number of the frame memories 5105 to 5108.
  • the frame memory controller 5112 does not store the decoded image in the frame memories 5105 to 5108.
  • the decoded image data stored in the frame memory 5113 is externally output at predetermined timings (e.g., for every predetermined number of frames or at predetermined time intervals) .
  • the motion decoder 5109 decodes a motion vector and reference frame information.
  • the motion compensator 5110 calculates prediction errors with reference to the frame memories 5105 to 5108 accordingly, and outputs them to the adder 5104.
  • the entropy decoder 5101 decodes input encoded data to reconstruct a quantization result .
  • the inverse quantizer/inverse transformer 5102 inversely quantizes the quantization result and computes the inverse transforms of integer orthogonal transformation, thus reconstructing prediction errors .
  • the reconstructed prediction errors are added for respective blocks by the adder 5104 to obtain a reconstructed image, which is stored in the frame memory 5113. In this case, no write access is made to the frame memories 5105 to 5108. Finally, the image data stored in the frame memory 5113 is externally output.
  • the decoding process is a known process for decoding the encoded result of the encoding process based on motion compensation, except for the process for storing the decoded image of the representative frame in each of the frame memories 5105 to 5108, and the image to be referred to which is that of the representative frame.
  • Fig. 15 is a flowchart of the aforementioned decoding process according to this embodiment .
  • the input unit 5100 refers to and analyzes the header information input prior to encoded data (step S5101). Especially, the unit 5100 analyzes and holds the number of frames that can be referred to. It is checked if the decoding process for all frames is complete (step S5102). If frames to be decoded still remain, the flow advances to step S5103.
  • the input unit 5100 inputs encoded data of each frame (step S5103).
  • the input unit 5100 separates the frame number of the representative frame of this frame, which is appended to this encoded data, from the encoded data, and inputs it to the representative frame analyzer 5111.
  • the representative frame analyzer 5111 outputs data of this frame number to the frame memory controller 5112 (step S5104) .
  • the motion decoder 5109, motion compensator 5110, entropy decoder 5101, inverse quantizer/inverse transformer 5102, and intra-predictor 5103 or adder 5104 decode the input encoded data according to the decoding method of the H.264 encoding method (step S5105).
  • Step S511 Data of the decoded result, i.e., decoded image data is output to the frame memory 5113.
  • the frame memory controller 5112 is notified of the frame number of the representative frame of the currently decoded frame in step S5104.
  • the frame memory controller 5112 can determine with reference to the frame number of the currently decoded frame whether or not the currently decoded frame is the representative frame. With this determination process, if the currently decoded frame is the representative frame, the flow advances to step S5107.
  • a frame memory that stores the frame of the oldest frame number of the frame memories 5105 to 5108 is specified (step S5107), and the decoded data image data of the decoded representative frame is stored in the specified frame memory (step S5108).
  • the decoded image data output to the frame memory 5113 is externally output (step S5109).
  • the decoding apparatus according to this embodiment is implemented by hardware having the arrangement shown in Fig. 14.
  • the functions of the respective units shown in Fig. 14 are implemented by software, and a program of this software is installed in a computer such as a PC (personal computer), WS (workstation), or the like. When a CPU of this computer executes this program, the computer executes the aforementioned decoding process.
  • Fig. 16 is a block diagram showing the arrangement of a moving image encoding apparatus according to this embodiment .
  • Reference numeral 5200 denotes a frame memory which holds an image of each frame (input image) that forms a moving image to be encoded.
  • Reference numeral 5202 denotes a frame counter, which is counted up every time one frame is output from the frame memory 5200. That is, the frame counter 5202 counts the number of frames output from the frame memory 5200.
  • Reference numeral 5201 denotes a selector which switches an output destination to the intra-predictor when the remainder obtained when the count value of the frame counter 5202 is divided by an integer M is N ( ⁇ M) . Otherwise, the selector 5201 switches the output destination to the differentiator 53.
  • Reference numeral 5218 denotes a representative frame determination unit which determines a representative frame.
  • Reference numeral 5224 denotes a representative frame flag generator which outputs the frame number of the representative frame to the output unit 516 when the representative frame determination unit 5218 determines the representative frame. Note that the representative frame flag generator 5224 holds the notified latest frame number, and outputs the currently held frame number until it receives a new number.
  • the moving image encoding process to be executed by the moving image encoding apparatus with the above arrangement will be described below.
  • the representative frame flag generator 5224 prior to encoding, the representative frame flag generator 5224 generates a code indicating that encoded data of interest adopts a method that refers to a representative frame, and outputs that code to the output unit 516. Also, the output unit 516 generates and outputs header information which represents the maximum number of frames that can be used in reference.
  • the frame memories are reset to zero.
  • the count value held by the frame counter 5202 is reset to zero.
  • the frame memory 5200 receives image data of each frame, and outputs it to the selector 5201.
  • the frame counter 5202 counts up its count value by one for each output.
  • the representative frame determination unit 5218 always refers to this count value.
  • the unit 5218 determines , as a representative frame, a frame output from the frame memory 5200 when the remainder obtained upon dividing the count value by 5 is 1, and determines as a non-representative frame, a frame output from the frame memory 5200 in other cases.
  • the frame data output from the frame memory 5200 is output to the transformer/quantizer 54 as a prediction error via the intra-predictor 52 or differentiator 53 depending on whether that frame data is the representative or non-representative frame. If this prediction difference is that of the representative frame (when the representative frame determination unit 5218 determines the representative frame), the inverse quantizer/inverse transformer 57 and adder 58 are activated to decode the image of the representative frame, as in the fifth embodiment. In addition, an instruction is issued to the frame memory controller 517 to overwrite the decoded image data and frame number on a frame memory that stores the oldest data of the frame memories 59 to 512.
  • the representative frame flag generator 5224 outputs, to the output unit 516, the frame number of the representative frame which is output from the representative frame determination unit 5218 when it determines the representative frame.
  • the prediction difference is not that of the representative frame (when the representative frame determination unit 5218 determines a non-representative frame), without being activated the inverse quantizer/inverse transformer 57 and adder 58, as in the fifth embodiment, an instruction is issued to the frame memory controller 517 not to store the decoded image in the frame memories 59 to 512.
  • the selector 5201 performs intra-frame encoding at given intervals, input frame data is input to the intra-predictor 52 for respective macroblocks, and undergoes prediction for respective blocks.
  • the transformer/quantizer 54 that result undergoes integer orthogonal transformation and quantization by the transformer/quantizer 54, and the quantization result is input to the entropy encoder 55.
  • the entropy encoder 55 entropy-encodes the input quantization result, and stores encoded data in the buffer 56 for each frame.
  • the inverse quantizer/inverse transformer 57 obtains a decoded image from the input quantization result .
  • the decoded image data and frame number are stored in a free frame memory or a frame memory which stores the image data of the oldest representative frame of the frame memories 59 to 512. In case of the non-representative frame, the decoded image is not stored in the frame memories 59 to 512.
  • the output unit 516 appends the frame number of the representative frame output from the representative frame flag generator 5224 to the encoded data stored in the buffer 56, and outputs that encoded data.
  • inter-frame encoding is executed in the same manner as in the fifth embodiment. That is, the selector 5201 switches the output destination of frame data to the differentiator 53.
  • the frame memory 5200 inputs frame data to the selector 5201 and motion estimator 513 for respective macroblocks.
  • the motion estimator 513 reads out the contents of the frame memories 59 to 512 to calculate a motion vector and frame number.
  • the motion compensator 514 generates a predicted image, which is input to the differentiator 53.
  • the differentiator 53 calculates a prediction error, which is input to the transformer/quantizer 54 that performs integer orthogonal transformation and quantization.
  • the quantization result is input to the entropy encoder 55.
  • the entropy encoder 55 entropy-encodes the input quantization result , and stores encoded data in the buffer 56 for each frame.
  • the output unit 516 appends the frame number of the representative frame output from the representative frame flag generator 5224 to the encoded data stored in the buffer 56, and outputs that encoded data.
  • Fig. 17 is a flowchart of the aforementioned encoding process. Note that the same step numbers in
  • Fig. 17 are assigned to the same processing steps as in Fig. 13, and a description thereof will be omitted.
  • the process in step S51 is executed, and. the counter value held by the frame counter 5200 is reset to zero (step S5200). If the encoding process has not been executed for all frames, the flow advances to step S54 to execute the same process as in step S54 described above so as to output frame data from the frame memory 5200 to the respective units (step S54). Every time frame data for one frame is output from the frame memory 5200, the counter vale held by the frame counter 5202 is counted up by one (step S5201).
  • step S54 If the remainder obtained upon dividing the current count value by M is 1 (i.e., if the frame number of the frame output from the frame memory 5200 in step S54 is (N*M+1)), it is determined that the frame output from the frame memory 5200 in step S54 is a representative frame, and the flow advances to step S5204.
  • the representative frame flag generator 5224 outputs the frame number of this representative frame output from the representative frame determination unit 5218 to the output unit 516 (step S5204).
  • the aforementioned respective units perform intra-frame encoding (intra-encoding) to this representative frame (step S5205).
  • the encoded result is decoded to generate a decoded image, which is stored in a frame memory according to the above conditions of the frame memories 59 to 512 (step S5206).
  • the image data of the representative frame stored in the frame memory in this way is referred to upon encoding subsequent frames to be encoded.
  • step S5203 the frame output from the frame memory 5200 in step S54 is not a representative frame, and the flow advances to step S5203, and the respective units perform inter-frame encoding for this non-representative frame (step S5203). Since the encoded result is output to the buffer 56 independently of whether the frame to be encoded is a representative or non-representative frame, the output unit 516 appends "the frame number of representative frame" output from the representative frame flag generator 5224 to this encoded result, and externally outputs it (step S5206).
  • a moving image encoding apparatus has the same arrangement as in the third embodiment. A case will be explained hereinafter wherein the moving image encoding apparatus according to this embodiment implements a moving image encoding process different from that according to the third embodiment .
  • the moving image encoding apparatus encodes a moving image, which is sensed by the camera 306 and is recorded in the storage device 305, and stores the encoded result in the memory 301 or outputs it onto the network 309 via the communication I/F 308, as in the third embodiment.
  • the H.264 encoding method will be exemplified as the encoding method.
  • the present invention is not limited to such specific method, and may be applied to any other encoding methods as long as they encode data with reference to a plurality of frames (i.e. , two or more frames). For the sake of simplicity, three frames each before and after the frame of interest are referred to. However, the present invention is not limited to this .
  • a software program for making the CPU 300 executes a moving image encoding process (to be described later), a software program for making a communication process (to be described later) , and the like from the storage device 305, and loads them onto the memory 301.
  • Fig. 18 shows an example of the memory map in the memory 301. As shown in Fig. 18, areas for storing the moving image encoding software and communication software are assured on the memory 301, and an area for storing an OS (operating system) for making the CPU 300 execute processes that control these software programs is also assured.
  • the memory 301 includes image areas (1 to 7; to be described later), code areas (1 to 3), and a working area.
  • the reason why the number of code areas is 3 is that the number of frames to be referred to is 3. If another number of frames to be referred to is set, the number of code areas changes accordingly.
  • Fig. 19 is a flowchart of the process implemented when the CPU 300 executes these moving image encoding software and communication software.
  • the CPU 300 generates header information including information indicating that the encoding method to be executed below uses a representative frame, and information indicating the maximum number of frames to be referred to, and stores it on the memory 301 (step S5301) .
  • the encoding process of all frames is determining whether or not complete, when the encoding process of all frames is not complete yet, the flow advances to step S5303 to check if the encoding process for three frames is complete (step S5303). If the encoding process for three frames is not complete yet, the flow advances to step S5304.
  • the CPU 300 reads out data for one frame of the moving image data sensed by the camera 306 from the storage device 305, and stores it in one image area of the memory 301 (step S5304). If the process in step S5304 is the first one, data is stored in image area 1.
  • the frame data input in step S5304 is encoded by one of inter-frame encoding and intra-frame encoding (step S5305).
  • step S5305 the frame data is encoded by inter-frame encoding.
  • the encoded result in step S5305 is stored in one code area (step S5306). If the process in step S5306 is the first one, data is stored in code area 1.
  • the image of the frame encoded in step S5305 is decoded by executing the same process as in the above embodiment, and the decoded result (decoded image) is stored in one of image areas 2 to 4 (step S5307). Note that the frame numbers of frame data stored in respective image areas are referred to, and the decoded image is stored in the image area that stores frame data with the oldest frame number.
  • a distortion value between the image stored in step S5306 and the decoded image obtained by decoding this image in step S5307 is calculated (step S5308).
  • the absolute values of differences of pixel values between positionally corresponding pixels in two images are calculated, and the sum total value of the absolute values of differences for respective pixels is calculated to obtain a distortion value between the images .
  • the method of calculating a distortion value is not particularly limited. In brief, an index indicating a difference between a source image and an image, which is once encoded and then decoded, need only be obtained as a value. The flow returns to step S5303 to repeat the subsequent processes.
  • step S5309 The CPU 300 refers to the distortion values calculated in step S5308 for three encoded frames, and determines a frame with a smallest distortion value as a representative frame which is a frame that represents the three encoded frames (step S5309).
  • the CPU 300 appends the frame number of the determined representative frame to the encoded data stored in the code areas (step S5310).
  • the CPU 300 stores a decoded image of the representative frame (which is stored in one of image areas 2 to 4 and can be uniquely selected by the frame number) in an image area which stores frame data with the oldest frame number of image areas 5 to 7 (step S5311).
  • the CPU 300 By executing the communication software, the CPU 300 outputs encoded data stored in the code areas on the memory 301 to the network 309 via the communication interface 308 (step S5312).
  • the output destination of the encoded data of respective frames in step S5312 is not limited to this.
  • the encoded data may be stored in predetermined areas in the memory 301 or may be saved in the storage device 304, 305, or the like.
  • the flow then returns to step S5302 to encode the next three frames , thus repeating the subsequent processes. If no next three frames are available, the process according to the flowchart shown in Fig. 19 ends .
  • This embodiment will explain a decoding apparatus which decodes the encoded result output from the moving image encoding apparatus according to the eighth embodiment .
  • the decoding apparatus according to this embodiment is an apparatus which has the same arrangement as that of the third embodiment.
  • the decoding apparatus according to this embodiment and the moving image encoding apparatus according to the eighth embodiment may be integrated, and the encoding and decoding processes may be done in a single apparatus.
  • the encoded results must be held in the apparatus (e.g., in predetermined areas in the memory 301) .
  • the decoding apparatus according to this embodiment and the moving image encoding apparatus according to the eighth embodiment may be arranged as independent apparatuses.
  • the encoded results must be output onto the network 309 via the communication interface 308 and must be input to the decoding apparatus according to this embodiment.
  • the essential process to be executed is the same in either of the above cases .
  • Fig. 20 shows an example of the memory map in the memory 301. As shown in Fig.
  • the memory 301 includes image areas (1 to 7; to be described later), a code area, and a working area.
  • image area 1 stores a decoded image
  • image areas 2 to 7 store reference images .
  • images are stored in image areas 2 to 7 in the order named.
  • Fig. 21 is a flowchart of the process implemented when the CPU 300 executes these moving image decoding software and communication software.
  • the CPU 300 refers to and interprets the header information which is input externally or internally (e.g., from the memory) prior to encoded data, and executes various initialization processes (step S5401). It is checked if the decoding process for all frames is complete (step S5402). If the decoding process for all frames is not complete yet, the flow advances to step S5403 to input encoded data of each frame to the code area on the memory 301 (step S5403). The CPU 300 then separates the frame number of. the representative frame of this frame, which is appended to this encoded data, from the encoded data (step S5404). The CPU 300 applies the decoding process to the input encoded data, as described in the sixth embodiment, and stores the decoded image in image area 1 (step S5405).
  • step S5404 whether or not the frame decoded in step S5404 is a representative frame is determined by referring to the frame number written in the working area, i.e., by checking if that frame number matches the frame number separated in step S5404 (step S5406). If the decoded frame is a non-representative frame, the flow advances to step S5407.
  • the CPU 300 specifies an image area which stores frame data with the oldest frame number of image areas 2 to 4 (step S5407), and copies the image stored in image area 1 to the specified area (step S5408).
  • the flow advances to step S5409.
  • the CPU 300 specifies an image area which stores frame data with the oldest frame number of image areas 5 to 7 (step S5409), and copies the image stored in image area 1 to the specified area (step S5410).
  • the CPU 300 outputs the decoded image data stored in image area 1 (step S5411).
  • the output destination is not particularly limited.
  • the decoded image data may be output to the display device 307 and may be displayed.
  • Fig. 23 is a block diagram showing the arrangement of a moving image encoding apparatus according to this embodiment . In this embodiment , a description will be given based on the H.264 encoding method, but temporally discontinuous reference frames are to be referred to unlike in H.264.
  • image data of a frame will also be referred to as frame data.
  • This frame data (an image of a frame) includes data indicating the frame number of this frame.
  • reference numerals 619 to 621 denote frame memories, which store images of representative frames selected by a process to be described later of images of respective frames (input images) of an externally input moving image.
  • Reference numeral 61 denotes a selector which selects an output destination in accordance with the intra-frame encoding or inter-frame encoding mode.
  • Reference numeral 62 denotes an intra-predictor which performs intra-prediction based on the H.264 encoding method; and 63, a differentiator for obtaining motion prediction errors.
  • Reference numeral 64 denotes a transformer/quantizer which performs integer orthogonal transformation, and quantizes input coefficients; and 67, an inverse quantizer/inverse transformer which performs the reverse process of the transformer/quantizer 64.
  • Reference numeral 65 denotes an entropy encoder which encodes the quantization result of the transformer/quantizer 64; and 66, a buffer which stores generated codes for a plurality of frames.
  • Reference numerals 69, 610, 611, and 612 denote frame memories each of which stores locally decoded image data and the frame number for each frame .
  • Reference numeral 613 denotes a motion estimator which extracts an optimal motion vector from the corresponding frame of an input image and decoded image data; and 614, a motion compensator which generates a predicted image on the basis of the motion vector calculated by the motion estimator 613 and information of the corresponding frame.
  • Reference numeral 615 denotes a motion encoder which encodes motion information on the basis of the calculated motion vector and information of the corresponding frame.
  • Reference numeral 617 denotes a frame memory controller which controls inputs/outputs of the frame memories 69 to 612; and 616, an output unit which outputs encoded data stored in the buffer 66.
  • Reference numeral 618 denotes a similarity determination unit which calculates a similarity level by comparing an input image and an image of the representative frame.
  • Reference numeral 622 denotes a representative frame flag generator which outputs the frame number of the representative frame to the output unit 616.
  • the moving image encoding operation in the moving image encoding apparatus with the above •arrangement will be described below.
  • the representative frame flag generator 622 Prior to encoding, the representative frame flag generator 622 generates a code indicating that encoded data of interest adopts a method that refers to a representative frame, and outputs that code to the output unit 616. This code is encoded as user data to maintain compatibility to the H.264 encoding method.
  • the present invention is not limited to this, and such information may be embedded as a digital watermark.
  • the output unit 616 generates and outputs header information which represents the maximum number of frames that can be used in reference. At this time, the number of frames that can be used in reference is 3. Also, the frame memories are reset to zero. Images of respective frames (input images) that form a moving image are sequentially input to the selector 61 and similarity determination unit 618.
  • the similarity determination unit 618 calculates similarity values by comparing the input image and image data of the representative frames stored in the frame memories 619, 620, and 621. In this embodiment, as a calculation method of the similarity level, the absolute values of differences of pixel values between positionally corresponding pixels in two images are calculated, and the sum total value of the absolute values of differences for respective pixels is calculated.
  • the similarity determination unit 618 calculates similarity levels between the input image and the images of the three representative frames .
  • the unit 618 compares a minimum one of the calculated three similarity levels with a predetermined threshold. If the minimum similarity is equal to or larger than the predetermined threshold, the unit 618 determines this input image as an image of a representative frame. On the other hand, if the minimum similarity is smaller than the predetermined threshold, the unit 618 determines this input image as an image of a non-representative frame.
  • the selector 61 switches the output destination of the input image, which is determined as the representative frame by the similarity determination unit 618, to the intra-predictor 62.
  • the selector 61 switches the output destination of the input image, which is determined as the non-representative frame by the similarity determination unit 618, to the differentiator 63. Also, when the similarity determination unit 618 determines that the input image is the image of the representative frame, it stores this input image in a frame memory which stores frame data with the oldest frame number of the frame memories 619 to 621. On the other hand, when the similarity determination unit 618 determines that the input image is the image of the representative .frame, it notifies the representative frame flag generator 622 of the frame number of this input image. The representative frame flag generator 622 outputs the notified frame number to the output unit 616.
  • the representative frame flag generator 622 holds the notified latest frame number, and outputs the currently held frame number until it receives a new number.
  • intra-frame encoding is applied to image data of the representative frame.
  • the input image is input to the intra-predictor 62 for respective macroblocks, and undergoes prediction for respective blocks .
  • the prediction result is input to the transformer/quantizer 64, which performs integer orthogonal transformation and quantizes its coefficients .
  • the quantization result is input to the entropy encoder 65 and inverse quantizer/inverse transformer 67.
  • the entropy encoder 65 entropy-encodes the input quantization result, and stores encoded data in the buffer 66 for respective frames.
  • the inverse quantizer/inverse transformer 67 obtains a decoded image from the input quantization result .
  • the decoded image is stored in a frame memory that stores the representative frame with the oldest frame number of the frame memories 69 to 612 in accordance with an instruction from the frame memory controller 617.
  • the output unit 616 appends "the frame number of the representative frame" from the representative frame flag generator 622 to encoded data stored in the buffer 66 and externally outputs the data.
  • inter-frame encoding is applied to a non-representative frame.
  • the input image data is input to the selector 61 and motion estimator 613 for respective macroblocks.
  • the motion estimator 613 reads out images held in the frame memories 69 to 612, and calculates a motion vector as the motion estimator 13 in the first embodiment.
  • the motion compensator 614 generates a predicted image as the motion compensator 14 in the first embodiment, and inputs it to the differentiator 63. Since the differentiator 63 receives the input image of the non-representative frame from the selector 61, as described above, it calculates a difference between this image and predicted image, i.e., a prediction error.
  • the calculated prediction error is input to the transformer/quantizer 64, which quantizes the prediction error by integer orthogonal transformation. Furthermore, the quantization result is entropy-encoded by the entropy encoder 65.
  • the entropy-encoded result is stored in the buffer 66 for each frame.
  • the motion vector is encoded by the motion encoder 615 and is stored in. the buffer 66 as the motion encoder 15 in the first embodiment.
  • the output unit 616 outputs the encoded data stored in the buffer 66 together with the frame number of the representative frame output from the representative frame flag generator 622.
  • Fig. 24 is a flowchart of the aforementioned moving image encoding process according to this embodiment .
  • the output unit 616 generates and outputs header information including information indicating that the encoding method to be executed below uses a representative frame, and information indicating the maximum number of frames to be referred to (step S61).
  • step S62 It is then checked if the encoding process for all frames to be encoded is complete (step S62). If frames to be encoded still remain, the flow advances to step S63 to sequentially input frame data to be encoded to the similarity determination unit 618 and selector 61 (step S63). As described above, the similarity determination unit 618 calculates similarity levels between the input image and the images of the representative frames stored in the frame memories 619 to 621 (step S64), and checks if the minimum similarity level is equal to or smaller than the predetermined threshold (step S65). If the minimum similarity level is equal to or larger than the predetermined threshold, the flow advances to step S66.
  • the input image is determined as an image of a representative frame, and the intra-frame encoding process is applied to this input image (step S66), as described above.
  • This encoding process includes a process for decoding the encoded input image and storing the decoded image in a frame memory that stores the frame of the oldest frame number of the frame memories 69 to 612, as in the first embodiment.
  • the representative frame flag generator 622 outputs the frame number of this input image to the output device 616 (step S67). Also, the input image input in step S63 in one of the frame memories 619 to 621 (step S68) .
  • the flow advances to step S69.
  • the input image is determined as an image of a non-representative frame, and the inter-frame encoding process is applied to this input image (step S69), as described above. Since the encoded result in either step- S66 or S69 is output to the buffer 66, as described above, the output unit 616 appends the frame number output from the representative frame flag generator 622 to this encoded result (step S611), and externally outputs that result (step S612) . With a series of selection operations described above, frames for a long period of time can be referred to using a fewer number of frames , and the memory size can be suppressed upon encoding.
  • Fig. 25 is a block diagram showing the arrangement of a moving image encoding apparatus according to this embodiment.
  • Reference numeral 6200 denotes a frame memory which holds images of respective frames (input images) that form a moving image to be encoded.
  • Reference numeral 6213 denotes a motion estimator which compares the input image and image data in the frame memories 69 to 612 and outputs a motion vector and reference frame number. Also, the motion estimator 6213 outputs an error at that time.
  • Reference numeral 6202 denotes a similarity determination unit, which receives an error upon motion vector search from the motion estimator 6213, calculates similarity levels (to be described later), and determines if the input image is a representative frame.
  • Reference numeral 6201 denotes a selector which selects the differentiator 63 as a normal output destination.
  • the selector 6201 switches the output destination to the intra-predictor at predetermined intervals after the last timing when the similarity determination unit 6202 determines the representative frame.
  • the representative frame flag generator 622- prior to encoding, the representative frame flag generator 622- generates a code indicating that encoded data of interest adopts a method that refers to a representative frame, and outputs that code to the output unit 616. Furthermore, the output unit 616 generates and outputs header information which represents the maximum number of frames that can be used in reference.
  • the frame memory controller 617 deletes a frame, which is separated apart 15 frames or more from the input image input from the frame memory 6200, of those held by the frame memories 69 to 612. This process is attained by referring to the frame numbers of respective frames held by the frame memories 69 to 612, and that of the input image input from the frame memory 6200, and deleting each frame data having a frame number which is different 15 or more from the frame number of the input image. As a result, the motion estimator 6213 cannot refer to a frame which is separated apart 15 frames or more from the input image input from the frame memory 6200.
  • Image data of each frame is input to the frame memory 6200, and is input to the motion estimator 6213 prior to encoding (to be described later).
  • the motion estimator 6213 calculates motion vectors for respective macroblocks for all images stored in the frame memories 69 to 612.
  • the motion vector is calculated by a block matching method. In this method, a difference in a pixel position where the sum of the absolute values of differences for respective pixels between the corresponding portion of the reference image and input portion becomes minimum is used as a motion vector.
  • the minimum absolute value sum at that time is input to the similarity determination unit 620.2 as an error.
  • the similarity determination unit 6202 calculates the sum total of the absolute values of the generated errors for each frame. This sum total is used as a similarity level.
  • the unit 6202 determines that the input image is a non-representative frame. If the sum total of the absolute values of errors is larger than the predetermined threshold, the unit 6202 determines that the input image is a representative frame. When it is determined that the input image is a representative frame, the similarity determination unit 6202 notifies the representative frame flag generator 622 of the frame number of this input image. The representative frame flag generator 622 outputs the notified frame number to the output unit 616.
  • the image of the representative frame or the input image upon making intra-frame encoding at predetermined intervals as in the 10th embodiment is input to the intra-predictor 62 for respective macroblocks and undergoes prediction for respective blocks.
  • the prediction result undergoes integer orthogonal transformation and quantization by the transformer/quantizer 64, and the quantization result is input to the entropy encoder 65 and inverse quantizer/inverse transformer 67.
  • the entropy encoder 65 entropy-encodes the input quantization result, and stores the encoded result in the buffer 66 for each frame.
  • the inverse quantizer/inverse transformer 67 obtains a decoded image from the input quantization result in the same manner as in the 10th embodiment . Since the frame memory controller 617 receives the determination result of the similarity determination unit 6202, it can determine if the decoded image from the inverse quantizer/inverse transformer 67 is that of the representative frame.
  • the decoded image is that of the representative frame
  • this input image is stored in a frame memory that stores the frame with the oldest frame number of the frame memories 69 to 612.
  • the output unit 616 appends "the frame number" from the representative frame flag generator 622 to encoded data stored in the buffer 66 and externally outputs the data.
  • inter-frame encoding is applied to a non-representative frame
  • the same process as in the 10th embodiment is executed. That is, input image data of a non-representative frame is input to the selector 6201 and motion estimator 6213 for respective macroblocks.
  • the motion estimator 6213 reads out images held in the frame memories 69 to 612 to obtain motion vectors as in the 10th embodiment.
  • the motion compensator 614 generates predicted images as in the 10th embodiment and inputs them to the differentiator 63. Since the differentiator 63 receives the input image of the non-representative frame from the selector 6201, as described above, it calculates a difference between this image and predicted image, i.e., a prediction error. The calculated prediction error is input to the transformer/quantizer 64, which quantizes the prediction error by integer orthogonal transformation. Furthermore, the quantization result is entropy-encoded by the entropy encoder 65. The entropy-encoded result is stored in the buffer 66 for each frame. Also, the motion vector is encoded by the motion encoder 615 and is stored in the buffer 66 as in the 10th embodiment.
  • Fig. 26 is a flowchart of the aforementioned moving image encoding process according to this embodiment. Note that the same step numbers in Fig. 26 are assigned to the same processing steps as in Fig. 24, and a description thereof will be omitted.
  • step S6200 the frame memory controller 617 deletes a frame, which is separated apart 15 or more frames from the input image input from the frame memory 6200, of those which are held by the frame memories 69 to 612, as described above.
  • step S6201 substantially the same process as in step S68 in Fig. 24 is executed except that the storage destination is different.
  • the image of the representative frame is stored in the frame memories 69 to 612 in step S6201.
  • the frame memory size on the encoding side can be reduced by referring to locally decoded images, and size and cost reductions of the circuit can be achieved.
  • the similarity calculation uses an error after motion compensation, a similarity level can be determined in consideration of image motion, thus improving the precision. Since the representative frame undergoes intra-frame encoding, deterioration of the image quality due to propagation of errors can be prevented, and can be minimized.
  • a moving image encoding apparatus has the same arrangement as in the third embodiment . A case will be explained hereinafter wherein the moving image encoding apparatus according to this embodiment implements a moving image encoding process different from that according to the third embodiment .
  • the moving image encoding apparatus encodes a moving image, which is sensed by the camera 306 and is recorded in the storage device 305, and stores the encoded result in the memory 301 or outputs it onto the network 309 via the communication I/F 308, as in the third embodiment.
  • the H.264 encoding method will be exemplified as the encoding method.
  • the present invention is not limited to such specific method, and may be applied to any other encoding methods as long as they encode data with reference to a plurality of frames -(i.e. , two or more frames). For the sake of simplicity, three frames each before and after the frame of interest are referred to. However, the present invention is not limited to this .
  • a keyboard or mouse not shown
  • this instruction is supplied to the CPU 300, which initializes the respective units.
  • the CPU 300 reads out software programs and data such as a software program for making the CPU 300 execute a moving image encoding process (to be described later), a software program for making a communication process (to be described later) , and the like from the storage device 305, and loads them onto the memory 301.
  • a software program for making the CPU 300 execute a moving image encoding process (to be described later)
  • a software program for making a communication process to be described later
  • Fig. 27 is a flowchart of the process implemented when the CPU 300 executes these moving image encoding software and communication software.
  • header information including information indicating that the encoding method to be executed below uses a representative frame, and information indicating the maximum number of frames to be referred to is generated, and is held in the memory 301 (step S6301) .
  • the encoding process of all frames is determining whether or not complete, when the encoding process of all frames is not complete yet, the flow advances to step S6303.
  • the CPU 300 reads out data for one frame of the moving image data sensed by the camera 306 from the storage device 305, and stores it in image area 1 of the memory 301 (step S6303).
  • the CPU 300 calculates the similarity levels of respective images stored in image areas 4 to 7 with the image stored in image area 1 (step S6304).
  • the method of calculating the similarity levels is not particularly limited.
  • the absolute values of differences of pixel values between positionally corresponding pixels in two images are calculated, and the sum total value of the absolute values of differences for respective pixels is calculated, thus obtaining a similarity level between the two images . It is checked if the value of the minimum similarity level of those which are calculated for the images stored in image areas 4 to 7 is equal to or smaller than a predetermined threshold (step S6305). If the value of the minimum similarity level is equal to or smaller than the predetermined threshold, the flow advances to step S6306. It is determined that the image stored in image area 1 is that of the representative frame, and the frame number of this representative frame is output (stored), to a predetermined area (e.g., working area) in the memory 301 (step S6306).
  • a predetermined area e.g., working area
  • step S6307 intra-frame encoding is applied to the image of this representative frame as in the above embodiment, and the encoded result is stored in the code area (step S6307).
  • the encoded frame image is decoded by executing the same process as in the above embodiment , and the decoded result (decoded image) is stored in one of image area 5 and subsequent image areas (step S6308).
  • the frame numbers of frame data stored in respective image areas are referred to, and the decoded image is stored in the image area that stores frame data with the oldest frame number. If the value of the minimum similarity level is equal to or larger than the predetermined threshold, the flow advances to step S6309. It is determined that the image stored in image area 1 is that of a non-representative frame.
  • intra-frame or inter-frame encoding is applied to this image of the non-representative frame, and the encoded result is stored in the code area (step S6309).
  • the image of the encoded frame is decoded by executing the same process as in the above embodiment, and the decoded result (decoded image) is stored in one of image areas 2 to 4 (step S6310). Note that the frame numbers of frame data stored in respective image areas are referred to, and the decoded image is stored in the image area that stores frame data with the oldest frame number.
  • the CPU 300 By executing the communication software, the CPU 300 appends the frame number stored in step S6306 to encoded data of respective frames stored in the code areas on the memory 301 and outputs them to the network 309 via the communication interface 308 (step S6311).
  • the output destination of the encoded data of respective frames in step S6311 is not limited to this.
  • the encoded data may be stored in predetermined areas in the memory 301 or may be saved in the storage device 304, 305, or the like.
  • the flow then returns to step S6302 to encode the next frame, thus repeating the subsequent processes. If no next frame is available, the process according to the flowchart shown in Fig. 27 ends.
  • This embodiment relates to a decoding apparatus which decodes the encoded result output from the moving image encoding apparatus according to the 12th embodiment.
  • the decoding apparatus according to this embodiment is an apparatus which has the same arrangement as that of the third embodiment .
  • the decoding apparatus according to this embodiment and the moving image encoding apparatus according to the 12th embodiment may be integrated, and the encoding and decoding processes may be done in a single apparatus.
  • the encoded results must be held in the apparatus (e.g., in predetermined areas in the memory 301).
  • the decoding apparatus according to this embodiment and the moving image encoding apparatus according to the 12th embodiment may be arranged as independent apparatuses.
  • the encoded results must be output onto the network 309 via the communication interface 308 and must be input to the decoding apparatus according to this embodiment.
  • the essential process to be executed is the same in either of the above cases .
  • Fig. 28 is a flowchart of the process implemented when the CPU 300 executes these moving image decoding software and communication software.
  • the CPU 300 refers to and interprets the header information which is input externally or internally (e.g., from the memory) prior to encoded data, and executes various initialization processes (step S6401). It is checked if the decoding process for all frames is complete (step S6402). If the decoding process for all frames is not complete yet, the flow advances to step S6403 to input encoded data of each frame to the code area on the memory 301 (step S6403).
  • the CPU 300 then separates the frame number of the representative frame of this frame, which is appended to this encoded data, from the encoded data (step S6404).
  • the CPU 300 applies the decoding process to the input encoded data, as described in the sixth embodiment, and stores the decoded image in image area 1 (step S6405).
  • the CPU 300 decodes the data with reference to reference images stored in image areas 2 to 7. Also, the CPU 300 stores the frame number of the decoded frame in the working area.
  • step S6404 whether or not the frame decoded in step S6404 is a representative frame is determined by referring to the frame number written in the working area, i.e., by checking if that frame number matches the frame number separated in step S6404 (step S6406). If the decoded frame is a non-representative frame, the flow advances to step S6407.
  • the CPU 300 specifies an image area which stores frame data with the oldest frame number of image areas 2 to 4 (step S6407), and copies the image stored in image area 1 to the specified area (step S6408).
  • step S6409 the flow advances to step S6409.
  • the CPU 300 specifies an image area which stores frame data with the oldest frame number of image areas 5 to 7 (step S6409), and copies the image stored in image area 1 to the specified area (step S6410). Then, the CPU 300 outputs the decoded image data stored in image area 1 (step S6411).
  • the output destination is not particularly limited.
  • the decoded image data may be output to the display device 307 and may be displayed.
  • the flow returns to step S6402 to repeat the subsequent processes. With a series of selection operations, a temporally farther frame is held as a representative frame, and both temporally closer frames are referred to, thus obtaining a reconstructed image with a smaller information volume.
  • the objects of the present invention are also achieved by supplying a recording medium (or storage medium) , which records a program code of a software program that can implement the functions of the above-mentioned embodiments to the system or apparatus, and reading out and executing the program code stored in the recording medium by a computer (or a CPU or MPU) of the system or apparatus.
  • a recording medium or storage medium
  • the program code itself read out from the recording medium implements the functions of the above-mentioned embodiments, and the recording medium which stores the program code constitutes the present invention.
  • the functions of the above-mentioned embodiments may be implemented not only by executing the readout program code by the computer but also by some or all of actual processing operations executed by an operating system (OS) running on the computer on the basis of an instruction of the program code. Furthermore, the functions of the above-mentioned embodiments may be implemented by some or all of actual processing operations executed by a CPU or the like arranged in a function extension card or a function extension unit, which is inserted in or connected to the computer, after the program code read out from the recording medium is written in a memory of the extension card or unit. When the present invention is applied to the recording medium, that recording medium stores program codes corresponding to the aforementioned flowcharts . As many apparently widely different embodiments of the present invention can be made without departing from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments thereof except as defined in the claims .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
PCT/JP2005/004721 2004-03-11 2005-03-10 Encoding apparatus, encoding method, decoding apparatus, and decoding method Ceased WO2005088981A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10/586,352 US8064518B2 (en) 2004-03-11 2005-03-10 Encoding apparatus, encoding method, decoding apparatus, and decoding method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2004069486A JP4612797B2 (ja) 2004-03-11 2004-03-11 符号化装置、符号化方法
JP2004-069486 2004-03-11

Publications (1)

Publication Number Publication Date
WO2005088981A1 true WO2005088981A1 (en) 2005-09-22

Family

ID=34964036

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2005/004721 Ceased WO2005088981A1 (en) 2004-03-11 2005-03-10 Encoding apparatus, encoding method, decoding apparatus, and decoding method

Country Status (3)

Country Link
US (1) US8064518B2 (https=)
JP (1) JP4612797B2 (https=)
WO (1) WO2005088981A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8687694B2 (en) 2005-10-05 2014-04-01 Panasonic Corporation Reference picture selection method and apparatus
CN106454005A (zh) * 2015-08-11 2017-02-22 三星电子株式会社 电子装置和电子装置的图像编码方法

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4591657B2 (ja) * 2003-12-22 2010-12-01 キヤノン株式会社 動画像符号化装置及びその制御方法、プログラム
JP2005295505A (ja) * 2004-03-12 2005-10-20 Canon Inc 動画像符号化装置及び動画像復号装置及びそれらの制御方法、並びに、コンピュータプログラム及びコンピュータ可読記憶媒体
JP4702928B2 (ja) * 2004-03-12 2011-06-15 キヤノン株式会社 動画像符号化装置及び復号装置及びその制御方法、並びにコンピュータプログラム及びコンピュータ可読記憶媒体
KR101215614B1 (ko) * 2005-09-09 2012-12-26 삼성전자주식회사 영상의 부호화 및 복호화 장치와, 그 방법, 및 이를수행하기 위한 프로그램이 기록된 기록 매체
FR2894759A1 (fr) * 2005-12-12 2007-06-15 Nextamp Sa Procede et dispositif de tatouage sur flux
JP4677351B2 (ja) * 2006-02-17 2011-04-27 キヤノン株式会社 動き補償器、動き補償処理方法、コンピュータプログラム及び記憶媒体
JP4799504B2 (ja) * 2006-08-10 2011-10-26 キヤノン株式会社 画像復号化装置及びその制御方法
CN101502096B (zh) 2006-08-10 2013-06-26 佳能株式会社 图像解码设备
KR100834443B1 (ko) * 2007-02-27 2008-06-04 삼성전자주식회사 비디오 코덱을 위한 메모리 구조 및 메모리 액세스 방법
US8358703B2 (en) 2007-03-23 2013-01-22 Thomson Licensing Modifying a coded bitstream
WO2008154041A1 (en) * 2007-06-14 2008-12-18 Thomson Licensing Modifying a coded bitstream
JP2010004142A (ja) * 2008-06-18 2010-01-07 Hitachi Kokusai Electric Inc 動画像符号化装置、復号化装置、符号化方法及び復号化方法
CN102473298B (zh) 2009-07-01 2015-08-19 日本电气株式会社 提取代表特征的系统和方法
KR101328826B1 (ko) * 2009-11-24 2013-11-13 엘지디스플레이 주식회사 액정표시장치와 그 로컬 디밍 방법
US9747944B2 (en) * 2015-03-31 2017-08-29 Bose Corporation Zeotrope animation disc assembly
KR102833301B1 (ko) * 2019-05-10 2025-07-18 한국전자통신연구원 영상 인식 장치, 영상 인식 장치의 동작 방법, 그리고 영상 인식 장치를 포함하는 컴퓨팅 장치

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0658057A2 (en) * 1993-12-13 1995-06-14 Sharp Kabushiki Kaisha Moving image coder
WO2001026379A1 (en) * 1999-10-07 2001-04-12 World Multicast.Com, Inc. Self adapting frame intervals

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4974071A (en) * 1987-04-28 1990-11-27 Canon Kabushiki Kaisha Color image data encoding apparatus having improved resolution/efficiency characteristics
JPH02177766A (ja) * 1988-12-28 1990-07-10 Canon Inc ２値画像の階層的符号化方式
US5521717A (en) * 1992-02-03 1996-05-28 Canon Kabushiki Kaisha Image processing apparatus and method therefor
US6486981B1 (en) * 1993-07-27 2002-11-26 Canon Kabushiki Kaisha Color image processing method and apparatus thereof
JP3376129B2 (ja) * 1993-12-27 2003-02-10 キヤノン株式会社 画像処理装置及びその方法
JP3408094B2 (ja) * 1997-02-05 2003-05-19 キヤノン株式会社 画像処理装置及びその方法
JP4261630B2 (ja) * 1998-02-04 2009-04-30 キヤノン株式会社 画像符号化装置及び方法、画像符号化プログラムが記録されたコンピュータ可読記録媒体
DE69937816T2 (de) * 1998-04-28 2008-12-24 Canon K.K. Datenverarbeitungsgerät und Verfahren
US6643382B1 (en) * 1998-08-31 2003-11-04 Canon Kabushiki Kaisha Image processing method, image processing apparatus, image processing system and memory medium
SG75189A1 (en) * 1998-12-04 2000-09-19 Canon Kk Image processing apparatus method therefor and recording medium storing image processing program
US6342904B1 (en) * 1998-12-17 2002-01-29 Newstakes, Inc. Creating a slide presentation from full motion video
US6782049B1 (en) * 1999-01-29 2004-08-24 Hewlett-Packard Development Company, L.P. System for selecting a keyframe to represent a video
US6968088B2 (en) * 2000-03-28 2005-11-22 Canon Kabushiki Kaisha Modification of detected quantization step size from the encoded bitstream based on a region of interest (ROI) bitmask
JP3895145B2 (ja) * 2001-10-15 2007-03-22 日本電信電話株式会社 画像符号化装置、画像復号装置並びにその方法及び画像符号化プログラム、画像復号プログラム
US7162101B2 (en) * 2001-11-15 2007-01-09 Canon Kabushiki Kaisha Image processing apparatus and method
US20050028213A1 (en) * 2003-07-31 2005-02-03 International Business Machines Corporation System and method for user-friendly fast forward and backward preview of video
JP4591657B2 (ja) * 2003-12-22 2010-12-01 キヤノン株式会社 動画像符号化装置及びその制御方法、プログラム
JP4702928B2 (ja) * 2004-03-12 2011-06-15 キヤノン株式会社 動画像符号化装置及び復号装置及びその制御方法、並びにコンピュータプログラム及びコンピュータ可読記憶媒体
US7529417B2 (en) * 2004-07-09 2009-05-05 Canon Kabushiki Kaisha Apparatus, method and storage medium for image encoding/decoding using shape-based coefficient interpolation

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0658057A2 (en) * 1993-12-13 1995-06-14 Sharp Kabushiki Kaisha Moving image coder
WO2001026379A1 (en) * 1999-10-07 2001-04-12 World Multicast.Com, Inc. Self adapting frame intervals

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
CHANG A ET AL: "A NOVEL APPROACH TO FAST MULTI-FRAME SELECTION FOR H.264 VIDEO CODING", ISCAS 2003. PROCEEDINGS OF THE 2003 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS. BANGKOK, THAILAND, MAY 25 - 28, 2003, IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS, NEW YORK, NY : IEEE, US, vol. VOL. 2 OF 5, 25 May 2003 (2003-05-25), pages II - 704, XP001168642, ISBN: 0-7803-7761-3 *
SIKORA T: "MPEG STANDARDS SECTION 1 DIGITAL VIDEO CODING STANDARDS", DIGITAL CONSUMER ELECTRONICS HANDBOOK, 1997, pages 803 - 823,I, XP001059402 *
TAMBANKAR A ET AL: "An overview of H.264 / MPEG-4 part 10", VIDEO/IMAGE PROCESSING AND MULTIMEDIA COMMUNICATIONS, 2003. 4TH EURASIP CONFERENCE FOCUSED ON 2-5 JULY 2003, PISCATAWAY, NJ, USA,IEEE, 2 July 2003 (2003-07-02), pages 1 - 51, XP010650106, ISBN: 953-184-054-7 *
TUDOR P N: "TUTORIAL MPEG-2 VIDEO COMPRESSION", ELECTRONICS AND COMMUNICATION ENGINEERING JOURNAL, INSTITUTION OF ELECTRICAL ENGINEERS, LONDON, GB, vol. 7, no. 6, December 1995 (1995-12-01), pages 257 - 264, XP000545121, ISSN: 0954-0695 *
YI J LIANG ET AL: "Low-latency video transmission over lossy packet networks using rate-distortion optimized reference picture selection", PROCEEDINGS 2002 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING. ICIP 2002. ROCHESTER, NY, SEPT. 22 - 25, 2002, INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, NEW YORK, NY : IEEE, US, vol. VOL. 2 OF 3, 22 September 2002 (2002-09-22), pages 181 - 184, XP010607938, ISBN: 0-7803-7622-6 *
ZHANG X-D ET AL: "Dynamic selection and effective compression of key frames for video abstraction", PATTERN RECOGNITION LETTERS, NORTH-HOLLAND PUBL. AMSTERDAM, NL, vol. 24, no. 9-10, June 2003 (2003-06-01), pages 1533 - 1542, XP004410690, ISSN: 0167-8655 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8687694B2 (en) 2005-10-05 2014-04-01 Panasonic Corporation Reference picture selection method and apparatus
CN106454005A (zh) * 2015-08-11 2017-02-22 三星电子株式会社 电子装置和电子装置的图像编码方法

Also Published As

Publication number Publication date
JP4612797B2 (ja) 2011-01-12
JP2005260588A (ja) 2005-09-22
US20070110158A1 (en) 2007-05-17
US8064518B2 (en) 2011-11-22

Similar Documents

Publication Publication Date Title
US6532306B1 (en) Image predictive coding method
WO2005088981A1 (en) Encoding apparatus, encoding method, decoding apparatus, and decoding method
EP1528813B1 (en) Improved video coding using adaptive coding of block parameters for coded/uncoded blocks
US8213501B2 (en) Video encoding method, video decoding method, video encoding program, video decoding program, video encoding apparatus, and video decoding apparatus
US8619859B2 (en) Motion estimation apparatus and method and image encoding apparatus and method employing the same
KR100334703B1 (ko) 디지탈화상복호장치
JP5469193B2 (ja) 対象ブロックの処理方法、そのシステム、コンピュータ読取可能媒体
CN101779463B (zh) 用于处理图像的方法以及相应的电子设备
JP2000278688A (ja) 動きベクトル検出装置およびその方法と画像処理装置
JP3031152B2 (ja) 動き予測プロセッサ及び動き予測装置
JP2025530109A (ja) ビデオ符号化方法、エンコーダ、及びデコーダ
JP2002204459A (ja) 動画像符号化方法及び装置
JPH06217273A (ja) 画像データ符号化方法および装置
JPH07143487A (ja) 画像符号化装置
US20250330593A1 (en) Video decoding method, video encoding method, decoder, encoder and computer-readable storage medium
EP4099698A1 (en) Method, apparatus and device for constructing motion information list in video coding and decoding
JP2002010269A (ja) 動きベクトル検出方法及び、動画像符号化装置
JP2002232892A (ja) 画像符号化装置
CN117459731A (zh) 图像编码方法、编码器和计算机可读存储介质
JP2008072608A (ja) 画像符号化装置及び画像符号化方法
JP2012178669A (ja) 動画像符号化装置、動画像符号化方法及び動画像符号化プログラム
CN111107353A (zh) 视频压缩方法及视频压缩器
JPH0628444B2 (ja) 画像符号化方式
JPH05252503A (ja) 高能率符号化装置

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
WWE Wipo information: entry into national phase

Ref document number: 2007110158

Country of ref document: US

Ref document number: 10586352

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Country of ref document: DE

WWP Wipo information: published in national office

Ref document number: 10586352

Country of ref document: US

122 Ep: pct application non-entry in european phase