JP2010081241A - Device and method for encoding moving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently perform interframe prediction using motion compensation with a small computational quantity in the use of prediction among a plurality of reference frames. <P>SOLUTION: Face detection is performed on the image frame of moving image data, face parts such as eyes, a nose and a mouth are detected further from a detected face, and a state is determined for each of the detected face parts. The determination result of the states of the face parts is preserved in association with information indicating the image frame on which the face detection is performed. Then, the states of the face parts detected from the image frame of an encoding object are compared with the preserved states of the face parts, and an image frame corresponding to the matching state of the face parts is retrieved and used as the reference frame of the face parts. Then, each of the face parts is prediction encoded by a motion compensation block unit using the reference frame retrieved for each of the face parts. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a moving image encoding device and a moving image encoding method, and more particularly to a moving image encoding device and a moving image encoding method for encoding moving image data using interframe prediction based on motion compensation.

  In recent years, the resolution of moving image data has been increased, and a video of 1920 pixels × 1080 pixels called a full high-definition video is used in digital terrestrial broadcasting, for example, in comparison with a conventionally used video of 720 pixels × 480 pixels. Is increasing. Since such high-resolution moving image data has a huge amount of data transmitted per unit time, a highly efficient compression coding technique is required for the conventional technique.

  In response to these demands, standardization of an encoding / compression method using inter-frame prediction using correlation between images is underway by activities of ITU-T SG16 and ISO / IEC JTC1 / SC29 / WG11. Among these, H.264 is an encoding method that is said to realize the most efficient encoding at present. H.264 / MPEG-4 PART10 (AVC) (hereinafter referred to as H.264). H. H.264 encoding and decoding specifications are described in, for example, Patent Document 1.

  This H. One of the techniques newly introduced in H.264 is a technique for selecting a reference image used for inter-frame prediction from a plurality of images (hereinafter referred to as “multi-reference inter-frame prediction”). H. In the H.264 system, error accumulation is suppressed by performing orthogonal transform processing using Hadamard transform and integer precision DCT, compared to the conventional MPEG-1 and MPEG-2 systems. At the same time, intra-frame predictive coding and inter-frame predictive coding using motion compensation are performed to realize more accurate predictive coding.

  In a conventional encoding method such as the MPEG-1 method or the MPEG-2 method (hereinafter referred to as an MPEG encoding method), forward prediction and backward prediction can be used when performing motion prediction. Here, the forward prediction refers to a prediction method for predicting an image frame positioned later in time from an image frame positioned earlier in time. The backward prediction is a prediction method for predicting an image frame positioned temporally earlier from an image frame positioned temporally later. In the backward prediction, it is possible to predict an image frame that skips the previous encoding based on the current image frame. Reverse prediction is used in conjunction with forward prediction (called bi-directional prediction) to achieve a higher compression rate for the image frame to be encoded.

  In the MPEG encoding method, an image frame encoded by forward prediction is called a P picture, and an image frame encoded by bidirectional prediction is called a B picture. In addition, an image picture in which encoding is completed with only the image frame without using inter-frame prediction is called an I picture.

In the forward prediction and bidirectional prediction in the MPEG encoding method, a reference frame to be referred to when performing motion prediction is determined in advance for an image frame to be processed. As an example, when encoding is performed in units of GOP composed of one I picture, four P pictures, and ten B pictures, each picture is encoded when the subscripts are in the order of input (display) of image frames. The order is as follows.
_{I 3 B 1 B 2 P 6} B 4 B 5 P 9 B 7 B 8 P 12 B 10 B 11 P 15 B 13 B 14

In this case, the P ₆ picture is predictively encoded using the I ₃ picture as a reference frame, and the B ₄ and B ₅ pictures are predictively encoded using the I ₃ picture and the P ₆ picture as a reference frame. Similarly, the P ₉ picture is predictively encoded using the P ₆ picture as a reference frame, and the B ₇ and B ₈ pictures are predictively encoded using the P ₆ and P ₉ pictures as a reference frame.

  In forward prediction and backward prediction in this MPEG encoding method, an image frame that is temporally adjacent to an image frame to be processed is used as a reference frame that is referred to when performing motion prediction. Many. For example, as described above, the P picture is predictively encoded using the immediately preceding I picture or P picture as a reference frame. In addition, the B picture is subjected to predictive coding using the immediately preceding and immediately following I and P pictures or the immediately preceding and immediately following P pictures as reference frames. This is because the image correlation between the image frame to be processed and the image frame positioned in the vicinity in time is high in many cases.

  However, in these MPEG encoding schemes, if there is a sudden change in image between image frames, there is a possibility that the advantage of inter-frame prediction using motion compensation cannot be utilized. This is because, when there is an abrupt change in the image, the correlation with the image of the image frame to be encoded becomes low even in an image frame located in the vicinity in time. For example, when shooting a moving image that captures the facial expression of a person, if the subject person blinks his eyes or laughs suddenly and opens his mouth suddenly, the image will change in a short time and motion compensation will be used. The advantage of the inter-frame prediction cannot be utilized, and the compression efficiency may be reduced.

  H. H.264 addresses this problem by introducing multi-reference inter-frame prediction, which performs inter-frame prediction using a plurality of reference frames for one encoding target image frame. According to the prediction between multiple reference frames, a reference frame can be flexibly selected for each block with respect to an image frame to be processed. For example, in the case of a P picture, it can be traced back to a maximum of 15 P pictures, and an optimal picture can be selected for each motion compensation block and used as a reference frame.

In this way, H.C. In H.264, an image that minimizes an error between an input image and an already encoded image is selected from a plurality of images and used as a reference frame, so that inter-frame prediction using motion compensation is performed. It can be performed. As a result, when compressing and encoding moving image data, the correlation between images between the image frame to be encoded and the reference frame at a position close in time to the image frame as described above is low. However, efficient encoding is possible.
JP 2005-167720 A

  However, if a calculation for selecting an image frame that minimizes an error from the input image frame is always performed for a plurality of already encoded image frames, the amount of calculation increases in proportion to the number of image frames to be referenced. Resulting in. As a result, there is a problem that the time required for encoding becomes enormous. In particular, in a device such as a digital video camera that requires encoding in real time for shooting, there is a risk that the calculation may not be in time.

  In addition, in the case of a device designed for portable use such as a digital video camera, an increase in calculation load leads to an increase in the amount of battery to be driven, so that the influence on the shooting time cannot be ignored. was there.

  As described above, conventionally, in the case of using the multi-reference inter-frame prediction, it has been difficult to efficiently perform the inter-frame prediction using the motion compensation with a small amount of calculation.

  The present invention is a moving image that can efficiently execute inter-frame prediction using motion compensation with a small amount of computation, particularly when moving image data including a human face is encoded using inter-reference inter-frame prediction. An object is to provide an image encoding device and a moving image encoding method.

  In order to solve the above-described problem, the present invention performs motion-compensated inter-frame predictive encoding using a result of motion detection performed on a reference frame in units of blocks obtained by dividing an encoding target image frame. A moving image encoding apparatus capable of selecting a frame from a plurality of reference frame candidates, wherein the input image frame storing unit temporarily stores the input image frame, and the image frame stored in the input image frame storing unit A reference candidate frame storage unit that stores a plurality of reference candidate frames generated based on the image data, a face region detected from the image frame output from the input image frame storage unit, and further detects facial parts included in the face region, The determination means for determining the state of the part, and the state of the face part determined by the determination means, the information indicating the image frame including the face part The face part information storage means stored in association with the face parts information storage means, the face parts state detected and determined by the determination means from the encoding target frame output from the input image frame storage means match, and A search unit that searches for information indicating an image frame that can be referred to by the encoding target frame; and information that indicates an image frame searched by the search unit among a plurality of reference candidate frames stored in the reference candidate frame storage unit. A reference frame determining unit that determines a corresponding reference candidate frame as a reference frame of a block corresponding to the face part, and using the reference frame determined by the reference frame determining unit, between motion compensation frames for the encoding target frame A video encoding apparatus that performs predictive encoding.

  In addition, the present invention performs motion compensation inter-frame predictive encoding using the result of motion detection performed on the reference frame in units of blocks obtained by dividing the image frame to be encoded, and converts the reference frame into a plurality of reference frames. A video encoding method that can be selected from candidates, an input image frame storing step for temporarily storing an input image frame in the input image frame storing means, and an image frame stored in the input image frame storing means A reference candidate frame storing step for storing a plurality of reference candidate frames generated based on the reference candidate frame storing unit; and a face part included in the face region by detecting a face region from the image frame output from the input image frame storing unit. Further, a determination step for detecting and determining the state of the face part, and the state of the face part determined in the determination step Is stored in the face part information storage unit in association with information indicating the image frame including the face part, and the encoding target frame output from the input image frame storage unit from the face part information storage unit A search step for searching for information indicating an image frame in which the states of the face parts detected and determined in the determination step match and the encoding target frame can be referred to, and a plurality of reference candidate frame storage means A reference frame determination step for determining a reference candidate frame corresponding to information indicating the image frame searched in the search step among the reference candidate frames as a reference frame of a block corresponding to the face part, and the reference frame determination step A motion compensation frame for the encoding target frame using the reference frame determined in step A moving picture coding method and performing predictive coding.

  Since the present invention has the above-described configuration, particularly when moving image data including a human face is encoded using multiple reference inter-frame prediction, inter-frame prediction using motion compensation can be performed with a small amount of computation, and Can be executed efficiently.

  Hereinafter, embodiments of the present invention will be described. In the moving picture coding apparatus applied to the present invention, motion compensation interframe predictive coding is performed using the result of motion detection performed on the reference frame in units of blocks obtained by dividing the coding target image frame. At this time, the reference frame can be selected from a plurality of reference frame candidates.

  In such a moving image coding apparatus, in the present invention, face detection is performed on an image frame of moving image data, and face parts such as eyes, nose and mouth are further detected from the detected face and detected. Determine the state of each face part. The determination result of the state of the face part is stored in association with information indicating the image frame for which face detection has been performed. Then, the state of the face part detected from the image frame to be encoded is compared with the state of the stored face part, the image frame corresponding to the state of the matching face part is searched, and the reference frame of the face part is used. Use it. Then, each of the face parts is subjected to motion compensation interframe prediction coding in units of motion compensation blocks using the reference frames searched for each face part.

  As a result, even when each facial part of the face that is the subject of the moving image moves suddenly, optimal predictive coding can be performed for each facial part. Further, the reference frame is searched using the face part state stored when the face part is detected for the image frame to be encoded. Therefore, it is possible to efficiently perform inter-frame prediction encoding using motion compensation with a small amount of computation.

  FIG. 1 shows an exemplary configuration of an encoding apparatus 100 according to an embodiment of the present invention. The encoding apparatus 100 performs motion detection on the supplied baseband moving image data in units of blocks obtained by dividing one screen into a predetermined size, and performs interframe predictive encoding using motion compensation. Encoding uses orthogonal transformation using Hadamard transform and integer precision DCT, quantization for transform coefficients, intraframe prediction coding using intraframe prediction coding and motion compensation, and further entropy coding. To do.

  Hereinafter, the Hadamard transform and orthogonal transform using integer precision DCT are referred to as integer transform, and intra-frame prediction coding and inter-frame prediction coding are referred to as intra coding and inter coding, respectively.

  By inter coding, a P picture is formed that performs prediction with a reference frame positioned temporally before a motion compensation block that is a unit of motion compensation. In addition, a B picture that predicts up to two reference frames before and / or after in time series with respect to the motion compensation block is also formed by inter coding. Further, an I picture is formed by intra coding. In this way, a plurality of types of pictures having different temporal reference relationships are formed by inter coding and intra coding. Further, in inter-frame predictive encoding, moving image data is encoded as data having a GOP structure in which these I picture, P picture, and B picture are arranged in a predetermined manner.

For example, when the encoding apparatus 100 forms 1 GOP with 15 frames including one I picture, four P pictures, and 10 B pictures, the frames input to the encoding apparatus 100 are processed in the following order: A picture type is assigned. The subscript indicates the order of input or display.
B ₁ B ₂ I ₃ B ₄ B ₅ P ₆ B ₇ B ₈ P ₉ B ₁₀ B ₁₁ P ₁₂ B ₁₃ B ₁₄ P ₁₅

Here, since the B picture can be subjected to predictive coding using a past picture and a future picture in time series, the order of the B picture is changed with respect to the I picture and the P picture. For example, it is performed in the following order. Note that the B ₁ picture and the B ₂ picture following the I ₃ picture are predictively encoded using the I ₃ picture and the P ₁₅ picture in the immediately preceding GOP.
_{I 3 B 1 B 2 P 6} B 4 B 5 P 9 B 7 B 8 P 12 B 10 B 11 P 15 B 13 B 14

  The encoding apparatus 100 is controlled by a CPU (not shown) according to a predetermined program. The CPU may control the encoding device 100 exclusively, or may control a higher-order system in which the encoding device 100 is incorporated. The CPU includes a ROM and a RAM (not shown), operates as a work memory according to a program stored in advance in the ROM, and controls each unit of the encoding device 100.

  Baseband moving image data 50 is input to the encoding device 100 in units of image frames in the input order described above, and temporarily stored in the current frame storage unit 10 including a frame memory as input image frame storage means. Saved. The image frames stored in the current frame storage unit 10 are rearranged in the above-described encoding order, and are divided into macro blocks of a predetermined size (for example, 16 pixels × 16 pixels) and read for encoding. For example, the macro block is scanned in the horizontal direction from the left end to the right end of the screen, and is read repeatedly in the vertical direction. Also, coordinate information in the image frame is defined for the macroblock in accordance with, for example, the scan order.

  Furthermore, an image frame corresponding to the image data read in units of macroblocks of the moving image data 50 is read from the current frame storage unit 10 and supplied to the face detection unit 32. Note that an image frame corresponding to image data read from the current frame storage unit 10 in units of macroblocks for encoding is hereinafter referred to as an encoding target frame.

  The face detection unit 32 detects a face region including a human face for the encoding target frame supplied from the current frame storage unit 10. Information indicating the face area detected by the face detection unit 32 is supplied to the facial expression recognition unit 33 together with identification information indicating the encoding target frame for which face detection has been performed. The facial expression recognition unit 33 determines the state of each part (hereinafter referred to as a face part) included in the face based on the information indicating the face area supplied from the face detection unit 32. Here, it is assumed that the face part is a part that is considered to frequently move in the face. Examples of such a part in the face include the left eye, the right eye, and the mouth. For example, the facial expression recognition unit 33 determines at least one state among these left eye, right eye, and mouth. The face detection unit 32 and the facial expression recognition unit 33 constitute a determination unit.

  The determination result of each face part state is associated with the coordinate information of the macroblock including the corresponding face part and the identification information of the encoding target frame, and is accumulated in a memory (not shown) included in the facial expression recognition unit 33. Is remembered. The facial expression recognition unit 33 also constitutes face part information storage means. Details of processing in the facial expression recognition unit 33 will be described later.

  Various methods for detecting the face area by the face detection unit 32 can be considered. For example, a method described in Japanese Patent Laid-Open No. 2001-309225 can be used. First of all, the image data includes a central portion that is likely to include skin based on color and shape, and a peripheral region that is likely to include hair based on color and shape. Search for. Based on the result, the first face candidate detection algorithm is used to search for a region that is likely to include a face using a pattern recognition operator. Then, a face is detected in combination with the second algorithm for confirming the presence of the face in the face candidate area obtained by the first algorithm by pattern matching.

  Further, as a method of analyzing the state information of each face part in the face area by the facial expression recognition unit 33, the following method can be considered. First, binarization is performed with “0” for the skin color area of the face and “1” for areas other than the face skin color area. Then, the center of gravity of the face is detected from the skin color area of the face, and the position of the hole obliquely above the center of gravity is determined as the eye area. If a hole cannot be detected, it is determined that the eyes are closed. Further, from the general structure of the human body, a predetermined position on the vertical bisector between the right eye and the left eye below the center of gravity of the face area is defined as the mouth area. If the ratio of the mouth area to the face area is greater than or equal to a predetermined value, it is determined that the mouth is open.

  On the other hand, image data read from the current frame storage unit 10 in units of macroblocks is input to the subtracted input of the subtractor 11 and is supplied to the motion detection unit 23. The motion detection unit 23 detects a motion vector in the image data supplied from the current frame storage unit 10, and outputs the detected motion vector information to the inter prediction unit 22 and the entropy encoding unit 16.

  The subtractor 11 subtracts predicted image data output from the switch 26 described later from the image data input to the subtracted input to generate image residual data. The image residual data is converted into DCT coefficients by orthogonal transform processing such as Hadamard transform or integer precision DCT in the orthogonal transform unit 12.

  The DCT coefficient is quantized by the quantization unit 13 using a predetermined quantization parameter. The quantization parameter is a parameter having a predetermined relationship with the quantization step when the DCT coefficient is quantized. For example, the quantization parameter is determined so that the logarithm of the quantization parameter and the quantization step is proportional. The quantized value output from the quantizing unit 13 is supplied to the entropy coding unit 16.

  The quantized value output from the quantization unit 13 is also supplied to the inverse quantization unit 17. The quantized value is inversely quantized by the inverse quantization unit 17 and inversely orthogonally transformed by the inverse orthogonal transform unit 18 to be locally decoded image data. The predicted image data output from the switch 26 is added to the local decoded image data by the adder 19 to form restored image data. The restored image data is stored in the frame memory 24 and is also stored in the restored image frame storage unit 30 including a frame memory after the coding distortion is reduced by the deblocking filter 20. The restored image frame storage unit 30 as reference candidate frame storage means can store restored image data for a plurality of frames.

  A reference frame determination unit 31 as a search unit and a reference frame determination unit selects and determines data to be used as a reference frame. In the embodiment of the present invention, the reference frame determination unit 31 selects a reference frame from the restored image data stored in the restored image frame storage unit 30 based on the determination result of the facial part state in the facial expression recognition unit 33. You can select and decide.

  That is, the reference frame determination unit 31 compares the determination result of the facial part state for the encoding target frame in the facial expression recognition unit 33 with the determination result of the facial part state stored in the facial expression recognition unit 33. As a result of the comparison, the face part states stored in the facial expression recognizing unit 33 are searched for those that match the determination result of the face part state for the encoding target frame. Then, among the restored image frames stored in the restored image frame storage unit 30, a restored image frame corresponding to the face part state obtained as a search result is determined as a reference frame and stored in the reference frame storage unit 21.

  The process in the reference frame determination unit 31 is performed for each face part. That is, the reference frame can be determined for each face part. Details of processing in the reference frame determination unit 31 will be described later.

  The intra prediction unit 25 performs an intra-frame prediction process using the restored image data stored in the frame memory 24, and generates predicted image data. The intra prediction image data output from the intra prediction unit 25 is supplied to the input terminal 26A of the switch 26.

  The motion detection unit 23 performs motion detection of the image data supplied from the current frame storage unit 10 in units of macroblocks using the reference frame determined by the reference frame determination unit 31. The inter prediction unit 22 performs inter-frame prediction processing based on the restored image data stored in the reference frame storage unit 21 and the motion vector detected by the motion detection unit 23, and generates inter prediction image data. The inter prediction image data is supplied to the input terminal 26B of the switch 26.

  The switch 26 selects which of intra prediction and inter prediction is used. One of the intra prediction image data output from the intra prediction unit 25 and the inter prediction image data output from the inter prediction unit 22 is selected, and the selected prediction image data is supplied to the subtraction input of the subtractor 11. At the same time, it is supplied to the adder 19.

  The entropy encoding unit 16 entropy encodes the quantization parameter supplied from the quantization unit 13 and the motion vector information output from the motion detection unit 23. Further, the entropy encoding unit 16 further includes information indicating whether intra encoding or inter encoding has been performed (macroblock type), and information indicating the reference frame used in inter prediction in units of macroblocks. Entropy encoding. The output of the entropy encoding unit 16 is output from the encoding device 100 as, for example, an encoded stream in which codes are arranged according to the screen arrangement order.

  Next, reference frame determination processing by the reference frame determination unit 31 will be described in more detail. FIG. 2 is a flowchart showing an example of reference frame determination processing according to the embodiment of the present invention. Each step of FIG. 2 is executed and / or controlled by a CPU (not shown) that controls the entire encoding apparatus 100, for example.

  In step S10, the face detection unit 32 detects a face area in the encoding target frame. In the next step S11, the facial expression recognition unit 33 detects the face parts included in the face area detected by the face detection unit 32, and determines the state of each detected face part.

  As an example, face detection is performed on the encoding target frame 200 as shown in FIG. 3A, and the left eye, right eye, and mouth face parts are detected from the detected face area. In FIG. 3A and the following similar figures, a block indicated by a lattice in the encoding target frame 200 is assumed to be a macro block, and the block coordinates of the block at the upper left corner are (0, 0).

  In the present embodiment, as an example of facial expression in each face part, for each face part of the left eye, right eye, and mouth, the state information “0” is set when open, and the state information “1” is set when closed. The analysis result shall be saved. In the example of FIG. 3A, for each face part in the face area detected from the encoding target frame 200, the left eye 210 and the right eye 211 are open, and the mouth 212 is closed. It is determined at 33. Therefore, as illustrated in FIG. 3B, the state information of the left eye 210 and the right eye 211 is “0”, and the state information of the mouth 212 is “1”.

  Further, the left eye 210 is a rectangular area whose diagonal coordinates are indicated by block coordinates (3, 3) and (4, 4), and the right eye 211 is diagonally indicated by block coordinates (5, 3) and (6, 4). Included in the rectangular area. Further, the mouth 212 is included in a rectangular area whose diagonal coordinates are indicated by block coordinates (4, 5) and (7, 5).

  Returning to the flowchart of FIG. 2, when the facial expression recognition unit 33 determines the state of each facial part in step S11, the process proceeds to step S12. In step S12 and subsequent steps, processing for determining a reference frame is sequentially performed for each face part. Here, the face parts are processed in the order of left eye, right eye, and mouth.

  In step S <b> 12, the reference frame determination unit 31 compares the state of the facial part to be determined in the encoding target frame with the state of the facial part stored in the facial expression recognition unit 33. The face part corresponding to the face part to be determined may be determined based on, for example, the coordinate information of the face part or the positional relationship of the face part in the face area. The reference frame determination unit 31 acquires facial part state information corresponding to a frame that is positioned closest to the encoding target frame in the facial part state information stored in the facial expression recognition unit 33. To do.

  Hereinafter, the restored image frame stored in the restored image frame storage unit 30 is referred to as a reference candidate frame. That is, the reference frame determination unit 31 compares the face part state information determined for the encoding target frame with the face part state information stored in the facial expression recognition unit 33 based on the result of comparison. A restored image frame is read from the storage unit 30. Using the restored image frame as a reference frame, motion detection by the motion detection unit 23 and inter prediction by the inter prediction unit 22 are performed.

  In addition, the time closest to the encoding target frame is based on the picture type of the encoding target frame, and the reference candidate frame that can be referred to by the encoding target frame is temporally related to the encoding target frame. The closest thing to. For example, if the picture type of the encoding target frame is a P picture, the reference candidate frame indicates a past P or I picture that is temporally closest to the encoding target frame. For example, if the picture type of the encoding target frame is a B picture, the reference candidate frame indicates a past B, P, or I picture that is temporally closest to the encoding target frame.

  Based on the result of the comparison in step S12, it is determined in step S13 whether or not the state information of the corresponding face part matches between the encoding target frame and the reference candidate frame. If it is determined that they match, the process proceeds to step S14, and the reference frame determination unit 31 determines a reference candidate frame as a reference frame for the face part to be determined. Then, the process proceeds to step S15.

  On the other hand, if it is determined in step S13 that the state information of the face part does not match between the encoding target frame and the reference candidate frame, the process proceeds to step S16. In step S16, it is determined whether or not the reference candidate frame compared in step S12 is the last reference candidate frame. Here, the last reference candidate frame refers to a reference candidate frame that is set to be used as a reference candidate frame and that is farthest in time from the encoding target frame. If it is determined that the frame is not the last reference candidate frame, the process returns to step S12, and the process for the face part to be determined is performed on the reference candidate frame that is next in time.

  On the other hand, if it is determined in step S16 that the process for the last reference candidate frame has been completed for the face part to be determined, the process proceeds to step S17. In other words, in this case, the face part whose state information matches the face part to be determined does not exist in all the reference candidate frames set to be used as the reference candidate frame. In this case, in step S17, a reference candidate frame positioned closest in time to the encoding target frame is determined as a reference frame. Then, the process proceeds to step S15.

  In step S15, it is determined whether or not the determination has been completed for all the face parts detected in the encoding target frame in step S11 described above. If it is determined that the determination is not completed, the process returns to step S12, and the next face part is processed. On the other hand, when it is determined that the determination has been completed, a series of processes for the encoding target frame is completed.

  The process of the flowchart of FIG. 2 will be described more specifically with reference to FIGS. As an example, as illustrated in FIG. 4, it is assumed that two reference candidate frames 201 and 202 are set for the encoding target frame 200. The reference candidate frame 201 is assumed to be a frame immediately before the encoding target frame 200 in terms of time. The reference candidate frame 202 is assumed to be a frame farther in time than the reference candidate frame 201 with respect to the encoding target frame 200.

  In the reference candidate frame 201, as illustrated in FIG. 5A, the left eye 210, the right eye 211, and the mouth 212 are closed in each face part in the face area. Therefore, as illustrated in FIG. 5B, the state information of the left eye 210, the right eye 211, and the mouth 212 is set to “0”. In addition, the left eye 210 indicates a rectangular area where diagonal coordinates are indicated by block coordinates (3, 3) and (4, 3), and the right eye 211 indicates diagonal coordinates by block coordinates (4, 3) and (5, 3). Included in the rectangular area. Further, the mouth 212 is included in a rectangular area whose diagonal coordinates are indicated by block coordinates (3, 5) and (5, 5).

  On the other hand, as illustrated in FIG. 6A, the reference candidate frame 202 is in a state where the left eye 210, the right eye 211, and the mouth 212 are open for each face part in the face area. Therefore, as illustrated in FIG. 6B, the state information of the left eye 210, the right eye 211, and the mouth 212 is set to “1”. In addition, the left eye 210 indicates a rectangular area where diagonal coordinates are indicated by block coordinates (2, 3) and (3,4), and the right eye 211 indicates diagonal coordinates by block coordinates (4, 3) and (5, 3). Included in the rectangular area. Further, the mouth 212 is included in a rectangular area whose diagonal coordinates are indicated by block coordinates (3, 5) and (5, 5).

  A case where the face part to be determined is the left eye 210 will be described as an example. In the encoding target frame 200, the left eye 210 is open and the state information is “1” (see FIGS. 3A and 3B). In contrast, the left eye 210 of the reference candidate frame 201 immediately before the encoding target frame 200 is closed, and the state information is “0” (FIG. 5A and FIG. 5B). reference). Therefore, as a result of the comparison in step S12 described above, it is determined that the state information of both does not match (step S13). Therefore, the reference frame determination unit 31 suspends the left eye 210 from setting the reference candidate frame 201 as the reference frame, and the process proceeds to step S16.

  In step S16, it is determined that the reference candidate frame 201 is not the last reference candidate frame. Then, the process returns from step S16 to step S12, and the state information of the left eye 210 is compared with the encoding target frame 200 for the reference candidate frame 202 which is the next reference candidate frame. The left eye 210 of the reference candidate frame 202 is open, and the state information is “1” (see FIG. 6A and FIG. 6B). Therefore, as a result of the comparison in step S12 described above, it is determined that the state information of both coincides (step S13), and the reference candidate frame 202 is determined as the reference frame of the encoding target frame 200 (step S14).

  As described above, in the embodiment of the present invention, it is possible to determine a reference candidate frame whose facial expression is close to the encoding target frame as a reference frame. Therefore, high encoding efficiency can be realized even in an encoding device in which the number of reference frames that can be referred to in motion vector detection is limited.

  In addition, when there is no reference candidate frame whose facial expression is close to the encoding target frame, it is considered that the change from the encoding target frame is the least, and the reference candidate that is closest in time to the encoding target frame A frame can be determined as a reference frame. As a result, even when there is no reference candidate frame whose facial expression is close to the encoding target frame, there is a high possibility that deterioration in image quality during encoding can be suppressed.

  As described above, in the present embodiment, the states of the left eye, the right eye, and the mouth as the facial parts are classified into two states of open or closed, but this is not limited to this example. For example, the number of states may be further increased depending on how the left eye, the right eye, and the mouth open. Thereby, the comparison of the face part state between the encoding target frame and the reference candidate frame can be performed in more detail.

  In this case, even if the face part state does not necessarily match between the encoding target frame and the reference candidate frame, a reference candidate frame whose face part state is closer than the encoding target frame by a predetermined amount or more is determined as a reference frame. You may do it. When there is no reference candidate frame whose face part state is close to a predetermined value or more with respect to the encoding target frame, the reference candidate frame that is temporally closest to the encoding target frame is determined as a reference frame.

  In this embodiment, the face parts are the left eye, the right eye, and the mouth, and it is determined whether or not the states of these three face parts match. However, this is not limited to this example. For example, it is also conceivable to use other parts of the face such as the nose and eyebrows as face parts to determine the state match. The position of the nose can be specified based on the positional relationship between the left eye, the right eye, and the mouth, and the positions of two holes and shadows between the left eye, the right eye, and the mouth. The position of the eyebrows can be specified from the positions of the left eye and the right eye.

  In the flowchart of FIG. 2 described above, the reference candidate frame determination process is performed for each face part, but this is not limited to this example. For example, each face part may be determined for each reference candidate frame. More specifically, first, determination processing for each face part is performed on the reference candidate frame that is temporally closest to the encoding target frame. If the reference frames are not determined for all the facial parts, a determination process is performed on the facial parts for which the reference frame is not determined for the reference candidate frame temporally next to the encoding target frame. This process is repeated until a reference frame is determined for each face part.

<Other embodiments>
The above-described embodiments can also be realized in software by a computer (or CPU, MPU, etc.) of a system or apparatus.

  Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto.

  A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code.

  The program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers.

  That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention.

  In addition, a storage medium in which the computer program for realizing the above-described embodiment is encrypted and distributed is distributed, and key information for decrypting is supplied to a user who satisfies a predetermined condition, and the user's computer Installation may be allowed. The key information can be supplied by being downloaded from a homepage via the Internet, for example.

  In addition, the computer program for realizing the above-described embodiment may use an OS function already running on the computer.

  Furthermore, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU included in the expansion board. Good.

It is a block diagram which shows the structure of an example of the encoding apparatus by embodiment of this invention. It is a flowchart which shows a process of an example of the reference frame determination by embodiment of this invention. It is a figure for demonstrating the face part state of an encoding object frame. It is a figure for demonstrating an encoding object frame and a reference candidate frame. It is a figure for demonstrating the face part state of a reference candidate frame. It is a figure for demonstrating the face part state of a reference candidate frame.

Explanation of symbols

10 current frame storage unit 21 reference frame storage unit 22 inter prediction unit 23 motion detection unit 30 restored image storage unit 31 reference frame storage unit 32 face detection unit 33 facial expression recognition unit 100 encoding device 200 encoding target frames 201 and 202 Candidate frame

Claims (11)

Perform motion compensation interframe predictive coding using the result of motion detection performed on the reference frame in units of blocks obtained by dividing the image frame to be encoded, and select the reference frame from a plurality of reference frame candidates A video encoding device,
Input image frame storage means for temporarily storing the input image frame;
Reference candidate frame storage means for storing a plurality of reference candidate frames generated based on the image frames stored in the input image frame storage means;
Determination means for detecting a face area from the image frame output from the input image frame storage means, further detecting a face part included in the face area, and determining a state of the face part;
Face part information storage means for storing the state of the face part determined by the determination means in association with information indicating the image frame including the face part;
From the face part information storage unit, the state of the face part detected and determined by the determination unit from the encoding target frame output from the input image frame storage unit matches, and the encoding target frame is Search means for searching for information indicating the image frame that can be referred to;
Among the plurality of reference candidate frames stored in the reference candidate frame storage unit, the reference candidate frame corresponding to information indicating the image frame searched by the search unit is selected as the block corresponding to the face part. Reference frame determining means for determining the reference frame;
An apparatus for encoding a moving image, wherein motion-compensated interframe prediction encoding is performed on the encoding target frame using the reference frame determined by the reference frame determining means. The moving image encoding apparatus according to claim 1, wherein the determination unit detects at least one of eyes or a mouth included in the face region as a face part. The moving image encoding apparatus according to claim 2, wherein the determination unit determines the degree of opening of the eyes or mouth as the state of the face part. 4. The moving image encoding apparatus according to claim 2, wherein the determination unit determines whether the eyes or mouth are open or closed as the state of the face part. The search unit performs the search for each of the face parts detected and determined from the encoding target frame by the determination unit for one of the encoding target frames output from the input image frame storage unit,
5. The reference frame determination unit according to claim 1, wherein the reference frame determination unit determines the corresponding reference candidate frame for each piece of information indicating the image frame searched for the face part by the search unit. The moving image encoding device according to any one of the preceding claims. The search means searches for information indicating the image frame in order from the temporally closer to the encoding target frame among the information indicating the image frame stored in the face part information storage means. The moving picture coding apparatus according to claim 1, wherein: If the information indicating the image frame in which the state of the face part matches is not searched, the search means includes the encoding target frame of the information indicating the image frame stored in the face part information storage means. 7. The moving image according to claim 1, wherein information indicating the image frame that can be referred to and that is temporally closest to the image frame is used as a search result. Encoding device. If the information indicating the image frame in which the state of the face part matches is not searched, the search means includes the encoding target frame of the information indicating the image frame stored in the face part information storage means. 7. The moving image encoding apparatus according to claim 1, wherein information indicating the image frame that can be referred to and has the closest face part state is used as a search result. . Motion detection means for performing motion detection on a reference frame of image data read in block units obtained by dividing the image frame from the input image frame storage means;
Motion compensation means for performing motion compensation inter-frame prediction on the image data in units of blocks based on the result of motion detection by the motion detection means;
The reference candidate frame is:
9. The method according to claim 1, wherein the image data read out in units of blocks from the input image frame storage unit is generated based on the image data predicted between motion compensation frames by the motion compensation unit. The moving image encoding device according to any one of the preceding claims. Perform motion compensation interframe predictive coding using the result of motion detection performed on the reference frame in units of blocks obtained by dividing the image frame to be encoded, and select the reference frame from a plurality of reference frame candidates A video encoding method comprising:
An input image frame storage step of temporarily storing the input image frame in the input image frame storage means;
A reference candidate frame storage step of storing a plurality of reference candidate frames generated based on the image frame stored in the input image frame storage unit in a reference candidate frame storage unit;
A determination step of detecting a face area from the image frame output from the input image frame storage means, further detecting a face part included in the face area, and determining a state of the face part;
A face part information storage step of storing the state of the face part determined in the determination step in a face part information storage unit in association with information indicating the image frame including the face part;
From the face part information storage unit, the state of the face part detected and determined in the determination step from the encoding target frame output from the input image frame storage unit matches, and the encoding target frame is A search step for searching for information indicating the image frame that can be referred to;
Among the plurality of reference candidate frames stored in the reference candidate frame storage unit, the reference candidate frame corresponding to the information indicating the image frame searched in the search step is selected from the block corresponding to the face part. A reference frame determining step for determining the reference frame;
A moving picture coding method, wherein motion-compensated interframe prediction coding is performed on the coding target frame using the reference frame determined in the reference frame determination step.   A program that causes a computer to function as the moving picture encoding apparatus according to any one of claims 1 to 9.

Images