JP2010161740A - Image coding device and image coding method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform speedy playback and easy editing by creating a scene with an expression such as a smiling face or a tear-stained face as a reference frame while suppressing reduction in the efficiency of coding. <P>SOLUTION: An image coding device includes: a face information creating means for creating face information for identifying a face by analyzing an input image signal consisting of a plurality of frames; a coding means for performing compression coding on the input image signal utilizing an inter-frame prediction scheme; a prohibition decision means for determining whether to prohibit reference jumping over frames when performing inter-frame prediction in the coding means on the basis of the face information created by the face information creating means; and a setting means for setting a reference frame to prohibit the jumping reference when the prohibition decision means determines that the face information meets a prohibition condition to prohibit the reference jumping over frames. Coding is performed in accordance with a random access enabled picture type (IDR picture). <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image encoding device and an image encoding method, and more particularly to a technique suitable for use in compression encoding an image by performing inter-frame prediction.

  As techniques for high-efficiency encoding of images, encoding systems such as MPEG1 and MPEG2 using JPEG compression techniques and motion prediction / compensation techniques have been established. Each manufacturer has developed and commercialized an imaging device such as a digital camera or a digital video camera, or a recording device such as a DVD recorder, which can record an image on a recording medium using these encoding methods.

  On the other hand, the user can easily view images using these imaging devices, recording devices, personal computers, DVD players, and the like.

  By the way, the digitized moving image has a huge amount of data. Therefore, research has been continued on a moving picture encoding method that can achieve higher compression than MPEG 1 and 2 and the like. In recent years, ITU-T (International Telecommunication Union Telecommunication Standardization Sector) and ISO (International Organization for Standardization) An encoding method called H.264 / MPEG-4 part 10 (hereinafter referred to as “H.264”) has been standardized.

  Here, H. Selection of a reference image used for picture type and inter-frame prediction in H.264 will be described with reference to FIGS. 12 (a) to 12 (c) and 13 (a) to 13 (b) show the input image sequence and its picture type. The upper row is the display order (displayed in order from the left), and the lower row is encoded. It is shown in order (encoded in order from the left).

  For example, in FIG. 12A, the P8 picture is the ninth P picture frame displayed. Also, the arrows in FIG. 12 indicate the reference relationship. For example, in the example illustrated in FIG. 12A, the P8 picture refers to the B0 picture. In the example shown in FIG. 12B, the B0 picture refers to the P2 picture and the B7 picture.

  H. There are three picture types of image frames in H.264. That is, in addition to the difference between the I picture that is encoded only from the information in the same frame, the P picture that is encoded using the difference between the temporally previous frame, and the temporally previous frame, the time In particular, there is a B picture that can use a difference from a later frame.

  H. In H.264, when performing inter-frame prediction, any frame and picture type in an image sequence can be used as a reference image. For example, as shown in FIG. 12A, the P picture (P8) can refer to not only the I picture but also the frame by skipping the I picture. Similarly, as shown in FIG. 12B, the B picture (B0) can be referred to not only the I picture but also the I picture.

  In this way, H.C. H.264 allows flexible references. As a result, as compared with a method in which only the I picture or the P picture immediately before the P picture can be referred to in the case of a P picture like MPEG2, the H. H.264 improves inter-frame prediction accuracy and improves coding efficiency.

  On the other hand, since the flexible reference as described above is allowed, In H.264, random access may not be performed quickly. As an example, a case where playback is performed from an I5 picture that is a frame in the middle of an image sequence by random access in FIG.

  When decoding is started from the I5 picture in the image sequence and the P8 picture is decoded, since the P8 picture refers to the B0 picture, it is necessary to decode the B0 picture in advance. Furthermore, since the B0 picture refers to the P2 picture and the B7 picture, it is necessary to decode the P2 picture and the B7 picture in advance in order to decode the B0 picture.

  Similarly, although not shown, since the P2 picture and the B7 picture also refer to other pictures, respectively, in order to decode the P2 picture and the B7 picture, it is necessary to decode the other pictures in advance. There is. As described above, even when it is desired to start reproduction from the I5 picture, since the reference is allowed to skip the I5 picture, it is necessary to start decoding retroactively to the data before the I5 picture. It becomes difficult to start playback quickly.

  Therefore, in order to solve this problem and realize quick random access, a method of periodically limiting I pictures has been proposed (see Patent Document 1). This restricted I picture is H.264. In H.264, this is called an IDR picture.

Here, the IDR picture will be described with reference to FIGS. 13 (a) and 13 (b).
The image sequence shown in FIGS. 13A and 13B is an image sequence in which the I5 picture is set as the IDR picture with respect to the image sequence similar to FIGS. 12A and 12B.

  When the I5 picture is set as the IDR picture, the frame memory in which the reference image is recorded is cleared when the picture is encoded. Therefore, pictures encoded after the IDR picture cannot refer to pictures encoded before the IDR picture. Similarly, a picture encoded before an IDR picture cannot refer to a picture encoded after the IDR picture.

  In the example shown in FIG. 13A, a P picture (such as P8) or a B picture (such as B6) encoded after an IDR (IDR5) picture is a P picture (P2) encoded before the IDR picture. Etc.) and B picture (B0 etc.) cannot be referred to.

  Conversely, in the example shown in FIG. 13B, a P picture (such as P2) or a B picture (such as B0) encoded before an IDR (IDR5) picture is a P picture encoded after the IDR picture. A picture (such as P8) or a B picture (such as B6) cannot be referenced.

  As a result, if playback is started from the IDR picture, it is not necessary to decode back to the image data before the IDR picture, so that it is possible to realize playback with quick random access.

  In addition, since the reference skipping the IDR picture is prohibited, for example, cut editing using the IDR picture as a cut frame becomes possible without performing re-encoding processing. In order to perform editing using such an IDR picture, a method is proposed in which a scene that is considered to be important for a photographer is determined in accordance with a change in image motion, and an IDR picture is set (Patent Document 2). reference).

JP 2003-199112 A JP 2006-157893 A

  As described above, H.P. In the H.264 coding scheme, random access can be quickly performed by using an IDR picture that restricts the reference relationship of inter-frame prediction. Therefore, a large number of IDR pictures need to be set in order to perform quick reproduction and easy editing from any place in the image sequence.

  However, by setting an IDR picture, the reference relationship is limited as described above. For this reason, if a large number of IDR pictures are set, the coding efficiency may be lowered. That is, it is desirable to set the IDR picture to the minimum necessary in consideration of encoding efficiency.

  When an IDR picture is set periodically as in Patent Document 1, a frame that is not required for random access is also set as an IDR picture, which may reduce the coding efficiency. Further, as in Patent Document 2, in the method of setting an IDR picture by determining a scene that seems to be important for a photographer according to a change in image motion, all scenes that are considered important for the photographer are determined. It is difficult.

  For example, a scene in which a person laughs or crys is considered an important scene. However, in the method of determining an important scene based on a change in image movement as in Patent Document 2, a scene of a smile or a crying face Is difficult to detect.

  The present invention has been made in view of the above-described problems, so that rapid reproduction and easy editing can be performed from a scene having a facial expression such as a smile or a crying face while suppressing a decrease in encoding efficiency. The purpose is to do.

  An image encoding apparatus according to the present invention is an image encoding apparatus that compresses and encodes an input image signal composed of a plurality of frames, and generates face information for analyzing the input image signal and generating face information for identifying a face. Means, encoding means for compressing and encoding the input image signal using an inter-frame prediction method, and face information generated by the face information generating means for the encoding target frame in the encoding means. Based on the prohibition determination means for determining whether to prohibit reference in inter-frame prediction that skips the encoding target frame and the prohibition determination means, it is determined to prohibit reference that skips the encoding target frame. Setting means for setting the frame to be encoded to a reference frame that prohibits interlaced reference.

  An image encoding method of the present invention is an image encoding method for compressing and encoding an input image signal composed of a plurality of frames, and generating face information for analyzing the input image signal and generating face information for identifying a face A step of compressing and encoding the input image signal using an inter-frame prediction method, and the face information created in the face information creation step for the encoding target frame in the encoding step. Based on the prohibition determination step for determining whether to prohibit reference in inter-frame prediction that skips the encoding target frame, and in the prohibition determination step, it is determined that reference that skips the encoding target frame is prohibited. And a setting step of setting the encoding target frame to a reference frame that prohibits interlaced reference.

  The computer program of the present invention is a computer program for causing a computer to execute a process of compressing and encoding an input image signal composed of a plurality of frames, and generating face information for identifying the face by analyzing the input image signal An information creating step, a coding step for compressing and coding the input image signal using an inter-frame prediction method, and a face created in the face information creating step for a frame to be coded in the coding step Based on the information, in a prohibition determination step for determining whether or not reference in inter-frame prediction that skips the encoding target frame is prohibited, and prohibiting reference that skips the encoding target frame in the prohibition determination step If it is determined, a setting process for setting the encoding target frame to a reference frame that prohibits interlaced reference. Characterized in that to execute the door to the computer.

  According to the present invention, encoding is performed by setting the minimum necessary image frame as a reference frame according to the degree of facial expression, so a scene with a high degree of facial expression while suppressing a decrease in encoding efficiency. Can be quickly reproduced and easily edited. Thereby, for example, it is possible to easily start from the smile frame, or to perform editing with the smile frame as a starting point.

(First embodiment)
FIG. 1 is a block diagram illustrating an embodiment of the present invention and a configuration example of an image encoding device. The image encoding apparatus according to the present embodiment is an apparatus that performs encoding by setting a reference frame according to the degree of facial expression. Hereinafter, a configuration example of the image encoding device of the present embodiment will be described with reference to FIG.

  The image encoding apparatus according to the present embodiment includes an encoding unit 101, a face determination unit 102, and a reference frame setting determination unit 103. The encoding unit 101 performs compression encoding of an input video signal (input image signal), generates an encoded stream, and outputs the encoded stream. The encoding method in this embodiment is H.264. H.264 encoding scheme and MPEG2 encoding scheme, and other encoding schemes using inter-frame prediction schemes. The H.264 encoding method will be described as an example.

  The face determination unit 102 analyzes the input video signal, analyzes the face of the subject, creates face information for identifying the face, and outputs the created face information. Details of the face information will be described later. When the reference frame setting determination unit 103 determines that the encoding target frame to be encoded by the encoding unit 101 is to be encoded as the reference frame according to the face information output from the face determination unit 102, the reference frame setting information Is output to the encoding unit 101.

  Here, the reference frame is a frame in which a picture type capable of quick random access is set by prohibiting reference in inter-frame prediction that skips the reference frame. H. In the H.264 encoding method, the reference frame is an IDR picture frame. In the MPEG encoding method, the reference frame is an I picture frame.

Next, operations of the encoding unit 101, the face determination unit 102, and the reference frame setting determination unit 103 will be described in detail.
First, a configuration example of the encoding unit 101 will be described in detail with reference to FIG.
FIG. 2 is a block diagram illustrating a configuration example of the encoding unit 101. As shown in FIG. 2, the encoding unit 101 includes a frame rearrangement unit 201, a subtracter 202, an integer conversion unit 203, a quantization unit 204, an entropy encoding unit 205, an inverse quantization unit 206, and an inverse integer conversion unit 207. Etc. The adder 208 includes a first frame memory 209 and a second frame memory 213, an intra prediction unit 210, a first switch 211, and a second switch 217. Furthermore, a deblocking filter 212, an inter prediction unit 214, a motion detection unit 215, and a picture type determination unit 216 are included.

  The encoding unit 101 in the image encoding apparatus of the present embodiment configured as described above configures a block by dividing an input video signal, performs an encoding process on a block basis, and outputs an encoded stream To do.

Next, the encoding process performed by the encoding unit 101 will be described.
First, the frame rearrangement unit 201 rearranges the video signals input in the display order in the encoding order. The subtracter 202 subtracts the predicted image data from the input image data and outputs the image residual data to the integer conversion unit 203. The generation of predicted image data will be described later.

  The integer transform unit 203 performs orthogonal transform processing on the image residual data output from the subtracter 202 and outputs transform coefficients to the quantization unit 204. The quantization unit 204 quantizes the transform coefficient output from the integer transform unit 203 using a predetermined quantization parameter. The entropy encoding unit 205 receives the transform coefficient quantized by the quantization unit 204, entropy encodes it, and outputs it as an encoded stream.

  On the other hand, the transform coefficient quantized by the quantization unit 204 is also used for generating the predicted image data described above. The inverse quantization unit 206 inversely quantizes the transform coefficient quantized by the quantization unit 204. The inverse integer transform unit 207 performs inverse integer transform on the transform coefficient inversely quantized by the inverse quantization unit 206 and outputs it as decoded image residual data.

  The adder 208 adds the decoded image residual data output from the inverse integer transform unit 207 and the predicted image data, and outputs the result as reconstructed image data. The reconstructed image data output from the adder 208 is recorded in the frame memory 209. At the same time, when the deblocking filter process is performed on the reconstructed image data, it is recorded in the second frame memory 213 via the deblocking filter 212. Further, when the deblocking filter process is not performed, it is recorded in the second frame memory 213 without going through the deblocking filter 212.

  The first switch 211 is a selection unit that selects whether to perform deblocking filter processing on the reconstructed image data output from the adder 208. Among the reconstructed image data, data that may be referred to in the subsequent prediction is stored in the first frame memory 209 or the second frame memory 213 for a while.

  The intra prediction unit 210 performs intra-frame prediction processing using the reconstructed image data recorded in the first frame memory 209, and generates predicted image data. Also, the inter prediction unit 214 performs inter-frame prediction processing based on the motion vector information detected by the motion detection unit 215 using the reconstructed image data recorded in the second frame memory 213, and generates predicted image data To do. Here, the motion detection unit 215 detects a motion vector in the input image data, and outputs the detected motion vector information to the entropy encoding unit 205 and the inter prediction unit 214, respectively.

  The picture type determination unit 216 outputs the picture type to be encoded to the intra prediction unit 210, the inter prediction unit 214, and the second switch 217. When the reference frame setting determination unit 103 determines that the encoded frame is to be a reference frame, the picture type is determined as the reference frame. Otherwise, the picture type of the frame is determined as a picture type that conforms to the encoding method.

  When it is determined that the encoded frame is a reference frame, the picture type of the frame is determined to be an I picture, and a jump reference prohibition flag is added to the frame. Then, the prohibition determination may be performed based on the presence or absence of the skip reference prohibition flag, and the inter prediction unit 214 may determine a reference relationship that does not skip the I picture.

  The second switch 217 is a selection unit for selecting whether to use predicted image data generated by the intra prediction unit 210 or predicted image data generated by the inter prediction unit 214 as predicted image data. That is, it has a function for selecting whether to use intra prediction or inter prediction.

  The second switch 217 is controlled according to the picture type determined by the picture type determination unit 216. As a result, either the output from the intra prediction unit 210 or the output from the inter prediction unit 214 is selected, and the selected predicted image data is output to the subtracter 202 and the adder 208. The above is the description regarding the encoding unit 101.

Next, the face determination unit 102 will be described in detail with reference to FIG. 3, FIG. 4, and FIG.
FIG. 3 is a block diagram illustrating a configuration example of the face determination unit 102. As shown in FIG. 3, the face determination unit 102 includes a face detection unit 301, a face recognition history data recording unit 302, a face recognition unit 303, a facial expression determination unit 304, and a switch 305.

Next, face determination processing performed by the face determination unit 102 will be described.
First, the face detection unit 301 detects at least one face of a subject included in a frame of an input video signal, that is, an encoding target frame, and uses the center coordinates of the face in the frame as the reference coordinates of the face, And direction information are detected and calculated for each face and output.

  The face recognition history data recording unit 302 records the face image data detected by the face detection unit 301 and “face ID” set by the face recognition unit 303 described later. The face recognition unit 303 records the face of the subject of the input video signal in the face recognition history data recording unit 302 based on the information indicating the center coordinates, size, and direction of the face output from the face detection unit 301. It is determined whether the face matches the face. Then, “face ID”, which is information for identifying the face, is output as face information for each face, and the face image necessary for face recognition processing and the “face ID” corresponding to the face are output to the face. The data is output to the recognition history data recording unit 302.

  If it is determined that the face of the subject of the input video signal does not match the face recorded in the face recognition history data recording unit 302, a new “face ID” is set for the determined face. . On the other hand, if it is determined that the face of the subject of the input video signal matches the face recorded in the face recognition history data recording unit 302, the same as the face recorded in the face recognition history data recording unit 302 “ “Face ID” is set to the calculated face.

  The face recognition history data recorded in the face recognition history data recording unit 302 may be cleared for each stream. The facial expression determination unit 304 determines the facial expression of the subject included in the video signal based on the information indicating the center coordinates, size, and direction of the face output from the face detection unit 301, and determines the facial expression type and facial expression index. Is output.

  Examples of facial expressions include a smile, an angry face, and a crying face. The facial expression index is an index representing the degree of facial expression. In the present embodiment, a facial expression index representing the degree of facial expression in a plurality of stages is calculated, for example, a value that varies in the range of 0 to 10. For example, a smile expression index of 0 is a face that is not laughing and is a so-called true face. On the other hand, a smile expression index of 10 is a laughing face. The switch 305 is a selection unit that selects whether to include the “face ID” information output from the face recognition unit 303 in the face information.

  Note that a known method such as object detection can be used as the face detection method performed by the face detection unit 301, and thus detailed description thereof is omitted in the present embodiment. Further, as the face recognition method performed by the face recognition unit 303, for example, a known method such as object recognition can be used, and thus detailed description thereof is omitted in the present embodiment. The facial expression determination by the facial expression determination unit 304 uses, for example, a known facial expression determination method that is determined according to the relative position or shape of each part (eyes, nose, mouth, etc.) of the face in the face area. Omitted.

The face information output from the face detection unit 301, the face recognition unit 303, and the facial expression determination unit 304 by the above method will be described with reference to FIGS.
FIG. 4 is a diagram showing a video signal of frame number 0, and FIG. 5 is a diagram showing face information output from the face determination unit 102 for each frame, obtained from frame numbers 0 and 1. The face information is shown. In the example of FIGS. 4 and 5, the case where one face is included in the frame will be described for the sake of simplicity. However, a plurality of faces may be included in the frame.

  In the video signal of frame number 0 as shown in FIG. 4, the face detection unit 301 detects the face within the dotted line, (x, y) = (960, 540) as the center coordinate of the face, and (x_size, y_size) = (370, 370), and the face information “right” is output as the direction.

  The facial expression determination unit 304 outputs facial information of “smile” as the facial expression type and “5” as the facial expression index because the face in the dotted line in FIG. The face recognition unit 303 sets a new “face ID” for the face detected by the face detection unit 301 because no face information is recorded in the face recognition history data at frame number 0, and “face ID 0 ”Is output. In the example of FIG. 5, since the subject having the same face as frame number 0 is included in frame number 1, the same “face ID” “face ID 0” as frame number 0 is output as face information of frame number 1. Has been.

  As described above, it is possible to know information about a plurality of faces included in the video signal for each frame from the face information output from the face determination unit 102. Furthermore, it is possible to determine whether or not a face detected in the past matches with the “face ID” output from the face recognition unit 303. That is, by performing face recognition for determining whether or not the first face of the current frame matches the second face of the past frame, it is determined that the prohibition condition prohibiting the reference that skips the frame is satisfied. In this case, it is not determined that the first face determined to match the second face within the predetermined period from the second face frame satisfies the prohibition condition for prohibiting reference that skips the frame. Like that.

  Next, the reference frame setting determination unit 103 will be described in detail with reference to FIG. 6, FIG. 7, FIG. 8, FIG. The reference frame setting determination unit 103 determines whether to perform encoding by setting a reference frame according to the face information output from the face determination unit 102, and outputs the reference frame setting information.

  First, reference frame setting when there is one face in one frame will be described with reference to FIG. In the case of FIG. 6, the switch 305 is in an OFF state, and the reference frame setting determination unit 103 does not use the “face ID” output from the face recognition unit 303.

  FIG. 6 is a diagram showing temporal changes in facial expression index and reference frame setting. In the example of FIG. 6, when the user sets a facial expression index threshold and the facial expression index exceeds the facial expression index threshold, the reference frame setting determination unit 103 outputs reference frame setting information.

  It is assumed that the facial expression index threshold is set to “8” in the example of FIG. When a video signal is input to the encoding unit 101 and the face determination unit 102 at time t0, encoding processing and face information output are started.

  Then, at time t1, the facial expression index exceeds the facial expression index threshold, and at time t2, the facial expression index maintains a state where it exceeds the facial expression index threshold for a predetermined period (t2-t1 period). Therefore, the reference frame setting determination unit 103 determines to set a reference frame, and outputs the reference frame setting information to the encoding unit 101. In addition, during the period from time t3 to time t4, since the expression index does not exceed the expression index threshold, the reference frame setting determination unit 103 does not output the reference frame setting information.

  At time t4, the facial expression index exceeds the facial expression index threshold, but the facial expression index is smaller than the facial expression index threshold at time t5 without maintaining the state where the facial expression index exceeds the facial expression index threshold for a predetermined period. It has become. For this reason, in such a case, the reference frame setting determination unit 103 does not output the reference frame setting information.

  As described above, when the period during which the facial expression index exceeds the facial expression index threshold is a short period, the minimum necessary frame can be set as the reference frame by not setting the reference frame. Needless to say, however, the reference frame may be set even when the period during which the facial expression index exceeds the facial expression index threshold is short. In this case, when the facial expression index exceeds the facial expression index threshold, the reference frame setting determination unit 103 immediately determines that the reference frame is set, and outputs the reference frame setting information to the encoding unit 101.

  Next, reference frame setting when there are a plurality of faces present in one frame as face information will be described with reference to FIG. In the case of FIG. 7, the switch 305 is in an OFF state, and the reference frame setting determination unit 103 does not use the “face ID” output from the face recognition unit 303.

  FIG. 7 is a diagram showing temporal changes in expression indices and reference frame settings for three faces. In the example of FIG. 7, the user sets the expression index threshold of the expression index and the face number threshold of the number of faces, and the reference frame is set when the number of faces whose expression index exceeds the expression index threshold exceeds the face number threshold. The determination unit 103 outputs reference frame setting information.

  In the example of FIG. 7, the facial expression index threshold is set to a first threshold that sets “8” as the first threshold. The face number threshold of the number of faces is set to a second threshold value that sets “3” as the second threshold value. When a video signal is input to the encoding unit 101 and the face determination unit 102 at time t0, encoding processing and face information output are started.

  In the period from time t1 to time t2, since the facial expression index of one face exceeds the facial expression index threshold, but does not reach the face number threshold “3”, the reference frame setting determination unit 103 stores the reference frame setting information. Do not output.

  At time t3, the facial expression indexes of the three faces simultaneously exceed the facial expression index threshold, and at time t4, the state where the facial expression indexes of the three faces exceed the facial expression index threshold for a predetermined period (t4-t3 period) is maintained. Yes. For this reason, the reference frame setting determination unit 103 determines to set a reference frame at time t4, and outputs the reference frame setting information to the encoding unit 101.

  Next, reference frame setting according to the face direction when there are a plurality of faces in one frame will be described with reference to FIG. In the case of FIG. 8, the switch 305 is in the OFF state, and the reference frame setting determination unit 103 does not use the “face ID” output from the face recognition unit 303.

  FIG. 8 is a diagram showing temporal changes in expression indices and directions and reference frame settings for three faces. In the example of FIG. 8, the user sets a facial expression index threshold for the facial expression index and a face number threshold for the number of faces. When the total number of faces facing in the same direction among the faces whose facial expression index exceeds the facial expression index threshold exceeds the face number threshold, that is, the second threshold, the reference frame setting determination unit 103 Output frame setting information.

  In the example of FIG. 8, it is assumed that the facial expression index threshold is set to “8” and the face number threshold is set to “3”. When a video signal is input to the encoding unit 101 and the face determination unit 102 at time t0, encoding processing and face information output are started. In the period from time t1 to time t2, the facial expression indexes of the three faces exceed the facial expression index threshold. However, the reference frame setting determination unit 103 of the present embodiment performs face direction detection that detects the number of faces for each face direction. Therefore, when the face direction is different from “right”, “front”, and “left”, it is determined that the prohibition condition prohibiting the reference that skips the frame is satisfied, and the reference that prohibits the jump reference is determined. Set the frame.

  At time t3, the orientations of the two faces are changed, and the directions of the three faces are all “right”. At the time t4, the facial expression indices of the three faces facing the same “right” direction simultaneously exceed the facial expression index threshold, and at the time t5, the facial expression indices 3 are directed to the same “right” direction for a predetermined period (t5-t4 period). The facial expression index of one face is over the expression index threshold. Therefore, the reference frame setting determination unit 103 determines to set a reference frame, and outputs the reference frame setting information to the encoding unit 101.

  Next, reference frame setting according to main face information determined from face information will be described with reference to FIG. In the case of FIG. 9, the switch 305 is in an OFF state, and the reference frame setting determination unit 103 does not use the “face ID” output from the face recognition unit 303.

  FIG. 9 is a diagram showing temporal changes in facial expression indices and main face information and reference frame settings for three faces. In the example of FIG. 9, when the user sets a facial expression index threshold for the facial expression index, and the facial expression index exceeds the facial expression index threshold, the reference frame setting determination unit 103 outputs the reference frame setting information.

  In the example of FIG. 9, it is assumed that the facial expression index threshold is set to “8”. The main face is a face that the viewer (user) pays attention to. In the present embodiment, an example in which the reference frame setting determination unit 103 performs main face determination from the center coordinates, size, and direction of the face included in the face information will be described, but the main face determination is not limited to this. For example, in this embodiment, the reference frame setting determination unit 103 determines a face whose face center coordinates are close to the center of the frame, the face size is large, and the face direction is the front, as the main face.

  When a video signal is input to the encoding unit 101 and the face determination unit 102 at time t0, encoding processing and face information output are started. In the period from time t1 to time t2, the facial expression index of one face exceeds the facial expression index threshold value, but since it is not determined to be the main face, the reference frame setting determination unit 103 outputs the reference frame setting information. do not do.

  Also, in the period from time t3 to time t4, one face is determined to be the main face, but since the expression index does not exceed the expression index threshold, the reference frame setting determination unit 103 outputs the reference frame setting information. do not do.

  At time t5, one face is determined as the main face, and the expression index of the face determined as the main face is maintained in a state where it exceeds the expression index threshold for a predetermined period (t6-t5 period). . Therefore, the reference frame setting determination unit 103 determines to set a reference frame, and outputs the reference frame setting information to the encoding unit 101.

  Next, reference frame setting according to the “face ID” included in the face information will be described with reference to FIG. In the case of FIG. 10, the switch 305 is in the ON state, and the reference frame setting determination unit 103 uses the “face ID” output from the face recognition unit 303.

  FIG. 10 is a diagram showing temporal changes in expression indices and reference frame settings for three faces that can be identified by “face ID”. In the example of FIG. 10, the user sets a facial expression index threshold for the facial expression index, and when the facial expression index exceeds the facial expression index threshold, the reference frame setting determination unit 103 outputs the reference frame setting information. In the example of FIG. 10, it is assumed that the facial expression index threshold is set to “8”.

  When a video signal is input to the encoding unit 101 and the face determination unit 102 at time t0, encoding processing and face information output are started. At time t1, the facial expression index of “face ID 0” exceeds the facial expression index threshold, and at time t2, the facial expression index of face ID 0 exceeds the facial expression index threshold for a predetermined period (t2-t1 period). ing. Therefore, the reference frame setting determination unit 103 determines to set a reference frame, and outputs the reference frame setting information to the encoding unit 101.

  At time t3, the expression index of “Face ID1” exceeds the expression index threshold, and at time t4, the expression index of “Face ID1” exceeds the expression index threshold for a predetermined period (t4-t3 period). Is maintained. Therefore, the reference frame setting determination unit 103 determines to set a reference frame, and outputs the reference frame setting information to the encoding unit 101.

  At time t5, the expression index of “face ID 0” exceeds the expression index threshold, and at time t6, the expression index of “face ID 0” exceeds the expression index threshold within a predetermined period (t6 to t5 period). Is maintained. However, since the predetermined period has not elapsed since the reference frame setting at the previous time t2, the reference frame setting determination unit 103 determines prohibition unless the reference frame is set, and the reference frame setting information is sent to the encoding unit 101. Do not output.

  Thus, if the same face is within a predetermined period from the previous reference frame setting, the reference frame setting determination unit 103 sets the minimum necessary frame as the reference frame by not outputting the reference frame setting information. be able to.

  Note that the reference frame setting determination unit 103 may determine to set a reference frame in accordance with the “expression type” included in the face information. For example, it may be determined that the reference frame is set only when the facial expression index of the smile exceeds the expression index threshold, and it is not determined that the reference frame is set even if the facial expression index of the crying face exceeds the expression index threshold.

Next, a process for setting a reference frame according to face information will be described with reference to FIG.
First, in step S1101, the reference frame setting method selected by the user is determined. The reference frame setting method is a method used by the reference frame setting determination unit 103 for setting determination of a reference frame.

  In the present embodiment, there is a reference frame setting method described with reference to FIGS. 6, 7, 8, 9, and 10. In step S <b> 1102, the video signal is input to the encoding unit 101 and the face determination unit 102. In step S1103, the face determination unit 102 performs face determination processing by analyzing the video signal, and outputs the processing result as face information.

  In step S1104, the reference frame setting determination unit 103 determines whether to set a reference frame based on the face information and the reference frame setting method selected by the user. If it is determined that the reference frame is set as a result of the determination, the process proceeds to step S1105, and the encoding unit 101 sets the reference frame.

  On the other hand, as a result of the determination in step S1104, if the reference frame setting determination unit 103 determines not to set a reference frame based on the face information and the reference frame setting method selected by the user, the process proceeds to step S1106. In step S1106, the encoding unit 101 sets a picture type conforming to the encoding method.

  Note that the encoding unit 101 may include a code amount control unit that controls a generated code amount, and may perform code amount control for a reference frame set according to reference frame setting information. At this time, the code amount of the reference frame set based on the face information may be larger than the reference frame based on other conditions such as set in accordance with the encoding method. Thereby, for example, the image quality of the reference frame set in the smile scene is improved as compared with the normal reference frame.

  In the embodiment described above, the input image signal is H.264. The picture type of the reference frame is set to the IDR picture by compressing and coding in accordance with the H.264 coding system. Then, when it is determined that the face information meets the prohibition condition for prohibiting the reference that skips frames, a reference frame that prohibits the jump reference is set. However, the input image signal may be compressed and encoded in accordance with the MPEG encoding method, and the picture type of the reference frame may be set to I picture.

  According to the present embodiment, encoding is performed by setting the minimum necessary image frame as a reference frame in accordance with the degree of facial expression. As a result, it is possible to obtain an encoded stream that can be quickly reproduced and easily edited from a scene with a high degree of facial expression while suppressing a decrease in encoding efficiency as compared with the related art.

(Other embodiments according to the present invention)
Each means constituting the image coding apparatus according to the above-described embodiment of the present invention can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable recording medium recording the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowchart shown in FIG. 11) for executing each step in the above-described image encoding method is directly or remotely supplied to a system or apparatus. In addition, this includes a case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Various recording media can be used as a recording medium for supplying the program. For example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD- R).

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer may perform part or all of the actual processing. The functions of the above-described embodiments can be realized.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

1 is a block diagram illustrating an exemplary configuration of an image encoding device according to an embodiment of the present invention. It is a block diagram which shows embodiment of this invention and shows the structural example of an encoding part. It is a block diagram which shows embodiment of this invention and shows the structural example of a face determination part. It is a figure which shows embodiment of this invention and demonstrates a mode that face information is output from a video signal. It is a figure which shows embodiment of this invention and demonstrates an example of the content of face information. It is a figure which shows embodiment of this invention and demonstrates the 1st example of the reference | standard frame setting method. It is a figure which shows embodiment of this invention and demonstrates the 2nd example of the reference | standard frame setting method. It is a figure which shows embodiment of this invention and demonstrates the 3rd example of the reference | standard frame setting method. It is a figure which shows embodiment of this invention and demonstrates the 4th example of the reference | standard frame setting method. It is a figure which shows embodiment of this invention and demonstrates the 5th example of the reference | standard frame setting method. It is a flowchart which shows embodiment of this invention and demonstrates the control procedure of this invention. A conventional example is shown. 2 is a diagram illustrating selection of a reference image used for picture type and inter-frame prediction in H.264. It is a figure which shows a prior art example and demonstrates an IDR picture.

DESCRIPTION OF SYMBOLS 101 Encoding part 102 Face determination part 103 Reference | standard frame setting determination part 201 Frame rearrangement part 202 Subtractor 203 Integer conversion part 204 Quantization part 205 Entropy encoding part 206 Inverse quantization part 207 Inverse integer conversion part 208 Adder 209 1st 1 frame memory 213 2nd frame memory 210 intra prediction unit 211 first switch 217 second switch 212 deblocking filter 214 inter prediction unit 215 motion detection unit 216 picture type determination unit 301 face detection unit 302 face recognition history data Recording unit 303 Face recognition unit 304 Expression determination unit 305 Switch

Claims (17)

In an image encoding device that compresses and encodes an input image signal composed of a plurality of frames,
Face information creating means for analyzing the input image signal and creating face information for identifying a face;
Encoding means for compressing and encoding the input image signal using an inter-frame prediction method;
Prohibition to determine whether to prohibit reference in inter-frame prediction that skips over the encoding target frame based on the face information generated by the face information generation unit with respect to the encoding target frame in the encoding unit A determination means;
A setting unit configured to set the encoding target frame as a reference frame that prohibits jumping reference when it is determined by the prohibition determination unit that the reference skipping the encoding target frame is prohibited. Image encoding device. Face detection means for detecting a face from a frame included in the input image signal;
Facial expression determination means for determining facial expression detected by the face detection means,
2. The image according to claim 1, wherein the prohibition determination unit determines whether or not to prohibit the reference skipping the encoding target frame according to the degree of the facial expression determined by the facial expression determination unit. Encoding device.   The image coding apparatus according to claim 2, wherein the face detection unit calculates at least a reference coordinate of the face and a size of the face.   3. The image coding apparatus according to claim 2, wherein the facial expression determination unit calculates a facial expression index representing the degree of facial expression in a plurality of stages, and determines the facial expression based on the facial expression index. The facial expression determination means includes first threshold value setting means for setting a first threshold value related to the facial expression index,
The image encoding device according to claim 4, wherein the prohibition determination unit determines that reference that skips the encoding target frame is prohibited when the facial expression index exceeds the first threshold. The facial expression determination means includes second threshold setting means for setting a second threshold relating to the number of faces in a frame included in the input image signal,
The prohibition determination unit determines that reference that skips the encoding target frame is prohibited when the total number of faces whose facial expression index exceeds the first threshold exceeds the second threshold. The image encoding device according to claim 5.   The image coding apparatus according to claim 4, wherein the face detection unit includes a face direction detection unit that detects the number of faces for each face direction.   The prohibition determination unit is configured such that, among faces whose facial expression index exceeds the first threshold, the number of faces facing the same direction detected by the face direction detection unit exceeds a preset threshold. The image encoding apparatus according to claim 7, wherein it is determined that reference that skips the encoding target frame is prohibited. A main face determination means for determining a main face that is a face that is viewed by the viewer;
The prohibition determining means determines that reference that skips the encoding target frame is prohibited when the facial expression index of the main face determined by the main face determination means exceeds the first threshold. The image encoding device according to claim 5.   The said prohibition determination means determines to prohibit the reference which skipped the said encoding object frame, when the state where the said facial expression index exceeds the said 1st threshold value is maintained for a predetermined period. The image encoding device described. Face recognition means for identifying the face detected by the face detection means and determining whether or not the first face of the current frame matches the second face of the past frame;
Within a predetermined period from the frame of the second face determined to prohibit the reference that skipped the encoding target frame by the prohibition determination unit, the prohibition determination unit determines that the face recognition unit and the second face are The image coding apparatus according to claim 2, wherein the first face determined to match is not determined to prohibit interlaced reference.   The image coding apparatus according to claim 2, wherein the facial expression determination means determines at least one facial expression among a smile, a crying face, and an angry face. The encoding means includes code amount control means for controlling the amount of code generated,
The code amount control unit is configured to set a code amount for the reference frame to be set based on other conditions when the prohibition determining unit determines that the reference that skips the encoding target frame is prohibited. The image encoding device according to claim 1, wherein the image encoding device is also increased. The encoding means converts the input image signal to H.264. H.264 encoding and compression encoding,
2. The image encoding apparatus according to claim 1, wherein the picture type of the reference frame is set to an IDR picture. The encoding means compresses and encodes the input image signal in accordance with an MPEG encoding method,
The picture coding apparatus according to claim 1, wherein the picture type of the reference frame is set to I picture. In an image encoding method for compressing and encoding an input image signal composed of a plurality of frames,
A face information creation step of analyzing the input image signal and creating face information for identifying a face;
An encoding step of compressing and encoding the input image signal using an inter-frame prediction method;
Prohibition of determining whether to prohibit reference in inter-frame prediction that skips over the encoding target frame based on the face information generated in the face information generation step with respect to the encoding target frame in the encoding step A determination process;
And a setting step of setting the encoding target frame to a reference frame that prohibits jumping reference when it is determined in the prohibition determination step that the reference skipping the encoding target frame is prohibited. Image coding method. In a computer program for causing a computer to execute a process of compressing and encoding an input image signal composed of a plurality of frames,
A face information creation step of analyzing the input image signal and creating face information for identifying a face;
An encoding step of compressing and encoding the input image signal using an inter-frame prediction method;
Prohibition of determining whether to prohibit reference in inter-frame prediction that skips over the encoding target frame based on the face information generated in the face information generation step with respect to the encoding target frame in the encoding step A determination process;
Causing the computer to execute a setting step of setting the encoding target frame to a reference frame that prohibits jumping reference when it is determined in the prohibition determination step that the reference skipping the encoding target frame is prohibited. A featured computer program.

Images