JP5173858B2 - Moving image processing apparatus, moving image processing method, program, and storage medium - Google Patents

Description

  The present invention relates to a moving image processing apparatus, a moving image processing method, a program, and a storage medium, and in particular, a moving image processing apparatus capable of selecting a still image including a specific facial expression from moving image data including a human face, The present invention relates to a moving image processing method, a program, and a storage medium.

  In the field of digital cameras, there is a technology that detects the person's face from the video being shot during framing and automatically releases the shutter when a specific facial expression such as a smile is detected. Are known.

  For example, Patent Document 1 discloses an apparatus that detects a face image using each partial image cut out from a captured image. This apparatus calculates an index value indicating the probability that each partial image is a face image, extracts partial images whose index value is equal to or greater than a predetermined threshold as face image candidates, and determines threshold values of index values for all candidates. To detect a face image.

  Japanese Patent Application Laid-Open No. 2004-228561 detects a human face from a captured image, evaluates the facial expression of the detected face, calculates a facial expression evaluation value, and the facial expression evaluation value exceeds a predetermined threshold value. Discloses an apparatus for recording an image signal on a recording medium. The facial expression evaluation value indicates the degree of proximity to one facial expression between different facial expressions such as a smile and a normal facial expression.

JP 2007-108990 A JP 2008-042319 A

  However, although the apparatus of Patent Document 1 can detect a human face from an image, it cannot determine a facial expression and records a still image when a predetermined facial expression such as a smile is obtained. Control is impossible.

  On the other hand, the apparatus of Patent Document 2 can be controlled to record a still image when a predetermined facial expression such as a smile appears using the facial expression evaluation value. However, since the device of Patent Document 2 records an image when the facial expression evaluation value exceeds a threshold value, if the facial expression evaluation value further increases after recording, an image with a better facial expression cannot be recorded. There is a problem.

  An object of the present invention is to provide a moving image processing apparatus, a moving image processing method, a program, and a storage medium that can select a still image having a higher expression evaluation value from input moving image data.

In order to achieve the above object, a moving image processing apparatus according to the present invention includes a face detection unit that detects a human face from a plurality of image frame data constituting moving image data, and a facial expression in which the detected facial expression is a specific expression. A facial expression evaluation unit for calculating a facial expression evaluation value indicating the degree of proximity to the facial expression evaluation unit, and a control unit for executing processing for obtaining still image data when a high facial expression evaluation value is calculated by the facial expression evaluation unit In the moving image processing apparatus, the difference between the expression evaluation value for the image frame data currently being processed and the expression evaluation value for the image frame data at a time point earlier than the image frame data being currently processed is calculated, and the calculated difference value with the evaluation difference value determination unit that determines whether the difference is less than a predetermined reference value, the control unit, the difference value which the evaluation difference value determination unit is calculated smaller than a predetermined reference value And executes the process of acquiring the still image data when it is determined that.

In order to achieve the above object, a moving image processing method of the present invention includes a face detection step of detecting a human face from a plurality of image frame data constituting moving image data, and the detected facial expression is a specific expression. A facial expression evaluation step for calculating a facial expression evaluation value indicating the degree of proximity to the facial expression evaluation step, and a control step for executing a process for acquiring still image data when a high facial expression evaluation value is calculated by the facial expression evaluation step In the moving image processing method, a calculation step for calculating a difference value between the expression evaluation value for the image frame data being currently processed and the expression evaluation value for the image frame data at a time point before the image frame data being currently processed difference value is a determining evaluation difference value determination step of determining whether or not smaller than the predetermined reference value, in the control step, the And executes the processing of the difference value calculated by the price difference value determining step obtains the still image data when it is determined that the predetermined reference value smaller.

To achieve the above object, the program of the present invention includes a face detection step for detecting a human face from a plurality of image frame data constituting moving image data, and how much the detected facial expression is a specific facial expression. A moving image processing comprising a facial expression evaluation step for calculating a facial expression evaluation value indicating a degree of closeness, and a control step for executing processing for acquiring still image data when a high facial expression evaluation value is calculated by the facial expression evaluation step In a program for causing a computer to execute the method, a calculation step of calculating a difference value between a facial expression evaluation value for the currently processed image frame data and a facial expression evaluation value for the image frame data at a time point before the currently processed image frame data When the calculated difference value is a determining evaluation difference value determination step of determining whether or not smaller than a predetermined reference value For example, in the control step, and executes a process of the evaluation difference value determination difference value calculated in the step to obtain the still image data when it is determined that the predetermined reference value smaller.

  According to the present invention, still image data having a higher expression evaluation value can be acquired as compared with the conventional technique in which an image is simply selected by comparison with a threshold value.

1 is a block diagram schematically showing a configuration of a moving image processing apparatus according to a first embodiment of the present invention. It is a flowchart which shows operation | movement of the evaluation difference value determination part in FIG. It is a graph which shows the change with time of the facial expression evaluation value for demonstrating the effect | action of the evaluation difference value determination part in FIG. It is a flowchart which shows operation | movement of the evaluation difference value determination part of the moving image processing device which concerns on the 2nd Embodiment of this invention. It is a graph which shows the change with time of the facial expression evaluation value for demonstrating the effect | action of the evaluation difference value determination part of 2nd Embodiment. It is a flowchart which shows operation | movement of the evaluation difference value determination part of the moving image processing device which concerns on the 3rd Embodiment of this invention. It is a graph which shows the change with time of the facial expression evaluation value for demonstrating the effect | action of the evaluation difference value determination part of 3rd Embodiment. It is a block diagram which shows roughly the structure of the moving image processing apparatus which concerns on the 4th Embodiment of this invention. It is a block diagram which shows the detailed structure of the evaluation difference value determination part in FIG. It is a flowchart which shows the process of the evaluation difference value integration part in FIG. FIG. 9 is a graph in which changes in facial expression evaluation values output from the facial expression evaluation unit for each image frame data are plotted for each of the three facial image regions in order to explain a specific example of processing of the moving image processing apparatus in FIG. An example is shown in which three people entering three face areas are smiling at a timing close to each other. FIG. 9 is a graph in which changes in facial expression evaluation values output from the facial expression evaluation unit for each image frame data are plotted for each of the three facial image regions in order to explain a specific example of processing of the moving image processing apparatus in FIG. An example is shown in which one of three people in three face areas is smiling.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, the moving image processing apparatus according to the first embodiment of the present invention will be described.

  FIG. 1 is a block diagram schematically showing a configuration of a moving image processing apparatus 100 according to the first embodiment of the present invention. The moving image processing apparatus 100 in FIG. 1 receives the captured moving image data, includes a person from a plurality of image frame data constituting the moving image data, and the expression of the person is a specific expression, Then, image frame data indicating a smile is detected. The moving image processing apparatus 100 has a function of executing a process of acquiring still image data and recording it on a recording medium when smile image data is detected.

  The moving image processing apparatus 100 includes a control unit 110 for controlling the entire apparatus, and an imaging unit 120 for inputting moving image data and still image data. The moving image processing apparatus 100 includes a smile image detection unit 130 that detects an image in which a person's smile is recorded from image frame data constituting the moving image data input from the imaging unit 120, and a smile image detection unit 130 that is input from the imaging unit 120. And an image encoding unit 140 that encodes still image data. Furthermore, the moving image processing apparatus 100 includes a recording medium 150 such as a memory card or a hard disk for recording the encoded image data, and an operation unit 160 including switches for the user to operate. .

  The control unit 110 controls the photographing unit 120 to input image data in accordance with the operation of the operation unit 160 by the user, and controls the smile image detection unit 130 and the image encoding unit 140.

  The imaging unit 120 includes an imaging lens and an imaging element such as a CMOS sensor that detects a subject image formed by the imaging lens, and outputs image data as continuous image frame data. Note that the imaging unit 120 switches the shooting mode between the moving image mode and the still image mode according to an instruction from the control unit 110. In the moving image mode, images are continuously captured at a relatively low resolution, and in the still image mode, a single image is captured at a relatively high resolution.

  The smile image detection unit 130 detects a person's face from the image frame data captured in the moving image mode, and how close the detected facial expression is to a specific expression (smile in this example). An expression evaluation unit 132 that calculates an expression evaluation value indicating the degree of the expression. The smile image detection unit 130 includes an evaluation difference value determination unit 133. The evaluation difference value determination unit 133 calculates a difference value between the expression evaluation value for the image frame data currently being processed and the expression evaluation value for the image frame data at a time point before the image frame data currently being processed. It is determined whether or not the difference value satisfies a predetermined condition.

  The control unit 110 performs processing for acquiring still image data when the evaluation difference value determination unit 133 determines that a predetermined condition is satisfied among the image frame data having a high expression evaluation value calculated by the expression evaluation unit 132. It has a function of executing and recording on a recording medium. When it is determined that the predetermined condition is satisfied, the control unit 110 outputs a recording instruction to the imaging unit 120 and the image encoding unit 140 at that timing. As a result, the imaging unit 120 captures an image in the still image mode, and the image data is encoded as still image data by the image encoding unit 140 and recorded on the recording medium 150.

  Next, the operation of the moving image processing apparatus 100 configured as described above will be described. When the user turns on the moving image processing apparatus 100 by operating the operation unit 160, the control unit 110 initializes the imaging unit 120 and then controls the imaging unit 120 to start shooting in the moving image mode. The captured moving image is previewed on a display unit such as an LCD panel (not shown). In preview display, a fixed number of frames are displayed per second, and the captured moving image data is discarded after being displayed.

  When the user operates the operation unit 160 to instruct the start of recording, the control unit 110 initializes the smile image detection unit 130 and then inputs image frame data of moving image data captured by the imaging unit 120 to the face detection unit 131. (Input step).

  The face detection unit 131 analyzes the input image frame data and determines whether or not a human face exists in the image frame data (still image) (face detection step). The method described in Patent Document 1 can be used for the determination. When the face detection unit 131 determines that a face exists in the input image frame data, the face detection unit 131 identifies the face area, and calculates a face reliability value indicating the probability that the object in the face area is a human face. (Face reliability calculation step).

  Information on the face area in the image frame data detected by the face detection unit 131 is input to the expression evaluation unit 132, and a face reliability value for the face region is input to the evaluation difference value determination unit 133. The facial expression evaluation unit 132 calculates a facial expression evaluation value indicating the degree of smile on the input face area image (facial expression evaluation step). The method described in Patent Document 2 can be used for calculating the facial expression evaluation value. The evaluation difference value determination unit 133 adjusts the threshold based on the input face reliability value (threshold setting step). Specifically, when the face reliability value calculated by the face detection unit 131 is large, the threshold value is changed to a small value, and when the face reliability value is small, the threshold value is changed to a large value.

  The facial expression evaluation value calculated by the facial expression evaluation unit 132 is input to the evaluation difference value determination unit 133. The evaluation difference value determination unit 133 determines whether or not the input facial expression evaluation value is larger than a threshold value, and if so, the input facial expression evaluation value and the facial expression evaluation value for the previous image frame data Is calculated (calculation step). Then, it is determined whether or not the calculated difference value satisfies a predetermined condition (evaluation difference value determination step). Here, for example, when the difference value is smaller than a certain reference value, it is determined that a predetermined condition is satisfied. When the evaluation difference value determination unit 133 determines that the predetermined condition is satisfied (the difference value is smaller than the reference value), the evaluation difference value determination unit 133 outputs a smile detection signal to the control unit 110 and does not satisfy the predetermined condition (the difference value is If it is determined that it is greater than the reference value, a smile non-detection signal is output to the control unit 110. In general, the change in facial expression is continuous, and when the facial expression evaluation value approaches the maximum value, the difference value of the facial expression evaluation value from the immediately preceding image frame data is considered to be small. Therefore, by determining the acquisition timing of still image data based on the difference value, still image data with a facial expression evaluation value close to the maximum value can be acquired. Details of the operation of the evaluation difference value determination unit 133 will be described later.

  When the smile detection signal is input, the control unit 110 executes processing for acquiring still image data (control step). Specifically, the control unit 110 switches the photographing unit 120 to the still image mode, acquires image data, and inputs the image data to the image encoding unit 140. The image encoding unit 140 performs still image encoding processing such as JPEG compression on the input image data. The control unit 110 records the still image data subjected to the encoding process on the recording medium 150 as a still image file. When the recording of the still image file is completed, the control unit 110 ends the shooting, switches the mode of the shooting unit 120 to the moving image mode, and restarts the preview display.

  On the other hand, when the face detection unit 131 determines that the face of the person does not exist in the image frame data, the face detection unit 131 notifies the expression evaluation unit 132 that there is no face area, and the expression evaluation unit 132 determines the evaluation difference value. The determination unit 133 is notified that no face area exists. The evaluation difference value determination unit 133 outputs a smile non-detection signal to the control unit 110 because the face area does not exist in the image frame data. The control unit 110 ends the process for the image frame data to be processed, and starts the process for the next image frame data.

  FIG. 2 is a flowchart showing the operation of the evaluation difference value determination unit 133 in FIG. Hereinafter, the operation of the evaluation difference value determination unit 133 will be described using the flowchart of FIG. The process of FIG. 2 is executed for each image frame data until a smile image is detected when the user instructs to record a smile image.

  First, the evaluation difference value determination unit 133 initializes an evaluation value holding variable REv, which is an internal variable for holding the facial expression evaluation value of the immediately preceding image frame data necessary for obtaining the difference value, to −1 (Step S1). S101). Subsequently, the evaluation difference value determination unit 133 determines whether or not the facial expression evaluation value for the image frame data currently being processed is input from the facial expression evaluation unit 132 (step S102). If no facial area is detected from the image frame data and no facial expression evaluation value is input (No in step S102), the evaluation value holding variable REv is initialized to −1 (step S111), and the controller 110 smiles. A non-detection signal is output (step S112). Then, the process returns to step S102 to process the next image frame data. When the facial expression evaluation value is input (Yes in step S102), the facial expression evaluation value related to the current image frame data to be processed input from the facial expression evaluation unit 132 is set in the evaluation value variable Ev that is an internal variable. (Step S103). Then, the face reliability value input from the face detection unit 131 is set to a face reliability variable Rv that is an internal variable (step S104).

  Next, the evaluation difference value determination unit 133 determines whether or not the evaluation value holding variable REv is larger than −1 (step S105). When the evaluation value holding variable REv is not larger than −1 (No in step S105), the variable has just been initialized, the evaluation value relating to the immediately preceding image frame data is not stored, and the difference value cannot be obtained. . Therefore, the value of the evaluation value variable Ev is set in the evaluation value holding variable REv (step S113), a smile non-detection signal is output to the control unit 110 (step S112), and the next image frame data is returned to step S102. Process.

  On the other hand, if the evaluation value holding variable REv is larger than −1 (Yes in step S105), it means that the evaluation value related to the immediately previous image frame data is stored, and therefore the threshold is set according to the value of the face reliability variable Rv. A value THv is determined (step S106). In the present embodiment, according to the correspondence shown in Table 1 below, any one of the constants THv0, THv1, and THv2 is set as the threshold value THv according to the value of the face reliability variable Rv. That is, the threshold value THv decreases when the value of the face reliability variable Rv is large, and the threshold value THv increases when the value of the face reliability variable Rv is small.

  Next, the evaluation difference value determination unit 133 takes the difference between the value of the evaluation value holding variable REv and the value of the evaluation value variable Ev, and sets it in the difference value variable D (step S107). As a result, a difference value between the expression evaluation value of the image frame data currently being processed and the expression evaluation value of the immediately previous image frame data is obtained.

  Then, the evaluation difference value determination unit 133 determines whether or not the value of the evaluation value variable Ev exceeds the threshold value THv (step S108). In the present embodiment, when it is determined that the value of the evaluation value variable Ev exceeds the threshold value THv (Yes in step S108), it is determined that the detected facial area expression is a smile. In step 206, a threshold value is set in accordance with the face reliability value. Since a threshold value for smile detection is increased for a face region having a small face reliability value, the false detection rate may be lowered. it can.

  If the facial expression evaluation value is smaller than the threshold value (No in step S208), it is determined that the face area is not a smile, and the process returns to step S102 through steps S113 and S112 to process the next image frame data. .

  If it is determined that the face is a smile (Yes in step S108), the evaluation difference value determination unit 133 determines whether or not the value of the difference value variable D is greater than a predetermined reference value Dc (step S109). When the value of the difference value variable D is not smaller than the reference value Dc (No in Step S109), it is considered that the facial expression evaluation value has not reached a sufficiently high value, and the process returns to Step S102 through Steps S113 and S112. Process next image frame data. If the value of the difference value variable D is smaller than the reference value Dc (Yes in step S109), it is considered that the facial expression evaluation value has reached a sufficiently high value, and a smile detection signal is output to the control unit 110 (step S110). ) End the process.

  According to the processing in FIG. 2, it is possible to detect suitable smile image frame data from the moving image data input from the imaging unit 120 based on the difference value of the facial expression evaluation values. Therefore, by taking a still image at the time when the image frame data is detected, a still image with a suitable smile can be taken.

  Next, a specific example of the process of FIG. 2 will be described with reference to FIG. FIG. 3 is a graph showing the change of the facial expression evaluation value with time for explaining the operation of the evaluation difference value determination unit 133 in FIG. The polygonal line in the figure is a plot of facial expression evaluation values output for each image frame data including a face image from the facial expression evaluation unit 132 with black circles. What is the facial expression evaluation value in the captured moving image data? Shows how it changes. In this example, it is assumed that the same face reliability is output from the face detection unit 131 for all image frame data. Therefore, the threshold value is constant.

  When the user instructs to record an image, the smile image detection unit 130 is initialized, and image frame data is sequentially input from the imaging unit 120 to the smile image detection unit 130. The input image frame data undergoes processing in the face detection unit 131 and the facial expression evaluation unit 132, and the facial expression evaluation values shown in FIG. 3 are sequentially input to the evaluation difference value determination unit 133. For the first facial expression evaluation value shown in FIG. 3, since the image frame data immediately before the difference value calculation target is not input, it is determined No in step S105 of FIG. 2, and the process proceeds to step S112 to smile. A non-detection signal is output. Thereafter, the facial expression evaluation value is sequentially input to the evaluation difference value determination unit 133 up to the image frame data of (a), and the difference value with the facial expression evaluation value of the immediately previous image frame data is calculated (step S107). The value is less than the threshold value. For this reason, it is determined No in step S108, and the process proceeds to step S112, where a smile non-detection signal is output. Since the facial expression evaluation value after the image frame data of (b) is larger than the threshold value, it is determined Yes in step S108, and the difference value of the facial expression evaluation value is compared with the reference value Dc in step S109. In the image frame data of (b), (c), and (d), the difference value is larger than the reference value Dc. Therefore, it is determined in step S109 that the facial expression evaluation value has not reached a sufficiently high value, and the smile is determined in step S112. A non-detection signal is output. Since the difference value becomes smaller than the predetermined value Dc during the processing of the image frame data in (e), it is determined in step S109 that the facial expression evaluation value has reached a sufficiently high value, and the smile detection signal is controlled in step 210. The evaluation difference value determination unit 133 is output to the unit 110 and ends the process.

  When the smile detection signal is input, control unit 110 starts still image shooting processing. When the imaging process is started, the image data output from the imaging unit 120 is not input to the face detection unit 131 but is input to the image encoding unit 140. Therefore, in the period after (e) in FIG. Although the facial expression evaluation value is not calculated, it is shown as an example of the evaluation value when the facial expression evaluation value is calculated. When the shooting process is started, the control unit 110 controls the imaging unit 120 to adjust the aperture, emit the flash, control the exposure time, and the like, so that actual shooting is performed at a timing after (e). Therefore, the image data is output from the imaging unit 120 at a timing delayed from (e), for example, the timing (f), and image data having a large expression evaluation value can be encoded and recorded.

  Next, a moving image processing apparatus according to the second embodiment of the present invention will be described. Since the configuration and operation of the present embodiment are basically the same as those of the first embodiment described above, description of the overlapping configuration and operation will be omitted, and different configurations and operations will be described below.

  In the first embodiment, the reference value Dc to be compared with the difference value of the evaluation value is a constant. In the present embodiment, this reference value is used as the difference value of the facial expression evaluation value calculated when processing the immediately preceding image frame data. Based on this, the value of the reference value Dc is dynamically determined. The difference from the first embodiment is only the processing in the evaluation difference value determination unit 133, and the rest is the same as the configuration shown in FIG.

  FIG. 4 is a flowchart showing the operation of the evaluation difference value determination unit 133 of the moving image processing apparatus according to the second embodiment. Detailed description of steps having the same contents as in FIG. 2 is omitted.

  The evaluation difference value determination unit 133 initializes the evaluation value holding variable REv to −1 (step S201), and holds a difference value for holding the difference value of the facial expression evaluation value calculated when processing the previous image frame data. The variable Dp is initialized to 0 (step S202). Subsequently, the evaluation difference value determination unit 133 determines whether a facial expression evaluation value for the image frame data currently being processed is input (step S203). When the facial expression evaluation value is not input (No in step S203), the evaluation value holding variable REv is initialized to −1 (step S214), the difference value holding variable Dp is initialized to 0 (step S215), and control is performed. A smile non-detection signal is output to unit 110 (step S216). Then, the process returns to step S203 to process the next image frame data. If a facial expression evaluation value has been input (Yes in step S203), the facial expression evaluation value relating to the current image frame data to be processed is set in the evaluation value variable Ev (step S204). Then, the face reliability value input from the face detection unit 131 is set in the face reliability variable Rv (step S205).

  Next, the evaluation difference value determination unit 133 determines whether or not the evaluation value holding variable REv is larger than −1 (step S206). When the evaluation value holding variable REv is not larger than −1 (No in Step S206), the value of the evaluation value variable Ev is set in the evaluation value holding variable REv (Step S217). Then, a smile non-detection signal is output to the control unit 110 (step S216), and the process returns to step S203 to process the next image frame data.

  On the other hand, when the evaluation value holding variable REv is larger than −1 (Yes in Step S206), the threshold value THv is determined according to the value of the face reliability variable Rv (Step S207). Then, the difference between the value of the evaluation value holding variable REv and the value of the evaluation value variable Ev is taken and set in the difference value variable D (step S208).

  Then, the evaluation difference value determination unit 133 determines whether or not the value of the difference value holding variable Dp is set, that is, whether or not the difference value of the facial expression evaluation value is calculated for the previous image frame data. Is determined (step S209). When the difference value of the facial expression evaluation value with respect to the previous image frame data is calculated (Yes in step S209), whether or not the value of the evaluation value variable Ev exceeds the threshold value THv, that is, the facial expression Whether or not is a smile is determined (step S210). If it is determined that the user is smiling (Yes in step S210), the reference value Dc is calculated from the value of the difference value holding variable Dp. In this example, a value obtained by multiplying the value of the difference value holding variable Dp by 1/3 as a constant ratio is used as the reference value Dc.

  After setting the reference value Dc, the evaluation difference value determination unit 133 determines whether or not the value of the difference value variable D is smaller than the set reference value Dc (step S212). When the value of the difference value variable D is smaller than the reference value Dc (Yes in step S212), the evaluation difference value determination unit 133 considers that the facial expression evaluation value has reached a sufficiently high value, and detects a smile to the control unit 110. A signal is output (step S213), and the process ends.

  On the other hand, when the difference value of the facial expression evaluation value for the previous image frame data has not been calculated (No in step S209), the value of the difference value variable D is set to the difference value holding variable Dp (step S218). . Thereafter, the evaluation value variable Ev is set to the evaluation value holding variable REv (step S217), a smile non-detection signal is output to the control unit 110 (step S216), and the process returns to step S203 to process the next image frame data. To do. Also when the facial expression evaluation value is smaller than the threshold value (No in Step S210) and when the value of the difference value variable D is larger than the set reference value Dc (No in Step S212), Steps S218 and S217 are performed. , S216 is executed.

  According to the processing in FIG. 4, it is possible to detect suitable smile image frame data from the moving image data input from the imaging unit 120 based on the difference value of the facial expression evaluation values. Therefore, by taking a still image at the time when the image frame data is detected, a still image with a suitable smile can be taken.

  Next, a specific example of the process of FIG. 4 will be described with reference to FIG. FIG. 5 is a graph showing changes in facial expression evaluation values over time for explaining the operation of the evaluation difference value determination unit 133 of the second embodiment shown in FIG. In this example as well, it is assumed that the same face reliability is output from the face detection unit 131 for all image frame data. Therefore, the threshold value is constant.

  When the user instructs image recording, facial expression evaluation values shown in FIG. 5 are sequentially input to the evaluation difference value determination unit 133. For the first facial expression evaluation value shown in FIG. 5, the image frame data immediately before the difference value calculation target is not input. Therefore, No is determined in step S206 in FIG. 4, the first facial expression evaluation value is set in the evaluation value holding variable (step S217), and a smile non-detection signal is output (step S216). Since the previous facial expression evaluation value is set for the second facial expression evaluation value, it is determined as Yes in step S206, and the difference value is calculated (step S208). However, since the difference value with respect to the immediately preceding facial expression evaluation value is not set, it is determined No in step S209, the value of the difference value variable D is set to the difference value holding variable Dp (step S218), and the smile non-detection signal is set. It is output (step S216).

  The facial expression evaluation values from the third to (a) shown in FIG. 5 are determined as Yes in steps S206 and S209, but are not determined in step S210 because the threshold value is not exceeded, and the smile is not determined. A detection signal is output (step S216).

  Since the facial expression evaluation value after (b) in FIG. 5 exceeds the threshold value, it is also determined as Yes in step S210, and the reference value Dc is calculated based on the difference value with respect to the previous facial expression evaluation value (step S211). The difference value of the current facial expression evaluation value is compared (step S212). At this stage, the reference value Dc is obtained by Dc = D (a) / 3 based on the difference value D (a) with respect to the previous facial expression evaluation value (a). However, in the example of FIG. 5, since the difference value D (b) with respect to the facial expression evaluation value of (b) is larger than the determined reference value Dc, it is determined No in step S212, and a smile non-detection signal is output. (Step S216).

  Thereafter, for the facial expression evaluation value in (c), the difference value D (c) is compared with the reference value Dc = D (b) / 3 calculated based on the difference value with respect to the previous facial expression evaluation value in (b). The For the facial expression evaluation value of (d), the difference value D (d) is compared with the reference value Dc = D (c) / 3 calculated based on the difference value with respect to the facial expression evaluation value of the previous (c). The In either case, No is determined in step S212, and a smile non-detection signal is output (step S216).

  At the time of processing the image frame data of (e), the difference value D (e) becomes smaller than the reference value Dc = D (d) / 3 calculated based on the difference value with respect to the immediately preceding facial expression evaluation value of (d). Therefore, Yes is determined in step S212, a smile detection signal is output (step S213), and the evaluation difference value determination unit 133 ends the process.

  When the smile detection signal is input, the control unit 110 controls the image capturing unit 120 to adjust the aperture, emit the flash, control the exposure time, and the like, so that actual shooting is performed at a timing after (e). Done. Therefore, the image data is output from the imaging unit 120 at a timing delayed from (e), for example, the timing (f), and image data having a large expression evaluation value can be encoded and recorded.

  Next, a moving image processing apparatus according to the third embodiment of the present invention will be described. Since the configuration and operation of this embodiment are basically the same as those of the second embodiment described above, description of the redundant configuration and operation will be omitted, and different configurations and operations will be described below.

  In the second embodiment, the reference value Dc to be compared with the difference value of the evaluation value is calculated based on the difference value of the facial expression evaluation value calculated when processing the previous image frame data. On the other hand, in the present embodiment, this reference value is determined based on the average value of the difference values of the facial expression evaluation values calculated at the time of processing a plurality of previous (three in this example) image frame data. The difference from the first and second embodiments is only the processing in the evaluation difference value determination unit 133, and the other configuration is the same as that shown in FIG.

  FIG. 6 is a flowchart illustrating the operation of the evaluation difference value determination unit 133 of the moving image processing apparatus according to the third embodiment. Detailed description of steps having the same contents as those in FIGS. 2 and 4 is omitted.

  The evaluation difference value determination unit 133 initializes the evaluation value holding variable REv to −1 (step S301). Subsequently, the evaluation difference value determination unit 133 stores difference value holding variables Dp1, Dp2, and Dp2 for holding the difference values of the facial expression evaluation values calculated when processing the previous, second, and third previous image frame data. Dp3 is initialized to 0 (steps S302, 303, and 304). Furthermore, the evaluation difference value determination unit 133 determines whether or not a facial expression evaluation value for the image frame data currently being processed is input (step S305). When the facial expression evaluation value is not input (No in step S305), the evaluation difference value determination unit 133 initializes the evaluation value holding variable REv to −1 (step S316), and the difference value holding variables Dp1, Dp2, Dp3 is initialized to 0 (steps S317, 318, and 319). Then, the evaluation difference value determination unit 133 outputs a smile non-detection signal to the control unit 110 (step S320), returns to step S305, and processes the next image frame data. When the facial expression evaluation value is input (Yes in step S305), the evaluation difference value determination unit 133 sets the facial expression evaluation value related to the current image frame data to be processed in the evaluation value variable Ev (step S306). . Then, the evaluation difference value determination unit 133 sets the face reliability value input from the face detection unit 131 to the face reliability variable Rv (step S307).

  Next, the evaluation difference value determination unit 133 determines whether or not the evaluation value holding variable REv is larger than −1 (step S308). When the evaluation value holding variable REv is not larger than −1 (No in Step S308), the evaluation difference value determining unit 133 sets the value of the evaluation value variable Ev in the evaluation value holding variable REv (Step S321). Then, the evaluation difference value determination unit 133 outputs a smile non-detection signal to the control unit 110 (step S320), returns to step S305, and processes the next image frame data.

  On the other hand, when the evaluation value holding variable REv is larger than −1 (Yes in Step S308), the threshold value THv is determined according to the value of the face reliability variable Rv (Step S309). Then, the difference between the value of the evaluation value holding variable REv and the value of the evaluation value variable Ev is taken and set in the difference value variable D (step S310).

  Next, the evaluation difference value determination unit 133 determines whether or not the value of the difference value holding variable Dp3 is set, that is, whether or not the difference value of the facial expression evaluation value is calculated for the previous three image frame data. Is determined (step S311). If the difference value of the facial expression evaluation value with respect to the previous three image frame data has been calculated (Yes in step S311), whether or not the value of the evaluation value variable Ev exceeds the threshold value THv, that is, the facial expression Whether or not is a smile is determined (step S312). If it is determined that the user is smiling (Yes in step S312), the value of the reference value Dc is calculated from the values of the difference value holding variables Dp1, Dp2, and Dp3. In this example, a value obtained by multiplying the average value of the difference value holding variables Dp1, Dp2, and Dp3 by 1/3 as a constant ratio is used as the reference value Dc.

  After setting the reference value Dc, the evaluation difference value determination unit 133 determines whether or not the value of the difference value variable D is smaller than the set reference value Dc (step S314). If the value of the difference value variable D is smaller than the reference value Dc (Yes in step S314), the evaluation difference value determination unit 133 considers that the facial expression evaluation value has reached a sufficiently high value, and detects a smile to the control unit 110. A signal is output (step S315), and the process ends.

  If the difference value of the facial expression evaluation value for the previous three image frame data has not been calculated (No in step S311), the difference value holding variables Dp3, Dp2, and Dp1 are the first difference value Dp2, Dp1, respectively. The value of D is set (steps S322, 323, 324). Then, the value of the evaluation value variable Ev is set in the evaluation value holding variable REv (step S321), a smile non-detection signal is output to the control unit 110 (step S320), and the process returns to step S305 to process the next image frame data. To do. Steps S322 and 323 are also performed when the facial expression evaluation value is smaller than the threshold value (No in Step S312) and when the value of the difference value variable D is larger than the set reference value Dc (No in Step S314). , 324, 321 and 320, the process returns to step S305.

  According to the processing in FIG. 6, it is possible to detect suitable smile image frame data from the moving image data input from the imaging unit 120 based on the difference value of the facial expression evaluation values. For this reason, a still image with a suitable smile can be taken by taking a still image when the image frame data is detected.

  Next, a specific example of the process of FIG. 6 will be described with reference to FIG. FIG. 7 is a graph showing changes with time of facial expression evaluation values for explaining the operation of the evaluation difference value determination unit 133 of the third embodiment shown in FIG. In this example as well, it is assumed that the same face reliability is output from the face detection unit 131 for all image frame data. Therefore, the threshold value is constant.

  When the user instructs image recording, facial expression evaluation values shown in FIG. 7 are sequentially input to the evaluation difference value determination unit 133. For the first facial expression evaluation value shown in FIG. 7, the image frame data immediately before the difference value calculation target is not input. Therefore, No is determined in step S308 of FIG. 4, the first facial expression evaluation value is set in the evaluation value holding variable (step S321), and a smile non-detection signal is output (step S320). Since the previous facial expression evaluation value is set for the second facial expression evaluation value, it is determined Yes in step S308, and the difference value is calculated (step S310). The difference value for is not set. For this reason, it is determined No in step S311, the value of the difference value variable D is set in the difference value holding variable Dp1 (step S324), and a smile non-detection signal is output (step S320). At the time of steps S322 and 323, the values of the difference value holding variables Dp1 and Dp2 remain at the initial value 0. Therefore, even if these processes are finished, the values of the difference value holding variables Dp2 and Dp3 are 0.

  Since the value of the difference value holding variable Dp3 is 0 up to the third facial expression evaluation value in FIG. 7, it is determined No in step S311.

  When the facial expression evaluation value in FIG. 7A is input, Yes is determined in steps S308 and S311, but since the threshold value is not exceeded, No is determined in step S312 and a smile non-detection signal is output. (Step S320).

  Since the facial expression evaluation value after (b) in FIG. 7 exceeds the threshold value, it is also determined as Yes in step S312, and the reference value Dc is calculated based on the difference value with respect to the previous three facial expression evaluation values (step S313). The current facial expression evaluation value is compared with the difference value (step S314). At this stage, Dc = (D (a−2) + D (a−1) based on the difference values D (a−2), D (a−1), and D (a) with respect to the previous three facial expression evaluation values. ) + D (a)) / 9, the reference value Dc is obtained. However, in the example of FIG. 7, since the difference value D (b) with respect to the facial expression evaluation value of (b) is larger than the obtained reference value Dc, No is determined in step S314, and a smile non-detection signal is output. (Step S320).

  Thereafter, for the facial expression evaluation value of (c), the reference value Dc = (D (a−1) + D (a) where the difference value D (c) is calculated based on the difference values with respect to the previous three facial expression evaluation values. + D (b)) / 9. For the facial expression evaluation value of (d), the difference value D (d) is calculated based on the reference value Dc = (D (a) + D (b) + D () based on the difference values for the previous three facial expression evaluation values. c) Compared with / 9. In either case, No is determined in step S314, and a smile non-detection signal is output (step S320).

  At the time of processing the image frame data of (e), the difference value D (e) is a reference value Dc = (D (b) + D (c) + D (d) calculated based on the difference values for the previous three facial expression evaluation values. )) / 9 or less. For this reason, Yes is determined in step S314, a smile detection signal is output (step S315), and the evaluation difference value determination unit 133 ends the process.

  When the smile detection signal is input, the control unit 110 controls the image capturing unit 120 to adjust the aperture, emit the flash, control the exposure time, and the like, so that actual shooting is performed at a timing after (e). Done. Therefore, the image data is output from the imaging unit 120 at a timing delayed from (e), for example, the timing (f), and image data having a large expression evaluation value can be encoded and recorded.

  In the first to third embodiments, it is assumed that the shooting mode is different between the moving image shooting for preview display and the shooting of the still image. A part of the image frame data may be used as it is as still image data. In that case, it is also possible to encode and record the image frame data at the time (e) when the smile detection signal is input.

  Next, a moving image processing apparatus according to the fourth embodiment of the present invention will be described. Since the configuration and operation of the present embodiment are basically the same as those of the first embodiment described above, description of the overlapping configuration and operation will be omitted, and different configurations and operations will be described below.

  The moving image processing apparatuses of the first to third embodiments are assumed to have one face image in the image frame data, but the moving image processing apparatuses of the fourth embodiment It is assumed that there are a plurality of face images in the image frame data.

  FIG. 8 is a block diagram schematically showing the configuration of a moving image processing apparatus 100A according to the fourth embodiment of the present invention, and FIG. 9 shows the detailed configuration of the evaluation difference value determination unit in FIG. It is a block diagram. Although the basic configuration of the entire apparatus is the same as that in FIG. 1, the configuration of the smile image detection unit 170 is different from that of the first embodiment, and therefore, different reference numerals are given. Since other configurations are the same as those in FIG. 1, the same reference numerals are given and redundant descriptions are omitted.

  The smile image detection unit 170 of the present embodiment detects image frame data in which a person's smile is recorded from the moving image data input from the imaging unit 120. The smile image detection unit 170 detects a region of a plurality of human face portions from the image frame data, and a facial expression evaluation indicating the degree of smile with respect to the plurality of face regions detected by the face detection unit 171. A facial expression evaluation unit 172 for calculating the respective values. The smile image detection unit 170 includes an evaluation difference value determination unit 173. The evaluation difference value determination unit 173 calculates, for each face region, a difference value between the facial expression evaluation value for the currently processed image frame data and the facial expression evaluation value for the image frame data at a time point before the currently processed image frame data. It is determined whether or not the difference value satisfies a predetermined condition.

  When the user operates the operation unit 160 to turn on the power of the moving image processing apparatus 100A, the control unit 110 initializes the imaging unit 120 and then controls the imaging unit 120 to start shooting in the moving image mode. The captured moving image is previewed on a display unit (not shown). When the user operates the operation unit 160 to start recording, the control unit 110 initializes the smiley image detection unit 170 and then inputs image frame data of moving image data captured by the imaging unit 120 to the face detection unit 171. To do.

  The face detection unit 171 can detect a plurality of face regions from the input image frame data, analyzes the input image frame data, and determines whether a human face exists in the image frame data. . If the face detection unit 171 determines that there is a face in the input image frame data, the face detection unit 171 identifies the face area and calculates a face reliability value indicating the probability that the image in the face area is a human face. . When there are a plurality of faces, a plurality of face areas are specified, and a face reliability value is calculated for each face area.

  The face detection unit 171 has a function of adding an ID number to the detected face area, and is unique to the face area determined to be the same person as the face area detected from the continuous image frame data. ID numbers can be added. In order to identify the same person for the face area detected between successive image frame data, for example, the face area information detected in the previous image frame data and the current image frame data are detected. Compared with the information of the face area thus made. As a result of the comparison, it is possible to determine that face areas whose position and size differences in the image frame data are not more than a certain value are the same person.

  When the face detection unit 171 determines that a human face does not exist in the image frame data, the face detection unit 171 notifies the expression evaluation unit 172 that there is no face area, and the expression evaluation unit 172 evaluates the evaluation difference value. The determination unit 173 is notified that there is no face area. The evaluation difference value determination unit 173 outputs a smile non-detection signal to the control unit 110 because the face area does not exist in the image frame data, and the control unit 110 ends the processing for the image frame data and performs the next image frame data. The process for is started.

  When the face detection unit 171 determines that a human face is present in the image frame data, information on each face region in the image frame data detected by the face detection unit 171 and information including an ID number of each face region Is input to the facial expression evaluation unit 172. Further, the face reliability value for each face area is input to the evaluation difference value determination unit 173. The facial expression evaluation unit 172 calculates a facial expression evaluation value indicating the degree of smile for each input face area image.

  The calculated facial expression evaluation value and face reliability value for each face area are input to the evaluation difference value determination unit 173 in association with the ID number. The evaluation difference value determination unit 173 calculates, for each face area, the difference value between the expression evaluation value of the face area included in the immediately previous image frame data and the expression evaluation value of the face area included in the image frame data currently being processed. It is calculated and it is determined whether or not the difference value of each face area satisfies a predetermined condition. Then, the evaluation difference value determination unit 173 determines whether or not a smile is detected as a whole by integrating these determinations.

  For example, as shown in FIG. 9, the evaluation difference value determination unit 173 of the present embodiment is input together with the facial expression evaluation value so that the differential value of the facial expression evaluation values of the three face images can be detected independently. An ID discriminating unit 173a for detecting the ID number and distributing the input facial expression evaluation value is provided. The evaluation difference value determination unit 173 calculates the difference value from each facial expression evaluation value distributed according to the ID number, and determines whether or not a predetermined condition is satisfied. Another evaluation difference value determination unit 173b, 173c, 173d is provided. Also, the evaluation difference value determination unit 173 integrates the evaluation difference values output from the ID-specific evaluation difference value determination units 173b, 173c, and 173d to obtain a smile detection signal at a timing at which a suitable facial expression evaluation value is obtained for the entire image. Is provided to an evaluation difference value determination integration unit 173e.

  The ID number and the expression evaluation value for each face area input from the expression evaluation unit 172 and the face reliability value input from the face detection unit 171 are input to the ID determination unit 173a, and each ID is determined according to the ID number. It is input to the separate evaluation difference value determination units 173b, 173c, 173d.

  In addition, when the face area corresponding to the ID number is not detected, information indicating that the face area is not detected is input to each evaluation difference value determining unit for each ID. Further, the ID determination unit 173a outputs the number of valid IDs indicating the number of ID numbers in which face areas are detected to the evaluation difference value determination integration unit 173e.

  The operations of the ID-specific evaluation difference value determination units 173b, 173c, and 173d are the same as those of the evaluation difference value determination unit 133 according to the first embodiment. That is, it is determined whether or not the difference value of the facial area facial expression evaluation value for the corresponding ID number is in a predetermined state, and the detection result is output to the evaluation difference value determination integration unit 173e. The evaluation difference value determination integration unit 173e uses the detection result from each ID evaluation difference value determination unit and the number of face areas input from the ID determination unit 173a to determine a suitable facial expression evaluation value for the entire image data. And outputs the detection result to the control unit 101.

  As in the first embodiment, the control unit 110 controls the image encoding unit 140 according to the output of the evaluation difference value determination unit 173 to encode the image frame data, and the encoded still image Data is recorded on the recording medium 150.

  FIG. 10 is a flowchart showing the operation of the evaluation difference value determination integration unit 173e in FIG. Hereinafter, the operation of the evaluation difference value determination integration unit 173e will be described using the flowchart of FIG.

  When the evaluation difference value determination integration unit 173e starts the process, the evaluation difference value determination integration unit 173e initializes a face number variable NUM for holding the number of face regions in which a smile is detected to 0 (step S401). Subsequently, the evaluation difference value determination integration unit 173e determines whether or not a smile detection signal for the face area of ID1 is input from the first ID-specific evaluation difference value determination unit 173b (step S402). When the smile detection signal for the face area of ID1 is input (Yes in step S402), the face number variable NUM indicating the number of face areas where the smile is detected is incremented by 1 (step S403), and the face where the smile is detected It is determined whether or not the number of areas is 1 (step S404). If the number of face areas in which a smile is detected is 1, that is, if a smile is detected for the first time after starting the process, the internal timer is reset (step S405).

  The evaluation difference value determination integration unit 173e executes the same processing as the above-described steps S402 to 405 also for the second ID-specific evaluation difference value determination unit 173c and the third ID-specific evaluation difference value determination unit 173d (step). S406-409, S410-413).

  When the processing on the signal from each evaluation difference value determination unit for each ID is completed, the evaluation difference value determination integration unit 173e confirms the number of face regions in which a smile is detected. It is determined whether or not the number of face areas is 0 (step S414). If no smile is detected for any face area (Yes in step S414), a smile non-detection signal is output to the control unit 110. Then, the process returns to step S402 to process the next image data.

  If it is determined that a smile is detected (No in step S414), it is determined whether the number of face areas in which a smile is detected matches the number of valid IDs input from the ID determination unit 173a. (Step S415). As a result of the determination, if it is determined that all the face areas are smiling (Yes in step S415), the evaluation difference value determination integrating unit 173e outputs a smile detection signal to the control unit 110 (step S418). The process is terminated.

  If it is determined that all the detected face areas are not smiling (No in step S415), it is determined whether or not the value of the internal timer is greater than a predetermined value T0v (step S416). . If the timer value is not greater than the predetermined value T0v (No in step S416), a smile non-detection signal is output to the control unit 110 (step S417), and the process returns to step S402 to process the next image frame data. . If it is larger than the predetermined value T0v (Yes in step S416), a smile detection signal is output to the control unit 110, and the process ends.

  According to the processing of FIG. 10, in the following case, the control unit 110 can be instructed to acquire a still image after a predetermined period has elapsed. That is, when the difference values of the facial expression evaluation values of the face area included in the image frame data all satisfy a predetermined condition, or after the difference value first satisfies a predetermined condition in one face area, a predetermined period This is a case where a difference value for another face does not satisfy a predetermined condition.

  Hereinafter, a specific example of processing of the moving image processing apparatus 100A according to the present embodiment will be described with reference to FIGS. 11 and 12 are graphs in which changes in facial expression evaluation values output from the facial expression evaluation unit 132 for each image frame data are plotted for each of the three facial image regions. In the figure, the broken line indicates the change in facial expression evaluation value of the face image with ID1, the facial expression evaluation value of the face image with ID2 in the solid line, and the facial expression evaluation value of the face image with ID2 in the dashed line.

  FIG. 11 is a graph showing changes in facial expression evaluation values in an example in which three people entering three face areas are smiling at a timing close to each other. The first ID-specific evaluation difference value determination unit 173b detects that the difference value of the facial expression evaluation value is smaller than a predetermined reference value at the time point (a), and sends a smile detection signal to the evaluation difference value determination integration unit 173e. Is output. The evaluation difference value determination integration unit 173e receives the smile detection signal for ID1, and resets the timer in step S405 of FIG.

  Thereafter, the evaluation difference value determination integration unit 173e receives smile detection signals for ID2 and ID3 at the timings (b) and (c) in FIG. The evaluation difference value determination integration unit 173e determines that smile detection signals for all detected face areas have been input before the timer value reset at the timing of (a) becomes larger than the value of TOv (step S415). Detect with. Then, the evaluation difference value determination integration unit 173e outputs a smile detection signal to the control unit 110 (step S418). The control unit 110 receives the smile detection signal at the timing (c), starts the imaging process, and acquires image data output from the imaging unit 120 at a timing delayed by Δt from (c), thereby Image data having a large facial expression evaluation value for the face area can be encoded and recorded.

  FIG. 12 is a graph showing changes in facial expression evaluation values in an example in which only one person out of the three face areas is smiling. The first ID-specific evaluation difference value determination unit 173b detects that the difference value of the facial expression evaluation value is smaller than a predetermined reference value at the time point (a), and sends a smile detection signal to the evaluation difference value determination integration unit 173e. Is output. The evaluation difference value determination integration unit 173e receives the smile detection signal for ID1, and resets the timer in step S405 of FIG.

  Since the evaluation difference value determination integration unit 173e does not receive the smile detection signal for the other face area from the timing (a) in FIG. 11 to the timing (a ′) when the time of TOv has elapsed, step S416 in FIG. It detects that the timer value has become larger than the value of TOv. The evaluation difference value determination integration unit 173e outputs a smile detection signal to the control unit 110 at the timing (a ′) (step S418). The control unit 110 receives the smile detection signal at the timing (a ′) and starts the imaging process, and acquires image data output from the imaging unit 120 at a timing delayed by Δt from (a ′). As a result, the image data can be encoded and recorded before the facial expression evaluation value of the ID1 face area becomes small.

  The object of the present invention is achieved by executing the following processing. That is, a storage medium that records a program code of software that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus is stored in the storage medium. This is the process of reading the code.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

  Moreover, the following can be used as a storage medium for supplying the program code. For example, floppy (registered trademark) disk, hard disk, magneto-optical disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW, magnetic tape, nonvolatile memory card, ROM or the like. Alternatively, the program code may be downloaded via a network.

  Further, the present invention includes a case where the function of the above-described embodiment is realized by executing the program code read by the computer. In addition, an OS (operating system) running on the computer performs part or all of the actual processing based on an instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Is also included.

 Furthermore, a case where the functions of the above-described embodiment are realized by the following processing is also included in the present invention. That is, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  Further, the present invention includes a case where the functions of the above-described embodiments are realized by executing the program code read by the computer. In addition, there is a case where the OS running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing.

  In this case, the program is supplied by downloading directly from a storage medium storing the program or from another computer or database (not shown) connected to the Internet, a commercial network, a local area network, or the like.

100 moving image processing apparatus 110 control unit 120 moving image decoding unit 130 smile detection unit 131 face detection unit 132 facial expression evaluation unit 133 evaluation difference value determination unit 140 image encoding unit 150 recording medium 160 operation unit

Claims (13)

A face detection unit that detects a human face from a plurality of image frame data constituting moving image data, and a facial expression evaluation that calculates a degree of expression evaluation indicating how close the detected facial expression is to a specific facial expression A moving image processing apparatus comprising: a control unit that executes a process of acquiring still image data when a high facial expression evaluation value is calculated by the facial expression evaluation unit;
A difference value between the expression evaluation value for the currently processed image frame data and the expression evaluation value for the image frame data at a time point earlier than the currently processed image frame data is calculated, and the calculated difference value is a predetermined reference value An evaluation difference value determination unit that determines whether or not the difference is smaller , and the control unit acquires still image data when the evaluation difference value determination unit determines that the calculated difference value is smaller than a predetermined reference value A moving image processing apparatus characterized in that the processing is performed. An imaging unit that inputs the moving image data; and a recording medium that records the image data input by the imaging unit as still image data; and the control unit calculates the difference value calculated by the evaluation difference value determination unit. 2. The moving image processing apparatus according to claim 1, wherein when it is determined that the image data is smaller than a predetermined reference value, the image data captured by the imaging unit is recorded on the recording medium. The predetermined reference value, moving image processing apparatus according to claim 1 or 2, characterized in that a constant. The predetermined reference value, moving image processing apparatus according to claim 1 or 2, characterized in that a value which is a certain percentage of the calculated difference value immediately before. The predetermined reference value, moving image processing apparatus according to claim 1 or 2, characterized in that a value which is a certain percentage of the average of a plurality of difference values calculated immediately before. The evaluation difference value determination unit, a moving image processing apparatus according to any one of claims 1 to 5, characterized in that to calculate the difference values for the facial expression evaluation value exceeds a predetermined threshold.   The face detection unit calculates a face reliability value indicating the reliability of whether or not the object is a person's face, and the evaluation difference value determination unit performs the above described step as the face reliability value decreases. 7. The moving image processing apparatus according to claim 6, wherein the threshold value is changed to a large value.   When a plurality of faces are included in one image frame data, the face detection unit detects a plurality of faces, and the expression evaluation unit calculates respective expression evaluation values for the detected plurality of faces, The moving image processing apparatus according to claim 1, wherein the evaluation difference value determination unit calculates and determines a difference value for each face from the plurality of facial expression evaluation values. When the control unit determines that all the difference values calculated from the facial expression evaluation values of the plurality of faces detected by the face detection unit are smaller than a predetermined reference value , the still image 9. The moving image processing apparatus according to claim 8, wherein processing for acquiring data is executed. The control unit determines that the difference value for one of the plurality of faces detected by the face detection unit is smaller than a predetermined reference value after the evaluation difference value determination unit is within a predetermined period. 10. The moving image processing apparatus according to claim 9, wherein when it is determined that the difference value with respect to the face is smaller than a predetermined reference value, a process of acquiring still image data is performed after a predetermined period of time. Face detection step for detecting a human face from a plurality of image frame data constituting moving image data, and facial expression evaluation for calculating a facial expression evaluation value indicating how close the detected facial expression is to a specific facial expression In a moving image processing method comprising: a step; and a control step of executing processing for acquiring still image data when a high facial expression evaluation value is calculated by the facial expression evaluation step ,
A calculation step for calculating a difference value between the expression evaluation value for the image frame data being currently processed and the expression evaluation value for the image frame data at a time point before the image frame data being currently processed; and the calculated difference value is predetermined. of a determining evaluation difference value determination step of determining whether or not the reference value is smaller than, and in the control step, when the difference value calculated by the evaluation difference value determination step determines a predetermined reference value smaller than A moving image processing method characterized by executing processing for acquiring still image data. Face detection step for detecting a human face from a plurality of image frame data constituting moving image data, and facial expression evaluation for calculating a facial expression evaluation value indicating how close the detected facial expression is to a specific facial expression In a program for causing a computer to execute a moving image processing method comprising: a step; and a control step of executing a process of acquiring still image data when a high facial expression evaluation value is calculated in the facial expression evaluation step ,
A calculation step for calculating a difference value between the expression evaluation value for the image frame data being currently processed and the expression evaluation value for the image frame data at a time point before the image frame data being currently processed; and the calculated difference value is predetermined. of a determining evaluation difference value determination step of determining whether or not the reference value is smaller than, and in the control step, when the difference value calculated by the evaluation difference value determination step determines a predetermined reference value smaller than A program for executing processing for acquiring still image data.   A computer-readable storage medium storing the program according to claim 12.

Images