JP4857162B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and a recording medium, and more particularly to a technique suitable for use in generating a thumbnail image of a captured moving image.

  In recent years, video recorders using recording media such as optical disks, magnetic disks, and semiconductor memories have become common. Since these recording media can be accessed randomly as compared with conventional tape media, a method of displaying a list of recorded cuts and selecting cuts to be reproduced is generally used. In the cut list display, an image called a thumbnail image obtained by reducing the representative image selected from the image frame being cut is used.

  Conventionally, as the thumbnail image, a reduced image generated from a top frame of a cut or a frame image after a predetermined time from the top of the cut is used. The reason for using a reduced image after a predetermined time is that, for example, in a TV drama or the like, the top of the cut is a black image, and the actual video may be edited using fade-in. This is because it is not necessarily effective for identification in display.

  On the other hand, Patent Document 1 proposes a method for generating a scene representative screen from a screen showing a specific person such as an actor as a database search method for use in a video archive or the like. Since the method described in Patent Document 1 is intended for classification and search by actor name, if a target person appears in the scene, the target person will be identified regardless of the main content of the scene. Associated.

  FIG. 14 is a flowchart showing a procedure for creating a person representative image for database search disclosed in Patent Document 1. In the flowchart illustrated in FIG. 14, a frame in which a designated person appears is extracted as a representative image for each scene from a series of recorded video images.

  First, in step S1401, processing of a target video image is started. In step S1402, the flag is reset to zero. In step S1403, a frame to be processed is read out. Thereafter, it is determined in step S1404 whether a scene change has been detected. If a scene change is detected as a result of this determination, the process advances to step S1405 to display the representative image of the previous scene, and the flag is reset to 0 in step S1406. On the other hand, if the result of determination in step S1404 is that a scene change has not been detected, processing proceeds to step S1407.

  In step S1407, it is determined whether or not the flag is 0. If the result of this determination is 0, the representative image is stored in step S1408, and the flag is set to 1 in step S1409. On the other hand, if the result of determination in step S1407 is not 0, processing proceeds to step S1410.

  Then, a face image is detected in step S1410, and the detection result is determined in step S1411. If a face image is detected as a result of this determination, the process advances to step S1412 to update the representative image, and the process returns to step S1402. If no face image is detected as a result of the determination in step S1411, the process returns to step S1402 to repeat the process.

JP 2001-167110 A

  The conventional method of generating a fixed thumbnail image as described above is an effective method because the screen at the beginning of the scene can be confirmed when recording a TV drama or the like with a video recorder with a TV tuner. However, when displaying a list of cuts with a home video camera, it is not always effective.

  Shooting using a home video camera is not in line with the scenario as in a TV drama, and in most cases the subject is recorded. For this reason, it is desirable to use a list that reflects to some extent what the main subject is rather than the flow of the scene in the list display in the home video camera. With the conventional fixed representative image frame setting, it is difficult to generate a thumbnail image suitable for this application, and a video camera capable of generating a thumbnail image mainly intended for home use. Was desired.

  Further, the method for generating a representative image by a specific person proposed in Patent Document 1 is a representative image generation method reflecting a subject. However, this method is intended for database search by actor name, and is a method for determining a representative image depending on whether or not a specific person appears regardless of the main contents of the cut.

  For example, a representative image of a specific person is generated even under conditions that are likely to be taken with a home video camera, such as when the main subject is a landscape and the specific person accidentally appears at the end of the cut. For this reason, this method is not necessarily effective for generating thumbnails for displaying a list on a home video camera. There has been a demand for a video camera having a function of generating a thumbnail image that can reflect not only the presence of a specific person but also the main subject in the cut.

  SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and it is an object of the present invention to prevent the inconvenience of generating a thumbnail image using a person who appears accidentally as a representative image and to generate a thumbnail image reflecting a main subject. Yes.

The image processing apparatus of the present invention detects the characteristic amount of the face of a human product is Ru contained in movies image that is entered, provided with a characteristic quantity detecting means for recognizing whether the person is a specific person A feature amount storage unit that stores a feature amount of the face of the specific person to be recognized by the feature amount detection unit in a storage unit, a recording unit that records an image on a recording medium, from the time of recording start of the recording means images recorded by, in a short first predetermined time period than up to the point of end of recording, the feature amount of the face of the specific person which is previously stored in the storage unit based on the case where the specific person from the recorded image is recognized by the feature quantity detecting unit, the recording from the video frame a particular person has been recognized in the image in the first predetermined time period thumbnail as a representative image of the video It generates Le image, when said first of said specific person from said recorded video within a predetermined time period has not been recognized by the feature quantity detecting unit, from the time of the recording start said first predetermined And a thumbnail image generating means for generating the thumbnail image from a video frame at the time when a second predetermined period shorter than the period has elapsed .

The image processing method of the present invention detects the characteristic amount of the face of a human product is Ru contained in movies image that is entered, provided with a feature amount detection process for recognizing whether the person is a specific person A feature amount storing step of storing a feature amount of the face of the specific person to be recognized in the feature amount detecting step in a storage unit, a recording step of recording a video on a recording medium, from the time of start of recording video recorded in the recording step, in a short first predetermined time period than up to the point of end of recording, the feature amount of the face of the specific person which is previously stored in the storage unit Based on the recorded video, when the specific person recognizes in the feature amount detection step, the recording is performed from the video frame in which the specific person in the video within the first predetermined period is recognized. thumbnail as a representative image of the video was It generates Le image, when said first of said specific person from said recorded video within a predetermined time period has not recognized in the feature amount detection step, from the time of the recording start said first predetermined And a thumbnail image generation step of generating the thumbnail image from a video frame at the time when a second predetermined period shorter than the period has elapsed .

Program of the present invention detects the characteristic amount of the face of a human product is Ru contained in movies image is entered, a feature amount detection step for recognizing whether the person is a specific person, the characteristic A feature amount storing step of storing in the storage unit a feature amount of the face of the specific person to be recognized in the amount detecting step, a recording step of recording an image on a recording medium, and a recording of the image recorded in the recording step wherein from the time of the start, in a short first predetermined time period than up to the point of recording end, on the basis of the feature amount of the face of a specific person stored in advance in the storage unit, from the recorded image When a specific person recognizes in the feature amount detection step, a thumbnail image is generated as a representative image of the recorded video from a video frame in which the specific person is recognized in the video within the first predetermined period And the first If the specific person does not recognize in the feature amount detection step from the recorded video within a predetermined period, a second predetermined period shorter than the first predetermined period from the recording start time And a thumbnail image generation step of generating the thumbnail image from the video frame at the time when elapses .

  The recording medium of the present invention is characterized by recording the program described above.

According to the present invention, even it is possible to prevent a disadvantage that generates a thumbnail image of natural yelling-through person as a representative image, the or another object main subject is a specific person be detected photographing Can be determined. Accordingly, a thumbnail image reflecting the main subject can be generated.

(First embodiment)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration example of a video recording processing circuit of the image processing apparatus 100 according to the present embodiment. The image processing apparatus 100 is a video camera, for example, and performs audio recording processing together with video. Since the voice is not directly related to the contents of the present invention, the description is omitted.

  A video signal is input to the first terminal 101 from an imaging unit including an optical system such as a lens, a sensor such as a CCD, and a camera signal processing circuit. The input video signal is compressed and encoded by the moving image encoding circuit 104 using an encoding method such as MPEG2. The details of the compression encoding process are not directly related to the contents of the present invention, and thus the description thereof is omitted.

  The multiplexer 105 multiplexes audio data, subcode data, and the like with the compressed and encoded moving image data to create AV data. Note that description of video camera components other than these video recording processes is omitted. The AV data multiplexed by the multiplexer 105 is recorded on the recording medium 107 by the media recording circuit 106. Examples of the recording medium include an optical disk, a magnetic disk, a solid-state memory, and the like, but any medium can be used as long as it can record thumbnail images for listing together with AV data. The system controller 103 controls each block according to a recording start instruction input from the second terminal 102 and controls recording of AV data.

  On the other hand, the video signal input via the first terminal 101 is stored in the frame memory 108 for each cycle of face image detection processing described later by the representative image generation control circuit 116 controlling the first switch 109. Is done. Note that this cycle may be performed every frame if the face image detection processing capability is high, or may be intermittent depending on the processing capability and a necessary detection interval.

  The feature amount detection circuit 112 detects an image feature amount for a video frame (video signal) stored in the frame memory 108. In the present embodiment, an example using an image feature amount for determining a human face image will be described. For facial image determination using image features, an algorithm using a structure specific to facial images such as the hue of the partial image, eyes, nose, and mouth has been proposed, and has already been put to practical use in personal authentication of portable devices. Yes. In the present embodiment, face image determination using an image feature amount is performed based on the algorithm. Since the present invention is not directly related to the feature amount detection method, a detailed description of an already known algorithm for face image determination is omitted.

  In the feature amount storage memory (storage unit) 114, feature amounts necessary for recognition of a specific person are stored in advance. For example, before recording with the image processing apparatus 100, a specific person such as a family is imaged as a subject, and at that time, the representative image generation control circuit 116 functions as a feature amount storage unit, and the second switch 113 is set to the writing side. To control. Thereby, the image feature quantity output from the feature quantity detection circuit 112 can be stored in the feature quantity storage memory 114. These controls are performed by the representative image generation control circuit 116 controlling each unit based on an instruction sent from the system controller 103.

  The feature amount comparison circuit 115 compares the image feature amount output from the feature amount detection circuit 112 with the image feature amount stored in the feature amount storage memory 114, thereby identifying a specific person in the currently captured video frame. It is determined whether or not a face image is included. The feature amount comparison circuit 115 and the feature amount detection circuit 112 function as feature amount detection means. Based on the recording start information output from the system controller 103, the recording time counting circuit 117 counts the elapsed time from the start of recording to the video frame for which face image determination has been performed.

  The representative image generation control circuit 116 determines the video frame to be a representative image according to the feature amount comparison result of the face image by the feature amount comparison circuit 115 and the elapsed time from the start of recording counted by the recording time counting circuit 117. To decide. Then, by controlling the third switch 118, the image of the video frame to be the representative image is stored in the representative image memory 110. Details of the representative image determination method based on the feature amount comparison result and the elapsed time from the start of recording will be described later with reference to a flowchart.

  The thumbnail image generation circuit 111 performs thumbnail image data reduction or other necessary conversion processing on the representative image stored in the representative image memory 110 to generate thumbnail image data. At this time, when a video frame including a face image of a specific person is selected as a representative image, additional information such as the name of the specific person stored in advance can be added to the thumbnail image data. The generated thumbnail image data is recorded on the recording medium 107 by the media recording circuit 106.

FIG. 2 is a flowchart illustrating an example of a processing procedure for determining a representative image based on a face image detection result and an elapsed time from the start of recording and creating thumbnail image data in the present embodiment.
First, in step S201, processing is started. Next, in step S202, the representative image generation control circuit 116 resets the face image detection flag (Flg) holding the face image detection state to “0” indicating the undetected state. Further, the elapsed time t counted by the recording time counting circuit 117 is reset to “0”.

  Next, in step S203, recording of a video signal is started under the control of the system controller 103. In step S204, the representative image generation control circuit 116 controls the first switch 109 to write the video frame F (t) for performing the face image detection process in the frame memory 108.

  Next, in step S205, the representative image generation control circuit 116 compares the elapsed time t with a first time period T1 that is an effective period of face image detection. As a result of this comparison, when the elapsed time t is within the first time period T1, the process proceeds to the face image detection process after step S206. On the other hand, as a result of the comparison in step S205, if the elapsed time t exceeds the first time period T1, the face image detection process is skipped and the process proceeds to step S212.

  Note that, by comparing the elapsed time t in step S205 with the face image detection period (first time period) T1, whether the main subject of the current shooting is a person to be detected or another subject. Determine. This determination is based on the fact that shooting using a home video camera that is not shooting according to a scenario often starts shooting after aiming at the main subject.

  In step S206, the representative image generation control circuit 116 checks the face image detection flag (Flg) to determine whether face image detection has already been performed. As a result of this determination, if the face image detection flag (Flg) is set to “1” indicating that detection has been completed, the following detection process is skipped and the process proceeds to step S212. On the other hand, if the result of determination in step S206 is that the face image detection flag (Flg) is “0” and no face image has been detected, the process proceeds to face image feature amount detection processing in step S207.

  Next, in step S207, the feature amount detection circuit 112 detects a face image feature amount D (F (t)) for determining a face image of a specific person that has already been stored, from the video frame F (t). Thus, the feature quantity P is output to the feature quantity comparison circuit 115. In step S208, the feature amount comparison circuit 115 compares the face image feature amount M and the feature amount P of the specific person stored in the feature amount storage memory 114 using a predetermined function C (P, M). The comparison result K is calculated.

  Next, in step S209, the feature amount comparison circuit 115 determines whether or not a face image of a specific person has been detected in the video frame F (t) by comparing the comparison result K with a predetermined threshold value Kth. . As a result of the determination, if the comparison result K is equal to or greater than the predetermined threshold value Kth and a face image is detected, the process proceeds to step S210. In step S210, the representative image generation control circuit 116 controls the third switch 118 to write the video frame F (t) as a representative image in the representative image memory 110. In step S211, the representative image generation control circuit 116 sets the face image detection flag (Flg) to “1” indicating that detection has been performed.

  On the other hand, as a result of the determination in step S209, if the comparison result K is less than the predetermined threshold value Kth and no face image is detected, the processing in steps S210 and S211 below is skipped and the process proceeds to step S212. In this way, face image detection is executed until the elapsed time t exceeds the first time period T1, and the detected head face image is stored in the representative image memory 110.

  Next, in step S212, the representative image generation control circuit 116 calculates the second time period T2 that is the representative image frame position and the elapsed time t when the face image is not detected within the first time period T1. Compare. As a result of this comparison, when the elapsed time t exceeds the second time period T2, the processing of the following steps S213 and S214 is skipped and the process proceeds to step S215.

  On the other hand, when the elapsed time t is within the second time period T2 as a result of the comparison in step S212, the representative image generation control circuit 116 checks the face image detection flag (Flg) in step S213. As a result of this check, when the face image detection flag (Flg) is set to “1” indicating detection, the processing of the following step S214 is skipped and the process proceeds to step S215.

  On the other hand, if the face image detection flag (Flg) is “0” as a result of the check in step S213 and no face image is detected, the process proceeds to step S214, and the representative image generation control circuit 116 F (t) is written into the representative image memory 110. As described above, when the face image is not detected, the content stored in the representative image memory 110 is updated by the latest video frame F (t) until the elapsed time t exceeds the second time period T2. .

  Next, in step S215, the representative image generation control circuit 116 determines whether or not a recording stop instruction has been issued from the system controller 103 by a user operation or the like. If the result of this determination is that there is no instruction to stop recording, processing proceeds to step S216 and processing continues. In step S216, the recording time counting circuit 117 increments the elapsed time t by a predetermined detection unit. As described above, when the face image detection processing capability is high and the required detection interval is short, the increment is performed every frame, and when the detection interval is long, the increment is performed so as to be a predetermined interval process. After the increment, the process returns to step S204 to repeat the process.

  On the other hand, if recording stop is instructed as a result of the determination in step S215, the process proceeds to step S217, and the thumbnail image generation circuit 111 generates thumbnail image data from the video frames stored in the representative image memory 110. In step S218, the media recording circuit 106 records the generated thumbnail image data on the recording medium 107, and ends the process.

  As shown in the flowchart of FIG. 2, the face image detection process after step S207 is performed only during a period in which the elapsed time t is equal to or shorter than the first time period T1. Therefore, the face image detection process itself can be stopped after the elapsed time t exceeds the first time period T1. In this case, the feature quantity detection circuit 112, the feature quantity comparison circuit 115, and the like can be mounted to cope with energy saving such as stopping the clock supply. By limiting the actual detection time to a short time at the beginning of recording in this way, even when the image processing apparatus 100 of the present embodiment is applied to a portable device such as a video camera, it is possible to suppress the influence on battery consumption. is there.

  FIG. 3 is a diagram showing a specific example of selecting a representative image by the above processing. In the example illustrated in FIG. 3, description will be made assuming that the default representative image position (second time period) T2 <face image detection period (first time period) T1 when no face image is detected.

  In the example shown in FIG. 3A, since the elapsed time t <T2 <T1 and the face image of the specific person is detected within the face image detection period T1, the video frame at the position of the elapsed time t is selected as the representative image. Is done. In the example shown in FIG. 3B, since T2 <elapsed time t <T1 and a face image of a specific person is detected within the face image detection period T1, the video frame at the position of the elapsed time t is used as a representative image. Selected. On the other hand, in the example shown in FIG. 3C, T2 <T1 <elapsed time t, and the face image of the specific person was not detected within the face image detection period T1, so the video frame at the default representative image position T2 is representative. Selected as an image.

  FIG. 4 is a diagram illustrating another specific example of selecting a representative image. In the example shown in FIG. 4, description will be made assuming that the face image detection period (first time period) T1 <the default representative image position (second time period) T2 when no face image is detected.

  In the example shown in FIG. 4A, since the elapsed time t <T1 <T2 and the face image of the specific person is detected within the face image detection period T1, the video frame at the position of the elapsed time t is selected as the representative image. Is done. On the other hand, in the example shown in FIG. 4B, since T1 <elapsed time t <T2 and the face image of the specific person was not detected within the face image detection period T1, the video frame at the default representative image position T2 is representative. Selected as an image. In the example shown in FIG. 4C, T1 <T2 <elapsed time t, and the face image of the specific person was not detected within the face image detection period T1, so the video frame at the default representative image position T2 is representative. Selected as an image.

  As described above, according to this embodiment, as described above, in home video cameras, shooting is often started after aiming at a main subject. Based on this, the face image detection result and the elapsed time from the start of shooting are used to determine whether the main subject of shooting is a person to be detected or another subject and reflect the intention of shooting It is possible to create a thumbnail image.

(Second Embodiment)
FIG. 5 is a block diagram illustrating a configuration example of a video recording processing circuit of the image processing apparatus 500 in the present embodiment.
A video signal is input to the first terminal 501 from an imaging unit including an optical system such as a lens, a sensor such as a CCD, and a camera signal processing circuit. The input video signal is compressed and encoded by the moving image encoding circuit 504 using an encoding method such as MPEG2. The multiplexer 505 multiplexes audio data, subcode data, and the like on the compressed and encoded moving image data. Similar to the first embodiment, description of the video camera components other than the video recording processing is omitted.

  The AV data multiplexed by the multiplexer 505 is recorded on the recording medium 507 by the media recording circuit 506. The system controller 503 controls each block according to a recording start instruction input from the second terminal 502 and controls recording of AV data.

  On the other hand, the video signal input via the first terminal 501 is stored in the frame memory 508 for each cycle of face image detection processing described later by the representative image generation control circuit 516 controlling the first switch 513. Is done. Note that this cycle may be performed every frame if the face image detection processing capability is high, or may be intermittent depending on the processing capability and a necessary detection interval.

  The feature amount detection circuit 512 detects an image feature amount for a video frame (video signal) stored in the frame memory 508. In the present embodiment, an example using an image feature amount for determining a human face image will be described.

  The feature amount storage memory (storage unit) 514 stores in advance feature amounts necessary for recognition of a specific person. For example, before recording with the image processing apparatus 500, a specific person such as a family is imaged as a subject. At that time, the representative image generation control circuit 516 functions as a feature amount storage unit, and the second switch 518 is set to the writing side. To control. As a result, the image feature amount output from the feature amount detection circuit 512 can be stored in the feature amount storage memory 514. These controls are performed by the representative image generation control circuit 516 controlling each unit based on an instruction sent from the system controller 503.

  The feature amount comparison circuit 515 compares the image feature amount output from the feature amount detection circuit 512 with the image feature amount stored in the feature amount storage memory 514, thereby identifying the specific person in the currently captured video frame. It is determined whether or not a face image is included. The feature quantity comparison circuit 515 and the feature quantity detection circuit 512 function as a feature quantity detection unit. Based on the recording start information output from the system controller 503, the recording time counting circuit 517 counts the elapsed time from the recording start to the video frame for which the face image determination is performed.

  The representative image generation control circuit 516 is a video frame to be a representative image according to the feature amount comparison result of the face image by the feature amount comparison circuit 515 and the elapsed time from the start of recording counted by the recording time counting circuit 517. To decide. Then, index information for specifying the video frame at the time of reproduction is stored in the representative image index memory 519. As the index information, a frame number from the head of the recorded cut is used. Unlike the first embodiment, this embodiment holds representative image candidates by storing index information instead of storing actual representative images. As a result, the memory capacity required for shooting can be reduced.

  The representative image is read by reproducing the moving image data recorded on the recording medium 507 using the index information after the recording is completed. The system controller 503 controls the media playback circuit 520 after the recording is completed, and reads data necessary for playback of the video frame specified by the index information from the recording media 507 at this time.

  Video data is extracted from the read AV data by the demultiplexer 521, and a moving image is reproduced from the moving image data compressed and encoded by the moving image decoding circuit 522. Then, the video frame specified by the index information in the reproduced moving image is stored in the representative image memory 510 by controlling the third switch 523.

  The thumbnail image generation circuit 511 performs thumbnail image data reduction or other necessary conversion processing on the representative image stored in the representative image memory 510 to generate thumbnail image data. At this time, when a video frame including a face image of a specific person is selected as a representative image frame, additional information such as the name of the specific person stored in advance can be added to the thumbnail image data. The generated thumbnail image data is recorded on the recording medium 507 by the media recording circuit 506.

FIGS. 6A and 6B are flowcharts illustrating an example of a processing procedure for determining a representative image based on a face image detection result and an elapsed time from the start of recording and creating thumbnail image data in the present embodiment. .
First, in step S601, processing is started. In step S602, the representative image generation control circuit 516 resets the face image detection flag (Flg) that holds the face image detection state to “0” that represents the undetected state. Further, the elapsed time t counted by the recording time counting circuit 517 is reset to “0”.

  Next, in step S603, recording of a video signal is started under the control of the system controller 503. In step S604, the representative image generation control circuit 516 controls the first switch 513 to write the video frame F (t) for performing the face image detection process in the frame memory 508.

  Next, in step S605, the representative image generation control circuit 516 compares the elapsed time t with a first time period T1 that is an effective period of face image detection. As a result of this comparison, when the elapsed time t is within the first time period T1, the process proceeds to the face image detection process after step S606. On the other hand, if the elapsed time t exceeds the first time period T1 as a result of the comparison in step S605, the face image detection process is skipped and the process proceeds to step S612.

  In step S606, the representative image generation control circuit 516 checks the face image detection flag (Flg) to determine whether face image detection has already been performed. As a result of this determination, if the face image detection flag (Flg) is set to “1” indicating that detection has been completed, the following detection processing is skipped and the process proceeds to step S612. On the other hand, if the result of determination in step S606 is that the face image detection flag (Flg) is “0” and no face image has been detected, the process proceeds to face image feature amount detection processing in step S607.

  Next, in step S607, the feature amount detection circuit 512 detects a face image feature amount D (F (t)) for determining a face image of a specific person already stored from the video frame F (t). The feature amount P is output. In step S608, the feature amount comparison circuit 515 compares the face image feature amount M and the feature amount P of the specific person stored in the feature amount storage memory 514 using a predetermined function C (P, M). The comparison result K is calculated.

  Next, in step S609, the feature amount comparison circuit 515 determines whether or not a face image of a specific person has been detected in the video frame F (t) by comparing the comparison result K with a predetermined threshold value Kth. . If it is determined that the comparison result K is equal to or greater than the predetermined threshold value Kth and a face image is detected, the process proceeds to step S610. In step S610, the representative image generation control circuit 516 writes the index information of the video frame F (t) in the representative image index memory 519. In step S611, the representative image generation control circuit 516 sets the face image detection flag (Flg) to “1” indicating that detection has been performed. In this embodiment, the elapsed time t is used as index information.

  On the other hand, as a result of the determination in step S609, if the comparison result K is less than the predetermined threshold value Kth and no face image is detected, the processing in steps S610 and S611 below is skipped and the process proceeds to step S612. In this way, face image detection is executed until the elapsed time t exceeds the first time period T1, and index information for the detected first face image is stored in the representative image index memory 519.

  In step S612, the representative image generation control circuit 516 compares the second time period T2, which is a representative image frame position when the face image is not detected within the first time period T1, with the elapsed time t. As a result of this comparison, when the elapsed time t exceeds the second time period T2, the processing of the following steps S613 and S614 is skipped and the process proceeds to step S615.

  On the other hand, if the elapsed time t is within the second time period T2 as a result of the comparison in step S612, the representative image generation control circuit 516 checks the face image detection flag (Flg) in step S613. As a result of this check, when the face image detection flag (Flg) is set to “1” indicating that detection has been completed, the detection process of step S614 below is skipped and the process proceeds to step S615.

  On the other hand, as a result of the check in step S613, if the face image detection flag (Flg) is “0” and no face image is detected, the process proceeds to step S614. In step S614, the representative image generation control circuit 516 writes the index information of the current video frame F (t) in the representative image index memory 519. As described above, when the face image is not detected, the content stored in the representative image index memory 519 until the elapsed time t exceeds the second time period T2 is determined by the index information of the latest video frame F (t). Updated.

  Next, in step S615, the representative image generation control circuit 516 determines whether or not a recording stop is instructed from the system controller 503 by a user operation or the like. If the result of this determination is that there is no instruction to stop recording, processing proceeds to step S616 and processing continues. In step S616, the recording time counting circuit 517 increments the elapsed time t by a predetermined detection unit. As described above, when the face image detection processing capability is high and the required detection interval is short, the increment is performed every frame, and when the detection interval is long, the increment is performed so as to be a predetermined interval process. After the increment, the process returns to step S604 and the process is repeated.

  On the other hand, if recording stop is instructed as a result of the determination in step S615, the process proceeds to step S617, and the index information tm is read from the representative image index memory 519 under the control of the system controller 503. In step S618, the media playback circuit 520 reads the AV data recorded this time using the read index information tm, and plays back the video frame F (tm) selected as the representative image.

  Next, in step S619, the representative image generation control circuit 516 writes the video frame F (tm) in the representative image memory 510. In step S 620, the thumbnail image generation circuit 511 generates thumbnail image data from the video frame stored in the representative image memory 510. In step S621, the media recording circuit 506 records the generated thumbnail image data on the recording medium 507 and ends the process.

  As described above, according to this embodiment, as described above, in home video cameras, shooting is often started after aiming at a main subject. Based on this, the face image detection result and the elapsed time from the start of shooting are used to determine whether the main subject of shooting is a person to be detected or another subject and reflect the intention of shooting It is possible to create a thumbnail image.

(Third embodiment)
In the configuration of the second embodiment, the reading of the video frame serving as the representative image is performed after the recording is completed. Therefore, in the present embodiment, an example in which the face image detection period (first time period) T1 is changed according to the actual shooting time will be described.

  FIG. 8A and FIG. 8B are flowcharts illustrating an example of a processing procedure for determining a representative image based on a face image detection result and an elapsed time from the start of recording and creating thumbnail image data in the present embodiment. . The video recording processing circuit for executing each process in the flowcharts shown in FIGS. 8A and 8B is the same as that in the second embodiment, and a description thereof will be omitted.

  First, in step S801, processing is started. Next, in step S802, the representative image generation control circuit 516 resets the face image detection flag (Flg) holding the face image detection state to “0” indicating the undetected state. Further, the elapsed time t counted by the recording time counting circuit 517 is reset to “0”.

  Next, in step S803, recording of the video signal is started under the control of the system controller 503. In step S804, the representative image generation control circuit 516 controls the first switch 513 to write the video frame F (t) for performing the face image detection process in the frame memory 508.

  In step S805, the representative image generation control circuit 516 checks the face image detection flag (Flg) to determine whether face image detection has already been performed. As a result of this determination, if the face image detection flag (Flg) is set to “1” indicating detection completion, the following detection processing is skipped and the process proceeds to step S812. On the other hand, as a result of the determination in step S805, if the face image detection flag (Flg) is “0” and no face image is detected, the process proceeds to face image feature amount detection processing in step S806.

  In step S806, the feature amount detection circuit 512 detects a face image feature amount D (F (t)) for determining a face image of a specific person that has already been stored from the video frame F (t). The feature amount P is output to the feature amount comparison circuit 515. In step S807, the feature amount comparison circuit 515 compares the face image feature amount M and the feature amount P of the specific person stored in the feature amount storage memory 514 using a predetermined function C (P, M). The comparison result K is calculated.

  Next, in step S808, the feature amount comparison circuit 515 determines whether or not a face image of a specific person has been detected in the video frame F (t) by comparing the comparison result K with a predetermined threshold value Kth. . If it is determined that the comparison result K is equal to or greater than the predetermined threshold value Kth and a face image is detected, the process proceeds to step S809. In step S809, the representative image generation control circuit 516 writes the index information of the video frame F (t) in the representative image index memory 519. In step S810, the representative image generation control circuit 516 sets the face image detection flag (Flg) to “1” indicating that detection has been performed. In this embodiment, the elapsed time t is used as index information.

  On the other hand, if the result of determination in step S808 is that the comparison result K is less than the predetermined threshold value Kth and no face image is detected, the processing of steps S809 and S810 below is skipped and the process proceeds to step S812. In this way, index information for the detected first face image is stored in the representative image index memory 519. The difference between this embodiment and the second embodiment is that there is no face image detection time limit (first time period) T1 when video data is recorded.

  In step S812, it is determined whether or not recording stop is instructed from the system controller 103 by a user operation or the like. As a result of the determination, if recording stop is not instructed, the process proceeds to step S811, and the process is continued. In step S811, the recording time counting circuit 517 increments the elapsed time t by a predetermined detection unit. As described above, when the face image detection processing capability is high and the required detection interval is short, the increment is performed every frame, and when the detection interval is long, the increment is performed so as to be a predetermined interval process. After the increment, the process returns to step S804 to repeat the process.

  On the other hand, if the result of determination in step S812 is that recording stop has been instructed, processing proceeds to step S813, where the representative image generation control circuit 516 functions as a period changing means, and from the elapsed time T3 from recording start to recording stop, One time period T1 is calculated. In the present embodiment, the first time period T1 is set so that T1 = T3 / 3. That is, the effective period of face image detection is from the start of recording to the first third of the clip.

  Next, in step S814, the representative image generation control circuit 516 compares the second time period T2, which is the default representative image frame position, with the elapsed time T3 from the start of recording to the stop of recording to determine the second time period. The validity of the period T2 is verified. As a result of this comparison, if the elapsed time T3 is less than the second time period T2, there is no video frame when the second time period T2 has elapsed, so the process proceeds to step S815, and the representative image generation control circuit 516 is reached. Replaces the second time period T2 with an elapsed time T3. On the other hand, if the elapsed time T3 is equal to or longer than the second time period T2 as a result of the comparison in step S814, there is a video frame when the second time period T2 has elapsed, so step S815 is skipped. The process proceeds to S816.

  Next, in step S816, the representative image generation control circuit 516 checks the face image detection flag (Flg). As a result of this check, if the face image detection flag (Flg) is “0” and no face image is detected, the process proceeds to step S819, and the representative image generation control circuit 516 sets the time of the representative image index tm to the first time. It is assumed that the time period T2 is 2. On the other hand, if it is determined in step S816 that the face image detection flag (Flg) is set to “1” indicating detection, the process proceeds to step S817. In step S817, the index information tm is read from the representative image index memory 519 under the control of the system controller 503.

  Next, in step S818, the representative image generation control circuit 516 compares the time of the index information tm with the first time period T1 that is the effective period of face image detection. As a result of the comparison, if the time of the index information tm is within the first time period T1, the process proceeds to the process of reading the representative image frame after step S820. On the other hand, as a result of the comparison in step S818, if the time of the index information tm exceeds the first time period T1, the process proceeds to step S819, and the representative image generation control circuit 516 sets the time of the representative image index tm to the second time. It is assumed that the time period T2.

  Next, in step S820, the media playback circuit 520 reads the AV data recorded this time using the index information tm, and plays back the video frame F (tm) selected as the representative image. In step S821, the representative image generation control circuit 516 writes the video frame F (tm) in the representative image memory 510. In step S822, the thumbnail image generation circuit 511 generates thumbnail image data from the video frames stored in the representative image memory 510. In step S823, the media recording circuit 506 records the generated thumbnail image data on the recording medium 507 and ends the process.

  FIG. 7 is a diagram showing a specific example of selecting a representative image by the above processing. In the present embodiment, the face image detection period (first time period T1) is described as T1 = T3 / 3.

  In the example shown in FIG. 7A, T1 <T2 <time tm when the face image is detected, and the face image of the specific person is not detected within the face image detection period T1, so the position of the second time period T2 Are selected as representative images. On the other hand, in the example shown in FIG. 7B, T2 <time tm <T1 when the face image is detected, and since the face image of the specific person is detected within the face image detection period T1, time tm when the face image is detected. Is selected as a representative image.

  As described above, in the example shown in FIGS. 7A and 7B, the time from the start of recording to the detection of the face image is the same, but the recording time T3 of the entire clip from the start of recording to the stop of recording. Is different. For this reason, video frames selected as representative images are different. In the example shown in FIG. 7A, a specific person is captured near the end of the clip. In this case, the specific person is not determined to be the main subject. On the other hand, in the example shown in FIG. 7B, the specific person is shown within 1/3 from the top of the clip. In this case, it is determined that the specific person is likely to be the main subject.

  As described above, according to this embodiment, as described above, in home video cameras, shooting is often started after aiming at a main subject. Based on this, the face image detection result and the elapsed time from the start of shooting are used to determine whether the main subject of shooting is a person to be detected or another subject and reflect the intention of shooting It is possible to create a thumbnail image.

(Fourth embodiment)
In the present embodiment, an example will be described in which processing is performed on video clip data after shooting as well as when shooting with a video camera or the like. FIG. 9 is a block diagram illustrating a configuration example of a video recording processing circuit of the image recording apparatus 900 according to the present embodiment. In this embodiment, components for recording are omitted in order to describe processing for recorded video data. However, it goes without saying that the present invention can also be applied to an image processing apparatus such as a video camera. Yes.

  In FIG. 9, under the control of the system controller 903, the media playback circuit 920 reads AV data of the target video clip from the recording medium 907. Video data is extracted from the read AV data by the demultiplexer 921, and a moving image is reproduced from the moving image data compressed and encoded by a moving image decoding circuit 922, for example, using an encoding method such as MPEG.

  The reproduced video frame of the moving image is stored in the frame memory 908. In the first to third embodiments described above, real-time processing was performed at the time of shooting. On the other hand, in the present embodiment, it is possible to reproduce a video frame in synchronization with the feature amount detection process by controlling the process itself read from the recording medium 907 by the system controller 903. For this reason, even if the feature amount detection processing capability is not high, it is possible to perform processing for each frame.

  The feature amount detection circuit 912 detects an image feature amount for the video frame stored in the frame memory 908. In the present embodiment, an example using an image feature amount for determining a human face image will be described. In the feature amount storage memory (storage unit) 914, feature amounts necessary for recognition of a specific person are stored in advance. About the means to memorize | store this feature-value, it carries out in the procedure similar to the 1st-3rd embodiment, for example.

  The feature amount comparison circuit 915 compares the image feature amount output from the feature amount detection circuit 912 with the image feature amount stored in the feature amount storage memory 914 to thereby add a specific person to the video frame currently being reproduced. It is determined whether or not a face image is included. The feature amount comparison circuit 915 and the feature amount detection circuit 912 function as feature amount detection means. Based on the reproduction start information output from the system controller 903, the reproduction time counting circuit 917 counts the elapsed time from the reproduction start to the video frame for which face image determination has been performed.

  The representative image generation control circuit 916 selects a video frame to be a representative image according to the feature amount comparison result of the face image by the feature amount comparison circuit 915 and the elapsed time from the start of reproduction counted by the reproduction time counting circuit 917. decide. Then, by controlling the first switch 909, the determined video frame is stored in the representative image memory 910.

  The thumbnail image generation circuit 911 generates thumbnail image data by reducing the video frames stored in the representative image memory 910 and performing other necessary conversion processing. At this time, when a video frame including a face image of a specific person is selected as a representative image frame, additional information such as the name of the specific person stored in advance can be added to the thumbnail image data. The generated thumbnail image data is recorded on the recording medium 907 by the media recording circuit 906.

FIG. 10 is a flowchart illustrating an example of a processing procedure for determining a representative image based on a face image detection result and an elapsed time from the start of recording and creating thumbnail image data in the present embodiment.
First, in step S1001, processing is started. Next, in step S1002, the representative image generation control circuit 916 resets the face image detection flag (Flg) holding the face image detection state to “0” indicating the undetected state. Further, the elapsed time t counted by the reproduction time counting circuit 917 is reset to “0”.

  In step S1003, the playback of the video frame is started under the control of the system controller 903. In step S1004, the representative image generation control circuit 916 writes the video frame F (t) for performing the face image detection process in the frame memory 908.

  Next, in step S1005, the representative image generation control circuit 916 compares the elapsed time t with a first time period T1 that is an effective period of face image detection. As a result of the comparison, when the elapsed time t is within the first time period T1, the process proceeds to the face image detection process after step S1006. On the other hand, as a result of the comparison in step S1005, when the elapsed time t exceeds the first time period T1, the face image detection process is skipped and the process proceeds to step S1012.

  Next, in step S1006, the representative image generation control circuit 916 determines whether or not face image detection has already been performed by checking the face image detection flag (Flg). As a result of this determination, if the face image detection flag (Flg) is set to “1” indicating detection completion, the following detection processing is skipped and the process proceeds to step S1012. On the other hand, as a result of the determination in step S1006, if the face image detection flag (Flg) is “0” and no face image is detected, the process proceeds to the face image feature amount detection process in step S1007.

  Next, in step S1007, the feature amount detection circuit 912 detects the face image feature amount D (F (t)) for determining the face image of the specific person already stored, and detects it from the video frame F (t). The feature amount P is output to the feature amount comparison circuit 915. In step S1008, the feature amount comparison circuit 915 compares the face image feature amount M and the feature amount P of the specific person stored in the feature amount storage memory 914 using a predetermined function C (P, M). The comparison result K is calculated.

  Next, in step S1009, the feature amount comparison circuit 915 determines whether or not a face image of a specific person is detected in the video frame F (t) by comparing the comparison result K with a predetermined threshold value Kth. . As a result of the determination, if the comparison result K is equal to or greater than the predetermined threshold value Kth and a face image is detected, the process proceeds to step S1010. In step S1010, the representative image generation control circuit 916 writes the video frame F (t) in the representative image memory 910. In step S1011, the representative image generation control circuit 916 sets the face image detection flag (Flg) to “1” indicating detection.

  On the other hand, as a result of the determination in step S1009, if the comparison result K is less than the predetermined threshold value Kth and no face image is detected, the processing in steps S1010 and S1011 below is skipped and the process proceeds to step S1012. In this manner, face image detection is executed until the elapsed time t exceeds the first time period T1, and the detected head face image is stored in the representative image memory 910.

  Next, in step S1012, the representative image generation control circuit 916 calculates the second time period T2 that is the representative image frame position and the elapsed time t when the face image is not detected within the first time period T1. Compare. As a result of this comparison, when the elapsed time t exceeds the second time period T2, the processing of the following steps S1013 and S1014 is skipped and the process proceeds to step S1015.

  On the other hand, when the elapsed time t is within the second time period T2 as a result of the comparison in step S1012, the representative image generation control circuit 916 checks the face image detection flag (Flg) in step S1013. As a result of this check, when the face image detection flag (Flg) is set to “1” indicating detection, the processing of the following step S1014 is skipped and the process proceeds to step S1015.

  On the other hand, if the face image detection flag (Flg) is “0” as a result of the check in step S1013 and no face image is detected, the process proceeds to step S1014, and the representative image generation control circuit 916 displays the current video frame. F (t) is written into the representative image memory 910. As described above, when the face image is not detected, the content stored in the representative image memory 910 is updated with the latest video frame F (t) until the elapsed time t exceeds the second time period T2.

  Next, in step S1015, it is determined whether or not a recording stop is instructed from the system controller 903 by a user operation or the like. As a result of the determination, if recording stop is not instructed, the process proceeds to step S1016 and the process is continued. In step S1016, the reproduction time counting circuit 917 increments the elapsed time t by a predetermined detection unit. As described above, when the face image detection processing capability is high and the required detection interval is short, the increment is performed every frame, and when the detection interval is long, the increment is performed so as to be a predetermined interval process. After the increment, the process returns to step S1004 to repeat the process.

  On the other hand, if recording stop is instructed as a result of the determination in step S1015, the process advances to step S1017, and the thumbnail image generation circuit 911 generates thumbnail image data from the video frames stored in the representative image memory 910. In step S1018, the media recording circuit 906 records the generated thumbnail image data on the recording medium 907, and ends the process.

  In the present embodiment, thumbnail image data is generated for recorded video data, and real-time performance is not required as compared with the case where processing is performed at the time of recording, so that hardware costs can be suppressed.

  As described above, according to this embodiment, as described above, in home video cameras, shooting is often started after aiming at a main subject. Based on this, the face image detection result and the elapsed time from the start of shooting are used to determine whether the main subject of shooting is a person to be detected or another subject and reflect the intention of shooting It is possible to create a thumbnail image.

(Fifth embodiment)
FIG. 11 is a block diagram illustrating a configuration example of a video recording processing circuit of the image processing apparatus 1100 according to the present embodiment.
A video signal is input to the first terminal 1101 from an imaging unit including an optical system such as a lens, a sensor such as a CCD, and a camera signal processing circuit. The input video signal is compressed and encoded by a moving image encoding circuit 1104 using an encoding method such as MPEG2. The multiplexer 1105 multiplexes audio data, subcode data, and the like on the compressed and encoded moving image data. Similar to the first to third embodiments, description of video camera components other than these video recording processes is omitted.

  The AV data multiplexed by the multiplexer 1105 is recorded on the recording medium 1107 by the media recording circuit 1106. The system controller 1103 controls each block according to a recording start instruction input from the second terminal 1102, and controls recording of AV data.

  On the other hand, the video signal input via the first terminal 1101 is stored in the frame memory 1108 for each cycle of face image detection processing to be described later by the representative image generation control circuit 1116 controlling the first switch 1109. Is done. Note that this cycle may be performed every frame if the face image detection processing capability is high, or may be intermittent depending on the processing capability and a necessary detection interval.

  A feature amount detection circuit 1112 detects an image feature amount for a video frame (video signal) stored in the frame memory 1108. In the present embodiment, an example using an image feature amount for determining a human face image will be described.

  In the feature amount storage memory (storage unit) 1114, feature amounts necessary for recognition of a specific person are stored in advance. For example, a specific person such as a family is imaged as a subject before recording by the video camera. At that time, the representative image generation control circuit 1116 functions as a feature amount storage unit, and controls the second switch 1113 to the writing side. As a result, the image feature amount output from the feature amount detection circuit 1112 can be stored in the feature amount storage memory 1114. These controls are performed by the representative image generation control circuit 1116 controlling each unit based on an instruction sent from the system controller 1103.

  The feature amount comparison circuit 1115 compares the image feature amount output from the feature amount detection circuit 1112 with the image feature amount stored in the feature amount storage memory 1114, so that a specific person is added to the currently captured video frame. It is determined whether or not a face image is included. The feature quantity comparison circuit 1115 and the feature quantity detection circuit 1112 function as feature quantity detection means. Based on the recording start information output from the system controller 1103, the recording time counting circuit 1117 counts the elapsed time from the recording start to the video frame for which the face image determination is performed.

  On the other hand, the facial expression determination circuit 1124 detects a facial expression feature amount related to a human facial expression from the video frame stored in the frame memory 1108 and determines a facial expression based on the facial expression feature amount. It is known that identification of specific facial expressions such as “blink”, “laughter”, “cry”, “anger”, etc. is possible by determining facial expressions based on facial expression features. In this embodiment, human blink detection is performed. An example using this will be described. Even a video frame in which a specific person is captured is not desirable as a representative image. Therefore, the effectiveness of selecting the representative image is enhanced by detecting “blink”.

  The representative image generation control circuit 1116 uses the facial image feature amount comparison result by the feature amount comparison circuit 1115, the elapsed time from the recording start counted by the recording time counting circuit 1117, and the blink detection result by the facial expression determination circuit 1124. In response, a video frame to be a representative image is determined. Then, by controlling the third switch 1118, the image of the video frame to be the representative image is stored in the representative image memory 1110. Details of the representative image determination method based on the feature amount comparison result, the elapsed time from the start of recording, and the blink detection result will be described later with reference to a flowchart.

  The thumbnail image generation circuit 1111 performs thumbnail image data reduction or other necessary conversion processing on the representative image stored in the representative image memory 1110 to generate thumbnail image data. At this time, when a video frame including a face image of a specific person is selected as a representative image frame, additional information such as the name of the specific person stored in advance can be added to the thumbnail image data. The generated thumbnail image data is recorded on the recording medium 1107 by the media recording circuit 1106.

FIG. 12 is a flowchart illustrating an example of a processing procedure for determining a representative image based on a face image detection result, an elapsed time from the start of recording, and a facial expression determination result, and creating thumbnail image data in the present embodiment.
First, in step S1201, processing is started. Next, in step S1202, the representative image generation control circuit 1116 resets the face image detection flag (Flg) holding the face image detection state to “0” indicating the undetected state. Further, the elapsed time t counted by the recording time counting circuit 1117 is reset to “0”. Further, the blink detection flag (Eflg) for detecting “blink” is reset to “0” in the initial state.

  In step S1203, recording of the video signal is started under the control of the system controller 1103. In step S1204, the representative image generation control circuit 1116 writes the frame F (t) for the face image detection process in the frame memory 1108.

  In step S1205, the representative image generation control circuit 1116 compares the elapsed time t with a first time period T1 that is an effective period of face image detection. As a result of the comparison, when the elapsed time t is within the first time period T1, the process proceeds to the face image detection process after step S1206. On the other hand, if the elapsed time t exceeds the first time period T1 as a result of the comparison in step S1205, the face image detection process is skipped and the process proceeds to step S1212.

  Next, in step S1206, the feature amount detection circuit 1112 detects a face image feature amount D (F (t)) for determining a face image of a specific person that has already been stored from the video frame F (t). The feature amount P is output to the feature amount comparison circuit 1115. In step S1207, the feature amount comparison circuit 1115 compares the face image feature amount M and the feature amount P of the specific person stored in the feature amount storage memory 1114 using a predetermined function C (P, M). The comparison result K is calculated.

  Next, in step S1208, the feature amount comparison circuit 1115 determines whether or not a face image of a specific person has been detected in the video frame F (t) by comparing the comparison result K with a predetermined threshold value Kth. . As a result of the determination, if the comparison result K is less than the predetermined threshold value Kth and no face image is detected, the processing of the following steps S1209 to S1211 is skipped and the process proceeds to step S1212. On the other hand, if the result of determination in step S1208 is that the comparison result K is greater than or equal to the predetermined threshold value Kth and a face image is detected, the process proceeds to step S1209.

  In step S1209, the representative image generation control circuit 1116 checks the face image detection flag (Flg) to determine whether face image detection has already been performed. As a result of this determination, if the face image detection flag (Flg) is “0” and no face image has been detected, the process proceeds to step S1210. In step S1210, the representative image generation control circuit 1116 writes the video frame F (t) as a representative image in the representative image memory 1110. In step S1211, the representative image generation control circuit 1116 sets the face image detection flag (Flg) to “1” indicating that detection has been performed.

  Next, in step S1212, the representative image generation control circuit 1116 calculates a second time period T2 that is a representative image frame position and an elapsed time t when the face image is not detected within the first time period T1. Compare. As a result of this comparison, when the elapsed time t exceeds the second time period T2, the processing of the following steps S1213 and S1214 is skipped and the process proceeds to step S1215.

  On the other hand, as a result of the comparison in step S1212, if the elapsed time t is within the second time period T2, the representative image generation control circuit 1116 checks the face image detection flag (Flg) in step S1213. As a result of this check, if the face image detection flag (Flg) is set to “1” indicating detection, the detection processing in the following step S1214 is skipped and the process proceeds to step S1215.

  On the other hand, if the face image detection flag (Flg) is “0” as a result of the check in step S1213 and no face image is detected, the process advances to step S1214, and the representative image generation control circuit 1116 F (t) is written into the representative image memory 1110. As described above, when the face image is not detected, the content stored in the representative image memory 1110 is updated with the latest video frame F (t) until the elapsed time t exceeds the second time period T2.

  On the other hand, if the result of determination in step S1209 is that the face image detection flag (Flg) has been set to “1” indicating detection, the process proceeds to blink detection processing from step S1219 to step S1223.

  In step S1219, the facial expression determination circuit 1124 checks the blink detection flag (Eflg). As a result of this check, if the blink detection flag (Eflg) is “1”, it indicates that a representative image including a face image of a specific person who has not performed “blink” has already been selected. For this reason, the process from the following steps S1220 to S1223 is skipped and the process proceeds to step S1215.

  On the other hand, if the result of the check in step S1219 is that the blink detection flag (Eflg) is “0”, it indicates that a representative image including a face image of a specific person who has not “blinks” has not yet been selected. Yes. Therefore, the process proceeds to step S1220 in order to determine whether or not to blink the current video frame. In step S1220, the facial expression determination circuit 1124 detects the facial expression feature amount H (F (t)) of the video frame F (t). Then, the facial expression feature amount E related to “blink” is set as a detection result.

  Next, in step S1221, the facial expression determination circuit 1124 compares the detected facial expression feature amount E related to “blink” with a predetermined threshold Eth, thereby “blinks” the face image of the video frame F (t). It is determined whether or not. As a result of this determination, if it is determined that expression feature amount E <predetermined threshold Eth and “blink” is being performed, the processing of steps S1222 and S1223 below is skipped and the process proceeds to step S1215.

  On the other hand, as a result of the determination in step S1221, if it is determined that expression feature amount E ≧ predetermined threshold Eth and “blink” is not performed, the process proceeds to step S1222. In step S1222, the representative image generation control circuit 1116 writes the video frame F (t) as a representative image in the representative image memory 1110. In step S1223, the representative image generation control circuit 1116 sets the blink detection flag (Eflg) to “1”.

  In this way, face image detection is performed until the elapsed time t exceeds the first time period T1, and among the detected face images, the first video frame that is not “blink” is stored in the representative image memory 1110. Is done. If it is determined that the plurality of video frames have not been “blinked” up to the first time period T1, the first video frame among them is stored in the representative image memory 1110. If no face image is detected, the last processed video frame within a range not exceeding the second time period T2 is stored in the representative image memory 1110.

  Next, in step S1215, it is determined whether or not recording stop is instructed from the system controller 1103 by a user operation or the like. If the result of this determination is that there is no instruction to stop recording, processing proceeds to step S1216 and processing continues. In step S1216, the recording time counting circuit 1117 increments the elapsed time t by a predetermined detection unit. As described above, when the face image detection processing capability is high and the required detection interval is short, the increment is performed every frame, and when the detection interval is long, the increment is performed so as to be a predetermined interval process. After the increment, the process returns to step S1204 to repeat the process.

  On the other hand, if recording stop is instructed as a result of the determination in step S1215, the process advances to step 1217, and the thumbnail image generation circuit 1111 generates thumbnail image data from the video frames stored in the representative image memory 1110. In step S1218, the media recording circuit 1106 records the generated thumbnail image data on the recording medium 1107, and ends the process.

  FIG. 13 is a diagram showing a specific example of selecting a representative image by the above processing. In the example illustrated in FIG. 13, description will be made assuming that the default representative image position (second time period) T2 <face image detection period (first time period) T1 when no face image is detected.

  In the example shown in FIG. 13A, first, a face image of a specific person who “blinks” is detected at the elapsed time te, but later “blink” at the elapsed time t (t <T1). A face image of a specific person who has not performed is detected. For this reason, the video frame at the position of the elapsed time t is selected as the representative image.

  In the example shown in FIG. 13B, the face image of the specific person who “blinks” is detected at the elapsed time t (t <T1), but “blink” within the face image detection period T1. No face image is detected. For this reason, the video frame at the position of the elapsed time t is selected as the representative image.

  Further, in the example shown in FIG. 13C, T2 <T1 <elapsed time t, and the face image of the specific person was not detected within the face image detection period T1, so that the video frame at the default representative image position T2 is Selected as a representative image.

  As described above, according to the present embodiment, as described above, in a home video camera, shooting is often started after aiming at a main subject. Based on this, the face image detection result and the elapsed time from the start of shooting are used to determine whether the main subject of shooting is a person to be detected or another subject and reflect the intention of shooting Thumbnail image creation can be realized.

  In the present embodiment, it can be performed when video is recorded, and processing can be performed from recorded video data. In particular, in the latter case, since real-time processing is not necessary, processing by software is also possible. It is also possible to create thumbnail image data by combining the detection of a specific person and facial expression detection, excluding unwanted video frames such as blinking, and selecting more optimal representative images It is.

(Other embodiments according to the present invention)
Each means constituting the image processing apparatus and each step of the image processing method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable recording medium (storage medium) recording (storing) the program are included in the present invention.

  In addition, the present invention can be implemented as a system, apparatus, method, program, recording medium (storage medium), or the like, and can be applied to a system including a plurality of devices. It may also be applied to an apparatus consisting of a single device.

  Note that the present invention corresponds to the software program that implements the functions of the above-described embodiments (in the embodiment, the flowcharts shown in FIGS. 2, 6-1, 6-2, 8-1, 8-2, 10, 12). Including the case where the program is supplied directly or remotely to the system or apparatus. This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium (storage medium) for supplying the program include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. Further, the computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium (storage medium) such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  As another method, the program of the present invention is encrypted, stored in a recording medium such as a CD-ROM, distributed to users, and encrypted from a homepage via the Internet to users who have cleared predetermined conditions. Download the key information to be solved. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  Furthermore, as another method, a program read from a recording medium (storage medium) is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

It is a block diagram which shows the structural example of the video processing circuit of the image processing apparatus in the 1st Embodiment of this invention. It is a flowchart which shows an example of the process sequence which produces the thumbnail image data in the 1st Embodiment of this invention. It is a figure which shows the specific example which selects the representative image in the 1st Embodiment of this invention. It is a figure which shows the other specific example which selects the representative image in the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the video processing circuit of the image processing apparatus in the 2nd Embodiment of this invention. It is a flowchart which shows an example of the process sequence which produces the thumbnail image data in the 2nd Embodiment of this invention. It is a flowchart which shows an example of the process sequence which produces the thumbnail image data in the 2nd Embodiment of this invention. It is a figure which shows the specific example which selects the representative image in the 3rd Embodiment of this invention. It is a flowchart which shows an example of the process sequence which produces the thumbnail image data in the 3rd Embodiment of this invention. It is a flowchart which shows an example of the process sequence which produces the thumbnail image data in the 3rd Embodiment of this invention. It is a block diagram which shows the structural example of the video processing circuit of the image processing apparatus in the 4th Embodiment of this invention. It is a flowchart which shows an example of the process sequence which produces the thumbnail image data in the 4th Embodiment of this invention. It is a block diagram which shows the structural example of the video processing circuit of the image processing apparatus in the 5th Embodiment of this invention. It is a flowchart which shows an example of the process sequence which produces the thumbnail image data in the 5th Embodiment of this invention. It is a figure which shows the specific example which selects the representative image in the 5th Embodiment of this invention. It is a flowchart which shows an example of the process sequence which selects the representative image in the past.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 1st terminal 102 2nd terminal 103 System controller 104 Moving image encoding circuit 105 Multiplexer 106 Media recording circuit 107 Recording medium 108 Frame memory 109 1st switch 110 Representative image memory 111 Thumbnail image generation circuit 112 Feature amount detection circuit 113 Second switch 114 Feature amount storage memory 115 Feature amount comparison circuit 116 Representative image generation control circuit 117 Recording time coefficient circuit 118 Third switch

Claims (8)

Detect a characteristic amount of the face of the entered human product that is part of the image movies, the person is an image processing apparatus having a feature amount detection means for recognizing whether a specific person,
Feature quantity storage means for storing in the storage unit the feature quantity of the face of the specific person to be recognized by the feature quantity detection means;
Recording means for recording video on a recording medium;
From the time of recording start of the recording means images recorded by, in a short first predetermined time period than up to the point of end of recording, the feature amount of the face of the specific person which is previously stored in the storage unit based on the case where the specific person from the recorded image is recognized by the feature quantity detecting unit, the recording from the video frame a particular person has been recognized in the image in the first predetermined time period A thumbnail image as a representative image of the recorded video, and when the specific person is not recognized by the feature amount detection means from the recorded video within the first predetermined period, the recording start time It is characterized in that a thumbnail image generating means for generating the thumbnail images from the first predetermined short second predetermined image frame at the time the period of time than the period from Image processing apparatus. The image processing apparatus according to claim 1, further comprising a period changing unit that changes the first predetermined period according to a time from a recording start to a recording end of the recorded video . The feature quantity detecting unit, the first after a lapse of the predetermined time period, the image processing apparatus according to claim 1 or 2, characterized in that stopping the detection of the feature quantity of the face of pre-Symbol human product. The thumbnail image generating means, the image processing apparatus according to any one of claim 1 to 3, characterized in that additional information is added about a particular person that is recognized by the feature quantity detecting unit to the thumbnail image. Further comprising determining facial expression determination unit facial expressions of the inputted human product that is part of the image,
When there are a plurality of video frames in which the specific person is recognized by the feature amount detection unit from the video within the first predetermined period, the thumbnail image generation unit determines the facial expression determined by the facial expression determination unit. The image processing apparatus according to claim 1, wherein the thumbnail image is generated from a video frame corresponding to a predetermined facial expression. Detect a characteristic amount of the face of the entered human product that is part of the image movies, the person is an image processing method having a feature amount detection step for recognizing whether a specific person,
A feature amount storage step of storing in a storage unit a feature amount of the face of the specific person to be recognized in the feature amount detection step;
A recording process for recording video on a recording medium;
From the time of start of recording video recorded in the recording step, in a short first predetermined time period than up to the point of end of recording, the feature amount of the face of the specific person which is previously stored in the storage unit Based on the recorded video, when the specific person recognizes in the feature amount detection step, the recording is performed from the video frame in which the specific person in the video within the first predetermined period is recognized. When a thumbnail image is generated as a representative image of the recorded video and the specific person does not recognize from the recorded video in the feature amount detection step within the first predetermined period, from the time of the recording start characterized in that a thumbnail image generating process of generating the thumbnail image from the video frame of the time the first short second predetermined time period than the predetermined period of time Image processing method. Detect a characteristic amount of the face of a human product is Ru contained in movies image is entered, a feature amount detection step for recognizing whether the person is a specific person,
A feature amount storage step of storing in a storage unit a feature amount of the face of the specific person to be recognized in the feature amount detection step;
A recording process for recording video on a recording medium;
From the time of start of recording video recorded in the recording step, in a short first predetermined time period than up to the point of end of recording, the feature amount of the face of the specific person which is previously stored in the storage unit Based on the recorded video, when the specific person recognizes in the feature amount detection step, the recording is performed from the video frame in which the specific person in the video within the first predetermined period is recognized. When a thumbnail image is generated as a representative image of the recorded video and the specific person does not recognize from the recorded video in the feature amount detection step within the first predetermined period, from the time of the recording start wherein to execute the thumbnail image generation step of generating a thumbnail image to a computer from a short second predetermined image frame at the time the period of time than the first predetermined time period Program, characterized in that. A computer-readable recording medium, wherein the program according to claim 7 is recorded.

Images