JP4835545B2 - Image reproducing apparatus, imaging apparatus, image reproducing method, and computer program - Google Patents

Description

  The present invention relates to an image reproduction device, an imaging device, an image reproduction method, and a computer program. More specifically, the present invention relates to an image reproduction device, an imaging device, an image reproduction method, and a computer program that detect a human face in fast-forwarding or rewinding processing of moving image data and executing display processing of the detected face information.

  Recently, large-capacity media such as DVDs and hard disks (HD) are used as moving image data recording media. DVDs and hard disks (HD) are used as recording media such as video cameras and TV broadcasts.

  In many cases, moving image data recorded on such media is recorded after being compressed (encoded) in accordance with a predetermined compression algorithm such as MPEG compression. At the time of reproduction, a decoding process is performed and displayed on the display unit. That is, a data recording / reproducing device using such a medium is provided with a codec that performs data compression processing (encoding) and decoding processing (decoding). Note that the reproduction-only device includes a decoder that executes only the decoding process.

  In a device that reproduces moving image content, such as a general DVD player or HD recorder, when fast-forward playback processing or rewind playback processing is executed, one or two images are decoded and reproduced and displayed per second. It has become. For example, Japanese Patent Laid-Open No. 2006-269066 discloses a conventional technique that discloses DVD playback, search, and scan processing.

  That is, when performing fast forward playback processing and rewind playback processing of data recorded on a general DVD player or HD recorder, temporally discrete image display is performed.

  Such a time-spaced image display is caused by a limitation in the performance of a decoder that performs image decoding processing, but fast-forward and rewind playback images, for example, 30 frames / second, etc. Even when the image is displayed at a high frame rate, the image display changes rapidly, and there is also a reason that the user feels uncomfortable because it cannot be visually recognized.

On the other hand, however, the following problems occur when discrete display of only one or two frames per second is performed.
(A) Because the number of displayed images is small, it is difficult to cue the desired scene. (B) It is difficult to adjust the fast-forward / rewind speed because it is difficult to feel the fast-forward / rewind speed. Inability to move quickly These inconveniences occur.
JP 2006-269066 A

  The present invention has been made in view of the above-described problems. When executing special playback processing such as fast-forward playback and rewind playback of a moving image, the present invention determines a face area included in the image, and An image reproducing device, an imaging device, and an image that can facilitate confirmation of a scene desired by a user by displaying still images of face images detected together with fast-forward playback and rewind playback images It is an object to provide a reproduction method and a computer program.

The first aspect of the present invention is:
An image playback device that executes playback processing of moving image data,
A face detection unit for detecting a face area from image frames constituting moving image data;
A shot score calculation unit for calculating a score of the face detected by the face detection unit;
A best shot image selection unit that selects a face image with a high score calculated by the shot score calculation unit as a best shot image from each of preset image frame sections;
A display image control unit that executes control for displaying the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward reproduction image or the rewind reproduction image of the moving image data;
Under the control of the display image control unit, a display unit that displays a fast-forward reproduction image or a rewind reproduction image of moving image data, and a best shot image,
An image reproducing apparatus having the above.

  Furthermore, in one embodiment of the image reproduction device of the present invention, the shot score calculation unit, for each of the faces detected by the face detection unit, a face direction score corresponding to the direction of the face, and a face detection indicating the face-likeness At least one of the scores is calculated as a sub-score, and a shot score calculation process as a total score is executed based on the calculated sub-score.

  Furthermore, in an embodiment of the image reproduction device of the present invention, the shot score calculation unit sets a weight corresponding to each of the subscores, and calculates a weighted sum of the subscores as a shot score as a total score. It is characterized by that.

  Furthermore, in one embodiment of the image reproduction device of the present invention, the display image control unit pastes the best shot image on a face area in a fast-forward reproduction image or a rewind reproduction image of the moving image data. It is the structure which performs the control displayed on this.

  Furthermore, in one embodiment of the image reproducing device of the present invention, the display image control unit performs display control so that a moving speed of the best shot image displayed on the display unit is equal to or lower than a preset upper limit speed. It is characterized by being.

  Furthermore, in one embodiment of the image reproduction device of the present invention, the display image control unit executes luminance level or contrast control of a fast-forward reproduction image or a rewind reproduction image of moving image data displayed on the display unit to perform luminance It is characterized in that the display control is performed while suppressing the level change.

  Furthermore, in an embodiment of the image reproduction device of the present invention, the display image control unit controls the best shot image to be displayed in a region different from a display region of the fast-forward reproduction image or the rewind reproduction image of the moving image data. It is the structure which performs.

  Furthermore, in one embodiment of the image reproduction device of the present invention, the face detection unit is configured to detect a face area from an image frame constituting moving image data and to identify the detected face. The display image control unit is configured to control to display the face identification information detected by the face detection unit together with the best shot image displayed on the display unit.

Furthermore, the second aspect of the present invention provides
An imaging device that performs recording and playback processing of moving image data,
A face detection unit for detecting a face area from image frames constituting moving image data input from an imaging unit;
A data recording unit that records the face area detection information of the face detection unit as tag information corresponding to the moving image data on a recording medium together with the moving image data;
A shot score calculation unit that detects a face region using the tag information when reproducing moving image data recorded in the data recording unit, and calculates a score of the detected face;
A best shot image selection unit that selects a face image with a high score calculated by the shot score calculation unit as a best shot image from each of preset image frame sections;
A display image control unit that executes control for displaying the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward reproduction image or the rewind reproduction image of the moving image data;
Under the control of the display image control unit, a display unit that displays a fast-forward reproduction image or a rewind reproduction image of moving image data, and a best shot image,
There exists in an imaging device characterized by having.

Furthermore, the third aspect of the present invention provides
An image reproduction method in an image reproduction apparatus that performs reproduction processing of moving image data,
A face detection step in which a face detection unit executes detection of a face region from image frames constituting moving image data;
A shot score calculating unit that calculates a score of the face detected by the face detecting unit;
The best shot image selection unit selects a face image having a high score calculated by the shot score calculation unit as a best shot image, and selects from each preset image frame section; and
A display image in which a display image control unit executes control to display the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward playback image or the rewind playback image of the moving image data. Control steps;
A display step for displaying a fast-shot playback image or a rewind playback image of the moving image data together with a best shot image under the control of the display image control unit;
An image reproduction method characterized by comprising:

  Furthermore, in an embodiment of the image reproduction method of the present invention, the shot score calculation step includes a face direction score corresponding to the face direction and a face detection indicating the face-likeness for each of the faces detected by the face detection unit. In this step, at least one of the score is calculated as a sub-score, and a shot score calculation process as a total score is executed based on the calculated sub-score.

  Furthermore, in one embodiment of the image reproduction method of the present invention, the shot score calculating step is a step of setting a weight corresponding to each of the sub-scores and calculating a weighted sum of the sub-scores as a shot score as a total score. It is characterized by that.

  Furthermore, in one embodiment of the image reproduction method of the present invention, the display image control step includes a step of pasting the best shot image on a face area in a fast-forward reproduction image or a rewind reproduction image of the moving image data. It is the step which performs the control displayed on this.

  Furthermore, in one embodiment of the image reproduction method of the present invention, the display image control step performs display control so that a moving speed of the best shot image displayed on the display unit is equal to or lower than a preset upper limit speed. It is characterized by being.

  Furthermore, in one embodiment of the image reproduction method of the present invention, the display image control step executes luminance level or contrast control of a fast-forward reproduction image or a rewind reproduction image of moving image data displayed on the display unit to perform luminance control. It is a step of performing display control in which a change in level is suppressed.

  Furthermore, in one embodiment of the image reproduction method of the present invention, the display image control step is a control for displaying the best shot image in a separate area from the display area of the fast-forward reproduction image or the rewind reproduction image of the moving image data. It is the step which performs.

  Furthermore, in one embodiment of the image reproduction method of the present invention, the face detection step is a step of performing detection of a face region from an image frame constituting moving image data and identifying the detected face. In the display image control step, the face identification information detected in the face detection step is controlled to be displayed together with the best shot image displayed on the display unit.

Furthermore, the fourth aspect of the present invention provides
In the image reproduction device, a computer program for executing reproduction processing of moving image data,
A face detection step for causing the face detection unit to detect a face area from image frames constituting moving image data;
A shot score calculation step for causing the shot score calculation unit to calculate the score of the face detected by the face detection unit;
A best shot image selection step for causing the best shot image selection unit to select a face image having a high score calculated by the shot score calculation unit as a best shot image from each of preset image frame sections;
A display image that causes the display image control unit to execute control to display the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward playback image or the rewind playback image of the moving image data. Control steps;
Under the control of the display image control unit on the display unit, a display step of displaying a fast-forward reproduction image or a rewind reproduction image of the moving image data and a best shot image, and
There is a computer program characterized by comprising:

  The computer program of the present invention is, for example, a computer program that can be provided by a storage medium or a communication medium provided in a computer-readable format to a general-purpose computer system that can execute various program codes. . By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of one embodiment of the present invention, face area detection is performed from image frames constituting moving image data, a score of the detected face is calculated, and a face image with a high score is preliminarily set as a best shot image. Each of the set image frame sections is selected and displayed in the face area portion or another area of the fast-forward playback image or the rewind playback image of the moving image data. According to this configuration, a face image of a person that is likely to attract the user's attention is displayed as one still image in a predetermined frame section (for example, 300 frames), and the viewer can easily identify the person. The scene in which the person appears can be easily grasped.

  Hereinafter, the details of an image playback device, an imaging device, an image playback method, and a computer program according to embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a functional block diagram of an image reproduction apparatus according to an embodiment of the present invention. As shown in FIG. 1, the image reproduction apparatus 100 of the present invention includes an image input unit 101, a face detection unit 102, a shot score calculation unit 103, a best shot image selection unit 104, a display image control unit 105, and an image display unit 106. Have.

  The image input unit 101 inputs the decoded moving image data recorded on a recording medium such as a DVD or HD. There is a decoder that executes an image decoding process in the preceding stage of the image input unit 101, and the image input unit 101 inputs the moving image data decoded through the decoder.

  In this embodiment, a fast-forward playback image or a rewind playback image is displayed on the image display unit 106 of the image playback device 100. That is, image display is executed at a playback rate faster than the normal playback rate. The high-speed rate moving image data is input to the image input unit 101 and supplied from the image input unit 101 to the display image control unit 106 and the face detection unit 102.

  The face detection unit 102 performs face detection in each frame constituting the input moving image data, and outputs the detection result to the shot score calculation unit 103. The shot score calculation unit 103 calculates a “shot score” that is an index indicating whether the face image detected by the face detection unit 102 is an image that is easy for the user to see. The shot score is calculated from the orientation of the face image, the degree of shielding, and the like, and is used for selecting an image to be pasted on the screen as a representative image of the person in the fast-forward image. The shot score information calculated by the shot score calculation unit 103 is input to the best shot image selection unit 104.

  Using the shot score information calculated by the shot score calculation unit 103, the best shot image selection unit 104 uses a past frame (for example, the past 300 frames (about about the time of normal playback) from the image displayed on the image display unit 106. In a 10-second image frame or the like, a process of selecting the face image with the highest shot score as the best shot of the person is executed.

  The display image control unit 105 inputs moving image data from the image input unit 101 and performs adjustment processing of a fast-forward reproduction image or a rewind reproduction image displayed on the image display unit 106. In fast-forward and rewind playback, high-rate image display is performed. Therefore, if display is performed at the same luminance level as during normal playback, there is a risk of causing a photosensitive seizure to the viewer. The display image control unit 105 executes control of the display image such as lowering the contrast of the background image so that the change in luminance of the display image is sufficiently small to the extent that no photosensitive seizure occurs. That is, the display image control unit 105 performs display control in which the luminance level or contrast control of the fast-forward playback image or the rewind playback image of the moving image data displayed on the image display unit 106 is executed to suppress a change in the brightness level.

  In addition, the display image control unit 105 pastes the still image of the best shot face selected by the best shot selection unit 104 on the fast-forward or rewind playback image, for example, on the face area portion of the fast-forward or rewind playback image. The data is output and displayed on the display unit 106.

  The best shot image selection unit 104 uses the shot score information calculated by the shot score calculation unit 103 to use a frame in the past of a certain section from the image displayed on the image display unit 106 (for example, the past 300 frames (during normal playback) In the image frame of about 10 seconds, etc.), the face image with the highest shot score is selected as the best shot of the person, but it is pasted to the character by adjusting the section for selecting the best shot. For example, if the interval for selecting the best shot is lengthened, it will appear to the user that a still image is stuck, and if the interval is shortened, the video will be stuck. Looks like.

As for the position where the image is pasted, the display image control unit 105 sets the face position change speed [v] to a preset threshold [[ vlimit], ie,
v <vlimit
It is good also as a structure which controls control of the face moving speed to be. An example of a constraint equation for the face position change speed [v] is shown in the following (Formula 1).

... (Formula 1)
In the above formula,
t ₁ , t ₂ : frame time satisfying t ₁ <t ₂ ,
x (t), y (t): x-coordinate data and y-coordinate data of the face position in the xy image plane at time t,
It is.

  In addition to the restriction on the face position change speed [v], the face movement control may be performed using restrictions such as the relationship between the face change position and the spline curve in time.

  FIG. 2 shows an example of a fast-forward image displayed on the image display unit 106. Compared to conventional analog recording type VTR and tape type DV, HDD recorder, DVD, and PC digital content playback application can read digital data of one screen completely, so there is no noise even in fast-forward images A beautiful image can be read. Therefore, face recognition processing can be performed even in an image being fast-forwarded.

  On the other hand, in general, in order to select a scene desired by a user in contents such as a drama, a movie, and a variety program, it is only necessary to know a rough background and characters. However, in general, the characters are not always facing the front and may not be clearly visible, so that depending on the timing of the displayed image, it may be difficult for the user to identify the characters' faces.

  Moreover, the screen at the time of fast-forwarding of the present general HDD recorder or DVD player mainly displays one or two images per second in a time discrete manner. Although this is limited by the performance of the image decoder, even if a fast-forward image can be displayed at a full frame of 30 fps, the image display changes rapidly, and there is also a reason that the user cannot recognize visually and feels uncomfortable.

  Here, by making the position change and texture change of the image of the face part of the character that attracts the user's attention sufficiently small and easy to see, and by reducing the brightness change of the background part other than the face to the light-sensitive seizure countermeasure level, The user can grasp the characters while viewing the rough background information. This facilitates adjustment of the fast-forward speed and movement to a desired scene.

[Shot score calculation]
Details of the shot score calculation process in the shot score calculation unit 103, that is, a process of selecting a representative face image to be pasted on the display image from the detected face image will be described.

FIG. 3 shows a face image and score setting example detected by the face detection unit 102. The face image shown in FIG. 3 is as follows.
A: Face image with a shadow on the face by a hat B: Face image facing diagonally upper right C: Face image facing front D: Face image facing left

In addition, for each of these faces, setting examples of [Face Angle], [Face Direction Score], [Face Detection Score], and [Shot Score] are shown.
[Shot Score] is a total score applied to select a representative face image to be pasted on the display image.
[Face Direction Score] and [Face Detection Score] are sub-scores used to calculate [Shot Score], which is an overall score.
[Face angle] is used as a calculation parameter of [Face direction score] as a sub-score.
Hereinafter, a method of calculating a shot score that serves as an index for selecting a representative face image will be described.

The example described below is
(A) a face direction score based on the direction of the detected face;
(B) a face detection score based on the face image likeness of the detected face;
This is an example of calculating a shot score as an overall score using these two types of sub-scores. Note that this score calculation example is an example, and if there is an index indicating the ease of identification of the character's face other than the above (a) and (b), other sub-scores participating in the calculation of the shot score are used. It is good also as a setting to do. Less than,
(A) a face direction score based on the direction of the detected face;
(B) a face detection score based on the face image likeness of the detected face;
Details of the sub-score calculation processing will be described.

((A) Face direction score)
The face detection unit 102 determines a front face and a side face during face detection processing. For example, the learning data of the front face and the side face is held, the detected face image and the learning data are collated, and the detected face image determines the front face and the side face. As a technique for recognizing the face direction together with such face detection, for example, the technique disclosed in the following document can be applied.
[Kotaro Sabe, Kenichi Hidai, "Learning a real-time arbitrary posture face detector using pixel difference features", Proceedings of the 10th Image Sensing Symposium, pp. 547-552, 2004]

Using this face direction recognition technology, the direction of the detected face image is obtained, and face direction information is output to the shot score calculation unit 103. The shot score calculation unit 103 calculates the face direction score using the face direction angle information input from the face detection unit 102. Since the face is easier for the user to identify the face closer to the front, a higher score is assigned. An example of a calculation formula of [Face Direction Score] using the face direction angle in the Yaw direction is shown in (Formula 2) below.
Face direction score = (90− (face detection angle [deg]) / 90
... (Formula 2)

The face image and score setting example shown in FIG. 3 shows the face direction score calculated by applying the above calculation formula (Formula 2) based on the face angle information.
The face images A and C facing the front have a face angle = 0 deg, the face direction score = 1, and the highest score value is set. The face image B has a face angle = 36 deg, and the face direction score = 0. The face image D is set to face angle = −45 deg and face direction score = 0.5.

((B) Face detection score)
The face detection unit 102 determines whether the target image is a face or a non-face by performing threshold processing on a score obtained in face detection processing from an image frame. This process is described in the above-mentioned document [Kotaro Sabe, Kenichi Hidai, "Learning a Real-Time Arbitrary Posture Face Detector Using Pixel Difference Features", Proc. 547-552, 2004]. In this document, the score obtained in the face detection process is described as a “weighted majority value”. This score generally represents the likelihood of a face image, and the higher the score, the more clearly it can be used as an index for determining that the face is reflected clearly.

In the face image and score setting example shown in FIG. 3, the face detection score is set as a normalized value in the range of 0 to 1.0.
Since the face image A has a hat shadow on the top of the face, the face is detected but the score is low. Regarding B and D, since the face is not the front, the face detection is more difficult than the front face, so the score is slightly lower. Since C is a front face and has no shielding, the face detection score is high.

(Shot score)
The shot score calculation unit 103 obtains a shot score as a total score indicating the representative degree of the face image from the face direction score and the face detection score, which are the sub-scores. When each sub-score is normalized data, a shot score as a total score can be obtained by simply adding or multiplying each sub-score. If the importance level of each score is obtained by statistical processing, the shot score can be defined by a weighted sum.

Here, in obtaining a shot score [Sall] as a total score from a plurality of sub-scores,
(A) Weight [Wdir] for the face direction score as the first sub-score,
(B) Weight [Wfd] for the face detection score which is the second sub-score,
Set these subscore-corresponding weights,
Each sub-score is multiplied by a weight, and a score-corresponding weighted sum is calculated to obtain a shot score [Sall] that is an overall score. That is,
(A) Weight [Wdir] for the face direction score [Sdir] as the first sub-score,
(B) Weight [Wfd] for the face detection score [Sfd] as the second sub-score,
Is set, the shot score [Sall] is calculated according to the following equation (Equation 3).
Shot score [Sall] = (Sdir × Wdir) + (Sfd × Wfd)
... (Formula 3)
A shot score [Sall] as a total score is obtained from a plurality of sub-scores according to the above formula (Formula 3).

In the face image and score setting example shown in FIG.
(A) Weight [Wdir] = 0.6 for the face direction score [Sdir] as the first sub-score,
(B) Weight [Wfd] = 0.4 for the face detection score [Sfd] as the second sub-score,
The values of shot scores when these subscore weights are set are shown.

In the example shown in FIG.
A: Face image with shadow on face by hat: Shot score = (1 × 0.6) + (0.3 × 0.4) = 0.72
B: Face image facing diagonally upper right: Shot score = (0.6 × 0.6) + (0.7 × 0.4) = 0.64
C: Face image facing front: Shot score = (1 × 0.6) + (0.9 × 0.4) = 0.96
D: Face image facing left: Shot score = (0.5 × 0.6) + (0.8 × 0.4) = 0.62
These shot scores are calculated. In this example, since the face image C has the highest shot score value, it is selected as a representative image.

  In this way, the shot score information calculated by the shot score calculation unit 103 is input to the best shot image selection unit 104, and the best shot image selection unit 104 uses the shot score information calculated by the shot score calculation unit 103. The face image having the highest shot score in the past frame (for example, the past 300 frames (for example, an image frame of about 10 seconds during normal playback)) from the image displayed on the image display unit 106 is the best shot of the person. The display image control unit 105 selects the best shot face still image selected by the best shot selection unit 104 as the fast forward or rewind reproduction image, for example, the face area of the fast forward or rewind reproduction image. Affixed to the part, output to the image display unit 106 and displayed.

[Display image creation processing]
Next, display image creation processing executed by the display image control unit 105 will be described. The display image control unit 105 performs display screen adjustment of the fast forward or rewind screen, that is, as described above, a process of reducing the contrast in order to prevent photosensitivity attacks (photosensitivity). Further, according to the shot score information calculated by the shot score calculation unit 103, the face image with the highest shot score (best shot image) selected by the best shot image selection unit 104 is pasted to the face position on the fast forward or rewind screen. , Output and display on the image display unit 106. The display image control unit 105 executes control to display the best shot image selected for each image frame section on the image display unit 106 in synchronization with the display of the fast-forward playback image or the rewind playback image of the moving image data.

  A specific display example is shown in FIG. As shown in FIG. 4, the image displayed on the image display unit 106 is a display image in which still images 211 and 212 of a human face are pasted on a high-rate playback image 201 by fast forward or rewind. The face still images 211 and 212 are face images (best shot images) having the highest shot score selected by the best shot image selection unit 104 according to the shot score information calculated by the shot score calculation unit 103, respectively. The face image with the highest shot score in a frame in a certain interval past the image displayed in 106 (for example, the past 300 frames (such as an image frame of about 10 seconds during normal playback)).

As described above, the position where the face still images 211 and 212 are pasted is the position of the face of the high-rate playback image 201 by fast-forwarding or rewinding. Since it becomes difficult to follow, the display image control unit 105 sets the face position change speed [v] to be equal to or less than a preset threshold value [vlimit], that is,
v <vlimit
It is good also as a structure which controls control of the face moving speed to be. The constraint formula for the face position change speed [v] is, for example, according to (Formula 1) described above, and the display image control unit 105 changes the display position of the still images 211 and 212 of the face according to the constraint. Process.

  Further, as in the image reproduction device 230 shown in FIG. 5, the face detection unit of the image reproduction device described above with reference to FIG. 1 may have a face identification function. The face detection and face identification unit 231 shown in FIG. 5 detects a face area included in the image frame input from the image input unit 101, and further performs a face identification process for identifying who the detected face is. Execute. The face database 232 stores data in which facial feature information and human identification information such as a person name are associated with each other, and the face detection and face identification unit 231 detects from an image frame input from the image input unit 101. The face area feature information and the face feature information stored in the face database 232 are collated, a person with a high matching degree is determined to be the person of the detected face, and the person name information registered in the face database 232 Are output to the shot score calculation unit 103 together with the detected face information.

  The person name is further input to the display image control unit 105 through the best shot image selection unit 104 as information corresponding to the face image of the best shot. The display image control unit 105 displays the person name detected by the face detection and face identification unit 231 together with the face image (best shot image) having the highest shot score selected by the best shot image selection unit 104 on the fast forward or rewind screen. The image is pasted on the face position and output and displayed on the image display unit 106.

  A specific display example is shown in FIG. As shown in FIG. 6, the image displayed on the image display unit 106 has a human face still image 211, 212 pasted on a high-rate playback image 201 by fast-forwarding or rewinding. It becomes a display image in which a person name is pasted as the person identification information 251 and 252 specified in the identification unit 231. The face still images 211 and 212 are face images (best shot images) having the highest shot score selected by the best shot image selection unit 104 according to the shot score information calculated by the shot score calculation unit 103, respectively. The face image with the highest shot score in a frame in a certain interval past the image displayed in 106 (for example, the past 300 frames (such as an image frame of about 10 seconds during normal playback)).

  As another display example, the fast-forward screen and the detected face image may be displayed in a separate window as shown in FIG. In the example shown in FIG. 7, still images 311 and 312 of human faces included in the reproduction image 301 are displayed in an independent display area different from the high-rate reproduction image 301 by fast-forwarding or rewinding. It is a display image displaying a person name as person identification information 351, 352 of a person specified by the detection and face identification unit 231.

  In the case of this display example, since the face image does not move on the screen, the position of the character in the image cannot be grasped, but there is an advantage that it is easy to grasp who is appearing.

  Note that face recognition processing such as face detection and face identification processing may be performed in parallel when image display is performed on the image display unit 106, that is, when fast-forward playback or rewind playback is performed. If the device is a video camera or other device, it is executed during video recording processing or when the device is not in use, and the image data is recorded on the recording media as face information and face identification information as tag information corresponding to each image frame. It is good also as a structure recorded with it. The tag information may be read when fast-forwarding or rewinding reproduction is performed, and display data pasted with data such as a still image of a face or a person name may be generated.

  That is, the face detection unit detects a face area from the image frame constituting the moving image data to be recorded in the moving image recording process on the recording medium, and the data recording unit uses the detection information as tag information to record the recording medium. To record.

  The shot score calculation unit uses the tag information when reproducing the moving image data recorded in the data recording unit, detects the face area, calculates the score of the detected face, and the best shot image selection unit A face image with a high score calculated by the score calculation unit is selected as a best shot image from each preset image frame section, and the display image control unit synchronizes with the display of the fast-forward playback image or the rewind playback image of the moving image data. Then, control for displaying the best shot image selected for each image frame section on the display unit is executed.

  When face detection or face recognition processing is performed during fast forward or rewind playback, a storage medium for storing tag information and the like is unnecessary, and a mechanism for managing tag information is also unnecessary, but calculation processing can be executed in real time. Requires a fast computing unit. On the other hand, when the face recognition process is performed when the device is in a quiet period, there is no need to increase the processing speed of the face detection or the face recognition process.

  In the above-described embodiment, the section of the image displaying the face image (best shot image) having the highest shot score selected by the best shot image selection unit 104 is, for example, 300 frames (an image of about 10 seconds during normal reproduction). Frame), and a processing example in which one best shot is selected every 300 frames has been described. However, the time interval for selecting the best shot image is fast-forward playback that is controlled by the user in order to provide consistent image display. A method of changing according to the speed and the rewind playback speed can be considered. The same applies to parameters such as the constraint value of the face position change speed.

  For example, regarding the time interval for selecting the best shot image, control is performed such that 10 seconds at the standard speed is set when fast-forwarding is performed at 5 times speed, and 20 seconds at the standard speed is set when fast-forwarding is performed at 10 times speed. By doing so, the update frequency of the face image can be made constant regardless of the fast-forwarding speed.

  In this case, information from the input unit for inputting user operation information is input to the best shot image selection unit 104 and the display image control unit 105, and the best shot image selection unit 104 responds to this input information. A process of changing the selection time interval and controlling the moving speed of the face image displayed by the display image control unit 105 is performed.

According to the image reproducing apparatus of the present invention, the user can visually view an image with a high frame rate (for example, about 30 frames per second) without a burden during fast-forwarding.
(A) You will be able to cue accurately without skipping the scene you want to see. (B) It will be easier to grasp the fast-forward speed visually, and you can move quickly to the desired scene. Brought about.

  In the configuration of the present invention, the facial image of the character that is likely to attract the attention of the user is displayed as one still image in a predetermined frame section (for example, 300 frames), and the update frequency is low (1 per second). , About two frames), and is displayed as an image, so that a user as a viewer can easily identify a person. In addition, as described above, the movement speed restriction is applied to the human face image, so that the user can easily follow the movement with the eyes, and the background image is checked for countermeasures against light-sensitive seizures (photosensitivity). Since the background image is controlled so as to suppress a large change in luminance over time, the visual burden on the user is reduced. Since the user can easily grasp the scene situation and the fast-forward speed visually, the user can easily adjust the fast-forward speed and control the stop timing.

  Finally, an example of a hardware configuration of a video camera that is an imaging apparatus as an example of the image reproducing apparatus of the present invention will be described with reference to FIG. The video camera 500 inputs a shooting signal acquired by the imaging unit 501 such as a lens or a charge coupled device (CCD) to the analog signal processing unit 502, performs processing such as noise removal in the analog signal processing unit 502, and performs A / D The conversion unit 503 converts the digital signal. The digitally converted data is subjected to digital signal processing such as data compression in the digital signal processing unit 504 and then recorded in a recording device 515 configured by, for example, a flash memory. Further, it is displayed on the monitor 517 and the viewfinder (EVF) 516.

  The operation unit 511 includes, for example, a shutter provided in the camera body, and switches for starting processing such as a playback designation switch for data recorded in the recording device 515 and fast-forward playback and rewind playback.

  Control unit 510 has a CPU, and executes control of various processes executed by the video camera in accordance with a program stored in advance in memory (ROM) 520 or the like. A memory (EEPROM) 512 is a non-volatile memory, and stores image data, various auxiliary information, programs, and the like. A memory (ROM) 520 stores programs, calculation parameters, and the like used by the control unit (CPU) 510. The memory (RAM) 521 stores programs used in the control unit (CPU) 510, parameters that change as appropriate during execution thereof, and the like.

  When a processing request such as fast-forward playback or rewind playback of data recorded in the recording device 515 is input from the operation unit 511, it has been described above with reference to FIGS. Processing is executed. That is, the image data stored in the recording device 515 is input to the digital signal processing unit 504, and decoding is performed.

  The face detection unit 505 detects a face area from the decoded image and inputs the detection result to the digital signal processing unit 504. The digital signal processing unit 504 performs processing such as the above-described calculation of the shot score, selection of the best shot image, selection of the selected best shot image, and the like to the fast forward or rewind playback image displayed on the monitor 517. Paste the best shot image and output.

  In the above description, the calculation of the shot score, the selection of the best shot image, and the control of the display image have been described as being executed by the digital signal processing unit 504. However, these processes may be executed by the control unit 510. Moreover, it is good also as a structure which performs both in cooperation.

  Further, the face detection unit 505 illustrated in FIG. 8 may be set as a processing unit that performs face recognition. For example, as a configuration in which the face database described above with reference to FIG. 5 is set in a part of the data recording area of the memory (EEPROM), the face detection unit 505 performs face identification along with face detection and is detected. The person name corresponding to the face may be output to the monitor 517 as the display unit via the digital signal processing unit 504.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and can be installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the embodiment of the present invention, the face area is detected from the image frame constituting the moving image data, the score of the detected face is calculated, and the face having a high score is obtained. An image is selected from each image frame section set in advance as a best shot image and displayed in the face area portion of the fast-forward playback image or the rewind playback image of the moving image data, or in another area. According to this configuration, a face image of a person that is likely to attract the user's attention is displayed as one still image in a predetermined frame section (for example, 300 frames), and the viewer can easily identify the person. The scene in which the person appears can be easily grasped.

It is a block diagram explaining the structure of the image reproduction apparatus which concerns on one Example of this invention, and the image reproduction process to perform. It is a figure which shows the example of a display image of the fast-forward image displayed on the image display part of the image reproduction apparatus which concerns on one Example of this invention. It is a figure which shows the face image detected in the face detection part of the image reproduction apparatus which concerns on one Example of this invention, and a score setting example. It is a figure which shows an example of the display image displayed on the image display part of the image reproduction apparatus which concerns on one Example of this invention. It is a block diagram explaining the structure of the image reproduction apparatus which concerns on one Example of this invention, and the image reproduction process to perform. It is a figure which shows the example of a display image of the fast-forward image displayed on the image display part of the image reproduction apparatus which concerns on one Example of this invention. It is a figure which shows the example of a display image of the fast-forward image displayed on the image display part of the image reproduction apparatus which concerns on one Example of this invention. It is a figure explaining the hardware structural example of the video camera as an image reproduction apparatus which concerns on one Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image reproduction apparatus 101 Image input part 102 Face detection part 103 Shot score calculation part 104 Best shot image selection part 105 Display image control part 106 Image display part 201 Reproduction image 211,212 Still image of face 230 Image reproduction apparatus 231 Face detection and Face identification unit 232 Face database 251, 252 Person identification information 301 Playback image 311, 312 Face still image 351, 352 Person identification information 500 Video camera 501 Imaging unit 502 Analog signal processing unit 503 A / D conversion unit 504 Digital signal processing unit 505 Face detection unit 510 Control unit 511 Operation unit 512 Memory 515 Recording device 516 Viewfinder 517 Monitor 520 Memory (ROM)
521 Memory (RAM)

Claims (18)

An image playback device that executes playback processing of moving image data,
A face detection unit for detecting a face area from image frames constituting moving image data;
A shot score calculation unit for calculating a score of the face detected by the face detection unit;
A best shot image selection unit that selects a face image with a high score calculated by the shot score calculation unit as a best shot image from each of preset image frame sections;
A display image control unit that executes control for displaying the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward reproduction image or the rewind reproduction image of the moving image data;
Under the control of the display image control unit, a display unit that displays a fast-forward reproduction image or a rewind reproduction image of moving image data, and a best shot image,
An image reproducing apparatus comprising: The shot score calculation unit
For each of the faces detected by the face detection unit,
At least one of a face direction score corresponding to the face direction and a face detection score indicating the likelihood of a face is calculated as a subscore, and a shot score calculation process as a total score is executed based on the calculated subscore. The image reproducing apparatus according to claim 1, wherein: The shot score calculation unit
The image reproducing device according to claim 2, wherein a weight corresponding to each of the sub-scores is set, and a weighted sum of the sub-scores is calculated as a shot score as a total score. The display image control unit
2. The control according to claim 1, wherein the best shot image is pasted on a face area in the fast-forward playback image or the rewind playback image of the moving image data and displayed on a display unit. Image playback device. The display image control unit
5. The image reproducing device according to claim 4, wherein display control is performed so that a moving speed of the best shot image displayed on the display unit is equal to or lower than a preset upper limit speed. The display image control unit
The display control is performed by executing a luminance level or contrast control of a fast-forward reproduction image or a rewind reproduction image of moving image data displayed on the display unit to suppress a change in luminance level. The image reproducing device described in 1. The display image control unit
2. The image playback apparatus according to claim 1, wherein the best shot image is configured to execute control for displaying the best shot image in a region different from a display region of the fast-forward playback image or the rewind playback image of the moving image data. The face detection unit
The face area is detected from the image frames constituting the moving image data, and the detected face is identified.
The display image control unit
The image reproducing apparatus according to claim 1, wherein control is performed to display face identification information detected by the face detection unit together with a best shot image displayed on a display unit. An imaging device that performs recording and playback processing of moving image data,
A face detection unit for detecting a face area from image frames constituting moving image data input from an imaging unit;
A data recording unit that records the face area detection information of the face detection unit as tag information corresponding to the moving image data on a recording medium together with the moving image data;
A shot score calculation unit that detects a face region using the tag information when reproducing moving image data recorded in the data recording unit, and calculates a score of the detected face;
A best shot image selection unit that selects a face image with a high score calculated by the shot score calculation unit as a best shot image from each of preset image frame sections;
A display image control unit that executes control for displaying the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward reproduction image or the rewind reproduction image of the moving image data;
Under the control of the display image control unit, a display unit that displays a fast-forward reproduction image or a rewind reproduction image of moving image data, and a best shot image,
An imaging device comprising: An image reproduction method in an image reproduction apparatus that performs reproduction processing of moving image data,
A face detection step in which a face detection unit executes detection of a face region from image frames constituting moving image data;
A shot score calculating unit that calculates a score of the face detected by the face detecting unit;
The best shot image selection unit selects a face image having a high score calculated by the shot score calculation unit as a best shot image, and selects from each preset image frame section; and
A display image in which a display image control unit executes control to display the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward playback image or the rewind playback image of the moving image data. Control steps;
A display step for displaying a fast-shot playback image or a rewind playback image of the moving image data together with a best shot image under the control of the display image control unit;
An image reproduction method comprising: The shot score calculating step includes:
For each of the faces detected by the face detection unit,
At least one of a face direction score corresponding to the face direction and a face detection score indicating the likelihood of a face is calculated as a sub-score, and a process of calculating a shot score as a total score is executed based on the calculated sub-score. The image reproduction method according to claim 10. The shot score calculating step includes:
The image reproducing method according to claim 11, wherein a weight corresponding to each of the sub-scores is set, and a weighted sum of the sub-scores is calculated as a shot score as a total score. The display image control step includes:
11. The step of executing control for pasting the best shot image on a face area in a fast-forward playback image or a rewind playback image of the moving image data and displaying the best shot image on a display unit. Image playback method. The display image control step includes:
The image reproduction method according to claim 13, wherein display control is performed so that a moving speed of the best shot image displayed on the display unit is equal to or lower than a preset upper limit speed. The display image control step includes:
11. The step of performing display control in which luminance level or contrast control of a fast-forward reproduction image or a rewind reproduction image of moving image data displayed on the display unit is executed to suppress a change in luminance level. The image reproduction method described in 1. The display image control step includes:
11. The image reproduction method according to claim 10, wherein the best shot image is displayed in a region different from a display region of the fast-forward reproduction image or the rewind reproduction image of the moving image data. The face detection step includes
A step of performing detection of a face region from image frames constituting moving image data and identifying the detected face;
The display image control step includes:
The image reproduction method according to claim 10, wherein control for displaying the face identification information detected in the face detection step together with a best shot image displayed on a display unit is performed. In the image reproduction device, a computer program for executing reproduction processing of moving image data,
A face detection step for causing the face detection unit to detect a face area from image frames constituting moving image data;
A shot score calculation step for causing the shot score calculation unit to calculate the score of the face detected by the face detection unit;
A best shot image selection step for causing the best shot image selection unit to select a face image having a high score calculated by the shot score calculation unit as a best shot image from each of preset image frame sections;
A display image that causes the display image control unit to execute control to display the best shot image selected for each image frame section on the display unit in synchronization with the display of the fast-forward playback image or the rewind playback image of the moving image data. Control steps;
Under the control of the display image control unit on the display unit, a display step of displaying a fast-forward reproduction image or a rewind reproduction image of the moving image data and a best shot image, and
A computer program characterized by comprising: