JP5665380B2 - Image processing apparatus, image processing apparatus control method, program, and recording medium - Google Patents

Description

  The present invention relates to an image processing apparatus that outputs an image from recorded data, a control method for the image processing apparatus, a program, and a recording medium.

  2. Description of the Related Art Conventionally, some video cameras, moving image playback apparatuses, and the like have a function of displaying a moving image representative image called a thumbnail for each moving image data when displaying a list of the captured moving image data. By viewing the thumbnail, the user can grasp the contents of each moving image data without reproducing a plurality of moving image data.

  As a conventional thumbnail creation method, a method of using the first frame of moving image data as a thumbnail is generally used. By using the first frame of the moving image data as a thumbnail, the processing for determining the thumbnail becomes very simple.

  However, when a plurality of moving images starting from the same scene and having different contents are taken, using the thumbnail creation method described above results in similar thumbnails that are difficult to identify during playback. With regard to such a problem, for example, a method for suppressing the creation of similar thumbnails for a plurality of moving image data has been proposed (see Patent Document 1). In Patent Document 1, by extracting frames having different characteristics from a plurality of moving image data and using them as thumbnails, a plurality of moving images starting from similar scenes can be easily identified.

JP 2007-006129 A

  However, in the above conventional example, the condition that the creation of similar thumbnails is suppressed for a plurality of video data is used, so that different thumbnails are created even for videos with exactly the same content, It cannot be said that it represents well.

  In addition, as described above, the method of using the first frame of moving image data as a thumbnail also creates the same thumbnail for moving image data that starts from the same scene and has different contents, so that the contents of the moving image can be identified. Have difficulty.

The present invention has been made in view of the above problems, and can reduce the difficulty of content identification that occurs when a thumbnail is created from the first frame of a moving image, and faithfully represents the content of the moving image. The purpose is to provide a technology that enables easy-to-understand thumbnails to be created.

To achieve the above object, the present invention provides a first representative frame extracting means for extracting a first representative frame from moving image data based on a first condition, and the first condition from the moving image data. A second representative frame extracting means for extracting a second representative frame based on a different second condition, and associating the extracted first representative frame and second representative frame as thumbnails with the moving image data Control means for storing; and face recognition means for recognizing a person's face from the video data; and the second representative frame extraction means recognizes a face of the video data by the face recognition means. A frame is extracted as the second representative frame, and the first representative frame extracting means extracts a frame whose face is not recognized by the face recognition means from the moving image data. Extracted as a table frame, wherein the control means is characterized by storing the image obtained by synthesizing the said first representative frame second representative frame in association with the moving image data as the thumbnail.

According to the present invention, it is possible to reduce the difficulty of content identification that occurs when a thumbnail is created from the first frame of a moving image. In addition , it is possible to create an easy-to-understand thumbnail that faithfully represents the content of the moving image.

1 is a block diagram illustrating a schematic configuration of a video camera that is an example of an image processing apparatus according to an embodiment of the present invention. It is a flowchart of the thumbnail creation processing in the first embodiment. (A) is a flowchart showing details of the first representative frame extraction process in step S130 of FIG. 2, and (b) is a flowchart showing details of the second representative frame extraction process in step S140 of FIG. It is a flowchart which shows the detail of the process which produces the thumbnail in step S150 of FIG. It is a figure which shows an example of the thumbnail when moving image data is displayed as a list on the monitor in the first embodiment. 12 is a flowchart illustrating an example of processing for creating a thumbnail from two representative frames in the second embodiment. (A) is a 1st representative frame, (b) is a 2nd representative frame, (c) is a figure which shows the thumbnail produced by synthesize | combining a 1st representative frame and a 2nd representative frame. It is a figure which shows an example of the thumbnail when moving image data is displayed as a list on the monitor in the second embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a schematic configuration of a video camera which is an example of an image processing apparatus according to an embodiment of the present invention.

  In FIG. 1, reference numeral 101 denotes a lens, and reference numeral 102 denotes an image sensor such as a CCD or CMOS that converts light collected by the lens 101 into electric charges. Reference numeral 103 denotes a signal processing circuit that converts the charge generated by the image sensor 102 into a video signal of a predetermined format. A lens control circuit 104 controls the lens 101. A CPU 105 controls the entire video camera system. Reference numeral 112 denotes a RAM which serves as a work area for the CPU 105. Reference numeral 113 denotes a ROM having a program necessary for controlling the video camera.

  Reference numeral 106 denotes a recording medium such as a DVD, a hard disk, or a nonvolatile memory. Reference numeral 107 denotes a recording medium control circuit that controls input / output of data between the CPU 105 and the recording medium 106. Reference numeral 108 denotes a monitor that outputs (displays) an image being shot and a reproduced image. Reference numeral 109 denotes a reproduction circuit for displaying the moving image data read from the recording medium 106 on the monitor 108 as an image. Reference numeral 110 denotes an OSD (On Screen Display) circuit for displaying menus and shooting information on a monitor. Reference numeral 111 denotes an operation switch for the user to operate the video camera. Reference numeral 114 denotes a real-time clock that transmits date information and time information to the CPU 105. Reference numeral 115 denotes an external input / output unit, which can output captured images to an external monitor or recorder, or input images from another video camera or player.

  In the present embodiment, the CPU 105 is assumed to have a known face recognition function for recognizing a person's face from the photographed subject.

[First Embodiment]
In the first embodiment, an example will be described in which two frames of a main person frame and a landscape frame are used as thumbnails from moving image data recorded at the time of shooting with a video camera.

  First, a method for extracting a main person frame and a landscape frame will be briefly described.

When creating a representative frame of the main person, the face with the longest face recognition time is determined as the main face among the subjects photographed using the face recognition function, and the frame including the main face is the most A frame facing the front is extracted as a representative frame of the main person. That is, these are distinguished from the first condition described later as the second condition. For example, when the CPU 105 performs face recognition, the face detection time is compared with the previously recognized face detection time, and the face with the long detection time is recorded in the RAM 112 as the main face. When a new face is recognized, the CPU 105 compares the previously recorded main face with the newly recognized face detection time. If the newly recognized face detection time becomes longer, the CPU 105 selects the new face as the main face. A new face is recorded in the RAM 112. Finally, the frame with the face with the longest detection time in the moving image is recorded in the RAM 112 as the main face, and the frame with the face facing most front is extracted from the recorded frames as the representative frame. To do. In the present embodiment, the detection time of the recognized face is used as the main face determination criterion. However, the present invention is not limited to this, and the number of times of detection may be used. Moreover, the face of the person selected by the user may be sufficient.

When creating a landscape representative frame, a frame having the widest angle of view among photographed subjects is used as the landscape representative frame. That is, these are the first conditions. Specifically, the CPU 105 determines the width of the angle of view based on a value calculated from the zoom magnification and the distance to the subject calculated by AF. For example, when the zoom is 1x and the distance to the subject is 1, and the angle of view is (A, B), the angle of view is increased under the condition that the zoom is Cx and the distance to the subject is D. When calculated, the angle of view is (A / C × D, B / C × D). Then, the CPU 105 calculates the angle of view for each frame and records the calculated value in the RAM 112. When there is a frame whose face is not recognized, a frame with the widest angle of view is extracted as a representative frame of the landscape among the frames whose face is not recognized. If there is no frame whose face is not recognized, the frame with the widest angle of view is extracted as the representative frame of the landscape.

  As described above, in the first embodiment, based on the shooting information indicating which face is the main face having the longest face recognition time and which frame is the most facing frame among the main faces. Determine the representative frame of the main person. Also, the representative frame of the landscape is determined based on the shooting information indicating which frame has the widest angle of view.

  Next, the thumbnail creation process in the first embodiment will be described with reference to FIG. This thumbnail creation process is realized by the CPU 105 developing and executing a program stored in the ROM 113 on the RAM 112.

  The flow in FIG. 2 starts when the power of the video camera is turned on by the operation switch 111.

  First, in step S100, the CPU 105 determines whether or not video recording has started. If the recording has started, the process proceeds to step S110, and if not, the process returns to step S100.

  In step S110, the CPU 105 records the captured moving image data on the recording medium 106, and the field of view, the main face, and the face direction that are necessary to identify the main person and the landscape are facing the front. Three pieces of shooting information of the current frame are recorded in the RAM 112. That is, in step S110, shooting information is acquired and recorded in the RAM 112 by the main person frame and landscape frame extraction method described above. Then, the process proceeds to step S120.

  In step S120, the CPU 105 determines whether the recording is finished. When the recording is finished, the process proceeds to step S130, and when the recording is not finished, the process returns to step S110.

  In step S130, the CPU 105 executes a first representative frame extraction process, and proceeds to step S140. Step S130 is an example of a first representative frame extraction unit. Details of the extraction process of the first representative frame will be described later.

  In step S140, the CPU 105 executes a second representative frame extraction process, and proceeds to step S150. Step S140 is an example of second representative frame extraction means. Details of the second representative frame extraction process will be described later.

  In step S150, the CPU 105 executes a process of creating a thumbnail from the first representative frame and the second representative frame, and ends this process. Details of step S150 will be described later.

  Next, the details of the first representative frame extraction process in step S130, the second representative frame extraction process in step S140, and the thumbnail creation process in step S150 will be described.

  FIG. 3A is a flowchart showing details of the first representative frame extraction process in step S130 of FIG. In the first embodiment, it is assumed that the first representative frame extraction process extracts a landscape.

  In FIG. 3A, in step S131, the CPU 105 refers to the main face information in the shooting information and determines whether there is a frame whose face is not recognized from the recorded moving image data. That is, the CPU 105 functions as a first determination unit. As a result, when it is determined that there is a frame whose face is not recognized, the process proceeds to step S132, and when it is determined that there is no frame whose face is not recognized, the process proceeds to step S133.

  In step S132, the CPU 105 sets only a frame whose face is not recognized as the first representative frame candidate, and proceeds to step S134.

  In step S133, the CPU 105 sets all the frames as first representative frame candidates, and proceeds to step S134.

  In step S134, the CPU 105 refers to the information on the angle of view in the shooting information, sets the first representative frame candidate having the widest view angle as the first representative frame, and returns. The width of the angle of view is determined from the data recorded in the RAM 112.

  FIG. 3B is a flowchart showing details of the second representative frame extraction process in step S140 of FIG. In the first embodiment, it is assumed that the second representative frame extraction process extracts a representative person's face.

  In step S141, the CPU 105 refers to the shooting information and determines whether there is a frame whose face is recognized from the recorded moving image data. As a result, if it is determined that there is a face-recognized frame, the process proceeds to step S142. If it is determined that there is no face-recognized frame, the process returns.

  In step S142, the CPU 105 sets the frame including the face recognized the longest as the second representative frame candidate, and proceeds to step S143. The second representative frame candidate is a frame including a face recorded as a main face in the RAM 112.

  In step S143, the CPU 105 sets the second representative frame candidate having the face most facing the front as the second representative frame, and returns. The second representative frame is a frame recorded as a face frame close to the front in the RAM 112.

  FIG. 4 is a flowchart showing details of the process of creating a thumbnail in step S150 of FIG.

  In step S151, the CPU 105 determines whether or not both the first representative frame and the second representative frame have been extracted. If it is determined that both are extracted, the process proceeds to step S152 and both are extracted. If it is determined that it has not been performed, the process proceeds to step S153. The case where two are not extracted is a case where there is no frame whose face is recognized during recording and only the first representative frame is extracted.

  In step S152, the CPU 105 stores the extracted two representative frames as thumbnails.

  In step S153, the CPU 105 determines whether one representative frame has been extracted. If it is determined that one representative frame has been extracted, the process proceeds to step S154. The case where one representative frame is extracted means that the representative frame is obtained from only one of the first representative frame extraction process in FIG. 3A and the second representative frame extraction process in FIG. This is the case when extracted.

  If it is determined in step S153 that no representative frame has been extracted, the process proceeds to step S155. The case where no representative frame is extracted means that, for example, when recording moving image data input from the external input / output unit 115 to the present video camera, it is not possible to refer to necessary shooting information, and there is no representative frame. This is the case when it is not extracted.

  In step S154, the CPU 105 stores the first frame of the moving image and the extracted frame as thumbnails.

  In step S155, the CPU 105 stores the first frame of the moving image and the middle frame of the moving image as thumbnails. The middle frame of the movie is a frame that is exactly in the middle from the start to the end of shooting.

  The thumbnails created in steps S152, S154, and S155 are recorded on the recording medium 106 by the CPU 105 in association with the moving image data. When the moving image data is displayed in a list on the monitor 108, two thumbnails are displayed side by side with respect to one moving image data as shown in FIG. In the illustrated example, thumbnail 1 is the first representative frame and thumbnail 2 is the second representative frame. Further, the file name of the moving image data is displayed in the display portion of the moving image data 1 to 4.

  As described above, according to the first embodiment, a thumbnail is created from two representative frames of a main person and a landscape representative frame in the moving image data, so that the contents of the moving image are faithfully expressed. You can create thumbnails. In particular, by using a frame of a main person who is highly likely to be a main subject as one of the representative frames for creating a thumbnail, it is possible to clearly indicate who is the moving image. And since the representative frame of the landscape is determined under conditions different from the conditions for extracting the representative frame of the main person, even when the same person is shot, the difference in shooting location can be identified based on the landscape, and each video data Identification becomes easy.

  In the first embodiment, the landscape and the main face are extracted as the representative frames based on the angle of view information and the face recognition information. However, the representative frames may be extracted using other extraction methods. For example, the frames may be extracted under arbitrary conditions such as high brightness and loudness. It can be considered that the high brightness can be used when determining whether the sound is outdoor or indoor, and that the sound can be used when determining whether or not the loudness of the sound is rising.

  In the first embodiment, as shown in FIG. 5, the first representative frame and the second representative frame are displayed side by side, but the display method is not limited to this. For example, two may be arranged vertically, or two may be arranged in different sizes.

[Second Embodiment]
In the second embodiment, as in the first embodiment, two frames of the main person frame and the landscape frame are extracted from the video shot by the video camera, but they are not displayed side by side. An example of creating a thumbnail by combining two images will be described. Since the thumbnail creation process and the representative frame extraction process in the second embodiment are the same as those in the flowchart of FIG. 2 and the flowchart of FIG. 3 described in the first embodiment, description thereof will be omitted.

  Hereinafter, processing for creating a thumbnail from the first and second representative frames, which is processing in step S150 of FIG. 2 in the second embodiment, will be described with reference to FIG. Note that each processing in the thumbnail creation is realized by the CPU 105 developing and executing a program stored in the ROM 113 on the RAM 112.

  FIG. 6 is a flowchart illustrating an example of a process for creating a thumbnail from two representative frames in the second embodiment. Note that steps S151 and S153 in FIG. 6 are the same as steps S151 and S153 in FIG. 4 described in the first embodiment, and thus the description thereof is omitted.

  In step S252 of FIG. 6, the CPU 105 combines the extracted two representative frames and saves them as thumbnails. The method for synthesizing the representative frame will be specifically described using the example of the representative frame shown in FIG.

  FIG. 7A shows a first representative frame, and FIG. 7B shows a second representative frame. FIG. 7C shows a thumbnail created by combining the first representative frame and the second representative frame.

  First, the CPU 105 cuts out the face recognized portion in the second representative frame. Next, the CPU 105 enlarges or reduces the cut face portion, and composes it as shown in FIG. 7C on the upper right of the first representative frame (the entire range of the representative frame of the landscape may be used). . Then, the synthesized image is saved as a thumbnail.

  In step S254, the CPU 105 saves the extracted representative frame as a thumbnail.

  In step S255, the CPU 105 stores the first frame of the moving image as a thumbnail.

  The thumbnails created in steps S252, S254, and S255 are recorded on the recording medium 106 in association with the moving image data.

  FIG. 8 is a diagram showing an example of thumbnail display when moving image data is displayed as a list on the monitor 108.

  In FIG. 8, moving image data 1, 2, 4, 6, and 7 are frames in which a face is recognized during shooting, and thumbnails obtained by combining two representative frames are displayed. The moving image data 3, 5, and 8 have no frames whose faces are recognized during shooting, and one representative frame is displayed as a thumbnail.

  As described above, according to the second embodiment, by creating a thumbnail by synthesizing the representative frame of the main person in the moving image data and the representative frame of the landscape, it is easy to understand the moving image content in a limited display range. You can create thumbnails. Even if a part of the landscape is hidden, it does not interfere with grasping the contents, and even if the face is cut out, someone can understand and there is no problem even if there is no surroundings. The specified area can be used effectively.

  In the second embodiment, two representative frames are combined and output as shown in FIG. 7C, but the combining method is not limited to this. For example, the clipped main face may be combined with the lower left instead of the upper right, or the two representative frames may be combined with enlargement or reduction to an arbitrary size.

  As described above, according to the present invention, it is possible to create a thumbnail whose contents are easily understood by using two representative frames extracted under two different conditions. In addition, by using the main person in the moving image data as a thumbnail, it is possible to clearly indicate who the moving image was captured. Furthermore, since not only main persons but also representative frames are thumbnails, individual moving image data can be easily identified even when the same person is photographed. Then, by combining the two representative frames as a thumbnail, it becomes possible to create a thumbnail that is easier to understand in a limited display range.

  Note that the control of the CPU 105 may be performed by a single piece of hardware, or the entire apparatus may be controlled by a plurality of pieces of hardware sharing the processing.

  Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms without departing from the gist of the present invention are also included in the present invention. included. Furthermore, each embodiment mentioned above shows only one embodiment of this invention, and it is also possible to combine each embodiment suitably.

  Further, in the above-described embodiment, the case where the present invention is applied to a digital camera has been described as an example, but this is not limited to this example. That is, the present invention can be applied to any display control device that can be controlled to display a plurality of images simultaneously, such as a PC, a PDA, a mobile phone terminal, a portable video playback device, or a display provided in a printer device. is there.

  The present invention supplies software (programs) for realizing the functions of the above-described embodiments to a system or apparatus via a network or various recording media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. To be executed. In this case, the program and the recording medium storing the program constitute the present invention.

105 CPU
106 Recording medium 107 Recording medium control circuit 108 Monitor 109 Playback circuit 110 OSD circuit 111 Operation switch 112 RAM
113 ROM
115 External input / output unit

Claims (14)

First representative frame extracting means for extracting a first representative frame from moving image data based on a first condition;
Second representative frame extracting means for extracting a second representative frame from the moving image data based on a second condition different from the first condition;
Control means for storing the extracted first representative frame and second representative frame as thumbnails in association with the moving image data;
A face recognition means for recognizing a person's face from the video data;
The second representative frame extracting means extracts, as the second representative frame, a frame in which the face is recognized by the face recognition means from the moving image data.
The first representative frame extracting means extracts a frame whose face is not recognized by the face recognition means as the first representative frame from the moving image data,
The image processing apparatus, wherein the control unit stores an image obtained by combining the first representative frame and the second representative frame in association with the moving image data as the thumbnail.   2. The second representative frame extracting unit extracts a frame including a human face recognized by the face recognizing unit for the longest time from the moving image data as the second representative frame. An image processing apparatus according to 1.   The second representative frame extracting means selects a frame in which the face direction of the person is most facing among the frames including the face of the person recognized the longest by the face recognition means in the moving image data. The image processing apparatus according to claim 2, wherein the image is extracted as the second representative frame.   The second representative frame extracting means extracts, as the second representative frame, a frame including the face of the person selected by the user among the faces of the person recognized by the face recognition means in the moving image data. The image processing apparatus according to claim 1, wherein: Information on the result of recognition by the face recognition means is recorded in the recording means;
5. The second representative frame extraction unit according to claim 1, wherein the second representative frame extraction unit extracts the second representative frame with reference to information on the recognition result recorded in the recording unit. The image processing apparatus according to item.   6. The first representative frame extracting unit according to claim 1, wherein the first representative frame extracting unit extracts the first representative frame based on information on an angle of view when the moving image data is captured. Image processing apparatus.   The first representative frame extracting means extracts, as the first representative frame, a frame having the widest angle of view at the time of shooting among frames that are candidates for the first representative frame included in the moving image data. The image processing apparatus according to claim 6. Information on the angle of view at the time of shooting the moving image data is recorded in the recording means,
The image processing according to claim 6 or 7, wherein the first representative frame extraction unit extracts the first representative frame with reference to information on the angle of view recorded in the recording unit. apparatus.   One of the first representative frame extracting unit and the second representative frame extracting unit is configured to select the first representative frame extracting unit based on a condition different from that of the other representative frame extracting unit among the luminance level and the loudness. The image processing apparatus according to claim 1, wherein the representative frame or the second representative frame is extracted.   The control means performs control so as to store an image obtained by clipping a partial range of the second representative frame and superimposed on the first representative frame as the thumbnail in association with the moving image data. The image processing apparatus according to claim 1.   When there is no frame that matches the first condition and the first representative frame is not extracted, or when there is no frame that matches the second condition and the second representative frame is not extracted, 2. The control unit according to claim 1, wherein one of the first representative frame and the second representative frame extracted and the first frame of the moving image data are stored in association with the moving image data as thumbnails. The image processing apparatus according to any one of 1 to 10. A first representative frame extraction step of extracting a first representative frame from the moving image data based on a first condition;
A second representative frame extracting step of extracting a second representative frame from the moving image data based on a second condition different from the first condition;
A control step of storing the extracted first representative frame and second representative frame as thumbnails in association with the moving image data;
A face recognition step for recognizing a person's face from the video data,
It said second representative frame extraction step, among the moving image data, extracts a frame in which the face is recognized by the face recognition step as said second representative frame,
The first representative frame extraction step extracts, from the moving image data, a frame whose face is not recognized in the face recognition step as the first representative frame,
The control step stores the image obtained by combining the first representative frame and the second representative frame in association with the moving image data as the thumbnail.   The program for functioning a computer as each means of the image processing apparatus as described in any one of Claims 1 thru | or 11.   A computer-readable storage medium storing a program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 11.

Images