JP2011044882A - Image processing apparatus, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and an image processing program that produce a still image which can recognize the whole contents of a moving image more exactly. <P>SOLUTION: An image processing apparatus includes: a frame selecting portion 11a that selects a plurality of frames from a moving image that consists of a plurality of frames; an object extracting portion 11b that recognizes and extracts a predetermined object from the selected frames; and an image producing portion 11c that produces a still image including a plurality of objects extracted by the object extracting portion 11b. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing program.

  In general, a plurality of still images recorded on a recording medium are displayed as so-called thumbnails so that the contents of the recording medium can be easily grasped. When the same thing is performed on a moving image, since a moving image usually includes a large number of frames, an appropriate frame must be selected as a thumbnail image. For example, Patent Document 1 describes a content recording / playback apparatus that creates a plurality of thumbnail data from one recorded content. This content recording / playback apparatus displays a plurality of created thumbnail data on the screen as thumbnail candidates, and selects one of them as a representative thumbnail.

JP 2008-294938 A

  Even using the technique described in Patent Document 1, since each thumbnail image corresponds to one frame, it is difficult to grasp the entire content of the moving image from the thumbnail image.

The invention according to claim 1 is a selection means for selecting a plurality of frames from a moving image composed of a plurality of frames, an extraction means for recognizing and extracting a predetermined object image from the selected plurality of frames, and an extraction means. And an image creating unit that creates a still image including a plurality of object images extracted by the above.
The invention according to claim 5 is a selection step of selecting a plurality of frames from a moving image composed of a plurality of frames, an extraction step of recognizing and extracting a predetermined object image from the selected plurality of frames, and an extraction means An image processing program for causing a computer to execute an image creation step of creating a still image including a plurality of object images extracted by the method.

  According to the present invention, it is possible to create a still image capable of more accurately grasping the entire content of a moving image.

1 is a block diagram illustrating a circuit configuration of an image processing apparatus according to a first embodiment. It is a figure which shows the preparation procedure of the thumbnail image by an object synthetic | combination process. It is a figure which shows the preparation procedure of the thumbnail image by the best shot composition process. It is a flowchart of the process which produces a representative image from a moving image file. It is a flowchart of an object recognition process. It is a flowchart of an object synthetic | combination process. It is a flowchart of a best shot composition process.

(First embodiment)
FIG. 1 is a block diagram illustrating a circuit configuration of the image processing apparatus according to the present embodiment. The image processing apparatus 1 includes a control circuit 11, a flash memory 12, a DRAM 13, a hard disk drive 14 (hereinafter referred to as HDD 14), an external device interface (I / F) 15, an input device 16, and a display device 17. The flash memory 12 is a nonvolatile storage device in which a predetermined control program executed by the control circuit 11 is stored. The DRAM 13 is used for temporarily storing data used by the control program. The HDD 14 is a nonvolatile storage device and stores a plurality of moving image files.

  The external device I / F 15 is an interface for connecting an external device such as a digital camera. A user of the image processing apparatus 1 can connect a digital camera to the external device I / F 15 and capture a moving image file from the digital camera to the HDD 14. The input device 16 is various input devices such as a keyboard and a mouse, and outputs an operation signal to the control circuit 11 in accordance with a user operation. The display device 17 is a variety of display devices such as a liquid crystal monitor.

  The control circuit 11 includes a frame selection unit 11a, an object extraction unit 11b, and an image creation unit 11c. These functional units are realized by the control circuit 11 executing a predetermined control program stored in the flash memory 12.

  Next, the moving image file list display function by the image processing apparatus 1 will be described. The control circuit 11 has a list display function of a plurality of moving image files stored in the HDD 14 and a moving image file reproduction function. The moving image file list display function is a function for displaying still images (hereinafter referred to as representative images) corresponding to the respective moving image files as thumbnails. When any one of the thumbnail-displayed representative images is selected by the user, the control circuit 11 starts reproduction of the moving image file corresponding to the selected representative image.

  The image processing apparatus according to the present embodiment can create a representative image by a process called an object composition process and a representative image by a process called a best shot composition process. When displaying a list of moving image files, the control circuit 11 executes one of these two processes for each moving image file stored in the HDD 14.

  FIG. 2 is a diagram showing a procedure for creating a thumbnail image by object composition processing. FIG. 2A shows a moving image file 21 that is a target of an object composition process described below. The moving image file 21 is composed of frames F11 to F19.

  In the object composition processing, first, the frame selection unit 11a selects a frame to be subjected to object composition processing from the frames F11 to F19. For example, the frame selection unit 11a may select all the frames included in the moving image file 21, or may select one frame every several frames. Further, the frame may be selected in consideration of detection of a scene change or the like. Here, it is assumed that the four frames F12, F14, F16, and F19 shown in FIG. 2B have been selected as objects for object composition processing.

  Next, the object extraction unit 11b recognizes an object from these four selected frames. An object is a predetermined object image included in each frame. A known technique such as pattern matching can be used for object recognition. In the example shown in FIG. 2B, the car object OA1 is displayed from the frame F12, the human object OB2 and the dog object OC1 are displayed from the frame F14, and the human object OB2 and the dog object OC2 are displayed from the frame F16. The dog object OC3 and the house object OD1 are recognized from the frame F19.

  After recognizing these objects, the object extraction unit 11b determines the identity of these objects. For example, when a moving image file is created by shooting 30 frames per second, the same object is repeatedly shot in a plurality of consecutive frames. The object extraction unit 11b determines whether or not the plurality of recognized objects are objects based on one object as in the above example. A known image recognition technique is used to determine the identity of the object. For example, for a human object, a known face recognition technique or the like can be used.

  For example, in FIG. 2B, the object extraction unit 11b determines that the person object OB1 recognized in the frame F14 and the person object OB2 recognized in the frame F16 are objects of the same person. From the four frames shown in FIG. 2 (b), it is similarly determined that the objects OC1, OC2, and OC3 are objects of the same object.

  Subsequently, the object extraction unit 11b extracts each object recognized from each frame. However, only one object of the same object is extracted based on the result of the identity determination. For example, the objects OC1, OC2, and OC3 are not extracted simultaneously from the plurality of frames shown in FIG. The object extraction unit 11b extracts only one of these objects.

  The reference for the object extraction unit 11b to select any one of the objects of the same object is the appearance order of the objects in the moving image file 21. For example, in FIG. 2B, the time-series order of the selected frame in the moving image file 21 is the order of frame F12, frame F14, frame F16, and frame F19. Therefore, among the objects OC1, OC2, and OC3 of the same object, the object having the earliest appearance order is the object OC1 recognized from the earliest frame F14 in the time series order in the moving image file 21. In this way, the object extraction unit 11b extracts the first appearing object from the same object.

  Finally, the image creation unit 11c creates a single still image by arranging a plurality of objects extracted by the object extraction unit 11b. An example of the created still image is shown in FIG. The still image P1 shown in FIG. 2C includes four objects OA1, OB1, OC1, and OD1 extracted by the object extraction unit 11b. These objects are arranged in order of appearance in the moving image file 21 from left to right. In addition, the background of the still image P1 is filled with a single color selected as the background color. This still image P1 is a representative image of the moving image file 21 created by the object composition process.

  FIG. 3 is a diagram showing a procedure for creating a thumbnail image by the best shot composition process. FIG. 3A shows a moving image file 22 that is a target of the best shot composition process described below. The moving image file 22 is composed of frames F21 to F29.

  In the best shot composition process, as in the object composition process, frame selection, object recognition, and identity determination are first executed. That is, first, a frame to be processed is selected by the frame selection unit 11a. Next, the object extraction unit 11b recognizes the object and determines whether each object is an object of the same object. The frame selection unit 11a detects the horizontal movement of the viewpoint from each frame, and selects a frame so that a panoramic image can be created.

  Thereafter, the object extraction unit 11b selects one of the objects corresponding to the recognized object with the highest appearance frequency as the main subject of the moving image file 22. Then, the object having the largest size among the objects corresponding to the main subject is selected as the main object. Hereinafter, a frame including the main object is referred to as a best shot of the main subject.

  The image creation unit 11c creates a panoramic image based on the frame selected by the frame selection unit 11a. Then, a still image is created using the panoramic image as a background and including no objects other than the main object. This still image is a representative image of the moving image file 22.

  The panorama image created by the image creation unit 11c is created so as not to include objects other than the main object as much as possible. For example, in the case of a moving image captured by tracking a moving object, it is conceivable that even if the background is a background of the object in a certain frame, the background is not a background. Therefore, by using such a frame, a panoramic image of the background can be created so as not to include objects other than the main object.

  FIG. 4 is a flowchart showing processing of the control circuit 11 that creates a representative image from a moving image file. A program for executing the processing of this flowchart is stored in the flash memory 12. First, in step S10, the frame selection unit 11a selects a frame to be processed later. In step S20, the object extraction unit 11b executes an object recognition process described later. In step S30, the control circuit 11 determines a representative image creation method. For example, when it is detected that the viewpoint of the moving image file is panned in the horizontal direction, it is determined that the representative image should be created by the best shot composition process, and in other cases, the representative image is synthesized with the object. It is determined that it should be created by processing.

  If the control circuit 11 determines in step S30 that the representative image should be created by the object composition process, the process proceeds to step S40. In step S40, the object extraction unit 11b and the image creation unit 11c execute an object composition process to be described later. On the other hand, when the control circuit 11 determines in step S30 that the representative image should be created by the best shot composition process, the process proceeds to step S50. In step S50, the object extraction unit 11b and the image creation unit 11c execute a best shot composition process to be described later.

  FIG. 5 is a flowchart of the object recognition process. This process is called in step S20 of the flowchart shown in FIG. In the object recognition process, an object is recognized from each frame selected in step S10 in FIG. In step S21, the object extraction unit 11b reads one unprocessed frame among the frames selected by the frame selection unit 11a. In step S22, the object extraction unit 11b calculates a feature amount necessary for object recognition from the frame read in step S21. In step S23, the object extraction unit 11b recognizes an object from the frame read in step S21. The feature amount calculated in step S22 is used for object recognition.

  Note that the feature amount calculation in step S22 and the object recognition in step S23 both use a known image recognition technique. For example, if object recognition is performed using pattern matching, first, in step S22, a feature amount representing a correlation with a processing target frame is calculated for each of a plurality of patterns prepared in advance. In step S23, a pattern that satisfies a predetermined criterion is selected based on the feature amount.

  In step S24, the object extraction unit 11b determines the identity between the object recognized so far and the object most recently recognized in step S23. In step S25, the object extraction unit 11b creates object information from the object recognized in step S23. Object information is information for extracting an object. The object information includes, for example, a unique identifier assigned to the object, information indicating whether the object is the same as the object recognized so far, a rectangle circumscribing the object, information for specifying the frame in which the object is recognized, etc. included. In step S26, the object extraction unit 11b records the object information created in step S24 in the DRAM 13. In step S27, the object extraction unit 11b determines whether or not the processing has been completed for all the frames selected by the frame selection unit 11a. If unprocessed frames remain, the process returns to step S21. On the other hand, when the process is completed for all frames, the object recognition process is terminated.

  FIG. 6 is a flowchart of the object composition process. This process is called in step S40 of the flowchart shown in FIG. In step S41, the object extraction unit 11b reads all the object information recorded in the DRAM 13 in step S25. In step S42, the object extraction unit 11b selects one unprocessed object in the order of appearance among all the objects included in the object information.

  In step S43, the object extraction unit 11b confirms information regarding the identity of the object included in the object information, and determines whether the same object as the object selected in step S42 has already been extracted. If it has been extracted, an affirmative determination is made in step S43, and the process proceeds to step S45. On the other hand, if a negative determination is made, the process proceeds to step S44. In step S44, the object extraction unit 11b extracts an object from the corresponding frame based on the object information. In step S45, the object extraction unit 11b determines whether the processing has been completed for all objects. If there is an unprocessed object, a negative determination is made, and the process returns to step S42. On the other hand, if a positive determination is made in step S45, the process proceeds to step S46. In step S46, the image creation unit 11c creates a still image including all the objects extracted in step S44.

  FIG. 7 is a flowchart of the best shot composition process. This process is called in step S50 of the flowchart shown in FIG. In step S51, the object extraction unit 11b reads all the object information recorded in the DRAM 13 in step S25. In step S52, the object extraction unit 11b totals the appearance frequency for each subject. In step S53, the object extraction unit 11b selects the subject having the highest appearance frequency, that is, the main subject that has been tabulated in step S52. In step S54, the object extraction unit 11b extracts the largest object among the objects corresponding to the main subject, that is, the main object from the corresponding frame based on the object information. In step S55, the image creation unit 11c creates a panoramic image that is the background image of the still image that is finally created. In step S56, the image creating unit 11c creates a still image that uses the panoramic image created in step S55 as a background and does not include objects other than the main object extracted in step S54.

According to the image processing apparatus according to the first embodiment described above, the following operational effects can be obtained.
(1) The image creation unit 11c creates a still image including a plurality of objects extracted by the object extraction unit 11b. Thereby, the entire content of the moving image file can be grasped more accurately from one still image.

(2) The object extraction unit 11b extracts only one of the plurality of objects when the plurality of objects recognized from each of the plurality of frames are the same object. As a result, even if there is an object that repeatedly appears, a still image that makes it easy to grasp the entire contents of the moving image file is created.

(3) The control circuit 11 creates a still image by switching between the object composition process and the best shot composition process. Thereby, an appropriate still image suitable for the characteristics of the moving image is created.

(4) The image creating unit 11c creates a panoramic image that does not include an object other than the main object when the panoramic image is used as the background of the still image. This creates a still image that makes it easy to determine what the main object is.

  The following modifications are also within the scope of the present invention, and one or a plurality of modifications can be combined with the above-described embodiment.

(Modification 1)
The creation of the representative image may be executed when the thumbnail display is performed, or may be executed when the moving image file is generated. For example, the above-described representative image creation process may be executed by a video camera. Further, the created representative image may be recorded on a recording medium as an image file associated with the moving image file.

(Modification 2)
In the description of the embodiment described above, the representative image is created by switching between the two processes of the object synthesis process and the best shot synthesis process. However, the representative image is created using only one of these processes. An image processing apparatus may be configured.

(Modification 3)
When the object extracting unit 11b recognizes more objects than a predetermined number, only some of the objects may be extracted. For example, an object having a predetermined size or less may not be extracted, or objects to be extracted may be limited based on the type and shape of the object. Further, for example, only a specific person may be extracted using a technique such as face recognition. Alternatively, a list of objects may be displayed and the user may select an object to be extracted.

(Modification 4)
As a criterion for selecting any one of the objects of the same object, a criterion other than the order of appearance of the objects may be used. For example, an object having the largest size may be selected, or an object having the highest contrast may be selected. Alternatively, an object may be selected using a feature amount calculated by predetermined image processing.

(Modification 5)
A technique for arranging a plurality of objects in a still image may be a technique other than arranging them in order of appearance from left to right. For example, it may be arranged randomly throughout the still image. Further, when the number of objects is large, some or all of the objects may be reduced and stored in a still image. Alternatively, the size of the still image may be increased.

(Modification 6)
Switching between the object composition process and the best shot composition process is not limited to the embodiment described above. For example, the user may be able to manually select which process to use, or the object composition process may be used when the number of objects appearing in the moving image file is greater than a predetermined number.

(Modification 7)
The still image created by the image creation unit 11c in the object composition process may have some background. For example, the background of any one of the frames selected by the frame selection unit 11a may be used, or a predetermined background image may be used.

(Modification 8)
The selection method of the main subject and the main object is not limited to the above-described embodiment. For example, a method may be used in which a frame in which the strongest edge is detected is set as the best shot, and an object included in the frame is set as a main object.

(Modification 9)
A plurality of still images may be created from one moving image. Still images obtained by connecting these still images vertically and horizontally may be used as representative images.

  The present invention is not limited to the above-described embodiment as long as the characteristics of the present invention are not impaired, and other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. .

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 11 Control circuit 11a Frame selection part 11b Object extraction part 11c Image creation part

Claims (8)

Selecting means for selecting a plurality of frames from a moving image composed of a plurality of frames;
Extracting means for recognizing and extracting a predetermined object image from the selected plurality of frames;
Image creating means for creating a still image including a plurality of object images extracted by the extracting means;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
When the plurality of object images recognized from each of the plurality of frames are the same object image, the extraction unit extracts only one of the plurality of object images. An image processing apparatus. The image processing apparatus according to claim 1 or 2,
The image processing device is characterized in that the image creating unit creates the still image with a background image created based on at least a part of the frame of the moving image as a background. The image processing apparatus according to claim 3.
The image processing apparatus according to claim 1, wherein the background image is a panoramic image created based on a plurality of frames included in the moving image. A selection step of selecting a plurality of frames from a moving image composed of a plurality of frames;
An extraction step of recognizing and extracting a predetermined object image from the selected plurality of frames;
An image creating step of creating a still image including a plurality of object images extracted by the extracting means;
An image processing program for causing a computer to execute. The image processing program according to claim 5,
In the extracting step, when a plurality of object images recognized from each of a plurality of frames are images of the same object, only one object image is extracted from the plurality of object images. Image processing program. In the image processing program according to claim 5 or 6,
The image processing program is characterized in that the image creating step creates the single still image with a background image created based on at least a part of the frame of the moving image as a background. The image processing program according to claim 7,
The image processing program according to claim 1, wherein the background image is a panoramic image created based on a plurality of frames included in the moving image.

Images