JP2011244343A - Electronic apparatus and dynamic image generating method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an electronic apparatus and a dynamic image generating method that can easily determine an object to which an effect is applied.SOLUTION: According to one embodiment, an electronic apparatus comprises object detection means, object display means, selection means, effect means, and dynamic image generation means. The object detection means analyzes a plurality of materials and detects objects included therein. The object display means determines a representative image for each object detected and displays a list of the representative images on a screen. The effect means applies an effect to a first object selected by the selection means. The video generation means generates a dynamic image using the materials including the first object to which the effect has been applied.

Description

  Embodiments described herein relate generally to an electronic device that generates a moving image and a moving image generation method applied to the device.

  In recent years, an image reproducing apparatus called a digital photo frame has been widely used. For example, the digital photo frame has a function of sequentially displaying a plurality of still images stored in a card-type storage medium at predetermined time intervals. For example, personal computers, digital cameras, and the like generally have a function of sequentially displaying a plurality of still images at predetermined time intervals, like a digital photo frame.

JP 2007-49387 A

  In addition, some of the above-described image playback devices have a function of displaying an effect on a still image. For example, an effect of sliding (switching in / out) the two still images is slid and switched.

  The effect is applied to, for example, a still image that is noticed by the user or an object in the still image. However, the task of determining and determining a still image to be noticed by the user and an object in the still image for each still image has been problematic for the user.

  An object of the present invention is to provide an electronic apparatus and a moving image generation method that can easily determine an object to which an effect is applied.

  According to the embodiment, the electronic device includes an object detection unit, an object display unit, a selection unit, an effect unit, and a moving image generation unit. The object detection means analyzes a plurality of materials and detects objects included in the plurality of materials. The object display means determines a representative image for each object from the detected objects, and displays the determined list of representative images on the screen. The effect means applies an effect to the first object selected by the selection means. The moving image generation means generates a moving image using the plurality of materials including the first object to which the effect has been applied.

1 is a block diagram illustrating a configuration of an electronic device according to an embodiment. 2 is an exemplary block diagram showing the system configuration of the electronic apparatus of the embodiment. FIG. 2 is an exemplary block diagram illustrating a functional configuration of a moving image reproduction application program executed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing an example of material information used by a moving image reproduction application program executed by the electronic apparatus of the embodiment. FIG. 4 is an exemplary view showing an example of object information used by a moving image reproduction application program executed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing an example of appearance frequency information used by a moving image reproduction application program executed by the electronic apparatus of the embodiment. FIG. 4 is an exemplary view showing an example of a face image selection screen displayed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing another example of a face image selection screen displayed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing an example of template information used by a moving image reproduction application program executed by the electronic apparatus of the embodiment. 6 is an exemplary view illustrating an example of effect information used by a moving image reproduction application program executed by the electronic apparatus of the embodiment. FIG. 4 is an exemplary view showing an example of synthetic moving image generation information used by a moving image reproduction application program executed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing another example of the composite moving image generation information used by the moving image reproduction application program executed by the electronic apparatus of the embodiment. FIG. FIG. 6 is an exemplary view showing an example of a moving image generated by the electronic apparatus of the embodiment. 4 is an exemplary view showing an example of a parameter setting screen displayed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary flowchart illustrating an example of a procedure of material analysis / moving image generation processing executed by the electronic apparatus of the embodiment. 6 is an exemplary flowchart illustrating an example of a priority object selection process performed by the electronic apparatus of the embodiment. 6 is an exemplary flowchart illustrating an example of the procedure of the highest priority object selection process which is executed by the electronic apparatus of the embodiment. 6 is an exemplary flowchart illustrating an example of a procedure of a synthetic moving image generation process which is executed by the electronic apparatus of the embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
FIG. 1 is a perspective view illustrating an appearance of an electronic apparatus according to an embodiment. This electronic apparatus is realized as a notebook type personal computer 10, for example. As shown in FIG. 1, the computer 10 includes a computer main body 11 and a display unit 12. The display unit 12 includes an LCD (liquid crystal display) 17. The display unit 12 is attached to the computer main body 11 so as to be rotatable between an open position where the upper surface of the computer main body 11 is exposed and a closed position covering the upper surface of the computer main body 11.

  The computer main body 11 has a thin box-shaped casing. On the upper surface of the computer main body 11, a keyboard 13, a power button 14 for powering on / off the computer 10, an input operation panel 15, a touch pad 16, Speakers 18A, 18B, etc. are arranged. Various operation buttons are provided on the input operation panel 15.

  Further, on the right side surface of the computer main body 11, for example, a USB (Universal Serial Bus) 2.0 standard USB cable or a USB connector 19 for connecting a USB device is provided.

FIG. 2 is a diagram showing a system configuration of the computer 10.
As shown in FIG. 2, the computer 10 includes a CPU (central processing unit) 101, a north bridge 102, a main memory 103, a south bridge 104, a GPU (Graphics Processing Unit) 105, a VRAM (video RAM: random access). memory) 105A, sound controller 106, BIOS-ROM (basic input / output system-read only memory) 107, LAN (local area network) controller 108, hard disk drive (HDD) 109, optical disk drive (ODD) 110, USB controller 111A , A card controller 111B, a wireless LAN controller 112, an embedded controller / keyboard controller (EC / KBC) 113, an EEPROM (electrically erasable programmable ROM) 114, and the like.

  The CPU 101 is a processor that controls the operation of each unit in the computer 10. The CPU 101 executes various application programs such as an operating system (OS) 201 and a moving image reproduction application program 202 that are loaded from the HDD 109 to the main memory 103. The moving image reproduction application program 202 is software for reproducing various digital content data (for example, photo movies) stored in the HDD 109 or the like, for example. The moving image reproduction application program 202 has a moving image generation function. This moving image generation function is a function for creating a composite moving image (moving image) using a material (digital content) such as a photograph stored in the HDD 109 or the like, for example. Further, the moving image generation function includes a function of analyzing a material used for the moving image. The moving image reproduction application program 202 reproduces a moving image created using the material and displays it on the screen (LCD 17).

  The CPU 101 also executes the BIOS stored in the BIOS-ROM 107. The BIOS is a program for hardware control.

  The north bridge 102 is a bridge device that connects the local bus of the CPU 101 and the south bridge 104. The north bridge 102 also includes a memory controller that controls access to the main memory 103. The north bridge 102 also has a function of executing communication with the GPU 105 via, for example, a PCI EXPRESS serial bus.

  The GPU 105 is a display controller that controls the LCD 17 used as a display monitor of the computer 10. A display signal generated by the GPU 105 is sent to the LCD 17.

  The south bridge 104 controls each device on a peripheral component interconnect (PCI) bus and each device on a low pin count (LPC) bus. Further, the south bridge 104 includes an IDE (Integrated Drive Electronics) controller for controlling the HDD 109 and the ODD 110. Further, the south bridge 104 has a function of executing communication with the sound controller 106.

  The sound controller 106 is a sound source device and outputs audio data to be reproduced to the speakers 18A and 18B. The LAN controller 108 is a wired communication device that executes IEEE 802.3 standard wired communication, for example, while the wireless LAN controller 112 is a wireless communication device that executes IEEE 802.11g standard wireless communication, for example. The USB controller 111A executes communication with an external device (connected via the USB connector 19) that supports, for example, the USB 2.0 standard. For example, the USB controller 111A is used for receiving an image data file stored in a digital camera, for example. The card controller 111B executes data writing and reading with respect to a memory card such as an SD card inserted into a card slot provided in the computer main body 11.

  The EC / KBC 113 is a one-chip microcomputer in which an embedded controller for power management and a keyboard controller for controlling the keyboard 13 and the touch pad 16 are integrated. The EC / KBC 113 has a function of turning on / off the computer 10 in accordance with the operation of the power button 14 by the user.

Next, with reference to FIG. 3, a functional configuration of the moving image reproduction application program 202 operating on the computer 10 having the above configuration will be described. Here, a configuration example for realizing a moving image generation function among the functions of the moving image reproduction application program 202 will be described. This moving image generation function is not limited to the material data 51 stored from an external device (digital camera, memory card, etc.) via the interface unit (the above-mentioned USB controller 111A, card controller 111B, etc.), The present invention can also be applied to material data 51 stored in a directory, material data 51 stored via a network, and the like. The material data 51 is, for example, still image data 301A, audio data 301B, moving image data 301C, and the like.
The moving image reproduction application program 202 includes a material input unit 21, a material analysis unit 22, and a moving image reproduction unit 23.

  The material input unit 21 stores the material data 51 in the material database 301 in the HDD 109 in response to the input of the material data 51 via the interface unit such as the USB controller 111A and the card controller 111B. The material database 301 is a database for storing material data 51 used for generated moving images. Specifically, the material database 301 stores material data 51, such as still image data 301A, audio data 301B, and moving image data 301C. The material data 51 stored in the material database 301 is used as a material candidate for a composite moving image. In addition, the material input unit 21 notifies the material analysis unit 22 that the material data 51 has been stored in the HDD 109.

  In response to the notification from the material input unit 21, the material analysis unit 22 starts processing for analyzing the material data 51. The material analysis unit 22 includes an object detection unit 221, an appearance frequency calculation unit 222, a priority object selection unit 223, and a top priority object selection unit 224. In the following, it is assumed that the material data 51 to be analyzed is still image data 301A.

  The object detection unit 221 executes an object detection process for detecting an object from the still image data 301A. The object can be detected by, for example, analyzing the feature of the still image data 301A and searching for a region (object image) having a feature similar to a feature sample of an object prepared in advance. An object feature sample is feature data obtained by statistically processing image features of a large number of objects. By the object detection process, the position (coordinates), size, frontality, and the like of each object image included in the still image 301A are detected. The object detection unit 221 also assigns an object image ID to each detected object image. The object detection unit 221 generates material information 302A including information added to the still image 301A and information of the detected object image, and stores the material information 302A in the material information database 302 in the HDD 109.

  FIG. 4 shows a configuration example of the material information 302A. The material information 302A includes a material ID, a file name, a file path, a file size, a generation date / time, an attribute, a group ID, an object image ID, object image information, and the like. “Material ID” indicates identification information uniquely assigned to the material data 51. “File name” indicates the name of the data file corresponding to the material data 51. “File path” indicates a location where the material data 51 is stored. “File size” indicates the data size of the material data 51. “Generation date / time” indicates time information indicating the date / time when the material data 51 was generated. “Attribute” indicates attribute information of the material data 51. “Group ID” indicates identification information uniquely assigned to a group to which the material data 51 belongs. This group ID represents, for example, the ID of the group to which the material data 51 classified based on the generation date / time belongs. The group ID represents, for example, the ID of a group to which the material data 51 classified based on the relevance belongs (the material data 51 with high relevance is classified into the same group). “Object image ID” indicates identification information uniquely assigned to an object image included in the material data 51. “Object image information” includes information indicating the image, position (coordinates), size, frontality, and the like of each object image.

  The object image is, for example, a face image. In that case, the object detection unit 221 executes a face image detection process for detecting a face image from the still image data 301A. Then, the object detection unit 221 stores material information 302A including information of the detected face image in the material information database 302. Note that the object image is not limited to a face image, and any object such as a person image showing a whole body or a part of the body of the person including the face image or an animal image showing an animal drawn area is drawn. Including the specified region.

  For example, in the example shown in FIG. 4, the material information 302A corresponding to the material data 51 with the material ID “1” has a file name “AAA.jpg”, a file path “AAA / AAA”, and a file size “ “200kByte”, generation date and time “2010/02/01”, attribute “AA”, and group ID “A”. In the material information 302A, six object images with object image IDs “1, 2, 3, 4, 5, 6” are detected from material data 51 with material ID “1”. Indicates. Further, the material information 302A also indicates information (image, position, size, frontality, etc.) of each of the six object images.

  The object detection unit 221 classifies the detected object image. The object detection unit 221 classifies the plurality of object images detected from the still image data 301A into groups for each object image estimated as the same object. Further, the object detection unit 221 may identify an object corresponding to the detected object image. In this case, the object detection unit 221 determines whether the detected object image is the object using, for example, an image feature sample of the object to be identified. The object detection unit 221 assigns an ID (object ID) for each group (that is, for each object) to the classified object image. This object ID is identification information uniquely assigned to the object. Then, the object detection unit 221 generates object information 302B indicating the information of the classified object image and stores it in the material information database 302.

  FIG. 5 shows a configuration example of the object information 302B. The object information 302B includes, for example, an object ID, an object image ID, and an attribute. “Object ID” indicates identification information uniquely assigned to an object. “Object image ID” indicates the ID of the object image corresponding to the object. When the object detection unit 221 detects a plurality of object images corresponding to the object from a plurality of input material data 51 (still image 301A), the “object image ID” includes the plurality of object images. A plurality of corresponding object image IDs are shown. “Attribute” indicates attribute information corresponding to the object ID. “Attribute” indicates, for example, the name of the object corresponding to the object ID. For example, when the object image is a face image, the “attribute” indicates the name of a person corresponding to the face image. Accordingly, the object information 302B indicates object image information (object image ID) corresponding to the object for each object.

  For example, in the example shown in FIG. 5, the object information 302B corresponding to the object having the object ID “1” indicates that the object image recognized as the object is the object image ID “1, 9, 12, 18,”. It represents six object images indicated by 25, 30 ″. The object information 302B indicates that the attribute of the object whose object ID is “1” is “Mr. A”. That is, it is understood that the face image (object image) recognized as the person (object) “Mr. A” is the image indicated by the object image ID “1, 9, 12, 18, 25, 30”. . As shown in FIG. 4, the images indicated by the object image IDs “1, 9, 12, 18, 25, 30” are detected from the material data 51 whose material IDs are “1” to “6”, respectively. Has been.

  The object detection unit 221 outputs the generated material information 302A and object information 302B to the appearance frequency calculation unit 222. The object detection unit 221 may notify the appearance frequency calculation unit 222 that the material information 302A and the object information 302B corresponding to the input material data 51 are stored in the material information database 302.

  The appearance frequency calculation unit 222 calculates statistical information of the detected object. For example, the appearance frequency calculation unit 222 calculates the appearance frequency for each object based on the material information 302A and the object information 302B generated by the object detection unit 221. For example, the appearance frequency calculation unit 222 calculates the number of object images for each object in one group based on the group ID and object image ID of the material information 302A and the object ID and object image ID of the object information 302B. Calculated as appearance frequency. Then, the appearance frequency calculation unit 222 generates appearance frequency information 302 </ b> C including the calculated appearance frequency and stores it in the material information database 302. Also, the appearance frequency calculation unit 222 outputs the generated appearance frequency information 302C to the priority object selection unit 223. Note that the appearance frequency calculation unit 222 may notify the priority object selection unit 223 that the appearance frequency information 302C has been generated.

  FIG. 6 shows a configuration example of the appearance frequency information 302C. The appearance frequency information 302C includes, for example, an appearance frequency ID, a group ID, an object ID, an object image ID, and an appearance frequency. “Appearance frequency ID” indicates identification information uniquely assigned to one piece of appearance frequency information. “Group ID” indicates identification information of the group for which the appearance frequency is calculated. “Object ID” indicates identification information of the object for which the appearance frequency is calculated. The “object image ID” indicates identification information of an object image in which the object specified by the “object ID” is recognized among the object images detected from the material data 51 belonging to the group specified by the “group ID”. . “Appearance frequency” indicates the number of object images indicated by “object image ID”.

  For example, in the example illustrated in FIG. 6, the appearance frequency information 302 </ b> C corresponding to the appearance frequency whose “appearance frequency ID” is “1” has the ID of the group for which the appearance frequency is calculated as “A”, Indicates that the ID is “1”. The appearance frequency information 302C is an object image in which an object with an “object ID” of “1” is recognized among object images detected from material data 51 belonging to a group with a “group ID” of “A”. It indicates that the corresponding “object image ID” is “1, 9, 12, 30”. That is, the object images in which the object having the “object ID” “1” is recognized are the four object images having the “object image ID” “1, 9, 12, 30”. Therefore, the appearance frequency information 302C indicates that the “appearance frequency” is “4”.

  As shown in FIG. 4, the material ID of the material data 51 whose “group ID” is “A” is “1, 2, 3, 6”. From the material data 51 belonging to the group “A”, Object images having an “object image ID” of “1” to “17” and an object image of “30” are detected. Further, as shown in FIG. 5, the “object image ID” of the object image in which the object whose “object ID” is “1” is recognized is “1, 9, 12, 18, 25, 30”. Therefore, as shown in FIG. 6, among the object images detected from the material data 51 belonging to the group with the group ID “A”, the object with the object ID “1” (that is, “Mr. A”) is recognized. The “object image ID” corresponding to the image is “1, 9, 12, 30”.

  Note that the appearance frequency calculation unit 222 may calculate the appearance frequency of the object not only for each group but for each period based on the generation date and time, for each attribute, and the like. In addition, the appearance frequency calculation unit 222 may calculate the appearance frequency of objects for a plurality of material data 51 that satisfy a predetermined condition based on a group, a period, an attribute, and the like.

  The priority object selection unit 223 selects a priority object (second object) that is a target to which an effect is to be preferentially applied based on the statistical information of the object detected by the object detection unit 221. Specifically, the priority object selection unit 223 selects, as priority objects, predetermined objects (for example, second) in descending order of appearance frequency based on the appearance frequency information 302C. The priority object selection unit 223 outputs information on the selected priority object to the moving image reproduction unit 23.

  For example, in the example illustrated in FIG. 6, the priority object selection unit 223 has the object ID “1” having the appearance frequency “4” and the appearance frequency “3” for the group having the group ID “A”. The object with the object ID “2” is selected as the priority object. Therefore, in the group “A”, the priority effect (second effect) is applied to the object images whose object image IDs are “1, 9, 12, 30” and “2, 10, 13”. Further, for example, the priority object selection unit 223 performs the object ID “1” having the appearance frequency “2” and the object ID “2” having the appearance frequency “2” for the group having the group ID “B”. Select the 2 ”object as the priority object. Therefore, in the group “B”, the priority effect (second effect) is applied to the object images whose object image IDs are “18, 25” and “26, 31”.

  Further, the highest priority object selection unit 224 selects the object designated by the user as the highest priority object (first object) to which the effect is applied with the highest priority. Specifically, first, the highest priority object selection unit 224 displays a list of objects using the material information 302A and the object information 302B. The highest priority object selection unit 224 displays, for example, an object selection screen 50 shown in FIG. On the object selection screen 50, still images 50A to 50D are displayed, and a frame surrounding an object detected from the still images 50A to 50D is drawn. By selecting an area surrounded by the frame, the user can designate an object corresponding to the area as the highest priority object.

  Further, the highest priority object selection unit 224 displays, for example, an object selection screen 52 shown in FIG. On the object selection screen 52, a list of representative images for each object detected from the material data 51 is displayed. Specifically, the object selection screen 52 displays a list of face images 51A to 51E for each person detected from the still image 301A. In that case, the highest priority object selection unit 224 selects a representative image (face image) for each person using the material information 302A and the object information 302B, and displays a list of the selected representative images on the screen. The user can designate an object (person) corresponding to the representative image as the highest priority object by selecting the representative image from the list.

  The highest priority object selection unit 224 uses the object selection screens 50 and 52 and the like to select an object designated by the user as the highest priority object. The highest priority object selection unit 224 outputs information on the selected highest priority object to the moving image reproduction unit 23.

  For example, in the example shown in FIGS. 5 and 6, when an object having an object ID “2” (that is, “Mr. B”) is selected, the highest priority object selecting unit 224 has an object ID “2”. Select an object as the highest priority object. Therefore, in the group “A”, the highest priority effect (first effect) is applied to the object images having the object image ID “2, 10, 13”. In the group “B”, the highest priority effect (first effect) is applied to the object image having the object image ID “26, 31”.

  With the above-described configuration, the material analysis unit 22 analyzes the input material data 51 and selects a priority object and a top priority object. When the highest priority object selected by the highest priority object selection unit 224 is the same as the priority object selected by the priority object selection unit 223, the priority object selection unit 223 changes the selected priority object. Specifically, based on the appearance frequency information 302C, the priority object selection unit 223 newly selects an object having the next highest appearance frequency after the currently selected object as a priority object.

  The moving image reproduction unit 23 generates a moving image in which an effect is applied to the selected priority object and the highest priority object. The moving image playback unit 23 includes an extraction key selection unit 231, a material information extraction unit 232, a template selection unit 233, an effect selection unit 234, a composite video generation unit 235, and a composite video output unit 236.

  The extraction key selection unit 231 selects an extraction key for extracting the material data 51 used for the composite moving image. Specifically, the extraction key selection unit 231 displays, for example, a list of groups into which the material data 51 is classified on the screen. The extraction key selection unit 231 determines a group selected by the user from the displayed list as an extraction key. For example, when the user selects a group whose group ID is “A”, the extraction key selection unit 231 determines the group ID “A” as an extraction key. The extraction key selection unit 231 may select a group satisfying a predetermined condition (for example, a group including the material data 51 with the latest generation date and time) from the list of groups into which the material data 51 is classified. In addition, the extraction key is not limited to the group into which the material data 51 is classified, and for example, the generation date and time of the material data 51, the attribute, the storage location (directory where the material data 51 is stored), and the like can be specified. . The extraction key selection unit 231 outputs the determined extraction key to the material information extraction unit 232.

  The material information extraction unit 232 refers to the material information 302A stored in the material information database 302, and selects the material data 51 used for the synthesized moving image based on the determined extraction key. In the example shown in FIG. 4, for example, when the extraction key is the group ID “A”, the material information extraction unit 232 has the material data whose material ID is “1”, “2”, “3”, “6”. 51 is selected as the material data 51 used for the synthesized moving image. The material information extraction unit 232 notifies the template selection unit 233 that the material data 51 used for the synthesized moving image has been selected.

  The template selection unit 233 selects a template (template data) used for the synthesized moving image. The template includes the number of material data 51 used for the composite moving image, the timing for outputting the material data 51 (for example, the timing for displaying the still image 301A, the timing for outputting the sound 301B, etc.), Specify the color scheme. The template data is stored in the HDD 109, for example. For example, the template selection unit 233 analyzes the material data 51 selected by the material information extraction unit 232 (the material data 51 used for the synthesized moving image). Then, the template selection unit 233 refers to the template information 303A, and determines a template suitable for the material data 51 selected by the material information extraction unit 232 as a template used for the synthesized moving image based on the analysis result. Further, the template selection unit 233 may determine, for example, a template selected by the user as a template used for the synthesized moving image.

  FIG. 9 shows a configuration example of the template information 303A. The template information 303A includes, for example, a template ID, a file name, a file path, and an attribute. “Template ID” indicates identification information uniquely assigned to a template. “File name” indicates the name of a file in which template data is stored. “File path” indicates a location (address) where the file indicated by “file name” is stored. “Attribute” indicates attribute information of the template. This attribute information is, for example, keywords such as “gorgeous” and “fun” indicating the feature (impression) of the template. For example, in the example illustrated in FIG. 9, the template information 303 </ b> A whose “template ID” is “1” has a “file name” of “III”, a “file path” of “III / III”, and an “attribute” of “gorgeous”. ".

  For example, the template selection unit 233 determines a template with the template ID “1” appropriate for the material data 51 selected by the material information extraction unit 232 as a template used for the synthesized moving image. Further, for example, the template selection unit 233 determines the template with the template ID “1” selected by the user as a template used for the synthesized moving image. The template selection unit 233 notifies the effect selection unit 234 that the template has been determined.

  The effect selection unit 234 selects an effect (effect data) used for the synthesized moving image. The effect applies a decorative effect to the material data 51 used in the synthesized moving image and the objects included in the material data 51. This effect includes, for example, zoom, rotation, slide-in / slide-out, superimposed display of images such as frames, fade-in / fade-out, and the like. The effect can be designed such that the impression recognized by the user is changed by changing the color, shape, motion (motion), object, or the like used. The effect data is stored in the HDD 109, for example.

  For example, the effect selection unit 234 selects the highest priority effect (first effect) to be applied to the highest priority object selected by the highest priority object selection unit 224 with reference to the effect information 303B. Similarly, the effect selection unit 234 refers to the effect information 303B and selects a priority effect (second effect) to be applied to the priority object selected by the priority object selection unit 223. Further, the effect selection unit 234 refers to the effect information 303B and selects a normal effect to be applied to an object (normal object) other than the highest priority object and the priority object. More specifically, for example, the effect selection unit 234 has a higher priority object (priority object) than an effect that adds a mark for distinguishing it from other objects or an effect applied to other objects. Select an effect with a high visual effect. Note that the effect selection unit 234 may select the effect designated by the user as the highest priority effect, the priority effect, or the normal effect.

  FIG. 10 shows a configuration example of the effect information 303B. The effect information 303B includes, for example, an effect ID, a file name, a file path, an effect level, and an attribute. “Effect ID” indicates identification information uniquely assigned to an effect (effect data). “File name” indicates the name of the file in which the effect data is stored. “File path” indicates a location (address) where the file indicated by “file name” is stored. “Effect level” indicates the level (priority) of an object suitable for the effect. For example, an effect with an effect level of 10 (high effect level) is applied to the highest priority object. For example, an effect whose effect level is 5 (effect level is an intermediate value) is applied to the priority object. Further, for example, an effect whose effect level is 1 (low effect level) is applied to a normal object. “Attribute” indicates attribute information of the effect. This attribute information is, for example, a keyword such as “conspicuous” or “cute” indicating the feature (impression) of the effect.

  For example, in the example shown in FIG. 10, the effect information 303B whose “effect ID” is “1” has the “file name” “MMM”, the “file path” “MMM / MMM”, and the “effect level” “ 10 ”and“ attribute ”are“ conspicuous ”. Since the effect level is “10”, the effect whose “effect ID” is “1” is suitable for the highest priority object.

  For example, the effect selection unit 234 selects the effect with the effect ID “1” as the highest priority effect. For example, the effect selection unit 234 selects the effect with the effect ID “3” as the priority effect. The effect selection unit 234 selects, for example, the effect with the effect ID “2” as the normal effect. The effect selection unit 234 notifies the composite video generation unit 235 that the effect used for the composite video has been determined.

  Note that the material extraction unit 232, template selection unit 233, and effect selection unit 234 describe the material data 51, template, and effect information selected in the composite video generation information 303C stored in the composite video database 303, respectively. The composite moving image database 303 is stored in the HDD 109, for example.

  11 and 12 show a configuration example of the composite moving image generation information 303C. The composite video generation information 303C includes, for example, a management ID, group ID, file name, top priority object, priority object, composite video template, top priority effect, priority effect, and normal effect. “Management ID” indicates identification information uniquely assigned to the material data 51 used for the composite moving image. “Group ID” indicates identification information of a group to which the material data 51 belongs. “File name” indicates the name of the file corresponding to the material data 51. “Highest priority object” indicates the object image ID of the object image selected as the highest priority object among the object images included in the material data 51. “Priority object” indicates the object image ID of the object image selected as the priority object among the object images included in the material data 51. “Synthetic moving image template” indicates a template ID of a template used for the synthetic moving image. “Highest priority effect” indicates an effect ID of an effect applied to the highest priority object. “Priority effect” indicates the effect ID of the effect applied to the priority object. “Normal effect” indicates an effect ID of an effect applied to a normal object.

  For example, when the extraction key selection unit 231 selects the group with the group ID “A” as the extraction key, the material information extraction unit 232 refers to the material information 302A illustrated in FIG. 4 and belongs to the group “A”. Information of material data 51 (material ID “1, 2, 3, 6”) is extracted. 10, the material information extraction unit 232 adds “AAA.jpg”, “ABB.jpg”, “ACC” to the “file name” of the material data 51 used for the composite video. .jpg ”and“ AFF.jpg ”are described. In addition, the material information extraction unit 232 describes “A” in the “group ID” of the material data 51 used for the synthesized moving image.

  Next, for example, when the object with the object ID “2” is selected as the top-priority object by the top-priority object selection unit 224, the material information extraction unit 232 includes the object image included in the material data 51 used for the composite video. Among them, the object image ID corresponding to the object ID “2” (in FIG. 6, the object image ID “2, 10, 13” having the group ID “A” and the object ID “2”) is combined with the composite moving image generation information 303C. Described in “Highest priority object”. Therefore, in FIG. 11, “2” is assigned to the highest priority object corresponding to the material data 51 with the management ID “1”, and “10” is assigned to the highest priority object corresponding to the material data 51 with the management ID “2”. “13” is described in the highest priority object corresponding to the material data 51 of “3”. The material data 51 with the management ID “4” is not described because the corresponding highest priority object (object image) does not appear.

  Further, for example, when the priority object selection unit 223 selects the object with the object ID “1” having the highest appearance frequency as the priority object, the material information extraction unit 232 is included in the material data 51 used for the composite video. Object image ID corresponding to the object ID “1” (in FIG. 6, the object image ID “1, 9, 12, 30” having the group ID “A” and the object ID “1” in FIG. 6), It is described in “priority object” of the composite video generation information 303C. Therefore, in FIG. 11, “1” is assigned to the priority object corresponding to the material data 51 with the management ID “1”, “9” is assigned to the priority object corresponding to the material data 51 with the management ID “2”, and the management ID “3”. "12" is described in the priority object corresponding to the material data 51 of "", and "30" is described in the priority object corresponding to the material data 51 of the management ID "4".

  Then, for example, when the template selection unit 233 determines that the template with the template ID “1” is a template used for the synthesized moving image, “1” is described in “synthesized moving image template” of the synthesized moving image generation information 303C.

  For example, when the effect selection unit 234 determines the effect with the effect ID “1” as the highest priority effect used for the synthesized moving image, “1” is described in the “highest priority effect” of the synthesized moving image generation information 303C. For example, when the effect selection unit 234 determines the effect with the effect ID “3” as the priority effect used for the synthesized moving image, “3” is described in the “priority effect” of the synthesized movie generation information 303C. For example, when the effect selecting unit 234 determines that the effect with the effect ID “2” is the normal effect used for the synthesized moving image, “2” is described in the “normal effect” of the synthesized moving image generation information 303C.

  Similarly, when the group with the group ID “B” is selected as the extraction key by the extraction key selection unit 231, the composite moving image generation information 303 </ b> C illustrated in FIG. 11 is created.

  The composite video generation unit 235 generates a composite video using the above-described composite video generation information 303C. First, the composite video generation unit 235 reads material data 51, template data, and effect data corresponding to the file name and ID described in the composite video generation information 303C from the HDD 109, for example. Then, the synthetic moving image generating unit 235 generates a synthetic moving image in which the material data 51 is displayed (output) based on, for example, the timing and arrangement specified in the template. In addition, the synthesized moving image generating unit 235 applies the effect selected by the effect selecting unit 234 to the object included in the displayed material data 51. Specifically, the composite video generation unit 235 applies the highest priority effect (the object of the composite video generation information 303C) to the highest priority object (the object image corresponding to the “highest priority object” of the composite video generation information 303C) appearing in the material data 51. Apply the effect corresponding to “Priority effect”. The composite video generation unit 235 applies the priority effect (the effect corresponding to the “priority effect” of the composite video generation information 303C) to the priority object (the object image corresponding to the “priority object” of the composite video generation information 303C) that appears in the material data 51. ). Then, the synthesized moving image generating unit 235 applies a normal effect (an effect corresponding to “normal effect” of the synthesized moving image generation information 303 </ b> C) to a normal object appearing in the material data 51. The composite video generation unit 235 outputs the generated composite video to the composite video output unit 236.

  FIG. 13 shows an example (screens 62 to 64) of a synthesized moving image in which an effect is applied to an object included in the material data 51 (still image 301A). The screen 61 is a top-priority object selection screen for selecting the top-priority object by the top-priority object selection unit 224. In this screen 61, as in the screen 52 shown in FIG. 8, a list of representative images for each person selected from the face images included in the still image 301A is displayed. Here, it is assumed that the user has selected the person 61A as the top priority object because the user has selected the person (representative image) 61A as the top priority object (main role). Further, it is assumed that the priority object selection unit 223 has selected a person 63A with a high appearance frequency as a priority object (quasi-main character).

  On the screen 62, an effect (highest priority effect) 62A having a high visual effect is applied to the person 61A selected as the highest priority object. On the screen 63, the priority effect 63B is applied to the person 63A selected as the priority object. Further, as shown in the screen 64, the person 61A selected as the highest priority object is given the highest priority effect 64A having a higher visual effect than the priority effect 64B applied to the person 63A selected as the priority object. Has been.

  Note that the composite video generation unit 235 may further select the material data 51 from the material 51 (the material data 51 used for the composite video) extracted by the material information extraction unit 232 to generate a composite video. This is performed, for example, when the number of materials extracted by the material information extraction unit 232 is larger than the number of materials used for the selected template. For example, the composite video generation unit 235 regards the material data 51 including at least one of the highest priority object and the priority object as the material data 51 with high importance, and prioritizes the material data 51 extracted by the material information extraction unit 232. To choose. In addition, an index (parameter) indicating the degree to which the material data 51 with high importance is selected is set, and material data with high importance is added to the material data 51 selected by the synthesized moving image generating unit 235 based on this index. The ratio including 51 may be determined.

  FIG. 14 shows an example of a parameter setting screen 71 for setting an index indicating the degree of selection of material data 51 with high importance. The parameter setting screen 71 shows the top priority object and the persons 71A to 71D selected as the priority object. Further, the parameter setting screen 71 displays a slider 71E for designating the degree to which the material data 51 with high importance is selected. In particular, in the parameter setting screen 71, the degree to which the material data 51 with high importance is selected is referred to as “person fluctuation”, and when the ratio of the material data 51 with high importance is increased, the slider 71E Set a smaller value. That is, when the ratio of the material data 51 having a high importance level is increased, more material data 51 in which the persons 71A to 71D appear is selected, and thus the possibility that the persons 71A to 71D appear is relative. It can be said that the material data 51 (material group) having a high “small fluctuation” is selected. On the other hand, when the ratio of the material data 51 having high importance is reduced, the value of the slider 71E is set to be large. That is, when the ratio of the material data 51 having high importance is reduced, the material data 51 other than the material data 51 in which the persons 71A to 71D appear can be easily selected, so that a person other than the persons 71A to 71D appears. It can be said that the material data 51 (material group) with a relatively high possibility of being large and having a large “human fluctuation” is selected.

  Note that the effect selection unit 234 selects the selected effect (highest priority effect, priority effect, normal effect) for the object (highest priority object, priority object, normal object) included in the material data 51 used for the composite video. May be applied. In this case, the effect selection unit 234 outputs the material data 51 including the object to which the effect has been applied to the synthesized moving image generation unit 235. The composite moving image generation unit 235 generates a composite moving image (moving image) using the material data 51 including the object to which the effect has been applied.

  The composite video output unit 236 outputs the composite video generated by the composite video generation unit 222. The synthesized moving image output unit 236 reproduces the synthesized movie and displays it on the screen (LCD 17). Further, the composite video output unit 236 may encode the composite video and store the encoded composite video file in a predetermined storage device (for example, the HDD 109).

  With the above configuration, the moving image playback application program 202 can easily select an object to which an effect is applied. Specifically, the highest priority object selection unit 224 displays a list of objects that are candidates for the highest priority object using the material information 302A, the object information 302B, and the like. For example, the highest priority object selection unit 224 displays a list of representative images for each person as a list of objects. The user can easily select an object (for example, a person of interest) to which an effect is to be applied from the displayed list. Therefore, the work by the user can be saved more than the method of selecting the object of interest and the effect applied to the object for each material data 51.

  Further, the priority object selection unit 223 generates appearance frequency information 302 </ b> C of objects included in the material data 51 by analyzing the input material data 51. Then, the priority object selection unit 223 selects an object (for example, a person) with a high appearance frequency as a priority object. By selecting an object having a high appearance frequency as a priority object, an object that is estimated to be highly interested by the user can be an object to which an effect is applied. Therefore, an effect can be applied to an object that the user is not aware of (not aware of), and a new moving image with unexpected findings can be provided to the user.

Next, the material analysis / moving image generation process executed by the moving image playback application program 202 will be described with reference to the flowchart of FIG.
First, the material analysis unit 22 analyzes the material data 51 input via the material input unit 21, and selects a priority object based on the analysis result (block B20). Next, the highest priority object selection unit 224 determines the object selected by the user as the highest priority object (block B30). Then, the synthesized moving image generating unit 236 generates a synthesized moving image (moving image) including the highest priority object and the priority object to which the effect has been applied (block B40). Hereinafter, the detailed procedure of each block will be described with reference to the flowcharts of FIGS. 16 to 18.

The flowchart of FIG. 16 shows an example of the priority object selection processing procedure corresponding to block B20 of FIG.
First, the object detection unit 221 determines whether there is unprocessed material data 51 (block B201). When there is unprocessed material data 51 (YES in block B201), the object detection unit 221 detects an object from the material data 51 (block B202). Specifically, first, the object detection unit 221 detects an area corresponding to the object from the material data 51. Then, the object detection unit 221 gives identification information (object image ID) to the detected area. Next, the object detection unit 221 recognizes an object corresponding to the detected area. Identification information (object ID) is given to the recognized object. Therefore, the object detection unit 221 derives the correspondence between the object ID and the object image ID for each detected area.

  Next, the object detection unit 221 generates material information 302A including information added to the material data 51 including the file name and generation date and time, and information on the object image (block B203). Then, the object detection unit 221 generates object information 302B including the object ID and the corresponding object image ID based on the recognition result of the object (block B204). When the object information 302B corresponding to the object ID is already stored in the material information database 302, the object detection unit 221 updates the object information 302B by adding the object image ID. After the object information 302B is generated (updated), the object detection unit 221 determines the block B201 again.

  When there is no unprocessed material data 51 (NO in block B201), the appearance frequency calculation unit 222 generates (updates) appearance frequency information 302C based on the material information 302A and the object information 302B (block B205). The appearance frequency calculation unit 222 generates appearance frequency information 302C indicating the number of appearances for each object. Then, the priority object selection unit 223 selects a priority object (second object) based on the appearance frequency information 302C (block B206). For example, the priority object selection unit 223 determines up to a predetermined number of objects as priority objects in descending order of appearance frequency. The priority object selection unit 223 outputs the object ID corresponding to the determined priority object to the moving image generation unit 235.

  Through the above processing, the priority object selection unit 223 can determine a priority object (second object) that is a target to which the effect is to be preferentially applied.

Next, the flowchart of FIG. 17 shows an example of the procedure of the highest priority object selection process corresponding to the block B30 of FIG.
First, the highest priority object selection unit 224 displays a list of objects using the material information 302A and the object information 302B (block B301). For example, as shown in FIG. 7, the top-priority object selection unit 224 displays a detected object (for example, a face image) so as to be selectable for each still image 301A. For example, the highest priority object selection unit 224 displays each object surrounded by a frame. Further, the highest priority object selection unit 224 may display a list of face images for each person detected from the still image 301A, for example, as shown in FIG.

  Next, the highest priority object selection unit 224 determines whether or not selection of the highest priority object (first object) has been completed (block B302). For example, the top-priority object selection unit 224 displays a message for confirming whether or not the selected object can be confirmed as the top-priority object. Then, in response to the input of an instruction that the user may confirm to the displayed message (for example, pressing the OK button), the top-priority object selection unit 224 determines that the selection has been completed. To do.

  When selection of the highest priority object is not completed (NO in block B302), the highest priority object selection unit 224 determines whether an object is selected from the displayed list (block B303). When an object is not selected (NO in block B303), the highest priority object selection unit 224 performs determination of block B302 again.

  When an object is selected (YES in block B303), the highest priority object selection unit 224 determines the selected object as the highest priority object (block B304). At this time, when the highest priority object has already been determined (when the previously selected object has been determined as the highest priority object), the highest priority object selection unit 224 selects the newly selected object as the highest priority. Update to determine priority object. Then, the highest-priority object performs the determination of block B302 again.

  When selection of the highest priority object has been completed (YES in block B302), the highest priority object selection unit 224 determines the highest priority object. The highest priority object selection unit 224 outputs the object ID corresponding to the determined highest priority object to the moving image generation unit 235.

  Through the above processing, the highest priority object selection unit 224 can select the highest priority object (first object) that is the target to which the effect is applied with the highest priority. When the highest priority object selected by the highest priority object selection unit 224 is the same as the priority object selected by the priority object selection unit 223, the priority object selection unit 223 changes the selected priority object. Specifically, based on the appearance frequency information 302C, the priority object selection unit 223 newly selects an object having the next highest appearance frequency after the currently selected object as a priority object.

Further, the flowchart in FIG. 18 shows an example of the procedure of the synthetic moving image generation process corresponding to the block B40 in FIG.
First, the extraction key selection unit 231 selects an extraction key for extracting the material data 51 used for the composite moving image (block B401). As the extraction key, for example, the generation date and time, the attribute, the group, and the storage location (such as the directory in which the material data 51 is stored) of the material data 51 can be designated. Also, the designated still image 301A can be used as the extraction key. The extraction key selection unit 231 displays a list of extraction key candidates using, for example, the material information 302A. In response to the user selecting an extraction key candidate to be used as an extraction key from the displayed list, the extraction key selection unit 231 determines the selected extraction key candidate as the extraction key. The extraction key selection unit 231 may determine an extraction key candidate that satisfies a predetermined condition as an extraction key.

  Next, the material information extraction unit 232 refers to the material information 302A and the like stored in the material information database 302, and selects the material data 51 used for the composite video based on the determined extraction key (block B402). . That is, the material information extraction unit 232 selects the material data 51 corresponding to the extraction key. For example, when one group is determined as the extraction key, the material information extraction unit 232 selects the material data 51 belonging to the group as the material data 51 used for the synthesized moving image. For example, when one still image 301A is determined as the extraction key, the material information extraction unit 232 selects the material data 51 related to the still image 301A as the material data 51 used for the composite moving image.

  And the template selection part 233 selects the template used for a synthetic | combination moving image (block B403). For example, the template selection unit 233 refers to the template information 303A and selects a template suitable for the material data 51 selected by the material information extraction unit 232. Moreover, the template selection part 233 may select the template designated by the user as a template used for a synthetic moving image.

  Next, the effect selection unit 234 selects an effect used for the synthesized moving image (block B404). For example, the effect selection unit 234 selects the highest priority effect (first effect) to be applied to the highest priority object selected by the highest priority object selection unit 224 with reference to the effect information 303B. Similarly, the effect selection unit 234 refers to the effect information 303B and selects a priority effect (second effect) to be applied to the priority object selected by the priority object selection unit 223. Further, the effect selection unit 234 refers to the effect information 303B and selects a normal effect to be applied to an object (normal object) other than the highest priority object and the priority object. Note that the effect selection unit 234 may select the effect designated by the user as the highest priority effect, the priority effect, or the normal effect.

  Then, the synthetic moving image generating unit 235 generates a synthetic moving image using the selected material data 51, template, and effect (block B405). The synthetic moving image generating unit 235 generates a synthetic moving image on which the material data 51 is displayed (output) based on the timing and arrangement specified in the template. In addition, the synthesized moving image generating unit 235 applies the effect selected by the effect selecting unit 234 to the object included in the displayed material data 51. Specifically, the composite moving image generation unit 235 applies the highest priority effect to the highest priority object appearing in the material data 51. The synthesized moving image generation unit 235 applies a priority effect to the priority object appearing in the material data 51. Then, the synthetic moving image generation unit 235 applies a normal effect to a normal object appearing in the material data 51. The composite video generation unit 235 outputs the generated composite video to the composite video output unit 236.

  Next, the composite video output unit 236 determines whether the output method of the composite video is playback or file output (block B406). When the output method for reproducing the synthesized video is selected (reproduction of block B406), the synthesized video output unit 236 reproduces the synthesized video and outputs it to the screen (block B407). On the other hand, when the output method for outputting the synthetic moving image as a file is selected (file output in block B406), the synthetic moving image output unit 236 outputs the synthetic moving image to a file in a predetermined format (block B408).

  Through the above processing, the composite moving image generation unit 235 can generate a composite moving image including the highest priority object and the priority object to which the designated effect is applied. Note that the highest priority object selection unit 224 and the priority object selection unit 223 may select the highest priority object and the priority object, respectively, after the material data 51 used for the composite video is selected by the material information extraction unit 232. . That is, the highest priority object selection unit 224 and the priority object selection unit 223 may select the highest priority object and the priority object from the objects included in the material data 51 used for the composite moving image.

  As described above, according to the present embodiment, an object to which an effect is applied can be easily determined. The highest priority object selection unit 224 determines, for example, the object selected by the user from the list indicating the object candidates as the highest priority object. Further, the priority object selection unit 223 determines, for example, an object with a high appearance frequency as a priority object. Then, the synthesized moving image generating unit 235 applies the designated effect to the highest priority object and the priority object that appear in the material data 51 used for the synthesized moving image. As a result, it is possible to easily create a moving image in which an effect is applied to an object that is noticed by the user (high importance).

  Note that all the procedures of the moving image generation process of the present embodiment can be executed by software. For this reason, the same effect as that of the present embodiment can be easily realized simply by installing and executing this program on a normal computer through a computer-readable storage medium storing a program for executing the procedure of moving image generation processing. be able to.

  Further, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 109 ... HDD, 202 ... Moving image reproduction application program, 21 ... Material input part, 22 ... Material analysis part, 221 ... Object detection part, 222 ... Appearance frequency calculation part, 223 ... Priority object selection part, 224 ... Highest priority object selection , 23: Video playback unit, 231 ... Extraction key selection unit, 232 ... Material information extraction unit, 233 ... Template selection unit, 234 ... Effect selection unit, 235 ... Synthetic video generation unit, 236 ... Synthetic video output unit, 301 ... Material database 301A: Still image data 301B Sound data 301C Moving image data 302 Material information database 302A Material information 302B Object information 302C Appearance frequency information 303 Synthetic video generator 303A ... Template information, 303B ... Effect information, 30 C ... composite moving picture generation information, 51 ... material data.

Claims (8)

Object detecting means for analyzing a plurality of materials and detecting objects included in the plurality of materials;
An object display means for determining a representative image for each object from the detected objects, and displaying a list of the determined representative images on a screen;
Selection means for the user to select the first object from the displayed list of representative images;
Effect means for applying an effect to the first object selected by the selection means;
An electronic apparatus comprising: a moving image generation unit configured to generate a moving image using the plurality of materials including the first object subjected to the effect. Statistical information calculating means for analyzing the detected object and calculating statistical information of the detected object;
A second object selecting means for selecting a second object from the objects included in the plurality of materials based on the statistical information;
The effect means applies an effect to the second object,
The electronic apparatus according to claim 1, wherein the moving image generating unit generates a moving image using the plurality of materials including the first object and the second object to which the effect has been applied. The statistical information includes an appearance frequency for each object,
The electronic device according to claim 2, wherein the second object selecting unit selects a second object having a high appearance frequency based on the appearance frequency for each object. The effect means applies a first effect to the first object, applies a second effect to the second object,
The moving image generation means generates a moving image using the plurality of materials including a first object to which the first effect is applied and a second object to which the second effect is applied. The electronic device described.   The electronic device according to claim 4, further comprising effect selecting means for selecting the first effect and the second effect.   The electronic device according to claim 1, wherein the material includes at least one of a still image, a moving image, and sound. A face image detecting means for analyzing a plurality of still images and detecting a face image included in the plurality of still images;
Face image display means for determining a representative image for each person from the detected face image, and displaying a list of the determined representative images on a screen;
A selection means for the user to select a person from the displayed list of representative images;
An effect of applying a first effect to a face image corresponding to a person selected by the selection means, and applying a second effect to a face image corresponding to a person with a high appearance frequency among the face images included in the plurality of still images. Means,
An electronic apparatus comprising: a moving image generation unit configured to generate a moving image using the plurality of still images including a face image subjected to the first effect and a face image subjected to the second effect. Analyzing multiple materials, detecting objects contained in the multiple materials,
Determining a representative image for each object from the detected objects, displaying a list of the determined representative images on a screen;
Selecting a first object from the displayed list of representative images;
Apply an effect to the selected first object,
A moving image generation method for generating a moving image using the plurality of materials including the first object to which the effect is applied.

Images