JP4261317B2 - Short film generating and reproducing apparatus and method thereof - Google Patents

Description

  The present invention relates to a short film generation / playback apparatus and method for generating and playing back a short video using at least one still picture.

  In the Japanese camera market in 2001, the number of digital still cameras shipped is expected to exceed the number of film cameras shipped, and in 2002 the gap is expected to widen further. In the recent mobile phone market, a large number of mobile phones with digital cameras have been released, and these models have become very popular. For this reason, the number of users of digital still cameras is increasing, and many of these users have a large amount of digital still pictures (hereinafter referred to as DSPs).

  These DSPs are generally stored in a disk drive of a personal computer. In order to view and manage this large amount of DSP, application software for image browsing and management called so-called digital camera software is commercially available. These applications have functions such as overall browsing with thumbnails and hierarchical management with directories, in addition to realizing conventional paper album images on a personal computer.

  In addition, in these applications, there is a function called a slide show as one function when viewing a DSP stored in a disk drive (see, for example, Patent Document 1). This slide show is a so-called “digital picture-story show” in which the DSP to be displayed is switched by a constant time interval or a trigger such as a user instruction as in the case of the slide projector. Depending on the application, BGM may be added, or transitions such as fade, wipe, and rotation may be added at the time of DSP switching to further improve entertainment.

  However, these applications are virtual realizations of a conventional paper album on a personal computer, and a slide show is basically the same as a picture-story show. No change. Therefore, it does not actively utilize the characteristics of digital data, nor does it provide the enjoyment unique to digital.

Therefore, in order to provide enjoyment unique to digital, an application for creating a short video (short film) like a movie from an input still image has been developed. This application automatically generates a short film by simply selecting a still image, BGM (music), and style set as materials. The style set defines a general image (color tone and effect) for the short film to be completed, and a visual effect (effect to promote still image, effect) according to the image. The short film produced by this application is groundbreaking in that still images are given motion-like motion, opening up a new world of entertainment using digital still images.
Japanese Unexamined Patent Publication No. 01-92875 (first page, FIG. 1)

However, the above application has the following problems.
First, when the user selects a still image, music, and style, sometimes unmatched BGM or DSP is selected for the style (effect), and the user feels uncomfortable when watching the short film. There is.
Secondly, the user has to select all the still images, music, and style, which is very troublesome. For example, aside from selecting a DSP that the user wants to use, it is troublesome to select music and style that match the image selected by the user. Conversely, when a user selects summer-like music, even if he / she wants to select a summer-like DSP and an active style, it is extremely difficult to select an appropriate image set from a huge DSP library. It is.

  Thirdly, the effect realized in the short film is to recognize the object such as “face” in the DSP and apply the visual effect to the DSP after considering its positional relationship. I don't mean. Therefore, it often occurs that a meaningless part is used for cutting and inserting, a person's face is cut, or a person zooms in on a meaningless part.

Fourthly, since it is assumed to be used on a personal computer, the user operates the keyboard, which is complicated and troublesome. In addition, along with the computerization of home appliances, it is desired to play back short films on home appliances such as televisions, but short films cannot be played back on home appliances.
Fifth, the visual effect is not applied after recognizing the type of object, for example, a facial part. Accordingly, the face part is cut off in the middle, or the order of sliding in / out by identifying the importance of the part is not determined, so that the viewer is uncomfortable.

  Sixth, the visual effect is not applied after recognizing the name of the object. As a result, all objects are treated equally, and a short film focused on an arbitrary person cannot be generated. Therefore, when a short film is generated and played using a still image taken at a party centering on a specific person such as a wedding reception or a birthday party, if it is a short film generated by a conventional application, Since the focus is out of focus and the main character is unknown, it may not meet the purpose depending on the playback location.

  Therefore, the present invention has been made in view of the above circumstances. If any one of elements such as a still image, music, and style is selected, another element that matches the element is determined and a short film is generated. Provided are a short film generation / playback apparatus and method capable of playing back and generating and playing back a short film with improved entertainment properties by reflecting an object in a still image in a visual effect. This is the first purpose.

Further, a second object of the present invention is to provide a short film generation / playback apparatus, a method thereof, and the like that can generate and play back a short film without requiring time and effort by a user.
Also provided is a short film generation / reproduction apparatus and method capable of generating and reproducing a short film with improved entertainment by determining a parameter for determining an effect operation based on object information of a still image. This is the third purpose.

  It is a fourth object of the present invention to provide a short film generation / playback apparatus and method capable of generating a short film focused on a specific object by recognizing the name of the object and reflecting it in the visual effect. And

  In order to achieve the first object, the short film generation / playback apparatus according to the present invention is a short film generation / playback apparatus that generates and plays back video using at least one still image. A database for storing a picture, an image feature amount representing the feature of the still image, a style representing an overall image of the generated video, a style feature amount representing the feature of the style, music, and a music feature amount representing the feature of the music Means for selecting any one of the still image, the music, and the style to be used when generating a video, and the still image, the music, and the style selected by the selection unit. The feature quantity reading means for reading out the feature quantity corresponding to any one from the database means and the feature quantity reading means A feature amount converting unit that converts the feature amount into another feature amount, and the still image, the music, and the music used when generating a video based on the other feature amount converted by the feature amount converting unit Of the styles, any one of the still image, the music, and the style that are not selected by the selection unit, and the still image, the music, and the style selected by the selection unit. And scenario generation means for generating a video scenario based on the still image, the music, and the style determined by the determination means.

  As a result, the user can create the scenario and the video simply by selecting one of the still image, music, and style used when creating the video. You don't have to choose all of your still images, music and styles.

Here, in the short film generation / playback apparatus according to the present invention, when the still image is selected by the selection unit, the feature amount conversion unit converts the image feature amount read by the feature amount reading unit into a music feature. It can also be characterized by conversion to a quantity and style feature quantity.
As a result, it is possible to determine the optimal music and style to be used when generating the video.

In the short film generation / playback apparatus according to the present invention, when music is selected by the selection unit, the feature amount conversion unit converts the music feature amount read by the feature amount reading unit into an image feature amount and It can also be characterized by conversion into style feature values.
As a result, it is possible to determine the optimal music and still image to be used when generating the video.

In the short film generation / playback apparatus according to the present invention, when a style is selected by the selection unit, the feature amount conversion unit converts the style feature amount read by the feature amount reading unit into an image feature amount and It can also be characterized by converting to music feature values.
Thereby, it is possible to determine the optimum still image and music to be used when generating the video.

  Specifically, in the short film generation / playback device according to the present invention, the database means further stores a theme of the video to be generated and a theme feature amount representing the characteristics of the theme, and the selection means stores the video. One of the still image, the music, the style, and the theme used for generation is selected, and the feature amount reading unit selects the still image, the music, the style, and the selected by the selection unit. One of the feature values of the theme may be read from the database unit.

  This allows you to expand your selection to 4 still images, music, styles, and themes, and you can simply select one of the still images, music, styles, and themes to use when creating your video. You can create scenarios and video.

In the short film generation / playback device according to the present invention, the feature amount conversion unit converts the theme feature amount read by the feature amount reading unit into an image feature amount, a music feature amount, and a style feature amount. Can also be characterized.
This makes it possible to determine the optimum still image, music, and style to be used when generating the video.

In the short film generation / reproduction device according to the present invention, the short film generation / reproduction device further includes a still image acquisition unit for acquiring a still image, and an image feature amount from the still image acquired by the still image acquisition unit. It is good also as providing the image feature-value extraction means which extracts.
As a result, it is possible to acquire the image feature amount that matches the still image while preventing the complexity of creating the image feature amount.

  Further, in the short film generation / playback apparatus according to the present invention, the short film generation / playback apparatus further includes object information extraction means for extracting object information from the still image acquired by the still image acquisition means. It can also be.

This makes it easy to select an optimal still image that constitutes a video using the extracted object information.
In the short film generation / playback apparatus according to the present invention, the style includes a predetermined number of effects and attributes that define what visual effect is used to play back one or more target still images. The scenario generation means may include a parameter, and the object information of a still image that is a target of the effect is associated with the effect.

This makes it possible to select a still image focusing on the object by generating a parameter for determining the action of the effect based on the object information of the still image, and to generate and play a short film with improved entertainment properties. it can.
Specifically, in the short film generation / playback device according to the present invention, the scenario generation means selects one effect from a group of effects included in the style one by one, and the selected effect is on the time axis. Effect placement means for sequentially placing the image on the time axis, and still image assignment means for assigning a still image that satisfies the feature amount of the image requested by the effect, based on the object information, for each effect placed on the time axis by the effect placement means And a parameter setting means for generating a scenario by describing a parameter indicating processing for an object that matches the effect placed on the time axis by the effect placement means and registering the created scenario in the database means Can be characterized.

As a result, by determining the parameters that determine the action of the effect based on the object information of the still image, a situation in which the object is divided is avoided, and a short film with improved entertainment characteristics focusing on the object is generated. And can be played.
In the short film generation / playback apparatus according to the present invention, the image feature amount required by the effect may be a feature amount of an object.

In the short film generation / playback device according to the present invention, the object may have at least one of a type, a color, a shape, and a number of objects.
In the short film generation / playback device according to the present invention, the short film generation / playback device further extracts a feature point indicating a characteristic portion included in the object, and registers the extracted feature point in the object information. A feature point extraction unit may be provided, and the parameter setting unit may generate a scenario by describing a parameter indicating processing for a position of a feature point included in the object.
This makes it possible to generate and play back a short film with improved entertainment that focuses on the feature points of the object.

  Further, in the short film generation / playback apparatus according to the present invention, the database means further stores personal authentication face information for specifying an individual's face, and the short film generation / playback apparatus further includes the object When the object extracted by the information extraction means is a human face, the face authentication means authenticates the name of the object using the face information for personal authentication, and registers the name of the authenticated object in the object information, The parameter setting means may generate a scenario by describing a parameter indicating processing for an object specified by a name.

  Thus, a short film focused on a specific object can be generated and reproduced by recognizing the name of the object and reflecting it in the visual effect.

In the short film generation / playback device according to the present invention, the short film generation / playback device further includes personal information storage means for storing personal information in which a person's name is associated with an attribute of the person, and face authentication means. The personal information corresponding to the name of the authenticated object is searched from the personal information, and the personal information search means for registering the searched personal attribute in the object information is included, and the parameter setting means includes the personal attribute It is also possible to generate a scenario by describing a parameter indicating processing for an object specified in (1).
Thereby, a short film having a visual effect for performing processing on an object specified by an individual attribute can be generated and reproduced.

In the short film generation / playback apparatus according to the present invention, the style includes a predetermined number of effects and attributes that define what visual effect is used to play back one or more target still images. It has a parameter, The scenario generation means may arrange the effect based on the attribute which the style has, and the attribute which the effect has.
This allows the visual effects to be determined in detail after the scenario skeleton has been determined by style.

In the short film generation / playback device according to the present invention, the effect is one of a basic effect including only one effect and an effect block including a combination of a plurality of basic effects, and the scenario generation The means may arrange the basic effects or the effect blocks based on the attributes of the style and the attributes of the effect.
This allows the visual effects to be determined more precisely after the scenario skeleton has been determined by style.

In the short film generation / playback device according to the present invention, the scenario generation means may assign a still image corresponding to the type of the effect to the effect.
As a result, an optimal still image can be assigned to the effect.
Further, in the short film generation / playback device according to the present invention, the short film generation / playback device further includes a short film selection means for selecting a video to be played back, and a video selected by the short film selection means. A short film playback unit that reads a scenario and the still image defined in the scenario and the music from the database unit and plays back the video based on the scenario may be provided.
As a result, a favorite short film can be selected from the created short films and reproduced.

  Moreover, in the short film production | generation reproduction | regeneration apparatus which concerns on this invention, the said short film production | generation reproduction | regeneration apparatus is further provided with the display means for displaying the said image reproduced | regenerated by the said short film reproduction | regeneration means, It is characterized by the above-mentioned. You can also.

  In order to achieve the second object, in the short film generating / reproducing apparatus according to the present invention, the short film generating / reproducing apparatus further displays the apparatus and the video connected to the apparatus. Operating means for operating the display device, wherein the short film reproducing means modulates and outputs a signal obtained by reproducing the video to an RF signal, and is provided in the operating means, to which the RF signal is assigned. When the predetermined channel button is pressed, the playback of the video is started.

In the short film generation / playback apparatus according to the present invention, the short film playback means switches the video being played to another video each time the channel button is pressed. You can also.
Moreover, in the short film production | generation reproduction | regeneration apparatus which concerns on this invention, the said short film reproduction | regeneration means can also reproduce | regenerate the said image | video several times simultaneously, and can output a plurality of said RF signals.

  Moreover, in the short film production | generation reproduction | regeneration apparatus which concerns on this invention, the said short film production | generation reproduction | regeneration apparatus is further the operation means for operating the display apparatus for displaying the said image | video connected to the said apparatus and the said apparatus. The short film playback means modulates and outputs a video signal as a video playback signal, and is provided in the operation means when a predetermined button to which the video signal is assigned is pressed, It is also possible to start playback of the video.

In the short film generation / playback apparatus according to the present invention, the short film playback means may switch the video being played to another video each time the button is pressed. it can.
Moreover, in the short film production | generation reproduction | regeneration apparatus which concerns on this invention, the said short film production | generation reproduction | regeneration apparatus can also be a home server.

  Also, a short film generation / playback device that generates and plays back video using at least one still image for operating the device and a display device for displaying video connected to the device. The operation means and a signal that reproduces the video are modulated and output as an RF signal, and a short circuit that is provided in the operation means and starts playing the video when a predetermined button assigned with the RF signal is pressed. A film reproducing means may be provided.

  The short film generation / reproduction device according to the present invention is a short film generation / reproduction device that generates and reproduces video using at least one still image, and represents the characteristics of the still image and the still image. Database means for storing image features, music, music features representing the music features, and effects defining which visual effect is to be used to reproduce one or more target still images, and generating video Selecting means for selecting any one of the still image and the music to be used when performing the processing, and a feature amount representing any feature of the still image and the music selected by the selecting means. A feature amount reading unit that reads from the feature amount, a feature amount conversion unit that converts the feature amount read by the feature amount reading unit into another feature amount, and the feature Determination to determine the still image and the music that are not selected by the selection unit among the still image and the music that are used when generating the video based on the other feature amount converted by the conversion unit, respectively. And a predetermined number of the effects and parameters used when generating a video based on one of the still image and the music selected by the selection unit and the still image and the music determined by the determination unit Style generating means for generating a style representing an overall image of the video to be generated, and either the still image or the music selected by the selecting means, the still image determined by the determining means, and A video scenario is generated based on the music and the style generated by the style generation means. It can be characterized by and a scenario generation unit that.

  As a result, the user can create a scenario and video by simply selecting one of the still image and music used when creating the video. You don't have to choose all of your music and style. In addition, it is possible to determine the optimum style used when generating the video.

  Further, in the short film generation and reproduction apparatus according to the present invention, a short film generation and reproduction apparatus that generates and reproduces video using at least one still image, and a still image acquisition unit that acquires a still image; From the database means for storing the still picture and the still picture acquired by the still picture acquisition means, an object included in the still picture is extracted, and object information including the position of the extracted object is registered in the database means. An object information extracting unit may be provided.

  In order to achieve the third object, the short film generation / playback apparatus according to the present invention is a short film generation / playback apparatus that generates and plays back video using at least one still image, Input means for inputting a still image input from the outside, and a predetermined number of effects and parameters that define what visual effect is used to reproduce the still image and one or more target still images. Database means for storing the style used and music used for the video to be generated, and scenario generation means for generating a video scenario based on a predetermined style and characteristics of the music, the input means from the outside Every time a still image is input, an object included in the still image is extracted, and object information including the position of the extracted object is extracted. Object information extracting means to be registered in the database means, wherein the scenario generating means selects one effect from the group of effects stored in the database means one by one, and selects the selected effect on the time axis For each effect placed on the time axis by the effect placement means, and based on the object information registered in the database means, a still image that satisfies the feature amount of the image requested by the effect A parameter for registering the generated scenario in the database unit by describing a parameter indicating processing for an object that matches the effect placed on the time axis by the effect placement unit And setting means It is also possible to.

As a result, by determining the parameters that determine the action of the effect based on the object information of the still image, a situation in which the object is divided is avoided, and a short film with improved entertainment characteristics focusing on the object is generated. And can be played.
In the short film generation / playback apparatus according to the present invention, the image feature amount required by the effect may be a feature amount of an object.

This makes it easy to determine parameters for determining the effect operation based on the object information of the still image.
In the short film generation / playback device according to the present invention, the object may have at least one of a type, a color, a shape, and a number of objects.

Accordingly, it becomes easy to determine a parameter for determining the effect operation based on at least one of the object information of the still image, that is, the type, color, shape, and number of the object.
In the short film generation / playback device according to the present invention, the input means further extracts a feature point indicating a characteristic portion included in the object, and extracts the feature point that registers the extracted feature point in the object information. Means for generating a scenario by describing a parameter indicating processing for a position of a feature point included in the object.
This makes it possible to generate and play back a short film with improved entertainment that focuses on the feature points of the object.

  In order to achieve the fourth object, in the short film generation / playback apparatus according to the present invention, the database means further stores personal authentication face information for specifying a personal face, The input means further authenticates the name of the object using the face information for personal authentication when the object extracted by the object information extraction means is a human face, and registers the name of the authenticated object in the object information The parameter setting unit may generate a scenario by describing a parameter indicating processing for an object specified by a name.

  Thus, a short film focused on a specific object can be generated and reproduced by recognizing the name of the object and reflecting it in the visual effect.

Further, in the short film generation / playback device according to the present invention, the short film generation / playback device further includes personal information storage means for storing personal information in which an individual name is associated with an attribute of the individual, and the input The means further includes a personal information search means for searching a personal attribute corresponding to the name of the object authenticated by the face authentication means from the personal information, and registering the searched personal attribute in the object information. The means may generate the scenario by describing a parameter indicating processing for an object specified by an individual attribute.
Thereby, a short film having a visual effect for performing processing on an object specified by an individual attribute can be generated and reproduced.

Further, in the short film generation / playback system according to the present invention, the short film generation / playback apparatus that generates and plays back video using at least one still image, and the short film generation / playback apparatus via a network are connected. A short film generation / reproduction system comprising: a short film generation / reproduction apparatus, wherein the short film generation / reproduction apparatus includes a still image, an image feature amount of the still image, a style representing an overall image of the generated image, Database means for storing the style feature quantity of the style, music and the music feature quantity of the music, and selection means for selecting any one of the still image, the music and the style used when generating the video , The still image selected by the selection means, the music, and the style A feature amount reading unit that reads out a feature amount corresponding to any one of the database unit, a feature amount conversion unit that converts the feature amount read out by the feature amount reading unit into another feature amount, and Of the still image, the music, and the style used when generating the video based on the other feature amount converted by the feature amount conversion unit, the still image, the music not selected by the selection unit Determining means for determining each of the style, the still image selected by the selecting means, the music and the style, and the still image, music and style determined by the determining means. A scenario generating means for generating a video scenario based on the video scenario selected by the short film playback device. And communication means for reading out the still image and the music defined in the scenario from the database means and transmitting them to the short film playback device via the network, the short film playback device comprising: Short film selection means for performing selection, communication means for communicating with the short film generation / playback apparatus via the network, a scenario of a video received from the short film generation / playback apparatus, and the scenario A storage means for temporarily holding the still image and the music; a scenario of the video selected by the short film selection means; and the still image and the music defined in the scenario are read from the storage means; A short fill that plays the video based on the scenario And a display means for displaying the video reproduced by the short film reproducing means.
Thus, the user can easily carry only the short film playback device, so that the video can be played on the go.

  The present invention can be realized not only as such a short film generation / reproduction device, but also as a short film generation / reproduction method including steps characteristic of the short film generation / reproduction device. These steps can be realized as a program for causing a computer to execute these steps. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.

  As is clear from the above description, according to the short film generation / playback apparatus according to the present invention, one of the three materials (still image, music, style) or theme required for generating the short film is selected. By selecting, the remaining material that matches the feature amount is determined and a short film is generated, so the user can save time spent searching in a huge database, making it very easy and easy There is an effect that a short film can be generated.

In addition, since the object information is extracted from the still image and the short film reflecting the object information is created, it is possible to provide a visual effect with higher entertainment properties.
Therefore, by using the features of short film materials such as still images, music, and styles, it is possible to generate short films more easily, easily, and to extract object information in still images. Therefore, the practical value is extremely high in that the effect can be utilized more effectively.

Furthermore, since the user can play the short film only by pressing a predetermined button, the user can save time and easily view the short film.
In addition, since the parameters that determine the action of the effect can be determined based on the object information of the still image, a situation in which the object is divided is avoided, and a short film that improves the entertainment property focusing on the object is provided. Can be generated and played.

Further, according to the short film generation / playback device according to the present invention, the scenario is generated by describing the parameter indicating the processing for the position of the feature point included in the object, so that the entertainment property focusing on the feature point of the object is improved. The produced short film can be generated and reproduced.
Further, according to the short film generation / playback device according to the present invention, the scenario is generated by describing the parameter indicating the processing for the object specified by the name, so that the object name is recognized and reflected in the visual effect. By doing so, a short film focused on a specific object can be generated and reproduced.

Furthermore, according to the short film generation / playback device according to the present invention, the scenario is generated by describing the parameter indicating the processing for the object specified by the personal attribute, so the processing for the object specified by the personal attribute is performed. A short film having the visual effect to be produced can be generated and reproduced.
Therefore, according to the present invention, it is possible to perform various types of image processing focusing on still image objects, and it is possible to generate and play a short film with high entertainment properties. Value is extremely high.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings.
(Embodiment 1)
FIG. 1 is a functional block diagram showing the configuration of the short film generating / reproducing apparatus according to Embodiment 1 of the present invention.
The short film generation / playback device 1001a is a device that generates and plays back a short video (short film) using one or more input still images. As shown in FIG. A database unit 1102, a playback unit 1103, an internal bus 1104, and a short film generation unit 1105a are provided. In addition, a display device 1002 is connected to the playback unit 1103. The display device 1002 is a device for displaying an image, such as an LCD or a cathode ray tube, and displays a reproduced short film.

  The still image includes image data and metadata indicating the attribute, and most of the metadata is data generated by a device such as a digital still camera that generates the still image. Typical data of this metadata includes shooting date and time, shooting location (position information by GPS), shooting conditions (shutter speed, focal length, exposure time, etc.) (hereinafter collectively referred to as “semantic information”). These specifications are formulated by JEIDA (Japan Electronics Industry Promotion Association) as EXIF.

The input unit 1101 is a unit that inputs a still image from a medium on which a still image is recorded, and includes a still image input unit 1106, an image feature amount extraction unit 1107, and an in-image object extraction unit 1108.
The database unit 1102 includes at least one data recording device, stores still images, music, and one or more preset styles, and has a function of searching for data in the database 1109.

  The still image input unit 1106 reads a still image recorded on the medium into a memory (not shown) in the short film generation / playback device 1001a and records it in the database 1109. Examples of media include semiconductor memory media such as compact flash (R) memory card and SD memory card (R), optical disk media such as CD-R / ROM / RW and DVD-R / RAM / RW, tape, flexible disk, etc. Magnetic media, magneto-optical disks such as MO.

The image feature amount extraction unit 1107 extracts an image feature amount from the read image data. Here, the image feature amount is an index indicating hue, color spread, color position, brightness, brightness position, and the like, and a color histogram, a frequency histogram, and the like are calculated to extract them.
The in-image object extracting unit 1108 extracts an object such as a person from the input image data, and generates object information. Here, the object information includes position information represented by the coordinates of the center of the circle with which the extracted object is in contact and the size of the radius, and the type of the object. The type of object includes whether it is a human face or not. In this case, for example, information for identifying an individual can be provided as the object type. Note that the position information of the object can be expressed by the coordinates of the start point and end point of one diagonal line of the quadrangle, for example, without depending on the method of expressing it by the center and radius of the circle.

The short film generation unit 1105a includes an image selection unit 1113, an image feature amount reading unit 1114, an image-style feature amount conversion unit 1115, an image-music feature amount conversion unit 1118, a style determination unit 1116, a music determination unit 1119, and a scenario generation. Part 1117 is provided.
The image selection unit 1113 provides a user with a GUI (graphical user interface) for selecting a still image. The image feature amount reading unit 1114 reads out the feature amount of the still image selected by the user using the image selection unit 1113 from the database 1109. The image-style feature amount conversion unit 1115 converts the image feature amount read by the image feature amount reading unit 1114 into a style feature amount. The style determination unit 1116 searches the style recorded in the database 1109 based on the style feature amount, and determines an optimal style. On the other hand, the image-music feature value conversion unit 1118 converts the image feature value read by the image feature value reading unit 1114 into a music feature value. The music determination unit 1119 searches the music recorded in the database 1109 based on the music feature amount, and determines the optimal music. Then, the scenario generation unit 1117 generates a scenario using the still image selected by the user, the style determined by the style determination unit 1116, and the music determined by the music determination unit 1119.

FIG. 2 is a conceptual diagram showing the relationship between styles and scenarios.
A style includes a “header” that describes items and attributes common to all the short films to be generated, and an arbitrary number of “collections of effects” collected under one concept. This effect defines what visual effect is used to reproduce one or more target still images, and describes a function that provides an actual visual effect and its attributes. At this time, in the “effect set” of the style, the permutation on the time axis between the effects and the pointer to the target still image are not fixed. The attributes include, for example, a speed attribute (eg, “slow”, “normal”, “fast”), an array attribute (eg, “prologue”, “body”, “epilogue”, “anywhere”). These attributes are referred to when determining the effect arrangement order in a scenario described later.

The header includes a name, a color tone, a common effect, a scenario configuration, a rhythm, a style feature amount, and the like. Of these, color tone and common effect are common setting items for all short films.
The scenario configuration is an attribute related to the effect arrangement. For example, when the scenario configuration is <“prologue”-“body”-“epilogue”>, based on the arrangement attribute of the effect, 1) a prologue, 2) Arrange the effects in the order of the main theme and 3) epilogue. The rhythm indicates a ratio of arranging effects having speed attributes “fast” and “slow” when arranging the effects. For example, if the rhythm is <2-1>, after arranging two effects having the attribute “fast”, the effect having the speed attribute “slow” is always arranged. The feature quantity of the style is expressed in five levels for each of the five keywords of “strength”, “dynamic feeling”, “exhilaration”, “simpleness”, and “softness”.

FIG. 3 is a diagram schematically showing an example of an effect with a storyboard.
In this effect, as shown in FIGS. 3A to 3E, a face whose spotlight is an object is searched for one image in which a person is captured. Next, as shown in FIG. 3 (f), the face is located and the spotlight is adjusted to the face. Further, as shown in FIG. 3G, only the spotlight portion is cut out and the other portions are erased. Finally, as shown in FIG. 3 (h), the spotlight portion is zoomed in and enlarged to the full screen.

As shown in FIG. 2, the scenario is such that the effects are picked up from the “collection of effects” and arranged in accordance with the time length of the music, and the still image used for each effect is determined.
The playback unit 1103 includes a short film selection unit 1110 and a short film playback unit 1112.

  The short film selection unit 1110 provides a user with a GUI for selecting a short film to be reproduced. The short film playback unit 1112 reads out the scenario of the selected short film and the still image and music defined there from the database 1109, decodes the scenario, plays back the short film, and outputs it to the display device 1002.

Next, an operation when a still image is input by the short film generation / playback apparatus 1001a configured as described above will be described.
FIG. 4 is a flowchart showing an algorithm when inputting a still image in the input unit 1101.
The still image input unit 1106 reads the still image recorded on the medium into the memory in the short film generation / playback device 1001a (S1101). Next, the image feature amount extraction unit 1107 extracts the image feature amount of the read image data (S1102). Further, the in-image object extracting unit 1108 extracts an object in the still image and generates object information (S1103).

  Finally, the image feature amount extracted by the image feature amount extraction unit 1107 and the object information generated by the in-image object extraction unit 1108 are added to the metadata of the still image and stored in the database 1109 together with the still image data. It is registered (S1104). The image data storage format may be a compression format represented by the JPEG format or an uncompressed format.

Next, an operation when a short film scenario is generated by the short film generation / playback apparatus 1001a will be described.
FIG. 5 is a flowchart showing an algorithm when generating a scenario in the short film generation unit 1105.
First, the image selection unit 1113 accepts determination of a selection range of still images (one or more) to be created as a short film created from the user (S1201). The still image used in the short film is a still image within this range. The image feature amount reading unit 1114 reads the image feature amount and semantic information of the selected still image (group) from the database 1109 (S1202).

Further, the image feature amount reading unit 1114 obtains the most occupied color in each still image based on the color histogram of the read image feature amount. The number of colors is counted and the largest number of colors is determined as the representative color of the still image (group) (S1203).
Next, the image-music feature value conversion unit 1118 converts the representative color determined by the image feature value reading unit 1114 into a music sensitivity image (S1204). Table 1 is a correspondence table showing an example of correspondence between color sensibility images and music sensibility images with respect to colors. Here, the music sensitivity image is preset based on the color sensitivity image for each color, and the music sensitivity image is determined if the representative color is determined. Music sensibility images, as well as style feature quantities, are divided into five levels for each of the five keywords of “strength”, “dynamic feeling”, “exhilaration”, “simpleness”, and “softness”. It is represented.

次に、画像−音楽特徴量変換部１１１８は、予め設定された所定の方法で音楽感性イメージを音楽特徴量に変換する（Ｓ１２０５）。表２は、音楽特徴量の一例を示す表である。本実施の形態１では、音楽特徴量として、「テンポ」、「ビート白色性」、「基本ビート」、「ビート強度１」、「ビート強度２」、「ビート強度比」、「平均音数」、「スペクトル変化度」の８つを用いている。

Next, the image-music feature amount conversion unit 1118 converts the music sensitivity image into the music feature amount by a predetermined method set in advance (S1205). Table 2 is a table showing an example of the music feature amount. In the first embodiment, the music feature values are “tempo”, “beat whiteness”, “basic beat”, “beat strength 1”, “beat strength 2”, “beat strength ratio”, “average number of sounds”. , “Spectrum change degree” are used.

次に、音楽決定部１１１９は、画像特徴量読出部１１１４により読み出された意味情報中の撮影日時から撮影季節を、撮影位置から大まかな撮影場所を取得する。ここで、大まかな撮影場所とは、例えば海の近く、山の中、街中、地方/市内、日本/国外、など情報である。そして、音楽決定部１１１９は、音楽特徴量をキーとしてデータベース１１０９を検索し、絞られた音楽の中から取得した撮影季節、撮影場所を考慮して、一つの音楽を決定する（Ｓ１２０６）。例えば、７月の海の近くで撮られた静止画であるならば、音楽特徴量で絞られた音楽の中からよりアップテンポな曲を選ぶ。

Next, the music determination unit 1119 acquires the shooting season from the shooting date and time in the semantic information read by the image feature reading unit 1114 and the approximate shooting location from the shooting position. Here, the rough shooting location is, for example, information such as near the sea, in the mountains, in the town, in the region / city, in Japan / outside Japan, and the like. Then, the music determination unit 1119 searches the database 1109 using the music feature amount as a key, and determines one music in consideration of the shooting season and shooting location acquired from the narrowed music (S1206). For example, if it is a still image taken near the sea in July, a song with a higher uptempo is selected from music narrowed down by the music feature amount.

  On the other hand, the image-style feature value conversion unit 1115 converts the representative color determined by the image feature value reading unit 1114 into a style feature value (S1207). Table 3 is a correspondence table showing an example of correspondence between the color sensitivity image and the style feature amount with respect to the color. Here, the style feature amount is preset based on the color sensitivity image for each color in the same manner as the music sensitivity image, and the style feature amount is determined when the representative color is determined.

次に、スタイル決定部１１１６は、スタイル特徴量と意味情報をキーとしてデータベース１１０９を検索し、スタイルを決定する（Ｓ１２０８）。
シナリオ生成部１１１７は、上記のように決定された静止画、音楽およびスタイルを用いてシナリオを生成する（Ｓ１２０９）。

Next, the style determination unit 1116 searches the database 1109 using the style feature quantity and semantic information as keys, and determines a style (S1208).
The scenario generation unit 1117 generates a scenario using the still image, music, and style determined as described above (S1209).

Hereinafter, scenario generation performed by the scenario generation unit 1117 will be described in detail.
FIG. 6 is a flowchart showing an algorithm for determining the effect arrangement.
First, the scenario generation unit 1117 reads the music to be BGM determined by the music determination unit 1119, and obtains the time length (S1301). Next, the scenario generation unit 1117 reads the style determined by the style determination unit 1116, and obtains the scenario configuration and rhythm which are the attributes (S1302).

When the scenario configuration and rhythm are acquired, the following processing is repeated until it exceeds 95% of the music time length (S1303). In the first embodiment, the effect arrangement is performed until it exceeds 95% of the music time length, but the numerical value of 95% can be changed.
The scenario generation unit 1117 determines the array attribute of the effect to be arranged next based on the scenario configuration and the arrangement history (S1304). For example, when the scenario configuration is <“prologue”-“body”-“epilogue”>, the effect placed within 0-30% from the beginning of the music is an effect having an array attribute of “prologue” or “anywhere” It is. In addition, the effect arranged 30 to 80% from the beginning of the music is an effect having an array attribute of “body” or “anywhere”. The effect arranged at 80 to 100% from the beginning of the music is an effect having an array attribute of “epilogue” or “anywhere”.

  Further, the scenario generation unit 1117 refers to the arrangement history to obtain the speed attribute ratio “slow”: “fast” of the effects already arranged. Based on this ratio and the ratio indicated by the rhythm, the speed attribute of the effect to be arranged next is determined (S1304). For example, when the rhythm is <3-1>, the weighting at the time of selection is determined so that the ratio of the speed attribute of the effect “slow”: “fast” is 3 to 1, and the speed attribute of “slow” is provided. Make sure the effect is not repeated more than 4 times.

Next, the scenario generation unit 1117 determines an effect to be arranged from the “effect set” of the style based on the logical product of the determined arrangement attribute and speed attribute (S1305). Then, the scenario generation unit 1117 adds the determined effect to the end of the scenario effect array, and updates the array history (S1306).
In the process of determining the effect arrangement attribute and the speed attribute and the process of determining the effect to be arranged, if the effect arrangement is performed until it exceeds 95% of the music time length (S1307), the scenario generation unit 1117 The speed attribute is read, and an effect to be subjected to temporal adjustment is determined (S1308). For example, when the motion attribute of the style is “slow”, all the arranged effects are targets for extending the execution time. When the movement attribute of the style is “normal”, the effect whose speed attribute is “slow” or “normal” is the target. When the motion attribute of the style is “fast”, the effect whose effect speed attribute is “slow” is targeted.

FIG. 7 is a conceptual diagram showing the effect arrangement with respect to the music time before and after the temporal adjustment, and is an example in the case where the speed attribute of the style is “fast”.
The scenario generation unit 1117 extends the execution time of the effect determined as the temporal adjustment target, and adjusts the total execution time of the effect array to be the same as the music (S1309). As a result, the effects are arranged in accordance with the music time length.

Next, the scenario generation unit 1117 searches the database 1109 for a still image having an image feature amount defined in each effect and determines it. The scenario generation unit 1117 determines the link between each effect and a still image as described above, and describes the relationship between the still image, music, and style as a scenario.
As described above, the scenario generation unit 1117 generates a scenario (FIG. 5, step S1209). Finally, the scenario generation unit 1117 stores the generated scenario in the database 1109 (S1210).

Next, an operation when reproducing the short film generated as described above will be described.
The short film selection unit 1110 of the playback unit 1103 acquires a list of short films registered in the database unit 1102 and displays it to the user. When the user selects a short film that the user wants to see, the short film selection unit 1110 instructs to read the scenario of the selected short film and the music and still images defined there. The short film playback unit 1112 decodes the scenario read from the database 1109, plays back the short film using the read music and still images, and outputs the short film to the display device 1002.

  As described above, based on the image feature amount extracted from the still image selected by the user, the music and the optimal style as the optimum BGM for the still image are determined and the short film is created. When creating a film, it is possible to avoid the hassle of selecting the optimal music from among a large amount of music and selecting the optimal style from a plurality of styles. In addition, it is possible to create a short film suitable for a still image simply by selecting the still image, and a creation environment comfortable for the user can be constructed.

  In addition, by arranging the effects as described above, it is possible to make the finished short film feel a story. For example, when creating a short film using a family trip still image, the prologue prepares an effect that allows an overview of the entire image using a plurality of still images. The main theme is an effect that makes each still image appear slowly, and in the epilogue, an effect that can make use of the group photo of the family. As a result, the sharpness of the short film can be given and the user can feel the story.

In addition, since the effects can be arranged in consideration of the balance between the slow effect and the uptempo effect, it is possible to create a short film that does not bore the user.
In the first embodiment, focusing on only the color histogram among the image feature amounts of the still image, the music feature amount and the style feature amount are converted based on the color sensitivity image of the representative color of the still image. ing. Not only the color histogram but also other feature quantities such as the spatial frequency and the number of objects may be focused and converted into music feature quantities and style feature quantities. Also, a method of converting them into music feature values and style feature values by combining them may be used. Further, a method may be used in which a characteristic still image is extracted and the music feature value and the style feature value are determined using the selected still image group as a representative image, regardless of the aggregation method.

  In the first embodiment, the effect arrangement is determined based on the scenario configuration and rhythm, which are style attributes. However, the present invention is not limited to this. For example, based on the tempo of the music feature amount, in the slow part, select an effect that can slowly show each still image with slow movement, and conversely, in the up-tempo part, multiple effects It is also possible to select a dynamic effect using a still image.

(Embodiment 2)
Next, a case where a user selects music to generate a short film that matches the music image will be described.
FIG. 8 is a block diagram showing another configuration of the short film generating / reproducing apparatus according to Embodiment 2 of the present invention. In addition, the same number is attached | subjected to the part corresponding to the short film production | generation reproduction | regeneration apparatus 1001a of Embodiment 1, and the description is abbreviate | omitted.

In the short film generation / reproduction device 1001b shown in FIG. 8, the configuration of the short film generation unit 1105b is different from the short film generation unit 1105a that generates a short film that matches the image of the image from the image. It is the same.
The short film generation unit 1105b is a device that generates a short film that matches the image of the music when the user selects music, and includes a music selection unit in addition to the style determination unit 1116 and the scenario generation unit 1117 described above. 1120, a music feature quantity reading unit 1121, a music-image feature quantity conversion unit 1122, a music-style feature quantity conversion unit 1124, and an image determination unit 1123.

  The music selection unit 1120 provides a user with a GUI for selecting music. The music feature quantity reading unit 1121 reads the music feature quantity selected by the user from the database 1109. The music-image feature value conversion unit 1122 converts the music feature value read by the music feature value reading unit 1121 into an image feature value. The image determination unit 1123 searches for a still image recorded in the database 1109 based on the image feature amount, and determines an optimal still image. On the other hand, the music-style feature value conversion unit 1124 converts the music feature value read by the music feature value reading unit 1121 into a style feature value. The style determination unit 1116 searches the style recorded in the database 1109 based on the style feature amount, and determines an optimal style. Then, the scenario generation unit 1117 generates a scenario using the music selected by the user, the still image determined by the image determination unit 1123, and the style determined by the style determination unit 1116.

Next, an operation when a short film scenario is generated by the short film generation unit 1105b configured as described above will be described.
FIG. 9 is a flowchart showing an algorithm when a scenario is generated in the short film generation unit 1105b.
First, the user uses the music selection unit 1120 to select a song (music 1) to be used as the BGM of the short film to be created (S1401). The music feature amount reading unit 1121 reads the music feature amount of the selected music 1 from the database 1109 (S1402). In the second embodiment, the music feature value is read from the database 1109. However, the music feature value reading unit 1121 has a function of extracting the music feature value, and the music feature value is directly selected from the music. May be extracted. Next, the music feature amount reading unit 1121 converts the read music feature amount into a music sensitivity image (S1403).

  The music-style feature quantity conversion unit 1124 uses the converted music sensitivity image as it is as the style feature quantity (S1404). This is because, in the second embodiment, both of the music sensitivity image and the style feature amount are 5 for each of the five keywords of “strength”, “dynamic feeling”, “exhilaration”, “simpleness”, and “softness”. This is because the level is expressed in stages. Then, the style determination unit 1116 searches the database 1109 using the style feature quantity (= music sensitivity image) as a key, and determines the style (S1405).

On the other hand, the music-image feature amount conversion unit 1122 converts the converted music sensitivity image into an image feature amount (S1406). The image determining unit 1123 searches for still images recorded in the database 1109 using the image feature amount as a key, and determines a predetermined number of still images (S1407).
The scenario generation unit 1117 generates a scenario in the same manner as in the first embodiment using the still image, music, and style determined as described above (S1408). Finally, the scenario generation unit 1117 stores the generated scenario in the database 1109 (S1409).

  As described above, since the optimum style and optimum still image for music are determined based on the music feature amount of the music selected by the user and the short film is created, the user can create the short film when creating the short film. It is possible to avoid the trouble of selecting an appropriate still image from a huge image database or selecting an optimal style from a plurality of styles. Moreover, it is possible to create a short film suitable for the music simply by selecting the music to be BGM, and a comfortable creation environment for the user can be constructed. Since a short film can be created simply by selecting music in this way, it is effective, for example, when generating a karaoke BGV.

  In the second embodiment, the image determination unit 1123 searches for a still image recorded in the database 1109 using the image feature amount as a key. However, the present invention is not limited to this. For example, music recorded in the database 1109 can be given semantic information, and still images recorded in the database 1109 can be searched based on the semantic information and image feature amounts. In this case, for example, if the semantic information of music includes a seasonal feeling of “summer”, it is possible to search by limiting the shooting date and time from July to September.

(Embodiment 3)
Next, a case where a user selects a style and generates a short film along the image of the style will be described.
FIG. 10 is a block diagram showing another configuration of the short film generating / reproducing apparatus according to Embodiment 3 of the present invention. In addition, the same number is attached | subjected to the part corresponding to the short film production | generation reproduction | regeneration apparatuses 1001a and 1001b of Embodiment 1, 2, and the description is abbreviate | omitted.

In the short film production / reproduction device 1001c shown in FIG. 10, the configuration of the short film production unit 1105c is different from the short film production units 1105a and 1105b shown in the first and second embodiments, and the others are the same.
The short film generation unit 1105c is a device that generates a short film according to an image of the style when the user selects a style. In addition to the image determination unit 1123, the music determination unit 1119, and the scenario generation unit 1117, A style selection unit 1125, a style feature amount reading unit 1126, a style-image feature amount conversion unit 1127, and a style-music feature amount conversion unit 1128 are provided.

  The style selection unit 1125 provides a user with a GUI for selecting a style. The style feature amount reading unit 1126 reads the feature amount of the style selected by the user from the database 1109. The style-image feature value conversion unit 1127 converts the style feature value read by the style feature value reading unit 1126 into an image feature value. The image determination unit 1123 searches for a still image recorded in the database 1109 based on the image feature amount, and determines an optimal still image. On the other hand, the style-music feature value conversion unit 1128 converts the style feature value read by the style feature value reading unit 1126 into a music feature value. The music determination unit 1119 searches the music recorded in the database 1109 based on the music feature amount, and determines the optimal music. Then, the scenario generation unit 1117 generates a scenario using the style selected by the user, the still image determined by the image determination unit 1123, and the music determined by the music determination unit 1119.

Next, an operation when a short film scenario is generated by the short film generating / reproducing apparatus 1001c configured as described above will be described.
FIG. 11 is a flowchart showing an algorithm at the time of scenario generation in the short film generation unit 1105c.
First, the user selects a style to be used in the short film to be created by the style selection unit 1125 (S1501). The style feature amount reading unit 1126 reads the selected style from the database 1109, and acquires the style feature amount described in the header portion of the style as shown in FIG. 2 (S1502). Next, the style-music feature value conversion unit 1128 converts the style feature value into a music feature value (S1503). In the third embodiment, a music sensitivity image matching the style is described as a style feature amount, and the music sensitivity image is converted into a music feature amount. Next, the music determination unit 1119 searches the database 1109 using the music feature amount as a key, and determines one music as the BGM of the short film (S1504).

On the other hand, the style-image feature value conversion unit 1127 converts the style feature value (= music sensitivity image) into an image feature value (S1505). Then, the image determination unit 1123 searches for still images recorded in the database 1109 using the image feature amount as a key, and determines a predetermined number of still images (S1506).
The scenario generation unit 1117 generates a scenario in the same manner as in the first and second embodiments using the still image, music, and style determined as described above (S1507). Finally, the scenario generation unit 1117 stores the generated scenario in the database 1109 (S1508).

  As described above, based on the style feature amount of the style selected by the user, the music and the optimum still image that are optimal for the style are determined and the short film is created. Therefore, when the user creates the short film, It is possible to avoid the trouble of selecting an appropriate still image and optimal music from a huge image and music database. In addition, it is possible to create a short film suitable for the style simply by selecting a style representing the overall image of the short film, and a comfortable creation environment for the user can be constructed.

  In the third embodiment, the music determination unit 1119 searches for music recorded in the database 1109 using the music feature amount as a key. However, the present invention is not limited to this. For example, the music recorded in the database 1109 can have semantic information, and the music determination unit 1119 can search the music recorded in the database 1109 based on the semantic information and the music feature amount. In this case, for example, the music determination unit 1119 acquires the current season from the current time information, and adds it to the key used when searching the database 1109, so that a song corresponding to the season can be obtained even with the same music feature amount. You can choose.

In the third embodiment, the style feature amount is represented by a music sensitivity image that matches the style, but the present invention is not limited to this. For example, the style feature value may be expressed using a keyword different from the music sensitivity image, and the style feature value may be converted into the music feature value based on a table comparing the style feature value and the music feature value.
Furthermore, in the third embodiment, the image determination unit 1123 searches for a still image recorded in the database 1109 using the image feature amount as a key. However, the present invention is not limited to this. For example, object information is defined as one of the style feature amounts, and the image determination unit 1123 can search for a still image recorded in the database 1109 based on the object information and the image feature amounts. In this case, it is preferable to select a still image including a lot of human (face) objects in the style of showing a still image centered on a snapshot.

(Embodiment 4)
Next, a case where a user selects a theme to generate a short film along the theme image will be described.
FIG. 12 is a block diagram showing another configuration of the short film generating / reproducing apparatus according to Embodiment 4 of the present invention. In addition, the same number is attached | subjected to the part corresponding to the short film production | generation reproduction | regeneration apparatuses 1001a-10001c of Embodiment 1-3, and the description is abbreviate | omitted.

In the short film production | generation reproduction | regeneration apparatus 1001d shown by FIG. 12, the structure of the short film production | generation part 1105d is different from the short film production | generation parts 1105a-1105c of the said Embodiment 1-3, and others are the same.
The short film generation unit 1105d is a device that generates a short film according to an image of the theme when the user selects a theme. The image determination unit 1123, the style determination unit 1116, the music determination unit 1119, and the scenario generation unit In addition to 1117, a theme selection unit 1129, a theme feature amount reading unit 1130, a theme-image feature amount conversion unit 1131, a theme-style feature amount conversion unit 1132, and a theme-music feature amount conversion unit 1133 are provided.

  Here, the theme is defined in advance by a developer or user, and an image feature amount, a music sensitivity image (music feature amount), and a style feature amount are determined according to the theme. Themes include, for example, seasons (spring / summer / autumn / winter, early summer, late autumn, etc.), a word with an ambiguous time designation such as recent events, or a specific year of the year such as 2000 or 1990. There are words that specify.

  The theme selection unit 1129 provides a user with a GUI for selecting a theme. The theme feature amount reading unit 1130 reads the theme definition file of the theme selected by the user from the database 1109. The theme-image feature amount conversion unit 1131 extracts the image feature amount described in the theme definition file. The image determination unit 1123 searches for a still image recorded in the database 1109 based on the image feature amount, and determines an optimal still image. The theme-style feature amount conversion unit 1132 extracts the style feature amount described in the theme definition file. On the other hand, the style determination unit 1116 searches the style recorded in the database 1109 based on the style feature amount, and determines the optimum style. The theme-music feature amount conversion unit 1133 extracts a music feature amount described in the theme definition file. On the other hand, the music determination unit 1119 searches the music recorded in the database 1109 based on the music feature amount and determines the optimum music. Then, the scenario generation unit 1117 generates a scenario using the still image determined by the image determination unit 1123, the style determined by the style determination unit 1116, and the music determined by the music determination unit 1119.

Next, an operation when a short film scenario is generated by the short film generating / reproducing apparatus 1001d configured as described above will be described.
FIG. 13 is a flowchart showing an algorithm at the time of scenario generation in the short film generation unit 1105d.
First, the user selects a theme to be used in the short film to be created by the theme selection unit 1129 (S1601). The theme feature amount reading unit 1130 reads the theme definition file of the selected theme from the database 1109 (S1602). Next, the theme-style feature quantity conversion unit 1132 extracts the style feature quantity described in the theme definition file (S1603). Then, the style determination unit 1116 searches the database using the style feature amount as a key, and determines the style (S1604).

Next, the theme-music feature amount conversion unit 1133 extracts the music sensitivity image described in the theme definition file, and converts this music sensitivity image into a music feature amount (S1605). Then, the music determination unit 1119 searches the database 1109 using the music feature amount as a key, and determines one music as the BGM of the short film (S1606).
Next, the theme-image feature amount conversion unit 1131 extracts the image feature amount described in the theme definition file (S1607). Then, the image determination unit 1123 searches for still images recorded in the database 1109 using the image feature amount as a key, and determines a predetermined number of still images (S1608).

The scenario generation unit 1117 generates a scenario in the same manner as in the first embodiment using the still image, music, and style determined as described above (S1609). Finally, the scenario generation unit 1117 stores the generated scenario in the database 1109 (S1610).
As described above, based on the theme selected by the user, the optimal style, optimal music, and optimal still image for the theme are determined and the short film is created, so when the user creates the short film, It is possible to avoid the trouble of selecting the optimum still image, music and style from among a large amount of images, music and style. Moreover, it is possible to create a short film suitable for the theme simply by selecting the theme, and a creation environment comfortable for the user can be constructed.

  In the fourth embodiment, the image determination unit 1123 searches for a still image recorded in the database 1109 using the image feature amount as a key. However, the present invention is not limited to this. For example, it is possible to provide a restriction condition in accordance with the theme in the theme definition file, and to search for a still image recorded in the database 1109 based on the restriction condition and the image feature amount. In this case, for example, if the theme is “summer”, the shooting date and time = June to September, the shooting location = mountain, the sea, the pool, the color tone = blue, white, etc. are the limiting conditions. You can select only a still image.

  In the first to fourth embodiments, the display device 1002 is connected to the short film generation / reproduction device 1001d. However, the present invention is not limited to this. For example, the display device 1002 is connected to the short film generation / reproduction device 1001d. It is also possible to provide a configuration.

(Embodiment 5)
Next, the association between the effect and the still image in the scenario generation unit 1117 will be described. Here, a case where a still image is assigned to the effect shown in FIG. 3 will be described. This effect shows the effect of searching for the face that the spotlight is an object for the single still image in which the person is captured as described above, finally finding the face, and finally zooming in. It is.

FIG. 14 is a flowchart showing an algorithm when the scenario generation unit 1117 assigns a still image to an effect and generates a visual effect based on the object information.
The scenario generation unit 1117 reads the attribute of the effect (S1701). The attribute of this effect is a preset still image search rule. For example, as shown in Table 4, “number of faces”, “position of face”, “face size”, “similar image” Number "etc. are set. In the effect shown in FIG. 3, since a still image in which a person is photographed alone can be shown effectively, for example, the search rule is set to “number of faces = 1”.

シナリオ生成部１１１７は、上記の検索ルールに基づいてデータベース１１０９を検索し、本エフェクトに割り当てるべき静止画を決定する（Ｓ１７０２）。次に、シナリオ生成部１１１７は、決定した静止画のオブジェクト情報をデータベース１１０９より読み出す（Ｓ１７０３）。シナリオ生成部１１１７は、エフェクトに必要なオブジェクト情報をエフェクトにセットする（Ｓ１７０４）。図３に示すエフェクトでは、オブジェクトである顔の位置とサイズが必要であるので、それらを静止画のオブジェクト情報から取得して、エフェクトにセットする。

The scenario generation unit 1117 searches the database 1109 based on the search rule described above, and determines a still image to be assigned to this effect (S1702). Next, the scenario generation unit 1117 reads the determined still image object information from the database 1109 (S1703). The scenario generation unit 1117 sets object information necessary for the effect in the effect (S1704). The effect shown in FIG. 3 requires the position and size of the face, which is an object, and is acquired from the object information of the still image and set in the effect.

  As described above, the effect shown in FIG. 3 selects a still image of a person alone from the database 1109 and sets face object information (coordinate position and radius of the center of gravity) in the effect. Knowing the position and size, the spotlight can look like searching for a face, finding the face, and finally zooming in.

  As described above, the still image having the best matching with the effect can be selected from the database, so that it is possible to provide a visual effect with higher entertainment. In addition, since an effect is given to a still image after recognizing objects in the still image including a face, it is possible to add a story and add a story.

(Embodiment 6)
In each of the first to fifth embodiments, the scenario generation unit 1117 first arranges the effects as described above, and then selects a still image having an image feature amount required by the effects from the database 1109 for the effects. By doing so, the link between the effect and the still image is established. On the other hand, in the sixth embodiment, after the still images are first arranged, the effect is selected based on the image feature amount and the object information of the still image, thereby determining the link between the effect and the still image. Is the case.

  FIG. 15 is a conceptual diagram showing assignment of effects to still images arranged on the time axis. Here, the still images A to F are arranged in the order registered in the database 1109 on the time axis. Further, when classified based on the image feature amount and the object information, the still image A is a snapshot with two subjects. Still images B to D are photographs with high similarity. The still image E is a landscape photograph in which no person is shown. The still image F is a snapshot that shows a single person. Table 5 is a table showing the effects corresponding to the classification based on the image feature amount.

An operation when the scenario generation unit 1117 generates a scenario using the determined still image, music, and style will be described.
FIG. 16 is a flowchart showing an algorithm for assigning effects to still images.
The scenario generation unit 1117 reads the determined still images (plurality) in the order registered in the database 1109 and arranges them on the time axis (S1801). Next, the scenario generation unit 1117 determines the similarity and type of still images based on the image feature amount and object information of each still image (S1802). Next, the scenario generation unit 1117 assigns effects corresponding to the similarity and type of each still image using a conversion table as shown in Table 5, for example, and determines the link between the effect and the still image (S1803). In the example shown in FIG. 15, “spotlight2” is selected for the still image A from the section of snapshots (two people). Similarly, visual effects such as “pan”, “still”, and “zoomin1” are selected for the still images B to D, E, and F. It should be noted that any effect may be selected at this time as long as it is within the same item, and may be selected according to a rule such as random or round robin.

最後に、シナリオ生成部１１１７は、配列の確定したエフェクトについて、例えばオブジェクトの位置、サイズ等のオブジェクト情報をセットする（Ｓ１８０４）。
以上のように、最初に静止画を配列した後、静止画の画像特徴量およびオブジェクト情報に基づいてエフェクトと静止画とのリンクを確定しているので、ユーザにより選択された静止画、または画像決定部１１２３により決定された静止画を全て素材として使用したシナリオを生成することが可能となる。また、静止画の画像特徴量およびオブジェクト情報に基づいて、静止画に適切なエフェクトを選択することができる。

Finally, the scenario generation unit 1117 sets object information such as the position and size of the object for the effect whose arrangement has been determined (S1804).
As described above, after the still images are first arranged, the link between the effect and the still image is determined based on the image feature amount and the object information of the still image. It is possible to generate a scenario using all the still images determined by the determination unit 1123 as materials. Further, it is possible to select an appropriate effect for a still image based on the image feature amount and object information of the still image.

In the sixth embodiment, when still images are arranged on the time axis, they are arranged in the order registered in the database 1109. However, the present invention is not limited to this. For example, the still images are arranged in the order of the date and time when the still images were taken. It doesn't matter.
In addition, without first arranging still images on the time axis, after establishing the link between the effect and the still image based on the image feature amount and object information of the still image, only the effect for which the link has been established is placed on the time axis. Scenarios may be generated by arranging them.

  In the first to sixth embodiments, the short film generation / reproduction devices 1001a to 1001d and the display device 1002 are separately configured. However, the present invention is not limited to this, and for example, all may be configured integrally. Absent.

(Embodiment 7)
FIG. 17 is a functional block diagram showing another configuration of the short film generation / playback device according to Embodiment 7 of the present invention.

  The short film generation / playback device 2001 generates a scenario of a short video (short film) using one or more input still images, and plays back a short film using the generated scenario. As shown in FIG. 17, the input unit 2010, the database 2020, the scenario generation unit 2030, the playback unit 2040, the display unit 2050, and the internal bus connecting these input units 2010 to the display unit 2050 2060 and the like. Here, only the scenario generation unit 2030 corresponding to the scenario generation unit 1117 is shown among the short film generation units 1105a to 1105d described above.

  Here, the scenario is not the moving image data itself as represented by MPEG, but the time-series visual effects (effects), parameters that define the operation of the effects, pointers to still images that are the targets of the effects, etc. It is described in a script language represented by XML. In the seventh embodiment, a scenario is generated by mapping still images to “effects” arranged on the time axis.

The input unit 2010 is a unit that inputs a still image from a digital camera or a medium on which a still image is recorded by the digital camera. The input unit 2010 includes a still image selection unit 2110, an object information extraction unit 2120, and the like.
The still image selection unit 2110 reads a still image recorded on the medium into a memory (not shown) in the short film generation / playback apparatus 2001 and registers it in the database 2020. Examples of the media include semiconductor memory media such as compact flash (R) memory card and SD memory card (R), optical disc media such as CD-R / ROM / RW, DVD-R / RAM / RW, tape, flexible disc, etc. Or a magneto-optical disk such as MO.

  Note that the still image 2210 registered in the database 2020 via the still image selection unit 2110 is, for example, JPEG format image data, including a still image main body and metadata indicating its attributes, and most of the metadata is , Data generated by a device such as a digital still camera that has generated a still image. Typical data of this metadata includes shooting date and time, shooting location (GPS position information), shooting conditions (shutter speed, focal length, exposure time, etc.) (hereinafter referred to as semantic information), and these are EXIF. The specifications are formulated by JEIDA (Japan Electronics Industry Promotion Association).

The object information extraction unit 2120 extracts an object such as a human face from the input image data, and generates object information. The generated object information is registered in the database 2020.
FIG. 18 is a diagram illustrating a configuration example of the object information 2220 registered in the database 2020.

  As shown in FIG. 18, the object information 2220 includes the coordinates of the center of the circle that the extracted object touches, the position information 2213 represented by the size of the radius, the object type 2212, and the corresponding still image file. Name 2211. The type of object is whether it is a human face or not. Note that the position information of the object can be expressed by the coordinates of the start point and the end point of one diagonal line of the quadrangle, for example, without depending on the method of expressing it by the center and radius of the circle.

  The database 2020 includes a still image 2210 input from the still image selection unit 2110 of the input unit 2010, object information 2220 generated by the object information extraction unit 2120, an effect table 2230, and music used as BGM during short film playback. 2240, a style 2250 that represents the overall image of the video to be generated, a theme 2260 of the video to be generated, a scenario 2270, and the like.

The database 2020 can be configured by a storage medium such as a hard disk or a RAM. In this case, a means for storing the still image 2210 or the like in the storage medium or reading it from the storage medium may be provided separately.
The effect table 2230 is a table showing the relationship between the effect name of each effect included in the effect group and the number of objects required by the effect.

FIG. 19 is a diagram illustrating a configuration example of the effect table 2230.
As shown in FIG. 19, the effect table 2230 includes an effect name column and a requested object number column.
In the effect name column, effect names that define what visual effect is used to reproduce one or more target still images, for example, spotlight eff1, zoom eff2, trimming eff3, crosstalk eff4, A list of slide-in eff5, slide-out eff6, and the like is stored. In the column of the number of requested objects, “1-4”, “1-4”, “1,2”, “2”, “any number OK”, corresponding to the spotlight eff1 to the slide-out eff6, “Any number of OK” is stored.

  For example, the spotlight eff1 is an effect that focuses on one object, and after the spotlight searches in the photo, it finally focuses on the target object. Photos are mapped. The slide-in eff5 and the slide-out eff6 are obtained by dividing a still image vertically, horizontally, or obliquely, sliding in or out vertically, horizontally, or obliquely, or dividing a quadrilateral photograph into four parts, and dividing each fragment into four parts. An effect that slides in or out of two corners, even if it is a picture with a large number of objects.

  Note that panning that moves the entire still image in either direction is stored in the effect name column, the position of the object is detected, and the farthest position is set as the start position, so that the movement becomes large and the appearance is good. When you get a visual effect or overlap two photos, store the montage that aligns the parts of the object in the effect name column, slide out the upper photo, you can see the lower photo like a montage You can get a visual effect, store zoom-in / out that zooms in / out around a specific object in the effect name column, and fade / in fade-in / out that disappears while alpha blending Stored in the field, read the coordinate position of the object, to obtain a visual effect to reduce the α value from a distant position, or to a specific object The focus in / out masking the background is stored in the effect name column, and the relationship between the center of gravity of the figure to be focused and the coordinate position of the object is determined. A visual effect set to the farthest position or the like may be obtained. In this case, the number of requested objects may be set according to the effect attribute.

The scenario 2270 is generated by picking up an effect from a set of effects to determine the arrangement in accordance with the time length of music, and mapping a still image to the effect arranged on the time axis. .
The scenario generation unit 2030 generates a scenario using a still image selected in advance by the user, a determined style and music, or a determined style, music, and theme, and registers the generated scenario in the database 2020. , An effect placement unit 2310, a still image assignment unit 2320, a parameter setting unit 2330, and the like.

  The effect placement unit 2310 is a block that extracts one effect from a set of effects based on a predetermined music and style or music, style and theme, and places the effect on the time axis. By repeating this, Form the skeleton of a scenario with a certain length of playback time. At this time, the length of the scenario playback time may be the length of the music to be BGM, or the time length defined by the user or the system.

  The still image allocation unit 2320 reads the description in the attribute for each effect of the skeleton of the scenario formed by the effect arrangement unit 2310, and searches the database 2020 for a still image having the required attribute. And map. More specifically, in the seventh embodiment, the still image allocation unit 2320 searches for a still image based on the number of objects requested by each effect, and allocates the still image obtained as a result of the search to the effect. . That is, mapping is performed. Therefore, when the effect is a spotlight, the still image assignment unit 2320 searches for a photograph in which one or more and four or less objects are captured, determines one photograph obtained as a result, and maps it to the spotlight. To do.

The parameter setting unit 2330 is called in a state where the mapping between the effect and the still image is determined, and determines a parameter for determining the operation of each effect in the scenario.
For example, taking the case where the effect is slide out eff6 as an example,
1. Direction in which the sliding block moves (vertical, horizontal, diagonal, four corners from the center)
2. 2. Width of block to slide out size By confirming the three parameters of the order in which the order block slides out, the operation of the slide out can be confirmed.

The playback unit 2040 reads a scenario, generates a plurality of frame data, and displays them in time series. The playback unit 2040 includes a short film selection unit 2410, a short film playback unit 2420, and the like.
The short film selection unit 2410 provides a user with a GUI for selecting a short film to be reproduced.

The short film playback unit 2420 reads the selected short film scenario and the still image and music defined there from the database 2020, decodes the scenario, plays back the short film, and outputs it to the display unit 2050. Thereby, the user can view as a moving image.
The display unit 2050 includes a device for displaying an image such as an LCD or a cathode ray tube, and a built-in speaker, and displays a reproduced short film image or the like.

Next, an operation when a still image is input by the short film generation / playback apparatus configured as described above will be described.
FIG. 20 is a flowchart illustrating an algorithm when inputting a still image in the input unit 2010.
The still image selection unit 2110 and the object information extraction unit 2120 repeatedly execute the following processing for each still image for all the still images recorded on the medium (S2011).

First, the still image selection unit 2110 reads the still image recorded on the medium into the memory in the short film generation / playback device 2002 (S2012), and registers the read still image in the database 2021 (S2013). The image data storage format may be a compression format represented by the JPEG format or an uncompressed format.
Next, the object information extraction unit 2120 extracts all the objects included in the read still image (human face in this embodiment 7), and generates object information composed of the position information of the object. (S2014). When the generation of the object information is finished, the input unit 2010 registers the object information in the database 2020 (S2015). As a result, object information 2220 (see FIG. 18) corresponding to the still image 2210 is registered in the database 2020.

When all the still images 2210 and the object information 2220 are registered in the database 2020 for all the still images (S2016), the input unit 2010 ends the still image data input process.
With the above processing, the position of the object and the like are obtained for all the still images to be subjected to image processing, and image processing centered on the object becomes possible.

Next, an operation when generating a short film scenario will be described.
FIG. 21 is a flowchart showing an algorithm until a scenario to be executed by each unit of the scenario generation unit 2030 is generated. This scenario generation process is executed after receiving an instruction from the user for the target range of the still image used on the short film, or after determining the music, style, and theme used on the short film.

  First, the effect placement unit 2310 selects one effect from a plurality of effects stored in the effect table 2230, and places it on the selected time axis (S2021). Specifically, the effect still, zoom, slide-in, slide-out,... Are arranged in this order according to the determined music time length. Thus, a scenario skeleton is formed.

When the arrangement of the effects on the time axis ends, the still image assignment unit 2320 and the parameter setting unit 2330 determine the still images to be mapped for all the effects arranged on the time axis, and set various parameters ( S2022).
First, the still image allocation unit 2320 uses the object information 2220 to search for still images that satisfy the number of requested objects using the number of objects in the effect attribute information as a key, and a plurality of still images obtained by the search. A still image selected from among them is determined as a still image to be mapped (S2023). Specifically, when the effect is slide-out eff6, the still image assigning unit 2320 searches for a photograph in which one or more objects are shown, determines one photograph obtained as a result, and assigns it to the slide-out eff6. Map.

Next, the parameter setting unit 2330 determines a parameter based on the object information corresponding to the mapped still image (S2024). Specifically, the parameter setting unit 2330 reads out object information (position information (center of gravity coordinates / size) of each object) corresponding to the mapped still image, and determines the parameter of each effect.
When the determination of still images to be mapped and the determination of parameters are completed for all effects (S2025), the scenario is completed, and the scenario generation unit 2030 registers the generated scenario in the database 2020 (S2026), thereby generating the scenario. The process ends.

Next, the parameter determination process in step S2024 will be described.
FIG. 22 is a flowchart showing a subroutine for explaining an algorithm by which parameter setting unit 2330 determines effect variations and parameters. In order to simplify the description, the parameter determination process will be specifically described by taking “slide out” as an example.

  Here, slide-out is an effect that divides a displayed still image into 1 to n blocks (n is a natural number of 2 or more) and moves the blocks in order to the outside of the screen. There are four variations of vertical, horizontal, diagonal, and four corners depending on the direction of out (see FIG. 23).

  1. In the vertical variation, the still image is divided into vertically long rectangular blocks by vertical lines, and the blocks are moved up or down (FIG. 23A).

  2. In the horizontal variation, the still image is divided into horizontally long rectangular blocks by a horizontal line and slid out to the right or left (FIG. 23B).

  3. In the diagonal variation, it is divided into blocks tilted to the right (trapezoids, both ends may be triangular) by a straight line of inclination: θ (0 ° <θ <90 °), and diagonally right upward or diagonally left The block is moved downward (FIG. 23 (c)).

  In the four-corner variation, the still image is equally divided into four by horizontal and vertical lines intersecting at the center of gravity of the still image, and each is moved in the direction of the nearest corner (and thus each block moves in four directions). (FIG. 23 (d)).

  It should be noted that, in the oblique variation, when θ = −90 ° and 0 °, it is equivalent to the horizontal and vertical variations, respectively, and therefore whether or not it can be implemented is considered in the same algorithm. In the seventh embodiment, the still image object is a human face.

First, the parameter setting unit 2330 uses the object information 2220, and the object O [i] (i = 0, 1, 2,...) Included in the still image mapped to the effect in the xy coordinate system. The barycentric coordinates x [i], y [i] and the size r [i] are read (S2401).
Next, the parameter setting unit 2330 obtains a block boundary line for the rotation angle: θ (θ = −90 °, θ = 0 °, 0 ° <θ <90 °), and sets vertical, horizontal, and diagonal flags. Set (S2402).

First, the parameter setting unit 2330 calculates the barycentric coordinates s [i], t [i] of each object O [i] in the rotational coordinate (s-ot) system (S2403).
FIG. 24 is a diagram for explaining a rotating coordinate system: sot coordinate system obtained by rotating the original coordinate system (x-o-y) by θ ° (0 ° <θ <90 °).
In FIG. 24A, an object O [i] is represented by coordinates of the center of gravity (x [i], y [i]) of the object in the x-o-y coordinate system and the radius of the circle circumscribing the object: r [ i]. H is the height of the still image. The coordinates (s [i], t [i]) in the so-t-t coordinate system can be obtained by the following formula.

重心座標の算出が終わると、パラメータ設定部２３３０は、各オブジェクトＯ［ｉ］をｔ軸上にマッピングする（Ｓ２４０４）。なお、図中に示されるｔ［ｉ］はオブジェクトＯ［ｉ］のｔ軸への写像である。これにより、各オブジェクトＯ［ｉ］がｔ軸上に並ぶことになる。

When the calculation of the barycentric coordinates ends, the parameter setting unit 2330 maps each object O [i] on the t-axis (S2404). Note that t [i] shown in the figure is a mapping of the object O [i] to the t-axis. As a result, the objects O [i] are arranged on the t-axis.

  Next, the parameter setting unit 2330 cannot place a point on the center-of-gravity coordinate t [i] on the t-axis of each object O [i] and the region of ± R centered on it, that is, a block parallel to the s-axis It is set as a prohibited area P [i] where a boundary line cannot be drawn (S2405). Here, R is larger than r [i], and is set to a value that reliably includes hair and the like.

When the setting of the prohibited area, that is, the setting of the prohibited band is completed, the parameter setting unit 2330 determines whether an object exists on the positive side on the t-axis with respect to each barycentric coordinate t [i] on the t-axis. After checking, a block boundary line is acquired, and flags in the vertical, horizontal, and diagonal directions are set (S2406).
First, the parameter setting unit 2330 determines whether there is an object on the positive side of the target object O [i] on the t-axis, that is, whether there is t [i + 1] (S2407). If t [i + 1] exists (Yes in S2407), it is determined whether or not there is a gap between the object O [i] and the forbidden band of the positive object O [i + 1] on the t-axis (S2408).

  If there is a gap (Yes in S2408), it means that the still image can be divided by a straight line (tilt: θ) that passes through the points between the gaps and is parallel to the s-axis (see FIG. 24B). It is possible to divide a still image at a block boundary line without dividing. Therefore, the parameter setting unit 2330 calculates the center of the gap, acquires the block boundary line BL [j] at θ parallel to the s-axis passing through the center of the gap (S2409), and flags the variation at θ. Set (S2410). Note that the block boundary line is defined by a position yBL [j] on the y-axis, a position on the x-axis, and θ.

  If the block boundary line can be acquired or if there is no gap between the prohibited bands (No in S2408), steps S2407 to S2410 are performed on the positive side on the t-axis for the positive object O [i + 1]. Repeat until no more objects exist. If the distance between the edge of the still image and the prohibited area of the positive object on the t-axis is wider than a predetermined threshold value, as shown in BL1 in FIG. You may make it provide a block boundary line in the middle.

When the acquisition of the block boundary line at a certain rotation angle θ and the flag is set in the variation at that θ (S2411), the rotation angle θ is changed, the block boundary line is acquired at the changed θ, and Run flag set on variation.
When the acquisition of the block boundary line at all the rotation angles θ and the setting of the flag to the variation at the θ are executed (S2412), the parameter setting unit 2330 displays the block boundary line parallel to the x-axis passing through the center of the still image. The still image is divided into four blocks by the block boundary line parallel to the y-axis (S2413). Then, the parameter setting unit 2330 determines whether or not all the objects O [i] can fit in any of the four blocks (S2414).

If all the objects O [i] can fit in any of the four blocks (Yes in S2414), since all the objects are not divided, a four-corner flag is set (S2415), and the process proceeds to step S2414.
On the other hand, if at least one of the objects O [i] does not fit in any of the four blocks (No in S2414), the object will be divided, so that the square flag is not set. The process proceeds to step S2414.

  When an executable variation is found, the parameter setting unit 2330 randomly selects one variation from the possible variations (S2416). In this case, we selected a random selection method using random numbers to select one of the variations, but by saving all the information necessary for implementing the possible variations and the history of the effects It is also possible to set to select so as not to suffer from the variation of the effect selected last time.

  When the variation is determined, the parameter setting unit 2330 determines the width of the block to be slid out based on the block boundary line, and determines the moving order and moving direction of the block to be slid out based on the coordinates and size of the object. (S2417). Specifically, except for the four-sided variation, if the block is divided into n blocks, the block that contains the object at the center is the nth (slide out last), and the farthest position from that block The block located at is the block that slides out to the first position. In the following, it is determined that the next farthest block is number 2, the next farthest block is number 3, and so on.

Note that the method for determining the order in which the blocks are slid out is not limited to this method, and conversely, the block including the object located at the center most may be the first. In the case of the four-angle variation, the four blocks are simultaneously slid out in four directions, so the order of the slide-out is irrelevant.
When the block width, moving order and moving direction, that is, all parameters necessary for the slide-out are determined, the process returns to the main routine. As a result, the scenario including the slide-out is registered in the database 2020.

FIG. 25 is a diagram illustrating a configuration example of a scenario 2270 registered in the database 2020. The scenario 2270 is described in the script as described above, but is illustrated in a table format for convenience of explanation.
As shown in FIG. 25, the scenario 2270 includes a start time and an end time indicating time allocation on the time axis, an effect allocated on the time axis, a still image allocated to each effect, and a parameter. Is done. When the effect is slide-out eff6, parameters include the selected variation (flag F0 = 1 indicating variation at θ and its θ), and the width of the block to be slid out (the position of the block boundary that defines the block) yBL [j] and θ), the order of the blocks to slide out (B0, B1,...), and the direction of the blocks to slide out (B0: diagonally right upward, B1: diagonally downward right,...).

  When a short film is played back using the scenario configured as described above, the short film selection unit 2410 of the playback unit 2040 instructs to read the short film scenario and the music and still images defined there, and plays back the short film. The unit 2420 plays a short film using music and still images according to the instructed scenario, and outputs the short film to the display unit 2050. In this case, the short film playback unit 2420 performs processing defined by parameters corresponding to the effect from the still image for each effect to generate a frame image, and plays back the short film.

As described above, according to the seventh embodiment, the parameters for determining the effect variation and the effect operation are determined based on the object information of the still image.
Therefore, a situation in which the face of a person who is an object is divided is avoided, and even with the same effect, it is possible to execute an effect of a variation optimal for a mapped still image. On the other hand, from the viewpoint of the viewer, even if the effect is the same, the operation differs depending on the mapped still image. As a result, the short film that the user sees during playback will appear as if different effects continue even if the same effects continue, and it will not get bored with a single pattern, producing highly entertaining content it can. Furthermore, even when mapping the same effect and still image, it is possible to select from a plurality of variations, and it is possible to realize different ways of showing.

In the invention according to the seventh embodiment, the conversion to the rotation coordinate system is performed in order to determine whether the variation is effective or not. In addition, a template is prepared for each variation. There is also a method for determining whether or not the straight line divided into blocks can be moved within a certain range so as not to overlap the object.
Further, in order to acquire a block boundary line having an inclination opposite to that in FIG. 23C, the rotation angle θ may be changed within a range of 90 ° <θ <180 °.

(Embodiment 8)
Next, a short film producing / reproducing apparatus according to Embodiment 8 of the present invention will be described.
FIG. 26 is a functional block diagram showing another configuration of the short film generation / playback device according to the eighth embodiment of the present invention. In addition, the same number is attached | subjected to the part corresponding to the short film production | generation reproduction | regeneration apparatus 2001 of Embodiment 7, and the description is abbreviate | omitted.

  As illustrated in FIG. 26, the short film generation / playback device 2002 includes an input unit 2011, a database 2021, a scenario generation unit 2031, a playback unit 2040, a display unit 2050, and an input unit 2011 to a display unit 2050. And an internal bus 2060 to be connected. The input unit 2011 further includes a face feature point extraction unit 2130 in addition to the still image selection unit 2110 and the object information extraction unit 2120. The scenario generation unit 2031 includes a still image allocation unit 2321 and a parameter setting unit 2331 in addition to the effect arrangement unit 2310. Here, as in the case of the seventh embodiment, only the scenario generation unit 2031 corresponding to the scenario generation unit 1117 is shown among the short film generation units 1105a to 1105d.

  By the way, although the input unit 2010 of the short film generation / playback device 2001 is configured to extract only the position of the object (person's face), the input unit 2011 of the short film generation / playback device 2002 includes the input unit 2010. In addition to the configuration, it further comprises a face feature point extraction unit 2130 that extracts facial feature points (face parts, such as eyes, nose, mouth, etc.), and by adding this face feature point extraction unit 2130, The point that a scenario corresponding to an effect (for example, montage) conscious of facial parts is created and a short film is reproduced using this scenario is greatly different from the short film generation and reproduction apparatus 2001.

FIG. 27 is a storyboard showing an example thereof.
As shown in FIGS. 27 (a) to 27 (g), in a storyboard, a so-called horizontal slide-out process that sequentially slides out a block of parts including the head, eyes, nose, and mouth is a so-called process. A montage-like effect is shown.
In this effect, by acquiring the position and size of the face, eyes, nose, and mouth, the width of the block is determined, and a visual effect like a montage is obtained.

In order to realize such processing, the effect table 2231, the object information 2221, and the scenario 2271 stored in the database 2021 are configured to realize a montage-like effect.
Specifically, as shown in FIG. 28, the effect table 2231 includes an effect name field, a requested object number field, and a requested object feature point field. Then, the montage eff10 is stored in the effect name column, "1" or "2" is stored in the column of the number of requested objects corresponding to the montage eff10, and the feature point column of the requested object is stored. The head, eyes, nose and mouth are stored.

FIG. 29 is a flowchart showing an algorithm for generating object information for realizing a montage-like effect.
The still image selection unit 2110 and the object information extraction unit 2120 repeatedly execute the following processing for each still image for all the still images recorded on the medium (S2011).

  First, the still image selection unit 2110 reads the still image recorded on the medium into the memory in the short film generation / playback apparatus 2001 (S2012), and registers the read still image in the database 2020 (S2013). Next, the object information extraction unit 2120 extracts all the objects included in the read still image (S2014), and registers the position information (coordinates / sizes) of the object in the object information (S2015).

  When the object is identified as a face, when the acquisition of the position information of the object is completed, the face feature point extraction unit 2130 extracts the face feature points (S2101), and registers the extracted face feature points in the object information. (S2102). Here, the face feature point information is information including the coordinates (a plurality) of feature points on the face. For example, the head of the eye, the corner of the eye, the apex of the nose, the ends of the mouth, the beginning and end of the eyebrows, and the like. As a technology for extracting feature points on the face, for example, in the 2002 Sensing Technology Application Seminar “Crisis Management and Security for Safety and Security” hosted by the Sensing Technology Application Study Group, There are technologies introduced in “Applications”, and such existing technologies may be used.

When the generation of all object information is completed, the input unit 2011 registers the object information in the database 2021. As a result, object information 2221 (see FIG. 30) corresponding to the still image 2210 is registered in the database 2021.
When registration of the still image 2210 and the object information 2221 with respect to the database 2021 is completed for all still images (S2016), the input unit 2011 ends the data input process.

FIG. 30 is a diagram showing a configuration example of the object information 2221 stored in the database 2021 by the data input process described above.
As shown in FIG. 30, the object information 2221 includes feature points in addition to the coordinates of the center of the circle that the extracted object touches, the position information represented by the size of the radius, the file name of the corresponding still image, and the like. It is comprised including. As described above, the feature points are the position coordinates of the start and end of the eyebrow, the corner of the eye, the apex of the nose, both ends of the mouth, and the eyebrows.

With the object information 2221 configured as described above, the position of the object, the facial feature point, and the like are acquired for all the still images to be subjected to image processing, and image processing centering on the object parts becomes possible.
Next, scenario generation processing will be described.
FIG. 31 is a flowchart showing an algorithm for generating a scenario to be executed by each unit of the scenario generation unit 2031. This scenario generation process is executed after receiving an instruction from the user for the target range of the still image used on the short film, or after determining the music, style, and theme used on the short film.

  First, the effect placement unit 2310 selects one effect from a plurality of effects stored in the effect table 2230, and places it on the selected time axis (S2021). Specifically, an effect montage, zoom, and the like are arranged in this order in accordance with the determined music time length. Thus, a scenario skeleton is formed.

When the arrangement of the effects on the time axis is completed, the still image assignment unit 2321 and the parameter setting unit 2331 determine still images to be mapped for all the effects arranged on the time axis, and set various parameters ( S2022).
First, the still image allocating unit 2321 uses the object information 2221 to search for still images that satisfy the number of requested objects using the number of objects and feature points in the effect attribute information as keys, and a plurality of images obtained by the search. A still image selected from the still images is determined as a still image to be mapped (S2023). Specifically, a still image having the number of objects and feature points required by the attribute of the effect is searched and mapped to the effect.

  Next, the parameter setting unit 2331 determines parameters based on object information corresponding to the mapped still image (S2024). Specifically, the parameter setting unit 2331 first reads out object information. Then, from the information about the head, eyes, nose, and mouth in the facial feature points, the coordinates and size of the center of gravity are acquired, and the width of the blocks to be slid out and the order in which the blocks slide out are determined. The order in which the blocks slide out may be random, but the part other than the face is the first, the block containing the mouth, the block containing the nose, the block containing the head, and the block containing the eyes at the end. If you slide out the block that contains the eye like this, you can achieve an effect with less discomfort to the viewer. In addition, the block boundary between parts is determined so that the division of parts is reduced in consideration of the importance of parts and the coordinates and size of the center of gravity. In addition, the position information of the object can be expressed by the coordinates of the start point and the end point of one diagonal line of the quadrangle, for example, without depending on the method of expressing by the center and radius of the circle.

For all effects, when the determination of the still image to be mapped and the determination of the parameters are completed (S2025), the scenario is completed. Therefore, the scenario generation unit 2031 registers the generated scenario in the database 2021 (S2026) to generate the scenario. The process ends.
FIG. 32 is a diagram illustrating a configuration example of the scenario 2271 registered in the database 2021.

  As shown in FIG. 32, the scenario 2271 includes a start time and an end time indicating time allocation on the time axis, an effect allocated on the time axis, a still image allocated to each effect, and a parameter. Is done. When the effect is montage, the parameters describe the width of the block to slide out, the order of the blocks to slide out, and the direction of the blocks to slide out.

  When a short film is played back using the scenario 2271 configured as described above, the short film selection unit 2410 of the playback unit 2040 instructs to read the short film scenario and the music and still images defined there, and the short film. The playback unit 2420 plays back the short film using music and still images according to the instructed scenario, and outputs the short film to the display unit 2050. In this case, the short film playback unit 2420 generates a frame image defined by parameters corresponding to the effect from the still image for each effect, and plays back the short film. When generating a frame image, parameters for determining the position and size of the block are determined based on face part information (head, eyes, nose, mouth) in the object (face) of the still image.

Therefore, according to the invention according to the eighth embodiment, although it is an effect cut off in the middle of the face, it can produce a visual effect with high entertainment characteristics that does not give a sense of incongruity to what it sees and includes further fun. .
As shown in FIG. 33 (a), when two faces are included in the still image mapped to the montage, the faces of the two people are trimmed so that the face sizes are almost the same. Normalize, align the parts of the object, overlay the two photos (see Fig. 33 (b)), slide out the top photo for each part block, and the bottom photo for each part You may make it perform the process of the montage style which can be seen (FIG.33 (c)-FIG.33 (h)).

By such processing, it is possible to produce a visual effect with further enhanced entertainment properties.
(Embodiment 9)
FIG. 34 is a block diagram showing a functional configuration of a short film generation / playback device 2003 according to Embodiment 9 of the present invention.

  As shown in FIG. 34, the short film generation / playback device 2003 includes an input unit 2012, a database 2022, a scenario generation unit 2032, a playback unit 2040, a display unit 2050, and these input units 2012 to 2050. And an internal bus 2060 to be connected. In addition, the same number is attached | subjected to the part corresponding to the short film production | generation reproduction | regeneration apparatus 2001 of Embodiment 7, and the description is abbreviate | omitted. Here, as in the case of the seventh and eighth embodiments, only the scenario generation unit 2032 corresponding to the scenario generation unit 1117 is shown among the short film generation units 1105a to 1105d.

The input unit 2012 includes a face authentication unit 2140 in addition to the still image selection unit 2110 and the object information extraction unit 2120. In addition to the effect placement unit 2310, the scenario generation unit 2032 includes a still image assignment unit 2322 and a parameter setting unit 2332.
By the way, although the input unit 2010 of the short film generation / playback device 2001 is configured to extract only the position of the object (human face), the input unit 2012 of the short film generation / playback device 2003 In addition to the configuration, a face authentication unit 2140 for authenticating who the object is is configured, and by adding the face feature point extraction unit 2130, an effect (for example, a spot is recognized) A scenario corresponding to “write” is created, and a short film is reproduced using this scenario, which is greatly different from the short film generation and reproduction apparatus 2001.

  The face authentication unit 2140 compares the face detected in the still image with the face information 2280 registered in advance in the database 2022, determines who the person is, and adds the name to the object information in addition to the face information. As a block to be added, an existing face authentication technique can be used. Here, applications using the face authentication technology are commercialized from various companies, such as FaceKey of OMRON Corporation and NeoFace of NEC Corporation. These applications are used in the security system, and compare the face image registered in advance with the face in the acquired image. If the person is a registered person, the door is locked, and the face authentication unit 2140 These existing face authentication techniques can be used.

  FIG. 35 is a storyboard showing an example of an effect (spotlight) focused on a specific person. This storyboard has a still image of three people (from left, Schiart, Nash, and Nikita) (see Fig. 35 (a)), and a spotlight moves around it (Fig. 35 (b) ~ FIG. 35 (d)) and finally, the state of focusing the spotlight on Mr. Nikita (see FIG. 35 (e)). Because of this effect, it is possible to collect the attention of what Nikita sees even if it is a photograph of multiple people including Nikita as the subject, so the completed short film is focused on Nikita. Become.

In this effect, the key is to obtain a visual effect that shines a spotlight on a specific person by authenticating who the person is.
In order to realize such processing, the effect table 2232, the object information 2222, and the scenario 2272 stored in the database 2022 are configured to realize an effect that shines a spotlight on a specific person.

  Specifically, as shown in FIG. 36, the effect table 2232 further includes an effect name column and a requested object number column, and further includes a column indicating whether or not one person can focus. One of “permitted” and “not permitted” is stored in the column of whether focus is possible for one person. For example, in the case of the spotlight eff1, “OK” is stored. In this case, it is possible to obtain a visual effect that focuses on one person.

FIG. 37 is a flowchart showing an algorithm for generating object information for realizing an effect of shining a spotlight on one person.
The still image selection unit 2110, the object information extraction unit 2120, and the face authentication unit 2140 repeatedly execute the following processing for each still image for all the still images recorded on the medium (S2011).

  First, the still image selection unit 2110 reads the still image recorded on the medium into the memory in the short film generation / playback device 2003 (S2012), and registers the read still image in the database 2022 (S2013). Next, the object information extraction unit 2120 extracts all the objects included in the read still image, and generates object information including position information of the objects (S2014, S2015).

  When the object is identified as a face, when the acquisition of the position information of the object is completed, the face authentication unit 2140 executes face authentication processing, acquires the name of each object (S2111), and uses the acquired name as the object. Information is registered (S2112). Specifically, face authentication unit 2140 compares face information 2280 for personal authentication registered in advance in database 2022 with face information passed from object information extraction unit 2120 to search for a name. If found, the name is registered in the database 2022 together with the face information as part of the object information. Needless to say, if the name is not found, the name is not registered.

When the generation of all the object information is completed, the input unit 2012 registers the object information in the database 2022. As a result, object information 2222 (see FIG. 38) corresponding to the still image 2210 is registered in the database 2022.
When all the still images 2210 and the object information 2222 are registered in the database 2022 for all the still images (S2216), the input unit 2012 ends the data input process.

FIG. 38 is a diagram showing a configuration example of the object information 2222 stored in the database 2022 by the data input process described above.
As shown in FIG. 38, the object information 2222 includes a name in addition to the coordinates of the center of the circle that the extracted object touches, the position information represented by the size of the radius, and the file name of the corresponding still image. Consists of including.

With the object information 2222 configured in this way, the position and name of each object are obtained for all still images to be subjected to image processing, and image processing centered on the name of the object specified by the user is possible. It becomes.
FIG. 39 is a flowchart showing an algorithm until a scenario to be executed by each unit of the scenario generation unit 2032 is generated. This scenario generation process is executed after receiving an instruction from the user for the target range of the still image used on the short film, or after determining the music, style, and theme used on the short film.

  First, the effect placement unit 2310 selects one effect that can execute a visual effect focused on one person from a plurality of effects stored in the effect table 2232, and places the selected effect on the selected time axis (S2021). . Specifically, the effect spotlight, zoom, and the like are arranged in this order in accordance with the determined music time length. Thus, a scenario skeleton is formed.

  When the effects are placed on the time axis, the still image assignment unit 2322 and the parameter setting unit 2332 search the database 2022 for photos of all the effects placed on the time axis and map them. A still image is determined and various parameters are set (S2022). At this time, not only a photograph of Nikita alone but also a photograph of a plurality of people including Nikita may be used.

  First, the still image allocation unit 2322 searches the effect table 2232 for a still image that satisfies the number of requested attribute information using the attribute information of the effect as a key, and selects one of the plurality of still images obtained by the search. The selected still image is determined as a still image to be mapped (S2023). Specifically, a still image having the number of objects required by the attribute of the effect and whether or not one (Mr. Nikita) can focus is searched and mapped to the effect.

  Next, the parameter setting unit 2332 determines parameters based on object information corresponding to the mapped still image (S2024). Specifically, the parameter setting unit 2332 reads the object information, sets the position and size of Mr. Nikita's face included therein, and determines the arrival position (destination) of the spotlight. In the case of a slide-out effect instead of a spotlight effect, you can make Nikita look longer on the viewer side by setting the block containing Nikita's face to slide out last. Can impress Nikita. On the other hand, in the case of the slide-in effect, by setting the block including Nikita's face to slide in first, Nikita can be shown long, and Nikita can be impressed.

When the determination of the still image to be mapped and the determination of the parameters are completed for all the effects (S2025), the scenario is completed. Therefore, the scenario generation unit 2032 registers the generated scenario in the database 2022 (S2026), and generates the scenario. The process ends.
FIG. 40 is a diagram illustrating a configuration example of the scenario 2272 registered in the database 2022.

  As shown in FIG. 40, the scenario 2272 includes a start time and an end time indicating time allocation on the time axis, an effect allocated on the time axis, a still image allocated to each effect, and a parameter. Is done. When the effect is a spotlight, an object (including position and size) to which the spotlight is applied is described in the parameter.

  When the short film is reproduced using the scenario 2272 configured as described above, the short film selection unit 2410 of the reproduction unit 2040 instructs to read the short film scenario and the music and still image defined therein, and the short film is reproduced. The playback unit 2420 plays back the short film using music and still images according to the instructed scenario, and outputs the short film to the display unit 2050. In this case, the short film playback unit 2420 generates a frame image defined by parameters corresponding to the effect from the still image for each effect, and plays back the short film. Then, when generating a frame image, a parameter for spotlighting the position information of the object (face) specified by the name of the still image is determined.

  As described above, according to the ninth embodiment, the name of the object in the still image can be recognized by the face authentication unit 2140. Therefore, the effect focused on the person using the photo including the arbitrary person. Can be executed. As a result, the completed short film is focused on an arbitrary person and can be used in accordance with the purpose of reproducing the short film. For example, since the wedding reception is a place where the bride and groom are the main characters, even if it is a short film generated using photos taken by many friends including them, it should be a short film focused on the bride and groom Is desirable. If the short film generation / reproduction device 2003 according to Embodiment 9 of the present invention is used, it is possible to select a photograph of the bride and groom and set them as the center of the effect. It is possible to easily produce a short film.

(Embodiment 10)
FIG. 41 is a block diagram showing a functional configuration of a short film generation / playback device 2004 according to Embodiment 10 of the present invention.
As shown in FIG. 41, the short film generation / reproduction device 2004 includes an input unit 2013, a database 2023, a scenario generation unit 2033, a reproduction unit 2040, a display unit 2050, and these input units 2013 to 2050. In addition to the internal bus 2060 to be connected, an address book 2070 is included. In addition, the same number is attached | subjected to the part corresponding to the short film production | generation reproduction | regeneration apparatus 2003 of Embodiment 9, and the description is abbreviate | omitted. Also, here, as in the case of the seventh and eighth embodiments, only the scenario generation unit 2033 corresponding to the scenario generation unit 1117 is shown among the short film generation units 1105a to 1105d.

  In the input unit 2012 of the ninth embodiment, the name of the object is acquired by the face authentication unit 2140. However, the input unit 2013 registers the name and personal data using the name obtained by the face authentication unit 2140. A personal information search unit 2150 for searching a look up table (for example, an address book), and adding attributes (for example, gender, age, hobby, affiliation, etc.) of objects (subjects) in still images as object information However, using the added object information, it is greatly different in that it is configured so that even the same effect can add fun, fun and entertainment as additional value. Yes.

As shown in FIG. 42, the address book 2070 includes attributes such as an address, a telephone number, an e-mail address, sex, age, date of birth, affiliation, and hobby in addition to the name.
FIG. 43 is a storyboard showing an example of an effect using gender. This effect is to display a character (a heart symbol in FIG. 43) that implies the relationship when the object is male and female, compared to the conventional slide-in effect. The block containing the male face and the block containing the female face slide in to generate a single picture (FIGS. 43A and 43B). After that, a character that implies the relationship between them, for example, a heart mark or the like is displayed (FIGS. 43C and 43D). On the other hand, when the sex is not male or female, as shown in FIG. 44, the faces of the two people are focused by a normal circle.

In order to realize such processing, the effect table 2233, the object information 2223, and the scenario 2273 stored in the database 2023 are configured to realize an effect in consideration of the gender of a specific person.
Specifically, as shown in FIG. 45, the effect table 2233 further includes a gender and character column for the requested object in addition to a column for the effect name and the number of requested objects. The requested object's gender column stores one man and one woman, and other combinations are possible, and the character column contains a “heart symbol” for men and women, and other combinations. In this case, “◯” is stored. This makes it possible to obtain a visual effect that displays a character that matches the sex combination.

FIG. 46 is a flowchart showing an algorithm for generating object information for realizing an effect of changing a character displayed by a combination of gender.
The still image selection unit 2110, the object information extraction unit 2120, the face authentication unit 2140, and the personal information search unit 2150 repeatedly execute the following processing for each still image for all the still images recorded on the medium (S2011). ).

  First, the still image selection unit 2110 reads the still image recorded on the medium into the memory in the short film generation / reproduction device 2004 (S2012), and registers the read still image in the database 2023 (S2013). Next, the object information extraction unit 2120 extracts all the objects included in the read still image, and generates object information including position information of the objects (S2014, S2015).

When the object is identified as a face, when the acquisition of the position information of the object is completed, the face authentication unit 2140 executes face authentication processing, acquires the name of each object (S2111), and uses the acquired name as the object. The information is registered in the information 2223 (S2112).
When the name is registered, the personal information search unit 2150 searches the address book 2070 by name, and adds personal information corresponding to the name to the object information 2223 (S2121). When the generation of all object information is completed, the input unit 2013 registers the object information 2223 in the database 2023. As a result, the object information 2223 corresponding to the still image 2210 (see FIG. 47) or the database 2023 is registered.

When all the still images 2210 and the object information 2223 are registered in the database 2023 for all the still images (S2016), the input unit 2013 ends the data input process.
FIG. 47 is a diagram showing a configuration example of the object information 2223 stored in the database 2023 by the data input process described above.

As shown in FIG. 47, the object information 2223 includes the coordinates of the center of the circle that the extracted object touches, the position information represented by the size of the radius, the file name of the corresponding still image, and the name of the object. In addition, it includes personal information, that is, an object attribute.
With the object information 2223 configured in this way, the position and name of each object are acquired for all the still images to be subjected to image processing, and image processing centered on the sex group of the object specified by the user Is possible.

FIG. 48 is a flowchart showing an algorithm until a scenario to be executed by each unit of the scenario generation unit 2033 is generated. This scenario generation process is executed after receiving an instruction from the user for the target range of the still image used on the short film, or after determining the music, style, and theme used on the short film.
First, the effect placement unit 2310 selects one effect that can execute a visual effect that matches a gender set designated by the user from a plurality of effects stored in the effect table 2233, and selects the selected time axis. It is arranged on the top (S2021). Specifically, the effect still, zoom, and the like are arranged in this order in accordance with the determined music time length. Thus, a scenario skeleton is formed.

  When the effects are placed on the time axis, the still image assignment unit 2323 searches for photos that match the sex group for all the effects placed on the time axis, determines the still images to be mapped, Parameters are set (S2211). Specifically, the attribute of the effect describes the feature quantity of a still image that should be mapped to the effect. Here, the number of objects = 2, the gender = male 1, and the female 1. At this time, it is not limited to a photograph showing two persons satisfying a specified sex group, but may be a photograph showing a plurality of persons including these two persons.

  When the mapping is completed, the parameter setting unit 2333 determines whether the object is male or female based on the object information corresponding to the mapped still image (S2213). If the object in the still image is male or female (Yes in S2213), a heart mark that is a character defined for male and female is assigned (S2214). If not (No in S2213), “◯”, which is a character defined for non-gender characters, is assigned (S2215).

When the character assignment ends, the scenario is completed, and the scenario generation unit 2033 registers the generated scenario in the database 2023 (S2026), and ends the scenario generation process.
When the short film is played back using the scenario 2273 configured as described above, the short film selection unit 2410 of the playback unit 2040 instructs to read the short film scenario and the music and still image defined there, and the short film is instructed. The playback unit 2420 plays back a short film using music and still images according to the instructed scenario, and outputs the short film to the display unit 2050. In this case, the short film playback unit 2420 generates a frame image defined by parameters corresponding to the effect from the still image for each effect, and plays back the short film. Then, when generating a frame image, a character specified by the sex group of the still image object is determined.

In the tenth embodiment, gender is used as the attribute of the object, but the scenario may be generated using other attributes such as age, date of birth, etc. The name, age, Other attributes such as date of birth may be displayed.
As described above, according to the invention of Embodiment 10, by using the name of the object in the still image obtained by the face authentication unit 2140 and linking with the address book, various attributes ( Gender, age, affiliation, hobby, etc.) can be added to the object information. Therefore, if such information is used, new entertainment properties can be found even with the same effect.

In the seventh to tenth embodiments described above, the case where the object is a human face has been described. However, the present invention is not limited to the case where the object is a human face, and the entire human figure, dog, cat, etc. Needless to say, it can be applied to all kinds of objects such as animals, plants such as apples and lemons, movable property such as desks and chairs, and real estate such as tourist information maps.
In the seventh embodiment, the still image allocating unit 2320 uses the object information registered in the database 2020 for each effect arranged by the effect arranging unit 2310 on the time axis. Although still images satisfying the number are allocated, the still image allocation unit 2320 in the present embodiment allocates still images that satisfy at least one of the type, color, shape, and number of objects required by the effect. Also good. In this case, the object information extraction unit 2120 may extract the color and shape of the object.

Further, the still image assigning unit 2320 assigns a still image that satisfies the image feature amount required by the effect (an image index indicating the hue, color spread, color position, brightness, brightness position, etc.). May be. In this case, the input unit 2013 may calculate a color histogram, a frequency histogram, or the like for each still image, and extract an image feature amount in advance.
(Embodiment 11)
Next, a case where a scenario, that is, a short film is generated by selecting an effect block will be described.

  In the first to tenth embodiments, a basic effect that visually shows one movement such as pan, still, and wipein is selected from the set of effects. In contrast to this, in the eleventh embodiment, it is configured such that an effect block is selected from a set of effect blocks obtained by collecting a plurality of one or more basic effect sequences. It is very different from the case.

The database stores a set of music, images, and effect blocks. In addition to the actual information, the music holds the music feature value and the image holds the image feature value as metadata.
As shown in FIG. 49, the set of effect blocks is a collection of a plurality of one or more basic effects. Each effect block is a combination of at least one basic effect that visually shows one movement, such as pan, still, and wipein, the same basic effect as this basic effect, or a basic effect that is different from this basic effect. is there. In a state where the effect block is included in the set (initial state), no parameters are determined when reproducing the effect block.

When configured with effect blocks in this way, the scenario generator
A scenario, that is, a short film can be generated by sequentially executing the following 1) to 4) based on either the music feature value or the image feature value.
1) Select effect blocks (plural) to be used to generate content.
2) Generate effect block parameters that determine content characteristics.
3) Arrange the effect blocks on the time axis to match the music.
4) Allocate (assign) an image that matches the effect block.

Here, an example of generating a scenario based on the music feature amount will be given.
Music features can be classified into two types: musical score information and emotional information. Musical information is information that can be acquired from the score, such as the tempo and change points. The tempo represents the speed of the rhythm of the entire music, and the change point is a point where the tune of the song changes or is characteristic of a percussion instrument, etc. Indicates the point where a special sound enters. Sensitive information, on the other hand, is information that people feel when they listen to the music, and the sensibility that they will feel when they hear the music is intense, dynamic, refreshing, simple, soft, etc. The sensitivity keywords are expressed by points based on a 10-level evaluation. It should be noted that the evaluation may be five or three stages instead of the ten-stage evaluation. On the other hand, an effect block is a permutation of one or more basic effects. In the initial state, only the permutation has been determined, and a length of time that is a parameter that defines all operations and appearances of an actual effect block. The start position, end position, hue, etc. are undefined.

The short film generation / playback apparatus has a plurality of effect blocks.
The basic effect visually shows one movement such as Pan, Slidein, Still, Wipein, and Focusin. The effect block defines a series of coherent movements such as one scene of a movie by arranging an arbitrary number of basic effects on the time axis.

For example, an effect block consisting of a sequence of four basic effects, Still → Still → Still → BoxOut, shows three pictures in a static state and then disappears while displaying the last one in a reduced size. It completes the scene.
Hereinafter, a configuration and an algorithm for automatically generating a short film (content) using an effect block and a photo as a material based on the music feature amount when music is selected will be described.

FIG. 51 is a block diagram showing a functional configuration of a short film generation / playback device 3000 according to Embodiment 11 of the present invention.
As shown in FIG. 51, the short film generation / playback device 3000 includes an effect block selection unit 3002, a parameter determination unit 3003, an effect block, in addition to the music feature amount, the image feature amount, and the effect block set described above. A block arrangement unit 3004, a photo assignment unit 3005, and a coordinate data generation unit 3006 are included. Here, only the database and the scenario generation unit are shown as the short film generation / playback device 3000.

FIG. 52 is a flowchart showing a short film generation process executed by each unit of the short film generation / reproduction device 3000.
The short film generating / reproducing apparatus 3000 first selects music to be reproduced as a short film BGM from the database (S3001). This selection method may be a method in which the short film generation / playback device 3000 provides an interface such as a GUI so that the user can select a favorite song by himself / herself, or the short film generation / playback device 3000 automatically selects from a database. It may be a method to elect. When the music selection is finished, the short film generation / playback device 3000 reads the music data and the data of the music feature amount from the database. If there is no music feature quantity in the database, the music feature quantity data may be extracted from the music data when the music data is read from the database.

  When the music data and the music feature value data are acquired, the effect block selection unit 3002 selects an effect block to be used for generating the current content from the music feature value from the set of effect blocks (see S3002, FIG. 49). For example, tempo information is given as one piece of tag information attached in advance to all effect blocks. Using this tempo information, the effect block selection unit 3002 selects an effect block that changes rapidly in the case of up-tempo music from the musical score information, and selects an effect block that changes little in the case of slow-tempo music.

  When the selection of the effect block ends, the parameter determination unit 3003 determines the parameter value of the selected effect block (see S3003, FIG. 50). More specifically, the parameter determination unit 3003 determines the color tone, frame pattern, background color, and the like of the entire content based on the sensibility information. Decide the search rule for the photo to be assigned. For example, if the emotional information is intense (2), dynamic (2), exhilarating (3), naive (8), soft (6), a scenario of a relaxed monochrome content In order to generate the parameters, parameters such as color tone = monochrome, background color = black, execution time = longer, motion = large, photo search rule = time series, and the like are determined.

A style is generated by the set of effect blocks selected in step S3002 and the set of parameters determined in step S3003. Here, this style is defined as an upper layer of the effect in this embodiment, but may be defined as the same layer.
When the parameters are determined, the effect block arrangement unit 3004 sequentially selects one of the plurality of effect blocks selected based on the musical score of the music feature value and arranges them on the time axis ( S3004). More specifically, the change of the effect is matched at the point where the tone of the song changes or the point where a characteristic sound such as a percussion instrument enters according to the change point information. For example, when the changing points are continuous, an effect block in which a still image to be displayed is continuously switched is arranged in a manner of still → still → still.

  When the arrangement of the effect blocks on the time axis is completed, the photo assignment unit 3005 assigns a photo (still image) to each effect block (S3005). More specifically, a photo in the database is searched based on the image feature amount required by the basic effect in the effect block and the search rule. For example, focusin is a basic effect that reduces the display area around the center point of a specific object in a photo. For example, a photograph that is 60% or more of the image is selected.

  When the photo assignment is completed, the coordinate data generation unit 3006 generates a coordinate value necessary for actually moving the basic effect from the image feature amount of the assigned photo (S3006). More specifically, FocusOut is taken as an example. FocusOut focuses on any point on the image, indicates the region of interest, and gradually enlarges and displays the graphic representing the focus, thereby displaying a still image. Although it is a basic effect that shows the whole, a coordinate value is required to represent the size of the first and last focus figures. If the figure is fixed to a square or circle, you can specify the center point and the size of the first and last radii. In order to acquire the coordinate value, the coordinate data generation unit 3006 uses the coordinate value from the object information of the image feature amount of the assigned image.

When the generation of the coordinate data is finished, the generation of the scenario is completed (S3007), and the short film generation / playback device 3000 stores the generated scenario, that is, the short film in the database, and ends the short film generation process.
In step S3002 and S3003, the music feature amount is used. However, an image feature amount can be used instead of the music feature amount. That is, in step S3002, the representative color extracted from the image is used as the image feature amount, and when the representative color is “red”, an effect block that changes drastically is selected, and when the representative color is “blue”, You may make it select the effect block with a loose change.

In step S3003, an effect block parameter is determined from the image feature amount. For example, when the representative color is “red”, dynamic content is generated, so that the color tone is normal, the background color is white, the execution time is short, the motion is large, the photo search rule is time series, and the like. May be.
Therefore, according to the invention according to the eleventh embodiment, the connection of the effects becomes smooth, and it is possible to produce a visual effect with high entertainment properties including further fun.

(Embodiment 12)
FIG. 53 is a functional block diagram showing the configuration of the short film generation / playback system according to the embodiment of the present invention. Note that portions similar to those in the above-described embodiments are given the same symbols, and description thereof is omitted.
In the short film generation / playback system of the twelfth embodiment, the playback unit 1103 shown in each of the above embodiments is configured as a short film playback device 1400 together with the display device 1002 separately from the short film generation / playback device. The point that the communication unit 1401 is provided in 1001e is different from each of the above embodiments, and the others are the same.

The communication unit 1401 of the short film generation / playback apparatus 1001e communicates with an external apparatus via the network 1402. On the other hand, the short film playback apparatus 1400 includes a short film selection unit 1110, a display unit 1111, a short film playback unit 1112, a storage unit 1403, and a communication unit 1404.
The short film selection unit 1110 provides a user with a GUI for selecting a short film to be reproduced. The short film playback unit 1112 reads the selected short film scenario and the still image and music defined there from the storage unit 1403, decodes the scenario, and plays back the short film. The communication unit 1404 performs communication with an external device via the network 1402. The storage unit 1403 temporarily holds data received from an external device via the communication unit 1404. The display unit 1111 is a device for displaying an image such as an LCD, and displays a reproduced short film.

Next, an operation when a short film is reproduced by the short film reproduction apparatus 1400 configured as described above will be described.
The short film selection unit 1110 acquires a list of short films registered in the database unit 1102 of the short film generation / playback device 1001e via the communication unit 1404 and the network 1402, and displays the list to the user. When the user selects a short film to be viewed, the short film selection unit 1110 transmits information including the name to the short film generation / playback device 1001e via the communication unit 1404, and the selected short film scenario, music, and stillness are selected. Request transmission of the image.

The communication unit 1401 of the short film generation / playback device 1001e reads the short film scenario, music, and still image requested by the short film playback device 1400 from the database unit 1102, and transmits them to the short film playback device 1400.
The communication unit 1404 of the short film playback device 1400 that has received the requested short film scenario, music, and still image stores the scenario, music, and still image in the storage unit 1403. Next, the short film playback unit 1112 reads the scenario, still image, and music of the selected short film from the storage unit 1403, decodes the scenario, plays back the short film, and outputs it to the display unit 1111.

  As described above, by configuring the playback unit 1103 as the short film playback device 1400 together with the display device 1002 separately from the short film generation / playback device, the user can carry only the short film playback device 1400. Further, the short film playback device 1400 can be configured as a mobile phone and can be easily carried. As a result, it can be shown to people who have met the short film created at home, and the joy and joy will increase dramatically.

(Embodiment 13)
Next, another embodiment in the case where a short film is reproduced by the short film production / reproduction device configured as in each of the above-described embodiments 1 to 12 will be described.
FIG. 54 is a block diagram showing the configuration of another embodiment of the short film generating / reproducing apparatus according to the present invention. In addition, about the part similar to said each Embodiment 1-12, the same symbol is attached | subjected and description is abbreviate | omitted.

The short film generation / playback device 1001f is connected to a television 1020 as a display device, and includes a remote control receiving unit (not shown) that receives an operation signal of the remote control 1004.
The reproduction unit 1103 of the short film generation / reproduction device 1001f modulates the reproduced short film into an RF signal and outputs the RF signal to the television 1020. Here, the RF signal is a signal modulated in the frequency band of the television channel. Usually, a U / VHF band is used as a frequency band and corresponds to a television channel (for example, 5, 7, etc.).

The television 1020 includes a channel selection unit 1021, a display unit 1022, and a remote control reception unit (not shown) that receives operation signals from the remote control 1004, and an antenna 1003 is connected to the channel selection unit 1021.
The channel selection unit 1021 includes a tuner, a demodulation circuit, and the like (not shown), receives the RF signal input from the short film generation / playback device 1001f and the signal from the antenna 1003, and selects and displays a desired frequency. Output to the unit 1022. The display unit 1022 is a device for displaying an image such as an LCD or a cathode ray tube, and displays a reproduced short film.

  The remote control 1004 includes a menu button 1041 and a plurality of channel buttons 1042. The user performs an operation with the remote controller 1004 and selects a short film theme with the menu button 1041. An RF signal input from the short film generation / playback apparatus 1001f is assigned to a predetermined channel (for example, 5) of the channel button 1042, and the channel button 1042 is used to select a short film and a normal television program. Do.

An operation when a short film is reproduced by the short film generation / reproduction device 1001f configured as described above will be described.
The user presses the channel (for example, 5) to which the RF signal input from the short film generation / playback apparatus 1001f is assigned with the channel button 1042 of the remote controller 1004. In response to this, the short film selection unit 1110 of the playback unit 1103 acquires the theme registered in the database unit 1102, modulates it as an RF signal as a theme selection screen, and outputs it to the television 1020. This theme selection will be described in detail later.

  The user selects a theme to be viewed with the menu button 1041 from the theme selection screen displayed on the display unit 1022. In response to this, the short film selection unit 1110 of the playback unit 1103 instructs to read a scenario of a short film corresponding to the selected theme and music and still images defined there. The short film playback unit 1112 decodes the scenario read from the database 1109, plays back the short film using the read music and still image, modulates the RF film, and outputs the RF signal to the television 1020.

Next, the theme selection will be described.
FIG. 55 is a display image diagram of the theme selection screen, and FIG. 56 is an image diagram of the user selection screen.
The theme selection, there is a manual selection and automatic selection, the user can select either in advance.

  Manual selection includes a case where a theme (default setting) preset by the user is used and a case where the user selects each time. Examples of default settings include fixed themes such as the previous theme and frequently used themes. On the other hand, when the user selects each time, selection is performed by operating the menu button 1041 from a plurality of themes presented as shown in FIG. 55, for example. As the theme, as described in the first embodiment, for example, there are themes such as “Nearly of last year”, “Recent events”, “Christmas” and “Athletic meet” as events.

  The automatic selection includes a case where the theme is changed according to the date and time, and a case where the theme is changed depending on the viewer. When changing the theme according to the date and time, for example, the corresponding theme is selected on the day of a regular event (birthday, Christmas, etc.). On the other hand, when changing the theme depending on the viewer, for example, the user name is selected on the user selection screen as shown in FIG. 56, and playback is performed mainly on the short film generated by the selected user or the short film in which the user is reflected. To do.

  As described above, since the playback output from the short film generation / playback apparatus 1001f is assigned to a predetermined channel, there is no distinction between the TV program and the short film in terms of user operation, and the short film can be viewed in substantially the same procedure as the TV operation. can do. Also, TV programs and short films can be selected and viewed seamlessly in the sense of so-called zapping (there is no clear purpose, and frequently switches channels to find interesting programs), saving user operations. it can.

  The above description of the operation is an example in which the user selects a theme every time by manual selection. In other cases, if the user presses the channel to which the RF signal from the short film generation / playback apparatus 1001f is assigned using the channel button 1042, the short film playback unit 1112 immediately plays back the short film, and the RF signal And output to the television 1020.

It is also possible to switch the short film being played each time the channel button assigned to the short film is pressed. In this case, the user can enjoy the short film without getting bored.
In addition, the display unit 1022 of the television 1020 can be configured as a parent-child screen so that playback of a short film can be displayed on, for example, a child screen, or vice versa.

(Embodiment 14)
FIG. 57 is a block diagram showing a configuration of another embodiment of the short film generating / reproducing apparatus according to the present invention. The same parts as those in the thirteenth embodiment are denoted by the same reference numerals, and the description thereof is omitted.
The fourteenth embodiment is different from the thirteenth embodiment in that a plurality of RF signal outputs from the short film generating / reproducing device 1001g are provided, and the other is the same.

The reproduction unit 1103 of the short film generation / reproduction device 1001g modulates the reproduced short film into an RF signal and outputs the RF signal to the television 1020, and a plurality of RF signal outputs are provided.
The channel selection unit 1021 of the television 1020 receives a plurality of RF signals input from the short film generation / playback device 1001g and the signal from the antenna 1003, selects a desired frequency, and outputs the selected frequency to the display unit 1022.

  A plurality of RF signals input from the short film generation / playback apparatus 1001g are assigned to each of a plurality of predetermined channels (for example, 5, 7, etc.) to the channel button 1042 of the remote controller 1004. That is, short films corresponding to different themes are assigned to different channels, for example. The theme selection is performed with the menu button 1041 for each channel.

As described above, since a plurality of RF signal outputs from the short film generating / reproducing apparatus 1001g are provided, the short film can be viewed with a sense of operation closer to that of a television where the short film is switched when the channel is changed.
Note that each time the currently selected channel button 1042 is pressed, it is possible to switch the short film being played while keeping the theme as it is. That is, the channel button 1042 selects a theme, and the short film switches each time the same button is pressed. In this case, the user can enjoy the short film without getting bored.

(Embodiment 15)
FIG. 58 is a block diagram showing a configuration of another embodiment of the short film generating / reproducing apparatus according to the present invention. The same parts as those in the thirteenth embodiment are denoted by the same reference numerals, and the description thereof is omitted.
The fifteenth embodiment is different from the thirteenth embodiment in that the short film reproduction output from the short film generating / reproducing apparatus 1001h is performed by a video signal, and the other is the same.

The reproduction unit 1103 of the short film generation / reproduction device 1001h converts the reproduced short film into a video signal and outputs the video signal to the television 1020.
The television 1020 receives the video signal input from the short film generation / playback device 1001h at a video input terminal (not shown).
The remote control 1004 includes a menu button 1041 and a short film button 1043. The user performs an operation with the remote controller 1004 and selects a short film theme with the menu button 1041. A short film is viewed with the short film button 1043. That is, when the short film button 1043 is pressed, the television 1020 selects and displays the video signal input from the short film generation / playback device 1001h. Here, it is also possible to switch the short film being played each time the short film button 1043 is pressed. In this case, the user can enjoy the short film without getting bored.

As described above, since the short film reproduction output from the short film generation / reproduction device 1001h is performed by the video signal, the short film can be viewed as if the video is operated.
In the thirteenth to fifteenth embodiments, the short film generating / reproducing apparatuses 1001e to 1001h and the television 1020 are separately configured. However, the present invention is not limited to this, and for example, all are integrally configured. It doesn't matter.

Further, the short film generation / reproduction devices 1001a to 1001h and 2001 to 2004 described in the above embodiments can be realized as, for example, a home server, a personal computer, a mobile phone device, a portable information terminal, a car navigation device, or the like. is there.
In each of the above embodiments, the still image is input from the medium on which the still image is recorded. However, the still image captured by the user's own device such as a mobile phone with a camera or attached to an e-mail is sent. It may be a still image coming or a still image received via a network. That is, it goes without saying that an acquired still image is included in the scope of the present invention regardless of the acquisition method.

  Furthermore, in the above embodiment, the effects and the effect blocks are arranged based on the time length of the music to be BGM. However, it goes without saying that this time length may be the time length specified by the user on the GUI or the like. Nor.

  The short film generation / playback apparatus of the present invention generates a short video with improved entertainment characteristics from a huge amount of still images and desired music held by a user in a personal computer or a portable information terminal device such as a mobile phone. Suitable for playing.

It is a block diagram which shows the structure of one Embodiment of the short film production | generation reproduction | regeneration apparatus based on this invention. It is the conceptual diagram which showed the relationship between the style and scenario in the said short film production | generation reproduction | regeneration apparatus. It is the figure which showed typically an example of the effect in the said short film production | generation reproduction | regeneration apparatus with a storyboard. It is a flowchart which shows the algorithm at the time of the still image input in the input part of the said short film production | generation reproduction | regeneration apparatus. It is a flowchart which shows the algorithm at the time of the production | generation of the scenario in the short film production | generation part of the said short film production | generation reproduction | regeneration apparatus. It is a flowchart which shows the algorithm which determines the arrangement | sequence of the effect in the said short film production | generation part. It is a conceptual diagram which shows the mode of the arrangement | sequence of the effect with respect to the music time before and after time adjustment is performed in the said short film production | generation part. It is a block diagram which shows the structure of other embodiment of the short film production | generation reproduction | regeneration apparatus based on this invention. It is a flowchart which shows the algorithm at the time of the production | generation of the scenario in the short film production | generation part of the said short film production | generation reproduction | regeneration apparatus. It is a block diagram which shows the structure of other embodiment of the short film production | generation reproduction | regeneration apparatus based on this invention. It is a flowchart which shows the algorithm at the time of the production | generation of the scenario in the short film production | generation part of the said short film production | generation reproduction | regeneration apparatus. It is a block diagram which shows the structure of other embodiment of the short film production | generation reproduction | regeneration apparatus based on this invention. It is a flowchart which shows the algorithm at the time of the production | generation of the scenario in the short film production | generation part of the said short film production | generation reproduction | regeneration apparatus. It is a flowchart which shows the algorithm when the scenario production | generation part of the said short film production | generation reproduction | regeneration apparatus produces | generates a visual effect by object information. It is a conceptual diagram which shows the assignment of the effect with respect to the still image arranged in the time axis | shaft in the scenario production | generation part of the said short film production | generation reproduction | regeneration apparatus. It is a flowchart which shows the algorithm which assigns an effect with respect to a still image in the scenario production | generation part of the said short film production | generation reproduction | regeneration apparatus. It is a functional block diagram which shows the structure of the short film production | generation reproduction | regeneration apparatus which concerns on Embodiment 7 of this invention. It is a figure which shows the structural example of the object information registered into the database. It is a figure which shows the structural example of an effect table. It is a flowchart which shows the algorithm at the time of the still image input in an input part. It is the flowchart which showed the algorithm until it produces | generates the scenario which each part of a scenario production | generation part performs. It is a flowchart for demonstrating the algorithm in which the parameter setting part determines the variation and parameter of an effect. It is a figure which shows four variations, vertical, horizontal, diagonal, and four corners in a slide-out. It is a figure explaining the rotation coordinate system: sot coordinate system which rotated the original coordinate system (x-o-y) (theta) degrees (0 degree <theta <90 degree). It is a figure which shows the structural example of the scenario registered into the database. It is a block diagram which shows the function structure of the short film production | generation reproduction | regeneration apparatus which concerns on Embodiment 8 of this invention. A storyboard showing an example of a montage. It is a figure which shows the structural example of an effect table. It is a flowchart which shows the algorithm which produces | generates the object information which implement | achieves the effect of a montage style. It is a figure which shows the structural example of object information. It is the flowchart which showed the algorithm until it produces | generates the scenario which each part of a scenario production | generation part performs. It is a figure which shows the structural example of the scenario registered into the database. It is a storyboard which shows the modification in a montage. It is a block diagram which shows the function structure of the short film production | generation reproduction | regeneration apparatus which concerns on Embodiment 9 of this invention. It is a storyboard showing an example of an effect (spotlight) focused on a specific person. It is a figure which shows the structural example of an effect table. It is a flowchart which shows the algorithm which produces | generates the object information which implement | achieves the effect which shines a spotlight on one person. It is a figure which shows the structural example of object information. It is the flowchart which showed the algorithm until it produces | generates the scenario which each part of a scenario production | generation part performs. It is a figure which shows the structural example of the scenario registered into the database. It is a block diagram which shows the function structure of the short film production | generation reproduction | regeneration apparatus which concerns on Embodiment 10 of this invention. It is a figure which shows the structural example of an address book. It is a storyboard showing an example of an effect using gender (male-female). It is a storyboard showing an example of an effect using gender (male-male). It is a figure which shows the structural example of an effect table. It is a flowchart which shows the algorithm which produces | generates the object information which implement | achieves the effect which changes the character displayed according to the combination of sex. It is a figure which shows the structural example of the object information stored in the database. It is the flowchart which showed the algorithm until it produces | generates the scenario which each part of a scenario production | generation part performs. It is a figure which shows the mode of the collection of an effect block, and the selection of an effect block. It is a figure which shows the mode of the parameter production | generation with respect to the selected effect block, and the setting of the produced | generated parameter value. It is a block diagram which shows the function structure of the short film production | generation reproduction | regeneration apparatus based on Embodiment 11 of this invention. It is a flowchart which shows the short film production | generation process which each part of a short film production | generation reproduction | regeneration apparatus performs. It is a block diagram which shows the structure of one Embodiment of the short film production | generation reproduction | regeneration system based on this invention. It is a block diagram which shows the structure of other embodiment of the short film production | generation reproduction | regeneration apparatus based on this invention. It is a display image figure of the theme selection screen in a display part. It is a display image figure of the user selection screen in a display part. It is a block diagram which shows the structure of other embodiment of the short film production | generation reproduction | regeneration apparatus based on this invention. It is a block diagram which shows the structure of other embodiment of the short film production | generation reproduction | regeneration apparatus based on this invention.

Explanation of symbols

1001a to 1001h,
2001-2004, 3000 Short film generation / reproduction device 1002 Display device 1020 Television 1022, 1111, 2050 Display unit 1030, 1117, 2030-2033 Scenario generation unit 1043 Short film button 1101, 2010-2013 Input unit 1103, 2040 Playback unit 1105a- DESCRIPTION OF SYMBOLS 1105d Short film production | generation part 1106 Still image input part 1107 Image feature-value extraction part 1108 In-image object extraction part 1109,2020-2023 Database 1110,2410 Short film selection part 1112,2420 Short film reproduction part 1113 Image selection part 1114 Image feature-value Reading unit 1115 Image-style feature value conversion unit 1116 Style determination unit 1118 Image-music feature value conversion unit 1119 Sound Determination unit 1120 Music selection unit 1121 Music feature amount reading unit 1122 Music-image feature amount conversion unit 1123 Image determination unit 1124 Music-style feature amount conversion unit 1125 Style selection unit 1126 Style feature amount reading unit 1127 Style-image feature amount conversion unit 1128 Style-music feature value conversion unit 1129 Theme selection unit 1130 Theme feature value reading unit 1131 Theme-image feature value conversion unit 1132 Theme-style feature value conversion unit 1133 Theme-music feature value conversion unit 1202 Image feature value extraction unit 1400 Short Film playback devices 1401 and 1404 Communication unit 1603 Image-style feature value conversion unit 1604 Style determination unit 1605 Image-music feature value conversion unit 1606 Music determination unit 2070 Address book 2110 Still image selection unit 2120 Object Information extraction unit 2130 face feature point extraction unit 2140 face authentication unit 2150 personal information search unit 2210 still image 2213 position information 2220 to 2223 object information 2230 to 2233 effect table 2240 music 2250 style 2260 theme 2270 to 2273 scenario 2280 face information 2310 effect Arrangement unit 2320 to 2323 Still image allocation unit 2330 to 2333 Parameter setting unit 3002 Effect block selection unit 3003 Parameter determination unit 3004 Effect block arrangement unit 3005 Photo allocation unit 3006 Coordinate data generation unit

Claims (7)

A short film generation / playback device that generates and plays back video using at least one still image,
An input means for inputting a still image input from the outside;
Storing said still image, a predetermined number that defines how to play with any visual effects one or more still images to be effect, and the effects table showing characteristics of an image each effect requires Database means to
Based on the predetermined style and music features of the image scenario, and a scenario generation unit for generating a scenario of the image,
The input means includes
Object information extraction means for extracting an object included in the still image every time a still image is input from the outside, and registering object information indicating the feature of the extracted object in association with the still image in the database means Have
The scenario generation means includes:
On the basis of the predetermined style and music feature, the selected one of the single effect from the Rue Effects table stored in the database means, effects arranged sequentially placing the selected effect on the time axis Means,
Each effect the effect positioning means are arranged on the time axis, still based on the object information and effects tables registered in said database means, associated with the characteristics of objects that satisfy the feature of the image to which the effect is required Still image assigning means for assigning images,
Parameter setting means for generating a scenario by describing a parameter indicating processing for an object in the still image that matches the effect arranged on the time axis by the effect arrangement means, and registers the generated scenario in the database means The short film production | generation reproduction | regeneration apparatus characterized by having. Features of the object, the short film generation reproduction apparatus according to claim 1, wherein the type of object, the color, that is at least one of the shape and number. The input means further comprises a feature point extracting means for extracting a feature point indicating a characteristic part included in the object and registering the extracted feature point in the object information,
Said parameter setting means, short film generation reproduction apparatus according to claim 1, wherein the generating the scenario by describing a parameter indicating a processing of a position of a feature point included in the object. The database means further stores personal authentication face information for specifying a personal face,
The input means further authenticates the name of the object using the personal authentication face information when the object extracted by the object information extraction means is a human face, and uses the authenticated object name as the object information. Having face recognition means to register,
Said parameter setting means, short film generation reproduction apparatus according to claim 1, wherein the generating the scenario by describing the parameters indicating the processing for the object identified by name. The short film generation / playback device further includes personal information storage means for storing personal information in which a person's name is associated with an attribute of the person,
The input means further includes a personal information search means for searching a personal attribute corresponding to the name of the object authenticated by the face authentication means from the personal information, and registering the searched personal attribute in the object information,
The short film generation / playback apparatus according to claim 4 , wherein the parameter setting unit generates a scenario by describing a parameter indicating processing for an object specified by an individual attribute. Using at least one still picture, a short film generation reproduction method for generating an image, play,
An input step for inputting a still image input from the outside;
Stores the still image, a predetermined number of effects defining what kind of visual effect is used to reproduce one or more target still images, and an effect table indicating image characteristics required by each effect Using database means to
Based on the predetermined style and music features of video scenarios, and a scenario generation step of generating a scenario of the image,
The input step includes
An object information extraction step of extracting an object included in the still image every time a still image is input from the outside, and registering object information indicating the characteristics of the extracted object in association with the still image in the database unit Including
The scenario generation step includes:
Effect placement step of selecting one effect from the effect table stored in the database means one by one based on the predetermined style and music characteristics, and sequentially placing the selected effect on the time axis When,
For each of the effects which are arranged on the time axis by the effect placement step, still based on the object information and effects tables registered in said database means, associated with the characteristics of objects that satisfy the characteristics of the image to which the effect is required A still image assignment step for assigning images,
It matches the placed on the time axis by the effect placement step effect, parameter settings and generates a scenario by describing the parameters indicating the processing for the object in the still image, and registers the generated scenario in the database means A method for producing and reproducing a short film, comprising: a step. A program for generating and playing back video using at least one still image,
An input step for inputting a still image input from the outside;
Stores the still image, a predetermined number of effects defining what kind of visual effect is used to reproduce one or more target still images, and an effect table indicating image characteristics required by each effect Using database means to
Based on the predetermined style and music features of video scenarios, and a scenario generation step of generating a scenario of the image,
The input step includes
An object information extraction step of extracting an object included in the still image every time a still image is input from the outside, and registering object information indicating the characteristics of the extracted object in association with the still image in the database unit Including
The scenario generation step includes:
Effect placement step of selecting one effect from the effect table stored in the database means one by one based on the predetermined style and music characteristics, and sequentially placing the selected effect on the time axis When,
For each of the effects which are arranged on the time axis by the effect placement step, still based on the object information and effects tables registered in said database means, associated with the characteristics of objects that satisfy the characteristics of the image to which the effect is required A still image assignment step for assigning images,
It matches the placed on the time axis by the effect placement step effect, parameter settings and generates a scenario by describing the parameters indicating the processing for the object in the still image, and registers the generated scenario in the database means a program characterized by executing the steps on a computer.

Images