JP7456232B2 - Photo movie generation system, photo movie generation device, user terminal, photo movie generation method, and program - Google Patents

Description

The present invention relates to a photo movie generation system and the like that generate and provide photo movies.

Among cloud photo storage services, a service is provided that automatically creates a photo movie using photos stored in cloud storage (Non-Patent Document 1). For example, in the cloud photo storage service of Non-Patent Document 1, when a user selects the photos or videos that he or she wants to make into a movie from the images saved in the cloud storage, and selects the desired theme from among several themes prepared in advance, , the movie is automatically generated on the service side.

Furthermore, a method has been reported for adding music that matches the impression of a captured video (Non-Patent Document 2). Non-Patent Document 2 describes how to extract features of a captured video (video features), extract features of music (music features), calculate the relationship between the video and the impression of melody/rhythm, and A method has been proposed to generate music that matches the impression of the user.

On the other hand, Patent Document 1 discloses an expression acquisition unit that acquires an acoustic expression representing an impression or timing in arbitrary acoustic data, an effect imparting unit that imparts a visual effect, etc. to an image, etc. according to the acoustic expression, and A sound reproducing device is described that includes a display that reproduces images and the like to which effects have been added. This makes it possible to acquire the acoustic representation of audio content such as commercially available music data, and to add visual effects and the like according to the audio expression.

Japanese Patent Application Publication No. 2013-114088

Google (registered trademark) Photos, "Create a movie", [online], [Searched on March 23, 2020], Internet <URL: https://photos.google.com/movies/create> Yurina Shimizu and 4 others, "Automatic material selection for video BGM based on impression estimation from video features", NICOGRAPH 2016, pp177-184

However, with the movie creation function of Non-Patent Document 1 mentioned above, the user can also select the music (BGM) to be added to the photo movie, but the music that can be selected is categorized by default category (dramatic, electronic, rockin'). ', upbeat,..., etc.), or DRM (Digital
Only songs that are not protected by Rights Management (digital rights management) are included. Furthermore, the proposed method in Non-Patent Document 2 is a method of generating music that matches the impression of a video. Therefore, in either method, music obtained from a commercially available CD or a music distribution service cannot be used as the BGM of a photo movie. On the other hand, the method disclosed in Patent Document 1 extracts acoustic expressions from commercially available music data and makes it possible to add visual effects according to the acoustic expressions. Since the content is fixed, the user may become bored after viewing the content several times.

The present invention has been made in view of the above problems, and it is an object of the present invention to provide a photo movie generation system, etc. that allows users to play their favorite songs with photo movies, and that allows users to enjoy listening to music. purpose.

A first invention for solving the above-mentioned problems is a photo movie generation system in which a server and a user terminal are communicatively connected via a network, and the user terminal specifies music to be used in the photo movie. music designation means that accepts and transmits the sound source data of the designated music or a music ID that is identification information of the music to the server; and music designation means that receives designation of a plurality of images to be used as image materials of a photo movie and transmits the designated image data. Alternatively, the server includes image specifying means for transmitting image specifying information to the server, and playback means for receiving and playing the photo movie transmitted from the server, and the server is configured to play the music specified in the user terminal. An acoustic feature data acquisition means for acquiring acoustic feature data, an image acquisition means for acquiring images to be used in the photo movie, and an image acquired by the image acquisition means are used to create a song based on the acoustic feature data. a photo movie generating means for generating a photo movie whose display changes according to a change in the image data and transmitting the generated photo movie to the user terminal; an effect template in which at least some of the effects are associated with a number of used images of 2 or more; a storage means for storing in advance an effect correspondence table in which a plurality of patterns of effect application examples are defined, the table associating acoustic feature amount data with the effect templates suitable therefor ; The storage means stores movie configuration information that is information constituting the photo movie and stores the number of used images acquired from the effect template and applied effects acquired from the effect correspondence table, based on the song ID. The photo movie generating means refers to the movie structure information stored in association with the specified song ID, and if the movie structure information is not stored in the storage means, the photo movie generating means An arbitrary pattern is read from the effect correspondence table, and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table, and a photo movie is created according to the effect correspondence table of the read pattern. This is a photo movie generation system that is characterized by generating a photo movie.

According to the first invention, when the server acquires acoustic feature data of a music specified in a user terminal and acquires an image to be used in a photo movie, the server uses the acquired image and uses the acoustic feature data based on the acoustic feature data. A photo movie whose display changes according to changes in music is generated and sent to the user terminal. The user terminal receives and generates a photo movie. As a result, a photo movie whose display changes according to changes in the song can be generated for a song specified by the user and provided to the user terminal, allowing the user to view his or her favorite song along with the photo movie. Therefore, the user can enjoy listening to music even more.

In the photo movie generation system of the first invention, the server includes a storage means for storing movie configuration information, which is information that configures the photo movie, in association with the song ID, and the photo movie generation means references the movie configuration information stored in association with a specified song ID and generates a photo movie with a different configuration from the movie configuration information stored in the storage means. This allows different movie representations to be generated for the same song and provided to the user terminal, providing a variety of entertainment to the user.

Further, the storage means of the server stores the acoustic feature data of the song in association with the song ID, and the acoustic feature data acquisition means obtains the acoustic feature data stored in the storage means. It is desirable that the system be characterized by: As a result, acoustic analysis can be omitted for songs that have been analyzed once, so a photo movie can be provided in a short time. Further, the acoustic feature data acquisition means of the server may obtain acoustic feature data of a music piece specified by the user terminal from a music distribution server communicatively connected via a network. As a result, acoustic feature data can be obtained without performing acoustic analysis processing on the server side, and the load on the server can be reduced.

Further, the server further includes an acoustic analysis unit that performs an acoustic analysis process on the song to obtain acoustic feature data, and the acoustic feature data acquisition unit obtains the acoustic feature data obtained by the acoustic analysis unit. You can do it like this. At this time, the server may acquire the sound source data of the song from a music distribution server communicatively connected via a network, or from the user terminal. Thereby, a photo movie matching the user's desired song can be generated.

Further, the photo movie generation means of the server generates the photo movie without adding a sound source, and outputs the photo movie by linking it with the song ID of the song, and the playback means of the user terminal generates the photo movie without adding a sound source, and the playback means of the user terminal generates the photo movie without adding a sound source. It is desirable that the sound source data of the song corresponding to the music ID is acquired, and the acquired sound source data and the photo movie are played back in synchronization. As a result, a photo movie with a smaller file size than a photo movie with music can be generated, and the storage capacity for storing the photo movie can be saved.

The server further includes image material generation means for generating image materials from the images acquired by the image acquisition means, and the photo movie generation means uses the image materials generated by the image material generation means. It is desirable to generate the photo movie.
Preferably, the user terminal further includes an image transmitting means for transmitting images to be used in the photo movie to the server, and the image acquiring means of the server receives the images transmitted from the user terminal. Thereby, a photo movie can be generated using an image specified by the user and an image material generated from the image.

A second invention uses an image acquisition means for acquiring an image, an acoustic feature data acquisition means for acquiring acoustic feature data of a song, and an image acquired by the image acquisition means, and uses the image acquisition means to acquire the acoustic feature data. a photo movie generating means for generating a photo movie whose display changes according to changes in music based on the music; an effect template in which at least some of the effects are associated with a number of used images of 2 or more; and the acoustic feature amount data. storage means for storing in advance an effect correspondence table in which a plurality of patterns of effect application examples are defined, the storage means being a table in which the effect templates are associated with the effect templates suitable for the photo movie; The movie configuration information, which is the information that constitutes the music file, and stores the number of used images obtained from the effect template and the applied effects obtained from the effect correspondence table, is linked to the song ID, which is the identification information of the song. The photo movie generating means refers to the movie structure information stored in association with the specified song ID, and if the movie structure information is not stored in the storage means, the photo movie generating means stores the movie structure information corresponding to the effect. An arbitrary pattern is read from the table, and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table and a photo movie is generated according to the effect correspondence table of the read pattern. This is a photo movie generation device characterized by the following.

According to the second invention, it is possible to generate a photo movie whose display changes according to changes in the music based on the acoustic feature data of the music.

A third invention provides an image acquisition means for acquiring an image, a sound source data acquisition means for acquiring sound source data of a song, an acoustic analysis means for analyzing the sound source data to acquire acoustic feature data, and the image acquisition means. a photo movie generating means for generating a photo movie whose display changes according to changes in music based on the acoustic feature amount data using the images acquired by the means; Preliminarily storing an effect correspondence table in which a plurality of patterns of effect application examples are defined, which is a table in which associated effect templates are associated with each other, and the acoustic feature amount data is associated with the effect template suitable therefor. storage means, the storage means stores information constituting the photo movie, including the number of used images obtained from the effect template and applied effects obtained from the effect correspondence table. Movie configuration information is stored in association with a song ID, which is identification information of a song, and the photo movie generating means refers to the movie configuration information stored in association with the specified song ID, and generates the movie configuration information. is not stored in the storage means, an arbitrary pattern is read from the effect correspondence table, and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table. This photo movie generation device is characterized in that it reads out and generates a photo movie according to an effect correspondence table of the read patterns .

According to the third invention, it is possible to perform acoustic analysis on the acquired sound source data to obtain acoustic feature data, and to generate a photo movie whose display changes according to changes in the song based on the acoustic feature data. becomes.

A fourth invention is a user terminal that can be communicatively connected to the server that is the photo movie generation device of the second or third invention via a network, which accepts a designation of a song, and receives sound source data of the specified song or music designation means for transmitting a music ID, which is identification information of a music, to the server ; image designation means for receiving designation of a plurality of images and transmitting designated image data or image designation information to the server; and the server. The user terminal is characterized in that it is provided with a reproduction means for receiving and reproducing a photo movie whose display changes according to changes in the music and which displays a plurality of images , transmitted from the user terminal.

According to the fourth invention, when a user terminal specifies a song and sends the music source data of the song or the song ID, which is identification information of the song, to the server, a photo movie whose display changes according to changes in the song is displayed. It becomes possible to receive it from the server and play it back.

A fifth invention is a photo movie generation method in a photo movie generation system in which a server and a user terminal are communicatively connected via a network , wherein the server has at least some effects that have a number of used images of 2 or more. storing in advance an effect correspondence table in which a plurality of patterns of effect application examples are defined, which is a table in which associated effect templates, acoustic feature amount data, and the effect templates suitable therefor are associated; a step in which the user terminal accepts a designation of a song to be used in the photo movie, and transmits a song ID, which is the sound source data of the specified song or identification information of the song, to the server; a step of accepting designations of a plurality of images to be used as image materials of the user, and transmitting the designated image data or designation information of the images to the server; a step of the server acquiring the images; a step of acquiring acoustic feature data of a song specified in the terminal; the server using the image and generating a photo movie whose display changes according to changes in the song based on the acoustic feature data; the user terminal receives the photo movie transmitted from the server and plays the photo movie, and the server uses the image and reproduces the acoustic feature data. In the step of generating a photo movie whose display changes according to changes in the music based on the information and transmitting it to the user terminal, the generated photo movie includes information that constitutes the photo movie and that is acquired from the effect template. Movie configuration information in which the number of used images and applied effects obtained from the effect correspondence table are stored is stored in the server in association with the song ID, and when generating a photo movie, If the movie configuration information is not stored in the server, an arbitrary pattern is read from the effect correspondence table, and if the movie configuration information is already stored in the This photo movie generation method is characterized in that, if stored in the server, an unused pattern is read from the effect correspondence table, and a photo movie is generated according to the effect correspondence table of the read pattern .

According to the fifth invention, when the server acquires the acoustic feature data of the music specified in the user terminal and acquires the image to be used in the photo movie, the server uses the acquired image and uses the acoustic feature data based on the acoustic feature data. A photo movie whose display changes according to changes in music is generated and sent to the user terminal. The user terminal receives and generates a photo movie. As a result, a photo movie whose display changes according to changes in the song can be generated for a song specified by the user and provided to the user terminal, allowing the user to view his or her favorite song along with the photo movie. Therefore, the user can enjoy listening to music even more.

A sixth aspect of the present invention provides a computer with image acquisition means for acquiring an image, acoustic feature data acquisition means for acquiring acoustic feature data of a song, and an image acquired by the image acquisition means; A program for causing the computer to function as a photo movie generating means for generating a photo movie whose display changes according to changes in music based on the above, wherein at least some of the effects correspond to the number of used images of 2 or more. A memory that stores in advance an effect correspondence table in which a plurality of patterns of effect application examples are defined, the table associating the attached effect template, the acoustic feature amount data, and the effect template suitable therefor. and the storage means stores information constituting the photo movie, including the number of used images obtained from the effect template and applied effects obtained from the effect correspondence table. Movie configuration information is stored in association with a song ID, which is identification information of a song, and the photo movie generating means refers to the movie configuration information stored in association with the specified song ID, and generates the movie configuration information. is not stored in the storage means, an arbitrary pattern is read from the effect correspondence table, and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table. This program is characterized in that it reads out a pattern and generates a photo movie according to an effect correspondence table of the read pattern .

According to the sixth invention, it is possible to cause a computer to function as the photo movie generating device according to the second invention.

A seventh aspect of the invention includes a computer, an image acquisition means for acquiring an image, a sound source data acquisition means for acquiring sound source data of a song, an acoustic analysis means for analyzing the sound source data to acquire acoustic feature data, and the image acquisition means. A program for causing the computer to function as a photo movie generating means for generating a photo movie whose display changes according to changes in music based on the acoustic feature data using images acquired by the means , the program comprising: A table that associates an effect template in which at least some of the effects are associated with a number of used images of 2 or more, and the acoustic feature amount data and the effect template that is suitable for the acoustic feature amount data, the table including the effect template that is associated with the number of images used, wherein at least some of the effects are associated with the number of images to be used, and the acoustic feature amount data and the effect template that is suitable for the effect template; A storage means for storing in advance an effect correspondence table in which an example is defined; The photo movie generating means stores movie configuration information storing applied effects obtained from the effect correspondence table in association with a song ID that is identification information of a song, and the photo movie generating means stores movie configuration information in which applied effects obtained from the effect correspondence table are stored, and the photo movie generating means associates it with a specified song ID. The stored movie configuration information is referred to, and if the movie configuration information is not stored in the storage means, an arbitrary pattern is read from the effect correspondence table, and if the movie configuration information is already stored in the storage means. If there is an effect correspondence table, the program reads an unused pattern from the effect correspondence table and generates a photo movie according to the effect correspondence table of the read pattern .

According to the seventh invention, it is possible to cause a computer to function as the photo movie generation device according to the third invention.

An eighth invention provides a computer that can be communicatively connected via a network to a server that executes the program of the sixth or seventh invention , receives a designation of a music piece, and transmits sound source data of the designated music piece or identification information of the music piece. music designation means for transmitting a certain music ID to the server; image designation means for accepting designation of a plurality of images and transmitting designated image data or image designation information to the server; and the music transmitted from the server. This is a program for functioning as a reproduction means for receiving and reproducing a photo movie whose display changes according to changes in the image and which displays a plurality of images .

The eighth invention allows a computer to function as the user terminal of the first and fourth inventions.

According to the present invention, it is possible to provide a photo movie generation system and the like that allows the user to play back a user's favorite music along with a photo movie, thereby allowing the user to enjoy listening to the music.

Diagram showing the overall configuration of photo movie generation system 1 Diagram showing the hardware configuration of server 2 Diagram showing the hardware configuration of user terminal 3 Diagram showing the functional configuration of photo movie generation system 1 A diagram showing an example of the data structure of acoustic feature data 45 Flowchart showing the flow of processing executed by the photo movie generation system 1 Flowchart showing the flow of acoustic analysis processing A diagram showing an example of an effect template 221 A diagram showing an example of an effect correspondence table 223 Flowchart showing the flow of photo movie generation processing A diagram showing examples of photo movies 61, 62, and 63 A diagram showing an example of movie configuration information 224 Diagram showing the overall configuration of the photo movie generation system 1A A diagram showing the hardware configuration of the music distribution server 4 Diagram showing the functional configuration of the photo movie generation system 1A A diagram showing an example of the data structure of sound source data 44 Flowchart showing the flow of processing executed by the photo movie generation system 1A

Hereinafter, preferred embodiments of the present invention will be described in detail based on the drawings.

[First embodiment]
FIG. 1 is a diagram showing the overall configuration of a photo movie generation system 1 according to a first embodiment of the present invention. As shown in FIG. 1, in the photo movie generation system 1 according to the present invention, a user terminal 3 and a server 2 are communicably connected via a network 5. The server 2 is a computer (photo movie generation device) that generates a photo movie in response to a request from the user terminal 3, and is communicatively connected to the user terminal 3.

The user terminal 3 is an electronic device used by the user, and is equipped with a dedicated application program (hereinafter referred to as a photo movie application) for using the photo movie generation system 1. Alternatively, the user terminal 3 is equipped with a browser that can view the website established by the server 2 on the network 5, and orders and receives photo movies by performing processing with the server 2 via the website. It can also be used as a thing. The user terminal 3 includes, for example, a smartphone, a tablet, a PC, a music player, a game console, and the like.

FIG. 2 is a diagram showing the configuration of the server 2. As shown in FIG. As shown in the figure, the server 2 can be realized by, for example, a computer configured by connecting a control section 201, a storage section 202, a communication section 203, and the like via a bus 204 and the like. However, the present invention is not limited to this, and various configurations can be adopted as appropriate.

The control unit 201 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The CPU calls programs stored in the storage unit 202, ROM, recording medium, etc. to a work memory area on the RAM and executes them, and drives and controls each unit connected via the bus 204. The ROM permanently stores programs such as a computer boot program and BIOS, data, and the like. The RAM temporarily stores loaded programs and data, and includes a work area used by the control unit 201 to perform various processes. The control unit 201 functions as each means of the server 2 by reading and executing the above programs.

The storage unit 202 is, for example, a storage device such as a hard disk drive, solid state drive, or flash memory. The storage unit 202 stores programs executed by the control unit 201, data necessary for program execution, an operating system, and the like. These program codes are read out by the control unit 201 as needed, moved to the RAM, and read out and executed by the CPU. Furthermore, as shown in FIG. 4, the storage unit 202 of the server 2 stores data necessary for generating a photo movie, such as an effect template 221, a general image material 222, an effect correspondence table 223, and movie configuration information 224. Ru. Details of these data will be described later.

The communication unit 203 has a communication control device, a communication port, etc., and controls communication with the network 5 and the like. The network 5 is a LAN (Local Area Network) or a WAN (Wide
Area Network), public communication lines such as the Internet, base stations, etc. The network 5 may be wired or wireless. The server 2 is communicatively connected to the user terminal 3 via the network 5 and is capable of transmitting and receiving various data.
The bus 204 is a path for transmitting and receiving control signals, data signals, etc. between each device.

FIG. 3 is a diagram showing the configuration of the user terminal 3. As shown in the figure, the user terminal 3 connects, for example, a control unit 301, a storage unit 302, a communication unit 303, a display unit 304, an input unit 305, a peripheral device I/F unit 306, an audio output unit 307, etc. to a bus 308, etc. This can be realized by a computer or the like configured by connecting. However, the present invention is not limited to this, and various configurations can be adopted as appropriate. The configurations of the control unit 301, storage unit 302, and communication unit 303 are similar to those of the control unit 201, storage unit 202, and communication unit 203 of the server 2.

The display unit 304 includes a display device such as a liquid crystal panel or a CRT monitor, and a logic circuit (such as a video adapter) for performing display processing in cooperation with the display device. Display information on a display device. Note that the input unit 305 and the display unit 304 may be touch panel displays in which an input device such as a touch panel is integrally provided on the display screen.

The input unit 305 is, for example, an input device such as a touch panel, a keyboard, a pointing device such as a mouse, or a numeric keypad, and outputs input data to the control unit 301.

The peripheral device I/F (interface) section 306 is a port for connecting peripheral devices, and transmits and receives data to and from the peripheral devices via the peripheral device I/F section 306. The peripheral device I/F section 306 is configured with a USB or the like, and usually has a plurality of peripheral device I/Fs. Connection with peripheral devices can be wired or wireless.

The audio output unit 307 outputs the audio data input from the control unit 301 from the speaker.

Note that the storage unit 302 of the user terminal 3 stores an application program (photo movie application) for implementing the process described later (FIG. 6), and the control unit 301 of the user terminal 3 performs the process described later according to this application program. Execute.

Next, the functional configuration of the photo movie generation system 1 will be explained with reference to FIG. 4. As shown in FIG. 4, in the photo movie generation system 1, the server 2 includes, in addition to the storage unit 202 described above, a sound source data acquisition unit 21, an acoustic analysis unit 22, an image acquisition unit 23, an image material generation unit 24, and a photo It includes a movie generation section 25 and the like. The user terminal 3 includes a music specifying section 31, an image specifying section 32, a transmitting section 33, a receiving section 34, etc. in addition to the above-mentioned storage section 302, display section 304, and audio output section 307.

The music designation unit 31 of the user terminal 3 receives music designation from the user. In the first embodiment, the music designation unit 31 designates a music piece from the sound source data stored in the storage unit 302 of the user terminal 3 or the sound source data read from a CD or the like by a user's operation.

The image specifying unit 32 receives a user's specification of an image to be used as an image material of a photo movie. The specified image may be an image stored in the storage unit 302 of the user terminal 3, or may be specified from the general image materials 222 stored in the storage unit 302 of the server 2. When specifying the general image material 222 stored in the storage unit 302 of the server 2, for example, the server 2 transmits an image selection screen for selecting the general image material 222 to the user terminal 3, and the user selects the general image material 222. accept.

The transmitting unit 33 acquires the sound source data of the music specified by the music specifying unit 31 and the image data specified by the image specifying unit 32 from the storage unit 302 and transmits them to the server 2 . If the designated image is the general image material 222 of the server 2, image designation information (image identification information, etc.) is transmitted.

In the storage unit 302, sound source data of a song is stored in association with a song ID, which is identification information of the song. The sound source data of a song is sound source data read from a CD, sound source data downloaded from a server of a music distribution service, or the like. Furthermore, the storage unit 302 stores image data. The image data is an image taken by a user with a camera, an image read with a scanner, an image downloaded via the network 5, or the like.

The song ID uses, for example, composite information such as album name + artist name + song title + playback time. In the music CD database (CDDB; Compact Disc DataBase), album names, artist names, and song titles are displayed on discs.
It is managed by linking it to an ID. The music application installed in the user terminal 3 calculates the Disc ID from the information recorded in the TOC (Table of Contents) area of the music CD, accesses the music CD database via the network 5, and stores the disc. The album name, artist name, and song title corresponding to the ID can be obtained. In this embodiment, information obtained by combining the album name, artist name, and song title obtained in this way with the playback time of the song, etc. is used as the song ID.

The sound source data acquisition unit 21 of the server 2 acquires the sound source data and song ID transmitted from the user terminal 3.

The acoustic analysis unit 22 performs acoustic analysis processing on the sound source data acquired by the sound source data acquisition unit 21 and acquires acoustic feature amount data 45.

Here, the acoustic analysis process performed by the acoustic analysis unit 22 and the acoustic feature data 45 will be described. FIG. 5 is a diagram showing an example of the acoustic feature data 45. In the acoustic analysis process, features such as the music structure (verse A, verse B, chorus, etc.), music tempo (BPM; Beats Per Minute), beat (time signature), and timing of pronunciation of specific instruments are analyzed. In addition to these, features such as chord progression (the flow of the musical melody), volume change (audible volume change), frequency center of gravity (degree of excitement in the music), and pitch power separation (complexity of the music) may also be analyzed.

Acoustic analysis processing is performed using known methods (for example, CRI Middleware Co., Ltd., “Ultra-high-speed, high-precision music analysis middleware BEATWIZ (registered trademark)”, [onine], [searched on March 15, 2020], the Internet). , <URL: https://www.cri-mw.co.jp/product/amusement/beatwiz/index.html>, or co-authored by Naoaki Horiuchi and 5 others, "Song
Surfing: A music playback system that lets you explore a music library using similar phrases", PIONEER R&D (Vol.17, No.2/2007), etc.) is an example, and is not limited thereto.The acoustic analysis unit 22 may also analyze feature amounts other than each item of the acoustic feature amount data 45 shown in FIG.

The acoustic analysis unit 22 outputs the acquired acoustic feature data 45 as a result of the acoustic analysis process to the photo movie generation unit 25, and stores it in the storage unit 202 in association with the song ID. Note that the acoustic analysis unit 22 only acquires the acoustic feature amount data 45 stored in the storage unit 202 for songs that have been subjected to acoustic analysis processing once. Thereby, the load on the server 2 required for acoustic analysis processing can be reduced, and the processing time can be reduced.

The image acquisition unit 23 acquires image data transmitted from the user terminal 3, or designation information for the general image material 222. When designation information for the general image material 222 is acquired, the image acquisition unit 23 acquires the general image material 222 from the storage unit 202 according to the designation information.

The image material generation section 24 generates an image material from the image data acquired by the image acquisition section 23. The image material generation unit 24 detects objects such as people, animals, and articles from image data such as photographs, cuts out the detected objects, and uses them as individual image materials. The image material generation section 24 outputs the generated image material to the photo movie generation section 25.

The photo movie generating unit 25 uses the images acquired by the image acquiring unit 23 or the image material generated by the image material generating unit 24 to generate a photo movie in which the display changes in response to changes in the music based on the acoustic feature data 45 acquired from the acoustic analysis unit 22. The photo movie generating unit 25 generates a photo movie in which the display changes in response to changes in the music by referring to the effect correspondence table 223 and the effect template 221 stored in the storage unit 202. Changes in the music are detected from the "music composition" and "timing of sound production of specific instruments" of the acoustic feature data 45, etc. Details of the photo movie generating process will be described later.

Note that the photo movie generated by the photo movie generation unit 25 is a photo movie with music or a photo movie without music. A photo movie with music is a photo movie to which the sound source data of a song specified by the user terminal 3 is added as BGM. That is, it is a photo movie that includes video data and audio data. A photo movie without music is a photo movie that includes only video data without adding a music sound source (BGM). When a photo movie without music is played, it is played in synchronization with the sound source data of the song.

When the photo movie generation unit 25 generates the photo movie, it compresses and encodes the data (video data) using the MPEG4 method or the like, and converts it into a data format that can be played back on the user terminal 3. Then, the movie configuration information 224, which is information representing the configuration of the photo movie, is stored in the storage unit 202 in association with the song ID, and the data of the photo movie is transmitted to the user terminal 3 in association with the song ID.

The receiving unit 34 of the user terminal 3 receives the photo movie transmitted from the server 2, and sends it to the reproduction processing unit 35.

The reproduction processing unit 35 reproduces the photo movie. When a photo movie without music is received, the playback processing unit 35 acquires the sound source data of the song corresponding to the song ID linked to the photo movie from the storage unit 302, and combines the acquired sound source data with the photo movie (without music). (Photo Movie). The playback processing unit 35 decodes the photo movie into pre-encoded video data and displays it on the display unit 304, and also synchronously plays back the sound source data and outputs it from the audio output unit 307.

When a photo movie with music is received, the playback processing unit 35 decodes the received photo movie data into pre-encoded video data and audio data, displays the video on the display unit 304, and displays the audio (sound source). ) is output from the audio output unit 307. Note that the photo movie may be played back by streaming playback or download playback.

Next, with reference to FIG. 6, the flow of processing executed by the photo movie generation system 1 will be described. In the following explanation, a photo movie application, which is an application for using the photo movie generation system 1, is installed in the storage unit 302 of the user terminal 3, and the control unit 301 of the user terminal 3 reads out this photo movie application. Describe the steps to take.

When the photo movie application is started on the user terminal 3, the control unit 301 of the user terminal 3 accepts the designation of a song to be used in the photo movie (step S101) and the designation of an image (step S102). In steps S101 and S102, the control unit 301 may display an input screen for selectively inputting songs and images.

In the first embodiment, the music that can be specified is music (sound source data) stored in the storage unit 302 of the user terminal 3 or sound source data of the music that can be read by a CD drive. Furthermore, images that can be specified include images stored in the storage unit 302 of the user terminal 3 and general image materials 222 stored in the server 2.

When a song and an image are specified, the control unit 301 acquires the specified sound source data and image data from the storage unit 302 (step S103), and transmits them to the server 2 (step S104). Note that when the general image material 222 is specified, the control unit 301 transmits the identification information of the specified general image material 222 to the server 2.

When the server 2 receives the sound source data and the image data (if the general image material 222 is specified, the identification information of the general image material 222) (step S105), the control unit 201 (acoustic analysis unit 22) Acoustic analysis processing is performed on the sound source data to obtain acoustic feature data 45 of the song (step S106).

FIG. 7 is a flowchart showing the flow of acoustic analysis processing. As shown in FIG. 7, the control unit 201 (acoustic analysis unit 22) acquires sound source data of a song (step S201), and executes acoustic analysis processing (step S202). In the acoustic analysis process, the tempo (BPM; Beats Per Minute) of the song, the beat (time signature), the song structure (A melody, B melody, chorus, etc.), the sound timing of a specific instrument, etc. are analyzed.

The control unit 201 stores the acoustic analysis results as acoustic feature data 45 in the storage unit 202 in association with the song ID (step S203). By storing the acoustic feature data 45 in association with the song ID in the server 2, a song that has already been analyzed does not need to be analyzed again from the second time onwards.

Further, the control unit 201 (image material generation unit 24) of the server 2 executes image material generation processing as necessary on the image data received in step S105, and cuts out image materials from the image data (step S107). Note that the received image data may be used as is as the image material.

The control unit 201 (photo movie generation unit 25) of the server 2 uses the image material and a predetermined effect template 221 to create a photo movie whose screen changes according to the acoustic feature data 45 (for example, a change in "song composition"). A movie is generated (step S108).

Here, the effect template 221 and the effect correspondence table 223 will be explained.
The effect template 221 is a program for adding a predetermined processing (effect) to an image, and as shown in FIG. 8, a plurality of effect templates 221 are assigned effect numbers. and the number of used images are stored in association with each other.

The effect template 221 includes, for example, "image switching" to sequentially switch between multiple display images, "monochrome to color" to change the image from monochrome to color, "transmittance change" to change the transmittance of the image, and image display position. ``Move screen'' to move the image within the screen, ``Rotate'' to rotate the image, ``Enlarge'' to enlarge the image, ``Shrink'' to reduce the image, and ``Change display ratio'' to change the display ratio of the image within the display screen. ", "Appearance" to make an image appear, "Move to enlarge" to move the image while enlarging it, "Rotate to enlarge" to rotate the image while enlarging it, "Move to shrink" to move the image while shrinking it, "Move to reduce" to move the image while enlarging it, ``reducing rotation'', which rotates while rotating, ``small image increase'', which gradually increases the number of small images, ``multiple image single screen display'', which displays multiple images on one screen, and multiple images displayed on one screen. There are "multiple images in one screen display + slide" which slides images.

Note that the effect template 221 shown in FIG. 8 is an example, and may include other effects. Furthermore, a plurality of patterns of parameters such as the amount of movement and the speed of change may be prepared for each effect. For example, for the effect "Enlargement", "Enlargement 1", "Enlargement 2", "Enlargement 3", etc. are prepared depending on the enlargement rate, and for the effect "Transmittance change", "Transmittance change 1" is prepared depending on the speed of change. ”, “transmittance change 2”, etc.

The control unit 201 (photo movie generation unit 25) refers to the effect correspondence table 223 and determines which effect to use when generating a photo movie. FIG. 9 is a diagram showing an example of the effect correspondence table 223. The effect correspondence table 223 is a table that associates musical composition elements (intro, A melody, B melody, C melody, solo, chorus, etc.) with effect templates 221 suitable for the elements, and as shown in FIG. pattern is defined.

In the example of FIG. 9, in the case of pattern (1), the control unit 201 (photo movie generation unit 25) applies the effect "monochrome → color" to the "intro" of the song, and applies the effect "monochrome → color" to the "A melody" of the song. Apply "Movement 3", apply the effect "Increase small images" to the "B melody" of the song, apply "Multiple images in one screen display 1" to the "C melody" of the song, and apply the effect "Movement 3" to the "Solo" of the song. Apply "Enlargement 1" and apply the effect "Image switching 1" at the "chorus" of the song.

Furthermore, in the case of pattern (2) in FIG. 9, the control unit 201 (photo movie generation unit 25) applies the effect "image switching 1" to the "intro" of the song, and applies the effect "shrinking" to the "A melody" of the song. Apply the effect "Rotation 1" to the "B melody" of the song, apply the effect "Screen movement" to the "C melody", and apply the effect "Enlargement 2" to the "Solo" of the song. Apply the effect "Image switching 2" at the "chorus" of the song. In this way, a plurality of patterns of effect application examples are defined in the effect correspondence table 223.

In step S108, the control unit 201 (photo movie generation unit 25) determines which pattern in the effect correspondence table 223 is used to generate the photo movie, for example, by the process shown in FIG. 10.

As shown in FIG. 10, the control unit 201 (photo movie generation unit 25) of the server 2 first determines whether or not the movie configuration information 224 corresponding to the song ID is stored in the storage unit 202 (step S301 ). The movie configuration information 224 is data that stores information for configuring the photo movie, such as image data used in the photo movie, image switching timing, and effects to be applied to the images (see FIG. 12). For photo movies already generated by the server 2, movie configuration information 224 is stored in the storage unit 202 in association with the song ID.

In step S301, if the movie configuration information 224 corresponding to the song ID of the sound source data transmitted from the user terminal 3 is not stored in the storage unit 202 (step S301; No), a photo movie has been created for the song. Since there is no history, the control unit 201 of the server 2 reads an arbitrary pattern from the effect correspondence table 223 (step S302), and generates a photo movie according to the effect correspondence table 223 of the read pattern (step S303).

On the other hand, in step S301, if the movie configuration information 224 corresponding to the song ID of the sound source data transmitted from the user terminal 3 is already stored in the storage unit 202 (step S301; Yes), the control unit 201 An unused pattern is read from the correspondence table 223 (step S304), and a photo movie is generated according to the effect correspondence table 223 of the read pattern (step S303). In this way, for a song for which a photo movie has been created, a photo movie of a different pattern is created from the second time onwards. Therefore, different movie expressions can be generated for the same song, and users can be provided with a variety of entertainment.

FIG. 11 is a diagram showing examples of various photo movies 61, 62, and 63 generated for songs (sound source data 44). The photo movie 61 in FIG. 11A is generated by using images 71, 72, 73, . . . acquired by the image acquisition unit 23 as they are. The displayed image is changed at the timing when the composition of the song (sound source data 44) changes (timing t1 when the melody A changes to the B melody, timing t2 when the melody B changes to the chorus, etc.).

Furthermore, the photo movie 62 in FIG. 11(b) is obtained by applying the effect template 221 to images 71, 72, 73, . . . acquired by the image acquisition unit 23. The image is changed at the timing when the music composition of the song (sound source data 44) changes (timing t1 when the A melody changes to the B melody, timing t2 when the B melody changes to the chorus), and the images are changed in each section (t0 to t1, t1 to At t2, t2~), a predetermined effect is applied to the image.

In the example of FIG. 11(b), the effect "monochrome→color" is applied to the image 71 in the section t0 to t1 of "A melody", and the monochrome image 71a is gradually changed to the color image 71b. Next, at time t1 when the melody A changes to the melody B, the image is switched and the effect is also changed to "reduction rotation 1". In the interval t1 to t2, an animation is generated in which the image 71 is used as the background and the image 72 is used as the foreground, and the foreground image 72 is reduced and rotated from the state of the image 72a to the state of the image 72b. Next, at time t2 when the B melody changes to the chorus, the image is switched and the effect is also changed to "reduction rotation 2". In the interval t2~, the image 71 is used as the background and the image 73 is used as the foreground, and an animation is generated in which the foreground image 73 is reduced and rotated from the state of the image 73a to the state of the image 73b.

The photo movie 63 in FIG. 11(c) has images 74, 76, 79, ... acquired by the image acquisition unit 23 as background images, and image materials 75, 77, 81, 82 (image material generation unit 24) in the foreground. It was generated using the image material (generated by The background image is changed at the timing when the composition of the song (sound source data 44) changes (timing t1 when the melody A changes to the B melody, timing t2 when the melody B changes to the chorus), and the background image is changed in each section (t0 to t1, t1). ~t2, t2~), a predetermined effect is applied to the foreground image.

In the example of FIG. 11(c), the image 74 is displayed as the background in the section t0 to t1 of the melody, and the effect "Screen movement" is applied to the foreground image material 75, and the image moves from image 75a to 75b to 75c. . Next, at time t1 when the melody A changes to the melody B, the background image is switched to the image 76, and the effect "alternate display" is applied to the foreground image material 77, so that the images 77a and 77b are displayed alternately. An animation is generated. Next, at time t2 when the B melody changes to the chorus, the background image is switched to image 79, and the foreground images are also changed to image materials 81 and 82. In the period t2~, the effect "appearance" is applied to the image materials 81 and 82 in the foreground, and the image materials 81 and 82 are displayed so as to appear in sequence in accordance with the sound timing of the bass drum.

When the control unit 201 (photo movie generating unit 25) generates a photo movie, it links the movie configuration information 224 of the generated photo movie to a music ID and stores it in the storage unit 202. As shown in FIG. 12, the movie configuration information 224 stores the music configuration, time, applied effects, number of images used, and file names of the images used, linked to the music ID.

The example in FIG. 12 is movie configuration information 224 of a photo movie that uses the song with the song ID "***** (album name + artist name + song title + playback time, etc.)" as BGM. The time of the section of the music composition "Intro" is "0-15", and the effect of "Monochrome→Color 1" is applied. The number of images used in this effect is "1", and the file name of the image used is "image15".

Further, the time of the section of the music composition "A melody" is "15-50", and the effect of "enlargement rotation 3" is applied. The number of images used in this effect is "2", and the file names of the images used are "image8" (enlarged and rotated image) for the foreground and "image2" for the background.

Further, the time of the section of the music composition "B melody" is "50-70", and the effect of "reduced rotation 2" is applied. The number of images used in this effect is "2", and the file names of the images used are "image8" for the foreground (an image to be reduced and rotated) and "image17" for the background.

Further, the time of the section of the music composition "Chorus" is "70-120", and the effect of "Image switching 5" is applied. The number of images used in this effect is "5", and the file names of the images used are "image10" (the first image displayed), "image11" (the second image displayed), and "image12". ” (the third displayed image), “image13” (the fourth displayed image), and “image14” (the fifth displayed image).

Further, the times (times) of the music composition "Bass Drum Timing" are "70, 110, 111, 112, 113", and the effect of "Appearance 5" is applied. The number of images used in this effect is "6", and the file names of the images used are "image15" for the background, "person 1" for the foreground (the image that appears at the first bass drum timing (70)), "Person 2" (image appearing at the second bass drum timing (110)), "Person 3" (image appearing at the third bass drum timing (111)), "Person 4" (image appearing at the fourth bass drum timing) "Person 5" (image appearing at the fifth bass drum timing (113)).

The items “music composition” and “time” of the movie composition information 224 correspond to the acoustic feature amount data 45 of the musical piece. The item "applied effect" is obtained from the effect correspondence table 223 shown in FIG. 9, and the item "number of images used" is obtained from the effect template 221 shown in FIG. 8. Which effect correspondence table 223 is to be used is determined by the control unit 201 (photo movie generation unit 25) executing the process shown in FIG. Further, the item “Used image file name” is arbitrarily determined by the control unit 201 (photo movie generation unit 25) from the images acquired by the image acquisition unit 23 and the images generated by the image material generation unit 24.

Returning to the explanation of FIG. 6.
When the photo movie is generated by the process in step S108, the server 2 transmits the generated photo movie to the user terminal 3 (step S109).

The user terminal 3 receives the photo movie transmitted from the server 2 (step S110), and plays the photo movie (step S111). If the received photo movie is a photo movie with music, the control unit 301 (playback processing unit 35) decodes and plays the received photo movie, thereby synchronously playing back the music and video.

Further, when the received photo movie is a photo movie without music, the control unit 301 (playback processing unit 35) reads the specified music from the storage unit 302 and plays it in synchronization with the photo movie. The control unit 301 (playback processing unit 35) of the user terminal 3 plays back the photo movie and the sound source data in synchronization, thereby playing back the photo movie whose screen changes according to changes in the music composition.

As explained above, in the photo movie generation system 1 of the first embodiment, when the user terminal 3 specifies music and images and transmits sound source data and image data to the server 2, the server 2 side generates an audio version of the sound source data. Analysis processing is performed to generate a photo movie based on the acoustic feature amount data 45. The user terminal 3 receives the photo movie generated by the server 2, and in the case of a photo movie without music, plays it in synchronization with the sound source data.

In this way, the photo movie generation system 1 can generate and provide to the user a photo movie whose screen changes according to changes in music that can be detected from the acoustic feature amount data 45. Further, the server 2 stores the movie configuration information 224 for songs for which photo movies have been previously generated, and generates photo movies of different patterns, so that it can provide a variety of entertainment to the user. Further, since the server 2 accumulates and stores acoustic feature data 45 for previously acquired sound source data, it is possible to generate a photo movie in a short time without performing acoustic analysis processing. Further, by generating a photo movie without music and playing it back in synchronization with the sound source data on the user terminal 3 side, the file size can be made smaller than that of a photo movie with music, and storage capacity and communication capacity can be reduced.

[Second embodiment]
Next, a photo movie generation system 1A according to a second embodiment will be described. In the second embodiment, a system configuration when the user terminal 3 does not have sound source data will be described.

Figure 13 is a diagram showing the overall configuration of a photo movie production system 1A according to the second embodiment of the present invention. As shown in the figure, in the photo movie production system 1A, a user terminal 3 and a server 2 are communicatively connected via a network 5. The user terminal 3 and the server 2 can also be communicatively connected via the network 5 to a music distribution server 4 having a music database 40. The hardware configuration of the server 2 and the user terminal 3 is the same as in the first embodiment. The same parts as in the first embodiment are given the same reference numerals, and duplicate explanations will be omitted.

FIG. 14 is a diagram showing the configuration of the music distribution server 4. As shown in FIG. As shown in the figure, the music distribution server 4 can be realized by, for example, a computer configured by connecting a control section 401, a storage section 402, a communication section 403, etc. via a bus 404 or the like. However, the present invention is not limited to this, and various configurations can be adopted as appropriate. The configurations of the control unit 401, storage unit 402, and communication unit 403 are similar to those of the control unit 201, storage unit 202, and communication unit 203 of the server 2. The music database 40 of the music distribution server 4 stores music source data 44 and acoustic feature data 45.

Next, the functional configuration of the photo movie generation system 1A of the second embodiment will be described with reference to FIG. 15. As shown in the figure, in the photo movie generation system 1A, the server 2 includes a storage section 202, an acoustic feature amount data acquisition section 26, an image acquisition section 23, an image material generation section 24, a photo movie generation section 25, and the like. The user terminal 3 includes a storage unit 302, a display unit 304, an audio output unit 307, a music designation unit 31, an image designation unit 32, a transmission unit 33, a reception unit 34, a sound source data acquisition unit 36, and the like. The music distribution server 4 includes a music database 40, a transmitter 41, and an acoustic analyzer 42.

The photo movie generation system 1A of the second embodiment differs from the photo movie generation system 1 of the first embodiment in that the server 2 is not provided with the sound source data acquisition section 21 and the acoustic analysis section 22, and the acoustic feature amount data The acquisition unit 26 is provided, the sound source data acquisition unit 36 is provided without storing sound source data in the storage unit 302 of the user terminal 3, and the music distribution server 4 includes a music database 40 and an acoustic analysis unit 42. This is the point.

The music distribution server 4 stores sound source data 44, acoustic feature data 45, song lyrics information, song bibliographic information, etc. of various songs in a song database 40 in association with song IDs. As shown in FIG. 16, the sound source data 44 is stored in association with a music ID.

The acoustic analysis unit 42 of the music distribution server 4 performs acoustic analysis processing on the songs stored in the song database 40, obtains acoustic feature data 45 as an analysis result, and stores it in the song database 40 in association with the song ID. do. The acoustic feature data 45 is the same as the acoustic feature data 45 shown in FIG.

The transmission unit 41 of the music distribution server 4 transmits the sound source data 44 to the user terminal 3 in response to a request from the user terminal 3. In addition, the transmission unit 41 transmits the sound source data 44 and the acoustic feature data 45 to the server 2 in response to a request from the server 2.

Upon receiving the song ID of the song specified by the song specifying section 31 of the user terminal 3, the acoustic feature data acquisition unit 26 of the server 2 transmits the acoustic feature data 45 of the song corresponding to the song ID to the music distribution server 4. Request and get.

The audio data acquisition unit 36 of the user terminal 3 requests and acquires the audio data 44 of the music specified by the music designation unit 31 from the music distribution server 4. When a photo movie without music is transmitted from the server 2, the playback processing unit 35 of the user terminal 3 acquires the audio data from the music distribution server 4 by the audio data acquisition unit 36 and plays it in sync with the received photo movie without music.

Next, with reference to FIG. 17, the flow of processing in the photo movie generation system 1A of the second embodiment will be described.
When the photo movie application is started on the user terminal 3, the control unit 301 of the user terminal 3 receives the designation of a song to be used in the photo movie (step S401) and the designation of an image (step S402). In steps S401 and S402, the control unit 301 may display an input screen for selectively inputting songs and images.

In the second embodiment, the music that can be specified is the music (sound source data 44) stored in the music database 40 of the music distribution server 4. Furthermore, the images that can be specified are the images stored in the storage unit 302 of the user terminal 3 and the general image materials 222 stored in the server 2, as in the first embodiment.

When a song and an image are specified, the control unit 301 acquires the specified image data from the storage unit 302 (step S403), and transmits it to the server 2 together with the song ID of the specified song (step S404). Note that when the general image material 222 is specified, the control unit 301 transmits the identification information of the specified general image material 222 to the server 2.

When the server 2 receives the song ID and the image data (if the general image material 222 is specified, the identification information of the general image material 222) (step S405), the control unit 201 (acoustic feature data acquisition unit 26) , requests the music distribution server 4 for acoustic feature amount data 45 corresponding to the received song ID (step S406).

When the music distribution server 4 receives a request for the acoustic feature data 45 corresponding to the song ID from the server 2 (step S407), it reads the acoustic feature data 45 corresponding to the requested song ID from the song database 40, and sends it to the server 2. (Step S408). The server 2 receives the acoustic feature data 45 corresponding to the song ID (step S409).

In step S408, if the corresponding acoustic feature data 45 is not stored in the music database 40 of the music distribution server 4, the music distribution server 4 executes acoustic analysis processing for the song with the requested song ID using the acoustic analysis unit 42 to obtain acoustic feature data 45. The music distribution server 4 associates the acoustic feature data 45 with the song ID, stores it in the music database 40, and transmits it to the server 2 that made the request.

Further, the control unit 201 (image material generation unit 24) of the server 2 executes image material generation processing as necessary on the image data received in step S405, and cuts out image materials from the image data (step S410). Note that the received image data may be used as is as the image material.

The control unit 201 (photo movie generation unit 25) of the server 2 uses the image material and a predetermined effect template 221 to create a photo movie whose screen changes according to the acoustic feature data 45 (for example, a change in the “song composition”). A movie (photo movie without music) is generated (step S411).

The method of generating a photo movie is the same as in the first embodiment. That is, as shown in FIG. 10, the control unit 201 (photo movie generating unit 25) determines whether movie configuration information 224 corresponding to a song ID specified from the user terminal 3 is already stored in the storage unit 202, and if it is not stored in the storage unit 202, generates a photo movie using an effect correspondence table 223 of any pattern from the effect correspondence table 223.

Further, if the movie configuration information 224 corresponding to the music ID specified from the user terminal 3 is already stored in the storage unit 202, the control unit 201 selects the effect correspondence table 224 of the unused pattern from the effect correspondence table 223. is read out (step S304), and a photo movie is generated according to the effect correspondence table 223 of the read pattern.

When the control unit 201 (photo movie generation unit 25) generates a photo movie, it stores the movie configuration information 224 of the generated photo movie in the storage unit 202 in association with the song ID. As shown in FIG. 12, the movie structure information 224 stores the song structure, time, applied effects, number of used images, and used image file name in association with the song ID. Further, the server 2 transmits the generated photo movie to the user terminal 3 (step S412).

When the user terminal 3 receives the photo movie from the server 2 (step S413), it performs playback processing. Since the received photo movie is a photo movie without music, the control unit 301 (playback processing unit 35) requests the music distribution server 4 for the sound source data 44 of the song ID of the song specified in step S401 (step S414).

When the music distribution server 4 receives a request for the sound source data 44 of the song ID from the user terminal 3 (step S415), the music distribution server 4 reads the sound source data 44 of the requested song ID from the song database 40 and transmits it to the user terminal 3 that made the request. (Step S416).

When the user terminal 3 receives the sound source data 44 from the music distribution server 4 (step S417), the control unit 301 (playback processing unit 35) of the user terminal 3 combines the sound source data 44 with the photo movie without music received in step S413. are synchronized and reproduced (step S418). With this, it is possible to play a photo movie whose screen changes according to changes in the music composition.

Note that the above example describes the procedure for generating a photo movie without music, but when requesting the server 2 to transmit the acoustic feature data 45 to the music distribution server 4 in step S406, the sound source data If the user also requests the transmission of 44 and obtains the sound source data 44 of the corresponding song, it is possible to generate a photo movie with music. When a photo movie with music is generated by the server 2, the user terminal 3 receives the photo movie with music, and the control unit 301 (playback processing unit 35) decodes and plays the received photo movie so that the music and video can be reproduced. are played simultaneously.

As described above, in the second embodiment, a photo movie generation system 1A has been described in which the user terminal 3 does not have sound source data, and the server 2 acquires acoustic feature data 45 from the music distribution server 4 to generate a photo movie. According to the photo movie generation system 1A of the second embodiment, the server 2 does not perform acoustic analysis processing, but acquires already analyzed acoustic feature data 45 from the music distribution server 4 and uses it to generate a photo movie, thereby reducing the processing burden on the server 2. Also, since sound source data 44 is not sent or received between the user terminal 3 and the server 2, the amount of communication is reduced, and the time required for communication is shortened.

Although preferred embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to such examples. For example, the music distribution server 4 does not have the acoustic analysis unit 42 or the acoustic feature data 45, but the server 2 includes the acoustic analysis unit 42, performs acoustic analysis processing on songs received from the music distribution server 4, and performs acoustic analysis processing on each song. A configuration may be adopted in which the acoustic feature amount data 45 is accumulated and stored.

Further, in the above-described embodiment, an example was shown in which the server 2 generates a photo movie whose display changes mainly according to changes in the song structure, but the present invention is not limited to this, and For example, a photo movie whose display changes according to changes in the tempo of the song, changes in the beat (time signature), etc. may be generated. It is clear that those skilled in the art can come up with various other changes or modifications within the scope of the technical idea disclosed in this application, and these naturally fall within the technical scope of the present invention. It is understood that

1...Photo movie generation system 2...Server 3...User terminal 4... -Music distribution server 5...Network 21...Sound source data acquisition unit 22...Acoustic analysis unit 23... ... Image acquisition section 24 ... Image material generation section 25 ... Photo movie generation section 26 ... Acoustic feature amount data acquisition section 31... Music designation section 32... Image specification section 33... Transmission section 34... Receiving section 35......Playback processing unit 36...Sound source data acquisition unit 40...Music database 41...Transmission Section 42...Acoustic analysis section 44...Sound source data 45...Acoustic feature data 61, 62, 63...Photo movie 221...Effect template 222...General image material 223...Effect support table 224...Movie configuration information 201, 301 ... Control section 202, 302 ... Storage section 203, 303 ... Communication section 304 ... Input section 305 ... Display section 306 ... ...Peripheral device I/F section 307 ...Audio processing section

Claims (16)

A photo movie generation system in which a server and a user terminal are communicatively connected via a network,
The user terminal is
music designation means that accepts a designation of a music piece to be used in a photo movie and transmits music ID, which is the sound source data of the designated music piece or identification information of the music piece, to the server;
image designating means for accepting designations of a plurality of images to be used as image materials for a photo movie, and transmitting designated image data or designation information of the images to the server;
Reproducing means for receiving and reproducing the photo movie transmitted from the server,
The server is
Acoustic feature data acquisition means for acquiring audio feature data of a music piece specified by the user terminal;
image acquisition means for acquiring images to be used in the photo movie;
a photo movie generating unit that uses the image acquired by the image acquisition unit to generate a photo movie whose display changes according to changes in music based on the acoustic feature data, and transmits the generated photo movie to the user terminal;
A table that associates an effect template in which at least some of the effects are associated with a number of used images of 2 or more, and the acoustic feature amount data and the effect template that is suitable for the acoustic feature amount data, the table including the effect template that is associated with the number of images used, wherein at least some of the effects are associated with the number of images to be used, and the acoustic feature amount data and the effect template that is suitable for the effect template; storage means for storing in advance an effect correspondence table in which examples are defined;
Equipped with
The storage means of the server stores movie configuration information that is information constituting the photo movie and stores the number of used images acquired from the effect template and applied effects acquired from the effect correspondence table. Store it in association with the song ID,
The photo movie generation means refers to movie configuration information stored in association with the designated music ID, and if the movie configuration information is not stored in the storage means, generates an arbitrary pattern from the effect correspondence table. and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table, and a photo movie is generated according to the effect correspondence table of the read pattern. Photo movie generation system. The storage means of the server stores the acoustic feature data of the song in association with the song ID,
2. The photo movie generation system according to claim 1 , wherein the acoustic feature amount data acquisition means acquires the acoustic feature amount data stored in the storage means. The acoustic feature data acquisition means of the server
2. The photo movie generating system according to claim 1, wherein the acoustic feature data of the music piece designated by the user terminal is acquired from a music distribution server connected to the user terminal via a network. The server further includes acoustic analysis means for performing an acoustic analysis process on the song to obtain acoustic feature data,
2. The photo movie generation system according to claim 1 , wherein the acoustic feature amount data acquisition means acquires the acoustic feature amount data obtained by the acoustic analysis means. 5. The photo movie generating system according to claim 4 , wherein the server acquires the sound source data of the music from a music distribution server connected to the server via a network. 5. The photo movie generation system according to claim 4 , wherein the server acquires sound source data of the song from the user terminal. The photo movie generating means of the server generates the photo movie without adding a sound source, and outputs the photo movie in association with the song ID of the song,
A claim characterized in that the playback means of the user terminal acquires sound source data of a song corresponding to a song ID linked to the photo movie, and plays the acquired sound source data and the photo movie in synchronization. A photo movie generation system according to any one of claims 1 to 6 . The server is
further comprising image material generation means for generating an image material from the image acquired by the image acquisition means,
The photo movie generation system according to any one of claims 1 to 7 , wherein the photo movie generation means generates the photo movie using the image material generated by the image material generation means. . The user terminal further includes image sending means for sending images to be used in the photo movie to the server,
9. The photo movie generation system according to claim 1, wherein the image acquisition means of the server receives images transmitted from the user terminal. image acquisition means for acquiring an image;
an acoustic feature data acquisition means for acquiring acoustic feature data of a piece of music;
a photo movie generating means for generating a photo movie in which a display changes in accordance with a change in music based on the acoustic feature data, using the images acquired by the image acquiring means;
a storage means for storing in advance an effect correspondence table in which an effect template in which at least some of the effects correspond to two or more images to be used and a table in which the acoustic feature data corresponds to the effect template suitable for the acoustic feature data, the effect correspondence table defining a plurality of patterns of effect application examples;
Equipped with
the storage means stores movie configuration information, which is information constituting the photo movie and includes the number of images used obtained from the effect template and the applied effects obtained from the effect correspondence table, in association with a song ID, which is identification information of a song;
The photo movie generation means refers to movie composition information stored in association with a specified song ID, and if the movie composition information is not stored in the storage means, reads an arbitrary pattern from the effect correspondence table, and if the movie composition information is already stored in the storage means, reads an unused pattern from the effect correspondence table and generates a photo movie according to the effect correspondence table of the read pattern. an image acquisition means for acquiring an image;
a sound source data acquisition means for acquiring sound source data of a song;
acoustic analysis means for analyzing the sound source data to obtain acoustic feature data;
a photo movie generating unit that uses the images acquired by the image acquiring unit and generates a photo movie whose display changes according to changes in music based on the acoustic feature data;
A table that associates an effect template in which at least some of the effects are associated with a number of used images of 2 or more, and the acoustic feature amount data and the effect template that is suitable for the acoustic feature amount data, the table including the effect template that is associated with the number of images used, wherein at least some of the effects are associated with the number of images to be used, and the acoustic feature amount data and the effect template that is suitable for the effect template; storage means for storing in advance an effect correspondence table in which examples are defined;
Equipped with
The storage means stores movie configuration information that constitutes the photo movie and stores the number of used images acquired from the effect template and applied effects acquired from the effect correspondence table. It is stored in association with the song ID which is identification information,
The photo movie generation means refers to movie configuration information stored in association with the designated music ID, and if the movie configuration information is not stored in the storage means, generates an arbitrary pattern from the effect correspondence table. and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table, and a photo movie is generated according to the effect correspondence table of the read pattern. Photo movie generator. A user terminal that can be communicatively connected to a server that is the photo movie generation device according to claim 10 or 11 via a network,
music designation means that receives a designation of a music piece and transmits the music ID, which is the sound source data of the designated music piece or identification information of the music piece, to the server;
image designating means for accepting designations of a plurality of images and transmitting designated image data or designation information of the images to the server;
Reproducing means for receiving and reproducing a photo movie whose display changes according to changes in the music and displays a plurality of images , which is transmitted from the server;
A user terminal comprising: A photo movie generation method in a photo movie generation system in which a server and a user terminal are communicatively connected via a network,
The server is a table that associates effect templates in which at least some of the effects are associated with a number of used images of 2 or more, and acoustic feature amount data and the effect templates that are suitable for the effect templates, the table including a plurality of patterns. storing in advance an effect correspondence table in which effect application examples of are defined;
a step in which the user terminal receives a designation of a song to be used in a photo movie, and transmits a song ID, which is the sound source data of the specified song or identification information of the song, to the server;
the user terminal receiving designation of a plurality of images to be used as image materials of a photo movie, and transmitting designated image data or designation information of the images to the server;
the server acquiring an image;
a step in which the server acquires acoustic feature data of a music piece specified in the user terminal;
The server uses the images to generate a photo movie whose display changes according to changes in music based on the acoustic feature data, and transmits it to the user terminal;
the user terminal receiving and playing the photo movie transmitted from the server;
including;
The step of the server using the image to generate a photo movie whose display changes according to changes in the music based on the acoustic feature data and transmitting it to the user terminal,
Regarding the generated photo movie, the movie configuration information, which is information constituting the photo movie and stores the number of used images obtained from the effect template and the applied effects obtained from the effect correspondence table, is Store it in the server in association with the song ID,
When generating a photo movie, refer to the movie configuration information stored in association with the specified song ID, and if the movie configuration information is not stored in the server, select any one from the effect correspondence table. A pattern is read out, and if the movie configuration information is already stored in the server, an unused pattern is read out from the effect correspondence table, and a photo movie is generated according to the effect correspondence table of the read pattern. How to generate a photo movie. computer,
image acquisition means for acquiring images;
acoustic feature data acquisition means for obtaining acoustic feature data of a song;
photo movie generation means for generating a photo movie whose display changes according to changes in music based on the acoustic feature data using the images acquired by the image acquisition means;
It is a program to function as
The computer,
A table that associates an effect template in which at least some of the effects are associated with a number of used images of 2 or more, and the acoustic feature amount data and the effect template that is suitable for the acoustic feature amount data, the table including the effect template that is associated with the number of images used, wherein at least some of the effects are associated with the number of images to be used, and the acoustic feature amount data and the effect template that is suitable for the effect template; storage means for storing in advance an effect correspondence table in which examples are defined;
function as
The storage means stores movie configuration information that constitutes the photo movie and stores the number of used images acquired from the effect template and applied effects acquired from the effect correspondence table. It is stored in association with the song ID which is identification information,
The photo movie generation means refers to movie configuration information stored in association with the designated music ID, and if the movie configuration information is not stored in the storage means, generates an arbitrary pattern from the effect correspondence table. and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table, and a photo movie is generated according to the effect correspondence table of the read pattern. program. computer,
image acquisition means for acquiring images;
a sound source data acquisition means for acquiring sound source data of a song;
acoustic analysis means for analyzing the sound source data to obtain acoustic feature data;
photo movie generation means for generating a photo movie whose display changes according to changes in music based on the acoustic feature data using the images acquired by the image acquisition means;
It is a program to function as
The computer,
A table that associates an effect template in which at least some of the effects are associated with a number of used images of 2 or more, and the acoustic feature amount data and the effect template that is suitable for the acoustic feature amount data, the table including the effect template that is associated with the number of images used, wherein at least some of the effects are associated with the number of images to be used, and the acoustic feature amount data and the effect template that is suitable for the effect template; storage means for storing in advance an effect correspondence table in which examples are defined;
function as
The storage means stores movie configuration information that constitutes the photo movie and stores the number of used images acquired from the effect template and applied effects acquired from the effect correspondence table. It is stored in association with the song ID which is identification information,
The photo movie generation means refers to movie configuration information stored in association with the designated music ID, and if the movie configuration information is not stored in the storage means, generates an arbitrary pattern from the effect correspondence table. and if the movie configuration information is already stored in the storage means, an unused pattern is read from the effect correspondence table, and a photo movie is generated according to the effect correspondence table of the read pattern. program. A computer that can be communicatively connected to a server that executes the program according to claim 14 or 15 via a network,
music designation means that receives a designation of a music piece and transmits the sound source data of the designated music piece or a music ID that is identification information of the music piece to the server;
image designating means for accepting designations of a plurality of images and transmitting designated image data or designation information of the images to the server;
Reproducing means for receiving and reproducing a photo movie whose display changes according to changes in the music and displays a plurality of images , which is transmitted from the server;
A program to function as

Images