JP7121988B2 - MOVIE CONTENT GENERATION METHOD AND GENERATION PROGRAM - Google Patents

Description

The present invention relates to a method and program for generating new video content from first material content including video information and audio information, and in particular, to a first material content created by a user himself and a preset second material content. The present invention relates to a method and program for generating new moving image content by synthesizing.

With the spread of smartphones in recent years, it has become possible to easily shoot videos without preparing expensive equipment. Also, in recent years, with the development of high-speed communication environments, it has become easy to open videos taken with smartphones to the public.

“TikTok (registered trademark)” is known as application software for smartphones that takes advantage of this situation. A user of TikTok can generate an original moving image and open it to the public according to the following procedure (see, for example, Non-Patent Documents 1 and 2).
(1) Select music as BGM from a plurality of preset music.
(2) Shoot a video that matches the selected BGM.
(3) Upload the video you shot.
In addition, before uploading the captured moving image, the user can decorate the captured moving image by applying one or more special effects selected from a plurality of special effects called filters or time effects.

"Tik Tok (app)", [online], Wikipedia Free Encyclopedia, [searched September 3, 2018], Internet <URL: https://ja.wikipedia.org/wiki/Tik_Tok_(%E3%82) %A2%E3%83%97%E3%83%AA)> "Basic usage of the free video creation app 'Tik Tok'-Introducing how to insert songs and save videos!", [online], April 18, 2018, Dohak, [September 3, 2018 day search], Internet <URL: https://dohack.jp/video/tik-tok>

In generating moving images using the above application software, the user is required to have an aesthetic sense when shooting moving images that match the background music or applying special effects to the shot moving images. For this reason, the above application software is very difficult for those who do not have confidence in their own aesthetic sense.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a moving image content generating method and a generating program that enable anyone to easily generate moving image content that looks good.

In order to solve the above problems, a video content generation method according to the present invention is a video content generation method for generating video content from first material content including video information and audio information,
a first reading step of reading the first material content from a storage device; a content separation step of separating the read first material content into the video information and the audio information; and based on the separated audio information. , an extraction start point determination step of determining N extraction start points (where N is an integer equal to or greater than 2); a second reading step of reading a second material content, which is music information, and synthesis information corresponding to the second material content from the storage device; and based on the synthesis information, a content synthesizing step of synthesizing the music information and the divided video information to generate the video content,
The synthesizing information includes a plurality of video switching points synchronized with the beat represented by the music information, the plurality of video switching points belong to one of N groups, and the content synthesizing step includes: each of the video switching points belonging to N is set as the starting point of any one of the N pieces of divided video information.

The video content generation method has a configuration in which, in the extraction start point determination step, the extraction start point is set to the top N times of a plurality of volume peak points included in the separated audio information. You may have

The moving image content generating method further comprises a command receiving step of receiving a command regarding selection of the first material content and the second material content, wherein in the first reading step, based on the command, one of the plurality of first material contents stored in the storage device, and one of the plurality of second material contents pre-stored in the storage device based on the command in the second reading step; The composition information corresponding to the one may be read.

Note that the synthesis information can be described in MIDI format. In this case, the N groups can be represented by musical scales.

In order to solve the above-described problems, a video content generation program according to the present invention causes an information processing apparatus to execute a video content generation method for generating video content from first material content including video information and audio information. A content generation program,
a first reading step of reading the first material content from a storage device into the information processing device; a content separation step of separating the read first material content into the video information and the audio information; an extraction start point determination step of determining N extraction start points (where N is an integer of 2 or more) based on the audio information; and determining each of the N extraction start points from the separated video information. a divided video extracting step of extracting N pieces of divided video information starting from , and a second reading step of reading second material content, which is music information, and synthesis information corresponding to the second material content from the storage device and executing a content synthesizing step of synthesizing the music information and the divided video information based on the synthesizing information to generate the video content,
The synthesizing information includes a plurality of video switching points synchronized with the beat represented by the music information, the plurality of video switching points belonging to one of N groups, and the content synthesizing step comprising: may be set as the start point of any one of the N divided video information.

wherein, in the extraction start point determination step, the moving image content generation program determines the extraction start point at a time corresponding to the top N of a plurality of volume peak points included in the separated audio information. may have a configuration.

The video content generation program causes the information processing device to further execute a command receiving step of receiving a command regarding selection of the first material content and the second material content, and in the first reading step, based on the command one of the plurality of first material contents pre-stored in the storage device is read, and in the second reading step, the plurality of second material contents pre-stored in the storage device based on the instruction; It may have a configuration in which one of the material contents and the synthesis information corresponding to the one are read.

Note that the synthesis information can be described in MIDI format. In this case, the N groups can be represented by musical scales.

According to the present invention, it is possible to provide a moving image content generating method and a generating program that enable anyone to easily generate moving image content that looks good.

FIG. 4 is a block diagram showing a schematic configuration of a smartphone executing a moving image content generation program according to an embodiment of the present invention; Fig. 3 is a flow diagram of a method for generating video content according to an embodiment of the present invention; 3 is a diagram for explaining a content separation process shown in FIG. 2; FIG. FIG. 3 is a diagram for explaining an extraction start point determination step shown in FIG. 2; FIG. 3 is a diagram for explaining a divided video extracting process shown in FIG. 2; FIG. 3 is a diagram showing the structure of music information and synthesis information read in a second reading step shown in FIG. 2; FIG. 3 is a diagram for explaining the first half of the content synthesizing process shown in FIG. 2; FIG. 3 is a diagram for explaining the second half of the content synthesizing process shown in FIG. 2; FIG. FIG. 5 is a flowchart of a moving image content generation method according to Modification 1 of the present invention; FIG. 11 is a block diagram showing a schematic configuration of a smartphone that is executing a moving image content generation program according to Modification 2 of the present invention; FIG. 10 is a flowchart of a moving image content generation method according to Modification 2 of the present invention; FIG. 11 is another flowchart of the moving image content generation method according to Modification 2 of the present invention. FIG. 11 is a block diagram showing a schematic configuration of a smartphone that is executing a moving image content generation program according to Modification 3 of the present invention; FIG. 10 is a diagram showing the structures of music information, synthesis information, and special effect information read in the second reading step of the moving image content generation method according to Modification 3 of the present invention;

An embodiment of a moving image content generation method and generation program according to the present invention will be described below with reference to the accompanying drawings.

[Example]
FIG. 1 shows a schematic configuration of the smartphone 10. As shown in FIG. As shown in the figure, the smartphone 10 includes an information processing device 11 consisting of an MPU (Micro Processor Unit), a storage device 12 consisting of a memory, a camera 13, a microphone 14, a touch panel display 15, and a speaker 16. I have.

The information processing device 11 has a first material content generation unit 20 and a moving image content generation unit 21 . Among these, the first material content generation unit 20 is a functional block formed in the information processing apparatus 11 when a moving image shooting program provided as standard is executed by the user. On the other hand, the moving image content generation unit 21 is a functional block formed within the information processing device 11 when the moving image content generation program according to the present embodiment is executed by the user.

The first material content generation unit 20 generates a first material content (moving image file) including video information and audio information based on the video signal output from the camera 13 and the audio signal output from the microphone, and Store in the storage device 12 .

In addition to the first material content, the storage device 12 further stores second material content, which is music information, and synthesis information corresponding thereto. The second material content and the synthesis information may be stored in advance, or may be stored when the moving image content generation program according to the present embodiment is executed by the user. The second material content and synthesis information will be described later in detail.

Note that in this embodiment, when the user executes the moving image content generation program according to this embodiment, one each of the first material content, the second material content, and the synthesis information are already stored in the storage device 12. and

The moving image content generation unit 21 is formed when the moving image content generation program according to the present embodiment is executed by the user, as described above. By executing the moving image content generating method according to the present embodiment, the moving image content generation unit 21 synthesizes the first material content and the second material content stored in the storage device 12 based on the synthesis information, and generates a new moving image. Generate content. In other words, the moving image content generation program according to this embodiment generates new moving image content by causing the information processing apparatus 11 to execute the moving image content generation method according to this embodiment.

The moving image content generation unit 21 can reproduce the generated moving image content via the touch panel display 15 and the speaker 16 . In addition, the moving image content generation unit 21 can upload the generated moving image content via a communication unit (not shown) or store it in the storage device 12 .

Next, a moving image content generation method (that is, operation of the moving image content generation unit 21) according to the present embodiment will be described with reference to FIG.

In the first reading step S<b>1 , the moving image content generation unit 21 reads the first material content from the storage device 12 . As described above, the first material content is a file related to moving images shot by the user himself/herself, and includes video information and audio information.

In a content separation step S2 that is executed after step S1, the moving image content generator 21 separates the first material content read in step S1 into video information and audio information (see FIG. 3).

In an extraction start point determination step S3 that is executed after step S2, the moving image content generation unit 21 analyzes the audio information separated in step S2, and determines the top N volume peak points included in the audio information. (where N is an integer equal to or greater than 2. In this embodiment, N=7) are specified, and the time corresponding to these is set as the extraction start point. As shown in FIG. 4, in this embodiment, times t1, t2, . . . , t6, t7 corresponding to volume peak points P1, P2, .

In the divided video extraction step S4 that is executed after step 3, the moving image content generation unit 21 extracts the seven extraction start points t1, t2, . , t6, t7 are the starting points, and seven pieces of divided video information V1, V2, . . . , V6, V7 are extracted (see FIG. 5). More specifically, the moving image content generator 21 extracts information between extraction start points t1 and t2 as divided video information V1, extracts information between extraction start points t2 and t3 as divided video information V2, and so on. , the information between the extraction start point t6 and t7 is extracted as the divided video information V6, and the information between the extraction start point t7 and the end of the video information is extracted as the divided video information V7.

In a second reading step S5 that is executed after step S4, the moving image content generation unit 21 reads the second material content and the synthesis information corresponding thereto from the storage device 12 .

FIG. 6A shows music information included in the read second material content. Also, FIG. 6B shows the read synthesis information. As is clear from these figures, the synthesized information includes a plurality of video switching points SW1, SW2, . , SW15, and SW16 belong to one of seven groups C3, D3, E3, F3, G3, A3, and B3. For example, video switching points SW1 and SW14 belong to group C3, and video switching points SW10, SW11, SW12, and SW13 belong to group F3.

Synthesis information is described in MIDI (Musical Instrument Digital Interface) format. By using the MIDI format, it is possible to easily describe the temporal positions of a plurality of video switching points belonging to any one of seven groups C3, D3, E3, F3, G3, A3 and B3. Note that C3, D3, E3, F3, G3, A3 and B3 are the notes of the third octave "Do", "Re", "Mi", "Fa", "So", "La" and "Si", respectively. ” means.

In the content synthesizing step S6 that is executed after the step S5, the moving image content generation unit 21 extracts the video switching points SW1, SW2, . The seven pieces of divided video information V1, V2, . . . , V6, V7 and the music information included in the second material content are synthesized.

More specifically, the moving image content generation unit 21 first divides the seven pieces of divided image information V1, V2, . and In this embodiment, this association is performed randomly, and as a result, group C3 and divided video information V1, group D3 and divided video information V3, group E3 and divided video information V5, group F3 and divided video information V6, group G3. and divided video information V2, group A3 and divided video information V4, and group B3 and divided video information V7 (see FIG. 7).

Subsequently, the moving image content generating unit 21 sets the video switching points SW1 and SW14 belonging to group C3 as the starting point of the divided video information V1, the video switching point SW6 belonging to the group D3 as the starting point of the divided video information V3, and sets the video switching point SW6 belonging to the group D3 as the starting point of the divided video information V3. The video switching point SW5 belonging to the group G3 is the starting point of the divided video information V5, the video switching points SW10, SW11, SW12, and SW13 belonging to the group F3 are the starting points of the divided video information V6, and the video switching point SW15 belonging to the group G3 is the starting point of the divided video information V6. The video switching points SW2, SW4, SW7, SW9, and SW16 belonging to the group A3 are the starting points of the divided video information V4, and the video switching points SW3 and SW8 belonging to the group B3 are the starting points of the divided video information V7. The divided video information V1, V2, .

At this time, for example, if the time between the video switching points S1 and S2 is shorter than the time of the divided video information V1, the moving image content generation unit 21 uses only the first part of the divided video information V1, If the time between S1 and S2 is longer than the time of the divided video information V1, the gap is filled with the last frame (still image) of the divided video information V1.

As described above, according to the moving image content generation method and generation program according to the present embodiment, even if the user does not perform work that requires an aesthetic sense, the voice information included in the first material content created by the user and the pre-generated The video information included in the first material content is edited so as to match the music information (BGM) included in the second material content based on the prepared synthesizing information, and new good-looking moving image content is obtained.

It should be noted that the moving image content generating method and generating program according to the present invention have a plurality of modified examples illustrated below.

[Modification 1]
The second reading step S5 may be performed before the first reading step S1 (see FIG. 9A), or may be performed concurrently with the first reading step S1-divided image extraction step S4. Good (see FIG. 9(B)). That is, in the present invention, it is sufficient that the first reading step S1—divided video extracting step S4 and the second reading step S5 are performed before the content synthesizing step S6.

[Modification 2]
As shown in FIG. 10, the storage device 12 may store a plurality of first material contents, a plurality of second material contents, and synthesis information corresponding to each of the plurality of second material contents. In this case, as shown in FIG. 11, it is necessary to execute the selection command receiving step S7 before the first reading step S1 and the second reading step S5.

In the selection command receiving step S7, the moving image content generation unit 21 causes the touch panel display 15 to display information on the plurality of first material contents and the second material contents stored in the storage device 12 as options, and causes the touch panel display 15 to be displayed. It accepts a command from the user regarding the selection of the first material content and the second material content input via. Then, in the first reading step S1, the moving image content generation unit 21 reads the first material content selected by the user. The same applies to the second reading step S5.

When executing the first reading step S1 and the second reading step S5 in series, a selection command receiving step S7 for receiving a command regarding selection of the first material content and the second material content may be performed before the first reading step S1. (See FIG. 12A), or a selection command receiving step S7A for receiving a command regarding selection of the first material content is performed before the first reading step S1, and selection of the second material content is performed. A selection command receiving step S7B for receiving a command regarding the selection command may be executed before the second reading step S5 (see FIG. 12(B)).

[Modification 3]
As shown in FIG. 13, the storage device 12 may further store special effect information corresponding to the second material content. In this case, the moving image content generation unit 21 reads the composition information and the special effect information along with the second material content in the second reading step S5. However, the moving image content generation unit 21 reads only the synthesis information together with the second material content for the second material content that does not have the special effect information.

14, the special effect information includes a plurality of special effect application periods EF1, EF2, . . . , EF6, EF7. , EF6, and EF7 belong to one of four groups C4, D4, E4, and F4.

Special effect application periods EF1, EF4, and EF6 belonging to group C4 are periods in which the special effect "enlargement" is applied, and special effect application period EF7 belonging to group D4 is a period in which the special effect "fade out" is applied, Special effect application periods EF2 and EF5 belonging to group E4 are periods in which the special effect "fast forward" is applied, and special effect application period EF3 belonging to group F4 is a period in which the special effect "slow motion" is applied.

Special effect information is described in MIDI format. By using the MIDI format, it is possible to easily describe the temporal range of the special effect application period belonging to one of the four groups C4, D4, E4 and F4. Note that C4, D4, E4 and F4 respectively mean the notes of the fourth octave "do", "re", "mi" and "fa".

In the content synthesizing step S6, the moving image content generation unit 21 applies a special effect according to the special effect information to the piece of jointed plural pieces of divided video information. Then, the moving image content generating unit 21 synthesizes the video information and the music information after applying the special effects.

To increase the number of types of special effects, for example, G4, A4, and B4, which mean "So", "La", and "Si" of the scale of the fourth octave, may be used. Also, the synthesis information and special effect information may be a single file described in MIDI format.

[Other Modifications]
(1) The moving image content generation unit 21 may determine the extraction start point based on something other than the volume peak point included in the audio information of the first material content. For example, the moving image content generation unit 21 may set the point at which the volume of the audio information suddenly increases/decreases as the extraction start point.

(2) After executing the content composition step S6, the moving image content generation unit 21 may receive a command as to whether or not it is necessary to re-randomly associate the N pieces of divided video information with the N groups. By redoing the association, various moving image contents can be generated without changing the first material content itself and the second material content itself.

(3) The moving image content generator 21 may associate the N pieces of divided video information with the N groups according to a predetermined rule instead of randomly.

(4) The moving image content generation unit 21 may receive an instruction from the user regarding determination of the extraction start point before executing the extraction start point determination step S3. This command includes specification of the range to be analyzed in the extraction start point determination step S3 and direct specification of the extraction start point. Note that when the user designates M extraction start points (where M is an integer of 1 or more and N or less), the moving image content generation unit 21 analyzes the audio information to obtain (NM) extraction start points. Determine the starting point. Further, when the extraction start point is designated by the user, the moving image content generation unit 21 divides the divided video information starting from the designated extraction start point into a group including many video switching points (in the example shown in FIG. 6, , to a group A3) including five video switching points SW2, SW4, SW7, SW9, and SW16.

(5) The moving image content generator 21 may receive a command regarding determination of the extraction start point after executing the content synthesizing step S6. When receiving the instruction, the moving image content generation unit 21 executes again the extraction start point determination step S3, the divided video extraction step S4, and the content synthesis step S6. Note that, after executing the content composition step S6, the moving image content generation unit 21 may receive a command regarding whether or not to redo the association in addition to the command regarding determination of the extraction start point.

(6) In the selection command receiving steps S7 and S7B, instead of presenting the user with information about the second material content, the moving image content generation unit 21 presents options related to the melody (for example, happy, sad, etc.) and options related to the tempo of the song. (eg, slow, fast, etc.) may be presented to the user. In this case, the user indirectly selects the second material content by selecting the tune and tempo.

(7) The audio information included in the first material content may consist of right audio information and left audio information. In this case, for example, the extraction start point can be determined based on one of the right audio information and the left audio information, or an average of these.

(8) Synthesis information and special effect information may be described in formats other than MIDI.

(9) The information processing device 11 is not limited to one provided in a smart phone.

10 Smartphone 11 Information Processing Device 12 Storage Device 13 Camera 14 Microphone 15 Touch Panel Display 16 Speaker 20 First Material Content Generation Unit 21 Video Content Generation Unit

Claims (8)

A video content generation method for generating video content from first material content including video information and audio information,
a first reading step of reading the first material content from a storage device;
a content separation step of separating the read first material content into the video information and the audio information;
an extraction start point determination step of determining N extraction start points (where N is an integer equal to or greater than 2) based on the separated audio information;
a divided image extracting step of extracting N pieces of divided image information starting from each of the N extraction start points from the separated image information;
a second reading step of reading second material content, which is music information, and synthesis information corresponding to the second material content from the storage device;
a content synthesizing step of synthesizing the music information and the divided video information based on the synthesizing information to generate the video content;
with
the synthesis information includes a plurality of video switching points synchronized with the beat represented by the music information;
The plurality of video switching points belong to one of N groups,
A moving image content generating method, wherein in the content synthesizing step, each of the video switching points belonging to the same group is set as a starting point of any of the N divided video information. 2. The moving image content according to claim 1, wherein, in said extraction start point determination step, the extraction start point is set to the top N times of a plurality of volume peak points included in said separated audio information. generation method. further comprising a command receiving step of receiving a command regarding selection of the first material content and the second material content;
In the first reading step, reading one of the plurality of first material contents pre-stored in the storage device based on the command;
In the second reading step, one of the plurality of second material contents stored in advance in the storage device and the synthesis information corresponding to the one are read based on the command. 3. The moving image content generating method according to claim 1 or 2. The synthesis information is described in MIDI format,
4. The moving image content generation method according to any one of claims 1 to 3, wherein the N groups correspond to musical scales. A video content generation program for causing an information processing apparatus to execute a video content generation method for generating video content from first material content including video information and audio information,
In the information processing device,
a first reading step of reading the first material content from a storage device;
a content separation step of separating the read first material content into the video information and the audio information;
an extraction start point determination step of determining N extraction start points (where N is an integer equal to or greater than 2) based on the separated audio information;
a divided image extracting step of extracting N pieces of divided image information starting from each of the N extraction start points from the separated image information;
a second reading step of reading second material content, which is music information, and synthesis information corresponding to the second material content from the storage device;
a content synthesizing step of synthesizing the music information and the divided video information based on the synthesizing information to generate the video content;
and
the synthesis information includes a plurality of video switching points synchronized with the beat represented by the music information;
The plurality of video switching points belong to one of N groups,
A moving image content generating program, wherein in the content synthesizing step, each of the video switching points belonging to the same group is set as a starting point of any one of the N divided video information. 6. The extraction start point according to claim 5, wherein, in said extraction start point determination step, a time corresponding to top N of a plurality of volume peak points included in said separated audio information is set as said extraction start point. video content generator. causing the information processing device to further execute a command receiving step of receiving a command regarding selection of the first material content and the second material content;
In the first reading step, one of the plurality of first material contents pre-stored in the storage device is read based on the command,
In the second reading step, one of the plurality of second material contents pre-stored in the storage device and the synthesis information corresponding to the one are read based on the command. 7. The moving image content generation program according to claim 5 or 6. The synthesis information is described in MIDI format,
8. The moving image content generation program according to any one of claims 5 to 7, wherein the N groups correspond to musical scales.

Images