CN114584803A - Video generation method and computer equipment - Google Patents

Abstract

The present invention provides a video generation method and computer equipment. The video generation method includes: acquiring an image set to be processed and audio to be processed, wherein the image set includes several images; audio, determining the target beat point corresponding to each image in the several images; inserting the image corresponding to the target beat point in each target beat point in the audio to generate a video. In the present invention, each image is inserted at the target beat point of the audio, which can improve the expressiveness of the video and obtain a video with better quality.

Description

A method for generating video and computer equipment

technical field

The present application relates to the technical field of video processing, and in particular, to a video generation method and computer equipment.

Background technique

Video production is a complex process. To get a high-quality video, you need to organize the images, select the appropriate audio, and then determine the insertion point of the image. This process has relatively high requirements on the technical ability of the producer.

For ordinary people who do not have video production skills, they usually select images manually, and play the background audio in the form of a slideshow. The video obtained in this way has poor expressiveness and low quality.

Therefore, the existing technology needs to be improved.

SUMMARY OF THE INVENTION

The invention provides a video generation method and computer equipment, so as to realize the continuous playback of similar images, and each image is inserted at the beat point of the audio, which can improve the expressive power of the video and obtain the video with better quality.

In a first aspect, an embodiment of the present invention provides a method for generating a video, including:

acquiring a set of images to be processed and audio to be processed, wherein the set of images includes several images;

Based on the image set and the audio, determining the target beat point corresponding to each image in the plurality of images;

Inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video.

In a further improvement scheme, determining the target beat point corresponding to each image in the plurality of images based on the image set and the audio, specifically includes:

Based on the similarity between any two images in the image set, sorting the several images to obtain an image insertion sequence;

obtain several beat points of the audio;

The target beat point corresponding to each image in the image insertion sequence is determined according to the image insertion sequence, the several beat points and the audio, wherein the target beat point is a beat point used in several beat points of the audio. at the beat point of the inserted image.

In a further improvement scheme, the plurality of images are sorted based on the similarity between any two images in the image set to obtain an image insertion sequence, which specifically includes:

Select an initial image in the image set, and set the insertion sequence number of the initial image as the first sequence number;

determining a non-initial image set corresponding to the initial image, wherein the non-initial image set includes several non-initial images;

Based on the similarity between each non-starting image and the starting image, a candidate image corresponding to the starting image is determined, and the insertion sequence number of the candidate image is set to be equal to the insertion sequence number of the starting image. the next serial number;

Taking the candidate image as the starting image, and continuing to perform the step of determining the non-starting image set corresponding to the starting image, until the respective insertion sequence numbers of all the images in the image set are determined;

The image insertion sequence corresponding to the image set is determined according to the corresponding insertion sequence numbers of all the images in the image set.

In a further improvement scheme, the determining the non-starting image set corresponding to the starting image specifically includes:

For the starting image, all images whose insertion sequence numbers are not determined are selected from the image set to obtain a non-starting image set corresponding to the starting image.

In a further improvement scheme, determining the candidate image corresponding to the starting image based on the similarity between each non-starting image in the set of non-starting images and the starting image specifically includes:

respectively calculating the similarity between each non-initial image in the non-initial image set and the initial image to obtain a similarity set;

The maximum similarity is selected from the similarity set, and the image corresponding to the maximum similarity is used as the candidate image corresponding to the initial image.

In a further improved solution, determining the target beat point corresponding to each image in the image insertion sequence according to the image insertion sequence, the several beat points and the audio, specifically includes:

Obtain the audio duration corresponding to the audio and the number of images of the several images;

An image insertion point corresponding to each image in the image insertion sequence is determined according to the number of images and the audio duration, wherein the duration between the two adjacent image insertion points is determined according to the audio duration and the audio duration. determined by the number of said images;

The target beat points corresponding to the several images are determined according to the several image insertion points and the several beat points.

In a further improved solution, determining the target beat points corresponding to the several images according to the several image insertion points and the several beat points specifically includes:

For each image insertion point, the beat point that is closest to the image insertion point is determined among the several beat points, and the beat point that is closest to the image insertion point is used as the target of the image corresponding to the image insertion point beat point.

In a further improvement scheme, the image corresponding to the target beat point is inserted into each target beat point in the audio to generate a video, which specifically includes:

For each target beat point, when the audio playback time reaches the target beat point, insert the image corresponding to the target beat point, and use the image as the image played between the target beat point and the next target beat point image frame.

In a further improvement scheme, before the acquiring the image set to be processed and determining the image insertion sequence based on the similarity between any two images in the image set, the method further includes:

acquiring an original image set, wherein the original image set includes multiple original images, and the multiple original images include at least one template image;

Determine to extract the target feature map corresponding to each original image in the original image set;

dividing the original image set into original image subsets of different categories based on all the determined target feature maps;

Any original image subset including the template image is used as the image set to be processed.

In a second aspect, the present invention provides a video generation device, comprising:

an acquisition unit, configured to acquire a set of images to be processed and audio to be processed, wherein the set of images includes several images;

a target beat point determination unit, configured to determine the target beat point corresponding to each of the several images based on the image set and the audio;

A video generation unit, configured to insert an image corresponding to the target beat point in each target beat point in the audio to generate a video.

In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

acquiring a set of images to be processed and audio to be processed, wherein the set of images includes several images;

Based on the image set and the audio, determining the target beat point corresponding to each image in the plurality of images;

Inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

acquiring a set of images to be processed and audio to be processed, wherein the set of images includes several images;

Based on the image set and the audio, determining the target beat point corresponding to each image in the plurality of images;

Inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video.

Compared with the prior art, the embodiment of the present invention has the following advantages:

The present invention provides a method for generating a video, comprising: acquiring an image set to be processed and audio to be processed, wherein the image set includes several images; and determining the several images based on the image set and the audio The target beat point corresponding to each image in the audio; insert the image corresponding to the target beat point in each target beat point in the audio to generate a video. In the present invention, each image is inserted at the beat point of the audio, which can improve the expressive power of the video and obtain a video with better quality.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a schematic flowchart of a method for generating a video in an embodiment of the present invention;

2 is a schematic diagram of a feature extraction network in an embodiment of the present invention;

3 is a schematic structural diagram of a video generation apparatus in an embodiment of the present invention;

FIG. 4 is an internal structural diagram of a computer device in an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and effects of the present invention clearer and clearer, the present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

It will be understood by those skilled in the art that the singular forms "a", "an", "the" and "the" as used herein can include the plural forms as well, unless expressly stated otherwise. It should be further understood that the word "comprising" used in the description of the present invention refers to the presence of stated features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components and/or groups thereof. It will be understood that when we refer to an element as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Furthermore, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combination of one or more of the associated listed items.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms, such as those defined in a general dictionary, should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, should not be interpreted in idealistic or overly formal meaning to explain.

After research, the inventor found that video production is a complicated process. To get a high-quality video, it is necessary to organize the images, select the appropriate audio, and then determine the insertion point of the image. This process has relatively high requirements on the technical ability of the producer. For ordinary people who do not have video production skills, they usually select images manually, and play the background audio in the form of a slideshow. The video obtained in this way has poor expressiveness and low quality.

In order to solve the above problem, in this embodiment of the present invention, an image set to be processed and audio to be processed are acquired, wherein the image set includes several images; based on the image set and the audio, it is determined that among the several images Each image corresponds to a target beat point; inserting an image corresponding to the target beat point in each target beat point in the audio to generate a video. In the present invention, each image is inserted at the beat point of the audio, which can improve the expressive power of the video and obtain a video with better quality.

A method for generating a video provided by an embodiment of the present invention can be applied to electronic equipment, and the electronic equipment may include: including a PC, a TV, a server, a mobile phone, a tablet computer, a handheld computer, and a Personal Digital Assistant (PDA). )Wait.

In the following, the content of the invention will be further illustrated by describing the embodiments in conjunction with the accompanying drawings.

Referring to FIG. 1, this embodiment provides a method for generating a video, including:

S1. Acquire an image set to be processed and audio to be processed, wherein the image set includes several images.

In this embodiment of the present invention, the set of images to be processed includes several images used for generating a video. The to-be-processed image set may be collected by a terminal executing the video generation method, or the to-be-processed image set may be obtained from a third-party device, or the image portion of the to-be-processed image set comes from For third-party devices, some are collected by terminals.

In this embodiment of the present invention, the audio is used to generate a video. In order to obtain a better video effect, the audio may be audio corresponding to music, and the audio is used as the background music of the video.

In this embodiment of the present invention, in order to make images in the image set to be processed have similar styles and obtain a video with a more harmonious picture, the images in the image set may be limited to belong to the same category. Specifically, images belonging to the same category may be selected from the original image set, and the selected images belonging to the same category may be used as the image set to be processed.

Specifically, before step S1, it includes:

M1. Acquire an original image set, wherein the original image set includes multiple original images, and the multiple original images include at least one template image.

In this embodiment of the present invention, the original image set includes a plurality of original images. Similarly, the original images may be acquired from a network, or the original images may be collected by a terminal. At least one template image is included in the plurality of original images. When selecting an image for generating a video from multiple original images, the priority of the template image is higher than that of the non-template image, and the non-template image refers to other original images in the multiple original images except the template image; That is, for a template image and a non-template image, the template image can be selected as the image for generating the video.

The template image may be a user-defined selection, that is, the user specifies a template image among multiple original images, and the template image may be the user's favorite original image, indicating that the user most wishes the template image to appear in the video. The template image may be multiple.

M2. Determine a target feature map corresponding to each original image in the original image set.

In this embodiment of the present invention, the target feature map corresponding to each original image can be determined by using a neural network model. The target feature map is a multi-channel image with an image size of 1*1 (only one pixel), the target feature map can be represented by a vector, and the dimension of the vector is the same as the number of channels of the target feature map. The value is the pixel value of the pixel in the channel corresponding to this dimension.

Input an original image into the neural network model, and the output item of the neural network model is the target feature map corresponding to the original image. The neural network model includes a feature extraction module and a fully connected module; the original image is input through the feature extraction module to obtain a feature map corresponding to the original image, and the feature map is input into the fully connected module, and the output item of the fully connected module is the target feature map .

In the embodiment of the present invention, the image size of the original image input to the neural network model needs to meet the input requirements of the neural network model, and the image size of all original images needs to be adjusted to a preset size in advance. For example, the preset size is 224*224.

In a specific implementation, the neural network model may be a VGG16 network model. The feature extraction network used to generate the feature atlas is shown in Figure 2. The neural network model includes five feature extraction modules and one fully connected module, wherein the five feature extraction modules include: a first feature extraction module, a second feature extraction module Feature extraction module, third feature extraction module, fourth feature extraction module and fifth feature extraction module.

The first feature extraction module includes: the first convolution layer c1, the second convolution layer c2 and the first pooling layer p1; the size of the convolution kernels of c1 and c2 are both 3*3*3, and the volumes of c1 and c2 The number of product kernels is 64, and the parameter of p1 is 2*2; the input item of the first feature extraction module is an original image, and the first feature extraction module extracts the features of the original image to obtain the first feature picture.

The second feature extraction module includes: the third convolution layer c3, the fourth convolution layer c4 and the second pooling layer p2; the size of the convolution kernels of c3 and c4 are both 3*3*3, the volumes of c3 and c4 The number of product kernels is 128, and the parameter of p2 is 2*2; the input item of the second feature extraction module is the first feature map, and the second feature extraction module extracts the features of the first feature map, to get the second feature map.

The third feature extraction module includes: the fifth convolutional layer c5, the sixth convolutional layer c6, the seventh convolutional layer c7 and the third pooling layer p3; the size of the convolution kernels of c5, c6 and c7 are all 3* 3*3, the number of convolution kernels of c5, c6 and c7 are all 256, and the parameter of p3 is 2*2; the input item of the third feature extraction module is the second feature map, and the third feature extraction The module extracts features of the second feature map to obtain a third feature map.

The fourth feature extraction module includes: the eighth convolution layer c8, the ninth convolution layer c9, the tenth convolution layer c10 and the fourth pooling layer p4; the size of the convolution kernels of c8, c9 and c10 are all 3* 3*3, the number of convolution kernels of c8, c9 and c10 are all 512, and the parameter of p4 is 2*2; the input item of the fourth feature extraction module is the third feature map, and the fourth feature extraction The module extracts features of the third feature map to obtain a fourth feature map.

The fifth feature extraction module includes: the eleventh convolutional layer c11, the twelfth convolutional layer c12, the thirteenth convolutional layer c13 and the fifth pooling layer p5; the size of the convolution kernels of c10, c11 and c12 are all is 3*3*3, the number of convolution kernels of c11, c12 and c13 are all 512, and the parameter of p5 is 2*2; the input item of the fifth feature extraction module is the fourth feature map, the first The five feature extraction module extracts the features of the fourth feature map to obtain the fifth feature map.

The fully connected module includes the first fully connected layer fc1, the second fully connected layer fc2 and the third fully connected layer fc3. The parameters of the first fully connected layer are: 1*1*4096, and the parameters of the second fully connected layer are 1* 1*4096, the parameters of the third fully connected layer are 1*1*1000. The fifth feature map is input into the fully connected module to obtain the target feature map corresponding to the original image. The target feature map can be represented in the form of a vector, for example, the target feature map can be represented as: {x1,x2,...,xn}.

M3. Divide the original image set into original image subsets of different categories based on all the determined target feature maps.

In the embodiment of the present invention, a classification method is used to divide the original image set into multiple original image subsets, and the categories corresponding to each original image subset are different from each other. For example, the original image set may be divided into different categories of original image subsets by a K-means algorithm. A subset of original images includes original images that are similar in style.

Specifically, k target feature maps are randomly selected as initial centroids in all target feature maps. For example, if k is set to 3, the initial centroids can be obtained as: u1, u2, and u3 respectively. The target feature maps other than the initial centroid in all target feature maps are recorded as the feature map to be classified, and for each feature map to be classified, the feature map to be classified and each initial centroid are calculated separately: u1, u2 and u3. The initial centroid corresponding to the minimum distance and the feature map to be classified are divided into the same class to obtain several classification sets.

For example, there are 10 target feature maps, including: t1, t2, ..., t10, three target feature maps are randomly selected from the 10 target feature maps: t1, t2 and t3, t1 is recorded as the initial centroid u1, and t2 Denote it as the initial centroid u2, and denote t3 as the initial centroid u3. The feature maps to be classified include: t4, t5,..., t10. For each feature map to be classified, the distance between the feature map to be classified and each initial centroid is calculated. For example, for t4, calculate the distance between t4 and u1, get d41, calculate the distance between t4 and u2, get d42, calculate the distance between t4 and u3, get d43, assuming d43 is the smallest, then t4 corresponds to The original image and the original image corresponding to u3 are divided into one category. The above calculation is performed on all feature maps to be classified, and three classification sets are obtained.

In the embodiment of the present invention, for each classification set in several classification sets, the classification centroid corresponding to the classification set is determined, and for each to-be-classified feature map in the above-mentioned to-be-classified feature map, the to-be-classified feature map and each to-be-classified feature map are calculated. The distance between a classification centroid, the classification centroid corresponding to the minimum distance and the feature map to be classified are divided into the same class to obtain several updated classification sets.

The classification centroid corresponding to the classification set can be determined by formula (1).

where Cj is the classification set, t is the target feature map belonging to the classification set Cj, and uj is the classification centroid.

In the embodiment of the present invention, the steps of: calculating the classification corresponding to the classification set: "for each classification set in several classification sets, then determine the classification centroid corresponding to the classification set", until the classification centroid is the same as the one calculated last time. If the classification centroids are the same, several classification sets are used as the original image subsets of different categories.

M4. Use any original image subset including the template image as the to-be-processed image set.

In this embodiment of the present invention, an original image subset including the template image is determined from several original image subsets, and if the number of original image subsets including the template image is greater than 1, randomization is performed in the multiple original image subsets including the template image. Identify a set of images to be processed. Alternatively, several template images have respective corresponding degrees of preference, and the degrees of preference may be set by the user, and the original image subset including the template image with the highest degree of preference is used as the image set to be processed. Alternatively, the set of images to be processed is determined in several original image subsets including template images in combination with the degree of preference and the number of original images in the original image subsets.

S2. Determine, based on the image set and the audio, a target beat point corresponding to each image in the plurality of images.

In the embodiment of the present invention, by analyzing the signal corresponding to the audio, several beat points corresponding to the audio can be determined, and the beat point is the moment when a musical instrument plays a specific note, such as a drum beat. The audio includes a large number of beat points, and any two different beat points correspond to different playback moments. The target beat point is substantially a beat point in the audio, and the target beat point is a beat point used for inserting an image among several beat points of the audio.

In the embodiment of the present invention, the playing sequence of the several images is determined first, and then the target beat points of the images are determined according to the playing sequence of the several images. The playback order of images in the video can be determined based on the similarity between any two images in several images, and similar images can be set to be played continuously, so that when the video is played, the style of the adjacent playing images will not appear too jumpy. Happening.

Specifically, step S2 includes:

S21. Based on the similarity between any two images in the image set, sort the several images to obtain an image insertion sequence.

In this embodiment of the present invention, the image insertion sequence includes several images and respective insertion sequence numbers corresponding to each image, and several images in the image insertion sequence may be arranged according to the insertion sequence numbers. When generating a video, insert several images into the audio in order of insertion sequence number from small to large.

In this embodiment of the present invention, for two adjacent images in the image insertion sequence, the two images are a first image and a second image respectively, wherein the first image is arranged before the second image, and the first image is the same as the second image. The similarity between the second images is greater than the similarity between any image arranged after the second image and the first image.

Specifically, step S21 includes:

S211. Select an initial image from the image set, and set the insertion sequence number of the initial image as the first sequence number.

In this embodiment of the present invention, the starting image may be randomly selected, and the insertion sequence number corresponding to the starting image is set as the first sequence number, and the first sequence number may be represented by a number, for example, the number 1 represents the first sequence number. a serial number.

S212. Determine a non-start image set corresponding to the start image, wherein the non-start image set includes several non-start images.

In this embodiment of the present invention, for the starting image, all images with undetermined insertion numbers are selected from the image set, and the images with undetermined insertion numbers are regarded as non-starting images, and based on all the non-starting images A non-start image set corresponding to the start image is obtained. That is to say, the non-starting image set includes several non-starting images, and the non-starting images are images for which insertion sequence numbers are not set.

For example, the image set includes images: r1, r2, r3, . . . , r8. In the preceding steps, it has been determined that the insertion sequence number of r1 is the first sequence number, then the insertion sequence number of r2, r3, ..., r8 has not been determined, so r2, r3, ..., r8 are non-initial images, and the non-initial image set is: { r2,r3,…,r8}.

S213. Determine a candidate image corresponding to the starting image based on the similarity between each non-starting image and the starting image, and set the insertion sequence number of the candidate image as the insertion sequence number of the starting image The sequence number after the sequence number.

In this embodiment of the present invention, "setting the insertion sequence number of the candidate image to the sequence number after the insertion sequence number of the starting image" means that the candidate image is located in the starting image in the image insertion sequence next. For each non-initial image, the similarity between the non-initial image and the initial initial image is calculated.

Specifically, step S213 includes:

S2131. Calculate the similarity between each non-initial image in the non-initial image set and the initial image, respectively, to obtain a similarity set.

In the embodiment of the present invention, for each non-initial image, the target feature map corresponding to the non-initial image and the target feature map of the initial image are obtained, and the relationship between each non-initial image and the initial image is calculated. The similarity between the images is obtained, and the similarity corresponding to each non-starting image is obtained, and then the similarity set is obtained.

Specifically, the similarity between the non-initial image and the initial image can be calculated by formula (2).

Among them, the characteristics of the starting image rx are tx={x1,x2,...,xn}, the characteristics of the non-starting image ry are ty={y1,y2,...yn}, SIM(x,y) is the starting image Similarity between rx and non-starting image ry.

S2132. Select the maximum similarity in the similarity set, and use the image corresponding to the maximum similarity as a candidate image corresponding to the initial image.

In the embodiment of the present invention, the similarity is represented by a numerical value, the smaller the numerical value, the smaller the similarity, and the greater the numerical value, the greater the similarity. The maximum similarity refers to the similarity with the largest numerical value among all the similarities. The image corresponding to the maximum similarity is taken as the candidate image corresponding to the starting image.

For example, r1 is the starting image, r2, r3,..., r8 are non-starting images, where the similarity between r3 and r1 is the maximum similarity in the similarity set, then r3 is used as the candidate image of r1, and r3 The corresponding insertion sequence number is the sequence number after the insertion sequence number of r1. Suppose the insertion sequence number corresponding to r1 is 1, indicating that the first inserted image is r1, then the insertion sequence number corresponding to r3 is 2, indicating that the second inserted image is r3, where the insertion sequence number "2" is the insertion sequence number "1" ' next. In the image insertion sequence, the image r3 is arranged one bit after the image r1.

S214. Use the candidate image as the starting image, and continue to perform the step of determining the non-starting image set corresponding to the starting image until the respective insertion sequence numbers corresponding to all the images in the image set are determined.

In this embodiment of the present invention, after steps S211 to S213, only the insertion sequence numbers of the two images are determined, and the insertion sequence numbers of other images in the image set to be processed also need to be determined. For the convenience of description, the candidate image is used as the starting image , and continue to perform step S212 to determine a candidate image corresponding to the starting image.

For example, after steps S211 to S123, it is determined that the insertion sequence number of r1 is 1 and the insertion sequence number of r3 is 2. Next, the candidate image corresponding to r3 needs to be determined, that is, the next image of r3 during playback. Taking r3 as the starting image, first determine the non-starting image corresponding to r3 (starting image). As explained above, the non-starting image is all the images whose insertion sequence number is not determined. In this example, since r1 and r3 have been determined Insert the sequence number, so the non-starting images corresponding to r3 include: r2, r4, r5, ..., r8. Calculate the similarity between r3 and each non-starting image (r2, r4, r5, ..., r8) respectively, and determine the candidate image corresponding to r3. Assuming that the candidate image corresponding to r3 is r7, the insertion sequence number of r7 is r3 The last bit of the corresponding insertion sequence number, in the image insertion sequence, r7 is arranged at the last bit of r3. Step S212 is then executed to determine the candidate images corresponding to r7 until the respective insertion sequence numbers corresponding to all the images in the image set are determined.

S215. Determine the image insertion sequence corresponding to the image set according to the respective insertion sequence numbers of all the images in the image set.

In the embodiment of the present invention, after determining the corresponding insertion sequence numbers of all the images, all images are sorted according to the respective insertion sequence numbers corresponding to all the images, so as to obtain the image insertion sequence. For an image in the image insertion sequence, the similarity between the image and the image arranged at the last two positions of the image is greater than the similarity between the image and the images arranged at the last two positions of the image.

For example, the images included in the image set are: r1, r2, r3, ..., r8. After step S211 to step S214, the corresponding insertion sequence number of each image in r1, r2, r3, ..., r8 can be obtained, and then the corresponding insertion sequence number can be obtained. The image insertion sequence is: r1, r3, r7, r6, r2, r8, r5, r4. Among them, the similarity between r7 and r6 is greater than the similarity between r7 and r2 (r8, r5, r4).

In this embodiment of the present invention, the greater the similarity between the two images, the more similar the styles between the two images are. Through the image insertion sequence determined by the similarity between any two images in the image set, the images are played according to the sequence of the image insertion sequence, so that images with similar styles can be played continuously, and the video picture is more harmonious.

S22. Acquire several beat points of the audio.

In this embodiment of the present invention, by analyzing the signal corresponding to the audio, several beat points corresponding to the audio can be determined. The beat is the moment when the instrument plays a particular note, such as a drum beat. The audio includes a large number of beat points, and any two different beat points correspond to different playback moments.

Specifically, step S22 includes:

S221. Acquire an initial signal corresponding to the audio.

In this embodiment of the present invention, the initial signal is a time domain signal corresponding to the audio, the abscissa of the initial signal is time, and the ordinate of the initial signal is the energy of the audio signal.

In this embodiment of the present invention, there is some noise in the initial signal. In order to obtain a more accurate beat point in the subsequent steps, and to reduce the amount of data processed in the subsequent steps, the initial signal may be preprocessed to Eliminate interfering signals and reduce the amount of data in the initial signal.

The process of preprocessing the initial signal includes:

Determine multiple central moments in the initial signal, for each central moment, calculate the cumulative value of all energy in the neighborhood of the central moment, and use the cumulative value as the amplitude corresponding to the central moment, and then obtain the preprocessing The post-processed initial signal is used instead of the pre-processed initial signal. Wherein, any two central moments in the plurality of central moments may intersect.

In this embodiment of the present invention, the neighborhood of the central time may be: from a preset time period before the central time to a neighborhood time period after the time. For example, the central moment is t0, the neighborhood of t0 is [t0-tr, t0+tr], and the neighborhood duration may be 10ms.

Specifically, the initial signal is preprocessed by formula (3).

Among them, in the preprocessed initial signal, Wt is the amplitude corresponding to the center time t; tr is the neighborhood duration, and tr can be set to 10ms; An is the amplitude corresponding to time n; t=t+10ms , that is, the amplitude corresponding to time t is determined every 10ms. For example, to determine the amplitude corresponding to 10ms in the preprocessed initial signal, it can be determined according to the corresponding amplitudes of all sampling points in [0, 20ms]; since t=t+10ms, then determine the preprocessed initial signal again The amplitude corresponding to 20ms in [10ms, 30ms] can be determined according to the corresponding amplitudes of all sampling points in [10ms, 30ms].

The amplitude corresponding to time n may be determined by sampling frequency f ₀ , and the sampling point corresponding to time n is n*f ₀ . The amplitude corresponding to the n*f _0th sampling point may be used as the amplitude corresponding to time n. The sampling frequency may be 50Hz.

S222. Perform low-pass filtering on the initial signal to obtain a first signal corresponding to the audio.

In the embodiments of the present invention, in theory, the beats in music are usually generated by low-frequency musical instruments, such as percussion instruments (bass drums, tambourines, etc.). Therefore, it is easier to determine several beat points by analyzing the low-frequency signal corresponding to the audio. First, low-pass filtering is performed on the initial signal to filter out signals larger than a low-frequency threshold, where the low-frequency threshold may be: 200 Hz. The initial signal may be processed through a Gaussian low-pass filter to obtain the first signal.

S223. Determine several target amplitudes according to the first signal.

In the embodiment of the present invention, the first signal is divided into a plurality of signal segments, the maximum amplitude corresponding to each signal segment is obtained, the obtained maximum amplitude corresponding to each signal segment is obtained, and the obtained maximum amplitude corresponding to each signal segment is obtained. The step of obtaining the corresponding maximum amplitude of each signal segment is performed a preset number of times; for each signal segment, if the maximum amplitude of the signal segment obtained by the preset number of times is the same, then the maximum amplitude of the signal segment is the same. value as the target amplitude for this signal segment.

For each execution, the specific process of obtaining the corresponding maximum amplitude of each signal segment is as follows:

The signal segment is determined by setting a window with a preset duration. Specifically, a plurality of signal segments are determined according to the preset duration, and the duration corresponding to each signal segment is the preset duration. For example, if the preset duration corresponding to the set window is L, the duration corresponding to each signal segment is L.

The window is slid over the first signal according to a preset step size to determine signal segments, and the maximum amplitude in each signal segment is determined. The preset step size is a step size in the time dimension, the preset step size may be greater than or equal to the preset duration, and when the preset step size may be greater than or equal to the preset duration, the Any two signal segments in the multiple signal segments do not intersect; the preset step size may be smaller than the preset duration, and when the preset step size is smaller than the preset duration, any one of the multiple signal segments Two adjacent signal segments intersect.

In this embodiment of the present invention, the preset value may be set to 20 times. That is to say, for each signal segment, it is necessary to repeat the preset value times to determine the maximum amplitude corresponding to the signal segment. As the target amplitude value corresponding to the signal segment; if there are any two different maximum amplitude values among the preset value and maximum amplitude values determined by the preset value times, the signal segment has no target amplitude value.

S224. For each target amplitude value in the several target amplitude values, take the time corresponding to the target amplitude value as the beat point corresponding to the target amplitude value.

In this embodiment of the present invention, for a target amplitude value, the target amplitude value is the maximum amplitude value in a signal segment of the first signal (the low frequency signal after filtering out the high frequency); the loudness of the beat point in the low frequency signal is the largest ( Amplitude is the largest), therefore, the beat point can be determined according to the target amplitude. The beat point is the moment when the musical instrument plays a specific note, and the moment corresponding to the target amplitude value is taken as the beat point corresponding to the target amplitude value.

S23. Determine a target beat point corresponding to each image in the image insertion sequence according to the image insertion sequence, the several beat points and the audio.

In this embodiment of the present invention, the image insertion sequence includes several images arranged in an insertion sequence, and the target beat point of each image is determined according to several beat points, the audio and the image insertion sequence. For any two adjacent images in the image insertion sequence, any two adjacent images include the first image played first and the second image played later, when the video playback time reaches the target beat point corresponding to the first image, the first image is inserted. , and insert the second image until the playback time of the video reaches the target beat point corresponding to the second image.

Specifically, step S23 includes:

S231. Obtain the audio duration corresponding to the audio and the number of images of the several images; determine the image insertion point corresponding to each image in the image insertion sequence according to the number of images and the audio duration, wherein the The duration between the two adjacent image insertion points is determined according to the audio duration and the number of images.

In this embodiment of the present invention, the audio duration is the audio duration, and after the video is generated, the video playing duration is equal to the audio duration. The number of images refers to the number of images in the image insertion sequence, that is, the number of images in the image set to be processed. The ratio of the audio duration to the number of images may be calculated to obtain the average playback duration of each image. An image insertion point corresponding to each image can be obtained according to the average playback duration and the audio duration. Wherein, the image insertion point corresponding to the first image in the image insertion sequence may be set as the start time of audio playback.

For example, if the audio duration is 20 seconds, and the image insertion sequence includes 4 images, the playback duration of each image is 5 seconds. Assuming that the image insertion sequence includes: g1, g2, g3 and g4, the image corresponding to g1 can be determined The insertion point is t=0 seconds, the image insertion point corresponding to g2 is t=5 seconds, the image insertion point corresponding to g3 is t=10 seconds, and the image insertion point corresponding to g4 is t=15 seconds.

S232. Determine the target beat points corresponding to the several images according to the several image insertion points and the several beat points.

In this embodiment of the present invention, for each image insertion point, the beat point that is closest to the image insertion point is determined from among the several beat points, and the beat point that is closest to the image insertion point is used as the image The target beat point of the image corresponding to the insertion point.

In the embodiment of the present invention, the so-called "closest distance" refers to the closest distance between the time corresponding to the target beat point and the time corresponding to the image insertion point.

For example, several beat points are: {j1,j2,j3,...,j20}, and several image insertion points are: {c1,c2,...,c5}, for cm, if cm=jn, then jn is taken as cm The target beat point of the corresponding image, where 1≦m≦5, 1≦n≦20; if cm≠jn, determine the jn closest to cm, and use jn as the target beat point of the image corresponding to cm.

For example, the image insertion point corresponding to g3 is t=10 seconds, if there is a beat point: t=10 seconds, then t=10 seconds is taken as the target beat point of the image corresponding to g3. The image insertion point corresponding to g3 is t=10 seconds, and among several beat points, the nearest beat point from t=10 seconds is t=11 seconds, then t=11 seconds is used as the target beat point of the image corresponding to g3.

S3. Insert an image corresponding to the target beat point at each target beat point in the audio to generate a video.

In the embodiment of the present invention, for each target beat point, when the audio playback time reaches the target beat point, an image corresponding to the target beat point is inserted, and the image is used as the target beat point and the next beat point. Image frames played between target beats.

For example, if the audio duration is 20 seconds, and the image insertion sequence includes 4 images, the playback duration of each image is 5 seconds. Assuming that the image insertion sequence includes: g1, g2, g3 and g4, the target corresponding to g1 can be determined The beat point is t=0 seconds, the target beat point corresponding to g2 is t=5.5 seconds, the target beat point corresponding to g3 is t=9.5 seconds, and the target beat point corresponding to g4 is t=16 seconds. When the playback time of the audio is 0 seconds, insert g1, and insert g1 in each frame of the playback time from 0 seconds to 5.5 seconds; when the playback time of the audio is 5.5 seconds, insert g2, and at the playback time of g2 is inserted in each frame from 5.5 seconds to 9.5 seconds; g3 is inserted when the playback time of the audio is 9.5 seconds, and g3 is inserted in each frame from the playback time of 9.5 seconds to 16 seconds; When the time is 16 seconds, g4 is inserted, and g4 is inserted for each frame from 16 seconds to 20 seconds in the playback time.

In the embodiment of the present invention, the generated video can realize: when the video is played, the same image is continuously played between two target beat points, and each target beat point is a beat point. For example, in the above example, g1 is continuously played from 0 seconds to 5.5 seconds, switched to g2 at 5.5 seconds, and g2 is continuously played from 5.5 seconds to 9.5 seconds, where 5.5 seconds is the beat point, that is, the realization of the beat point to switch the playback image.

Based on the above-mentioned method for generating a video, referring to FIG. 3 , an embodiment of the present invention further provides a video generating apparatus, including:

an acquisition unit, configured to acquire a set of images to be processed and audio to be processed, wherein the set of images includes several images;

a target beat point determination unit, configured to determine the target beat point corresponding to each of the several images based on the image set and the audio;

A video generation unit, configured to insert an image corresponding to the target beat point in each target beat point in the audio to generate a video.

In one embodiment, the present invention provides a computer device, the device may be a terminal, and the internal structure is shown in FIG. 4 . The computer equipment includes a processor, memory, a network model interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network model interface of the computer equipment is used to communicate with external terminals through the network model connection. The computer program, when executed by a processor, implements a method of generating a video. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer equipment , or an external keyboard, trackpad, or mouse.

Those skilled in the art can understand that the block diagram shown in FIG. 4 is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. shown in more or less components, or in combination with certain components, or with different arrangements of components.

An embodiment of the present invention provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

acquiring a set of images to be processed and audio to be processed, wherein the set of images includes several images;

Based on the image set and the audio, determining the target beat point corresponding to each image in the plurality of images;

Inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video.

Embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

acquiring a set of images to be processed and audio to be processed, wherein the set of images includes several images;

Based on the image set and the audio, determining the target beat point corresponding to each image in the plurality of images;

Inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (12)

1. a generation method of video, is characterized in that, comprises: acquiring a set of images to be processed and audio to be processed, wherein the set of images includes several images; Based on the image set and the audio, determining the target beat point corresponding to each image in the plurality of images; Inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video. 2. The method for generating a video according to claim 1, wherein the determining the target beat point corresponding to each image in the several images based on the image set and the audio, specifically comprises: Based on the similarity between any two images in the image set, sorting the several images to obtain an image insertion sequence; obtain several beat points of the audio; The target beat point corresponding to each image in the image insertion sequence is determined according to the image insertion sequence, the several beat points and the audio, wherein the target beat point is a beat point used in several beat points of the audio. at the beat point of the inserted image. 3. The method for generating a video according to claim 2, wherein the several images are sorted based on the similarity between any two images in the image set to obtain an image insertion sequence, specifically include: Select an initial image in the image set, and set the insertion sequence number of the initial image as the first sequence number; determining a non-initial image set corresponding to the initial image, wherein the non-initial image set includes several non-initial images; Based on the similarity between each non-starting image and the starting image, a candidate image corresponding to the starting image is determined, and the insertion sequence number of the candidate image is set to be equal to the insertion sequence number of the starting image. the next serial number; Taking the candidate image as the starting image, and continuing to perform the step of determining the non-starting image set corresponding to the starting image, until the respective insertion sequence numbers of all the images in the image set are determined; The image insertion sequence corresponding to the image set is determined according to the corresponding insertion sequence numbers of all the images in the image set. 4. The method for generating a video according to claim 3, wherein the determining a non-starting image set corresponding to the starting image specifically comprises: For the starting image, all images whose insertion sequence numbers are not determined are selected from the image set to obtain a non-starting image set corresponding to the starting image. 5 . The method for generating a video according to claim 3 , wherein the determining of the starting image is based on the similarity between each non-starting image in the set of non-starting images and the starting image. 6 . candidate images corresponding to the original image, including: respectively calculating the similarity between each non-initial image in the non-initial image set and the initial image to obtain a similarity set; The maximum similarity is selected from the similarity set, and the image corresponding to the maximum similarity is used as the candidate image corresponding to the initial image. 6 . The video generation method according to claim 2 , wherein the target corresponding to each image in the image insertion sequence is determined according to the image insertion sequence, the several beat points and the audio. 7 . Beat points, including: Obtain the audio duration corresponding to the audio and the number of images of the several images; An image insertion point corresponding to each image in the image insertion sequence is determined according to the number of images and the audio duration, wherein the duration between the two adjacent image insertion points is determined according to the audio duration and the audio duration. determined by the number of said images; The target beat points corresponding to the several images are determined according to the several image insertion points and the several beat points. 7. The method for generating a video according to claim 6, wherein determining the target beat points corresponding to the several images according to the several image insertion points and the several beat points, specifically comprising: For each image insertion point, the beat point that is closest to the image insertion point is determined among the several beat points, and the beat point that is closest to the image insertion point is used as the target of the image corresponding to the image insertion point beat points. 8. The method for generating video according to claim 1, wherein the image corresponding to the target beat point is inserted into each target beat point in the audio to generate a video, specifically comprising: For each target beat point, when the audio playback time reaches the target beat point, insert the image corresponding to the target beat point, and use the image as the image played between the target beat point and the next target beat point image frame. 9. The method for generating a video according to any one of claims 1 to 8, wherein the acquiring an image set to be processed, and determining an image based on the similarity between any two images in the image set Before the insertion sequence, also include: acquiring an original image set, wherein the original image set includes multiple original images, and the multiple original images include at least one template image; Determine the target feature map corresponding to each original image in the original image set; dividing the original image set into original image subsets of different categories based on all the determined target feature maps; Any original image subset including the template image is used as the image set to be processed. 10. A video generation device, comprising: an acquisition unit, configured to acquire a set of images to be processed and audio to be processed, wherein the set of images includes several images; a target beat point determination unit, configured to determine the target beat point corresponding to each of the several images based on the image set and the audio; A video generation unit, configured to insert an image corresponding to the target beat point in each target beat point in the audio to generate a video. 11. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the video according to any one of claims 1 to 9 when the processor executes the computer program. Steps in the build method. 12. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps in the method for generating a video according to any one of claims 1 to 9 are implemented .

Images