CN116074566B - Game video highlight recording method, device, equipment and storage medium - Google Patents

Abstract

The invention discloses a method, a device, equipment and a storage medium for recording a game video highlight, and relates to the technical field of video processing. The method comprises the steps of obtaining audio and video stream data generated in real time in a game process, obtaining a current image to be identified containing spectrum characteristic information through fast Fourier transform, frequency point amplitude coding and drawing processing based on a current audio frame in the audio and video stream data, inputting the image into a highlight frame classification model based on CNN and highlight audio frames to finish pretraining, obtaining a current classification result, and finally recording and storing at least one video frame in synchronization with the current audio frame when the highlight confidence in the classification result is greater than or equal to a preset threshold value to obtain a game video highlight frame fragment, so that highlight moment picture identification is indirectly carried out by replacing a plurality of video frames with one audio frame, and the purposes of greatly simplifying the process, improving the identification efficiency and reducing required computing resources can be achieved.

Description

A method, device, equipment and storage medium for recording game video highlights

Technical field

The invention belongs to the field of video processing technology, and specifically relates to a method, device, equipment and storage medium for recording exciting scenes of game videos.

Background technique

With the rapid development of computer technology, players have various demands for game experience. Among them, the most prominent one is: players want to review and watch their wonderful moments in the game, such as defeating opponents multiple times in a row. (such as: two consecutive defeats, three consecutive victories or five consecutive unbeatables, etc.), which means that the game running platform needs to automatically identify the wonderful moments during the game and record and save them so that they can be pushed to the player after the game is over. Do a review.

The existing highlight moment picture recognition scheme is mainly based on the video picture image to identify whether it is a highlight moment picture, that is, it is necessary to obtain a game video target highlight picture in advance, and then use the current frame of the game video as the image to be recognized in real time, using Perception Ha The hash algorithm obtains the hash value of the target highlight picture and the image to be identified respectively. Finally, when the distance between the hash value of the target highlight picture and the hash value of the image to be identified is less than the preset threshold, the image to be identified is As a wonderful moment picture. However, as the frame rate of game videos increases (for example, to 120 frames per second, or even higher), the aforementioned wonderful moment picture recognition solution will have the problem of being cumbersome and consuming a lot of computing resources. Therefore, how to provide a method that can turn the complexity into A new solution for identifying Jane's wonderful moments images in order to reduce the required computing resources is an urgent topic for those skilled in the field to study.

Contents of the invention

The purpose of the present invention is to provide a method, device, computer equipment, and computer-readable storage medium for recording highlight scenes of game videos, so as to solve the problems of existing highlight scene recognition solutions that have complicated processes and consume a large amount of computing resources.

In order to achieve the above objects, the present invention adopts the following technical solutions:

The first aspect provides a method for recording exciting scenes of game videos, including:

Obtain audio and video stream data generated in real time during the game;

Perform fast Fourier transform processing on the current audio frame in the audio and video stream data to obtain the current spectrum;

The K amplitudes in the current spectrum and corresponding to the K frequency points are respectively encoded as red, green and blue RGB three-channel color values to obtain the current data to be identified containing K RGB values, where K Represents a natural number not less than 64, and the K frequency points are equally spaced in the human auditory frequency domain;

According to the K RGB values of the current data to be recognized, draw the current image to be recognized with a pixel matrix of k*k, where k is a natural number not less than the square root of K;

The current to-be-recognized image is input into the pre-trained highlight frame classification model based on the convolutional neural network CNN and the highlight audio frames to obtain the current classification results, where the highlight audio frames refer to the same period as the highlight scenes of the game video target. Audio frames, and used to provide positive samples for wonderful frame classification training for the wonderful frame classification model;

When the highlight confidence in the current classification result is greater than or equal to the preset confidence threshold, at least one video frame in the audio and video stream data that is synchronized with the current audio frame is recorded and saved to obtain the game video highlights. Picture fragment, wherein the highlight confidence level refers to the confidence level of classifying the current audio frame as a highlight frame in the current classification result.

Based on the above content of the invention, a new solution is provided to indirectly identify the highlights of the game based on audio frames of the same period. That is, after obtaining the audio and video stream data generated in real time during the game, based on the current audio in the audio and video stream data Frame, through fast Fourier transform, frequency point amplitude encoding and drawing processing, the current image to be recognized containing spectral feature information is obtained, and then the image is input into the wonderful frame based on the convolutional neural network CNN and the wonderful audio frame to complete the pre-training Classify the model to obtain the current classification result. Finally, when the highlight confidence in the classification result is greater than or equal to the preset confidence threshold, perform a test on at least one video frame in the audio and video stream data that is synchronized with the current audio frame. Record and save to get the highlight clips of the game video. In this way, by using one audio frame instead of multiple video frames to indirectly identify the highlight moments, you can greatly simplify the process, improve the recognition efficiency and reduce the required computing resources, which is convenient for practical use. application and promotion.

In a possible design, K amplitudes corresponding to K frequency points are encoded into red, green, and blue RGB three-channel color values, including:

By converting numerical units, the K amplitudes are converted into numerical values to be converted that are in the same numerical unit and are respectively within the interval [0,16777215];

Convert the value to be converted from a decimal number into a binary number;

Add 0 to the binary number from left to right to obtain a 24-bit binary number;

Convert the first 8 binary digits of the 24-bit binary digits into decimal digits to obtain the red channel color value of the red, green, and blue RGB three-channel color values;

Convert the middle 8 binary digits of the 24-bit binary digits into decimal digits to obtain the green channel color value of the red, green, and blue RGB three-channel color values;

Convert the last 8 binary digits of the 24-bit binary digits into decimal digits to obtain the blue channel color value of the red, green, and blue RGB three-channel color values.

In a possible design, the CNN adopts Resnet50 network structure, Mobile-net network structure or VGG16 network structure.

In a possible design, when the highlight confidence in the current classification result is greater than or equal to the preset confidence threshold, at least one video frame in the audio and video stream data that is synchronized with the current audio frame is Record and save to get exciting game video clips, including:

When the highlight confidence in the current classification result is greater than or equal to the preset confidence threshold, it is determined whether the number of audio frames between the most recent highlight frame and the current audio frame is equal to zero, where the highlight confidence is Refers to the confidence of classifying the current audio frame as a highlight frame in the current classification result. The most recent highlight frame refers to the highlight frame that is located before the current audio frame in the audio and video stream data and corresponds to the highlight frame. Audio frames whose confidence is greater than or equal to the preset confidence threshold;

If it is determined that the number of audio frames is equal to zero, then record and save at least one video frame in the audio and video stream data that is synchronized with the current audio frame to obtain a game video highlight segment, otherwise it is further determined that the audio frame Whether the number is greater than or equal to the preset frame number threshold;

If it is determined that the number of audio frames is greater than or equal to the preset frame number threshold, at least one video frame in the audio and video stream data that is synchronized with the current audio frame is recorded and saved to obtain a game video highlight segment. , otherwise record and save at least one video frame in the audio and video stream data that is synchronized with the intermediate audio frame and the current audio frame to obtain the game video highlight clip, wherein the intermediate audio frame refers to the At least one audio frame located between the most recent highlight frame and the current audio frame in the audio and video stream data.

In a possible design, after obtaining the highlight clips of the game video, the method further includes:

Determine whether the most recent game video highlight clip and the latest game video highlight clip are sequentially consecutive;

If it is determined that they are continuous in time sequence, merge the two game video highlight clips into one game video highlight clip, otherwise further determine whether the duration of the most recent game video highlight clip is less than or equal to the preset duration threshold;

If it is determined that the duration is less than or equal to the preset duration threshold, then the saved most recent game video highlights are deleted.

In a possible design, the method further includes:

At the end of the game, all the game video highlight clips recorded during the game are collected to obtain at least one game video highlight clip;

For each of the game video highlight clips in the at least one game video highlight clip, the corresponding highlight confidence sum is calculated cumulatively according to the following formula:

In the formula, k represents a positive integer, GCT _k represents the sum of the confidence levels of the k-th game video highlight segment in the at least one game video highlight segment, and N _k represents the difference between the k-th game video highlight segment and the k-th game video highlight segment. The total number of frames of multiple audio frames collected simultaneously by the clip, n represents a positive integer, GC _{k, n} represents the highlight confidence of the nth audio frame among the multiple audio frames;

Sort the at least one game video highlight clip in descending order according to the sum of the highlight confidence levels to obtain a sequence of game video highlight clips;

Push the first M game video highlight clips in the game video highlight clip sequence to game players, where M represents a preset positive integer less than or equal to K, and K represents the total clips of the game video highlight clip sequence. number.

In a possible design, after obtaining the highlight clips of the game video, the method further includes:

Randomly select a video frame from the highlight footage of the game video;

Perform image processing on the video frame using a perceptual hashing algorithm to obtain image fingerprint information of the video frame;

Determine whether the number of different data digits between the image fingerprint information of the video frame and the image fingerprint information of the game video target highlight is greater than or equal to a preset digit threshold;

If so, delete the recorded and saved highlight footage of the game video.

In the second aspect, a device for recording highlights of game videos is provided, including a data acquisition module, a Fourier transform processing module, a frequency point amplitude coding module, an image drawing module to be recognized, a highlight frame classification module and a video frame saving module;

The data acquisition module is used to acquire audio and video stream data generated in real time during the game;

The Fourier transform processing module is communicatively connected to the data acquisition module, and is used to perform fast Fourier transform processing on the current audio frame in the audio and video stream data to obtain the current spectrum;

The frequency point amplitude encoding module is communicatively connected to the Fourier transform processing module, and is used to encode the K amplitude values in the current spectrum and corresponding to the K frequency points into red, green and blue RGB respectively. The channel color value is used to obtain the current data to be identified containing K RGB values, where K represents a natural number not less than 64, and the K frequency points are equally spaced in the human auditory frequency domain;

The image drawing module to be identified is communicatively connected to the frequency point amplitude encoding module, and is used to draw the current image to be identified with a pixel matrix of k*k according to the K RGB values of the current data to be identified, where, k is a natural number not less than the square root of K;

The highlight frame classification module is communicatively connected to the to-be-recognized image drawing module, and is used to input the current to-be-recognized image into a pre-trained highlight frame classification model based on the convolutional neural network CNN and highlight audio frames to obtain the current classification. As a result, the highlight audio frame refers to the audio frame that is in the same period as the target highlight picture of the game video, and is used to provide the highlight frame classification model with positive samples for highlight frame classification training;

The video frame saving module is respectively connected to the data acquisition module and the highlight frame classification module by communication, and is used to store the audio and video stream data when the highlight confidence in the current classification result is greater than or equal to the preset confidence threshold. At least one video frame that is in the same period as the current audio frame is recorded and saved to obtain a highlight video clip of the game video, wherein the highlight confidence level refers to classifying the current audio frame as in the current classification result. Confidence of highlight frames.

In a third aspect, the present invention provides a computer device, including a memory, a processor and a transceiver connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive voice signals, and the processing The computer program is configured to read the computer program and execute the game video highlight recording method as described in the first aspect or any possible design in the first aspect.

In a fourth aspect, the present invention provides a computer-readable storage medium. Instructions are stored on the computer-readable storage medium. When the instructions are run on a computer, the invention executes the first aspect or any possibility in the first aspect. Design the method for recording the wonderful scenes of game videos.

In a fifth aspect, the present invention provides a computer program product containing instructions that, when the instructions are run on a computer, cause the computer to execute the game video highlights described in the first aspect or any possible design in the first aspect. Screen recording method.

Beneficial effects of the above scheme:

(1) The present invention creatively provides a new solution for indirectly identifying the highlights of the game based on audio frames of the same period. That is, after obtaining the audio and video stream data generated in real time during the game, based on the current audio and video stream data in the For audio frames, through fast Fourier transform, frequency point amplitude encoding and drawing processing, the current image to be recognized containing spectral feature information is obtained, and then the image is input into the convolutional neural network CNN and the wonderful audio frame to complete the pre-trained wonderful The frame classification model obtains the current classification result. Finally, when the highlight confidence in the classification result is greater than or equal to the preset confidence threshold, at least one video frame in the audio and video stream data that is synchronized with the current audio frame is Record and save to get the highlight clips of the game video. In this way, by using one audio frame instead of multiple video frames to indirectly identify the highlight moments, you can greatly simplify the process, improve the recognition efficiency and reduce the required computing resources, which is convenient for Practical application and promotion;

(2) Adjacent clips can also be merged and isolated instantaneous clips eliminated to prevent the recorded game video clips from being too fragmented, so as to push content with high review value to players;

(3) At the end of the game, you can also calculate the retrospective value of each clip for all the game video highlights recorded during the game, and push the content with the most retrospective value to the game players based on the retrospective value. This can further Improve players' gaming experience and facilitate practical application and promotion.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic flowchart of a method for recording game video highlights provided by an embodiment of the present application.

Figure 2 is a schematic structural diagram of a device for recording game video highlights provided by an embodiment of the present application.

Figure 3 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

Detailed ways

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts. It should be noted here that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

It should be understood that although the terms first, second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, a first object may be referred to as a second object, and similarly a second object may be referred to as a first object, without departing from the scope of example embodiments of the invention.

It should be understood that the term "and/or" that may appear in this article is only an association relationship describing related objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, The existence of B alone or the existence of A and B at the same time; for another example, A, B and/or C can mean the existence of any one of A, B and C or any combination of them; for what may appear in this article The term "/and" describes another type of associated object relationship, indicating that there can be two relationships, for example, A/and B, which can mean: A alone exists or A and B exist simultaneously; in addition, for The character "/" that may appear in this article generally indicates that the related objects are an "or" relationship.

Example:

As shown in Figure 1, the method for recording game video highlights provided in the first aspect of this embodiment can be, but is not limited to, executed by a computer device with certain computing resources and capable of running game programs, such as a platform server, a personal computer (Personal Computer, PC refers to a multi-purpose computer with a size, price and performance suitable for personal use; desktops, laptops to small laptops and tablets and ultrabooks are all personal computers), smart phones, personal digital It is executed by electronic devices such as Personal Digital Assistant (PDA) or wearable devices. As shown in Figure 1, the method for recording game video highlights may include, but is not limited to, the following steps S1 to S6.

S1. Obtain the audio and video stream data generated in real time during the game.

In step S1, in order to provide players with a basic sense of game immersion, existing game programs will display the game video screen and the game audio sound synchronized with the game video screen to the player in real time during the game process. There are conventional techniques to obtain the audio and video stream data generated in real time during the game and used to display the aforementioned game video images and game audio sounds.

S2. Perform fast Fourier transform processing on the current audio frame in the audio and video stream data to obtain the current spectrum.

In the step S2, the current audio frame is the smallest unit of data used to display the current game audio sound, which may be specifically, but not limited to, an AAC (Advanced Audio Coding, Advanced Audio Coding) audio frame or an MP3 (Moving Picture Experts Group Audio Layer III, dynamic image experts compress standard audio layer III) audio frame. The former contains 1024 sampling points, and the latter contains 1152 sampling points. Therefore, the current audio frame can be obtained from the audio and video stream data in a conventional manner. audio frame. Since the frame playback duration of the current audio frame is negatively related to the audio sampling rate (that is, the higher the audio sampling rate, the shorter the frame playback duration), for example, when the audio sampling rate is 48kHz, the frame playback duration is 21.32 milliseconds, and when the audio When the sampling rate is 20.05kHz, the frame playback duration is 46.43 milliseconds. Therefore, in order to be able to replace more video frames for highlight moment picture recognition in the future, the audio sampling rate of the current audio frame is preferably a lower sampling rate, such as 8kHz. or 11.025kHz. In addition, the fast Fourier transform processing is an existing commonly used signal processing method, and will not be described again here.

S3. Encode the K amplitudes in the current spectrum and one-to-one correspondence with the K frequency points into red, green, and blue RGB three-channel color values to obtain the current data to be identified containing K RGB values, where , K represents a natural number not less than 64, and the K frequency points are equally spaced in the human auditory frequency domain.

In the step S3, since the current audio frame is used to display the current game audio sound to the player, the spectrum of the sound of the wonderful moment in the same period as the picture of the wonderful moment such as two consecutive defeats, three consecutive victories or five consecutive unparalleled moments. The characteristics must also be located in the human hearing frequency domain, and then it is preferable to set the K frequency points to be equally spaced in the human hearing frequency domain; the human hearing frequency range is generally 20Hz ~ 20kHz, if the frequency interval is set as an example is 10Hz, the value of K can be made to be 1999, which meets the design requirements of no less than 64. In addition, specific encoding methods may include, but are not limited to: converting the K amplitudes into numerical values to be converted under the same numerical unit and within the interval [0,16777215] by converting numerical units; The numerical value to be converted is converted from a decimal number to a binary number; the binary number is supplemented with 0s from left to right to obtain a 24-bit binary number; the first 8 binary digits in the 24-bit binary number are converted into a decimal number, Obtain the red channel color value in the red, green and blue RGB three-channel color values; convert the 8-bit binary digits in the 24-bit binary numbers into decimal numbers to obtain the red, green, and blue RGB three-channel color values. Green channel color value; convert the last 8 binary digits of the 24-bit binary digits into decimal digits to obtain the blue channel color value of the red, green, and blue RGB three-channel color values.

S4. According to the K RGB values of the current data to be recognized, draw the current image to be recognized with a pixel matrix of k*k, where k is a natural number not less than the square root of K.

In the step S4, the specific way of drawing the current image to be recognized may be, but is not limited to, the first RGB value as the /> OK and No./> RGB values of column pixels, where, /> It is a natural number between 1 and K, and floor() represents the lower rounding function. As for other pixels, common methods such as zero padding or average RGB value padding can be used to complement the value, so that a rectangle to be identified can be obtained. For example, if there are 1999 amplitude values, an image to be recognized can be obtained with a pixel matrix of 45*45; and if there are 396 amplitude values, an image to be recognized can be obtained with a pixel matrix of 20*20. In addition, considering that the size of the initially obtained image to be identified may be too small, the classification effect based on the convolutional neural network is not ideal. Therefore, after obtaining the current image to be identified, the method also includes: when K is smaller than the predetermined When a quantity threshold is set, the current images to be recognized are enlarged to obtain corresponding images to be recognized that have standard sizes. For example, zoom in to obtain an image with a size of 64*64.

S5. Input the current image to be recognized into the pre-trained highlight frame classification model based on the convolutional neural network CNN and the highlight audio frame to obtain the current classification result, where the highlight audio frame refers to the highlight picture of the game video target Audio frames of the same period are used to provide positive samples for wonderful frame classification training for the wonderful frame classification model.

In the step S5, the convolutional neural network CNN (Convolutional Neural Networks) is an existing feedforward neural network (Feedforward Neural Networks) based on convolution calculation and has a deep structure, including but not limited to having an input layer , convolution layer, activation layer, pooling layer, fully connected layer and output layer, and the normalized index Softmax function can be used through the output layer to perform image recognition classification; specifically, the CNN can, but is not limited to, use Resnet50 Network structure, Mobile-net network structure or VGG16 network structure, etc. The wonderful audio frames need to be obtained in advance. For example, the audio frames that are in the same period as the wonderful scenes of game video targets such as two consecutive defeats, three consecutive wins, or five consecutive unparalleled games can be used as the wonderful audio frames. The wonderful audio frames are used to provide positive samples for wonderful frame classification training for the wonderful frame classification model, including but not limited to: performing the fast Fourier transform processing on the wonderful audio frames to obtain the wonderful audio frames. Spectrum; then the K amplitudes in the wonderful audio spectrum and corresponding to the K frequency points are respectively encoded into red, green and blue RGB three-channel color values to obtain positive samples containing K RGB values. data; finally, according to the K RGB values of the positive sample data, a positive sample image with a pixel matrix of k*k is drawn. The specific pre-training method of the highlight frame classification model may, but is not limited to, include: inputting multiple positive sample images based on different multiple highlight audio frames into the classification model based on the CNN for training , and when the accuracy of the training set reaches the preset high value range and the change amplitude is less than the preset amplitude threshold during the training process, the adaptive gradient AdaGrad algorithm is used to adjust the learning rate, and then continues training until the learning rate adjustment amplitude is less than the preset amplitude threshold. When adjusting the threshold, training is stopped and a wonderful frame classification model that has completed training is obtained; the aforementioned adaptive gradient AdaGrad algorithm is an existing algorithm, which uses the sum of the square roots of the gradients in each iteration history to modify the learning rate. In addition, you can also refer to the aforementioned method of providing positive samples, and use some non-highlight audio frames to provide negative sample images for the wonderful frame classification model for the wonderful frame classification training, and during the model training process, these negative sample images are also used Enter the classification model based on the CNN for training to ensure the correctness of subsequent highlight frame classification; and some positive sample images and negative sample images can also be composed into a test sample set, so that after the model training is completed, the test can be used The sample set performs a highlight frame classification test on the highlighted frame classification model that has completed training to verify whether the highlighted frame classification can be accurately performed.

S6. When the highlight confidence in the current classification result is greater than or equal to the preset confidence threshold, record and save at least one video frame in the audio and video stream data that is synchronized with the current audio frame to obtain the game A video highlight segment, wherein the highlight confidence level refers to the confidence level of classifying the current audio frame as a highlight frame in the current classification result.

In step S6, the confidence level is conventional information output after classification and recognition. The preset confidence level threshold is used as a basis for judging whether to classify the current audio frame as a highlight frame, which can be 50% for example. . If it is found that the highlight confidence level is greater than or equal to the preset confidence threshold, it means that the current audio frame can be classified as a highlight frame. At this time, the content reflected in the audio and video stream data is different from the current audio frame. At least one video frame in the same period is a game video highlight, so the at least one video frame needs to be recorded and saved as a game video highlight segment. Since the frame playback duration of the current audio frame is long, for example, 46.43 milliseconds, if the video frame rate in the audio and video stream data is 120 frames per second, then the number of frames of the at least one video frame will be About 6 frames can be used to replace these 6 video frames with one audio frame to indirectly identify the highlight moments, thus greatly simplifying the process, improving recognition efficiency, and reducing the required computing resources.

In the step S6, it is considered that the frame playback time of an audio frame is only tens of milliseconds, while a game video highlight such as two-consecutive defeat, three-consecutive victory or five-consecutive peerless scene will last for several seconds or even more. In order to avoid Because some audio frames are identified as non-highlight frames within these seconds or more, the video frames synchronized with these audio frames are omitted to ensure the integrity of the game video highlights. Preferably, when the current classification result is When the highlight confidence is greater than or equal to the preset confidence threshold, record and save at least one video frame in the audio and video stream data that is synchronized with the current audio frame to obtain game video highlight segments, including but not limited to There are the following steps S61 to S63.

S61. When the highlight confidence in the current classification result is greater than or equal to the preset confidence threshold, determine whether the number of audio frames between the most recent highlight frame and the current audio frame is equal to zero, where the highlight confidence The degree refers to the confidence of classifying the current audio frame as a highlight frame in the current classification result, and the most recent highlight frame refers to the audio and video stream data that is located before and before the current audio frame. Audio frames corresponding to a highlight confidence level greater than or equal to the preset confidence threshold.

In the step S61, the previous most recent wonderful frame is the historical audio frame that was recently determined to be the wonderful frame.

S62. If it is determined that the number of audio frames is equal to zero, then record and save at least one video frame in the audio and video stream data that is synchronized with the current audio frame to obtain a highlight segment of the game video, otherwise further determine that the Whether the number of audio frames is greater than or equal to the preset frame number threshold.

In step S62, the preset frame number threshold can be determined in advance based on the audio frame playback duration and the general maintenance duration of the game video target highlight screen. For example, when the audio frame playback duration is 46.43 milliseconds, the preset frame The number threshold can be set to 21 frames, for example.

S63. If it is determined that the number of audio frames is greater than or equal to the preset frame number threshold, record and save at least one video frame in the audio and video stream data that is synchronized with the current audio frame to obtain game video highlights. Picture fragments, otherwise record and save at least one video frame in the audio and video stream data that is synchronized with the intermediate audio frame and the current audio frame to obtain the game video exciting picture fragments, wherein the intermediate audio frame refers to At least one audio frame located between the most recent highlight frame and the current audio frame in the audio and video stream data.

In the step S63, the intermediate audio frame is an audio frame that is identified as a non-highlight frame within the game video highlight screen maintenance time. Therefore, under certain conditions, by comparing the AND in the audio and video stream data Recording and saving the intermediate audio frame and at least one video frame in the same period as the current audio frame can ensure the integrity of the wonderful scenes of the game video.

After the step S6, it is also considered to use audio frames instead of multiple video frames in the same period to indirectly identify the highlight moments. In order to ensure the accuracy of this indirect recognition result, it is necessary to obtain the highlight moments of the game video. After the clip, a frame extraction check is performed, that is, preferably, after obtaining the highlight clips of the game video, the method also includes but is not limited to the following steps S71 to S74: S71. Randomly extract a video from the highlight clips of the game video. Frame; S72. Use perceptual hashing algorithm to perform image processing on the video frame to obtain the image fingerprint information of the video frame; S73. Determine the difference between the image fingerprint information of the video frame and the image fingerprint information of the game video target highlights. Whether the number of different data bits is greater than or equal to the preset number of bits threshold; S74. If so, delete the recorded and saved highlight clips of the game video. The aforementioned perceptual hashing algorithm is an existing algorithm used to generate fingerprint information from image data, and its principle will not be described again here. Based on the aforementioned steps S71 to S74, a frame can be drawn to check whether the game video highlight scene is a game highlight scene, if so, it will be retained, otherwise (that is, the image fingerprint information of the video frame and the image of the game video target highlight scene The fingerprint information is quite different) and the highlights of the game video need to be deleted. In addition, the preset number of bits threshold may be 10, for example.

Therefore, based on the method of recording game video highlights described in the aforementioned steps S1 to S6, a new solution for indirectly identifying highlights of the game based on audio frames of the same period is provided, that is, the audio and video stream data generated in real time during the game is obtained. Finally, based on the current audio frame in the audio and video stream data, through fast Fourier transform, frequency point amplitude encoding and drawing processing, the current image to be recognized containing spectral feature information is obtained, and then the image is input into the image based on the convolutional neural network. Network CNN and highlight audio frames are used to complete the pre-trained highlight frame classification model to obtain the current classification result. Finally, when the highlight confidence in the classification result is greater than or equal to the preset confidence threshold, the audio and video stream data and At least one video frame in the same period as the current audio frame is recorded and saved to obtain highlight clips of the game video. In this way, by using one audio frame instead of multiple video frames to indirectly identify highlight moments, the process can be greatly simplified and improved. The purpose of identifying efficiency and reducing required computing resources is to facilitate practical application and promotion.

Based on the technical solution of the first aspect, this embodiment also provides a possible design of how to merge adjacent segments and eliminate isolated instantaneous segments. That is, after obtaining the highlight clips of the game video, the method also includes: It is not limited to the following steps S81 to S83.

S81. Determine whether the most recent game video highlight clip and the latest game video highlight clip are sequentially continuous.

In the step S81, whether the two segments are continuous in time sequence can be determined based on the human eye reaction time (that is, generally 0.1 seconds to 0.4 seconds). For example, if the last frame of the most recent game video highlight segment If the timing difference between the first frame and the latest obtained game video highlight fragment is greater than or equal to 0.4 seconds, it can be considered that the two fragments are temporally discontinuous and cannot be merged into one fragment. Otherwise, they are allowed to be merged into one fragment.

S82. If it is determined that they are continuous in time sequence, merge the two game video highlight clips into one game video clip. Otherwise, further determine whether the duration of the most recent game video highlight clip is less than or equal to the preset duration threshold. .

In step S82, since the most recent game video highlight clip is not merged with the latest game video highlight clip, it is a clip with a fixed duration, and it is necessary to further compare the duration based on the result. To determine whether it is an isolated momentary clip, if so, it needs to be deleted because the highlight moment is too short. In addition, the preset duration threshold may be specifically determined based on the shortest duration of the highlight image, for example, 1 second.

S83. If it is determined that the duration is less than or equal to the preset duration threshold, delete the saved most recent game video highlights.

Based on the aforementioned possible design one, adjacent clips can be merged and isolated instantaneous clips can be eliminated to achieve the purpose of preventing the recorded game video highlight clips from being too fragmented, so as to push content with high review value to players.

Based on the technical solution of possible design one mentioned above, this embodiment also provides a possible design two of how to push high review value content to players, that is, the method also includes but is not limited to the following contents S91 to S94.

S91. At the end of the game, collect all game video highlight clips recorded during the game to obtain at least one game video highlight clip.

S92. For each of the game video highlight clips in the at least one game video highlight clip, accumulate and calculate the corresponding highlight confidence sum according to the following formula:

In the formula, k represents a positive integer, GCT _k represents the sum of the confidence levels of the k-th game video highlight segment in the at least one game video highlight segment, and N _k represents the difference between the k-th game video highlight segment and the k-th game video highlight segment. The total number of frames of multiple audio frames collected simultaneously by the clip, n represents a positive integer, GC _{k, n} represents the highlight confidence of the nth audio frame among the multiple audio frames.

In the step S92, based on the above formula, it can be known that the highlight confidence of each segment is always positively correlated with the corresponding number of audio frames in the same period and the excitement confidence of the audio frame in the same period respectively, so it can be used to reflect the retrospective value of the corresponding segment: Segment The longer it is and the more exciting scenes it contains, the higher the review value is.

S93. Sort the at least one game video highlight clip in descending order according to the sum of the highlight confidence levels to obtain a sequence of game video highlight clips.

S94. Push the first M game video highlight clips in the game video highlight clip sequence to game players, where M represents a preset positive integer less than or equal to K, and K represents the number of the game video highlight clip sequence. Total number of segments.

In step S94, if the value of K is 100, then the value of M can be, for example, 10, that is, the top 10 game video highlights with the highest review value are pushed to the game players.

Based on the aforementioned possible design 2, the retrospective value of each clip can be calculated separately at the end of the game for all game video highlight clips recorded during the game, and content with the most retrospective value can be pushed to game players based on the retrospective value. , which can further enhance the player’s gaming experience.

As shown in Figure 2, the second aspect of this embodiment provides a virtual device that implements the method for recording game video highlights described in the first aspect, possible design one or possible design two, including a data acquisition module, Fourier transform processing module, frequency point amplitude coding module, image drawing module to be recognized, highlight frame classification module and video frame saving module;

The data acquisition module is used to acquire audio and video stream data generated in real time during the game;

The Fourier transform processing module is communicatively connected to the data acquisition module, and is used to perform fast Fourier transform processing on the current audio frame in the audio and video stream data to obtain the current spectrum;

The frequency point amplitude encoding module is communicatively connected to the Fourier transform processing module, and is used to encode the K amplitude values in the current spectrum and corresponding to the K frequency points into red, green and blue RGB respectively. The channel color value is used to obtain the current data to be identified containing K RGB values, where K represents a natural number not less than 64, and the K frequency points are equally spaced in the human auditory frequency domain;

The image drawing module to be identified is communicatively connected to the frequency point amplitude encoding module, and is used to draw the current image to be identified with a pixel matrix of k*k according to the K RGB values of the current data to be identified, where, k is a natural number not less than the square root of K;

The highlight frame classification module is communicatively connected to the to-be-recognized image drawing module, and is used to input the current to-be-recognized image into a pre-trained highlight frame classification model based on the convolutional neural network CNN and highlight audio frames to obtain the current classification. As a result, the highlight audio frame refers to the audio frame that is in the same period as the target highlight picture of the game video, and is used to provide the highlight frame classification model with positive samples for highlight frame classification training;

The video frame saving module is respectively connected to the data acquisition module and the highlight frame classification module by communication, and is used to store the audio and video stream data when the highlight confidence in the current classification result is greater than or equal to the preset confidence threshold. At least one video frame that is in the same period as the current audio frame is recorded and saved to obtain a highlight video clip of the game video, wherein the highlight confidence level refers to classifying the current audio frame as in the current classification result. Confidence of highlight frames.

For the working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment, please refer to the method for recording game video highlights described in the first aspect, possible design one or possible design two, and will not be described again here.

As shown in Figure 3, the third aspect of this embodiment provides a physical device that implements the method for recording game video highlights as described in the first aspect, possible design one, or possible design two, including a memory connected by communication in sequence, A processor and a transceiver, wherein the memory is used to store a computer program, the transceiver is used to send and receive voice signals, and the processor is used to read the computer program and execute the first aspect, a possible design or a possible design 2. The method for recording highlights of game videos described in 2. For specific examples, the memory may include, but is not limited to, random access memory (Random-Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (FlashMemory), first-in-first-out memory (First Input First Output, FIFO) and/or First Input Last Output (FILO), etc.

For the working process, working details and technical effects of the aforementioned chip provided in the third aspect of this embodiment, please refer to the method for recording game video highlights described in the first aspect, possible design one or possible design two, and will not be described again here.

The fourth aspect of this embodiment provides a computer-readable storage medium that stores instructions for the method for recording game video highlights as described in the first aspect, possible design one, or possible design two, that is, the computer-readable storage medium Instructions are stored on the computer, and when the instructions are run on the computer, the method for recording game video highlights as described in the first aspect, possible design one, or possible design two is executed. Wherein, the computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memory, USB flash drives, and/or Memory Sticks. The computer may Is a general-purpose computer, special-purpose computer, computer network or other programmable device.

For the working process, working details and technical effects of the computer-readable storage medium provided in the fourth aspect of this embodiment, please refer to the method for recording game video highlights as described in the first aspect, possible design one or possible design two. Herein No longer.

The fifth aspect of this embodiment provides a computer program product containing instructions. When the instructions are run on a computer, the computer is caused to execute the game video highlights described in the first aspect, possible design one or possible design two. Screen recording method. Wherein, the computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

Finally, it should be noted that the above descriptions are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. A method for recording a video highlight of a game, comprising:

acquiring audio and video stream data generated in real time in the game process;

performing fast Fourier transform processing on the current audio frame in the audio-video stream data to obtain a current frequency spectrum;

respectively encoding K magnitudes which are in the current frequency spectrum and correspond to K frequency points one by one into RGB three-channel color values to obtain current data to be identified, wherein K represents a natural number which is not less than 64, and the K frequency points are distributed at equal intervals in a human auditory frequency domain interval;

drawing to obtain a current image to be recognized, wherein the pixel matrix of the current image to be recognized is K, according to K RGB values of the current data to be recognized, and K is a natural number not smaller than the square root of K;

inputting the current image to be identified into a highlight frame classification model which is trained in advance based on a convolutional neural network CNN and a highlight audio frame to obtain a current classification result, wherein the highlight audio frame is an audio frame synchronous with a game video target highlight picture and is used for providing a positive sample for highlight frame classification training for the highlight frame classification model;

When the highlight confidence in the current classification result is greater than or equal to a preset confidence threshold, recording and storing at least one video frame which is in the audio-video stream data and is synchronous with the current audio frame to obtain a game video highlight picture segment, wherein the highlight confidence is the confidence of classifying the current audio frame into the highlight frame in the current classification result.

2. The method of claim 1, wherein the encoding the K magnitudes corresponding to the K frequency points one to one into the RGB three-channel color values comprises:

transforming the K magnitudes into values of the same numerical unit and respectively in intervals by means of transforming the numerical units

A value to be converted in [0,16777215 ];

converting the value to be converted from decimal numbers to binary numbers;

0 is complemented on the binary digits from left to right to obtain 24-bit binary digits;

converting the first 8 digits in the 24 digits into decimal digits to obtain a red channel color value in the red, green and blue RGB three-channel color values;

converting the middle 8-bit binary digits in the 24-bit binary digits into decimal digits to obtain a green channel color value in the red, green and blue RGB three-channel color values;

And converting the last 8 binary digits in the 24-bit binary digits into decimal digits to obtain a blue channel color value in the red, green and blue RGB three-channel color values.

3. The game video highlight recording method according to claim 1, wherein the CNN adopts a network structure of Resnet50, a Mobile-net or a VGG 16.

4. The method for recording a video highlight of a game according to claim 1, wherein when the highlight confidence in the current classification result is greater than or equal to a preset confidence threshold, recording and storing at least one video frame in the audio-video stream data and in synchronization with the current audio frame to obtain a video highlight segment of the game, comprising:

when the highlight confidence in the current classification result is greater than or equal to a preset confidence threshold, judging whether the audio frame number between the previous latest highlight frame and the current audio frame is equal to zero, wherein the highlight confidence refers to the confidence of classifying the current audio frame into the highlight frame in the current classification result, and the previous latest highlight frame refers to the audio frame which is positioned before the current audio frame in the audio-video stream data and corresponds to the highlight confidence which is greater than or equal to the preset confidence threshold;

If the audio frame number is equal to zero, recording and storing at least one video frame which is in the audio-video stream data and is synchronous with the current audio frame to obtain a game video highlight frame segment, otherwise, further judging whether the audio frame number is greater than or equal to a preset frame number threshold value;

if the audio frame number is greater than or equal to the preset frame number threshold, recording and storing at least one video frame in the audio-video stream data and in the same period as the current audio frame to obtain a game video highlight frame segment, otherwise recording and storing at least one video frame in the audio-video stream data and in the same period as the middle audio frame and the current audio frame to obtain a game video highlight frame segment, wherein the middle audio frame refers to at least one audio frame in the audio-video stream data, which is positioned between the last highlight frame and the current audio frame.

5. The method of claim 1, wherein after obtaining the video highlight clip, the method further comprises:

judging whether the previous latest game video highlight frame segment is continuous with the latest obtained game video highlight frame segment in time sequence;

If the time sequence is continuous, merging the two game video highlight frame fragments into one game video highlight frame fragment, otherwise, further judging whether the duration of the previous latest game video highlight frame fragment is smaller than or equal to a preset duration threshold value;

and if the duration is less than or equal to the preset duration threshold, deleting the stored latest previous game video highlight frame fragment.

6. The method of claim 5, further comprising:

summarizing all game video highlight frame fragments recorded in the game process at the end of the game to obtain at least one game video highlight frame fragment;

for each game video highlight segment in the at least one game video highlight segment, accumulating and calculating according to the following formula to obtain a corresponding highlight confidence sum:

wherein k represents a positive integer, GCT _k A highlight confidence sum, N, representing a kth game video highlight clip in the at least one game video highlight clip _k Representing the total number of frames of a plurality of audio frames taken contemporaneously with the kth game video highlight segment, n representing a positive integer, GC _k,n Representing a highlight confidence level for an nth audio frame of the plurality of audio frames;

sequencing the at least one game video highlight frame segment according to the highlight confidence sum from high to low to obtain a game video highlight frame segment sequence;

pushing the first M game video highlight frame fragments in the game video highlight frame fragment sequence to a game player, wherein M represents a preset positive integer less than or equal to K, and K represents the total fragment number of the game video highlight frame fragment sequence.

7. The method of claim 1, wherein after obtaining the video highlight clip, the method further comprises:

randomly extracting a video frame from the game video highlight frame;

performing image processing on the video frame by adopting a perceptual hash algorithm to obtain image fingerprint information of the video frame;

judging whether the number of different data bits of the image fingerprint information of the video frame and the image fingerprint information of the game video target highlight is greater than or equal to a preset bit number threshold value;

if yes, deleting the recorded and saved game video highlight frame fragments.

8. The game video highlight image recording device is characterized by comprising a data acquisition module, a Fourier transform processing module, a frequency point amplitude encoding module, an image drawing module to be identified, a highlight frame classification module and a video frame storage module;

the data acquisition module is used for acquiring audio and video stream data generated in real time in the game process;

the Fourier transform processing module is in communication connection with the data acquisition module and is used for carrying out fast Fourier transform processing on the current audio frame in the audio-video stream data to obtain a current frequency spectrum;

the frequency point amplitude coding module is in communication connection with the Fourier transform processing module and is used for respectively coding K amplitude values which are in the current frequency spectrum and correspond to K frequency points one by one into RGB three-channel color values to obtain current data to be identified which contain K RGB values, wherein K represents a natural number which is not less than 64, and the K frequency points are distributed at equal intervals in a human auditory frequency domain interval;

the image drawing module to be identified is in communication connection with the frequency point amplitude encoding module and is used for drawing the current image to be identified with a pixel matrix of K according to K RGB values of the current data to be identified, wherein K is a natural number not smaller than the square root of K;

The highlight frame classifying module is in communication connection with the image drawing module to be recognized and is used for inputting the current image to be recognized into a highlight frame classifying model which is trained in advance based on a convolutional neural network CNN and a highlight audio frame to obtain a current classifying result, wherein the highlight audio frame is an audio frame synchronous with a game video target highlight picture and is used for providing a positive sample for highlight frame classifying training for the highlight frame classifying model;

the video frame storage module is respectively in communication connection with the data acquisition module and the highlight frame classification module, and is used for recording and storing at least one video frame in the audio and video stream data and in the same period as the current audio frame when the highlight confidence in the current classification result is greater than or equal to a preset confidence threshold value to obtain a game video highlight frame fragment, wherein the highlight confidence is the confidence of classifying the current audio frame into the highlight frame in the current classification result.

9. A computer device comprising a memory, a processor and a transceiver in communication connection in sequence, wherein the memory is configured to store a computer program, the transceiver is configured to transmit and receive data, and the processor is configured to read the computer program and perform the game video highlight recording method according to any one of claims 1 to 7.

10. A computer readable storage medium having instructions stored thereon which, when executed on a computer, perform the method of recording a game video highlight according to any one of claims 1 to 7.