CN110010164B

CN110010164B - Multi-channel audio and video automatic intelligent editing method

Info

Publication number: CN110010164B
Application number: CN201910452585.3A
Authority: CN
Inventors: 刘鑫; 庞炜; 陈挚; 黄启挺; 罗辉
Original assignee: Chengdu E Learning Technology Co ltd
Current assignee: Chengdu E Learning Technology Co ltd
Priority date: 2018-11-13
Filing date: 2019-05-28
Publication date: 2021-01-26
Anticipated expiration: 2039-05-28
Also published as: CN109637561A; CN110010164A

Abstract

The invention relates to the field of audio and video editing, and provides a multi-channel audio and video automatic intelligent editing method aiming at the problem that the existing audio and video editing method is not suitable for the teaching field, which comprises the following steps: A. loading a multi-channel audio and video file which is synchronously recorded in advance and a recording record file which is recorded correspondingly, wherein the multi-channel audio and video at least comprises a path of default audio and a path of default channel video picture; B. editing the multi-channel audio and video file according to the recording record file to obtain a result video picture and a result audio; C. and combining the result video picture and the result audio to obtain a result video file. Therefore, the user does not need to have the professional ability of audio and video editing, the complex editing work after the audio and video recording is finished is avoided, and the method and the device are suitable for audio and video editing in the teaching field.

Description

Multi-channel audio and video automatic intelligent editing method

Technical Field

The invention relates to the field of audio and video editing, in particular to a multi-channel audio and video automatic intelligent editing method.

Background

Most of the current teaching video and live broadcast pictures displayed to audiences are processed results, and the audio and video processing scheme generally has two schemes: real-time processing and post editing. The real-time processing is also called live broadcasting, multi-camera shooting or multi-channel video picture display, and one channel is selected from all channel pictures for display according to the content required to be expressed at the current time. The post-editing is to record multi-channel video pictures, and to manually complete splicing and cutting by post-editing software, and to add some transition effect contents. Real-time processing schemes are also commonly used in video conferencing systems and online live systems.

1. The real-time processing comprises the following steps: the system collects multiple paths of video pictures, and judges whether to switch display pictures or lenses with poor shielding effect through manual switching or artificial intelligent system equipment so as to achieve the best video content display effect. The manual switching means that the best watching lens picture is manually switched to be displayed for audiences by broadcasting the video conditions of each channel watched on site, the work needs to be completed through manual participation, and the influence of artificial subjective factors is large, and the scheme is mainly used for television station programs and is not suitable for education and teaching videos. The real-time judgment processing of the artificial intelligence system refers to that the shooting scene picture is subjected to depth analysis through the combination of artificial intelligence equipment and the system, and shooting and displaying are carried out by selecting a corresponding channel through capturing of a dynamic picture. The scheme is influenced by dynamic elements or people in a shot picture, and the system often judges by mistake. When shooting, interaction often exists on the spot, preset flows and ideas need to be adjusted timely, and for conference training, lecturers, teachers and the like, the system often cannot accurately understand and judge the contents and display pictures expressed by the teachers, so that misjudgment often occurs, and the displayed contents cannot be displayed. Such events are of great impact, especially in teaching processes.

2. And (3) post editing: and (4) manually participating the post-editing work of the collected multi-channel videos through post-processing software. Traditional audio and video recording software supports screen and multi-path external video recording, a recorded video result file is stored after recording is completed, a large number of redundant and invalid segments in the result file need to be deleted, the final result can be obtained only by manual editing and deletion, meanwhile, required segments in each path of video need to be screened, then synchronous processing is carried out, the workload is large, complexity and complexity are high, and the synchronization and continuity of the developed result are poor. In the daily teaching process, teachers have heavy teaching tasks, video courseware edited in free time and recorded along with the classroom is avoided, and in addition, many teachers do not have the later editing capacity of audio and video recording files, and the problems that time axis splicing pictures are asynchronous and the like easily occur.

Therefore, there is a need for an automatic editing method for processing audio and video to be suitable for the teaching field.

Disclosure of Invention

The technical problem to be solved by the invention is as follows: the method solves the problem that the existing audio and video editing method is not suitable for the teaching field, and provides a multi-channel audio and video automatic intelligent editing method. The invention can automatically edit each channel of the multi-channel audio/video file synchronously recorded in advance in the process of daily teaching, lecturing or training and lecturing according to the recorded recording file recorded in advance and then synthesize the multi-channel audio/video file to obtain the achievement video file.

The invention solves the technical problems and adopts the technical scheme that:

the multichannel audio and video automatic intelligent editing method comprises the following steps:

A. loading a multi-channel audio and video file which is synchronously recorded in advance and a recording record file which is recorded correspondingly, wherein the multi-channel audio and video at least comprises a path of default audio and a path of default channel video picture;

B. editing the multi-channel audio and video file according to the recording record file to obtain a result video picture and a result audio;

C. and combining the result video picture and the result audio to obtain a result video file.

Preferably, the multi-channel audio/video further comprises at least one non-default channel video picture; the step B comprises the following steps: obtaining two types of marking time points which are respectively a default channel cut-out time point and a default channel cut-in time point according to the recording record file, wherein the result video picture is set as a default channel video picture by default, and when the default channel cut-out time point exists, the result video picture is switched from the default channel video picture to a non-default channel video picture until the recording time reaches the default channel cut-in time point which is next to the default channel cut-out time point; when a default channel cut-in time point exists, switching the result video picture from a non-default channel video picture to a default channel video picture;

and/or the step B comprises obtaining a deletion starting time point and a deletion ending time point according to the recording record file; and deleting all audio and video segments from the deletion starting time point to the deletion ending time point immediately behind the deletion starting time point in the multi-channel audio and video file.

Preferably, when the default channel switch-out time point and the default channel switch-in time point are required to be obtained, the step a includes: detecting a human-computer interaction event in a source terminal of a default channel video picture in real time and recording a corresponding human-computer interaction time point when the default channel video picture is recorded, calculating a default channel cut-out time point and a default channel cut-in time point in real time according to the human-computer interaction time point, storing the default channel cut-out time point and the default channel cut-in time point into a recording file, wherein the step B of obtaining the default channel cut-out time point and the default channel cut-in time point according to the recording file comprises the following steps: reading a default channel switching-out time point and a default channel switching-in time point from a recording record file;

or, when the default channel switch-out time point and the default channel switch-in time point need to be obtained, the step a includes: when recording a default channel video picture, detecting a human-computer interaction event in a source terminal of the default channel video picture in real time, recording a corresponding human-computer interaction time point, and storing the human-computer interaction time point into a recording file; the step B of obtaining the default channel switch-out time point and the default channel switch-in time point according to the recording file includes: and calculating to obtain a default channel switching-out time point and a default channel switching-in time point according to the human-computer interaction time point in the recording file.

Preferably, the obtaining of the default channel cut-out time point and the default channel cut-in time point by real-time calculation according to the human-computer interaction time point includes: detecting the human-computer interaction in a source terminal of the default channel video image in real time and recording the corresponding human-computer interaction time point when recording the default channel video image, and the following processes:

s101, judging whether a newly-added man-machine interaction time point exists in a first preset time length from a current man-machine interaction time point, wherein an initial value of the current man-machine interaction time point is an initial moment of multichannel audio and video recording, if not, entering S103, and if yes, entering S102;

s102, setting the current human-computer interaction time point as the newly-added human-computer interaction time point, and entering the step S101;

s103, adding a new marking time point with the type of the default channel switching time point, and setting the value of the new marking time point as a time point corresponding to the current human-computer interaction time point plus a first preset time length;

s104, detecting a new man-machine interaction time point in real time after the latest marking time point is separated from the current man-machine interaction time point, setting the current man-machine interaction time point as the new man-machine interaction time point, adding a new marking time point with the type of default channel switching-in time point, and setting the value of the new marking time point as the new man-machine interaction time point or the time point corresponding to the new man-machine interaction time point minus the preset buffer time length; entering step S101;

or the like, or, alternatively,

the method for calculating the default channel cut-out time point and the default channel cut-in time point in real time according to the human-computer interaction time point comprises the following steps: detecting the human-computer interaction in a source terminal of the default channel video image in real time and recording the corresponding human-computer interaction time point when recording the default channel video image, and the following processes:

s201, judging whether a newly-added man-machine interaction time point exists in a first preset time length from a current man-machine interaction time point, if not, entering a step S203, if so, entering a step S202, wherein an initial value of the current man-machine interaction time point is an initial moment of multi-channel audio and video recording;

s202, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point with the type of the default channel cut-in time point, and setting the value of the added marked time point as the new man-machine interaction time point or the time point corresponding to the subtraction of the new added man-machine interaction time point and the preset buffer duration; setting the current human-computer interaction time point as the newly-added human-computer interaction time point, and entering step S201;

s203, setting an image time starting point as a time point corresponding to the current human-computer interaction time point plus a first preset time length, and if the type of the last marked time point is the default channel cut-in time point or no marked point exists, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as the time point corresponding to the current human-computer interaction time point plus the first preset time length;

s204, judging whether a newly-added man-machine interaction time point exists in a second preset time length after the image time starting point, and if so, entering the step S202; otherwise, go to step S205;

s205, judging whether the image of the default channel video image at the image time starting point is consistent with the image of the default channel video image at the image time starting point plus the second preset time length, if not, entering the step S206; otherwise, go to step S207;

s206, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point of which the type is the default channel cut-in time point, and setting the value of the new marked time point as the time point corresponding to the image time starting point plus a second preset time length; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; entering step S204;

s207, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as the time point corresponding to the image time starting point plus a second preset time length; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; entering step S204;

or the like, or, alternatively,

s301, judging whether a newly-added man-machine interaction time point exists in a first preset time length from a current man-machine interaction time point, if not, entering a step S303, if so, entering a step S302, wherein an initial value of the current man-machine interaction time point is an initial moment of multi-channel audio and video recording;

s302, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point with the type of the default channel cut-in time point, and setting the value of the added marked time point as the new man-machine interaction time point or the time point corresponding to the subtraction of the new added man-machine interaction time point and the preset buffer duration; setting the current human-computer interaction time point as the newly-added human-computer interaction time point, and entering the step S301;

s303, setting an image time starting point as a current human-computer interaction time point plus a time point corresponding to a first preset time length;

s304, judging whether a newly-added man-machine interaction time point exists in a second preset time length after the image time starting point, and if so, entering the step S302; otherwise, go to step S305;

s305, judging whether the image of the default channel video image at the image time starting point is consistent with the image of the default channel video image at the image time starting point plus a second preset time length, if not, entering a step S306, otherwise, entering a step S307;

s306, if the type of the last marked time point is the default channel cut-out time point, adding a new marked time point of which the type is the default channel cut-in time point, and setting the value of the new marked time point as an image time starting point; setting a new image time starting point as the image time starting point of the current time plus a second preset time length, and entering step S304;

s307, if the type of the last marked time point is the default channel cut-in time point or the marked time point does not exist, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as an image time starting point; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; the process advances to step S304.

Preferably, the calculating the default channel switching-out time point and the default channel switching-in time point according to the human-computer interaction time point in the recording file includes:

s401, judging whether a time interval between a current human-computer interaction time point and a next human-computer interaction time point next to the current human-computer interaction time point is greater than a first preset time length or not, wherein an initial value of the current human-computer interaction time point is an initial moment of multi-channel audio and video recording, and if so, entering S402; otherwise, go to step S403;

s402, adding a new marking time point with the type of the default channel switching time point, and setting the value of the new marking time point as a time point corresponding to the current human-computer interaction time point plus a first preset time length;

s403, if the type of the last marked time point is the default channel cut-out time point, adding a new marked time point with the type of the default channel cut-in time, and setting the value of the new marked time point as the next human-computer interaction time point or subtracting a preset buffering time length from the next human-computer interaction time point; setting the current human-computer interaction time point as the next human-computer interaction time point; entering step S401;

or the like, or, alternatively,

the step of calculating the default channel switching-out time point and the default channel switching-in time point according to the human-computer interaction time point in the recording file comprises the following steps:

s501, judging whether a time interval between a current human-computer interaction time point and a next human-computer interaction time point next to the current human-computer interaction time point is greater than a first preset time length or not, wherein an initial value of the current human-computer interaction time point is an initial moment of multi-channel audio and video recording, and if yes, entering the step S503; otherwise, go to step S502;

s502, if the type of the last marked time point is the default channel cut-out time point, adding a new marked time point of which the type is the default channel cut-in time point, and setting the value of the new marked time point as the next human-computer interaction time point or the value of the next human-computer interaction time point minus the preset buffering time length; setting the current human-computer interaction time point as the next human-computer interaction time point, and entering step S501;

s503, setting an image time starting point as a time point corresponding to the current human-computer interaction time point plus a first preset time length, and if the type of the last marked time point is the default channel cut-in time point or no marked point exists, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as the time point corresponding to the current human-computer interaction time point plus the first preset time length;

s504, judging whether a time point corresponding to the image time starting point plus a second preset time length is larger than or equal to the next man-machine interaction time point, if so, entering the step S502; otherwise, go to step S505;

s505, judging whether the image of the default channel video image at the image time starting point is consistent with the image of the default channel video image at the image time starting point plus a second preset time length, if not, entering the step S506; otherwise, entering step S507;

s506, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point of which the type is the default channel cut-in time point, and setting the value of the new marked time point as the time point corresponding to the image time starting point plus a second preset time length; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; entering step S504;

s507, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as the time point corresponding to the image time starting point plus a second preset time length; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; entering step S504;

or the like, or, alternatively,

s601, judging whether a time interval between a current human-computer interaction time point and a next human-computer interaction time point next to the current human-computer interaction time point is greater than a first preset time length or not, wherein an initial value of the current human-computer interaction time point is an initial moment of multi-channel audio and video recording, and if so, entering a step S603; otherwise, go to step S602;

s602, if the type of the last marked time point is the default channel cut-out time point, adding a new marked time point of which the type is the default channel cut-in time point, and setting the value of the new marked time point as the next human-computer interaction time point or the value of the next human-computer interaction time point minus the preset buffering time length; setting the current human-computer interaction time point as the next human-computer interaction time point, and entering step S601;

s603, setting an image time starting point as a current human-computer interaction time point plus a time point corresponding to a first preset time length;

s604, judging whether a time point corresponding to the image time starting point plus a second preset time length is larger than or equal to the next man-machine interaction time point, if so, entering the step S602; otherwise, go to step S605;

s605, judging whether the image of the default channel video image at the image time starting point is consistent with the image of the default channel video image at the image time starting point plus the second preset time length, if not, entering the step S606; otherwise, go to step S607;

s606, if the type of the last marked time point is the default channel cut-out time point, a new marked time point with the type of the default channel cut-in time point is added, and the value of the new marked time point is set as the image time starting point; setting a new image time starting point as the image time starting point of the current time plus a second preset time length, and entering step S604;

s607, if the type of the last marked time point is the default channel cut-in time point or the marked time point does not exist, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as the image time starting point; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; the process advances to step S604.

Preferably, the human-computer interaction event includes a screen touch instruction, a keyboard operation, a mouse operation, an electronic lecture control instruction, and a camera channel display operation instruction.

Further, the multi-channel audio and video also comprises N paths of non-default channel video pictures, wherein N is more than or equal to 2, and the corresponding N paths of non-default channel video pictures are respectively marked as a first path of non-default channel video picture to an Nth path of non-default channel video picture; the step B comprises the following steps: the achievement video picture is set as a default channel video picture in a default mode, when a default channel cut-in time point exists, the achievement video picture is switched to a first path of non-default channel video picture from the default channel video picture, the achievement video picture is switched to a second path of non-default channel video picture after the first path of non-default channel video picture is played for a preset alternation duration, and the like, the achievement video picture is switched to an Nth path of non-default channel video picture in a circulating mode, and then the achievement video picture is switched to the first path of non-default channel video picture until the default channel cut-in time point is reached; when a default channel cut-in time point exists, switching the result video picture from a non-default channel video picture to a default channel video picture;

and/or when a default channel switching-out time point exists, adding a switching-out special effect at the default channel switching-out time point; when a default channel cut-in time point exists, a cut-in transition special effect is added at the default channel cut-in time point.

Further, when the resulting video frame has a switch between the non-default channel videos, a transition special effect is added at the corresponding switching time point.

Preferably, when it is required to obtain the deletion start time point and the deletion end time point, the step a is preceded by: the default audio is recorded and sampled in real time to obtain real-time sound information and corresponding sampling time, a deletion starting time point and a deletion ending time point are calculated in real time according to the real-time sound information, the deletion starting time point and the deletion ending time point are stored in a recording file, and the step B of obtaining the deletion starting time point and the deletion ending time point according to the recording file comprises the following steps: reading a deletion starting time point and a deletion ending time point from the recording file;

or, when the deletion start time point and the deletion end time point need to be obtained, the step a includes: the method comprises the steps of obtaining sound information and corresponding sampling time of default audio by real-time sampling when the default audio is recorded, and storing the sound information and the corresponding sampling time into a recording file; the step B of obtaining the deletion start time point and the deletion end time point according to the recording file includes: and calculating to obtain a deletion starting time point and a deletion ending time point according to the sound information in the recording file and the corresponding sampling time.

Preferably, when the sound information includes a sound frequency, the calculating in real time according to the real-time sound information to obtain the deletion start time point and the deletion end time point includes: when recording the default audio, judging whether the sound frequency of the default audio obtained by sampling is lower than a preset low-frequency threshold in real time, if so, recording the corresponding sampling time as a mark input time, and if the sound frequency of the default audio in a third preset time period from the mark input time is lower than the preset low-frequency threshold, setting the time corresponding to the mark input time plus the third preset time period as a deletion starting time point; marking the sound frequency of the default audio after the mark input time plus a third preset time length is greater than or equal to a preset low-frequency threshold value and the sampling time closest to the mark input time as a deletion ending time point;

and/or when the sound information comprises sound loudness, the real-time calculation of the deletion start time point and the deletion end time point according to the real-time sound information comprises: judging whether the sound loudness of the default audio obtained by sampling is located in a preset loudness range in real time when the default audio is recorded, if not, recording the corresponding sampling time as a mark input time, and if the sound loudness of the default audio within a fourth preset duration from the mark input time is not located in the preset loudness range, setting the mark input time as a deletion starting time point; and marking that the loudness of the sound of the default audio after the mark input moment plus a fourth preset time length is in the preset loudness range and the sampling moment which is closest to the mark input moment is a deletion ending time point.

Preferably, when the sound information includes a sound frequency, the calculating, according to the sound frequency in the recording file and the corresponding sampling time, to obtain the deletion start time point and the deletion end time point includes: sequentially judging whether the sound frequency of the default audio obtained by sampling is lower than a preset low-frequency threshold value or not according to the time sequence, if so, recording the corresponding sampling time as a mark input time, and if the sound frequency of the default audio in a third preset time period from the mark input time is lower than the preset low-frequency threshold value, recording the time corresponding to the mark input time plus the third preset time period as a deletion starting time point; marking the sound frequency of the default audio after the mark input time plus a third preset time length is greater than or equal to a preset low-frequency threshold value and the sampling time closest to the mark input time as a deletion ending time point;

and/or when the sound information comprises sound loudness, calculating to obtain a deletion start time point and a deletion end time point according to the sound frequency and the corresponding sampling time in the recording file, wherein the method comprises the following steps: sequentially judging whether the sound loudness of the default audio obtained by sampling is in a preset loudness range according to the time sequence, if not, recording the corresponding sampling time as a mark input time, and if the sound loudness of the default audio in a fourth preset duration from the mark input time is not in the preset loudness range, marking the input time as a deletion starting time point; and marking that the loudness of the sound of the default audio after the mark input moment plus a fourth preset time length is in the preset loudness range and the sampling moment which is closest to the mark input moment is a deletion ending time point.

The invention has the beneficial effects that:

the invention analyzes the multi-channel audio and video files of the screen and each camera equipment channel according to the recording record file, thereby realizing automatic intelligent editing and distributing results. The complicated editing work after the audio and video recording is finished is avoided, and a user does not need to have the professional ability of audio and video editing; at the same time, the automatic editing function also saves the user a lot of valuable time for editing.

Drawings

Fig. 1 is a flowchart of a multichannel audio/video automatic intelligent editing method according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating an embodiment of a result video frame editing process;

FIG. 3 is a schematic diagram of another embodiment of a resulting video frame compilation;

FIG. 4 is a diagram of another embodiment of a resulting video frame compilation;

fig. 5 is a schematic diagram of outcome audio editing according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and the following embodiments.

The invention aims to solve the problem that the existing audio and video editing method is not suitable for the teaching field, and provides a multi-channel audio and video automatic intelligent editing method.

As shown in fig. 1, the multichannel audio/video automatic intelligent editing method comprises the following steps:

A. loading a multi-channel audio and video file which is synchronously recorded in advance and a recording file which is recorded correspondingly, wherein the multi-channel audio and video at least comprises a path of default audio and a path of default channel video picture;

the multi-channel audio and video file comprises a plurality of audio and video files which are independent from each other and have consistent duration, for example, a default audio is located in a default audio file, a default channel video picture is located in a default channel video picture file, for a teaching application environment, in general, the source of the default audio can be the sound of the whole teaching environment which is directly recorded, and the source of the default channel video picture can be a teaching screen video; when the multi-channel audio/video file comprises a plurality of channel video pictures, other video pictures except the default channel video picture can be from a camera channel for recording students and/or a camera channel for recording teachers and/or a camera channel for recording an experiment platform; it should be noted that the above-mentioned video pictures only include picture information and do not include sound information. It can be understood that the recording record file is information which is recorded in the recording process and is helpful for judging the state of the multi-channel audio and video.

B. Editing the multi-channel audio/video file according to the recording record file to obtain a result video picture and a result audio;

the display state of each audio and video in the multi-channel audio and video is judged according to the recording file to be edited, so that a preliminary result video picture and a result audio can be obtained, the problem that a teacher edits each audio and video independently is solved, time is wasted, and the operation level of the teacher is tested.

It will be appreciated that the video in the production video file includes both audio and visual. In order to obtain higher-quality audio and video, the step C can be preceded by: performing noise reduction processing on the result audio; step C may further include: and (4) coding and compressing the result audio and the result video, and combining the coded result video picture and the result audio to obtain a result video file.

And judging whether the default channel video picture is in an activated state, and taking the default video picture in the activated state as a result video picture, otherwise, taking the non-default channel video picture as the result video picture. Whether the default channel video picture is in an activated state or not can be marked through the default channel cut-out time and the default channel cut-in time point, so that switching of the achievement video picture is achieved, and therefore the multi-channel audio and video can further comprise at least one non-default channel video picture; step B may further comprise: obtaining two types of marking time points which are respectively a default channel cut-out time point and a default channel cut-in time point according to the recording record file, wherein the result video picture is set as a default channel video picture by default, and when the default channel cut-out time point exists, the result video picture is switched from the default channel video picture to a non-default channel video picture until the recording time reaches the default channel cut-in time point which is next to the default channel cut-out time point; when a default channel cut-in time point exists, switching the result video picture from a non-default channel video picture to a default channel video picture;

as explained below as an application scenario of the embodiment of the present invention, in the course of a training and teaching, a student is often required to watch screen explanation content, and correspondingly, a teaching screen video is recorded as a default channel video picture, but when watching a static screen for a long time, visual fatigue is easily generated, and at the same time, the processing and processing of the static image by the brain will be weakened, and the teaching effect will be reduced. When the default channel video picture is in the non-activated state, a picture switching mechanism is created to relieve visual fatigue and keep a brain efficient processing mode, the system can judge that the teacher mainly shows body actions to convey information to students at the moment, the moment corresponding to the non-activated state can be set as a default channel switching time point, and the default video picture is switched to a channel for collecting the video picture (belonging to the non-default channel video picture) of the teacher giving lessons.

In order to avoid matching caused by displaying a path of non-default channel video pictures, the following scheme can be adopted: the multi-channel audio and video comprises N paths of non-default channel video pictures, wherein N is more than or equal to 2, and the corresponding N paths of non-default channel video pictures are respectively marked as a first path of non-default channel video picture to an Nth path of non-default channel video picture; the step B comprises the following steps: the method comprises the steps that a result video picture is set as a default channel video picture in a default mode, when a default channel cut-out time point exists, the result video picture is switched to a first non-default channel video picture from the default channel video picture, the result video picture is switched to a second non-default channel video picture after the first non-default channel video picture plays a preset alternation duration, the result video picture is switched to the Nth non-default channel video picture in a circulating mode by analogy, the result video picture is switched to the first non-default channel video picture after the Nth non-default channel video picture plays the preset alternation duration, and the result video picture is switched to the first non-default channel video picture until the default channel cut; and when a default channel cut-in time point exists after the default channel cut-in point, switching the result video picture from the non-default channel video picture to the default channel video picture.

In order to clearly indicate the existence of the video picture switching action, the following processing can be carried out, and when the default channel switching-out time point exists, a switching-out transition special effect is added at the default channel switching-out time point; when a default channel cut-in time point exists, a cut-in transition special effect is added at the default channel cut-in time point. Similarly, when the resulting video frames are switched among the non-default channel videos, a transition special effect may be added at the corresponding switching time point.

The detection points of different types are stored in the recording record file in the teaching process of a teacher, the detection points appear alternately, the video result resources after automatic editing in the later stage are switched between the teaching screen video (serving as a default video channel) and each camera picture (serving as a non-default video channel) according to the information in the recording record file, focus transfer in the teaching process of the teacher is restored, through the focus transfer, the teaching information of the teacher can be well expressed, the visual fatigue of watching result courseware can be improved, the learning efficiency is improved, and the video effect is improved by adding a transfer special effect while the video channels are switched.

In order to determine whether the default audio is in an abnormal state, such as a mute state or other states of non-educational desired sounds such as coughing, the default audio may be determined to be in an abnormal state by obtaining a deletion start time point and a deletion end time point from the recorded recording file, and step B may include obtaining a deletion start time point and a deletion end time point from the recorded recording file; and deleting all audio and video segments between the deletion starting time point and the deletion ending time point immediately behind the deletion starting time point in the multi-channel audio and video file.

An application scenario as an embodiment of the present invention is: the teacher often has another scene in the classroom teaching process, when arranging the student to read a book and study, the classroom is in a quiet state, when lasting for a long time, the video recorded in the period of time does not need to be displayed in the achievement video, the video between the deletion starting time point and the deletion ending time point has no meaning in the subsequent video study, the period is automatically deleted in the post-processing process, and meanwhile, the effect of reducing the size of the achievement video is achieved.

As a further improvement, the default channel cut-out time point, the default channel cut-in time point, the deletion start time point and the deletion end time point can be stored in an editing record file and marked on a time axis of a multi-channel audio and video display interface, if a result video automatically and intelligently edited and released after the step C is defective or a teacher has an unsatisfactory place, the teacher can manually adjust the time point on the time axis according to the multi-channel audio and video display interface, so that the personalized needs of the user are met.

When the default channel switch-out time point and the default channel switch-in time point need to be obtained, how to obtain the default channel switch-out time and the default channel switch-in time point can adopt the following two ways.

Mode F: step A is preceded by: detecting a human-computer interaction event in a source terminal of a default channel video picture in real time and recording a corresponding human-computer interaction time point when the default channel video picture is recorded, calculating a default channel cut-out time point and a default channel cut-in time point in real time according to the human-computer interaction time point, storing the default channel cut-out time point and the default channel cut-in time point into a recording record file, and obtaining the default channel cut-out time point and the default channel cut-in time point according to the recording record file in the step B comprises the following steps: and reading the default channel switching-out time point and the default channel switching-in time point from the recording record file. In the method, the calculation of the default channel switching-out time point and the default channel switching-in time point is completed in real time at the time of recording the multi-channel audio and video.

Mode G: before step A, the method comprises the following steps: when recording a default channel video picture, detecting a human-computer interaction event in a source terminal of the default channel video picture in real time, recording a corresponding human-computer interaction time point, and storing the human-computer interaction time point into a recording file; the step B of obtaining the default channel switching-out time point and the default channel switching-in time point according to the recording record file comprises the following steps: and calculating to obtain a default channel switching-out time point and a default channel switching-in time point according to the human-computer interaction time point in the recording file. In this way, the calculation of the default channel switch-out time point and the default channel switch-in time point is completed in step B.

Compared with the mode G, the mode F does not need to occupy resources to store all the human-computer interaction time points, only two types of time points, namely the default channel cut-out time point and the default channel cut-in time point, can be stored in the recording file, resources are saved, meanwhile, a large amount of analysis and comparison in the later period are not needed to obtain the default channel cut-out time point and the default channel cut-in time point, and the later-period editing efficiency is improved.

It should be noted that the human-computer interaction event may include a screen touch instruction, a keyboard operation, a mouse operation, an electronic lecture control instruction, and a display operation of each camera channel.

A screen touch instruction: any touch and writing of a user on a screen mainly relates to content marking, dot circling, problem solving derivation and the like in the teaching process, and comprises touch information such as an infrared electronic whiteboard, an all-in-one machine, an electromagnetic induction writing screen, a digital board, a capacitance writing screen and the like;

and (3) keyboard operation: the user performs computer keyboard input, such as character input, software operation instructions, function keys and the like;

mouse operation: clicking events of left and right buttons of a mouse, dragging events, scrolling events, amplifying, reducing and copying operations of matched function keys and the like;

electronic lecture draft control instruction: the wireless page turner turns pages, blanks a screen, controls a flying mouse and other instructions for the lecture on the computer;

the camera shooting channel displays an operation instruction: and calling videos of all channels to be displayed in a screen in the teaching process of a user, such as picture display of a channel of the experimental camera shooting equipment for demonstration, picture display of a channel of the object display camera shooting equipment and the like.

As more specific embodiments of the mode F, the following specific embodiments may be included:

the first embodiment is as follows:

s104, detecting a new man-machine interaction time point in real time after the latest marking time point is separated from the current man-machine interaction time point, setting the current man-machine interaction time point as the new man-machine interaction time point, adding a new marking time point with the type of default channel switching-in time point, and setting the value of the new marking time point as the new man-machine interaction time point or the time point corresponding to the new man-machine interaction time point minus the preset buffer time length; the process advances to step S101.

The second embodiment is as follows:

s207, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as the time point corresponding to the image time starting point plus a second preset time length; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; the process advances to step S204.

The third concrete implementation mode:

As more specific embodiments of the mode G, there are included the following specific embodiments:

the fourth concrete implementation mode:

s403, if the type of the last marked time point is the default channel cut-out time point, adding a new marked time point with the type of the default channel cut-in time, and setting the value of the new marked time point as the next human-computer interaction time point or subtracting a preset buffering time length from the next human-computer interaction time point; setting the current human-computer interaction time point as the next human-computer interaction time point; the process advances to step S401.

The fifth concrete implementation mode:

s507, if the type of the last marked time point is the default channel cut-in time point, adding a new marked time point of which the type is the default channel cut-out time point, and setting the value of the new marked time point as the time point corresponding to the image time starting point plus a second preset time length; setting a new image time starting point as the image time starting point of the current time plus a second preset time length; the process advances to step S504.

The sixth specific implementation mode:

It can be understood that the preset buffering duration exists to avoid the phenomena of untimely observation and information capture caused by the abrupt change of the switching process.

It can be seen that the default channel cut-in time point and the default channel cut-out time point obtained finally in the first and fourth embodiments are the same, and the difference is that one is determined while recording, and the other is determined after recording is finished; similarly, the default channel cut-in time point and the default channel cut-out time point obtained finally in the second embodiment and the fifth embodiment are the same, and the default channel cut-in time point and the default channel cut-out time point obtained finally in the third embodiment and the sixth embodiment are the same.

The following explains the application scenarios of the first and fourth embodiments more clearly with reference to fig. 2, in which the multi-channel audio/video includes one default channel of audio, one default channel of video, one first non-default channel of video and one second non-default channel of video, and the four channels of audio/video have the same duration, where the default channel of video records t0 duration and comes from the first human-computer interaction time point M1, t0< t1, there is no new human-computer interaction time point in the first preset duration t1 after M1, M2 is the second human-computer interaction time point immediately after t3 is the preset rotation duration, t2 is the preset buffer time length, t5 is the duration from the second human-computer interaction time point to the video recording end time point, t5< t1, see M2-M1> t1, and the achievement video needs to be switched from the channel t0+ t1 to the first non-default channel of video, a new human-computer interaction time point is not available at t3 in the preset rotation duration of the first path of non-default channel video picture, the result video picture is switched to the second path of non-default channel video picture to be output, a second human-computer interaction time point M2 exists after a new human-computer interaction time point is available in the preset rotation duration of the second path of non-default channel video picture, t4+ t2 duration is continuously recorded, t0+ t1+ t3+ t4 is set as a default channel cut-in time point, the result video picture needs to be cut back to the default channel video picture, so that the result video picture is formed by sequentially splicing videos at four sections of slash shadows in the graph 2, and sequentially and respectively is a default channel video picture at t0+ t1 time period, a first path of non-default channel video picture at t3 time period, a second path of non-default channel video picture at t4 time period and a default channel video picture at t2+ t5 time period, and correspondingly, cut-out transition effects may be added at t0+ t1, rotation transition effects at t0+ t1+ t3, cut-in transition effects at t0+ t1+ t3+ t4, with the resulting audio still being the default audio component at the grid shading in the figure.

In the following, referring to fig. 3, a more clear explanation is made on application scenarios of the second embodiment and the fifth embodiment, where a multi-channel audio/video includes one channel of default audio, one channel of default channel video picture and one channel of first non-default channel video picture, the durations of three channels of audio/video are the same, the default channel video picture is recorded after t0 duration comes from a first human-computer interaction time point M1, there is no new human-computer interaction time point within a first preset duration t1 after M1, t0< t1, M2 is a second human-computer interaction time point immediately after t 6324, t6 is a second preset duration, it can be seen that M2-M1> t1, a result video picture needs to switch a channel at a time point corresponding to + t1 to a first non-default channel video picture, and it is assumed that an image of the default channel video picture at a time point corresponding to t0+ t1 is not consistent with a default channel image at a time point corresponding to t0+ t1+ t6, here, it is indicated that the default channel video frame is at a time when there is only a frame change in the drone interaction, for example, a demonstration video is played, where t0+ t1+ t6 is set as a default channel cut-in time point, and it is assumed that t0+ t1+ t6 passes through 2 × t6+ t7 to reach the second human-computer interaction time point M2, and the default channel video frame in the course is always changed, t5 is a time period from the second human-computer interaction time point to a video recording end time point, t5< t1, so that the resulting video frames are sequentially spliced by videos at three-segment slash shades in fig. 3, sequentially respectively a default channel video frame at t0+ t1 time period, a first non-default channel video frame at t6 time period and a default channel video frame at 2: + t6+ t7+ t5 time period, accordingly, a special effect transition may be added at t0+ t 6342 + t6, the resulting audio is still the default audio component at the grid shading in the figure.

As will be more clearly explained below with reference to fig. 4, in the application scenarios of the third and sixth embodiments, the multi-channel audio/video includes one default channel audio, one default channel video frame and one first non-default channel video frame, the durations of the three audio/video paths are the same, the default channel video frame is recorded with a duration of t0, and then comes from a first human-computer interaction time point M1, there is no new human-computer interaction time point within a first preset duration t1 after M1, t0< t1, M2 is a second human-computer interaction time point immediately after t0, t6 is a second preset duration, it is seen that M2-M1> t1, assuming that the image of the default channel video frame at the time point corresponding to t0+ t1 is inconsistent with the image of the default channel video frame at the time point corresponding to t0+ t1+ t6, and the default channel video frame reaches a second human-computer interaction time point M2 at time point t0+ t1+ t6, and the default channel video pictures in the period are consistent and changed, the result video picture is the default channel video picture, and the result audio is still the default audio component at the grid shadow in the figure.

When the deletion start time point and the deletion end time point are required to be obtained for audio editing, how to obtain the deletion start time point and the deletion end time point can be performed in the following two ways.

Mode H: step A is preceded by: the real-time sampling is carried out when the default audio is recorded to obtain real-time sound information and corresponding sampling time, the deletion starting time point and the deletion ending time point are obtained through real-time calculation according to the real-time sound information, the deletion starting time point and the deletion ending time point are stored in a recording file, and the step B of obtaining the deletion starting time point and the deletion ending time point according to the recording file comprises the following steps: reading a deletion starting time point and a deletion ending time point from the recording file;

mode K: step A is preceded by: real-time sampling is carried out when recording default audio to obtain sound information and corresponding sampling time of the default audio, and the sound information and the corresponding sampling time are stored in a recording file; the step B of obtaining the deletion start time point and the deletion end time point according to the recording file includes: and calculating to obtain a deletion starting time point and a deletion ending time point according to the sound information in the recording file and the corresponding sampling time.

That is, the mode H is to calculate the corresponding deletion start time point and deletion end time point when recording, and the mode K is to calculate the corresponding deletion start time point and deletion end time point after recording of the audio and video.

As for the mode H, more specific embodiments thereof include:

when the sound information includes sound frequency, the real-time calculation according to the real-time sound information to obtain a deletion start time point and a deletion end time point includes: when recording the default audio, judging whether the sound frequency of the default audio obtained by sampling is lower than a preset low-frequency threshold in real time, if so, recording the corresponding sampling time as a mark input time, and if the sound frequency of the default audio in a third preset time period from the mark input time is lower than the preset low-frequency threshold, setting the time corresponding to the mark input time plus the third preset time period as a deletion starting time point; marking the sound frequency of the default audio after the mark input time plus a third preset time length is greater than or equal to a preset low-frequency threshold value and the sampling time closest to the mark input time as a deletion ending time point;

More specific embodiments of mode K include:

when the sound information includes a sound frequency, calculating a deletion start time point and a deletion end time point according to the sound frequency in the recording file and the corresponding sampling time, including: sequentially judging whether the sound frequency of the default audio obtained by sampling is lower than a preset low-frequency threshold value or not according to the time sequence, if so, recording the corresponding sampling time as a mark input time, and if the sound frequency of the default audio in a third preset time period from the mark input time is lower than the preset low-frequency threshold value, recording the time corresponding to the mark input time plus the third preset time period as a deletion starting time point; marking the sound frequency of the default audio after the mark input time plus a third preset time length is greater than or equal to a preset low-frequency threshold value and the sampling time closest to the mark input time as a deletion ending time point;

It is to be understood that the sound between the deletion start time point and the deletion end time point is an abnormal sound, the deletion start time point and the deletion end time point obtained by the processing in the manner H and the manner K are consistent only in the inconsistent processing sequence, and in order to avoid a deletion error, the duration of the third preset duration may be longer than other preset durations, the operation of deleting the audios and videos at the deletion start time point and the deletion end time point may be placed after the video processing operation, and the video picture obtained after the switching of the video picture according to the default channel switch-out time point and the default channel switch-in time point is referred to as a preliminary video picture, which is for convenience of understanding, and is further explained below with reference to fig. 5.

As shown in fig. 5, after the time t8 passes through the default audio, the sound loudness of the default audio obtained by sampling at the time point M3 is detected to be not within the preset loudness range, the sound loudness of the default audio within the time t9 of the fourth preset duration from the time point t8 is not within the preset loudness range, and the sound loudness of the default audio detected at the time point M4 is within the preset loudness range, then M3 is the deletion start time point, and M4 is the deletion end time point; when the default audio reaches M5, detecting that the sound frequency of the default audio is lower than a preset low-frequency threshold, and the sound frequency of the default audio within a third preset time period t10 after M5 is lower than the preset low-frequency threshold, setting a time point corresponding to the third preset time period t10 after M5 as a deletion start time point, and setting an audio end time point as a deletion end time point, assuming that the sound frequencies of the default audio after the third preset time period t10 after M5 and after the audio end are all lower than the preset low-frequency threshold. And finally, splicing the obtained result audio by the content sequence of the two grid shadows in the default audio in the graph, and splicing the result video picture by the content sequence of the two slash shadows in the preliminary result video picture in the graph.

Claims

1. The method for automatically and intelligently editing the multi-channel audio and video is characterized by comprising the following steps of:

A. loading a multi-channel audio and video file which is synchronously recorded in advance and a recording file which is recorded correspondingly, wherein the multi-channel audio and video at least comprises a path of default audio and a path of default channel video picture and also comprises at least a path of non-default channel video picture;

B. editing the multi-channel audio and video file according to the recording record file to obtain a result video picture and a result audio: obtaining two types of marking time points which are respectively a default channel cut-out time point and a default channel cut-in time point according to the recording record file, wherein the result video picture is set as a default channel video picture by default, and when the default channel cut-out time point exists, the result video picture is switched from the default channel video picture to a non-default channel video picture until the recording time reaches the default channel cut-in time point which is next to the default channel cut-out time point; when a default channel cut-in time point exists, switching the result video picture from a non-default channel video picture to a default channel video picture; obtaining a deletion starting time point and a deletion ending time point according to the recording record file; deleting all audio and video segments from the deletion starting time point to the deletion ending time point which is next to the deletion starting time point in the multi-channel audio and video file;

C. combining the result video picture and the result audio to obtain a result video file;

when the default channel switch-out time point and the default channel switch-in time point need to be obtained, the step a includes: detecting a human-computer interaction event in a source terminal of a default channel video picture in real time and recording a corresponding human-computer interaction time point when the default channel video picture is recorded, calculating a default channel cut-out time point and a default channel cut-in time point in real time according to the human-computer interaction time point, storing the default channel cut-out time point and the default channel cut-in time point into a recording file, wherein the step B of obtaining the default channel cut-out time point and the default channel cut-in time point according to the recording file comprises the following steps: reading a default channel switching-out time point and a default channel switching-in time point from a recording record file;

2. The method of claim 1, wherein the calculating the default channel switch-out time point and the default channel switch-in time point in real time according to the human-computer interaction time point comprises: detecting the human-computer interaction in a source terminal of the default channel video image in real time and recording the corresponding human-computer interaction time point when recording the default channel video image, and the following processes:

or the like, or, alternatively,

3. The method of claim 1, wherein calculating the default channel switch-out time point and the default channel switch-in time point according to the human-computer interaction time point in the recording file comprises:

or the like, or, alternatively,

4. The method of claim 1, wherein the human-computer interaction event comprises a screen touch command, a keyboard operation, a mouse operation, an electronic lecture control command, and a camera channel presentation operation command.

5. The method according to any one of claims 1 to 4, wherein the multi-channel audio/video further comprises N non-default channel video pictures, N is greater than or equal to 2, and the corresponding N non-default channel video pictures are respectively marked as a first non-default channel video picture to an Nth non-default channel video picture; the step B comprises the following steps: the achievement video picture is set as a default channel video picture in a default mode, when a default channel cut-in time point exists, the achievement video picture is switched to a first path of non-default channel video picture from the default channel video picture, the achievement video picture is switched to a second path of non-default channel video picture after the first path of non-default channel video picture is played for a preset alternation duration, and the like, the achievement video picture is switched to an Nth path of non-default channel video picture in a circulating mode, and then the achievement video picture is switched to the first path of non-default channel video picture until the default channel cut-in time point is reached; when a default channel cut-in time point exists, switching the result video picture from a non-default channel video picture to a default channel video picture;

6. The method of claim 5, wherein when a resulting video picture has a switch between non-default channel videos, a rotation transition special effect is added at a corresponding switch time point.

7. The method of claim 1, wherein when a deletion start time point and a deletion end time point are required, the step a is preceded by: the default audio is recorded and sampled in real time to obtain real-time sound information and corresponding sampling time, a deletion starting time point and a deletion ending time point are calculated in real time according to the real-time sound information, the deletion starting time point and the deletion ending time point are stored in a recording file, and the step B of obtaining the deletion starting time point and the deletion ending time point according to the recording file comprises the following steps: reading a deletion starting time point and a deletion ending time point from the recording file;

8. The method of claim 7, wherein when the sound information includes a sound frequency, the calculating a deletion start time point and a deletion end time point in real time from real-time sound information includes: when recording the default audio, judging whether the sound frequency of the default audio obtained by sampling is lower than a preset low-frequency threshold in real time, if so, recording the corresponding sampling time as a mark input time, and if the sound frequency of the default audio in a third preset time period from the mark input time is lower than the preset low-frequency threshold, setting the time corresponding to the mark input time plus the third preset time period as a deletion starting time point; marking the sound frequency of the default audio after the mark input time plus a third preset time length is greater than or equal to a preset low-frequency threshold value and the sampling time closest to the mark input time as a deletion ending time point;

9. The method according to claim 7 or 8, wherein when the sound information includes a sound frequency, the calculating a deletion start time point and a deletion end time point according to the sound frequency and a corresponding sampling time in the recorded file comprises: sequentially judging whether the sound frequency of the default audio obtained by sampling is lower than a preset low-frequency threshold value or not according to the time sequence, if so, recording the corresponding sampling time as a mark input time, and if the sound frequency of the default audio in a third preset time period from the mark input time is lower than the preset low-frequency threshold value, recording the time corresponding to the mark input time plus the third preset time period as a deletion starting time point; marking the sound frequency of the default audio after the mark input time plus a third preset time length is greater than or equal to a preset low-frequency threshold value and the sampling time closest to the mark input time as a deletion ending time point;