JP7343378B2 - editing system - Google Patents

Description

TECHNICAL FIELD The present invention relates to an editing system that is capable of editing and retransmitting broadcast video mainly used in broadcast stations and the like.

2. Description of the Related Art In recent years, editing systems have been put into practical use at broadcasting stations and the like, in which video data is stored as editing material in a video server for the material, non-linear editing is performed, and the data is sent out for broadcasting.

As a conventional editing system, for example, referring to Patent Document 1, a technology is described that includes a processing target recognition unit that recognizes each video part and audio part to be processed in order to identify an object in a video. .

On the other hand, conventional editing systems also store broadcast simultaneously recorded video (hereinafter simply referred to as "broadcast video"), which is a simultaneous recording of the broadcast video. When retransmitting such broadcast video through staggered distribution, rebroadcasting, etc., it is necessary to mask unnecessary areas added during broadcasting and generate video material data (processed video data) that is close to the original video. Examples of this unnecessary area include additional displays such as "L-shaped", time display, images and subtitles showing weather forecasts, emergency reports, tsunami warning information, disaster information, etc. (Hereinafter, these additional displays will be referred to as (referred to as "L-shape, etc."). In addition, "L-shaped" is called "L-shaped screen", for example, the original program broadcast screen is reduced somewhat to the lower right corner, and the left and upper sides of the excess screen are used as L-shaped space. This refers to video processing that displays information about disasters, etc. In addition to the L-shaped screen, the additional display also includes a U-shaped additional display that surrounds all the sides and displays in a reduced size.
In this case, it was necessary to perform masking, trimming, etc. of unnecessary areas such as the L-shape by manual editing.

JP2019-62381A

However, in conventional editing systems, when unnecessary areas such as the L-shape are manually edited, image quality deterioration is unavoidable due to trimming errors, encoding and mask processing during simultaneous recording, partial enlargement due to trimming, etc.

The present invention has been made in view of this situation, and an object of the present invention is to solve the above-mentioned problems.

The editing system of the present invention is an editing system that retransmits a broadcast video, and includes an unnecessary area specifying means for specifying an unnecessary area that is continuously displayed at a specific location of the broadcast video, and an unnecessary area specifying means that identifies the unnecessary area by the unnecessary area specifying means. an unnecessary area processing means for creating a processed video in which the unnecessary area has been processed to delete or make it less conspicuous from the broadcast video; and storage from the processed video and/or the broadcast video processed by the unnecessary area processing means. The image processing apparatus is characterized by comprising an original video specifying means for specifying the original image that has been processed, and an image quality improving means for increasing the image quality of the processed image based on the original image specified by the original image specifying means.
The editing system of the present invention is characterized in that the unnecessary area identifying means identifies the unnecessary area using a model that has learned the characteristics of the area to be deleted.
In the editing system of the present invention, the image quality improving means performs edge enhancement or synthesis on the processed video using edge information and/or color information based on the original video, and/or synthesis by cutting out the original video. The feature is that the image quality is improved by doing this.
The editing system of the present invention is characterized in that the original video identifying means extracts common points in the images of the processed video and the original video, and identifies the original video based on the extracted common points. do.
The editing system of the present invention includes: a deletion part specifying means for analyzing the audio corresponding to the broadcast video and specifying a deletion part; an original audio specifying means for identifying the original audio corresponding to the original video; and the original audio The apparatus is characterized by further comprising a sound quality enhancement processing means for enhancing the sound quality of the deletion portion of the voice specified by the deletion portion specifying means based on the original voice specified by the specifying means.
The editing system of the present invention is characterized in that the deletion location identifying means performs audio analysis using a specific model and identifies the location of the alarm sound in the audio.

According to the present invention, a processed video is created in which an unnecessary area that is continuously displayed in a specific part of a broadcast video is deleted or made inconspicuous, and the original video is extracted from the processed video and/or the broadcast video. By specifying video data and increasing the quality of processed video based on this original video data, it is possible to provide an editing system that can suppress image quality deterioration during retransmission.

1 is a system configuration diagram showing a schematic configuration of an editing system X according to an embodiment of the present invention. 3 is a flowchart showing the flow of retransmission processing according to an embodiment of the present invention. FIG. 3 is a conceptual diagram showing image quality improvement processing in the retransmission processing shown in FIG. 2; FIG. 3 is a conceptual diagram showing high-quality sound processing in the retransmission processing shown in FIG. 2; FIG. 2 is a conceptual diagram of retransmission by a conventional video server system.

<Embodiment>
[Control configuration of editing system X]
Embodiments of the present invention will be described below with reference to the drawings.
Editing system X is an editing system (video server system) used at broadcasting stations and the like. The editing system X can retransmit the broadcast video included in the broadcast video data 200 by staggered distribution, rebroadcasting, or the like. At this time, the editing system X can delete unnecessary parts of the previously broadcast video.
According to FIG. 1, the editing system X includes an analysis device 1, a storage server 2, a recording device 3, and an editing device 4 connected through a network 5.

The analysis device 1 is a device for analyzing the contents of broadcast video data 200 etc. stored in the storage server 2. For example, the analysis device 1 performs image component analysis including various filter processing and OCR (Optical Character Recognition), a convolutional neural network, a GAN (Generative Adversarial Network), and an RNN on a video (image) included in video data. (Recurrent Neural Network), LSTM (Long short term memory network), other multilayer neural networks, kernel machines, decision trees, Bayesian networks, HMM (Hidden Markov Model), and other statistical methods, etc. It is a device that performs calculations such as intelligence. Furthermore, the analysis device 1 can also analyze the audio data 300 using audio component analysis or AI.
The detailed configuration of the analysis device 1 will be described later.

The recording device 3 is a device that records image data, audio data 300, etc., and encodes (converts) these data into various codecs captured by the image using an image and audio encoder.
In this embodiment, the recording device 3 records and encodes uncompressed image data captured by an imaging unit 30, which will be described later, for example. The recording device 3 may also record image data from a server, VTR, or other device located at another station via a dedicated line or network 5, or may import image data in a file such as MXF (Media eXchange Format). You can also record it by doing so. The video encoding method (codec) used for encoding by the encoder is, for example, MPEG2, H. 264, H. Although it is possible to use H.265 or the like, it is not limited to this. The recording device 3 can transmit the encoded data to the storage server 2 or the transmission equipment for reproduction.

The storage server 2 is a device such as a server that stores broadcast video data 200 and transmits it to other devices. In this embodiment, the storage server 2 functions as a material video server that stores broadcast video data 200, original video data 220, etc. of recorded materials (material video, material files) recorded by the recording device 3. In addition, the storage server 2 may include a multiplexing function using a multiplexer (MUX).
Details of the data stored in the storage server 2 will be described later.

The editing device 4 is a so-called general-purpose nonlinear editing machine (device). The editing device 4 performs editing processing such as rendering editing and cut editing. Among these, rendering editing is a process of editing the broadcast video data 200 stored in the storage server 2 while actually rendering it. Cut editing is a process of clipping without rendering.

In this embodiment, the editing device 4 includes a display section, a keyboard, a pointing device, an operating device, etc. (not shown). Furthermore, the editing device 4 includes an editing control means (editing means) that is a computer that actually performs this editing work, a display unit (display) that displays broadcast video data 200, an editing timeline, etc., and a display that displays editing instructions. It is equipped with an operation panel (operation means) for inputting information.

The editing device 4 is a device that can edit broadcast video data 200, original video data 220, etc. by referring to the storage server 2. The editing device 4 can perform cut editing, rendering editing, etc. by having the user operate the operation panel and specify a portion to be edited. Then, the editing device 4 transmits the edited information such as the edited broadcast video data 200 and the original video data 220 to the storage server 2 for storage.

The editing information used in these editing processes includes, for example, the video frame position of the portion to be processed, the coordinates on the video, the position range of the audio sample, the content of the processing, and the like. When the processing target is video, the types of editing processing described above include various image effects, connections between clips and their effects, brightness and color adjustment processing, fade-in, fade-out, volume adjustment, and the like.

The network 5 includes a LAN (Local Area Network) connecting each device, an optical fiber network, and c. It is a communication means that interconnects devices such as a link, wireless LAN (WiFi), and mobile phone network to perform communication. The network 5 may be a dedicated line, an intranet, the Internet, etc., or a mixture of these may constitute a VPN (Virtual Private Network). Furthermore, the network 5 may be connected using various protocols using an IP network such as TCP/IP or UDP.

In addition to this, the editing system X also includes transmission equipment (devices) including a transmission server for a general-purpose broadcasting station. These devices output (on-air) the material video recorded in the storage server 2 and the broadcast video recorded in the storage server 2. In addition, it is also possible to play back broadcast video for preview purposes.

To explain more specifically, the analysis device 1 includes a control unit 10 as a part of hardware resources.

The control unit 10 is an information processing unit that realizes a functional unit to be described later and executes each process of the retransmission process of this embodiment. The control unit 10 includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), a DSP (Digital Signal Processor), and an ASIC (Application Specific It consists of processors, processors for specific applications), etc. Thereby, the control unit 10 can perform image component analysis, audio component analysis, and AI processing for video and audio at high speed using batch processing or the like.

The storage server 2 includes a storage unit 11 as part of its hardware resources.

The storage unit 11 is a non-temporary recording medium. The storage unit 11 is configured as a video storage such as, for example, an SSD (Solid State Disk), an HDD (Hard Disk Drive), a magnetic cartridge, a tape drive, an optical disk array, or the like.
This video storage stores, for example, material video data (material data), video data of a broadcast video such as a completed program, broadcast video data 200 that is a broadcast video, and the like. The files stored in the storage server 2 are transferred to a playback device according to the broadcast schedule of the program, or are used in program editing processing by the editing device 4. Details of these data will be described later.
In addition, the storage unit 11 also includes general ROM (Read Only Memory), RAM (Random Access Memory), and the like. These stores include programs for processing executed by the storage server 2 and the control unit 10 of the analysis device 1, databases, temporary data, and other various files.

The recording device 3 includes an imaging section 30 (imaging means).

The imaging unit 30 is an imaging device such as a camera using a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) element. The imaging unit 30 may be built into the recording device 3 or may be a connected external camera.
The imaging unit 30 digitally converts the captured image and transmits it to the recording device 3 as, for example, HD-SDI standard image data. At this time, audio data from a microphone attached to the imaging unit 30 or provided externally may also be transmitted to the recording device 3 almost at the same time. Alternatively, these image data and audio data can also be transmitted to the recording device 3 via a mixer or various equipment.

Next, the functional configuration of the analysis device 1 and the details of the data stored in the storage server 2 will be explained.
The control unit 10 includes an unnecessary area specifying means 100, an unnecessary area processing means 110, an original video specifying means 120, a high image quality improving means 130, a deletion part specifying means 140, an original audio specifying means 150, and a high sound quality processing means 160. .
The storage unit 11 stores broadcast video data 200, processed video data 210, original video data 220, audio data 300, processed audio data 310, and original audio data 320.

The unnecessary area specifying means 100 acquires the broadcast video data 200 from the recording device 3 and specifies an unnecessary area that is continuously displayed at a specific part of the broadcast video included in the broadcast video data 200. At this time, the unnecessary area specifying means 100 analyzes the content of the video, for example, and specifies the position on the video as an L-shape or the like as an unnecessary area.
Specifically, the unnecessary area identifying means 100 can identify unnecessary areas using a model that has learned the characteristics of the area to be deleted. This model may use image component analysis or AI, for example.

Processed video data 210 is created in which the unnecessary area identified by unnecessary area identifying means 100 is deleted from broadcast video data 200 or processed to make it less noticeable.
The unnecessary area processing means 110 processes the identified unnecessary area, for example, by automatically trimming, enlarging, mask editing, etc., or any combination thereof (hereinafter simply referred to as "mask processing") to make it less noticeable. .
The unnecessary area processing means 110 stores the created processed video data 210 in the storage server 2.

The original video identifying means 120 identifies the original video data 220 stored in the storage server 2 from the processed video data 210 and/or the broadcast video data 200 processed by the unnecessary area processing means. For example, the original video identifying means 120 analyzes the video content of the processed video data 210 and/or the broadcast video data 200, and stores the video data (material) that is the material of the broadcast video stored in the storage unit 11 of the storage server 2. data) to identify the original video data 220 used for broadcasting.
More specifically, the original video identifying means 120 extracts common points in the image between the processed video and the original video included in the original video data 220, and based on the extracted common points, the original video data 220 contains the processed video. It is possible to identify the original video.

The image quality improving means 130 improves the image quality of the processed video data 210 based on the original video data 220 specified by the original video specifying means 120.
Specifically, the image quality improvement unit 130 performs edge enhancement or synthesis using edge information and/or color information based on the original video included in the original video data 220, and/or for each processed video of the processed video data 210. Alternatively, high image quality can be achieved by cutting out and synthesizing the original video included in the original video data 220.

Deletion location specifying means 140 analyzes audio data 300 corresponding to broadcast video and identifies a deletion location.
Specifically, the deletion location specifying means 140 performs audio analysis using a specific model and identifies the location of the alarm sound in the audio.

Original audio identifying means 150 identifies original audio data 320 corresponding to original video data 220.
The original audio specifying means 150 identifies the original audio data 320 that corresponds to the original video data 220 used for broadcasting, for example, by comparing it with the audio material data stored in the storage server 2.

Based on the original audio data 320 specified by the original audio specifying means 150, the high-quality sound processing means 160 enhances the sound quality of the deletion portion of the audio specified by the deletion portion specifying means 140.
The high-quality sound processing unit 160 can improve the sound quality by, for example, synthesizing the opposite phase of the alarm sound and/or synthesizing by cutting out the original audio data 320.

Broadcast video data 200 is data of broadcast video. In this embodiment, the broadcast video data 200 includes broadcast video such as a broadcast simultaneous recording video that is a simultaneous recording of a video sent out during broadcasting. In this embodiment, the broadcast video data 200 uses, for example, an MXF format file. MXF is a type of container format file that stores so-called business video files. Specifically, MXF is used in broadcasting equipment such as camcorders, recording/playback machines, nonlinear editing machines, and playout equipment, and is used to wrap data in various formats such as video and audio together with metadata. I can do it. In this embodiment, this metadata can include, for example, data on identified unnecessary areas, feature data in the video, data on common points in the image with the original video, and the like. Further, the metadata can include, for example, frame rate, frame size, creation date, photographer of the imaging unit 30, and various information about the material video. As this various information, it is possible to use, for example, title, content, playback time, scene information, object information including people in the video, shooting location, shooting date and time, and the like.

Processed video data 210 is video data that has been processed from broadcast video data 200 by deleting unnecessary areas or making them less noticeable. This processed video data 210 may also be data in MXF format, or data in an intermediate format for editing before being finally processed into data for transmission at a transmission facility. Alternatively, the processed video data 210 may be data in the same format as the material data, such as the original video data 220. Furthermore, the processed video data 210 may be sent out after being enhanced in image quality with the original video data 220 as described above.

The original video data 220 is material data stored in the storage server 2. The original video data 220 includes program data used in the actual broadcast video data 200, data of its materials, and the like. The original video data 220 and the broadcast video data 200 may have different video formats, and may have higher image quality than the broadcast video data 200, such as with lower compression or non-compression. That is, the format of the original video data 220 may be a format other than the MXF format or may be an original format. Furthermore, the original video data 220 may be a video stream recorded as material data from the recording device 3 and multiplexed.
Additionally, in this embodiment, the original video data 220 may include feature data in the video, data on common points in the image with the broadcast video, and the like.

Audio data 300 is audio data corresponding to broadcast video data 200. The audio data 300 includes broadcast audio such as broadcast simultaneous recording, which is a simultaneous recording of broadcast audio. This broadcast sound may include, for example, a warning sound such as a chime, a buzzer, or a short sound to draw attention to a location such as an L-shape. Specifically, the audio data 300 may be collected as a stream included in a container format such as MXF format, for example. Alternatively, this broadcast audio may be, for example, a WAV format file with various quantization bit numbers and frequencies, a file in various compression formats, or an audio stream format file. As described later, the audio data 300 may be processed with an alarm sound having an opposite phase, or may be replaced with the original audio data 320.

Processed audio data 310 is audio data included in processed video data 210.
In this embodiment, the processed audio data 310 may be the same data as the audio data 300. The processed audio data 310 may be enhanced in quality with the audio data 300 as described above, and may be sent together with the processed video data 210.
The processed audio data 310 may also be a WAV format file, a file in various compression formats or an audio stream format, data in an intermediate format for editing, or the like.

The original audio data 320 is audio data corresponding to the original video data 220. The original audio data 320 may also be a WAV format file, a file in various compression formats or an audio stream format, data in an intermediate format for editing, or the like.

Here, each of the above-mentioned functional units is realized by the control program and the like stored in the storage unit 11 being executed by the control unit 10.
Note that each of these functional units may be configured as a circuit using an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like.

[Resend processing of editing system X]
Next, the retransmission process using the editing system X according to the embodiment of the present invention will be described in more detail with reference to FIGS. 2 to 4.
In the retransmission process of this embodiment, a high image quality process is performed to improve the image quality of broadcast video data 200 such as a broadcast simultaneous recorded video, and a high sound quality process is performed to improve the sound quality of audio data 300 such as a broadcast simultaneous recorded video. Execute. These processes may be executed concurrently using threads, processes, or the like.
The retransmission process by the editing system X will be explained below using the flowcharts in FIG.

First, the image quality improvement process in the retransmission process will be explained in detail step by step using the flowchart of FIG. 2(a) and FIG. 3.

In step S100, the unnecessary area specifying means 100 performs initial processing.
According to FIG. 3A, the unnecessary area specifying means 100 causes the model to learn the characteristics of the area to be deleted, such as an L-shape, for each broadcasting station or the like. For this reason, for example, the unnecessary area identifying means 100 takes advantage of the fact that the shape of characters included in the L character, the shape of the time display, the display position of subtitles, the font, etc. can only be modified to a specific pattern. Use as a model. In other words, since video server systems are generally owned and operated by a single broadcasting station, processing methods such as the L-shape, time display shape, subtitle display position, font, etc. are limited to a certain extent. You can take advantage of being limited to patterns. This is because the video server system is operated for each broadcasting station, and the format used for inserting L characters and the like has a certain degree of regularity.

Specifically, when performing image component analysis, the unnecessary area specifying means 100 detects, for example, a specific image component included in an L-shape or the like. This indicates that, for example, in image component analysis, a specific image component included in an L-shape or the like is detected.
Alternatively, when using AI, the unnecessary area specifying means 100 can learn in advance that a specific pattern such as an L character, a time display, etc. are unnecessary areas for each broadcasting station or the like. This indicates, for example, that the AI recognizes an L-shape or the like indicating a specific pattern as an area to be deleted.

On the other hand, unnecessary area specifying means 100 acquires broadcast video data 200. Specifically, the transmission video transmitted from the transmission equipment at the time of broadcasting is simultaneously recorded, and the broadcast simultaneously recorded video is stored in the storage unit 11 as broadcast video data 200.

Next, in step S101, the unnecessary area specifying means 100 performs unnecessary area specifying processing.
According to FIG. 3A, the unnecessary area specifying means 100 specifies an unnecessary area that is continuously displayed at a specific part of the broadcast video included in the broadcast video data 200.
As a method for deleting L-characters etc., video analysis is first required. In the present embodiment, the unnecessary area identifying means 100 identifies unnecessary areas using a model that has learned the characteristics of the area to be deleted. Specifically, the unnecessary area identification means 100 analyzes the entire broadcast video data 200, and identifies accidental occurrences of similar images that occur in a part of the program and images that have been intentionally inserted. Distinguish between L-shape, etc. That is, the unnecessary area specifying means 100 can specify the position of an unnecessary L-shape or the like based on the contents of the broadcast video data 200.
The unnecessary area identifying means 100 may use the above-mentioned image component analysis or AI as this model. With these, it is possible to identify the presence or absence of an L-shape, etc. included in the video, and the range of its unnecessary area.

Here, the unnecessary area specifying means 100 does not need to analyze every frame of the broadcast video data 200, and may thin out frames for analysis. This thinning interval can be set depending on the characteristics of the analysis model, etc.
To be more specific, a characteristic of the L-shape etc. is that it is basically displayed continuously for a long period of time throughout the broadcast. That is, when the entire broadcast video data 200 includes an L-shape or the like, the unnecessary area identifying means 100 can identify the unnecessary area by analyzing one arbitrary frame of video.

However, the L-shape etc. may not be the entire story. Furthermore, there is a possibility that the display of the letter L or the like may be canceled while a commercial is being aired. For this reason, the unnecessary area specifying means 100 can also specify unnecessary areas by analyzing every frame every about 5 seconds. In this case, if the presence or absence of an L-shape or the like changes in the broadcast video data 200, the unnecessary area specifying means 100 can calculate the point of change by analyzing each frame before and after the location where the unnecessary area exists. It is possible. Further, if the size of the L-shape or the like changes at a change point, the unnecessary area specifying means 100 can specify the area range of the L-shape or the like for each frame.

Further, the unnecessary area specifying means 100 analyzes the broadcast video data 200 using metadata and OCR, analyzes the context of character strings included in L characters, etc., and deletes them depending on the content of the information included. It is also possible to determine whether the information is the original content or information that originally existed in the video content.
The unnecessary area specifying means 100 can store the locations identified as unnecessary areas in the metadata of the broadcast video data 200.

FIG. 3A shows an example in which an unnecessary area A1 that is an L-shaped area, an unnecessary area A2 that is a time display area, and an unnecessary area A3 that is a map area are identified.

Next, in step S102, the unnecessary area processing means 110 determines whether there is an unnecessary area. For example, if the unnecessary area specified by the unnecessary area specifying means 100 is set in the metadata of the broadcast video data 200, the unnecessary area processing means 110 determines Yes.
If Yes, the unnecessary area processing means 110 advances the process to step S103.
In the case of No, the unnecessary area processing means 110 ends the image quality improvement process in the retransmission process.

If there is an unnecessary area, the unnecessary area processing means 110 performs unnecessary area processing processing in step S103.
Unnecessary area processing means 110 creates processed video data 210 in which unnecessary areas are deleted from broadcast video data 200 or processed to make them less noticeable. As a process for deleting or making the unnecessary area less noticeable, the unnecessary area processing means 110, for example, edits the identified L-shaped unnecessary area so that it is not directly displayed on the screen, and creates processed video data 210. . Specifically, the unnecessary area processing unit 110 performs editing such as automatically trimming the L-shape, enlarging areas other than the L-shape, and displaying the entire screen. With this, it is possible to delete the L-shaped display from the processed video data 210.

On the other hand, the unnecessary area processing means 110 automatically performs mask processing on unnecessary areas such as the specified time and subtitles, for example. In this case, the unnecessary region processing means 110 performs editing such as blurring processing such as Gaussian blur on the unnecessary region. As a result, in the processed video data 210, displays such as the time, subtitles, and maps become inconspicuous or invisible.
That is, the unnecessary area processing means 110. It is possible to create processed video data 210 that has been automatically masked.

FIG. 3(b) shows an example of processed video data 210 in which unnecessary areas have been deleted or made less noticeable.

Next, in step S104, the original video identifying means 120 performs original video identifying processing.
It is highly possible that programs broadcast using a video server system use video stored within the video server system. For this purpose, the video stored in the video server system is searched and identified.
Specifically, the original video identifying means 120 identifies the original video data 220 stored in the storage server 2 from the processed video data 210 and/or the broadcast video data 200 processed by the unnecessary area processing means. More specifically, the original video identifying means 120 is capable of extracting common points between the video included in the processed video data 210 and the original video included in the original video data 220.

Here, the original video specifying means 120, for example, performs component analysis on characteristic data in the processed video data 210 and the original material video data, and stores the analyzed data as metadata or the like in each of them. The original video identifying means 120 can extract common points by comparing the feature data of the processed video data 210 and the original material video data in chronological order. The feature data in the video can include, for example, text information, screen color information, drawn object information, thumbnail image information, and the like. Alternatively, the original video specifying means 120 can also directly cause the AI to learn the processed video data 210 and the original material video data, and perform a process of comparing them based on the extracted common points.

That is, the original video identifying means 120 identifies the original video included in the original video data 220 based on the extracted common points in the processed video data 210 and/or the broadcast video data 200.
This search makes it possible to identify the original video used for broadcasting.

FIG. 3C shows an example of original video data 220 specified in correspondence to processed video data 210 and broadcast video data 200.

Next, in step S105, the image quality improvement unit 130 determines whether or not the original video data 220 has been identified. The image quality improving means 130 determines Yes when the original video identifying means 120 is able to identify the original video data 220 from the broadcast video data 200.
If Yes, the image quality improvement unit 130 advances the process to step S106.
In the case of No, the image quality improvement unit 130 ends the image quality improvement process in the retransmission process.

If the original video data 220 can be identified, the image quality improving means 130 performs high image quality processing in step S106.
The high quality sound processing means 160 improves the quality of the processed video based on the original video data 220 specified by the original video specifying means 120.
The image quality improvement unit 130 performs super resolution processing and image quality improvement processing on the processed video data 210. Specifically, the image quality improvement unit 130 can improve the image quality by cutting out and combining original videos included in the original video data 220.

3(c) and 3(e) show examples in which a part or the entire screen of the original video is cut out from the original video data 220 and combined with the processed video data 210 by overwriting or the like.

Further, the image quality improvement unit 130 can perform edge enhancement or synthesis on the processed video data 210 using edge information and/or color information based on the original video included in the original video data 220.

FIGS. 3(d) and 3(e) show examples in which edge information and color information are extracted from this original video data 220 and combined with processed video data 210.

In addition, the image quality improving means 130 can also improve the image quality of the processed video data 210 using AI such as GAN.

Thereafter, this high-quality processed video data 210 is retransmitted by the transmission equipment. At this time, the audio data 300 that has been subjected to the high-quality sound processing described below may be retransmitted as a file in a container format such as MXF format. Note that if an unnecessary area is not identified and the processed video data 210 is not generated, it is also possible to retransmit the broadcast video data 200 as is.
With the above steps, the image quality improvement process in the retransmission process is completed.

Next, the high-quality sound processing in the retransmission processing will be explained step by step in detail using the flowchart of FIG. 2(b) and FIG. 4.

First, in step S110, the deletion location specifying means 140 performs initial processing.
The deletion portion specifying means 140 sets, for example, a model to be searched from the audio data 300 as a specific model, similarly to the above-described video quality enhancement process. Here, as mentioned above, video server systems are generally owned and operated by a single broadcasting station, so the superimposed audio is limited to a certain specific pattern. Is possible. This indicates, for example, that characteristics such as a specific melody, voice pattern, voice frequency change, etc. are recognized as objects to be deleted.
In this embodiment, an example of setting a model for an alarm sound will be described. This model may be learned and set by AI using, for example, a statistical model such as HMM, or a time series model such as RNN or LSTM.

Next, in step S111, the deletion location identifying means 140 performs deletion location identification processing.
Deletion location specifying means 140 analyzes audio data 300 corresponding to broadcast video data 200 and identifies a deletion location. For example, the deletion location specifying means 140 acquires the audio data 300 associated with the container format video stream of the broadcast video data 200 from the storage server 2 and analyzes it.

According to FIG. 4A, the deletion location specifying means 140 analyzes the audio data 300 using a specific model and identifies the location of the alarm sound in the audio. As a method for analyzing the audio data 300, a mechanical audio component analysis may be performed or AI may be used. In addition, the location of the alarm sound is not simply a location where only the alarm sound is recorded in the audio data 300, but may be a location where the alarm sound is superimposed on other sounds. At this time, the deletion point specifying means 140, for example, divides the audio data 300 into windows of several milliseconds to several hundred milliseconds and performs FFT (Fast Fourier Transform) to search for the position of the alarm sound pattern. Specifically, the deletion location specifying means 140 can specify the location of the alarm sound in the audio using, for example, a statistical model such as HMM, AI such as RNN or LSTM, or the like. This alarm sound may be specified for the entire audio data, at specific intervals, or limited to a portion of the original video data 220 that corresponds to an L-shape or the like.

Next, in step S112, the original audio specifying means 150 determines whether there is a deleted portion.
If Yes, the process advances to step S113.
In the case of No, the high-quality sound processing of the retransmission processing ends.

If there is a deleted portion, the original audio specifying means 150 performs original audio specifying processing in step S113.
As with video, programs broadcast using a video server system are highly likely to use audio stored within the video server system. Therefore, during audio analysis, it is possible to search the original audio data 320 stored within the video server system.

According to FIG. 4(b), the original audio identifying means 150 identifies, for example, original audio data 320 corresponding to the original video data 220. Through this search, it is possible to specify the original audio used in the broadcast.

Next, in step S114, the high-quality sound processing means 160 determines whether or not the original audio data 320 has been identified.
If Yes, the high-quality sound processing means 160 advances the process to step S115.
In the case of No, the high-quality sound processing means 160 advances the process to step S116.

If the superimposition of the alarm sound is detected and the original audio data 320 can be identified, the high-quality sound processing unit 160 performs copy high-quality processing in step S115.
Based on the original audio data 320 specified by the original audio specifying means 150, the high-quality sound processing means 160 enhances the sound quality of the deletion portion of the audio specified by the deletion portion specifying means 140. For example, the high-quality sound processing unit 160 replaces the range in which the alarm sound is included in the audio data 300 with the corresponding range in the original audio data 320.

According to FIGS. 4(b) and 4(c), the high-quality sound processing means 160 specifies the deletion of the audio of the audio data 300, replaces the deleted portion with the corresponding portion of the original audio data 320, and Set and execute editing contents such as erasing the alarm sound. This processing can also be executed by a dedicated processor such as a DSP included in the control unit 10. Furthermore, at this time, the high-quality sound processing means 160 may adjust the output level of the audio using an effect such as a compressor.
Thereafter, the high-quality sound processing unit 160 ends the high-quality sound processing of the retransmission process.

If the superimposition of the alarm sound is detected but the original audio data 320 cannot be identified, the high-quality sound processing unit 160 performs inverted high-quality sound processing in step S116.

According to FIG. 4(d), the high-quality sound processing unit 160 deletes the alarm sound by synthesizing the waveform data of the opposite phase obtained by inverting the phase of the alarm sound with the audio data 300 at an appropriate output level. Alternatively, the high-quality sound processing means 160 can also delete the alarm sound by performing special filter processing such as removing frequency components of the alarm sound. Alternatively, the high-quality sound processing unit 160 can also delete the alarm sound by using AI that has been trained to mute the alarm sound. Furthermore, after deletion, the high-quality sound processing means 160 may adjust the output level of the audio.

After these processes are completed, the processed audio data 300 is associated with the processed video data 210 and retransmitted by the transmission equipment. Here, if there is no deleted portion, the unprocessed audio data 300 is retransmitted. Note that the broadcast video data 200 may be retransmitted in association with the processed or unprocessed audio data 300.
With the above steps, the high quality sound processing of the retransmission processing is completed.

By configuring as described above, the following effects can be obtained.
According to FIG. 5, conventionally, when retransmitting for rebroadcasting or redistribution based on broadcast video recorded simultaneously, video processing was performed to delete unnecessary elements such as L characters. Such video deletion processing requires manual editing each time, which creates a workload for the operator and also lacks the speed of redistribution. In addition, since editing is done manually, if the area to be deleted is incorrectly set, if more than necessary area is deleted, the angle of view may become unnatural, or conversely, if the deletion area is narrow, the L-shaped part may be deleted. There was a possibility that the background color of the text would bleed through and remain, making the video unsuitable for broadcast. In addition, compression noise and the like may occur due to re-encoding in the area reduced by the L-shape. Furthermore, when the area reduced by this L-shape is enlarged again, the resolution of the video may be impaired (blurring may occur) when compared with the original broadcast video. On the other hand, areas where the time and other information have been masked (blurred) create boundaries with surrounding images, resulting in extremely unnatural images. In the first place, it is assumed that many of the broadcast simulcast videos are recompressed in order to preserve the broadcast video, so the image quality was inferior compared to the original.
These may create a sense of discomfort in the video, resulting in a video that is not suitable for broadcasting.

On the other hand, the editing system X according to the embodiment of the present invention is an editing system that retransmits the broadcast video included in the broadcast video data 200. An unnecessary area identifying means 100 identifies unnecessary areas that are continuously displayed, and processed video data 210 is created in which the unnecessary area identified by the unnecessary area identifying means 100 is deleted from the broadcast video data 200 or processed to make it less noticeable. an unnecessary area processing means 110 for processing, an original video specifying means 120 for specifying stored original video data 220 from the processed video data 210 and/or broadcast video data 200 processed by the unnecessary area processing means, and an original video specifying means. The video processing apparatus is characterized by comprising an image quality enhancing means 130 for enhancing the image quality of the processed image based on the original image data 220 specified by 120.

With this configuration, when rebroadcasting broadcast video, unnecessary parts of the previously broadcast video are deleted. That is, the content of the video is analyzed, the position of unnecessary areas such as at L time is specified, and the identified unnecessary areas are automatically masked to make them less noticeable. Then, the video content is analyzed and compared with the original video data 220 to identify the original video used for broadcasting. Then, based on the original video data 220 stored in the storage server 2, a video close to the original broadcast video is restored.
In this way, when the original video data 220 used for broadcasting can be identified, by referring to the original video data 220, image quality can be improved with lower load and higher accuracy than in the past. This makes it possible to suppress image quality deterioration during retransmission of broadcast video and improve image quality. Furthermore, automatic editing can save labor in the workflow from simultaneous recording of broadcast video to retransmission. In addition, as an editing system that can automatically edit, it can reduce the workload of the operator and improve costs.

The editing system X according to the embodiment of the present invention is characterized in that the unnecessary area identifying means 100 identifies unnecessary areas using a model that has learned the characteristics of the area to be deleted.
With this configuration, it is possible to reliably identify unnecessary areas. In other words, the video server system operates for each broadcast station, and the shape of characters such as L letters of the broadcast recorded video, the shape of the time display, the display position of subtitles, the processing of fonts, etc., and the insertion format are There is some regularity. This kind of L-shape, etc. that shows a specific pattern can be trained as a model of the target area to be deleted, detect specific components contained in the L-shape, etc., and delete unnecessary areas with high precision. It becomes possible to specify the area. This makes it possible to reliably achieve high image quality through automatic editing.

In order to improve the broadcast quality of broadcast-recorded video, it is also possible to improve the image quality by applying super-resolution technology or the like. Here, especially in the blurred area, the resolution is intentionally processed to be extremely low. Furthermore, even if the blur range is precisely matched to the shape of the time, subtitles, etc., it is difficult to restore pixel information of the original video that is lost due to overwriting of the time, subtitles, etc. Even if advanced image prediction using AI or the like is performed for these images, it is extremely difficult to obtain the originally existing pixel information and resolution.

On the other hand, in the editing system The feature is that the image quality is improved by performing edge enhancement or synthesis using the original video data 220, and/or synthesis by cutting out the original video included in the original video data 220.
With this configuration, by performing edge enhancement and compositing using edge information and color information based on the original video data 220, and cutting out the original video included in the original video data 220, the pixel information that originally existed in the broadcast video It is possible to reproduce the sense of resolution. In other words, it is possible to restore a video that is close to the original video. Furthermore, by emphasizing and compositing edges and colors using the original video data 220, there is a possibility that the image quality can be made higher than that at the time of broadcast.

In the editing system X according to the embodiment of the present invention, the original video specifying means 120 extracts common points between the processed video included in the processed video data 210 and the original video included in the original video data 220, The feature is that the original video included in the original video data 220 is specified based on the extracted common points.
With this configuration, the common points in the images of the processed video data 210 and the original video data 220 are extracted in advance, the learning is performed based on the extracted common points, the comparison is made, and the original video data 220 is converted from the processed video data 210. It is possible to identify In this way, by analyzing the video content, comparing it with the stored video, and identifying the original video used for broadcasting, the search for the original video data 220 can be speeded up, and the accuracy of image quality restoration can be improved. can be done.

In the past, the alarm sound could not be completely deleted by manual editing such as reverse phase synthesis or filter processing, and components of the alarm sound remained as noise.
On the other hand, the editing system The original audio identifying means 150 identifies the corresponding original audio data 320, and based on the original audio data 320 identified by the original audio identifying means 150, the deletion location of the audio identified by the deletion location identifying means 140 is improved in sound quality. The present invention is characterized in that it further includes a high-quality sound processing means 160.

With this configuration, the audio content is analyzed and compared with the stored audio data 300 to identify the original audio data 320 that corresponds to the original video data 220 used for broadcasting. As a result, if the original audio data 320 used in the broadcast can be identified, by referring to this, the alarm sound added during the broadcast can be deleted with higher precision than usual. Thereby, noise originating from the alarm sound can be alleviated, and high quality sound can be reliably achieved.

The editing system X according to the embodiment of the present invention is characterized in that the deletion location specifying means 140 performs audio analysis using a specific model and identifies the location of the alarm sound in the audio.
With this configuration, the audio content is compared with the stored audio using a specific model that includes AI, and the audio of the original video used for broadcasting is identified, improving the accuracy of alarm sound removal. It becomes possible to do so.

In the above-described embodiment, an example has been described in which the broadcast video data 200 already stored in the storage server 2 is subjected to image quality enhancement processing, and the audio data 300 is subjected to audio quality enhancement processing.
However, it is also possible to perform processing in real time during or without recording. Furthermore, the high-quality sound processing for deleting the alarm sound can also be performed in real time during recording or without recording.

In the above-described embodiment, the broadcast video data 200 is analyzed, and the presence or absence of an unnecessary area is searched for and then identified.
However, the operator may specify that the broadcast video data 200 includes an L-shape or the like, or specify the position of the L-shape or the like on the video, the display start time, the display end time, and the like. With this configuration, analysis of the broadcast video data 200 can be omitted.

In the embodiment described above, the processed video data 210 is created from the broadcast video data 200, and the processed video data 210 is replaced with the original video data 220, edges and colors are emphasized, synthesized, etc.
However, it is also possible to directly process the broadcast video data 200 without creating the processed video data 210 for the broadcast video data 200. Alternatively, for example, it is also possible to create a copy of the broadcast video data 200 as the processed video data 210 without performing mask processing. In this case, it is possible to directly replace a frame with an unnecessary area with a frame of the original video data 220, or to replace the display position of the time, subtitles, etc. with the original video data 220 without performing mask processing. .
With this configuration, it is possible to reduce the effort required for mask processing and quickly improve image quality.

In the above-described embodiment, an example has been described in which the program data used in the broadcast video data 200 is used as the original video data 220.
However, even if it is difficult to identify the video itself used for broadcasting, it is also possible to use material data of a similar video showing a similar point as the original video data 220, for example. With this configuration, it is possible to achieve higher image quality with higher precision than in the past.
Note that this similar video can also be searched using the above-mentioned AI. Furthermore, using AI such as GAN, it is also possible to process this similar video into a video that is close to the video used in the actual broadcast.

On the other hand, in the above-described embodiment, it is described that anti-phase synthesis and filter processing of the alarm sound are performed only when the original audio data 320 cannot be specified.
However, the processed audio data 310 may be replaced with the original audio data 320 after performing anti-phase synthesis or filter processing.
With this configuration, there is no need to separate processing depending on whether or not the original audio data 320 can be specified, and the efficiency of improving sound quality can be improved.

In addition, in the embodiments described above, the audio data 300 is directly subjected to anti-phase synthesis, filter processing, or replaced with the original audio data 320.
However, it is also possible to configure the processed audio data 310 to be processed and output together with the processed video data 210 without processing the audio data 300.

In the embodiment described above, an example has been described in which the dedicated analysis device 1 executes the processing of each functional unit on the broadcast video data 200 stored in the storage server 2.
However, the processing of each of the above-mentioned functional units does not have to be performed by the analysis device 1, and may be performed by the editing device 4, the storage server 2, or the like.

In the embodiment described above, an example is described in which MXF is used as the container format file.
However, it is also possible to use container formats other than MXF, such as MKV. Furthermore, the recording format and recording format of the broadcast video data 200 may be MP4, AVI, other program stream (PS) formats, other transport stream formats (TS), etc., depending on system requirements. Furthermore, the broadcast video data 200 may be compressed using various codecs.

In addition, when replacing the original video data 220, the high image quality improvement means 130 and the high sound quality processing means 160 gradually change the original video data 220 and the original data by using a dissolve effect for video, a cross fade effect for audio, etc. It may be replaced with the audio data 320.
With this configuration, it is possible to alleviate the discomfort caused by discontinuity.

Further, the editing system according to the embodiment of the present invention can be applied to various devices that use video data. For example, the invention can be applied to encoders, decoders, editing machines, material servers, transmission servers, etc. as devices that use video data.

Note that the configuration and operation of the embodiment described above are merely examples, and it goes without saying that the configuration and operation of the embodiment can be modified and executed as appropriate without departing from the spirit of the present invention.

1 Analyzing device 2 Storage server 3 Recording device 4 Editing device 5 Network 10 Control section 11 Storage section 30 Imaging section 100 Unnecessary area specifying means 110 Unnecessary area processing means 120 Original video specifying means 130 High image quality means 140 Deletion part specifying means 150 Yuan Audio identification means 160 High quality sound processing means 200 Broadcast video data 210 Processed video data 220 Original video data 300 Audio data 310 Processed audio data 320 Original audio data A1, A2, A3 Unnecessary area X Editing system

Claims (6)

An editing system that retransmits broadcast video,
unnecessary area identifying means for identifying an unnecessary area that is continuously displayed at a specific location of the broadcast video;
unnecessary area processing means for creating a processed video in which the unnecessary area identified by the unnecessary area identification means is processed to delete or make it less noticeable from the broadcast video;
Original video identifying means for identifying a stored original video from the processed video and/or the broadcast video processed by the unnecessary area processing means;
An editing system comprising: an image quality improvement unit that increases the image quality of the processed video based on the original video specified by the original video identification unit. The unnecessary area identifying means includes:
The editing system according to claim 1, wherein the unnecessary area is identified by a model that has learned the characteristics of the area to be deleted. The image quality improvement means includes:
A claim characterized in that the image quality of the processed video is enhanced by performing edge enhancement or composition using edge information and/or color information based on the original video, and/or composition by cutting out the original video. The editing system according to item 1 or 2. The source video identifying means includes:
4. The method according to claim 1, wherein common points between the processed video and the original video are extracted, and the original video is identified based on the extracted common points. Editing system. Deletion location identifying means for analyzing audio corresponding to the broadcast video to identify deletion locations;
Original audio identifying means for identifying the original audio corresponding to the original video;
The method further comprises a high-quality sound processing means for enhancing the sound quality of the deletion portion of the voice specified by the deletion portion specifying means based on the original voice specified by the original voice specifying means. An editing system according to any one of claims 1 to 4. The deletion point specifying means includes:
6. The editing system according to claim 5, wherein a voice analysis is performed using a specific model to identify a location of an alarm sound in the voice.

Images