CN109889749B

CN109889749B - 3D high-definition operation recording and broadcasting method

Info

Publication number: CN109889749B
Application number: CN201910257890.7A
Authority: CN
Inventors: 许杰
Original assignee: Shanghai Qpeit Electronic Technology Co ltd
Current assignee: Shanghai Qpeit Electronic Technology Co ltd
Priority date: 2019-04-01
Filing date: 2019-04-01
Publication date: 2021-09-21
Anticipated expiration: 2039-04-01
Also published as: CN109889749A

Abstract

The invention discloses a 3D high-definition operation recording and broadcasting method, which comprises the following steps: s1, software and hardware, including a collection card with a GPU, a video card supporting CUDA, the existing operation recording and broadcasting equipment and corresponding software; s2, collecting audio and video signals by an acquisition card, and calling a GPU (graphics processing Unit) to process an audio and video recording function by an independent audio and video processing engine; processing the delay by a processing algorithm to reduce the delay, so that the delay of the 3D picture is less than 40ms, and the delay of the simulated 3D picture is less than 60 ms; s3, the GPU accelerates the 3D picture by adopting CUDA and processes the picture through the GPU or assists the CPU to process the picture through the GPU; and S4, adopting CUDA to assist the CPU to accelerate the processing through GPU in the post-processing of the recorded video. The invention adopts a single video processing engine, so that the 3D picture delay is less than 40ms, and the simulated 3D picture is less than 60 ms. According to the invention, the 3D picture is processed by the Nvidia GPU in an accelerating way by adopting the CUDA technology, and the picture is clear and smooth. The invention can greatly reduce the video processing time.

Description

3D high-definition operation recording and broadcasting method

Technical Field

The invention relates to a surgical recording and broadcasting technology, in particular to a 3D high-definition surgical recording and broadcasting method.

Background

The traditional medical operation observation and study is mainly used for organizing personnel to go to an operating room for study on site, or for recording the operation condition for teaching. The mode has the following defects: firstly, due to the space limitation of an operating room, the number of the visitors and visitors can be less, and the teaching and research efficiency is lower; secondly, the mess of personnel or air pollution in the operating room even affects the attention of operating doctors; thirdly, for the operation records of major operations and special operations, because the field video recorders do not know the focus of the researchers or students, the field of vision and the field of view breadth depth of some parts are not enough; in addition, because the camera is poor in shielding, when the ray equipment, the electric saw and other equipment are started in an operation, serious interference is caused, and the recording effect is poor or recording cannot be carried out.

And all audio and video synchronous live broadcast can be completed by arranging an Anlan operation live broadcast device in an operating room and a meeting place through operation recording and broadcast. However, the current surgical recording and broadcasting method has the following defects:

1. the picture has a longer delay. Most of the existing recording and broadcasting methods are delayed by more than 300 ms. The preview screen during the operation has a very high real-time requirement, and causes adverse consequences if the user does not pay attention to the preview screen.

2. The occupied system resources are high. The existing recording and broadcasting scheme occupies high computer system resources, the CPU occupancy rate is over 90 percent at most when a low workstation is configured and recording is performed, and computer crash can be caused sometimes. The invention adopts an independent processing engine, records the video by the acquisition card GPU, and does not occupy CPU resources.

3. The image and voice are less clear and smooth. The video of the existing recording and broadcasting scheme generally reaches 2Mbps, but images are sometimes jammed, the sound noise is large, and the highest resolution only supports 1920x 1080.

4. The post-processing time of the recorded video is too long. The existing recording and broadcasting scheme generally adopts CPU processing for the video processing in the later period, and the occupied time is too long.

Disclosure of Invention

In view of the above defects in the prior art, the technical problem to be solved by the present invention is to provide a 3D high definition surgical recording and broadcasting method, which can overcome the defects in the prior art that the picture has a long time delay, the occupied system resource is high, the image and voice are not clear and smooth, and the like.

In order to achieve the purpose, the invention provides a 3D high-definition surgical recording and broadcasting method, which comprises the following steps:

s1, software and hardware, including an acquisition card with Nvidia GPU, a video card supporting CUDA (computer Unified Device architecture), the existing operation recording and broadcasting equipment and corresponding software (the software includes a 3D high-definition operation recording and broadcasting system and supporting software thereof, the hardware includes a customized QP1200 acquisition card and a Nvidia GeForce GTX TITAN video card supporting CUDA); the method can be understood as replacing the acquisition card of the existing operation recording and broadcasting equipment with the acquisition card with the GPU and replacing the display card of the existing operation recording and broadcasting equipment with the display card supporting the CUDA.

S2, collecting the audio and video signals by the collecting card, calling the GPU of the collecting card by an independent audio and video processing engine (sound and video processing software) to process the audio and video recording function, thereby occupying no CPU resource and preventing the computer from crashing.

The delay is processed by a processing algorithm, and the delay is reduced by adopting an h.264 mvc algorithm, so that the 3D picture delay is less than 40ms, and the simulated 3D picture is less than 60 ms.

And S3, the 3D picture is accelerated by the GPU by adopting CUDA, and the video processing engine automatically adjusts the hue/brightness/contrast/saturation.

And S4, performing CUDA (compute unified device architecture) acceleration processing on the recorded video in the later period, so that the GPU assists the CPU processing, and the time is reduced.

The invention has the beneficial effects that:

1. the invention adopts a single video processing engine, so that the 3D picture delay is less than 40ms, and the simulated 3D picture is less than 60 ms.

2. According to the invention, the 3D picture is processed by the Nvidia GPU in an accelerating way by adopting the CUDA technology, and the picture is clear and smooth.

3. The code stream adopted by the invention can reach 50Mbps at most, and the general video code stream is set to be 12 Mbps. The method supports 4K ultra-high definition pictures, the highest resolution can reach 4096x2160, and the frame rate supports 60 fps. The video processing engine can automatically adjust the hue/brightness/contrast/saturation, so that the picture is clear and smooth.

4. The invention adopts CUDA accelerated processing to the video post-processing, so that the Nvidia GPU assists the CPU processing, and the time reduction can reach more than two thirds (the buffer dsp is collected originally, the delay is generally more than 120ms, and the delay time is only 40ms after the method of the invention is adopted, and is reduced by two thirds).

Drawings

Fig. 1 is a schematic flow chart of the process of collecting audio and video signals according to the present invention.

FIG. 2 is a process flow diagram of a 2D to 3D processing engine.

Detailed Description

The invention is further illustrated by the following examples in conjunction with the accompanying drawings:

referring to fig. 1-2, the present invention is composed of a software part and a hardware part. The software comprises a 3D high-definition surgery recording and broadcasting system and supporting software thereof. The hardware comprises a customized QP1200 acquisition card and an Nvidia GeForce GTX TITAN display card supporting CUDA.

And (3) operating environment:

CPU：Intel Core I7-7700

memory: more than 16G

Operating the system: windows 7 or more

Fig. 1 is a flow chart of the processing of acquiring audio and video signals according to the present invention, and the detailed process is as follows:

step 201, starting the whole operation recording and broadcasting system;

202, initializing acquisition equipment, wherein the acquisition equipment comprises an acquisition card, a GPU and the like;

step 203, collecting sound and video signals (in the case, referred to as audio and video) by a collecting card;

step 204, processing the collected images according to set parameters by an audio and video processing engine (software and hardware for processing audio and video), so as to automatically adjust hue/brightness/contrast/saturation, and process resolution, frame rate, de-interlacing mode and color space conversion;

the audio and video processing engine marks the time for audio and video according to the time for the audio and video to reach the acquisition card, and the acquisition card can stamp the time twice: the first time is the time when the audio and video enters the acquisition card, and the second time is the time when the audio and video completely enters the cache.

A program (the existing operation recording and broadcasting software) applied by a user processes audio and video according to the time stamp to ensure the synchronization of the audio and video;

meanwhile, by using a low-delay algorithm (in the embodiment, an h.264 mvc algorithm), the audio and video adopts ultra-low transmission delay, so that the 3D picture delay is ensured to be less than 40ms, and the simulated 3D picture is ensured to be less than 60 ms.

Step 205, judging whether a video needs to be recorded or not, which is mainly determined by an instruction received by the operation recording and broadcasting software;

step 206, if the video needs to be recorded, the operation recording and broadcasting system calls a GPU (graphics processing unit) acquisition card to process the recording function, so that CPU (central processing unit) resources are not occupied, and the blocking is prevented;

if the video recording is not required, then go to one or a combination of step 207, step 208, step 209;

step 207, if only the common video needs to be displayed, transmitting the common one-way 2D video signal to step 212, directly outputting the signal to a display preview screen through a display card, and entering a preview step 216.

And 208, if the 3D video needs to be synthesized, transmitting the two paths of audio and video to step 213, calling the Nvidia GPU by the CUDA through the step 213 to accelerate and synthesize the 3D video, transmitting the 3D video to a display preview picture, and entering a preview step 216.

Step 209, if the analog 3D program needs to be called, dividing the single-channel 2D video into two channels, namely a left channel video image 302 and a right channel video image 303, respectively, where the left channel video image 302 is an original unprocessed common video signal, and the right channel video image 303 is a video signal to be subjected to depth processing; then, the step 214 is carried out, and 2D to 3D processing is carried out through a 2D to 3D processing engine (2D to 3D software); the 2D to 3D processing engine uses CUDA technology to synthesize 3D in the graphics card with the Nvidia GPU assisting the CPU, and then transmits the 3D to the display for previewing, and then enters the preview step 216.

Step 210, after recording the video by the acquisition card GPU in step 206, judging whether the video after the path needs post-processing, if so, skipping to step 211, and accelerating the processing in the GPU by using a CUDA (compute unified device architecture) technology in step 211, so that most time can be saved, skipping to step 215 after the processing, and storing the processed video file; if no post-processing is required, the process jumps to step 219 and ends.

Step 217, judging whether OSD (On-Screen Display) is needed, if so, skipping to step 218, and processing the OSD in step 218 by adopting a GPU of the acquisition card; if not, the process jumps to step 219 and ends.

FIG. 2 is a process flow diagram of a 2D to 3D processing engine, the detailed process being as follows:

step 301, dividing the single-channel 2D video in step 209 into two channels, namely a left channel video image 302 and a right channel video image 303, wherein the left channel video image 302 is an original unprocessed common video signal, and the right channel video image 303 is a video signal to be subjected to depth processing;

the left video image 302 is directly transmitted to step 306 for processing; the right video image 303 is a video signal that needs to be processed.

Step 303, the right path video image 303 is transmitted to step 304, and step 304 extracts the depth information of the right path video image 303, and forms a depth map according to algorithm processing (h.264 mvc algorithm) through step 305.

Step 305, transmitting the processed video signal to step 306, so that the processed video signal and the left video signal 302 are subjected to image synchronization matching;

and 307, synthesizing the matched video images into 3D video images in the display card by using the CUDA technology and using the Nvidia GPU to assist the CPU, and then outputting the 3D video images.

The invention is not described in detail, but is well known to those skilled in the art.

The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.

Claims

1. A3D high-definition surgical recording and broadcasting method is characterized by comprising the following steps:

s1, software and hardware, including a collection card with a GPU, a video card supporting CUDA, the existing operation recording and broadcasting equipment and corresponding software;

s2, collecting audio and video signals by an acquisition card, and calling a GPU (graphics processing Unit) to process an audio and video recording function by an independent audio and video processing engine;

processing the delay by a processing algorithm to reduce the delay, so that the delay of the 3D picture is less than 40ms, and the delay of the simulated 3D picture is less than 60 ms;

s3, the GPU accelerates the 3D picture by adopting CUDA and processes the picture through the GPU or assists the CPU to process the picture through the GPU;

s4, adopting CUDA to assist CPU to accelerate processing through GPU in the post-processing of the recorded video;

step 201, starting the whole operation recording and broadcasting system;

202, initializing acquisition equipment;

step 203, collecting a sound signal and a video signal by a collecting card;

step 204, processing the collected images according to set parameters by the audio and video processing engine according to the sound and video signals;

the audio and video processing engine marks the time for audio and video according to the time for the audio and video to reach the acquisition card, and the acquisition card can stamp the time twice: the first time is the time when the audio and video enters the acquisition card, the second time is the time when the audio and video completely enters the cache, and the application program processes the audio and video according to the timestamp to ensure the synchronization of the audio and video;

step 205, judging whether a video needs to be recorded;

step 206, if the video needs to be recorded, the operation recording and broadcasting system calls a GPU processing and recording function of the acquisition card; if the video recording is not required, then go to one or a combination of step 207, step 208, step 209;

step 207, if only a common video needs to be displayed, transmitting a common one-way 2D video signal acquired by an acquisition card to step 212, directly outputting the common one-way 2D video signal to a display to preview a picture in step 212 through a display card, and entering a previewing step 216;

208, if the 3D video needs to be synthesized, transmitting the two paths of audio and video collected by the acquisition card to step 213, calling a GPU by a CUDA in step 213 to accelerate the synthesis of the 3D video, transmitting the 3D video to a display preview picture, and entering a preview step 216;

step 209, if the analog 3D program needs to be called, dividing the single-channel 2D video into two channels, namely a left channel video image and a right channel video image, wherein the left channel video image is an original unprocessed common video signal, and the right channel video image is to be subjected to depth processing video signals;

then, the process proceeds to step 214, and step 214 performs 2D to 3D conversion processing through the 2D to 3D processing engine; the 2D-to-3D processing engine utilizes the CUDA to drive the GPU to assist the CPU to synthesize 3D in the display card, then transmits the 3D to a display preview picture, and enters a preview step 216;

step 210, after recording the video through the acquisition card GPU in step 206, judging whether the video after the route needs post-processing, if so, jumping to step 211, accelerating the processing in the GPU by using CUDA in step 211, jumping to step 215 after the processing, and saving the processed video file in step 215; if no post-processing is required, the process jumps to step 219 and ends.

2. The 3D high definition surgical recording and broadcasting method according to claim 1, further comprising step 217, step 217 determining whether OSD processing is required, if so, skipping to step 218, step 218 processing OSD using GPU of acquisition card; if not, the process jumps to step 219 and ends.

3. The 3D high definition surgical recording and broadcasting method according to claim 1, wherein in step 214, the process of converting 2D to 3D is as follows:

step 301, directly transmitting the left video image in step 209 to step 306 for processing;

step 303, transmitting the right path of video image to step 304, extracting depth information of the right path of video image in step 304, and forming a depth map according to algorithm processing in step 305;

step 305, transmitting the processed video signal to step 306, so that the processed video signal and the left video signal are subjected to image synchronization matching;

and 307, driving the matched video images to a GPU by utilizing the CUDA to assist the CPU to synthesize the 3D video images in the display card, and then outputting the 3D video images.