CN111757087A - VR video processing method and device and electronic equipment - Google Patents
VR video processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN111757087A CN111757087A CN202010625574.3A CN202010625574A CN111757087A CN 111757087 A CN111757087 A CN 111757087A CN 202010625574 A CN202010625574 A CN 202010625574A CN 111757087 A CN111757087 A CN 111757087A
- Authority
- CN
- China
- Prior art keywords
- video
- processing
- resolution
- super
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000012545 processing Methods 0.000 claims abstract description 138
- 238000000034 method Methods 0.000 claims abstract description 24
- 238000012549 training Methods 0.000 claims description 23
- 238000003780 insertion Methods 0.000 claims description 11
- 230000037431 insertion Effects 0.000 claims description 11
- 238000013528 artificial neural network Methods 0.000 claims description 10
- 230000015556 catabolic process Effects 0.000 claims description 9
- 238000006731 degradation reaction Methods 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 8
- 238000005457 optimization Methods 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 5
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000007499 fusion processing Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005563 spheronization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/139—Format conversion, e.g. of frame-rate or size
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Studio Devices (AREA)
Abstract
The invention provides a processing method and a device of a VR video and electronic equipment, and relates to the technical field of image processing, wherein the method comprises the steps of obtaining the VR video to be processed; and performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video super-resolution model and a video frame interpolation model to obtain a processed VR video. According to the embodiment of the invention, the video super-resolution model and the video frame interpolation model are trained in advance, the VR video is subjected to super-resolution processing through the video super-resolution model so as to improve the video resolution, and the frame interpolation video processing is performed on the VR video through the video frame interpolation model so as to improve the video frame rate, so that the VR video with better definition and video image quality is obtained, and the watching experience of a user is improved.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a VR video processing method and apparatus, and an electronic device.
Background
With the development of VR (Virtual Reality) technology, users have higher and higher requirements for definition of video image quality provided by VR devices, and meanwhile, video resolution and frame rate that can be supported by VR related hardware products are higher and higher. For VR industry ecology, one important factor affecting VR development is the number of relevant VR videos and video quality. VR cameras are generally expensive, have high requirements for the shooting environment, are limited by the camera and the shooting conditions, and have a small number of currently shot videos that can be matched with high resolution and high frame rate VR hardware, whereas if a VR display device that is currently better is directly used to play a VR video shot by a camera with poor imaging performance, the VR display device has a disadvantage of poor subjective quality, which seriously affects the subjective experience of the user.
At present, no method is available for improving the quality of a VR video on the premise of not increasing the cost of VR shooting equipment.
Disclosure of Invention
In view of the above, the present invention provides a VR video processing method, an apparatus and an electronic device, which can improve the quality of a VR video without increasing the cost of a VR shooting device.
In a first aspect, an embodiment of the present invention provides a VR video processing method, including: acquiring a VR video to be processed; and performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video super-resolution model and a video frame interpolation model to obtain a processed VR video.
In a preferred embodiment of the present invention, the step of performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video hyper-resolution model and a pre-trained video frame interpolation model to obtain a processed VR video includes: performing super-resolution processing on the VR video through a pre-trained video super-resolution model to obtain a VR video with optimized resolution; performing frame interpolation video processing on the VR video with the optimized resolution ratio through a pre-trained video frame interpolation model to obtain a processed VR video; or, performing frame interpolation video processing on the VR video through a pre-trained video frame interpolation model to obtain a VR video with an optimized frame rate; and performing super-resolution processing on the VR video after the frame rate optimization through a pre-trained video super-resolution model to obtain the processed VR video.
In a preferred embodiment of the present invention, the video hyper-resolution model is obtained by training in the following manner: acquiring preset VR video original data; performing video quality degradation processing on the VR video original data to obtain VR video data with reduced definition; and training a preset neural network by taking the VR video data with reduced definition and the VR video original data as training sets to obtain a video hyper-resolution model.
In a preferred embodiment of the present invention, the step of performing video quality degradation processing on the VR video original data to obtain the VR video data with reduced definition includes: and carrying out video coding processing and/or video distortion deformation processing on the VR video original data to obtain VR video data with reduced definition.
In a preferred embodiment of the present invention, the step of obtaining the original data of the VR video includes: acquiring a plurality of initial videos shot by preset VR camera equipment at different visual angles; performing projection correction processing on the plurality of initial videos to obtain corrected videos; and carrying out image splicing processing on the corrected video to obtain a VR video to be processed with continuous pictures.
In a preferred embodiment of the present invention, the step of performing image stitching processing on the corrected video to obtain a VR video to be processed with continuous frames includes: extracting a feature vector of each video frame of the corrected video; performing feature matching and image transformation on the video frame according to the feature vector to obtain a video frame after image configuration; and carrying out image fusion processing on the video frame after image configuration to obtain the VR video to be processed with continuous pictures.
In a preferred embodiment of the present invention, the step of performing projection correction processing on the plurality of initial videos to obtain corrected videos includes: projecting the plurality of initial videos to a unified coordinate system through cylindrical projection or spherical projection; and correcting the rotation angle of the image in the initial video after projection to obtain a corrected video.
In a second aspect, an embodiment of the present invention further provides a processing apparatus for VR video, including: the VR video acquisition module is used for acquiring VR videos to be processed; and the super-resolution and frame insertion processing module is used for performing super-resolution processing and frame insertion video processing on the VR video through a pre-trained video super-resolution model and a pre-trained video frame insertion model to obtain the processed VR video.
In a preferred embodiment of the present invention, the super-divide and frame-insert processing module is further configured to: performing super-resolution processing on the VR video through a pre-trained video super-resolution model to obtain a VR video with optimized resolution; performing frame interpolation video processing on the VR video with the optimized resolution ratio through a pre-trained video frame interpolation model to obtain a processed VR video; or, performing frame interpolation video processing on the VR video through a pre-trained video frame interpolation model to obtain a VR video with an optimized frame rate; and performing super-resolution processing on the VR video after the frame rate optimization through a pre-trained video super-resolution model to obtain the processed VR video.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores computer-executable instructions capable of being executed by the processor, and the processor executes the computer-executable instructions to implement the VR video processing method.
In a third aspect, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the VR video processing method described above.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a VR video processing method, a VR video processing device and electronic equipment, wherein a VR video to be processed is obtained; and then performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video hyper-resolution model and a video frame interpolation model to obtain a processed VR video. In the mode, the video super-resolution model and the video frame insertion model are trained in advance, super-resolution processing is carried out on the VR video through the video super-resolution model, the video resolution is improved, frame insertion video processing is carried out on the VR video through the video frame insertion model, the video frame rate is improved, the VR video with better definition and video quality is obtained, and watching experience of a user is improved.
Additional features and advantages of the disclosure will be set forth in the description which follows, or in part may be learned by the practice of the above-described techniques of the disclosure, or may be learned by practice of the disclosure.
In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flowchart of a VR video processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a training process of a video hyper-resolution model according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a VR video processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Icon: 31-a to-be-processed VR video acquisition module; 32-a super-divide and frame-insert processing module; 41-a processor; 42-a memory; 43-bus; 44-communication interface.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In view of the problem that the number of videos shot at present and capable of being matched with high-resolution and high-frame-rate VR hardware is small, and there is no method for improving the image quality of a VR video without increasing the cost of VR shooting equipment, embodiments of the present invention provide a processing method, an apparatus, and an electronic device for a VR video, where the technology can be applied to image quality enhancement processing of an old VR video, or real-time processing of a generated video by a VR camera, or other scenes requiring VR video processing.
To facilitate understanding of the embodiment, a detailed description is first given of a VR video processing method disclosed in the embodiment of the present invention.
Referring to fig. 1, a flow chart of a VR video processing method according to an embodiment of the present invention is shown, and as can be seen from fig. 1, the VR video processing method includes the following steps:
step S102: and acquiring the VR video to be processed.
Here, the VR video, that is, the virtual reality video, is generally photographed by a plurality of cameras separated from each other so that a difference between the left and right eyes can be captured, and then the photographed videos are pieced together so as to create an image closer to the real world. Wherein, the user can immerse in video content 360 degrees usually when watching the VR video, reaches the effect of being personally on the scene, compares with ordinary 2D video, and the VR video makes watching that the user can the multi-angle.
In this embodiment, the VR video to be processed may be an existing old VR video, for example, a VR video captured by a VR camera device configured lower before, so that the VR video to be processed may be reprocessed to improve image quality and enrich high-quality VR video resources; moreover, the VR video to be processed may also be a VR video shot in real time, so that the image quality of the shot video is processed in real time and the VR video with enhanced image quality is output.
In one possible implementation, the VR video to be processed may be obtained through the following steps 11-13:
(11) a plurality of initial videos obtained by shooting of preset VR camera equipment at different visual angles are obtained.
Here, VR image pickup apparatuses may be VR cameras, VR panoramic cameras, and the like, and in general, when shooting VR video, it is necessary to set a plurality of VR image pickup apparatuses at different visual angles for shooting, so as to obtain initial video at a plurality of angles.
(12) And carrying out projection correction processing on the plurality of initial videos to obtain corrected videos.
Wherein, the plurality of initial videos can be projected to a unified coordinate system through cylindrical projection or spherical projection; and then, correcting the rotation angle of the image in the initial video after projection to obtain a corrected video. Here, since images taken by different VR cameras are not on the same plane, consistency of the geometric spatial structure in the actual scene can be maintained by projection correction.
(13) And carrying out image splicing processing on the corrected video to obtain a VR video to be processed with continuous pictures.
In actual operation, image splicing processing needs to be performed on each frame of a plurality of VR videos after projection correction, specifically, for each video frame of the corrected video, a feature vector of the video frame is extracted first; then, performing feature matching and image transformation on the video frame according to the feature vector to obtain a video frame after image configuration; and further carrying out image fusion processing on the video frame after image configuration to obtain a VR video to be processed with continuous pictures.
Step S104: and performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video super-resolution model and a video frame interpolation model to obtain a processed VR video.
In practical implementation, the video hyper-resolution model and the video frame interpolation model can be obtained by training a neural network, wherein the video hyper-resolution model is used for performing super-resolution processing on a video so as to improve the resolution of the video; and the video frame interpolation model is used for performing frame interpolation video processing on the video so as to improve the frame rate of the video. Therefore, VR videos obtained through super-resolution processing and frame interpolation video processing have better resolution and higher frame rate, and therefore video quality and definition are enhanced.
In one possible implementation, the super-resolution processing and the frame-interpolation video processing of the VR video can be implemented by the following steps 21 to 22:
(21) and performing super-resolution processing on the VR video through a pre-trained video hyper-resolution model to obtain the VR video with optimized resolution.
Here, Super Resolution (SR) is a process of Super Resolution reconstruction, which generates a high Resolution image by processing a low Resolution image. The super-resolution processing of the video image is to improve the spatial resolution of the video image, for example, the resolution of one picture is expanded from 352x288 to 704x576, so as to facilitate the user to view the picture on a large-sized display device.
When the VR video is subjected to super-resolution processing, the VR video to be processed is input into a pre-trained video hyper-resolution model, and the VR video with optimized resolution is output.
(22) And performing frame interpolation video processing on the VR video with the optimized resolution ratio through a pre-trained video frame interpolation model to obtain a processed VR video.
The purpose of the frame interpolation video processing is to synthesize a new intermediate frame in the video so as to improve the frame rate of the video. Here, the VR video with the optimized resolution is input to a video interpolation model trained in advance, and the VR video with the improved frame rate is output.
In another possible implementation manner, frame interpolation video processing may be performed on the VR video through a pre-trained video frame interpolation model to obtain a VR video with an optimized frame rate; and then performing super-resolution processing on the VR video after the frame rate optimization through a pre-trained video super-resolution model to obtain the processed VR video. This method can also implement the step S104, and obtain a VR video with improved resolution and frame rate.
According to the processing method of the VR video, in actual operation, image details of the VR video can be recovered based on a super-resolution processing technology and an interpolation video processing technology of deep learning, for example, in some application scenes, if a VR video transmission source cannot provide a high-resolution video due to objective limitation (for example, the acquisition capacity of a camera is insufficient, the network bandwidth is insufficient or the processing capacity of a source end is insufficient, and the like), if the processing capacity of a cloud end or a receiving end meets requirements, the video quality can be recovered by means of the super-resolution processing and interpolation video processing technologies, so that the VR video with higher quality can be presented to a user.
The embodiment of the invention provides a VR video processing method, which comprises the steps of firstly obtaining a VR video to be processed; and then performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video hyper-resolution model and a video frame interpolation model to obtain a processed VR video. In the mode, super-resolution processing is carried out on the VR video through the video hyper-resolution model, the resolution ratio of the video can be improved, frame inserting video processing is carried out on the VR video through the video frame inserting model, the video frame rate can be improved, the definition of the VR video is improved, the video image quality is enhanced, and the watching experience of a user is improved.
On the basis of the VR video processing method shown in fig. 1, the embodiment of the present invention describes in detail a training process of a video hyper-resolution model in the method, and as shown in fig. 2, it is a schematic diagram of a training flow of the video hyper-resolution model provided by the embodiment of the present invention, where training of the video hyper-resolution model includes the following steps:
step S202: and acquiring preset VR video original data.
Here, in the present embodiment, reference may be made to corresponding description of obtaining the VR video to be processed in step S102 in the foregoing embodiment, and the VR video raw data may also be obtained according to the above-mentioned manners in steps 11 to 13, which is not described herein again.
Step S204: and performing video quality degradation processing on the VR video original data to obtain VR video data with reduced definition.
In at least one possible embodiment, the VR video raw data may be subjected to a video coding process and/or a video warping process, so as to obtain VR video data with reduced definition.
The video coding means that a file in an original video format is converted into a file in another video format by a compression technology, and the video is compressed by video coding processing, so that the redundancy of space and time dimensions can be removed.
In addition, the video warping processing refers to performing geometric warping on a video image to obtain a video image after various image distortions are performed, where the manner of warping may include: displacement, amplification, rotation, wave transformation, and spheronization, among others.
Therefore, through video coding and/or distortion deformation, the quality degradation of the VR video original data can be realized, and the VR video data with reduced definition can be obtained.
Step S206: and training a preset neural network by taking the VR video data with reduced definition and the VR video original data as training sets to obtain a video hyper-resolution model.
In this embodiment, the VR video data with reduced definition obtained in steps S202 and S204 and the corresponding VR video raw data form a video group matched in pairs, and are used as a training set for training a neural network, where the VR video data with reduced definition is used as an input of the neural network, and the VR video raw data is used as an output of the neural network, and network parameters of the neural network are trained, so as to obtain a trained video hyper-differentiation model.
In addition, in consideration of the high distortion characteristic of the VR video and the difficulty in acquiring paired VR high-definition video and low-definition video in a real scene to use as a training data set of a network model, in this embodiment, a VR video with relatively high definition is acquired and subjected to video quality degradation processing, so that a VR video with low definition and low resolution corresponding to the VR video is obtained, and a video pair is formed by the VR video to train a neural network model, so that the problem of lack of training data during training of a video super-resolution model is effectively alleviated.
Through the steps S202 to S206, a video super-resolution model can be obtained through training, so that super-resolution processing can be performed on the VR video; in actual operation, the Super-Resolution processing can be performed on the VR video through a widely applicable, Efficient and Accurate conventional AI Super-Resolution network such as a wide-application, Efficient and Accurate Image Super-Resolution (WDSR) and an Enhanced Super-Resolution generation countermeasure network (ESRGAN), and the Resolution of the VR video can be improved as well.
Similarly, a preset neural network can be trained by acquiring a training data set of the video frame insertion to obtain a video frame insertion model. And then, performing super-resolution processing on the VR video to be processed according to the video super-resolution model obtained through training so as to improve the video resolution, and performing frame interpolation video processing on the VR video through the video frame interpolation model so as to improve the video frame rate of the VR video, so that the VR video with improved definition and image quality is obtained.
Corresponding to the processing method of the VR video shown in fig. 1, an embodiment of the present invention further provides a processing apparatus of the VR video, referring to fig. 3, which is a schematic structural diagram of a processing apparatus of the VR video, and as can be seen from fig. 3, the processing apparatus includes a to-be-processed VR video acquisition module 31 and a super-frame and inter-frame processing module 32, which are connected to each other, where the functions of the two modules are as follows:
a to-be-processed VR video obtaining module 31, configured to obtain a to-be-processed VR video;
and the super-resolution and frame-interpolation processing module 32 is configured to perform super-resolution processing and frame-interpolation video processing on the VR video through a pre-trained video super-resolution model and a pre-trained video frame-interpolation model to obtain a processed VR video.
The processing apparatus for VR video provided in this embodiment first obtains VR video to be processed; and then performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video hyper-resolution model and a video frame interpolation model to obtain a processed VR video. In the device, through training video hyper-resolution model and video interpolation model in advance, and carry out super resolution processing to the VR video through video hyper-resolution model to promote its video resolution, and carry out interpolation video processing to the VR video through video interpolation model, with improve its video frame rate, thereby obtain the VR video that definition and video image quality are all more excellent, promote user's watching experience.
In one possible implementation, the super-divide and frame-interpolation processing module 32 is further configured to: performing super-resolution processing on the VR video through a pre-trained video super-resolution model to obtain a VR video with optimized resolution; performing frame interpolation video processing on the VR video with the optimized resolution ratio through a pre-trained video frame interpolation model to obtain a processed VR video; or, performing frame interpolation video processing on the VR video through a pre-trained video frame interpolation model to obtain a VR video with an optimized frame rate; and performing super-resolution processing on the VR video after the frame rate optimization through a pre-trained video super-resolution model to obtain the processed VR video.
In another possible implementation, the video hyper-score model is trained by: acquiring preset VR video original data; performing video quality degradation processing on the VR video original data to obtain VR video data with reduced definition; and training a preset neural network by taking the VR video data with reduced definition and the VR video original data as training sets to obtain a video hyper-resolution model.
In another possible implementation manner, the step of performing video quality degradation processing on the VR video original data to obtain VR video data with reduced definition includes: and carrying out video coding processing and/or video distortion deformation processing on the VR video original data to obtain VR video data with reduced definition.
In another possible implementation manner, the step of obtaining the preset VR video raw data includes: acquiring a plurality of initial videos shot by preset VR camera equipment at different visual angles; performing projection correction processing on the plurality of initial videos to obtain corrected videos; and carrying out image splicing processing on the corrected video to obtain a VR video to be processed with continuous pictures.
In another possible embodiment, the step of performing image stitching processing on the corrected video to obtain a VR video to be processed with continuous frames includes: extracting a feature vector of each video frame of the corrected video; performing feature matching and image transformation on the video frame according to the feature vector to obtain a video frame after image configuration; and carrying out image fusion processing on the video frame after image configuration to obtain the VR video to be processed with continuous pictures.
In another possible embodiment, the step of performing projection rectification processing on the plurality of initial videos to obtain rectified videos includes: projecting the plurality of initial videos to a unified coordinate system through cylindrical projection or spherical projection; and correcting the rotation angle of the image in the initial video after projection to obtain a corrected video.
For a brief description, the embodiment of the processing apparatus for VR video does not refer to the corresponding content in the embodiment of the processing method for VR video, and therefore, the processing apparatus for VR video can be applied to other embodiments of the processing apparatus for VR video.
An embodiment of the present invention further provides an electronic device, as shown in fig. 4, which is a schematic structural diagram of the electronic device, where the electronic device includes a processor 41 and a memory 42, the memory 42 stores machine executable instructions that can be executed by the processor 41, and the processor 41 executes the machine executable instructions to implement the processing method for the VR video.
In the embodiment shown in fig. 4, the electronic device further comprises a bus 43 and a communication interface 44, wherein the processor 41, the communication interface 44 and the memory 42 are connected by the bus.
The Memory 42 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 44 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
The processor 41 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 41. The Processor 41 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory, and the processor 41 reads the information in the memory 42, and completes the steps of the VR video processing method of the foregoing embodiment in combination with the hardware thereof.
An embodiment of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the VR video processing method, and specific implementation may refer to the foregoing method embodiment, and is not described herein again.
The VR video processing method, the VR video processing apparatus, and the computer program product of the electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the VR video processing method described in the foregoing method embodiments, and specific implementations may refer to the method embodiments and are not described herein again.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (11)
1. A method for processing VR video, comprising:
acquiring a VR video to be processed;
and performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video super-resolution model and a video frame interpolation model to obtain a processed VR video.
2. The method for processing the VR video of claim 1, wherein the step of performing super-resolution processing and frame interpolation video processing on the VR video through a pre-trained video hyper-resolution model and a pre-trained video frame interpolation model to obtain the processed VR video includes:
performing super-resolution processing on the VR video through a pre-trained video super-resolution model to obtain a VR video with optimized resolution; performing frame interpolation video processing on the VR video with optimized resolution through a pre-trained video frame interpolation model to obtain a processed VR video;
alternatively, the first and second electrodes may be,
performing frame interpolation video processing on the VR video through a pre-trained video frame interpolation model to obtain a VR video with an optimized frame rate; and performing super-resolution processing on the VR video after the frame rate optimization through a pre-trained video super-resolution model to obtain a processed VR video.
3. The method of processing VR video of claim 1, wherein the video hyper-resolution model is trained by:
acquiring preset VR video original data;
performing video quality degradation processing on the VR video original data to obtain VR video data with reduced definition;
and training a preset neural network by taking the VR video data with reduced definition and the VR video original data as training sets to obtain a video hyper-resolution model.
4. The method of claim 3, wherein the step of performing video quality degradation on the VR video raw data to obtain the VR video data with reduced definition includes:
and carrying out video coding processing and/or video distortion deformation processing on the VR video original data to obtain VR video data with reduced definition.
5. The method of processing VR video of claim 1, wherein the step of obtaining the VR video to be processed comprises:
acquiring a plurality of initial videos shot by preset VR camera equipment at different visual angles;
performing projection correction processing on the plurality of initial videos to obtain corrected videos;
and carrying out image splicing processing on the corrected video to obtain a VR video to be processed with continuous pictures.
6. The method for processing the VR video of claim 5, wherein the step of performing image stitching on the corrected video to obtain the VR video to be processed in successive frames includes:
extracting a feature vector of each video frame of the corrected video;
performing feature matching and image transformation on the video frame according to the feature vector to obtain a video frame after image configuration;
and carrying out image fusion processing on the video frames after image configuration to obtain the VR video to be processed with continuous pictures.
7. The VR video processing method of claim 5, wherein the step of performing projection correction processing on the plurality of initial videos to obtain corrected videos includes:
projecting the plurality of initial videos to a unified coordinate system through cylindrical projection or spherical projection;
and correcting the rotation angle of the image in the initial video after projection to obtain a corrected video.
8. An apparatus for processing VR video, comprising:
the VR video acquisition module is used for acquiring VR videos to be processed;
and the super-resolution and frame insertion processing module is used for performing super-resolution processing and frame insertion video processing on the VR video through a pre-trained video super-resolution model and a pre-trained video frame insertion model to obtain the processed VR video.
9. The VR video processing apparatus of claim 8, wherein the super and inter frame processing module is further configured to:
performing super-resolution processing on the VR video through a pre-trained video super-resolution model to obtain a VR video with optimized resolution; performing frame interpolation video processing on the VR video with optimized resolution through a pre-trained video frame interpolation model to obtain a processed VR video;
alternatively, the first and second electrodes may be,
performing frame interpolation video processing on the VR video through a pre-trained video frame interpolation model to obtain a VR video with an optimized frame rate; and performing super-resolution processing on the VR video after the frame rate optimization through a pre-trained video super-resolution model to obtain a processed VR video.
10. An electronic device, comprising a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the VR video processing method of any of claims 1 to 7.
11. A computer-readable storage medium having stored thereon computer-executable instructions that, when invoked and executed by a processor, cause the processor to implement the VR video processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010625574.3A CN111757087A (en) | 2020-06-30 | 2020-06-30 | VR video processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010625574.3A CN111757087A (en) | 2020-06-30 | 2020-06-30 | VR video processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111757087A true CN111757087A (en) | 2020-10-09 |
Family
ID=72680385
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010625574.3A Pending CN111757087A (en) | 2020-06-30 | 2020-06-30 | VR video processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111757087A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114025202A (en) * | 2021-11-03 | 2022-02-08 | 北京字节跳动网络技术有限公司 | Video processing method, device and storage medium |
| CN114285958A (en) * | 2021-12-28 | 2022-04-05 | 维沃移动通信有限公司 | Image processing circuit, image processing method, and electronic device |
| WO2022111631A1 (en) * | 2020-11-30 | 2022-06-02 | 华为技术有限公司 | Video transmission method, server, terminal, and video transmission system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101079151A (en) * | 2006-10-13 | 2007-11-28 | 浙江师范大学 | 360 degree around panorama generation method based on serial static image |
| US20100027664A1 (en) * | 2008-08-04 | 2010-02-04 | Kabushiki Kaisha Toshiba | Image Processing Apparatus and Image Processing Method |
| US20140067869A1 (en) * | 2012-08-30 | 2014-03-06 | Atheer, Inc. | Method and apparatus for content association and history tracking in virtual and augmented reality |
| CN108322685A (en) * | 2018-01-12 | 2018-07-24 | 广州华多网络科技有限公司 | Video frame interpolation method, storage medium and terminal |
| CN108961186A (en) * | 2018-06-29 | 2018-12-07 | 赵岩 | A kind of old film reparation recasting method based on deep learning |
| CN111080528A (en) * | 2019-12-20 | 2020-04-28 | 北京金山云网络技术有限公司 | Image super-resolution and model training method, device, electronic equipment and medium |
-
2020
- 2020-06-30 CN CN202010625574.3A patent/CN111757087A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101079151A (en) * | 2006-10-13 | 2007-11-28 | 浙江师范大学 | 360 degree around panorama generation method based on serial static image |
| US20100027664A1 (en) * | 2008-08-04 | 2010-02-04 | Kabushiki Kaisha Toshiba | Image Processing Apparatus and Image Processing Method |
| US20140067869A1 (en) * | 2012-08-30 | 2014-03-06 | Atheer, Inc. | Method and apparatus for content association and history tracking in virtual and augmented reality |
| CN108322685A (en) * | 2018-01-12 | 2018-07-24 | 广州华多网络科技有限公司 | Video frame interpolation method, storage medium and terminal |
| CN108961186A (en) * | 2018-06-29 | 2018-12-07 | 赵岩 | A kind of old film reparation recasting method based on deep learning |
| CN111080528A (en) * | 2019-12-20 | 2020-04-28 | 北京金山云网络技术有限公司 | Image super-resolution and model training method, device, electronic equipment and medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022111631A1 (en) * | 2020-11-30 | 2022-06-02 | 华为技术有限公司 | Video transmission method, server, terminal, and video transmission system |
| CN114025202A (en) * | 2021-11-03 | 2022-02-08 | 北京字节跳动网络技术有限公司 | Video processing method, device and storage medium |
| CN114025202B (en) * | 2021-11-03 | 2024-05-07 | 抖音视界有限公司 | Video processing method, device and storage medium |
| CN114285958A (en) * | 2021-12-28 | 2022-04-05 | 维沃移动通信有限公司 | Image processing circuit, image processing method, and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111580765B (en) | Screen projection method, screen projection device, storage medium, screen projection equipment and screen projection equipment | |
| CN111757087A (en) | VR video processing method and device and electronic equipment | |
| US10499056B2 (en) | System and method for video processing based on quantization parameter | |
| US10003768B2 (en) | Apparatus and methods for frame interpolation based on spatial considerations | |
| US20080232716A1 (en) | Video Processing Method and Device for Depth Extraction | |
| CN112365407B (en) | Panoramic stitching method for camera with configurable visual angle | |
| JP2006270676A (en) | Panorama image generating program, panorama image generating apparatus, and panorama image generation method | |
| WO2021057689A1 (en) | Video decoding method and apparatus, video encoding method and apparatus, storage medium, and electronic device | |
| CN111062881A (en) | Image processing method and device, storage medium and electronic equipment | |
| CN109688413B (en) | Method and encoder for encoding a video stream in a video encoding format supporting auxiliary frames | |
| WO2023246926A1 (en) | Model training method, video encoding method, and video decoding method | |
| CN110392265B (en) | Inter-frame motion estimation method and device, electronic equipment and readable storage medium | |
| CN111787300B (en) | VR video processing method and device and electronic equipment | |
| US20130141531A1 (en) | Computer program product, computer readable medium, compression method and apparatus of depth map in 3d video | |
| KR101806840B1 (en) | High Resolution 360 degree Video Generation System using Multiple Cameras | |
| WO2021011804A1 (en) | Video frame pulldown based on frame analysis | |
| US10783670B2 (en) | Method for compression of 360 degree content and electronic device thereof | |
| CN116208812A (en) | Video frame inserting method and system based on stereo event and intensity camera | |
| EP2656310B1 (en) | Method for producing a panoramic image and implementation apparatus | |
| CN113409209B (en) | Image deblurring method, device, electronic equipment and storage medium | |
| CN111711818B (en) | Video image coding transmission method and device thereof | |
| JP2018207356A (en) | Image compression program, image compression device, and image compression method | |
| CN115914834A (en) | Video processing method and device | |
| CN113037947A (en) | Method for coding spatial information in continuous dynamic image | |
| CN111556304A (en) | Panoramic image processing method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201009 |
|
| RJ01 | Rejection of invention patent application after publication |