CN112866507B - Intelligent panoramic video synthesis method and system, electronic device and medium - Google Patents
Intelligent panoramic video synthesis method and system, electronic device and medium Download PDFInfo
- Publication number
- CN112866507B CN112866507B CN202110043474.4A CN202110043474A CN112866507B CN 112866507 B CN112866507 B CN 112866507B CN 202110043474 A CN202110043474 A CN 202110043474A CN 112866507 B CN112866507 B CN 112866507B
- Authority
- CN
- China
- Prior art keywords
- foreground
- video
- panoramic video
- panoramic
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001308 synthesis method Methods 0.000 title claims abstract description 24
- 238000005286 illumination Methods 0.000 claims abstract description 115
- 238000000034 method Methods 0.000 claims abstract description 34
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 23
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 23
- 238000013507 mapping Methods 0.000 claims description 53
- 230000003044 adaptive effect Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 8
- 238000010276 construction Methods 0.000 claims description 4
- 230000002194 synthesizing effect Effects 0.000 claims description 4
- 230000001131 transforming effect Effects 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 206010003591 Ataxia Diseases 0.000 description 1
- 206010010947 Coordination abnormal Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 208000016290 incoordination Diseases 0.000 description 1
- 239000012528 membrane Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 229910052755 nonmetal Inorganic materials 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006748 scratching Methods 0.000 description 1
- 230000002393 scratching effect Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
Abstract
The invention provides an intelligent panoramic video synthesis method, a system, electronic equipment and a medium, wherein the method comprises the steps of adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video; acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone; and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video. The method and the device improve the reality sense of panoramic video synthesis.
Description
Technical Field
The present invention relates to the field of video synthesis, and more particularly, to an intelligent panoramic video synthesis method, system, electronic device, and medium.
Background
The panoramic video background superposition technology takes a pre-shot panoramic video as a synthetic background, and renders and fuses a foreground video shot by a user in front of a green screen in real time, so that a cross-space-time effect of transferring a real foreground video to a real panoramic video background is realized. The panoramic video is shot by a panoramic camera consisting of a plurality of fisheye lenses, the foreground video is shot by a linear lens, and due to the difference between shooting equipment and a shooting environment, the direct superposition of the foreground and the background cannot meet the perspective relation and illumination consistency of a foreground object and the background, so that a synthesized picture is inconsistent with a picture observed in the real world, and the reduction of the truth is caused.
The panoramic video background superposition technology is independent of the virtual studio technology. The virtual studio mainly comprises technical means such as green screen image matting and camera tracking, and can digitally synthesize a virtual three-dimensional scene made by a computer and a character moving image shot by a television camera on site, so that characters and virtual backgrounds synchronously change, a fantasy picture which cannot be seen in real life is presented, the limitation of studio making processes such as traditional scenery, props, lamplight and fields is broken through, and novel visual experience is provided for users.
Although virtual studio technologies have been widely used, most researchers and practitioners in the related arts tend to design "illusion" CG scenes using virtual engines and merge them with realistic foreground videos, which may actually create an exclamatory virtual special effect, but if the keyed foreground videos are merged with real background videos of different time and space, the "real illusion" may have more impact on viewers.
Augmented Reality (AR) technology is a technology for skillfully fusing virtual information and a real world, and is used for implementing analog simulation processing on information which is difficult to experience in a space range of the real world originally on the basis of scientific technologies such as computers and the like, superposing the information in the real world, and enabling the information to be perceived by human senses in the process, so that the sensory experience beyond Reality is realized. AR is a fusion between a real environment and a virtual object, and does not involve a fusion between a real environment and a real object.
Long Ye et al, 2019, proposed a simple and easy panoramic video background overlay scheme, which is based on a UE4 engine, and pastes a panoramic video on the inner side of a three-dimensional sphere for playing, and captures a background picture with a virtual camera placed at the center of the sphere. The foreground picture that the cell-phone was shot is placed on HUD user control, has realized after scratching the image with the superposition of panorama video. The scheme combines the virtual studio technology with the panoramic video, realizes the fusion of the real foreground video and the real background video in different time and space, and the 'unreal reality' can bring more fresh watching experience to audiences.
Because panoramic video and foreground video shooting equipment parameters are different, the panoramic video and the foreground video are directly overlapped to generate obvious incoordination, and Long Ye et al try to realize tone consistency by manually adjusting the gain of each color channel of the foreground video. However, the method needs to be manually adjusted respectively for different content input videos, and intelligent adaptation cannot be realized. In addition, the perspective relation and position consistency of the foreground and the background cannot be effectively guaranteed by the scheme.
The traditional virtual studio technology synthesizes a real foreground into a three-dimensional virtual scene generated by computer modeling, and the AR combines a virtual model with a real background, so that the two technologies can provide novel visual experience for a user, but the texture, color and contour of the virtual model are obviously different from those of a real object, so that the telepresence of the user is reduced to a certain extent.
Long Ye et al propose a background superimposition scheme for panoramic video to solve the above problems, and automatically render sub-regions of the corresponding panoramic video as background video according to the pose information of the foreground camera, thereby realizing organic fusion of foreground characters and real background. However, the following problems are not solved in this solution: 1. because the panoramic video and the foreground video have different shooting equipment parameters, the foreground and background illumination of the synthesized video is inconsistent, and although the scheme of Long Ye and the like designs an HSV color adjusting function, the manual adjusting mode is too laborious and time-consuming, and an ideal picture cannot be intelligently synthesized in real time; 2. on the other hand, the scheme of Long Ye et al places the panoramic video inside the three-dimensional sphere, automatically realizes fisheye transformation through prior structural information, and finally superimposes the transformed two-dimensional video and the two-dimensional foreground video. However, the method cannot effectively describe the perspective relationship between the foreground video and the background sub-video, and when the viewing angle changes significantly, the normal perspective relationship between the foreground object and the background video cannot be maintained, which affects the reality of the synthesized picture.
Disclosure of Invention
In view of the above problems, the present invention provides an intelligent panoramic video synthesis method, system, electronic device, and medium for improving the reality of panoramic video synthesis, mainly aiming at the two problems of inconsistent foreground and background positions and inconsistent illumination in the panoramic video background superposition technology.
According to an aspect of the present invention, there is provided an intelligent panoramic video synthesis method, including:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video.
Optionally, the step of collecting illumination information of the panoramic video and adaptively adjusting the color tone of the foreground video includes:
improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
acquiring illumination information contained in an HDR image obtained by inverse tone mapping;
and adjusting the illumination information of the foreground video to enable the illumination information of the foreground video and the HDR image to be similar.
Optionally, the step of collecting illumination information of the panoramic video and adaptively adjusting the color tone of the foreground video further includes:
the collected illumination information generates ambient light in the three-dimensional model.
Optionally, the step of improving the dynamic range of the panoramic video illumination information by using the inverse tone mapping method includes:
mapping the normal dynamic range panoramic video to a high dynamic range according to
Wherein,
r is an expansion threshold, and brightness values greater than R are expanded, I w,max For the expanded maximum luminance, α is the decay index that controls the stretching of the tone curve, L d (x) Is the original brightness, L w (x) Is the expanded brightness.
Optionally, the step of adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video includes:
obtaining a panoramic video signal in a three-dimensional polar coordinate system;
obtaining foreground attachment points of a foreground video in a panoramic video to form an attachment point set;
converting the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate conversion mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
obtaining a bottom foreground signal point set in the foreground signal point set by a threshold judgment method;
obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
randomly selecting a foreground attachment point in the attachment point set, and rotating the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
wherein the step of superimposing the adaptively adjusted foreground video on the panoramic video comprises:
and synthesizing the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
According to another aspect of the present invention, there is provided an intelligent panoramic video composition system, comprising:
the position self-adaptive adjusting module is used for self-adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;
the acquisition module is used for acquiring illumination information of the panoramic video after the foreground video of the position adaptive adjustment module is adaptively adjusted, wherein the illumination information comprises brightness and tone;
the illumination self-adaptive adjustment module is used for carrying out self-adaptive adjustment on illumination information of the foreground video according to the illumination information of the panoramic video acquired by the acquisition module;
and the three-dimensional synthesis module is used for superposing the foreground video adjusted by the illumination self-adaptive adjustment module in the three-dimensional panoramic image space of the panoramic video.
Optionally, the acquisition module comprises:
the mapping unit is used for improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
the information acquisition unit is used for acquiring illumination information contained in the HDR image obtained by inverse tone mapping;
the illumination self-adaptive adjusting module adjusts illumination information of the foreground video, so that the illumination information of the foreground video is similar to that of the HDR image.
Optionally, the position adaptive adjusting module includes:
the first acquisition unit is used for acquiring a panoramic video signal in a three-dimensional polar coordinate system;
the attachment point set constructing unit is used for obtaining foreground attachment points of the foreground video in the panoramic video to form an attachment point set;
the foreground signal point set construction unit is used for transforming the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate transformation mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
the bottom foreground signal point set constructing unit is used for acquiring a bottom foreground signal point set in the foreground signal point set by a threshold judging method;
the positioning point obtaining unit is used for obtaining positioning points of the foreground video signals according to the bottom foreground signal point set;
the rotating unit randomly selects a foreground attachment point in the attachment point set, and rotates the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
the mapping radius obtaining unit is used for obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
the scaling unit is used for scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
the three-dimensional synthesis module synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
In addition, the present invention also provides an electronic device including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described intelligent panoramic video synthesis method.
In addition, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the above-mentioned intelligent panoramic video synthesis method.
The intelligent panoramic video synthesis method, system, electronic equipment and medium are based on HDR illumination collection and under-three-dimensional space position consistency calibration, and therefore authenticity of foreground video and panoramic video synthesis is improved efficiently.
Drawings
Other objects and results of the present invention will become more apparent and more readily appreciated as the same becomes better understood by reference to the following description taken in conjunction with the accompanying drawings. In the drawings:
fig. 1 is a schematic flowchart of an intelligent panoramic video synthesis method according to an embodiment of the present invention;
FIG. 2 is a schematic block diagram of an intelligent panoramic video synthesis system according to another embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of an electronic device for implementing an intelligent panoramic video synthesis method according to an embodiment of the present invention.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.
The invention provides an intelligent panoramic video synthesis method. Fig. 1 is a schematic flow chart of an intelligent panoramic video synthesis method according to an embodiment of the present invention. The method may be performed by an apparatus, which may be implemented by software and/or hardware.
In this embodiment, the intelligent panoramic video synthesis method includes:
s1, adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
s2, collecting illumination information of the panoramic video, and performing self-adaptive adjustment on the illumination information of the foreground video, wherein the illumination information comprises brightness and tone;
and S3, overlapping the foreground video subjected to self-adaptive adjustment in the panoramic video.
In one embodiment, step S2 comprises:
improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
acquiring illumination information contained in an HDR image obtained by inverse tone mapping;
and adjusting the illumination information of the foreground video to enable the illumination information of the foreground video and the HDR image to be similar.
Optionally, the method further comprises:
the collected illumination information generates ambient light in the three-dimensional model.
The intelligent panoramic video synthesis method provides an efficient reality improving method based on the technologies of HDR illumination collection, position consistency calibration in three-dimensional space and the like, so that a more realistic foreground/background synthesis scheme is provided for a panoramic video background superposition system.
In one embodiment, the step of improving the dynamic range of the panoramic video illumination information by using the inverse tone mapping method comprises:
mapping a normal dynamic range panoramic video to a high dynamic range according to
Wherein,
r is an extension threshold, luminance values greater than R are all extended, I w,max For maximum brightness after expansion, α is the decay index controlling the tone curve extension, L d (x) Is the original brightness, L w (x) Is the expanded brightness.
In one embodiment, the step of acquiring the illumination information included in the HDR image obtained by inverse tone mapping includes:
the UE4 collects illumination information included in the HDR image obtained by the inverse tone mapping using a sky illumination unit.
In one embodiment, the step of generating the ambient light in the three-dimensional model by the acquired illumination information includes:
and generating ambient light in the three-dimensional model by utilizing illumination information acquired by the sky illumination component.
In one embodiment, the adjusting the illumination information of the foreground video so that the illumination information of the foreground video and the HDR image is similar comprises:
adjusting illumination information for foreground video according to
Wherein C is a constant for preventing numerical errors when the denominator approaches 0,
respectively representing the weighted average values of three channels of red, green and blue of the HDR image, and the calculation method is
Wherein x represents a weighted average of the red/green/blue channels, x i Representing the ith normalized pixel value in the red/green/blue channel, d i The distance between the ith background pixel and the foreground;
respectively representing the average values of the red, green and blue channels of the foreground video,
is composed of
Is composed of
Is composed of
In one embodiment, step S1 comprises:
obtaining a panoramic video signal in a three-dimensional polar coordinate system;
obtaining foreground attachment points of a foreground video in a panoramic video to form an attachment point set;
converting the foreground video signal and the mask signal thereof to three-dimensional polar coordinates through coordinate conversion mapping to obtain a foreground signal point set consisting of all foreground video signals in the mask signal;
obtaining a bottom foreground signal point set in the foreground signal point set by a threshold judgment method;
obtaining positioning points of the foreground video signals according to the bottom foreground signal point set;
randomly selecting a foreground attachment point in the attachment point set, and rotating the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is overlapped with the foreground attachment point;
obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
and scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius.
In one embodiment, the step of transforming the foreground video signal and its mask signal onto three-dimensional polar coordinates by coordinate transformation mapping comprises:
converting two-dimensional image plane coordinates of the foreground video signal and the mask signal thereof into longitude and latitude coordinates;
and superposing the longitude and latitude coordinates into the three-dimensional polar coordinates according to the radius of the panoramic video signal distribution plane.
In one embodiment, step S3 comprises:
and synthesizing the mask signal, the foreground video signal and the panoramic video signal which are subjected to position self-adaptive adjustment and illumination self-adaptive adjustment in the three-dimensional sphere.
In one embodiment, the intelligent panoramic video synthesis method is implemented based on HDR illumination acquisition and UE4, and includes two parts of illumination consistency and position consistency adjustment, which are specifically as follows:
step S10, calibrating the position consistency of the foreground/background images, that is, adaptively adjusting the synthesis position of the foreground video in the panoramic video, specifically:
the existing panoramic video overlay system mainly performs overlay operation in a two-dimensional image space, that is, a two-dimensional image obtained by directly selecting a part of regions from a panoramic video and performing fisheye transformation is used as a background video and then overlaid with a two-dimensional foreground image. However, the above-mentioned superimposing method cannot effectively record and maintain the perspective relationship between the foreground object and the background video, and when the viewing angle changes significantly, the foreground video and the background video have obvious inconsistency of position and mapping relationship. In view of the above problems, the present invention directly performs video overlay in a three-dimensional panoramic image space, thereby effectively maintaining a perspective relationship and maintaining position consistency thereof while synthesizing a foreground object background signal, and specifically includes:
the panoramic video signal in the three-dimensional polar coordinate system is B s Panoramic video signalDistributed at radius r 0 On a plane of (i.e.
Effective (because the panoramic signal is distributed at radius r 0 So only when r = r 0 Has a value when r is other values, is 0 when r is other values, valid may refer to having a value) if and only if r = r 0 . Wherein, the coordinate r is the distance from a point M in the space to the origin O;
is an angle formed by a half plane passing through the z-axis and the point M and the coordinate plane zOx; theta is the angle between the line segment OM and the positive direction of the z-axis. Selecting a plurality of (K) marking points as a foreground video attachment point set in a panoramic video according to user interest
The attachment point can be selected by the user, the style of the image preferred by the user can be obtained through the picture stored by the user client, and the attachment point conforming to the style is selected. Wherein p is i For the ith point in the set of points, the radius of each point
Are all r 0 ；
The two-dimensional foreground signal is F, the mask signal corresponding to the foreground signal is recorded as M, and the F and the M are respectively transformed to the three-dimensional polar coordinate through coordinate transformation mapping g (-) to obtain F s 、M s Foreground and mask signals in three-dimensional polar coordinates, respectively:
F s ＝g(F),M s ＝g(M)
wherein the step of coordinate transformation mapping g (-) includes: firstly, converting a two-dimensional image plane coordinate into a longitude and latitude coordinate, and finally, converting the longitude and latitude coordinate into a longitude and latitude coordinate with the radius r = r 0 Mapping to three-dimensional polar coordinates.
For three-dimensional polar undersizeSet of all foreground signal points in the membrane signal
The infimum of its vertical angular coordinate theta is noted
inf is short for infimum, represents the largest lower bound of the set and is positioned in the foreground signal point set at the bottom
Can pass through
And | theta-theta min < TH) }, wherein TH represents the judgment threshold of the bottom foreground signal point, namely all horizontal angles are equal to theta min (θ min Is the infimum of the vertical angular coordinate theta) is less than TH, the foreground signal points with the difference less than TH are all judged as bottom foreground signals, TH is obtained according to the following formula:
sup is the abbreviation of supremum, representing the upper bound of the set minimum;
from a set of foreground signal points
Obtaining a foreground signal anchor point
Wherein
Randomly selecting attachment point set P of foreground video B One point in the foreground is taken as a foreground attachment point p B I.e. p B ∈P B According to p B And a foreground video signal anchor point p B Is a distance of
Rotating the foreground signal in three-dimensional polar coordinates to enable the rotated foreground signal to be located at a point p F Same background attachment point p B And (4) overlapping. Wherein d is θ ,
Are each p B And p F The two points are at the position of theta,
the distance in coordinates.
Is a point p B At the time of the theta-phase shift,
the value on the coordinates.
Let the height of the camera in the three-dimensional coordinate system be h, the attachment point p B At r = r 0 On a spherical surface of, i.e.
According to the vertical angular coordinate of the attachment point
Determine the mapping radius
Then, the foreground signal F under the three-dimensional polar coordinate is obtained s Scaled to r m Is a radius on the inner surface of a three-dimensional sphere.
Step S20, acquiring HDR image illumination and illuminating a scene, that is, adaptively adjusting illumination information of a foreground video according to illumination information of a panoramic video, specifically:
the foreground video and the panoramic video are shot at different time and space, the illumination difference is large, and the reality of the picture is influenced if the foreground video and the panoramic video are directly synthesized. The invention carries out self-adaptive adjustment on the tone of a foreground video by collecting the illumination information of a panoramic video, and specifically comprises the following steps:
mapping the shot general dynamic range (1000,
wherein
In the UE4, the sky illumination unit is used to collect illumination information included in the HDR image obtained by inverse tone mapping, the illumination information mainly includes the overall brightness and the tone, and the environment light is generated in the three-dimensional model by using the illumination information collected by the sky illumination unit.
The method for calculating the similarity S of the foreground video and the background video, which is set to be non-metal, has the roughness of 1, can reflect illumination information, can reflect light containing illumination information of the panoramic HDR image based on the physically rendered foreground video, and has similar brightness and tone foreground and background as the background video, comprises the following steps:
step S30, according to the mask signal M under the three-dimensional coordinate system s A foreground signal F s And background signal B s Carrying out synthesis:
wherein, C S In order to synthesize the image(s),
to be scaled to r m Is a foreground image on the inner surface of a three-dimensional sphere of radius.
Because the foreground and the panoramic video serving as the background are shot in different environments, some differences exist in aspects of tone distribution, brightness and the like, the illumination model based on the HDR image is built, and the position consistency in the three-dimensional space is calibrated, so that the illumination consistency and the position consistency of the foreground image and the panoramic video can be ensured, the reality of the synthesized image is increased, and the watching and using experience of a user is improved.
Fig. 2 is a functional block diagram of the intelligent panoramic video synthesis system according to the present invention.
The intelligent panoramic video synthesis system 100 of the present invention can be installed in an electronic device. Depending on the implemented functions, the intelligent panoramic video synthesis system in one embodiment may include a position adaptive adjustment module 10, an acquisition module 20, an illumination adaptive adjustment module 30, and a three-dimensional synthesis module 40. A module according to the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions of the respective modules/units are as follows:
the intelligent panoramic video composition system 100 includes:
the position self-adaptive adjusting module 10 is used for self-adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
the acquisition module 20 is used for acquiring illumination information of the panoramic video after the position adaptive adjustment module foreground video is adaptively adjusted, wherein the illumination information comprises brightness and tone;
the illumination self-adaptive adjusting module 30 is used for self-adaptively adjusting the illumination information of the foreground video according to the illumination information of the panoramic video acquired by the acquisition module;
and the three-dimensional synthesis module 40 is used for superposing the foreground video adjusted by the illumination self-adaptive adjustment module in the three-dimensional panoramic image space of the panoramic video.
In one embodiment, the acquisition module 20 includes:
the mapping unit is used for improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
the information acquisition unit is used for acquiring illumination information contained in the HDR image obtained by inverse tone mapping;
the self-adaptive adjusting module adjusts the illumination information of the foreground video, so that the illumination information of the foreground video is similar to that of the HDR image.
In one embodiment, the adaptive position adjustment module 10 includes:
the first acquisition unit is used for acquiring a panoramic video signal in a three-dimensional polar coordinate system;
the attachment point set constructing unit is used for obtaining foreground attachment points of the foreground video in the panoramic video to form an attachment point set;
the foreground signal point set construction unit is used for transforming the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate transformation mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
the bottom foreground signal point set constructing unit is used for acquiring a bottom foreground signal point set in the foreground signal point set by a threshold judging method;
the positioning point obtaining unit is used for obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
the rotating unit randomly selects a foreground attachment point in the attachment point set, and rotates the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
the mapping radius obtaining unit is used for obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
the scaling unit is used for scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
the three-dimensional synthesis module 40 synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
The intelligent panoramic video synthesis system optimizes the position consistency of the synthesized image by a foreground/background synthesis method facing to a three-dimensional panoramic image space; illumination consistency of the foreground and the background is achieved by using illumination collection and scene illumination; .
Fig. 3 is a schematic structural diagram of an electronic device for implementing an intelligent panoramic video synthesis method according to the present invention.
The electronic device 200 may include a memory 210, a processor 220, and a bus, and may further include a computer program, such as a panoramic video composition program 211, stored in the memory 210 and executable on the processor 220.
The memory 210 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 210 may in some embodiments be an internal storage unit of the electronic device 200, such as a removable hard disk of the electronic device 200. The memory 210 may also be an external storage device of the electronic device 200 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the electronic device 200. Further, the memory 210 may also include both an internal storage unit and an external storage device of the electronic device 200. The memory 210 may be used not only to store application software installed in the electronic device 200 and various types of data, such as codes of a panoramic video composition program, but also to temporarily store data that has been output or is to be output.
The processor 220 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same function or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 220 is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules (e.g., a panoramic video composition program, etc.) stored in the memory 210 and calling data stored in the memory 210.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 210 and at least one processor 220 or the like.
Fig. 3 shows only an electronic device having components, and those skilled in the art will appreciate that the configuration shown in fig. 3 is not limiting to the electronic device 200, and may include fewer or more components than shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device 200 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 220 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are implemented through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 200 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 200 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (e.g., a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 200 and other electronic devices.
Optionally, the electronic device 200 may further include a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device 200 and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The image and category attention based object detection program 211 stored by the memory 210 in the electronic device 200 is a combination of instructions that, when executed in the processor 220, may implement:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video.
Specifically, the specific implementation method of the processor 220 for the instruction may refer to the description of the relevant steps in the corresponding embodiment of fig. 1, which is not repeated herein.
Further, the modules/units integrated with the electronic device 200 may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. The computer-readable medium may include: any entity or device capable of carrying said computer program code, a recording medium, a usb-disk, a removable hard disk, a magnetic diskette, an optical disk, a computer Memory, a Read-Only Memory (ROM).
In addition, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium may be non-volatile or volatile, and the computer-readable storage medium includes a computer program, where the computer program is executed by a processor to implement the following operations:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video.
The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the above-mentioned intelligent panoramic video synthesis method, apparatus, and electronic device, and will not be described herein again.
According to the intelligent panoramic video synthesis method, the intelligent panoramic video synthesis device and the electronic equipment, the dynamic range of the panoramic video is improved by using an inverse tone mapping method, and the panoramic video is used as scene illumination after illumination collection, so that background illumination can be reflected by a foreground video, and the illumination consistency of a foreground and a background is realized; in order to keep the perspective relation of the foreground/background video in the synthesis process, the perspective relation of the foreground/background is maintained and recorded in the whole synthesis process by a foreground/background synthesis method facing to a three-dimensional panoramic image space.
In the above embodiments, an embodiment is given in which the position adaptive adjustment is performed on the background video first and then the illumination adaptive adjustment is performed, but the present invention is not limited to this, and the illumination adaptive adjustment may be performed first and then the position adaptive adjustment is performed, and the advantage of performing the position adaptive adjustment first as compared with performing the illumination adaptive adjustment first is that the influence on the illumination due to the different positions can be avoided.
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not to denote any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (6)
1. An intelligent panoramic video synthesis method is characterized by comprising the following steps:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
overlaying the adaptively adjusted foreground video in the panoramic video,
the step of collecting the illumination information of the panoramic video and performing self-adaptive adjustment on the color tone of the foreground video comprises the following steps:
improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
acquiring illumination information contained in an HDR image obtained by inverse tone mapping;
adjusting illumination information of the foreground video to make the illumination information of the foreground video and the HDR image similar,
the step of adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video comprises the following steps:
obtaining a panoramic video signal in a three-dimensional polar coordinate system;
obtaining foreground attachment points of a foreground video in a panoramic video to form an attachment point set;
converting the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate conversion mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
obtaining a bottom foreground signal point set in the foreground signal point set by a threshold judgment method;
obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
randomly selecting a foreground attachment point in the attachment point set, and rotating the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
obtaining a mapping radius according to the height of the shooting device and a foreground attachment point in the three-dimensional polar coordinate system;
scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
wherein the step of superimposing the adaptively adjusted foreground video on the panoramic video comprises:
and synthesizing the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
2. The intelligent panoramic video synthesis method according to claim 1, wherein the step of acquiring illumination information of the panoramic video and adaptively adjusting the color tone of the foreground video further comprises:
the collected illumination information generates ambient light in the three-dimensional model.
3. The intelligent panoramic video synthesis method according to claim 1, wherein the step of improving the dynamic range of the panoramic video illumination information by using the inverse tone mapping method comprises:
mapping the normal dynamic range panoramic video to a high dynamic range according to
Wherein,
r is an extension threshold, luminance values greater than R are all extended, I w,max For maximum brightness after expansion, α is the decay index controlling the tone curve extension, L d (x) Is the original brightness, L w (x) Is the expanded brightness.
4. An intelligent panoramic video synthesis system, comprising:
the position self-adaptive adjusting module is used for self-adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;
the acquisition module is used for acquiring illumination information of the panoramic video after the foreground video of the position adaptive adjustment module is adaptively adjusted, wherein the illumination information comprises brightness and tone;
the illumination self-adaptive adjustment module is used for carrying out self-adaptive adjustment on illumination information of the foreground video according to the illumination information of the panoramic video acquired by the acquisition module;
the three-dimensional synthesis module is used for superposing the foreground video adjusted by the illumination self-adaptive adjustment module in a three-dimensional panoramic image space of the panoramic video, and the acquisition module comprises:
the mapping unit is used for improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
the information acquisition unit is used for acquiring illumination information contained in the HDR image obtained by inverse tone mapping;
the illumination self-adaptive adjusting module adjusts illumination information of the foreground video to enable the illumination information of the foreground video to be similar to that of the HDR image, and the position self-adaptive adjusting module comprises:
the first acquisition unit is used for acquiring a panoramic video signal in a three-dimensional polar coordinate system;
the attachment point set constructing unit is used for acquiring foreground attachment points of the foreground video in the panoramic video to form an attachment point set;
the foreground signal point set construction unit is used for transforming the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate transformation mapping to obtain a foreground signal point set consisting of all foreground video signals in the mask signals;
the bottom foreground signal point set construction unit is used for obtaining a bottom foreground signal point set in the foreground signal point set through a threshold judgment method;
the positioning point obtaining unit is used for obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
the rotating unit randomly selects a foreground attachment point in the attachment point set, and rotates the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
the mapping radius obtaining unit is used for obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
the scaling unit is used for scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
the three-dimensional synthesis module synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
5. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the intelligent panoramic video compositing method of any of claims 1-3.
6. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the intelligent panoramic video composition method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110043474.4A CN112866507B (en) | 2021-01-13 | 2021-01-13 | Intelligent panoramic video synthesis method and system, electronic device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110043474.4A CN112866507B (en) | 2021-01-13 | 2021-01-13 | Intelligent panoramic video synthesis method and system, electronic device and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112866507A CN112866507A (en) | 2021-05-28 |
| CN112866507B true CN112866507B (en) | 2023-01-10 |
Family
ID=76003456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110043474.4A Active CN112866507B (en) | 2021-01-13 | 2021-01-13 | Intelligent panoramic video synthesis method and system, electronic device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112866507B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114003331A (en) * | 2021-11-10 | 2022-02-01 | 浙江博采传媒有限公司 | LED circular screen virtual reality synthesis method and device, storage medium and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007140834A (en) * | 2005-11-17 | 2007-06-07 | System Keikaku Kenkyusho:Kk | Program, information storage medium, photograph printer and photograph printing method |
| CN108027961A (en) * | 2015-08-31 | 2018-05-11 | 汤姆逊许可公司 | Method and apparatus for inverse tone mapping (ITM) |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9883155B2 (en) * | 2016-06-14 | 2018-01-30 | Personify, Inc. | Methods and systems for combining foreground video and background video using chromatic matching |
-
2021
- 2021-01-13 CN CN202110043474.4A patent/CN112866507B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007140834A (en) * | 2005-11-17 | 2007-06-07 | System Keikaku Kenkyusho:Kk | Program, information storage medium, photograph printer and photograph printing method |
| CN108027961A (en) * | 2015-08-31 | 2018-05-11 | 汤姆逊许可公司 | Method and apparatus for inverse tone mapping (ITM) |
Non-Patent Citations (1)
| Title |
|---|
| 马尔科夫随机场化的光照一致图像合成方法;黄美玉;《计算机辅助设计与图形学学报》;20150430;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112866507A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110650368B (en) | Video processing method and device and electronic equipment | |
| KR100260786B1 (en) | A system for implanting an image into a video stream | |
| JP7007348B2 (en) | Image processing equipment | |
| US11425283B1 (en) | Blending real and virtual focus in a virtual display environment | |
| US10719920B2 (en) | Environment map generation and hole filling | |
| JP2006053694A (en) | Space simulator, space simulation method, space simulation program and recording medium | |
| US11276150B2 (en) | Environment map generation and hole filling | |
| CN112446939A (en) | Three-dimensional model dynamic rendering method and device, electronic equipment and storage medium | |
| CN111866523A (en) | Panoramic video synthesis method and device, electronic equipment and computer storage medium | |
| CN114979689B (en) | Multi-machine-position live broadcast guide method, equipment and medium | |
| CN112866507B (en) | Intelligent panoramic video synthesis method and system, electronic device and medium | |
| CN110870304B (en) | Method and apparatus for providing information to a user for viewing multi-view content | |
| CN109427089B (en) | Mixed reality object presentation based on ambient lighting conditions | |
| CN115486091A (en) | System and method for video processing using virtual reality devices | |
| CN114066715A (en) | Image style migration method and device, electronic equipment and storage medium | |
| JP4046973B2 (en) | Information processing method and image mixing apparatus | |
| JP4323910B2 (en) | Image composition apparatus and method | |
| US10902669B2 (en) | Method for estimating light for augmented reality and electronic device thereof | |
| JPH11328443A (en) | System and method for generating three-dimensional panorama image and recording media therefor | |
| CN114332356A (en) | Virtual and real picture combining method and device | |
| CN112915536A (en) | Rendering method and device of virtual model | |
| Lee | Wand: 360∘ video projection mapping using a 360∘ camera | |
| TW202223842A (en) | Image processing method and device for panorama image | |
| CN116245741B (en) | Image processing method and related device | |
| WO2023160219A1 (en) | Light supplementing model training method, image processing method, and related device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |