CN112866507A - Intelligent panoramic video synthesis method and system, electronic device and medium - Google Patents
Intelligent panoramic video synthesis method and system, electronic device and medium Download PDFInfo
- Publication number
- CN112866507A CN112866507A CN202110043474.4A CN202110043474A CN112866507A CN 112866507 A CN112866507 A CN 112866507A CN 202110043474 A CN202110043474 A CN 202110043474A CN 112866507 A CN112866507 A CN 112866507A
- Authority
- CN
- China
- Prior art keywords
- foreground
- video
- panoramic video
- panoramic
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
Abstract
The invention provides an intelligent panoramic video synthesis method, a system, electronic equipment and a medium, wherein the method comprises the steps of adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video; acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone; and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video. The method and the device improve the reality sense of panoramic video synthesis.
Description
Technical Field
The present invention relates to the field of video synthesis, and more particularly, to an intelligent panoramic video synthesis method, system, electronic device, and medium.
Background
The panoramic video background superposition technology takes a pre-shot panoramic video as a synthetic background, and renders and fuses a foreground video shot by a user in front of a green screen in real time, so that a cross-space-time effect of transferring a real foreground video to a real panoramic video background is realized. The panoramic video is shot by a panoramic camera consisting of a plurality of fisheye lenses, the foreground video is shot by a linear lens, and due to the difference between shooting equipment and a shooting environment, the direct superposition of the foreground and the background cannot meet the perspective relation and illumination consistency of a foreground object and the background, so that a synthesized picture is inconsistent with a picture observed in the real world, and the reduction of the truth is caused.
The panoramic video background superposition technology is a virtual studio technology. The virtual studio mainly comprises technical means such as green screen image matting and camera tracking, and can digitally synthesize a virtual three-dimensional scene made by a computer and a character moving image shot by a television camera on site, so that characters and virtual backgrounds synchronously change, a fantasy picture which cannot be seen in real life is presented, the limitation of studio making processes such as traditional scenery, props, lamplight and fields is broken through, and novel visual experience is provided for users.
Although the virtual studio technology has been widely used, most researchers and practitioners in the related arts tend to design an "illusion" CG scene using a virtual engine and blend the "illusion" CG scene with a real foreground video, which may create an impressive virtual special effect.
Augmented Reality (AR) technology is a technology for skillfully fusing virtual information and a real world, and is used for implementing analog simulation processing on information which is difficult to experience in a space range of the real world originally on the basis of scientific technologies such as computers and the like, superposing the information in the real world, and enabling the information to be perceived by human senses in the process, so that the sensory experience beyond Reality is realized. AR is a fusion between a real environment and a virtual object and does not involve a fusion between a real environment and a real object.
Long Ye et al in 2019 propose a simple and easy panoramic video background overlay scheme, which is based on a UE4 engine, and the panoramic video is pasted on the inner side of a three-dimensional sphere to be played, and a virtual camera placed at the center of the sphere is used for capturing background pictures. The foreground picture that the cell-phone was shot places on HUD user control, has realized after scratching the image with the stack of panoramic video. The scheme combines the virtual studio technology with the panoramic video, realizes the fusion of the real foreground video and the real background video in different time and space, and the 'unreal reality' can bring more fresh watching experience to audiences.
Because panoramic video and foreground video shooting equipment parameters are different, the panoramic video and the foreground video are directly overlapped to generate obvious incoordination, and Long Ye et al try to realize tone consistency by manually adjusting the gain of each color channel of the foreground video. However, the method needs to be manually adjusted respectively for different content input videos, and intelligent adaptation cannot be realized. In addition, the scheme cannot effectively ensure the perspective relation and position consistency of the foreground and the background.
The traditional virtual studio technology synthesizes a real foreground into a three-dimensional virtual scene generated by computer modeling, and the AR combines a virtual model with a real background, so that the two technologies can provide novel visual experience for a user, but the texture, color and contour of the virtual model are obviously different from those of a real object, so that the telepresence of the user is reduced to a certain extent.
Long Ye et al propose a background superimposition scheme for panoramic video to solve the above problems, and automatically render sub-regions of the corresponding panoramic video as background video according to the pose information of the foreground camera, thereby realizing organic fusion of foreground characters and real background. However, the following problems are not solved in this solution: 1. because the parameters of panoramic video and foreground video shooting equipment are different, the illumination of the foreground and background of the synthesized video is inconsistent, and although the scheme of Long Ye and the like designs an HSV color adjusting function, the manual adjusting mode is too laborious and time-consuming, and an ideal picture cannot be intelligently synthesized in real time; 2. on the other hand, the scheme of Long Ye et al places the panoramic video inside the three-dimensional sphere, automatically realizes fisheye transformation through prior structural information, and finally superimposes the transformed two-dimensional video and the two-dimensional foreground video. However, the method cannot effectively describe the perspective relationship between the foreground video and the background sub-video, and when the viewing angle changes remarkably, the normal perspective relationship between the foreground object and the background video cannot be maintained, so that the reality of the synthesized picture is influenced.
Disclosure of Invention
In view of the above problems, the present invention provides an intelligent panoramic video synthesis method, system, electronic device, and medium for improving the reality of panoramic video synthesis, mainly aiming at the two problems of inconsistent foreground and background positions and inconsistent illumination in the panoramic video background superposition technology.
According to an aspect of the present invention, there is provided an intelligent panoramic video synthesis method, including:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video.
Optionally, the step of collecting illumination information of the panoramic video and adaptively adjusting the color tone of the foreground video includes:
improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
acquiring illumination information contained in an HDR image obtained by inverse tone mapping;
and adjusting the illumination information of the foreground video to enable the illumination information of the foreground video and the HDR image to be similar.
Optionally, the step of collecting illumination information of the panoramic video and adaptively adjusting the color tone of the foreground video further includes:
the collected illumination information generates ambient light in the three-dimensional model.
Optionally, the step of improving the dynamic range of the panoramic video illumination information by using the inverse tone mapping method includes:
mapping the normal dynamic range panoramic video to a high dynamic range according to
Wherein the content of the first and second substances,
r is an extension threshold, luminance values greater than R are all extended, Iw,maxFor maximum brightness after expansion, α is the decay index controlling the tone curve extension, Ld(x) Is the original brightness, Lw(x) Is the expanded brightness.
Optionally, the step of adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video includes:
obtaining a panoramic video signal in a three-dimensional polar coordinate system;
obtaining foreground attachment points of a foreground video in a panoramic video to form an attachment point set;
converting the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate conversion mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
obtaining a bottom foreground signal point set in the foreground signal point set by a threshold judgment method;
obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
randomly selecting a foreground attachment point in the attachment point set, and rotating the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is overlapped with the foreground attachment point;
obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
wherein the step of superimposing the adaptively adjusted foreground video on the panoramic video comprises:
and synthesizing the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
According to another aspect of the present invention, there is provided an intelligent panoramic video composition system, comprising:
the position self-adaptive adjusting module is used for self-adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;
the acquisition module is used for acquiring illumination information of the panoramic video after the foreground video of the position adaptive adjustment module is adaptively adjusted, wherein the illumination information comprises brightness and tone;
the illumination self-adaptive adjustment module is used for self-adaptively adjusting the illumination information of the foreground video according to the illumination information of the panoramic video acquired by the acquisition module;
and the three-dimensional synthesis module is used for superposing the foreground video adjusted by the illumination self-adaptive adjustment module in the three-dimensional panoramic image space of the panoramic video.
Optionally, the acquisition module comprises:
the mapping unit is used for improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
the information acquisition unit is used for acquiring illumination information contained in the HDR image obtained by inverse tone mapping;
the illumination self-adaptive adjusting module adjusts illumination information of the foreground video, so that the illumination information of the foreground video is similar to that of the HDR image.
Optionally, the position adaptive adjustment module includes:
the first acquisition unit is used for acquiring a panoramic video signal in a three-dimensional polar coordinate system;
the attachment point set constructing unit is used for acquiring foreground attachment points of the foreground video in the panoramic video to form an attachment point set;
the foreground signal point set construction unit is used for transforming the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate transformation mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
the bottom foreground signal point set constructing unit is used for acquiring a bottom foreground signal point set in the foreground signal point set by a threshold judging method;
the positioning point obtaining unit is used for obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
the rotating unit randomly selects a foreground attachment point in the attachment point set, and rotates the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
the mapping radius obtaining unit is used for obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
the scaling unit is used for scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
the three-dimensional synthesis module synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
In addition, the present invention also provides an electronic device including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described intelligent panoramic video composition method.
The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the above-mentioned intelligent panoramic video synthesis method.
The intelligent panoramic video synthesis method, system, electronic equipment and medium disclosed by the invention are based on HDR illumination acquisition and three-dimensional spatial lower position consistency calibration, and the reality of foreground video and panoramic video synthesis is efficiently improved.
Drawings
Other objects and results of the present invention will become more apparent and more readily appreciated as the same becomes better understood by reference to the following description taken in conjunction with the accompanying drawings. In the drawings:
fig. 1 is a schematic flowchart of an intelligent panoramic video synthesis method according to an embodiment of the present invention;
FIG. 2 is a block diagram of an intelligent panoramic video synthesis system according to another embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of an electronic device for implementing an intelligent panoramic video synthesis method according to an embodiment of the present invention.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides an intelligent panoramic video synthesis method. Fig. 1 is a schematic flow chart of an intelligent panoramic video synthesis method according to an embodiment of the present invention. The method may be performed by an apparatus, which may be implemented by software and/or hardware.
In this embodiment, the intelligent panoramic video synthesis method includes:
step S1, adaptively adjusting the position of a foreground video in the three-dimensional panoramic image space of the panoramic video;
step S2, collecting illumination information of the panoramic video, and carrying out self-adaptive adjustment on the illumination information of the foreground video, wherein the illumination information comprises brightness and tone;
and step S3, superposing the foreground video after the adaptive adjustment in the panoramic video.
In one embodiment, step S2 includes:
improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
acquiring illumination information contained in an HDR image obtained by inverse tone mapping;
and adjusting the illumination information of the foreground video to enable the illumination information of the foreground video and the HDR image to be similar.
Optionally, the method further comprises:
the collected illumination information generates ambient light in the three-dimensional model.
The intelligent panoramic video synthesis method provides an efficient reality sense improving method based on the technologies of HDR illumination collection, position consistency calibration in three-dimensional space and the like, so that a foreground/background synthesis scheme with more authenticity is provided for a panoramic video background superposition system.
In one embodiment, the step of improving the dynamic range of the panoramic video illumination information by using the inverse tone mapping method comprises:
mapping the normal dynamic range panoramic video to a high dynamic range according to
Wherein the content of the first and second substances,
r is an extension threshold, luminance values greater than R are all extended, Iw,maxFor maximum brightness after expansion, α is the decay index controlling the tone curve extension, Ld(x) Is the original brightness, Lw(x) Is the expanded brightness.
In one embodiment, the step of acquiring the illumination information included in the HDR image obtained by inverse tone mapping includes:
the UE4 collects illumination information included in the HDR image obtained by the inverse tone mapping using a sky illumination unit.
In one embodiment, the step of generating the ambient light in the three-dimensional model by the acquired illumination information comprises:
and generating ambient light in the three-dimensional model by utilizing illumination information acquired by the sky illumination component.
In one embodiment, the adjusting the illumination information of the foreground video so that the illumination information of the foreground video and the HDR image is similar comprises:
adjusting illumination information for foreground video according to
Wherein C is a constant for preventing numerical errors when the denominator approaches 0,
respectively representing the weighted average values of three channels of red, green and blue of the HDR image, and the calculation method is
Wherein x represents the weighted average of the red/green/blue channels, xiRepresenting the ith normalized pixel value in the red/green/blue channel, diThe distance between the ith background pixel and the foreground;
respectively representing the average values of the red, green and blue channels of the foreground video,
is composed of
Is composed of
Is composed of
In one embodiment, step S1 includes:
obtaining a panoramic video signal in a three-dimensional polar coordinate system;
obtaining foreground attachment points of a foreground video in a panoramic video to form an attachment point set;
converting the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate conversion mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
obtaining a bottom foreground signal point set in the foreground signal point set by a threshold judgment method;
obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
randomly selecting a foreground attachment point in the attachment point set, and rotating the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is overlapped with the foreground attachment point;
obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
and scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius.
In one embodiment, the step of transforming the foreground video signal and its mask signal onto three-dimensional polar coordinates by coordinate transformation mapping comprises:
converting two-dimensional image plane coordinates of the foreground video signal and the mask signal thereof into longitude and latitude coordinates;
and superposing the longitude and latitude coordinates into the three-dimensional polar coordinates according to the radius of the panoramic video signal distribution plane.
In one embodiment, step S3 includes:
and synthesizing the mask signal, the foreground video signal and the panoramic video signal which are subjected to position self-adaptive adjustment and illumination self-adaptive adjustment in the three-dimensional sphere.
In one embodiment, the intelligent panoramic video synthesis method is implemented based on HDR luminance acquisition and UE4, and includes two parts of illumination consistency and position consistency adjustment, which are as follows:
step S10, calibrating the position consistency of the foreground/background images, that is, adaptively adjusting the composite position of the foreground video in the panoramic video, specifically:
the existing panoramic video superposition system mainly performs superposition operation in a two-dimensional image space, namely directly selects a two-dimensional image obtained by fish-eye transformation of a part of regions from a panoramic video as a background video, and then superposes the two-dimensional image with a two-dimensional foreground image. However, the above-mentioned superimposing method cannot effectively record and maintain the perspective relationship between the foreground object and the background video, and when the viewing angle changes significantly, the foreground video and the background video have obvious inconsistency of position and mapping relationship. In view of the above problems, the present invention directly performs video overlay in a three-dimensional panoramic image space, thereby effectively maintaining a perspective relationship and maintaining position consistency thereof while synthesizing a foreground object background signal, and specifically includes:
the panoramic video signal in the three-dimensional polar coordinate system is BsThe panoramic video signal is distributed at a radius r0On a plane of (i.e. i
Effective (because the panoramic signal is distributed at radius r0So only when r is r0Has a value when r is other values, is 0 when r is other values, effectively can mean a value) if and only if r ═ r0. Wherein, the coordinate r is the distance from a point M in the space to the origin O;
is the angle formed by the half-plane passing through the z-axis and point M with the coordinate plane zOx; theta is the angle between the line segment OM and the positive direction of the z-axis. Selecting a plurality of (K) marking points as a foreground video attachment point set in a panoramic video according to user interest
The attachment point can be selected by the user himself or herself, orAnd obtaining the style of the image preferred by the user through the picture stored by the user client, and selecting the attachment point according with the style. Wherein p isiFor the ith point in the set of points, the radius of each point
Are all r0；
The two-dimensional foreground signal is F, the mask signal corresponding to the foreground signal is recorded as M, and the F and the M are respectively transformed to the three-dimensional polar coordinate through coordinate transformation mapping g (-) to obtain Fs、MsForeground and mask signals in three-dimensional polar coordinates, respectively:
Fs＝g(F),Ms＝g(M)
wherein the step of coordinate transformation mapping g (-) includes: firstly, converting the plane coordinate of the two-dimensional image into a longitude and latitude coordinate, and finally, converting the longitude and latitude coordinate into a longitude and latitude coordinate according to the radius r ═ r0Mapping to three-dimensional polar coordinates.
For all foreground signal point sets in mask signals under three-dimensional polar coordinates
The infimum of its vertical angular coordinate theta is noted
inf is short for infimum, represents the largest lower bound of the set and is positioned in the foreground signal point set at the bottom
Can pass through
And | theta-thetamin< TH), wherein TH represents a judgment threshold of a bottom foreground signal point, namely all horizontal angles are equal to thetamin(θminIs the infimum of the vertical angular coordinate theta) is less than THTH is obtained according to the following formula:
sup is the abbreviation of supremum, representing the upper bound of the set minimum;
from a set of foreground signal points
Obtaining a foreground signal anchor point
Wherein
Randomly selecting attachment point set P of foreground videoBOne point in the foreground is taken as a foreground attachment point pBI.e. pB∈PBAccording to pBAnd foreground video signal anchor point pBIs a distance of
Rotating the foreground signal in a three-dimensional polar coordinate to enable the rotated foreground signal to be located at a point pFSame background attachment point pBAnd (4) overlapping. Wherein d isθ,
Are each pBAnd pFThe two points are at the position of theta,
distance in coordinates.
Is a point pBAt the time of the theta-phase shift,
the value on the coordinates.
Let the height of the camera in the three-dimensional coordinate system be h, the attachment point pBAt r ═ r0On a spherical surface of, i.e.
According to the vertical angular coordinate of the attachment point
Determine the mapping radius
Then, the foreground signal F under the three-dimensional polar coordinate is obtainedsScaled to rmIs a radius on the inner surface of a three-dimensional sphere.
Step S20, HDR image illumination collection and scene illumination, that is, adaptive adjustment is performed on the illumination information of the foreground video according to the illumination information of the panoramic video, specifically:
the foreground video and the panoramic video are shot at different time and space, the illumination difference is large, and the reality of the picture is influenced if the foreground video and the panoramic video are directly synthesized. The invention carries out self-adaptive adjustment on the tone of a foreground video by collecting the illumination information of a panoramic video, and specifically comprises the following steps:
mapping the shot common dynamic range (1000:1) panoramic video to a high dynamic range (100000:1) by utilizing an inverse tone mapping method,
wherein
In the UE4, the lighting information included in the HDR image obtained by inverse tone mapping is collected by a sky lighting unit, and mainly includes the overall brightness and tone thereof, and the lighting information collected by the sky lighting unit is used to generate the ambient light in the three-dimensional model.
The method for calculating the similarity S of the foreground video and the background video, which is set to be non-metal, has the roughness of 1, can reflect illumination information, can reflect light containing illumination information of the panoramic HDR image based on the physically rendered foreground video, and has similar brightness and tone foreground and background as the background video, comprises the following steps:
step S30, according to the mask signal M under the three-dimensional coordinate systemsA foreground signal FsAnd background signal BsCarrying out synthesis:
wherein, CSIn order to synthesize the image, the image is synthesized,
to be scaled to rmIs a foreground image on the inner surface of a three-dimensional sphere of radius.
Because the foreground and the panoramic video serving as the background are shot in different environments, some differences exist in aspects of tone distribution, brightness and the like, the illumination model based on the HDR image is built, and the position consistency in the three-dimensional space is calibrated, so that the illumination consistency and the position consistency of the foreground image and the panoramic video can be ensured, the reality of the synthesized image is increased, and the watching and using experience of a user is improved.
Fig. 2 is a functional block diagram of the intelligent panoramic video synthesis system according to the present invention.
The intelligent panoramic video synthesis system 100 of the present invention can be installed in an electronic device. Depending on the implemented functions, the intelligent panoramic video synthesis system in one embodiment may include a position adaptive adjustment module 10, an acquisition module 20, an illumination adaptive adjustment module 30, and a three-dimensional synthesis module 40. A module according to the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the intelligent panoramic video composition system 100 includes:
the position self-adaptive adjusting module 10 is used for self-adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
the acquisition module 20 is used for acquiring illumination information of the panoramic video after the position adaptive adjustment module foreground video is adaptively adjusted, wherein the illumination information comprises brightness and tone;
the illumination self-adaptive adjusting module 30 is used for self-adaptively adjusting the illumination information of the foreground video according to the illumination information of the panoramic video acquired by the acquisition module;
and the three-dimensional synthesis module 40 is used for superposing the foreground video adjusted by the illumination self-adaptive adjustment module in the three-dimensional panoramic image space of the panoramic video.
In one embodiment, the acquisition module 20 includes:
the mapping unit is used for improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
the information acquisition unit is used for acquiring illumination information contained in the HDR image obtained by inverse tone mapping;
the adaptive adjusting module adjusts the illumination information of the foreground video, so that the illumination information of the foreground video is similar to that of the HDR image.
In one embodiment, the adaptive position adjustment module 10 includes:
the first acquisition unit is used for acquiring a panoramic video signal in a three-dimensional polar coordinate system;
the attachment point set constructing unit is used for acquiring foreground attachment points of the foreground video in the panoramic video to form an attachment point set;
the foreground signal point set construction unit is used for transforming the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate transformation mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
the bottom foreground signal point set constructing unit is used for acquiring a bottom foreground signal point set in the foreground signal point set by a threshold judging method;
the positioning point obtaining unit is used for obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
the rotating unit randomly selects a foreground attachment point in the attachment point set, and rotates the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
the mapping radius obtaining unit is used for obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
the scaling unit is used for scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
the three-dimensional synthesis module 40 synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
The intelligent panoramic video synthesis system optimizes the position consistency of the synthesized image by a foreground/background synthesis method facing to a three-dimensional panoramic image space; illumination consistency of the foreground and the background is achieved by using illumination collection and scene illumination; .
Fig. 3 is a schematic structural diagram of an electronic device for implementing an intelligent panoramic video synthesis method according to the present invention.
The electronic device 200 may include a memory 210, a processor 220, and a bus, and may further include a computer program, such as a panoramic video composition program 211, stored in the memory 210 and executable on the processor 220.
The memory 210 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 210 may in some embodiments be an internal storage unit of the electronic device 200, such as a removable hard disk of the electronic device 200. The memory 210 may also be an external storage device of the electronic device 200 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the electronic device 200. Further, the memory 210 may also include both an internal storage unit and an external storage device of the electronic device 200. The memory 210 may be used not only to store application software installed in the electronic device 200 and various types of data, such as codes of a panoramic video composition program, but also to temporarily store data that has been output or is to be output.
The processor 220 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 220 is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules (e.g., a panoramic video composition program, etc.) stored in the memory 210 and calling data stored in the memory 210.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 210 and at least one processor 220 or the like.
Fig. 3 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 3 does not constitute a limitation of the electronic device 200, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device 200 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 220 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are implemented through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 200 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 200 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (e.g., a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 200 and other electronic devices.
Optionally, the electronic device 200 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), or optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device 200 and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The image and category attention based object detection program 211 stored by the memory 210 in the electronic device 200 is a combination of instructions that, when executed in the processor 220, may implement:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video.
Specifically, the processor 220 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, and details thereof are not repeated herein.
Further, the modules/units integrated with the electronic device 200 may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
In addition, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium may be non-volatile or volatile, and the computer-readable storage medium includes a computer program, where the computer program is executed by a processor to implement the following operations:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video.
The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the above-mentioned intelligent panoramic video synthesis method, apparatus, and electronic device, and will not be described herein again.
According to the intelligent panoramic video synthesis method, the intelligent panoramic video synthesis device and the electronic equipment, the dynamic range of the panoramic video is improved by using an inverse tone mapping method, and the panoramic video is used as scene illumination after illumination collection, so that background illumination can be reflected by a foreground video, and the illumination consistency of a foreground and a background is realized; in order to keep the perspective relation of the foreground/background video in the synthesis process, the perspective relation of the foreground/background is maintained and recorded in the whole synthesis process by a foreground/background synthesis method facing to a three-dimensional panoramic image space.
In the above embodiments, an embodiment is given in which the position adaptive adjustment is performed on the background video first and then the illumination adaptive adjustment is performed, but the present invention is not limited to this, and the illumination adaptive adjustment may be performed first and then the position adaptive adjustment is performed, and the advantage of performing the position adaptive adjustment first as compared with performing the illumination adaptive adjustment first is that the influence on the illumination due to the different positions can be avoided.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. An intelligent panoramic video synthesis method is characterized by comprising the following steps:
adaptively adjusting the position of a foreground video in a three-dimensional panoramic image space of a panoramic video;
acquiring illumination information of a panoramic video, and performing self-adaptive adjustment on the illumination information of a foreground video, wherein the illumination information comprises brightness and tone;
and superposing the foreground video subjected to self-adaptive adjustment in the panoramic video.
2. The intelligent panoramic video synthesis method according to claim 1, wherein the step of acquiring illumination information of the panoramic video and adaptively adjusting the color tone of the foreground video comprises:
improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
acquiring illumination information contained in an HDR image obtained by inverse tone mapping;
and adjusting the illumination information of the foreground video to enable the illumination information of the foreground video and the HDR image to be similar.
3. The intelligent panoramic video synthesis method according to claim 2, wherein the step of acquiring illumination information of the panoramic video and adaptively adjusting the color tone of the foreground video further comprises:
the collected illumination information generates ambient light in the three-dimensional model.
4. The intelligent panoramic video synthesis method according to claim 2, wherein the step of improving the dynamic range of the panoramic video illumination information by using the inverse tone mapping method comprises:
mapping the normal dynamic range panoramic video to a high dynamic range according to
Wherein the content of the first and second substances,
r is an extension threshold, luminance values greater than R are all extended, Iw，maxFor maximum brightness after expansion, α is the decay index controlling the tone curve extension, Ld(x) Is the original brightness, Lw(x) Is the expanded brightness.
5. The intelligent panoramic video synthesis method of claim 1, wherein the step of adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video comprises:
obtaining a panoramic video signal in a three-dimensional polar coordinate system;
obtaining foreground attachment points of a foreground video in a panoramic video to form an attachment point set;
converting the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate conversion mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
obtaining a bottom foreground signal point set in the foreground signal point set by a threshold judgment method;
obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
randomly selecting a foreground attachment point in the attachment point set, and rotating the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is overlapped with the foreground attachment point;
obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
wherein the step of superimposing the adaptively adjusted foreground video on the panoramic video comprises:
and synthesizing the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
6. An intelligent panoramic video composition system, comprising:
the position self-adaptive adjusting module is used for self-adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;
the acquisition module is used for acquiring illumination information of the panoramic video after the foreground video of the position adaptive adjustment module is adaptively adjusted, wherein the illumination information comprises brightness and tone;
the illumination self-adaptive adjustment module is used for self-adaptively adjusting the illumination information of the foreground video according to the illumination information of the panoramic video acquired by the acquisition module;
and the three-dimensional synthesis module is used for superposing the foreground video adjusted by the illumination self-adaptive adjustment module in the three-dimensional panoramic image space of the panoramic video.
7. The intelligent panoramic video composition system of claim 6, wherein the capture module comprises:
the mapping unit is used for improving the dynamic range of the panoramic video illumination information by using an inverse tone mapping method to obtain an HDR image;
the information acquisition unit is used for acquiring illumination information contained in the HDR image obtained by inverse tone mapping;
the illumination self-adaptive adjusting module adjusts illumination information of the foreground video, so that the illumination information of the foreground video is similar to that of the HDR image.
8. The intelligent panoramic video synthesis system of claim 6, wherein the position adaptive adjustment module comprises:
the first acquisition unit is used for acquiring a panoramic video signal in a three-dimensional polar coordinate system;
the attachment point set constructing unit is used for acquiring foreground attachment points of the foreground video in the panoramic video to form an attachment point set;
the foreground signal point set construction unit is used for transforming the foreground video signals and the mask signals thereof to three-dimensional polar coordinates through coordinate transformation mapping to obtain a foreground signal point set consisting of all the foreground video signals in the mask signals;
the bottom foreground signal point set constructing unit is used for acquiring a bottom foreground signal point set in the foreground signal point set by a threshold judging method;
the positioning point obtaining unit is used for obtaining a positioning point of the foreground video signal according to the bottom foreground signal point set;
the rotating unit randomly selects a foreground attachment point in the attachment point set, and rotates the foreground video signal in a three-dimensional polar coordinate according to the distance between the foreground attachment point and a positioning point, so that the positioning point of the rotated foreground video signal is superposed with the foreground attachment point;
the mapping radius obtaining unit is used for obtaining a mapping radius according to the height of the shooting device and the foreground attachment point in the three-dimensional polar coordinate system;
the scaling unit is used for scaling the foreground video signal under the rotated three-dimensional polar coordinate to the inner surface of a three-dimensional sphere with the mapping radius as the radius;
the three-dimensional synthesis module synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the intelligent panoramic video compositing method of any of claims 1-5.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the intelligent panoramic video composition method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110043474.4A CN112866507B (en) | 2021-01-13 | 2021-01-13 | Intelligent panoramic video synthesis method and system, electronic device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110043474.4A CN112866507B (en) | 2021-01-13 | 2021-01-13 | Intelligent panoramic video synthesis method and system, electronic device and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112866507A true CN112866507A (en) | 2021-05-28 |
| CN112866507B CN112866507B (en) | 2023-01-10 |
Family
ID=76003456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110043474.4A Active CN112866507B (en) | 2021-01-13 | 2021-01-13 | Intelligent panoramic video synthesis method and system, electronic device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112866507B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114003331A (en) * | 2021-11-10 | 2022-02-01 | 浙江博采传媒有限公司 | LED circular screen virtual reality synthesis method and device, storage medium and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007140834A (en) * | 2005-11-17 | 2007-06-07 | System Keikaku Kenkyusho:Kk | Program, information storage medium, photograph printer and photograph printing method |
| US20170359559A1 (en) * | 2016-06-14 | 2017-12-14 | Personify, Inc. | Methods and systems for combining foreground video and background video using chromatic matching |
| CN108027961A (en) * | 2015-08-31 | 2018-05-11 | 汤姆逊许可公司 | Method and apparatus for inverse tone mapping (ITM) |
-
2021
- 2021-01-13 CN CN202110043474.4A patent/CN112866507B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007140834A (en) * | 2005-11-17 | 2007-06-07 | System Keikaku Kenkyusho:Kk | Program, information storage medium, photograph printer and photograph printing method |
| CN108027961A (en) * | 2015-08-31 | 2018-05-11 | 汤姆逊许可公司 | Method and apparatus for inverse tone mapping (ITM) |
| US20170359559A1 (en) * | 2016-06-14 | 2017-12-14 | Personify, Inc. | Methods and systems for combining foreground video and background video using chromatic matching |
Non-Patent Citations (1)
| Title |
|---|
| 黄美玉: "马尔科夫随机场化的光照一致图像合成方法", 《计算机辅助设计与图形学学报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114003331A (en) * | 2021-11-10 | 2022-02-01 | 浙江博采传媒有限公司 | LED circular screen virtual reality synthesis method and device, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112866507B (en) | 2023-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110650368B (en) | Video processing method and device and electronic equipment | |
| KR100260786B1 (en) | A system for implanting an image into a video stream | |
| JP2006053694A (en) | Space simulator, space simulation method, space simulation program and recording medium | |
| US11425283B1 (en) | Blending real and virtual focus in a virtual display environment | |
| US10719920B2 (en) | Environment map generation and hole filling | |
| US11276150B2 (en) | Environment map generation and hole filling | |
| CN106558017B (en) | Spherical display image processing method and system | |
| CN111866523B (en) | Panoramic video synthesis method and device, electronic equipment and computer storage medium | |
| CN112446939A (en) | Three-dimensional model dynamic rendering method and device, electronic equipment and storage medium | |
| CN112348937A (en) | Face image processing method and electronic equipment | |
| CN112866507B (en) | Intelligent panoramic video synthesis method and system, electronic device and medium | |
| CN109427089B (en) | Mixed reality object presentation based on ambient lighting conditions | |
| CN110870304A (en) | Method and apparatus for providing information to a user for viewing multi-view content | |
| CN114979689B (en) | Multi-machine-position live broadcast guide method, equipment and medium | |
| US10902669B2 (en) | Method for estimating light for augmented reality and electronic device thereof | |
| US11120606B1 (en) | Systems and methods for image texture uniformization for multiview object capture | |
| CN114332356A (en) | Virtual and real picture combining method and device | |
| CN112915536A (en) | Rendering method and device of virtual model | |
| TW202223842A (en) | Image processing method and device for panorama image | |
| CN115546041B (en) | Training method of light supplementing model, image processing method and related equipment thereof | |
| CN116245741B (en) | Image processing method and related device | |
| US11568552B2 (en) | Imaging systems and methods incorporating improved culling of virtual objects | |
| Lee | Wand: 360∘ video projection mapping using a 360∘ camera | |
| US20090322744A1 (en) | System and method for displaying pictures in digital photo frame | |
| CN113012015A (en) | Watermark adding method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |