CN108734791B - Panoramic video processing method and device - Google Patents
Panoramic video processing method and device Download PDFInfo
- Publication number
- CN108734791B CN108734791B CN201810276791.9A CN201810276791A CN108734791B CN 108734791 B CN108734791 B CN 108734791B CN 201810276791 A CN201810276791 A CN 201810276791A CN 108734791 B CN108734791 B CN 108734791B
- Authority
- CN
- China
- Prior art keywords
- multimedia
- panoramic
- sphere
- distance
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
Abstract
The invention provides a method and a device for processing a panoramic video, wherein the method comprises the following steps: receiving a multimedia element; determining a first location of a corresponding multimedia element on the surface of a panoramic ball, wherein a rectangular panoramic video image is rendered on the inner surface of the panoramic ball; determining a distance D for translating the first positioning to the direction of the center of the panorama ball; determining the target location of the multimedia elements in the inner space of the panorama ball according to the first location and the distance D; and adding a multimedia element to the target positioning part, wherein the visual surface of the multimedia element has a preselected datum point, the datum point is overlapped with the target positioning position, and the connecting line of the sphere center and the target positioning is vertical to the visual surface. The invention can improve the position change fault tolerance rate of the multimedia elements, and can flexibly and dynamically configure the multimedia elements by means of the received two-dimensional coordinate information, the multimedia elements and the like without being limited by the edition sending period.
Description
Technical Field
The invention relates to the technical field of panoramic videos, in particular to a method and a device for processing a panoramic video.
Background
At present, when multimedia elements are added to a panoramic application or a panoramic video, the main steps are as follows: firstly, a trained operator operates a Unity engine, manually selects a spatial position of a multimedia element to be added in a panorama ball of the Unity engine, and manually adds the multimedia element to a corresponding spatial position in the panorama ball, so as to obtain a position parameter of the multimedia element; then, technical personnel writes codes for adding the multimedia elements by using the complex position parameters to realize the client; and finally, presenting the version sent by the client to the user. Wherein a multimedia element is added in a certain spatial position in the panoramic video in the client presented to the user. It should be noted that, here, only the Unity engine is taken as an example for description, and in practical applications, the engine of the panorama sphere is not limited to this.
In this scheme, each time a multimedia element is added to the panoramic video, a publishing period is waited, that is, for example, a certain multimedia element is added to a certain spatial position in a client of version 1.0 published at time t 1; when the background wants to add the multimedia elements to the client again or modify the added multimedia elements, the multimedia elements can be added or modified in the client of the version 2.0 by releasing the client of the new version 2.0 only by waiting for T2 after a release period T elapses. The adding period of the multimedia elements depends on the version upgrading period of the client, and the support capability for temporary operation activities is poor.
In addition, in the scheme, because the multimedia elements are manually added to a certain spatial position in the panoramic ball by an operator on the engine side, the added positions of the multimedia elements are easy to be out of the expected requirements due to large error and low precision of manual addition, and the position change fault-tolerant rate is low; in addition, since the adding and modifying periods of the multimedia elements depend on the version upgrading period of the client, the positions, styles and the like of the added multimedia elements cannot be changed before the upgrading time is not reached, so that the requirements of dynamically configuring the multimedia elements cannot be met.
Therefore, in the prior art, when configuring multimedia elements in a panoramic video, the problems that the configuration time is limited by the edition sending period of a client, the position change fault tolerance rate of the multimedia elements and the multimedia elements cannot be dynamically added generally exist.
Disclosure of Invention
The invention provides a method and a device for processing a panoramic video, which aim to solve the problems that in the prior art, when multimedia elements are configured in the panoramic video, the configuration time is limited by the edition sending time of a client, the position change fault tolerance rate of the multimedia elements and the multimedia elements cannot be dynamically added.
In order to solve the above problem, according to an aspect of the present invention, the present invention discloses a method for processing a panoramic video, including:
receiving a multimedia element;
determining a first location on the surface of a panorama sphere corresponding to the multimedia elements, wherein a rectangular panorama video image is rendered on the inner surface of the panorama sphere;
determining a distance D for translating the first location toward a center of the panorama ball;
determining the target location of the multimedia elements in the inner space of the panorama ball according to the first location and the distance D;
and adding the multimedia elements to the target positioning part, wherein the visual surface of the multimedia elements has a preselected datum point, the datum point is overlapped with the target positioning position, and the connecting line of the sphere center and the target positioning is vertical to the visual surface.
Optionally, the determining a first location on the surface of the panorama sphere corresponding to the multimedia element comprises:
receiving two-dimensional coordinate information of the rectangular panoramic video image;
receiving a second position corresponding to the multimedia element in the rectangular panoramic video image;
rendering the rectangular panoramic video image to the inner surface of the panoramic ball according to a preset mapping relation between the plane coordinates and the spherical coordinates and the coordinate information of the rectangular panoramic video image;
and determining the corresponding first positioning of the second positioning on the surface of the panoramic ball according to the mapping relation.
Optionally, the determining a distance D to translate the first location toward a center of the panorama ball comprises:
and determining the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the size information of the multimedia elements and the sphere radius R of the panoramic sphere.
Optionally, the determining, according to the size information of the multimedia element and the sphere radius R of the panoramic sphere, a distance D for translating the first location in the direction of the center of the panoramic sphere includes:
determining a reference point on a visual surface of the multimedia element according to the size information of the multimedia element;
calculating the farthest distance L between the reference point and each edge point on the visible surface;
and calculating the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the farthest distance L and the sphere radius R.
Optionally, the determining a reference point on a visual surface of the multimedia element according to the size information of the multimedia element includes:
determining the geometric center of the visual surface of the multimedia element according to the size information of the multimedia element;
determining the geometric center as a reference point on the visual plane.
Optionally, before calculating the distance D for translating the first location to the center direction of the panoramic ball according to the farthest distance L and the radius R, the method further includes:
calculating the scaling M of the multimedia elements according to the size information of the panorama ball and the size information of the multimedia elements;
after the calculating the farthest distance L between the reference point and each edge point on the visible surface, the method further includes:
reducing the farthest distance L according to the scaling ratio M to obtain a scaling distance M x L;
reducing the multimedia elements according to the scaling M to obtain scaled multimedia elements;
calculating a distance D for translating the first location to a direction of a center of the panorama sphere according to the farthest distance L and the sphere radius R, including:
calculating a distance D for translating the first positioning to the spherical center direction of the panoramic sphere according to the zooming distance M and the spherical radius R;
the adding the multimedia element to the target location comprises:
adding the scaled multimedia elements to the target location.
Optionally, the calculating, according to the farthest distance L and the sphere radius R, a distance D for translating the first location toward a sphere center direction of the panoramic sphere includes:
according to another aspect of the present invention, the present invention also discloses a device for processing panoramic video, comprising:
the receiving module is used for receiving the multimedia elements;
the first determining module is used for determining first positioning corresponding to the multimedia elements on the surface of the panoramic ball, wherein a rectangular panoramic video image is rendered on the inner surface of the panoramic ball;
a second determining module, configured to determine a distance D for translating the first positioning toward the center of the panorama ball;
a third determining module, configured to determine, according to the first location and the distance D, a target location of the multimedia element in an internal space of the panorama ball;
and the adding module is used for adding the multimedia elements to the target positioning position, wherein the visual surface of the multimedia elements has a preselected datum point, the datum point is overlapped with the target positioning position, and the connecting line of the sphere center and the target positioning is vertical to the visual surface.
Optionally, the first determining module includes:
the first receiving submodule is used for receiving the two-dimensional coordinate information of the rectangular panoramic video image;
the second receiving submodule is used for receiving second positioning corresponding to the multimedia elements in the rectangular panoramic video image;
the rendering submodule is used for rendering the rectangular panoramic video image to the inner surface of the panoramic ball according to the mapping relation between the preset plane coordinates and the preset spherical coordinates and the coordinate information of the rectangular panoramic video image;
and the first determining submodule is used for determining the corresponding first positioning of the second positioning on the surface of the panoramic ball according to the mapping relation.
Optionally, the second determining module includes:
and the second determining submodule is used for determining the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the size information of the multimedia elements and the sphere radius R of the panoramic sphere.
Optionally, the second determining sub-module includes:
the determining unit is used for determining a reference point on a visual surface of the multimedia element according to the size information of the multimedia element;
the first calculating unit is used for calculating the farthest distance L between the reference point and each edge point on the visible surface;
and the second calculating unit is used for calculating the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the farthest distance L and the sphere radius R.
Optionally, the determining unit includes:
the first determining subunit is used for determining the geometric center of the visual surface of the multimedia element according to the size information of the multimedia element;
a second determining subunit, configured to determine the geometric center as a reference point on the visual surface.
Optionally, the apparatus further comprises:
the calculation module is used for calculating the scaling M of the multimedia elements according to the size information of the panoramic ball and the size information of the multimedia elements;
a first reducing module, configured to reduce the farthest distance L according to the scaling ratio M to obtain a scaling distance M × L;
the second reducing module is used for reducing the multimedia elements according to the scaling M to obtain scaled multimedia elements;
the second calculation unit includes:
a calculating subunit, configured to calculate, according to the zoom distance M × L and the sphere radius R, a distance D for translating the first positioning toward the center of the panoramic sphere;
the adding module comprises:
an adding sub-module for adding the scaled multimedia elements to the target location.
Optionally, the second calculating unit is configured to calculate the distance D according to the following formula:
compared with the prior art, the invention has the following advantages:
therefore, the embodiment of the invention only needs to receive the two-dimensional coordinate information of each pixel point in the rectangular panoramic video image and a certain two-dimensional coordinate selected from the rectangular panoramic video image and used for adding the multimedia elements, can accurately render the two-dimensional rectangular panoramic video image to the inner surface of the panoramic ball according to the mapping relation between the preset plane coordinate and the spherical coordinate, and uniquely determine the corresponding three-dimensional coordinate of the certain two-dimensional coordinate in the panoramic ball, namely uniquely determine the three-dimensional coordinate position used for adding the multimedia elements on the surface of the panoramic ball after rendering the panoramic video image. The method of the embodiment of the invention can receive the two-dimensional coordinate information used for adding the multimedia elements in the two-dimensional expansion map of the panoramic video and the two-dimensional coordinate information of the two-dimensional expansion map of the panoramic video at any time, thereby accurately determining the three-dimensional coordinate information used for adding the multimedia elements in the panoramic ball in the two-dimensional expansion map of the panoramic video at any time and improving the position change fault-tolerant rate of the multimedia elements; and the multimedia elements can be flexibly and dynamically configured by means of the received two-dimensional coordinate information, the multimedia elements and the like, and the method is not limited by the publishing period.
Drawings
FIG. 1 is a flow chart of the steps of an embodiment of a method of processing panoramic video according to the present invention;
FIG. 2 is a schematic diagram of the coordinate transformation of a panoramic video image according to the present invention;
FIG. 3 is a schematic diagram of a multimedia element of the present invention;
FIG. 4 is a schematic diagram of a method for calculating the target location of a multimedia element according to the present invention;
fig. 5 is a block diagram of a processing apparatus for panoramic video according to an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Referring to fig. 1, a flowchart illustrating steps of an embodiment of a method for processing a panoramic video according to the present invention is shown, where the method may be applied to a client, and specifically may include the following steps:
the types of the multimedia elements can comprise videos, images, models, texts or barrages; wherein, the video may include but is not limited to picture-in-picture video, scenario video, etc.; the images may include, but are not limited to, poster pictures, in-scene hint pictures, merchandise pictures, and the like; the model may include, but is not limited to, a 3D model of a computer image added to the scene; the characters include but are not limited to subtitles, prompt characters in scenes, commodity introduction characters and the like; the barrage is a comment of the user in the interaction, including but not limited to a text barrage.
As mentioned above, the multimedia element may be a tile (i.e. 2D), such as video, images, text and barrage, or a convex polyhedron (i.e. 3D), such as the 3D model described above.
In the invention, no matter whether the client side is published or not, or whether the current time reaches the publishing time point of the client side, at any time point, only multimedia elements are required to be configured for a certain frame of panoramic video image of the panoramic video of the client side, the server only needs to input the multimedia elements to be added to the client side, and manual arrangement and adding positions of the multimedia elements in a panoramic ball of a Unity engine are not required, so that the client side can receive the multimedia elements which are required to be added to the certain frame of panoramic video image of the panoramic video.
Of course, if the same multimedia element needs to be added to different or the same position of a multi-frame panoramic video image of a certain panoramic video, the implementation principle of the present invention is not changed, that is, the multimedia element corresponds to one frame panoramic video image or a multi-frame panoramic video image, or the same panoramic video, or different panoramic videos, and there is no limitation on the implementation of the scheme of the present invention.
The same or different multimedia elements may be added to different positions of different panoramic videos, which is not limited by the present invention.
the panoramic ball in this embodiment is a hollow structure, and the first positioning is consistent between the outer surface of the panoramic ball and the inner surface of the panoramic ball, wherein the thickness of the ball wall is negligible, so the first positioning can be understood as the position corresponding to the multimedia element on the inner/outer surface of the panoramic ball. Since the rectangular panoramic video image is rendered on the inner surface of the panorama ball, the first positioning essence is the position of the rectangular panoramic video image on the panorama ball, where the multimedia elements need to be added.
For the convenience of the reader to understand the technical solution of the present invention, the play principle of the panoramic video is briefly introduced here:
at present, 2D video viewed by a user is played by a current rectangular player, and when the 2D video needs to be converted into panoramic video for viewing, since a video stream is composed of multiple frames of video images, each frame of video image played by the rectangular player can be rendered to the inner surface of a panoramic ball. The camera is arranged at the center of the sphere of the panoramic ball and is connected with the visual equipment of the user, namely, the eyes of the user are equivalent to the center of the sphere of the camera, so that each frame of video image rendered in the panoramic ball can be seen, and the effect of seeing the panoramic video is achieved. It should be noted that, when the video image rendered from the rectangular player to the panorama ball is a normal rectangular image that we normally see without any stretching, the video image rendered to the panorama ball may have a problem that the image is squeezed and deformed. Therefore, the video image rendered from the rectangular player to the inner surface of the panoramic ball is a rectangular panoramic video image, namely a video image specially used for panoramic playing, and the video image does not have the problem of image extrusion deformation when being displayed on the inner surface of the panoramic ball.
Wherein the camera may be a camera within an engine (e.g., Unity engine) in which the panorama ball is located.
Wherein the first position may be determined in the following manner when adding multimedia elements to the rectangular panoramic video map. In the mode 1, a server calculates a first positioning corresponding to the multimedia element in a rendered rectangular panoramic video image on the surface of a panoramic ball in advance, and then sends the first positioning to a client; in the mode 2, the client determines the first positioning of the multimedia element in the rectangular panoramic video image rendered on the inner surface of the panoramic ball according to the second positioning of the multimedia element sent by the server in the rectangular panoramic video image of the rectangular player.
Wherein the first location is a three-dimensional location of a certain point in a rectangular panoramic video image located on the inner surface of the panoramic ball;
and the second location is a two-dimensional location of the point in the rectangular panoramic video map lying on the plane.
In one embodiment, the specific implementation of determining the first location on the surface of the panorama sphere corresponding to the multimedia element in manner 2 is set forth below:
receiving two-dimensional coordinate information of the rectangular panoramic video image;
receiving a second position corresponding to the multimedia element in the rectangular panoramic video image;
the two sets of information are received from the server, and the receiving order between the two sets of information is not limited in the present invention, and the receiving order between the two sets of information and the multimedia elements in step 101 is also not limited in the present invention.
Taking a rectangular panoramic video image with a 2:1 ratio as an example, as shown in the left diagram in fig. 2, when multimedia elements need to be added to a panoramic video of a client, the server side can mark the position where the multimedia elements need to be added, i.e. the second positioning, on a two-dimensional expansion diagram (i.e. a rectangular panoramic video image) of the panoramic video image. Specifically, a standard 2:1 panorama video development may be used as a basic coordinate plot chart, a central point is used as a (0,0) origin, and coordinate values of respective pixels are calibrated on a two-dimensional development for the entire panorama video shown in the left diagram in fig. 2. And selecting a coordinate point (x, y) on the two-dimensional expansion map, namely a second positioning, for adding the multimedia elements.
And then, sending the coordinate information of the two-dimensional expansion map of the panoramic video map and a second positioning for adding the multimedia elements in the two-dimensional expansion map to a client.
It should be noted that how to establish the planar rectangular coordinate system for the two-dimensional expansion map of the panoramic video image is not limited in the present invention, for example, in other embodiments, a coordinate system may be established with an arbitrary position in the two-dimensional expansion map as an origin, and is not limited to the central point of the two-dimensional expansion map as the origin.
Rendering the rectangular panoramic video image to the inner surface of the panoramic ball according to a preset mapping relation between the plane coordinates and the spherical coordinates and the coordinate information of the rectangular panoramic video image;
and determining the corresponding first positioning of the second positioning on the surface of the panoramic ball according to the mapping relation.
In this embodiment of the present invention, a mapping relationship between the plane rectangular coordinate of the 2:1 two-dimensional expansion diagram and the three-dimensional coordinate of the panorama sphere used in this embodiment may be configured on the client side in advance. The mapping relation is related to the size proportion of the two-dimensional expansion diagram and the size proportion of the panoramic ball, and in practical application, the mapping relation can be flexibly adjusted according to the difference of the two size proportions.
Thus, as shown in fig. 2, the two-dimensional expanded view in the left drawing of fig. 2 can be rendered on the panorama ball in the right drawing of fig. 2. In this step, the two-dimensional expansion map may be rendered to the inner surface of the panoramic ball according to the mapping relationship between the preset planar coordinates and the spherical coordinates and the coordinate values of the pixels calibrated on the two-dimensional expansion map, so that the coordinate value of each pixel in the two-dimensional expansion map has a unique three-dimensional coordinate value corresponding to the coordinate value on the surface of the panoramic ball.
Correspondingly, a certain two-dimensional coordinate (x, y), namely a second location, in the two-dimensional expansion map for adding the multimedia elements is calibrated, and a corresponding three-dimensional location, namely a first location (x ', y ', z '), on the surface of the panoramic ball is determined according to the mapping relation.
Therefore, the embodiment of the invention only needs to receive the two-dimensional coordinate information of each pixel point in the rectangular panoramic video image and a certain two-dimensional coordinate selected from the rectangular panoramic video image and used for adding the multimedia elements, can accurately render the two-dimensional rectangular panoramic video image to the inner surface of the panoramic ball according to the mapping relation between the preset plane coordinate and the spherical coordinate, and uniquely determine the corresponding three-dimensional coordinate of the certain two-dimensional coordinate in the panoramic ball, namely uniquely determine the three-dimensional coordinate position used for adding the multimedia elements on the surface of the panoramic ball after rendering the panoramic video image. The method of the embodiment of the invention can receive the two-dimensional coordinate information used for adding the multimedia elements in the two-dimensional expansion map of the panoramic video and the two-dimensional coordinate information of the two-dimensional expansion map of the panoramic video at any time, thereby accurately determining the three-dimensional coordinate information used for adding the multimedia elements in the panoramic ball in the two-dimensional expansion map of the panoramic video at any time and improving the position change fault-tolerant rate of the multimedia elements; and the multimedia elements can be flexibly and dynamically configured by means of the received two-dimensional coordinate information, the multimedia elements and the like, and the method is not limited by the publishing period.
since the multimedia elements are patches or convex polyhedrons, if the multimedia elements are only attached to the first location (on the surface of the panorama sphere), as shown in fig. 4, the multimedia elements 11 are attached to the first location C1(x ', y ', z '), such that the multimedia elements 11 are only attached to the outer surface of the panorama sphere, and the multimedia elements 11 cannot be captured by the camera at the center of the panorama sphere M, so that human eyes cannot see the multimedia elements 11 in the panorama video rendered on the inner surface of the panorama sphere. Therefore, in order to make the multimedia element 11 capable of being shot by the camera, the first positioning C1(x ', y ', z ') corresponding to the multimedia element 11 needs to be moved toward the inside of the panorama sphere. Specifically, as shown in fig. 4, it is necessary to determine a distance D for translating the first location C1(x ', y ', z ') in the direction of the center M of the panorama sphere (i.e., in the direction of the arrow shown in fig. 4), wherein the multimedia element 11 ' of the first location C1(x ', y ', z ') moved by the distance D is located as shown in fig. 4.
In one embodiment, step 103 may be implemented by S31. S31, determining the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the size information of the multimedia elements and the sphere radius R of the panoramic sphere.
In one embodiment, when performing S31, this may be achieved by:
s311, determining a reference point on the visual surface of the multimedia element according to the size information of the multimedia element;
wherein, the multimedia elements can be 2D patches and also can be 3D convex polyhedrons.
When the multimedia element is a 2D patch, for example, a picture, the picture side of the picture is a visible surface; when the multimedia element is a 3D convex polyhedron, one surface for the user to browse can be selected from a plurality of surfaces of the convex polyhedron, i.e. a visible surface, and certainly, in other embodiments, when the multimedia element is in a 3D shape, the visible surface can be a plurality of surfaces.
Since the multimedia element is intended to be viewed by the user as a viewing surface and the first location defined on the surface of the panorama sphere for adding the multimedia element is only a point, it is necessary to select a reference point in the viewing surface of the multimedia element for alignment with the first location of the surface of the panorama sphere.
Wherein the reference point may be selected based on size information of the multimedia element when the reference point is selected.
As shown in fig. 3, the multimedia element in the form of a 2D tile, where the shaded portion is a visual surface of the tile, when selecting the reference point, any one point in the shaded portion may be selected as the reference point according to a user instruction, or a geometric center of the shaded portion may be selected as the reference point according to a system configuration.
In one embodiment, when S311 is executed, the following steps may be performed:
determining the geometric center of the visual surface of the multimedia element according to the size information of the multimedia element;
determining the geometric center as a reference point on the visual plane.
Wherein, as shown in fig. 3, the geometric center N of the hatched patch can be determined as a reference point on the visual surface of the multimedia element.
In this way, in the embodiment of the present invention, the geometric center of the visual surface of the multimedia element is used as the reference point for the first positioning for aligning the surface of the panoramic ball, and compared with selecting other points on the visual surface as the reference points, the maximum distance L can be made smaller when the advancing distance D is calculated in the subsequent process, and the distance D advanced by the first positioning to the inside of the panoramic ball is also made smaller, so that when the multimedia element is finally added to the target positioning mentioned in the subsequent process, the fusion degree between the added multimedia element seen at the user viewing angle and the panoramic video on the inner surface of the panoramic ball is higher, the video viewing experience is improved, and the problem that the multimedia element is seen at the user viewing angle and disconnected from the panoramic video due to the larger advancing distance D is avoided.
S312, calculating the farthest distance L between the reference point and each edge point on the visible surface;
as shown in fig. 3, in order to calculate the farthest distance L quickly and accurately, a circle may be drawn outside the patch (shaded area) with the geometric center N as the center of the circle, and the radius of the circle may be reduced continuously until the first point of the patch coincides with the drawn circle, and the distance between the point (here, K, B) on the patch and the geometric center N, that is, the radius of the currently drawn circle is the farthest distance L between the reference point N and each edge point on the visible surface.
In other embodiments, if the reference point is not the geometric center N, but is point a in fig. 3, the farthest distance L is the distance between point a and point B, here L ', and then referring back to fig. 4, in order to move the visual surface of the tile all the way inside the panorama sphere, a larger L ' would make the distance D that the tile moves toward the center of the sphere too large, so that the user's perspective would see the multimedia elements out of focus from the panorama video.
And S313, calculating the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the farthest distance L and the sphere radius R.
Among them, in one embodiment, as shown in fig. 4, the distance D may be calculated by the following formula (1):
thus, in order to enable the visual surface of the multimedia element to be seen from the user visual angle at the center of the panorama sphere, the embodiment of the invention can accurately calculate the accurate moving distance of the first positioning corresponding to the multimedia element on the surface of the panorama sphere in the direction of the center of the panorama sphere, thereby accurately calculating the first positioning after corresponding displacement, namely the accurate position of target positioning, and the target positioning can enable the visual surface of the multimedia element to be seen from the user visual angle completely in the panorama sphere. The target location is uniquely determined based on the size information of the multimedia elements and the first location corresponding to the multimedia elements, and the problems that in the prior art, when the multimedia elements are directly and manually added to the spatial position in the panoramic ball, the adding position of the multimedia elements does not meet the expected requirement due to large error and low accuracy of manual addition, and the position change fault tolerance rate is low are solved.
104, determining the target location of the multimedia elements in the inner space of the panoramic ball according to the first location and the distance D;
as shown in fig. 4, the first location C1(x ', y ', z ') is translated toward the center M by a distance D to reach the target location C.
And 105, adding the multimedia elements to the target positioning position, wherein the visual surface of the multimedia elements has a preselected datum point N, the datum point N is overlapped with the target positioning C, and the connecting line of the sphere center and the target positioning is vertical to the visual surface.
As shown in fig. 3 and 4, the default multimedia element is perpendicular to the user's line of sight, i.e. the line between the reference point N and the center M of the panorama ball needs to be perpendicular to the visible surface (the visible surface of the tile) to which the multimedia element is added. Then theoretically the reference point N of the multimedia element 11 should be positioned to the first position C1(x ', y ', z ') of the surface of the panorama sphere, then the position of the multimedia element 11 is substantially determined; however, in practical applications, due to the particularity of the panorama ball, for the playing of the panorama video, it is equivalent to that the user's eyes are located at the center M of the panorama ball, so that the rectangular panorama video image played by the rectangular player is projected inward to the surface of the panorama ball, that is, the user can only see the content inside the panorama ball, and the content outside the panorama ball is not visible. Then, the visual surface of the multimedia element dynamically added according to the above positioning manner should be in an circumscribed relation with the panorama sphere, and after the inward movement distance D of the multimedia element is determined, it can be ensured that the visual surface of the added multimedia element 11' is completely placed inside the panorama sphere. The geometric formula of the basic formula (1) can calculate the first positioning inward movement distance D corresponding to the multimedia element 11, thereby ensuring that each point on the visual surface of the multimedia element enters the panoramic ball and can be seen by the user.
By means of the technical scheme of the embodiment of the invention, the multimedia elements are received at any time and the target positioning of the multimedia elements in the panoramic ball is automatically determined, so that the configuration time for configuring the multimedia elements for the panoramic video is not influenced by the edition sending period of a client, and the support capability of temporary operation activities is improved; the target location capable of putting down the multimedia elements can be accurately calculated according to the first location of the multimedia elements on the surface of the panoramic ball and the translation distance of the first location to the center of the ball, namely, the location information capable of putting down the multimedia elements is calculated in advance, and then the multimedia elements are configured, so that the problem of low fault tolerance rate of position change of the multimedia elements caused by manually and directly configuring the multimedia elements in the panoramic ball is solved, and the fault tolerance rate of the position change is improved; in addition, the invention can dynamically configure the multimedia elements by receiving the multimedia elements at any time and determining the target positioning of the multimedia elements in the panoramic ball, and can adjust the target positioning of the multimedia elements on line at any time according to the actual situation.
When the target positioning of the added multimedia elements needs to be adjusted, only the second positioning of the multimedia elements in the two-dimensional rectangular panoramic video image needs to be received again; or the first positioning corresponding to the multimedia element on the panoramic ball is received again, so that the target positioning of the multimedia element added in the panoramic video of the panoramic ball in the panoramic video image can be adjusted at any time.
Moreover, the multimedia elements to be added and the first positioning information or the second positioning information of the multimedia elements can be received at any time, so that the multimedia elements can be dynamically configured on the target positioning in the panoramic ball of the internally rendered panoramic video, and the requirement of dynamically configuring the multimedia elements can be met.
In the above embodiment, it is described by taking the multimedia element as a 2D patch as an example, when the multimedia element is a 3D model, for example, when the multimedia element is a convex polyhedron of 3D, and the visible surfaces are one, the method is similar to the 2D patch, and details are not repeated here.
And when the visual face of the convex polyhedron of 3D is a plurality of, that is, the user needs to see the multimedia element of 3D effect, then can all carry out the processing of above-mentioned scheme to every visual face of this multimedia element, and, in order to simulate the stereoscopic effect, the camera of installing in the centre of sphere position department of panorama ball then can be 2, simulates people's left eye and right eye respectively, like this, when the user receives the image data that 2 cameras were gathered through wearing equipment, just can see the effect of 3D multimedia element.
Alternatively, the user may, in one embodiment,
before performing S31, the method according to an embodiment of the present invention may further include:
and calculating the scaling M of the multimedia elements according to the size information of the panorama ball and the size information of the multimedia elements.
The technical method for calculating the scaling M is a known technique, and is not described herein again.
Accordingly, after S312, the method according to the embodiment of the present invention may further include:
reducing the farthest distance L according to the scaling ratio M to obtain a scaling distance M x L;
reducing the multimedia elements according to the scaling M to obtain scaled multimedia elements;
correspondingly, when S313 is executed, a distance D for translating the first positioning toward the center of the panorama sphere is calculated according to the zoom distance M × L and the sphere radius R;
that is to say that the first and second electrodes,
accordingly, when step 105 is performed, the zoom multimedia element is added to the target location.
Like this, when the size of multimedia element and the size of panorama ball are not in a dimension, can be through the size with multimedia element zoom when the size of panorama ball is unanimous to the size of panorama ball, add multimedia element to the target location department of panorama ball again to when facing the size of multimedia element and the size of panorama ball are not unified, can in time unify the processing, and then realize the accurate dynamic configuration to multimedia element.
It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.
Corresponding to the method provided by the foregoing embodiment of the present invention, referring to fig. 5, a block diagram of a structure of an embodiment of a processing apparatus for panoramic video according to the present invention is shown, which may specifically include the following modules:
a receiving module 51, configured to receive multimedia elements;
a first determining module 52, configured to determine a first position on a surface of a panorama sphere corresponding to the multimedia element, where a rectangular panorama video image is rendered on an inner surface of the panorama sphere;
a second determining module 53, configured to determine a distance D for translating the first positioning toward the center of the panoramic ball;
a third determining module 54, configured to determine, according to the first location and the distance D, a target location of the multimedia element in the inner space of the panorama ball;
an adding module 55, configured to add the multimedia element to the target location, where a visual surface of the multimedia element has a pre-selected reference point, the reference point overlaps with the target location, and a connection line between the center of sphere and the target location is perpendicular to the visual surface.
Optionally, the first determining module 52 includes:
the first receiving submodule is used for receiving the two-dimensional coordinate information of the rectangular panoramic video image;
the second receiving submodule is used for receiving second positioning corresponding to the multimedia elements in the rectangular panoramic video image;
the rendering submodule is used for rendering the rectangular panoramic video image to the inner surface of the panoramic ball according to the mapping relation between the preset plane coordinates and the preset spherical coordinates and the coordinate information of the rectangular panoramic video image;
and the first determining submodule is used for determining the corresponding first positioning of the second positioning on the surface of the panoramic ball according to the mapping relation.
Optionally, the second determining module 53 includes:
and the second determining submodule is used for determining the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the size information of the multimedia elements and the sphere radius R of the panoramic sphere.
Optionally, the second determining sub-module includes:
the determining unit is used for determining a reference point on a visual surface of the multimedia element according to the size information of the multimedia element;
the first calculating unit is used for calculating the farthest distance L between the reference point and each edge point on the visible surface;
and the second calculating unit is used for calculating the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the farthest distance L and the sphere radius R.
Optionally, the determining unit includes:
the first determining subunit is used for determining the geometric center of the visual surface of the multimedia element according to the size information of the multimedia element;
a second determining subunit, configured to determine the geometric center as a reference point on the visual surface.
Optionally, the apparatus further comprises:
the calculation module is used for calculating the scaling M of the multimedia elements according to the size information of the panoramic ball and the size information of the multimedia elements;
a first reducing module, configured to reduce the farthest distance L according to the scaling ratio M to obtain a scaling distance M × L;
the second reducing module is used for reducing the multimedia elements according to the scaling M to obtain scaled multimedia elements;
the second calculation unit includes:
a calculating subunit, configured to calculate, according to the zoom distance M × L and the sphere radius R, a distance D for translating the first positioning toward the center of the panoramic sphere;
the adding module 55 includes:
an adding sub-module for adding the scaled multimedia elements to the target location.
Optionally, the second calculating unit is configured to calculate the distance D according to the following formula:
for the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The method for processing a panoramic video and the device for processing a panoramic video provided by the present invention are described in detail above, and a specific example is applied in the text to explain the principle and the implementation of the present invention, and the description of the above embodiment is only used to help understanding the method of the present invention and the core idea thereof; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (12)
1. A method for processing panoramic video, comprising:
receiving a multimedia element;
determining a first location on the surface of a panorama sphere corresponding to the multimedia elements, wherein a rectangular panorama video image is rendered on the inner surface of the panorama sphere;
determining a distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the size information of the multimedia elements and the sphere radius R of the panoramic sphere;
determining the target location of the multimedia elements in the inner space of the panorama ball according to the first location and the distance D;
and adding the multimedia elements to the target positioning part, wherein the visual surface of the multimedia elements has a preselected datum point, the datum point is overlapped with the target positioning position, and the connecting line of the sphere center and the target positioning is vertical to the visual surface.
2. The method of claim 1, wherein determining a first location on the surface of the panoramic ball corresponding to the multimedia element comprises:
receiving two-dimensional coordinate information of the rectangular panoramic video image;
receiving a second position corresponding to the multimedia element in the rectangular panoramic video image;
rendering the rectangular panoramic video image to the inner surface of the panoramic ball according to a preset mapping relation between the plane coordinates and the spherical coordinates and the coordinate information of the rectangular panoramic video image;
and determining the corresponding first positioning of the second positioning on the surface of the panoramic ball according to the mapping relation.
3. The method according to claim 1, wherein said determining a distance D for translating said first position towards a center of said panorama sphere according to said multimedia element size information and a sphere radius R of said panorama sphere comprises:
determining a reference point on a visual surface of the multimedia element according to the size information of the multimedia element;
calculating the farthest distance L between the reference point and each edge point on the visible surface;
and calculating the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the farthest distance L and the sphere radius R.
4. The method of claim 3, wherein the determining the reference point on the visual surface of the multimedia element according to the size information of the multimedia element comprises:
determining the geometric center of the visual surface of the multimedia element according to the size information of the multimedia element;
determining the geometric center as a reference point on the visual plane.
5. The method of claim 3,
before calculating the distance D for translating the first location toward the center of the panorama sphere according to the farthest distance L and the sphere radius R, the method further comprises:
calculating the scaling M of the multimedia elements according to the size information of the panorama ball and the size information of the multimedia elements;
after the calculating the farthest distance L between the reference point and each edge point on the visible surface, the method further includes:
reducing the farthest distance L according to the scaling ratio M to obtain a scaling distance M x L;
reducing the multimedia elements according to the scaling M to obtain scaled multimedia elements;
calculating a distance D for translating the first location to a direction of a center of the panorama sphere according to the farthest distance L and the sphere radius R, including:
calculating a distance D for translating the first positioning to the spherical center direction of the panoramic sphere according to the zooming distance M and the spherical radius R;
the adding the multimedia element to the target location comprises:
adding the scaled multimedia elements to the target location.
6. The method of claim 3, wherein said calculating a distance D for translating said first location toward a center of said panoramic ball based on said farthest distance L and said radius R comprises:
7. an apparatus for processing a panoramic video, comprising:
the receiving module is used for receiving the multimedia elements;
the first determining module is used for determining first positioning corresponding to the multimedia elements on the surface of the panoramic ball, wherein a rectangular panoramic video image is rendered on the inner surface of the panoramic ball;
a second determining module, configured to determine, according to the size information of the multimedia element and the sphere radius R of the panoramic sphere, a distance D for translating the first location toward the center of the panoramic sphere;
a third determining module, configured to determine, according to the first location and the distance D, a target location of the multimedia element in an internal space of the panorama ball;
and the adding module is used for adding the multimedia elements to the target positioning position, wherein the visual surface of the multimedia elements has a preselected datum point, the datum point is overlapped with the target positioning position, and the connecting line of the sphere center and the target positioning is vertical to the visual surface.
8. The apparatus of claim 7, wherein the first determining module comprises:
the first receiving submodule is used for receiving the two-dimensional coordinate information of the rectangular panoramic video image;
the second receiving submodule is used for receiving second positioning corresponding to the multimedia elements in the rectangular panoramic video image;
the rendering submodule is used for rendering the rectangular panoramic video image to the inner surface of the panoramic ball according to the mapping relation between the preset plane coordinates and the preset spherical coordinates and the coordinate information of the rectangular panoramic video image;
and the first determining submodule is used for determining the corresponding first positioning of the second positioning on the surface of the panoramic ball according to the mapping relation.
9. The apparatus of claim 7, wherein the second determining module comprises:
the determining unit is used for determining a reference point on a visual surface of the multimedia element according to the size information of the multimedia element;
the first calculating unit is used for calculating the farthest distance L between the reference point and each edge point on the visible surface;
and the second calculating unit is used for calculating the distance D for translating the first positioning to the sphere center direction of the panoramic sphere according to the farthest distance L and the sphere radius R.
10. The apparatus of claim 9, wherein the determining unit comprises:
the first determining subunit is used for determining the geometric center of the visual surface of the multimedia element according to the size information of the multimedia element;
a second determining subunit, configured to determine the geometric center as a reference point on the visual surface.
11. The apparatus of claim 9, further comprising:
the calculation module is used for calculating the scaling M of the multimedia elements according to the size information of the panoramic ball and the size information of the multimedia elements;
a first reducing module, configured to reduce the farthest distance L according to the scaling ratio M to obtain a scaling distance M × L;
the second reducing module is used for reducing the multimedia elements according to the scaling M to obtain scaled multimedia elements;
the second calculation unit includes:
a calculating subunit, configured to calculate, according to the zoom distance M × L and the sphere radius R, a distance D for translating the first positioning toward the center of the panoramic sphere;
the adding module comprises:
an adding sub-module for adding the scaled multimedia elements to the target location.
12. The apparatus according to claim 9, wherein the second calculating unit is configured to calculate the distance D according to the following formula:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810276791.9A CN108734791B (en) | 2018-03-30 | 2018-03-30 | Panoramic video processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810276791.9A CN108734791B (en) | 2018-03-30 | 2018-03-30 | Panoramic video processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108734791A CN108734791A (en) | 2018-11-02 |
| CN108734791B true CN108734791B (en) | 2022-04-01 |
Family
ID=63940561
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810276791.9A Active CN108734791B (en) | 2018-03-30 | 2018-03-30 | Panoramic video processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108734791B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109982036A (en) * | 2019-02-20 | 2019-07-05 | 华为技术有限公司 | A kind of method, terminal and the storage medium of panoramic video data processing |
| CN111212293A (en) * | 2020-01-13 | 2020-05-29 | 聚好看科技股份有限公司 | Image processing method and display device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105979242A (en) * | 2015-11-23 | 2016-09-28 | 乐视网信息技术（北京）股份有限公司 | Video playing method and device |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100423007C (en) * | 2007-07-03 | 2008-10-01 | 浙江大学 | Modeling approachused for trans-media digital city scenic area |
| CN101877139A (en) * | 2009-04-30 | 2010-11-03 | 爱国者全景(北京)网络科技发展有限公司 | Method and system for realizing spacial hot spots in three-dimensional video panorama |
| US9277122B1 (en) * | 2015-08-13 | 2016-03-01 | Legend3D, Inc. | System and method for removing camera rotation from a panoramic video |
| CN105245838A (en) * | 2015-09-29 | 2016-01-13 | 成都虚拟世界科技有限公司 | Panoramic video playing method and player |
| HK1212857A2 (en) * | 2016-03-21 | 2016-06-17 | 有限公司 | Video display method and system for realistic scenes |
| US10249019B2 (en) * | 2016-05-06 | 2019-04-02 | Mediatek Inc. | Method and apparatus for mapping omnidirectional image to a layout output format |
| CN106101741B (en) * | 2016-07-26 | 2020-12-15 | 武汉斗鱼网络科技有限公司 | Method and system for watching panoramic video on network video live broadcast platform |
| CN106534670B (en) * | 2016-10-25 | 2019-07-09 | 成都通甲优博科技有限责任公司 | It is a kind of based on the panoramic video generation method for connecting firmly fish eye lens video camera group |
-
2018
- 2018-03-30 CN CN201810276791.9A patent/CN108734791B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105979242A (en) * | 2015-11-23 | 2016-09-28 | 乐视网信息技术（北京）股份有限公司 | Video playing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108734791A (en) | 2018-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11869205B1 (en) | Techniques for determining a three-dimensional representation of a surface of an object from a set of images | |
| US11170561B1 (en) | Techniques for determining a three-dimensional textured representation of a surface of an object from a set of images with varying formats | |
| US11076142B2 (en) | Real-time aliasing rendering method for 3D VR video and virtual three-dimensional scene | |
| US9298070B2 (en) | Participant collaboration on a displayed version of an object | |
| US11663733B2 (en) | Depth determination for images captured with a moving camera and representing moving features | |
| CN109584295A (en) | The method, apparatus and system of automatic marking are carried out to target object in image | |
| US20160180593A1 (en) | Wearable device-based augmented reality method and system | |
| CN107316273B (en) | Panoramic image acquisition device and acquisition method | |
| TWI547901B (en) | Simulating stereoscopic image display method and display device | |
| US20170195664A1 (en) | Three-dimensional viewing angle selecting method and apparatus | |
| US9467620B2 (en) | Synthetic camera lenses | |
| US11244423B2 (en) | Image processing apparatus, image processing method, and storage medium for generating a panoramic image | |
| CN108961423B (en) | Virtual information processing method, device, equipment and storage medium | |
| US10764493B2 (en) | Display method and electronic device | |
| WO2017128887A1 (en) | Method and system for corrected 3d display of panoramic image and device | |
| US20150326847A1 (en) | Method and system for capturing a 3d image using single camera | |
| Suenaga et al. | A practical implementation of free viewpoint video system for soccer games | |
| CN108734791B (en) | Panoramic video processing method and device | |
| US20170104982A1 (en) | Presentation of a virtual reality scene from a series of images | |
| US11212502B2 (en) | Method of modifying an image on a computational device | |
| CN113870213A (en) | Image display method, image display device, storage medium, and electronic apparatus | |
| US20230342973A1 (en) | Image processing method and apparatus, device, storage medium, and computer program product | |
| CN110060349B (en) | Method for expanding field angle of augmented reality head-mounted display equipment | |
| CN116708862A (en) | Virtual background generation method for live broadcasting room, computer equipment and storage medium | |
| CN108510433B (en) | Space display method and device and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |