CN114567742A - Panoramic video transmission method and device and storage medium - Google Patents
Panoramic video transmission method and device and storage medium Download PDFInfo
- Publication number
- CN114567742A CN114567742A CN202111616117.9A CN202111616117A CN114567742A CN 114567742 A CN114567742 A CN 114567742A CN 202111616117 A CN202111616117 A CN 202111616117A CN 114567742 A CN114567742 A CN 114567742A
- Authority
- CN
- China
- Prior art keywords
- panoramic video
- projection image
- video frame
- client
- sin
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0117—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving conversion of the spatial resolution of the incoming video signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
- H04N19/88—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
Abstract
The invention is suitable for the technical field of image processing, and provides a panoramic video transmission method, a panoramic video transmission device and a panoramic video storage medium, wherein the method comprises the following steps: the method comprises the steps of acquiring a first projection image of an interested area of a client user based on a current panoramic video frame, encoding the first projection image and the current panoramic video frame with reduced resolution ratio, and then sending the encoded first projection image and the current panoramic video frame to the client, thereby reducing network resource overhead and transmission delay in the panoramic video transmission process and ensuring the definition of the interested area.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a panoramic video transmission method, a panoramic video transmission device and a panoramic video storage medium.
Background
The panoramic video is a new data form, and compared with the traditional video, the panoramic video has the characteristics of large visual field, high resolution, large data volume and the like. In the prior art, a spliced panoramic video or an original fisheye video is usually encoded into a standard data stream format for data transmission, and a client decodes a data stream and then watches an interested area, but when transmitting the data with high resolution and high image quality, a large network resource overhead is generated, and a certain degree of network transmission delay is generated.
Disclosure of Invention
The invention aims to provide a panoramic video transmission method, a panoramic video transmission device and a storage medium, and aims to solve the problems of network resource waste and transmission delay in the panoramic video transmission process in the prior art.
In one aspect, the present invention provides a method for transmitting a panoramic video, including the following steps:
acquiring a first projection image of an interested area of a client user based on a current panoramic video frame;
and coding the first projection image and the current panoramic video frame with reduced resolution and then sending the coded first projection image and the current panoramic video frame to a client.
Optionally, the step of obtaining a first projection image of a region of interest of the client user based on the current panoramic video frame includes:
generating a longitude and latitude mapping table according to calibration parameters preset by the panoramic shooting device;
acquiring first coordinate information of each pixel of a second projection image in a target three-dimensional coordinate system, wherein the second projection image is a projection image obtained when a view is taken from the front of the panoramic shooting device according to the screen size of the client;
establishing a rotation matrix according to the Euler angles for characterizing the region of interest;
rotating each first coordinate information according to the rotation matrix to obtain each second coordinate information;
performing longitude and latitude mapping on the second coordinate information to obtain longitude and latitude information;
performing coordinate mapping on the longitude and latitude information by using the longitude and latitude mapping table to obtain third coordinate information of the region of interest under the current panoramic video frame;
and acquiring pixel values corresponding to the third coordinate information according to the current panoramic video frame to obtain the first projection image.
Optionally, the target three-dimensional coordinate system is a right-hand coordinate system, the right-hand coordinate system takes a point where the panoramic shooting device is located as an origin, and the first coordinate information is calculated as follows:
x＝(i-rayW×0.5)
y＝(j-rayH×0.5)
z＝tan(0.5π-0.5×fov×π/180°)×rayW×0.5
wherein (i, j) represents a two-dimensional coordinate of the second projection image in a two-dimensional projection coordinate system, i and j are positive numbers, the two-dimensional projection coordinate system takes the upper left corner of the image displayed on the screen of the client as an origin, the X axis and the Y axis of the two-dimensional projection coordinate system are respectively in the same direction as the X axis and the Y axis of the right-hand coordinate system, (X, Y, z) represents a three-dimensional coordinate of a two-dimensional coordinate point (i, j) in the right-hand coordinate system, fov represents a field angle of the image displayed on the screen of the client, rayW represents a width of the screen of the client, and rayH represents a height of the screen of the client;
the rotation matrix is calculated as follows:
r(0,0)＝cos(pitch)*cos(yaw)
r(0,1)＝sin(roll)*sin(pitch)*cos(yaw)-cos(roll)*sin(yaw)
r(0,2)＝cos(roll)*sin(pitch)*cos(yaw)+sin(roll)*sin(yaw)
r(1,0)＝cos(pitch)*sin(yaw)
r(1,1)＝sin(roll)*sin(pitch)*sin(yaw)+cos(roll)*cos(yaw)
r(1,2)＝cos(roll)*sin(pitch)*sin(yaw)-sin(roll)*cos(yaw)
r(2,0)＝-sin(pitch)
r(2,1)＝sin(roll)*cos(pitch)
r(2,2)＝cos(roll)*cos(pitch)
wherein R represents the rotation matrix, yaw, pitch, and roll represent yaw, pitch, and roll, respectively, of the region of interest;
the second coordinate information is calculated as follows:
x0＝x×r(0,0)+y×r(0,1)+z×r(0,2)
y0＝x×r(1,0)+y×r(1,1)+z×r(1,2)
z0＝x×r(2,0)+y×r(2,1)+z×r(2,2)
wherein (x0, y0, z0) represents three-dimensional coordinates of a projected point in the second coordinate information;
the longitude and latitude information calculation mode is as follows:
wherein, theta1Denotes longitude, θ2Indicating the latitude.
Optionally, the step of encoding the first projection image and the current panoramic video frame with reduced resolution and then sending the encoded first projection image and the current panoramic video frame to a client includes:
acquiring two fisheye images of a current panoramic video frame;
and combining the two fisheye images with reduced resolution with the first projection image, and encoding the combined images and then sending the encoded images to the client.
Optionally, the step of encoding the first projection image and the current panoramic video frame with reduced resolution and then sending the encoded first projection image and the current panoramic video frame to a client includes:
acquiring a plane projection image of a current panoramic video frame;
and combining the plane projection image with the reduced resolution with the first projection image, and encoding the combined image and then sending the encoded image to the client.
Optionally, the method further comprises:
and determining the resolution of the current panoramic video frame after the resolution is reduced according to the acquired target environment information.
In another aspect, the present invention provides an apparatus for transmitting a panoramic video, the apparatus comprising:
the system comprises a projected image acquisition unit, a first display unit and a second display unit, wherein the projected image acquisition unit is used for acquiring a first projected image of an interested area of a client user based on a current panoramic video frame;
and the video sending unit is used for coding the first projection image and the current panoramic video frame with reduced resolution and then sending the coded first projection image and the current panoramic video frame to a client.
Optionally, the projection image acquisition unit further includes:
the mapping table generating unit is used for generating a longitude and latitude mapping table according to calibration parameters preset by the panoramic shooting device;
a first information acquisition unit configured to acquire first coordinate information of each pixel of a second projection image in a target three-dimensional coordinate system, the second projection image being a projection image obtained when framing from the front of the panoramic shooting device according to a screen size of the client;
the rotation matrix establishing unit is used for establishing a rotation matrix according to the Euler angle for representing the region of interest;
the second information acquisition unit is used for rotating each piece of first coordinate information according to the rotation matrix to obtain each piece of second coordinate information; and
the longitude and latitude mapping unit is used for carrying out longitude and latitude mapping on the second coordinate information to obtain the longitude and latitude information;
the third information acquisition unit is used for carrying out coordinate mapping on the longitude and latitude information by using the longitude and latitude mapping table to obtain third coordinate information of the region of interest under the current panoramic video frame; and
and the projection image generation unit is used for acquiring pixel values corresponding to the third coordinate information according to the current panoramic video frame to obtain the first projection image.
In another aspect, the present invention further provides a panoramic shooting apparatus, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method when executing the computer program.
In another aspect, the present invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method as described above.
According to the method and the device, the first projection image of the region of interest of the client user is acquired based on the current panoramic video frame, and the first projection image and the current panoramic video frame with reduced resolution are coded and then sent to the client, so that the network resource overhead and transmission delay in the panoramic video transmission process are reduced, and the definition of the region of interest is ensured.
Drawings
Fig. 1A is a flowchart illustrating an implementation of a method for transmitting a panoramic video according to an embodiment of the present invention;
FIG. 1B is a schematic diagram of a right-hand coordinate system and a two-dimensional projection coordinate system according to an embodiment of the invention;
FIG. 1C is a diagram illustrating an example of displaying effect of an encoded image according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a panoramic video transmission apparatus according to a second embodiment of the present invention; and
fig. 3 is a schematic structural diagram of a panoramic shooting apparatus according to a third embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The following detailed description of specific implementations of the present invention is provided in conjunction with specific embodiments:
the first embodiment is as follows:
fig. 1A shows an implementation flow of a panoramic video transmission method according to a first embodiment of the present invention, and for convenience of description, only the relevant portions related to the first embodiment of the present invention are shown, which is detailed as follows:
in step S101, a first projection image of a region of interest of a client user is acquired based on a current panoramic video frame.
The embodiment of the invention is suitable for a panoramic shooting device, the panoramic shooting device can be a monitoring camera, a mobile phone, a tablet computer, wearable equipment, vehicle-mounted equipment, a notebook computer and the like, and the embodiment of the invention does not limit the specific type of the panoramic shooting device. The client can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a notebook computer, a super mobile personal computer, a server and other video playing devices.
In the embodiment of the invention, the client acquires the region of interest of the user and sends the acquired region of interest of the client user to the panoramic shooting device. The client side can be used for obtaining the interested areas input by the client side user, predicting the interested areas of the client side user in the next period in each interested area calculation period, and sending the predicted interested areas to the panoramic shooting device.
And the panoramic shooting device performs projection calculation on the current panoramic video frame according to the acquired region of interest of the client user to obtain a first projection image of the region of interest. The first projection image is generally a projection image of a spherical image of the region of interest in the current panoramic video frame in a straight-line projection mode.
Upon acquiring a first projection image of a region of interest of a client user based on a current panoramic video frame, optionally, generating a longitude and latitude mapping table according to calibration parameters preset by the panoramic shooting device, acquiring first coordinate information of each pixel of the second projection image in a target three-dimensional coordinate system, and then, establishing a rotation matrix according to the Euler angle for representing the region of interest, rotating each first coordinate information according to the rotation matrix to obtain each second coordinate information, performing longitude and latitude mapping on each second coordinate information to obtain each longitude and latitude information, performing coordinate mapping on each longitude and latitude information by using a preset longitude and latitude mapping table to obtain each third coordinate information of the interested area under the current panoramic video frame, and acquiring pixel values corresponding to the third coordinate information according to the current panoramic video frame to obtain a first projection image, so that the acquisition of the region-of-interest image is realized through coordinate conversion. The second projection image is a projection image obtained when a view is taken from the front of the panoramic shooting device according to the screen size of the client, the horizontal and vertical coordinates of the longitude and latitude mapping table respectively represent the longitude and latitude of a spherical surface with the panoramic shooting device as the center, the value in the longitude and latitude mapping table is the specific coordinate of the panoramic video frame, the calibration parameters can include internal and external parameters and distortion parameters, and the internal and external parameters can include parameters such as a field angle and a focal length.
As shown in fig. 1B, alternatively, the target three-dimensional coordinate system is a right-hand coordinate system, the right-hand coordinate system uses a point where the panoramic photographing apparatus is located as an origin, the right-front side of the panoramic photographing apparatus is a Z-axis direction, Xd, Yd, and Zd in fig. 1B are three axes of the right-hand coordinate system, and Od is an origin of the right-hand coordinate system, i.e., a center of a spherical model, the second projection image is obtained when viewing from the front side of the panoramic photographing apparatus according to the screen size of the client in the straight-line projection mode, two-dimensional coordinates of a projection point of the second projection image in the two-dimensional projection coordinate system are represented by (i, j), an X axis and a Y axis of the two-dimensional projection coordinate system are respectively in the same direction as the X axis and the Y axis of the right-hand coordinate system, the two-dimensional projection coordinate system uses an upper left corner of an image displayed on the client screen as an origin, Xp and Yp are two axes of the two-dimensional projection coordinate system in fig. 1B, and Od is a center of a circle, the first coordinate information is calculated as follows:
x＝(i-rayW×0.5)
y＝(j-rayH×0.5)
z＝tan(0.5π-0.5×fov×π/180°)×rayW×0.5
where i and j are positive numbers, (x, y, z) represents a three-dimensional coordinate of the two-dimensional coordinate point (i, j) in the right-hand coordinate system, fov represents a field angle of an image displayed on the client screen, rayW represents a width of the client screen, and rayH represents a height of the client screen.
After the second coordinate information is obtained, receiving the region of interest sent by the client, and calculating a rotation matrix according to the Euler angle of the region of interest, wherein the calculation mode of the rotation matrix is as follows:
r(0,0)＝cos(pitch)*cos(yaw)
r(0,1)＝sin(roll)*sin(pitch)*cos(yaw)-cos(roll)*sin(yaw)
r(0,2)＝cos(roll)*sin(pitch)*cos(yaw)+sin(roll)*sin(yaw)
r(1,0)＝cos(pitch)*sin(yaw)
r(1,1)＝sin(roll)*sin(pitch)*sin(yaw)+cos(roll)*cos(yaw)
r(1,2)＝cos(roll)*sin(pitch)*sin(yaw)-sin(roll)*cos(yaw)
r(2,0)＝-sin(pitch)
r(2,1)＝sin(roll)*cos(pitch)
r(2,2)＝cos(roll)*cos(pitch)
where R denotes a rotation matrix, yaw, pitch and roll denote the yaw, pitch and roll angles, respectively, of the region of interest.
After the rotation matrix is calculated, each piece of first coordinate information is rotated according to the rotation matrix to obtain each piece of second coordinate information, and the calculation mode of the second coordinate information is as follows:
x0＝x×r(0,0)+y×r(0,1)+z×r(0,2)
y0＝x×r(1,0)+y×r(1,1)+z×r(1,2)
z0＝x×r(2,0)+y×r(2,1)+z×r(2,2)
where (x0, y0, z0) represents the three-dimensional coordinates of the projected point in the second coordinate information.
After the second coordinate information is obtained, longitude and latitude mapping can be performed on each second coordinate information to obtain each longitude and latitude information, wherein the longitude and latitude information calculation mode is as follows:
wherein, theta1Denotes longitude, θ2Indicating the latitude.
After the longitude and latitude information is obtained, coordinate mapping can be carried out on the longitude and latitude information according to the longitude and latitude mapping table to obtain third coordinate information of the interesting region under the current panoramic video frame, finally, pixel values corresponding to the third coordinate information are obtained according to the current panoramic video frame, and the first projection image can be obtained based on the pixel values corresponding to the third coordinate information.
In step S102, the first projection image and the current panoramic video frame with reduced resolution are encoded and transmitted to the client.
In the embodiment of the invention, optionally, two fisheye images of the current panoramic video frame are obtained, the two fisheye images with reduced resolution and the first projection image are combined, and the combined image is encoded and then sent to the client, so that the user can be ensured to observe global information from the two fisheye images while the image definition of the region of interest of the user is ensured. Fig. 1C is an exemplary diagram of a display effect obtained by combining the two fish-eye images with reduced resolutions and the first projection image.
Optionally, a plane projection image of the current panoramic video frame is obtained, the plane projection image with the reduced resolution is combined with the first projection image, and the combined image is encoded and then sent to the client, so that the user can be ensured to observe global information from the plane projection image while ensuring the image definition of the region of interest of the user.
Optionally, a display mode selected by a user of the client is obtained, if the display mode is a fisheye display mode, the two fisheye images with reduced resolution and the first projection image are combined, the combined image is encoded and then sent to the client, and if the display mode is a tiled display mode, the planar projection image with reduced resolution and the first projection image are combined, the combined image is encoded and then sent to the client, so that the display mode of the panoramic video frame with reduced resolution is enriched.
The resolution of the current panoramic video frame after the resolution is reduced can be a fixed resolution, and can also be flexibly set by a client user, and optionally, the resolution of the current panoramic video frame after the resolution is reduced is determined according to the obtained target environment information, so that the flexibility of adjusting the resolution of the current panoramic video frame is improved. The target environment information may include a network environment of the client and/or a software and hardware environment, where the network environment may include a network bandwidth, a network transmission delay, and the like, and the software and hardware environment may include a data processing capability of the client, an energy consumption of the client, a minimum resolution specified by a user of the client, and the like.
In the embodiment of the invention, the first projection image of the interested area of the client user is obtained based on the current panoramic video frame, and the first projection image and the current panoramic video frame with reduced resolution are coded and then sent to the client, so that the network resource overhead and transmission delay in the panoramic video transmission process are reduced, and the definition of the interested area is ensured.
Example two:
fig. 2 shows a structure of a panoramic video transmission apparatus according to a second embodiment of the present invention, and for convenience of description, only the portions related to the second embodiment of the present invention are shown, which includes:
the system comprises a projected image acquisition unit, a first display unit and a second display unit, wherein the projected image acquisition unit is used for acquiring a first projected image of an interested area of a client user based on a current panoramic video frame; and
and the video sending unit is used for coding the first projection image and the current panoramic video frame with the reduced resolution and then sending the coded first projection image and the current panoramic video frame to the client.
Optionally, the projection image acquisition unit further includes:
the mapping table generating unit is used for generating a longitude and latitude mapping table according to calibration parameters preset by the panoramic shooting device;
a first information acquisition unit configured to acquire first coordinate information of each pixel of a second projection image in a target three-dimensional coordinate system, the second projection image being a projection image obtained when framing from the front of the panoramic shooting device according to a screen size of the client;
the rotation matrix establishing unit is used for establishing a rotation matrix according to the Euler angle used for representing the region of interest;
the second information acquisition unit is used for rotating each piece of first coordinate information according to the rotation matrix to obtain each piece of second coordinate information;
the longitude and latitude mapping unit is used for carrying out longitude and latitude mapping on each second coordinate information to obtain each longitude and latitude information;
the third information acquisition unit is used for carrying out coordinate mapping on the longitude and latitude information by using a longitude and latitude mapping table to obtain third coordinate information of the interested area under the current panoramic video frame; and
and the projection image generation unit is used for acquiring pixel values corresponding to the third coordinate information according to the current panoramic video frame to obtain a first projection image.
In the embodiment of the present invention, each unit of the panoramic video transmission apparatus may be implemented by a corresponding hardware or software unit, and each unit may be an independent software or hardware unit, or may be integrated into a software or hardware unit, which is not limited herein. For the specific implementation of each unit of the panoramic video transmission apparatus, reference may be made to the description of the foregoing method embodiment, and details are not repeated here.
Example three:
fig. 3 illustrates a configuration of a panoramic imaging apparatus according to a third embodiment of the present invention, and for convenience of description, only the portions related to the third embodiment of the present invention are illustrated.
The panorama photographing apparatus 3 of an embodiment of the present invention includes a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. The processor 30, when executing the computer program 32, implements the steps in the above-described method embodiments, for example, the steps S101 to S102 shown in fig. 1A. Alternatively, the processor 30, when executing the computer program 32, implements the functions of the units in the device embodiments described above, for example, the functions of the units 21 to 22 shown in fig. 2.
In the embodiment of the invention, the first projection image of the interested area of the client user is obtained based on the current panoramic video frame, and the first projection image and the current panoramic video frame with reduced resolution ratio are coded and then sent to the client, thereby reducing the network resource overhead and transmission delay in the panoramic video transmission process and ensuring the definition of the interested area.
Example four:
in an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps in the above-described method embodiments, e.g., steps S101 to S102 shown in fig. 1A. Alternatively, the computer program realizes the functions of the units in the above-described device embodiments, for example, the functions of the units 21 to 22 shown in fig. 2, when executed by the processor.
In the embodiment of the invention, the first projection image of the interested area of the client user is obtained based on the current panoramic video frame, and the first projection image and the current panoramic video frame with reduced resolution are coded and then sent to the client, so that the network resource overhead and transmission delay in the panoramic video transmission process are reduced, and the definition of the interested area is ensured.
The computer readable storage medium of the embodiments of the present invention may include any entity or device capable of carrying computer program code, a recording medium, such as a ROM/RAM, a magnetic disk, an optical disk, a flash memory, or the like.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A method for transmitting panoramic video, the method comprising the steps of:
acquiring a first projection image of an interested area of a client user based on a current panoramic video frame;
and coding the first projection image and the current panoramic video frame with reduced resolution and then sending the coded first projection image and the current panoramic video frame to a client.
2. The method of claim 1, wherein the step of obtaining a first projection image of a region of interest of a client user based on a current panoramic video frame comprises:
generating a longitude and latitude mapping table according to calibration parameters preset by the panoramic shooting device;
acquiring first coordinate information of each pixel of a second projection image in a target three-dimensional coordinate system, wherein the second projection image is a projection image obtained when a view is taken from the front of the panoramic shooting device according to the screen size of the client;
establishing a rotation matrix according to the Euler angles for characterizing the region of interest;
rotating each first coordinate information according to the rotation matrix to obtain each second coordinate information;
performing longitude and latitude mapping on the second coordinate information to obtain longitude and latitude information;
performing coordinate mapping on the longitude and latitude information by using the longitude and latitude mapping table to obtain third coordinate information of the region of interest under the current panoramic video frame;
and acquiring pixel values corresponding to the third coordinate information according to the current panoramic video frame to obtain the first projection image.
3. The method of claim 2, wherein the target three-dimensional coordinate system is a right-hand coordinate system, the right-hand coordinate system having an origin at a point where the panorama photographing apparatus is located, and the first coordinate information is calculated as follows:
x＝(i-rayW×0.5)
y＝(j-rayH×0.5)
z＝tan(0.5π-0.5×fov×π/180°)×rayW×0.5
wherein (i, j) represents a two-dimensional coordinate of the second projection image in a two-dimensional projection coordinate system, i and j are positive numbers, the two-dimensional projection coordinate system takes the upper left corner of the image displayed on the screen of the client as an origin, the X axis and the Y axis of the two-dimensional projection coordinate system are respectively in the same direction as the X axis and the Y axis of the right-hand coordinate system, (X, Y, z) represents a three-dimensional coordinate of a two-dimensional coordinate point (i, j) in the right-hand coordinate system, fov represents a field angle of the image displayed on the screen of the client, rayW represents a width of the screen of the client, and rayH represents a height of the screen of the client;
the rotation matrix is calculated as follows:
r(0,0)＝cos(pitch)*cos(yaw)
r(0,1)＝sin(roll)*sin(pitch)*cos(yaw)-cos(roll)*sin(yaw)
r(0,2)＝cos(roll)*sin(pitch)*cos(yaw)+sin(roll)*sin(yaw)
r(1,0)＝cos(pitch)*sin(yaw)
r(1,1)＝sin(roll)*sin(pitch)*sin(yaw)+cos(roll)*cos(yaw)
r(1,2)＝cos(roll)*sin(pitch)*sin(yaw)-sin(roll)*cos(yaw)
r(2,0)＝-sin(pitch)
r(2,1)＝sin(roll)*cos(pitch)
r(2,2)＝cos(roll)*cos(pitch)
wherein R represents the rotation matrix, yaw, pitch, and roll represent yaw, pitch, and roll, respectively, of the region of interest;
the second coordinate information is calculated as follows:
x0＝x×r(0,0)+y×r(0,1)+z×r(0,2)
y0＝x×r(1,0)+y×r(1,1)+z×r(1,2)
z0＝x×r(2,0)+y×r(2,1)+z×r(2,2)
wherein (x0, y0, z0) represents three-dimensional coordinates of a projected point in the second coordinate information;
the longitude and latitude information calculation mode is as follows:
wherein, theta1Denotes longitude, θ2Indicating the latitude.
4. The method of claim 1, wherein said step of encoding said first projected image and said reduced resolution current panoramic video frame for transmission to a client comprises:
acquiring two fisheye images of a current panoramic video frame;
and combining the two fisheye images with reduced resolution with the first projection image, and encoding the combined images and then sending the encoded images to the client.
5. The method of claim 1, wherein the step of encoding the first projected image and the reduced resolution current panoramic video frame for transmission to a client comprises:
acquiring a plane projection image of a current panoramic video frame;
and combining the plane projection image with the reduced resolution with the first projection image, and encoding the combined image and then sending the encoded image to the client.
6. The method of claim 1, wherein the method further comprises:
and determining the resolution of the current panoramic video frame after the resolution is reduced according to the acquired target environment information.
7. An apparatus for transmitting a panoramic video, the apparatus comprising:
the system comprises a projected image acquisition unit, a first display unit and a second display unit, wherein the projected image acquisition unit is used for acquiring a first projected image of an interested area of a client user based on a current panoramic video frame;
and the video sending unit is used for coding the first projection image and the current panoramic video frame with reduced resolution and then sending the coded first projection image and the current panoramic video frame to a client.
8. The apparatus of claim 7, wherein the projection image acquisition unit further comprises:
the mapping table generating unit is used for generating a longitude and latitude mapping table according to calibration parameters preset by the panoramic shooting device;
a first information acquisition unit configured to acquire first coordinate information of each pixel of a second projection image in a target three-dimensional coordinate system, the second projection image being a projection image obtained when framing from the front of the panoramic shooting device according to a screen size of the client;
the rotation matrix establishing unit is used for establishing a rotation matrix according to the Euler angle for representing the region of interest;
the second information acquisition unit is used for rotating each piece of first coordinate information according to the rotation matrix to obtain each piece of second coordinate information; and
the longitude and latitude mapping unit is used for carrying out longitude and latitude mapping on the second coordinate information to obtain the longitude and latitude information;
the third information acquisition unit is used for carrying out coordinate mapping on the longitude and latitude information by using the longitude and latitude mapping table to obtain third coordinate information of the region of interest under the current panoramic video frame; and
and the projection image generation unit is used for acquiring pixel values corresponding to the third coordinate information according to the current panoramic video frame to obtain the first projection image.
9. A panoramic photographing apparatus comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111616117.9A CN114567742A (en) | 2021-12-27 | 2021-12-27 | Panoramic video transmission method and device and storage medium |
| PCT/CN2022/141766 WO2023125353A1 (en) | 2021-12-27 | 2022-12-26 | Panoramic video transmission method and apparatus, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111616117.9A CN114567742A (en) | 2021-12-27 | 2021-12-27 | Panoramic video transmission method and device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114567742A true CN114567742A (en) | 2022-05-31 |
Family
ID=81712873
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111616117.9A Pending CN114567742A (en) | 2021-12-27 | 2021-12-27 | Panoramic video transmission method and device and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114567742A (en) |
| WO (1) | WO2023125353A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023125353A1 (en) * | 2021-12-27 | 2023-07-06 | 影石创新科技股份有限公司 | Panoramic video transmission method and apparatus, and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104735464A (en) * | 2015-03-31 | 2015-06-24 | 华为技术有限公司 | Panorama video interactive transmission method, server and client end |
| CN106101847A (en) * | 2016-07-12 | 2016-11-09 | 三星电子（中国）研发中心 | The method and system of panoramic video alternating transmission |
| CN108322727A (en) * | 2018-02-28 | 2018-07-24 | 北京搜狐新媒体信息技术有限公司 | A kind of panoramic video transmission method and device |
| CN113099245B (en) * | 2021-03-04 | 2023-07-25 | 广州方硅信息技术有限公司 | Panoramic video live broadcast method, system and computer readable storage medium |
| CN114567742A (en) * | 2021-12-27 | 2022-05-31 | 影石创新科技股份有限公司 | Panoramic video transmission method and device and storage medium |
-
2021
- 2021-12-27 CN CN202111616117.9A patent/CN114567742A/en active Pending
-
2022
- 2022-12-26 WO PCT/CN2022/141766 patent/WO2023125353A1/en unknown
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023125353A1 (en) * | 2021-12-27 | 2023-07-06 | 影石创新科技股份有限公司 | Panoramic video transmission method and apparatus, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023125353A1 (en) | 2023-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10595004B2 (en) | Electronic device for generating 360-degree three-dimensional image and method therefor | |
| US10437545B2 (en) | Apparatus, system, and method for controlling display, and recording medium | |
| US7570280B2 (en) | Image providing method and device | |
| CN110073662A (en) | The suggestion viewport of panoramic video indicates | |
| JP6672315B2 (en) | Image generation device and image display control device | |
| US10855916B2 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
| US20200304751A1 (en) | Image capturing device, image capturing system, image processing method, and recording medium | |
| JP2004187298A (en) | Plotting and encoding processing of panoramic image and omnidirection image | |
| US20190289206A1 (en) | Image processing apparatus, image capturing system, image processing method, and recording medium | |
| US11818492B2 (en) | Communication management apparatus, image communication system, communication management method, and recording medium | |
| TWI615808B (en) | Image processing method for immediately producing panoramic images | |
| WO2023125353A1 (en) | Panoramic video transmission method and apparatus, and storage medium | |
| JP2019075766A (en) | Image processing apparatus, photographing system, image processing method, and program | |
| JP2018033107A (en) | Video distribution device and distribution method | |
| US11102448B2 (en) | Image capturing apparatus, image processing system, image processing method, and recording medium | |
| JP2019080174A (en) | Image processing apparatus, imaging system, communication system, image processing method, and program | |
| CN115861514A (en) | Rendering method, device and equipment of virtual panorama and storage medium | |
| CN117115267A (en) | Calibration-free image processing method and device, electronic equipment and storage medium | |
| CN109961395A (en) | The generation of depth image and display methods, device, system, readable medium | |
| CN110786017B (en) | Distributed image generation method | |
| Liao et al. | Gpu parallel computing of spherical panorama video stitching | |
| US11388336B2 (en) | Horizontal calibration method and system for panoramic image or video, and portable terminal | |
| JP7219620B2 (en) | Delivery image generation method | |
| CN114866760A (en) | Virtual reality display method, equipment, system and readable storage medium | |
| WO2012157516A1 (en) | Video presentation system, video presentation method, program, and recording medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |