CN115243054A - Panoramic video image coding transmission method, panoramic video image acquisition method, related device and system - Google Patents
Panoramic video image coding transmission method, panoramic video image acquisition method, related device and system Download PDFInfo
- Publication number
- CN115243054A CN115243054A CN202210905773.9A CN202210905773A CN115243054A CN 115243054 A CN115243054 A CN 115243054A CN 202210905773 A CN202210905773 A CN 202210905773A CN 115243054 A CN115243054 A CN 115243054A
- Authority
- CN
- China
- Prior art keywords
- panoramic video
- image
- video image
- panoramic
- center
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/167—Position within a video image, e.g. region of interest [ROI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The application relates to the technical field of video coding and decoding and live webcasting, and provides a coding transmission method, an acquisition method, a device, a system, equipment and a medium for panoramic video images. The method and the device can improve the coding transmission efficiency and simultaneously keep the image center of the receiving end consistent with the transmitting end. The method comprises the following steps: the method comprises the steps that a panoramic video sending end obtains a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders, the first panoramic video image is converted to an image center through a target object in the image, a second panoramic video image and conversion and reduction information of the second panoramic video image are obtained, region-of-interest coding is carried out on the second panoramic video image according to a region-of-interest which is arranged in the image center and matched with the target object, a third panoramic video image is obtained, the third panoramic video image and the conversion and reduction information are sent to a panoramic video receiving end, and the panoramic video receiving end decodes the third panoramic video image and then obtains a fourth panoramic video image reduced by the image center according to the conversion and reduction information.
Description
Technical Field
The present application relates to the field of video encoding and decoding and live webcasting technologies, and in particular, to a method for encoding and transmitting a panoramic video image, a method for acquiring a panoramic video image, a system for transmitting a panoramic live video, an apparatus for encoding and transmitting a panoramic video image, an apparatus for acquiring a panoramic video image, an electronic device, and a computer-readable storage medium.
Background
ROI (region of interest) coding is a video coding technology based on a region of interest, namely, a quantization parameter value is reduced for a region of interest in an image, so that more code rates are allocated to improve the picture quality, and the quantization parameter value is improved for a region of no interest, so that less code rates are allocated, and the video code rate is reduced on the premise of not losing the overall quality of the image. The region-of-interest coding is performed based on a two-dimensional image frame, the region-of-interest needs to be set according to a target object (such as a human face), and the region-of-interest cannot be too large.
With the development of network live broadcast technology, a live broadcast platform provides a panoramic live broadcast service for users to enrich the propagation form of high-quality content. For panoramic video such as 360 ° and 720 ° required to be transmitted in such a scene, in the case of rendering by using an equidistant cylindrical projection manner, the distortion is more serious the farther away from the equator position, and various distortions may occur to the shape of some objects in the image, for example, a square may be stretched into a long strip in the top edge region of the image, while with the current encoding transmission manner for panoramic video images, a large amount of redundant regions may be generated when the region of interest is set for the square, resulting in low encoding transmission efficiency for the region of interest in the panoramic video image.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a method for encoding and transmitting a panoramic video image, a method for acquiring a panoramic video image, a panoramic live video transmission system, an apparatus for encoding and transmitting a panoramic video image, an apparatus for acquiring a panoramic video image, an electronic device, and a computer-readable storage medium.
In a first aspect, the present application provides a method for encoding and transmitting a panoramic video image. The method comprises the following steps:
acquiring a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders;
performing transformation processing on the first panoramic video image to transform a target object in the image to an image center to obtain a second panoramic video image and obtain transformation reduction information of the second panoramic video image;
carrying out region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the center of the image and matched with the target object to obtain a third panoramic video image;
and sending the third panoramic video image and the transformation and restoration information to a panoramic video receiving end so that the panoramic video receiving end can obtain a fourth panoramic video image restored by an image center according to the transformation and restoration information after decoding the third panoramic video image.
In one embodiment, the performing, on the first panoramic video image, a transformation process for transforming a target object in the image to an image center includes:
identifying a target object in the first panoramic video image; determining a geometric center of the target object in the first panoramic video image; and performing transformation processing on the first panoramic video image according to a transformation relation of moving the geometric center of the target object to the image center of the first panoramic video image.
In one embodiment, the panoramic picture is a panoramic picture of a target still-like scene; the transforming the first panoramic video image according to the transformation relation of moving the geometric center of the target object to the image center of the first panoramic video image includes:
when the first panoramic video image is a first frame image of a panoramic video, acquiring a transformation relation for moving the geometric center of the target object to the image center and taking the transformation relation as the transformation relation of the first frame image, and transforming the first panoramic video image according to the transformation relation of the first frame image; and when the first panoramic video image is a non-head frame image of the panoramic video, carrying out transformation processing on the first panoramic video image according to the transformation relation of the head frame image.
In one embodiment, the panoramic picture is a panoramic picture of a target motion scene; the transforming the first panoramic video image according to the transformation relation of moving the geometric center of the target object to the image center of the first panoramic video image includes:
when the first panoramic video image is the first image of the coded image group, acquiring a transformation relation for moving the geometric center of the target object to the image center and taking the transformation relation as the transformation relation of the first image, and transforming the first panoramic video image according to the transformation relation of the first image; when the first panoramic video image is a non-first image of an encoded image group, carrying out transformation processing on the first panoramic video image according to the transformation relation of the first image; and the size of the coding image group is set according to the motion characteristics of the target object in the target motion scene.
In one embodiment, before the region-of-interest coding is performed on the second panoramic video image according to the region-of-interest which is set at the center of the image and matched with the target object, the method further includes:
determining a circumscribed rectangle of the target object with a geometric center located at the image center of the second panoramic video image; and setting the region of interest in the center of the image according to the circumscribed rectangle.
In a second aspect, the present application provides a method for acquiring a panoramic video image. The method comprises the following steps:
receiving a third panoramic video image and transformation and restoration information from a panoramic video sending end;
decoding the third panoramic video image to obtain a decoded third panoramic video image;
performing image center restoration processing on the decoded third panoramic video image according to the conversion restoration information to obtain a fourth panoramic video image;
the third panoramic video image is obtained by the panoramic video sending end by coding the region of interest of the second panoramic video image according to the region of interest which is arranged in the image center and matched with the target object; the second panoramic video image and the conversion and restoration information are obtained by converting the target object in the image to the image center of the first panoramic video image by the panoramic video sending end; the first panoramic video image is obtained by projecting the panoramic picture according to an equidistant cylinder.
In a third aspect, the present application provides a panoramic live video transmission system. The system comprises: a main broadcasting end and a watching end of panoramic live broadcasting; wherein the content of the first and second substances,
the anchor terminal is used for encoding a first panoramic live video corresponding to a panoramic live video picture according to the panoramic video image encoding and transmitting method and transmitting the encoded first panoramic live video to the watching terminal; the watching end is used for decoding and restoring the panoramic video image acquisition method to obtain a second panoramic live video with the image center consistent with the first panoramic live video.
In a fourth aspect, the present application provides an apparatus for encoding and transmitting a panoramic video image. The device comprises:
the image acquisition module is used for acquiring a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders;
the center transformation module is used for carrying out transformation processing on the first panoramic video image to transform a target object in the image to an image center so as to obtain a second panoramic video image and obtain transformation reduction information of the second panoramic video image;
the image coding module is used for coding the region of interest of the second panoramic video image according to the region of interest which is arranged in the center of the image and matched with the target object to obtain a third panoramic video image;
and the image sending module is used for sending the third panoramic video image and the transformation and restoration information to a panoramic video receiving end so that the panoramic video receiving end can obtain a fourth panoramic video image restored by an image center according to the transformation and restoration information after decoding the third panoramic video image.
In a fifth aspect, the present application provides an apparatus for acquiring a panoramic video image. The device comprises:
the image receiving module is used for receiving a third panoramic video image and transformation and restoration information from the panoramic video sending end;
the image decoding module is used for decoding the third panoramic video image to obtain a decoded third panoramic video image;
the center restoration module is used for carrying out image center restoration processing on the decoded third panoramic video image according to the transformation restoration information to obtain a fourth panoramic video image;
the third panoramic video image is obtained by the panoramic video sending end by coding the region of interest of the second panoramic video image according to the region of interest which is arranged in the image center and matched with the target object; the second panoramic video image and the conversion and restoration information are obtained by converting the target object in the image to the image center of the first panoramic video image by the panoramic video sending end; the first panoramic video image is obtained by projecting the panoramic picture according to equidistant cylinders.
In a sixth aspect, the present application provides an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the following steps when executing the computer program:
acquiring a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders; performing transformation processing on the first panoramic video image to transform a target object in the image to an image center to obtain a second panoramic video image and transformation reduction information of the second panoramic video image; carrying out region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the center of the image and matched with the target object to obtain a third panoramic video image; and sending the third panoramic video image and the transformation and restoration information to a panoramic video receiving end so that the panoramic video receiving end can obtain a fourth panoramic video image restored by an image center according to the transformation and restoration information after decoding the third panoramic video image.
In a seventh aspect, the present application provides an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the following steps when executing the computer program:
receiving a third panoramic video image and transformation and restoration information from a panoramic video sending end; decoding the third panoramic video image to obtain a decoded third panoramic video image; performing image center restoration processing on the decoded third panoramic video image according to the transformation restoration information to obtain a fourth panoramic video image; the third panoramic video image is obtained by the panoramic video sending end by coding the region of interest of the second panoramic video image according to the region of interest which is arranged in the image center and matched with the target object; the second panoramic video image and the conversion and restoration information are obtained by converting the target object in the image to the image center of the first panoramic video image by the panoramic video sending end; the first panoramic video image is obtained by projecting the panoramic picture according to equidistant cylinders.
In an eighth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders; performing transformation processing on the first panoramic video image to transform a target object in the image to an image center to obtain a second panoramic video image and transformation reduction information of the second panoramic video image; carrying out region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the center of the image and matched with the target object to obtain a third panoramic video image; and sending the third panoramic video image and the transformation and restoration information to a panoramic video receiving end so that the panoramic video receiving end can obtain a fourth panoramic video image restored by an image center according to the transformation and restoration information after decoding the third panoramic video image.
In a ninth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
receiving a third panoramic video image and transformation and restoration information from a panoramic video sending end; decoding the third panoramic video image to obtain a decoded third panoramic video image; performing image center restoration processing on the decoded third panoramic video image according to the conversion restoration information to obtain a fourth panoramic video image; the third panoramic video image is obtained by the panoramic video sending end through region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the image center and matched with the target object; the second panoramic video image and the conversion and restoration information are obtained by converting the target object in the image to the image center of the first panoramic video image by the panoramic video sending end; the first panoramic video image is obtained by projecting the panoramic picture according to equidistant cylinders.
The panoramic video sending end obtains a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders, carries out conversion processing on the first panoramic video image to convert a target object in the image to an image center to obtain a second panoramic video image and conversion reduction information of the second panoramic video image, then carries out region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the image center and matched with the target object to obtain a third panoramic video image, sends the third panoramic video image and the conversion reduction information to the panoramic video receiving end, and the panoramic video receiving end decodes the third panoramic video image and obtains a fourth panoramic video image reduced by the image center according to the conversion reduction information. The scheme transforms the target object to the center of the image at a transmitting end, and at the moment, the target object does not have the condition of being stretched in the transformed image, so that compared with the image which is not transformed, the ROI coding is carried out on the image, redundant information does not exist, then corresponding transformation reduction information and the coded image are sent to a receiving end, the receiving end obtains a panoramic video image restored by the center of the image according to the transformation reduction information, the coding transmission efficiency is improved, and meanwhile the center of the image of the receiving end is kept consistent with that of the transmitting end.
Drawings
FIG. 1 is a diagram illustrating an application scenario of a correlation method in an embodiment of the present application;
fig. 2 is a schematic flowchart of a method for encoding and transmitting a panoramic video image according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a process of transforming a first panoramic video image according to an embodiment of the present disclosure;
fig. 4 is a schematic flowchart of a method for acquiring a panoramic video image according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a panoramic live video transmission system in an embodiment of the present application;
fig. 6 is a block diagram of a device for encoding and transmitting a panoramic video image according to an embodiment of the present application;
fig. 7 is a block diagram of a device for acquiring a panoramic video image according to an embodiment of the present application;
fig. 8 is an internal structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The coding transmission method and the acquisition method of the panoramic video image provided by the embodiment of the application can be applied to an application scene shown in fig. 1, the application scene can comprise a panoramic video sending end, a server and a panoramic video receiving end, and the panoramic video sending end can be used for coding the panoramic video image and then transmitting the coded panoramic video image to the panoramic video receiving end through the server. The panoramic video sending end can be an electronic device such as a smart phone and a personal computer which are provided with panoramic video shooting equipment; the panoramic video receiving end can be an electronic device such as a smart phone and a personal computer which are provided with the panoramic video display device; the server may be implemented as a stand-alone server or as a server cluster consisting of a plurality of servers.
The following sections sequentially describe the coding transmission method of the panoramic video image and the acquisition method of the panoramic video image according to the present application, with reference to the application scenario shown in fig. 1 and the related drawings.
In one embodiment, as shown in fig. 2, there is provided a method for encoding and transmitting a panoramic video image, where the method is performed by a panoramic video transmitting end in fig. 1, and the method includes:
step S201, a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders is obtained.
In this step, the panoramic video sending end can shoot and make a panoramic picture through the configured panoramic video shooting equipment, and then render the panoramic picture according to the equidistant cylindrical projection mode to obtain a panoramic video image corresponding to the panoramic picture, and record the panoramic video image as a first panoramic video image, and at this time, the shot panoramic picture is composed of a frame of two-dimensional first panoramic video image.
Step S202, the first panoramic video image is subjected to transformation processing of transforming the target object in the image to the center of the image, so that a second panoramic video image is obtained, and transformation and restoration information of the second panoramic video image is obtained.
Specifically, the panoramic video sending end may identify a target object for each frame of the first panoramic video image, for example, identify a face in each frame of the first panoramic video image as the target object, and the specific target object may be determined according to actual application requirements. For each frame of the first panoramic video image, after the panoramic video sending end identifies the target object, the related information of the target object in the first panoramic video image, such as image pixel information forming the target object, the geometric center of the target object, and the like, can be obtained. Then, in this step, the panoramic video sending end performs a transformation process on the first panoramic video image to transform the target object in the image to the image center, that is, transforms the position of the target object to the image center to obtain a second panoramic video image, in which the target object is located at the image center, and the second panoramic video image transforms the target object to the image center through this process, at this time, the distortion of the target object is minimum, and compared with the setting of an interested area, the redundant information can be greatly reduced, and the coding transmission efficiency is improved. In this step, the panoramic video sending end also obtains the transformation and restoration information of the second panoramic video image, and according to the transformation and restoration information, the position of each image pixel in the second panoramic video image can be restored to the position of each image pixel in the first panoramic video image, that is, the position of the target object in the image center in the second panoramic video image can be restored to the original position in the first panoramic video image, and simultaneously the target object can be restored to the image center of the first panoramic video image.
And step S203, carrying out region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the center of the image and matched with the target object, and obtaining a third panoramic video image.
In step S202, the target object is moved to the image center, so in this step, the panoramic video sending end may directly set an interested region at the image center of the second panoramic video image, and the interested region is matched with the target object, and may be set according to the shape and size of the target object in the image, for example, according to the size of the target object, the target object may be enlarged by several times in equal proportion, and the setting may be specifically performed according to the scene requirement. In this step, the panoramic video sending end performs region-of-interest coding on the second panoramic video image according to the region-of-interest set in the above manner, so as to obtain a coded second panoramic video image, which is recorded as a third panoramic video image.
For the region-of-interest setting, in some embodiments, before the sending end of the panoramic video performs region-of-interest encoding on the second panoramic video image according to the region-of-interest set in the center of the image and matched with the target object in step S203, the region-of-interest setting may be set through the following steps, which specifically include:
determining a circumscribed rectangle of the target object with the geometric center positioned in the image center of the second panoramic video image; and setting an interested area in the center of the image according to the circumscribed rectangle.
The panoramic video sending end may transform the geometric center of the target object to the image center of the second panoramic video image in step S202, so as to obtain the second panoramic video image in which the geometric center of the target object is located at the image center. In this embodiment, the panoramic video sending end determines a circumscribed rectangle of the target object whose geometric center is located at the image center of the second panoramic video image, and sets an interesting region matched with the target object in the image center of the second panoramic video image according to the size of the circumscribed rectangle. Illustratively, the panoramic video sending end can use the image center as the area center, and magnify several times in equal proportion according to the size of the external rectangle to obtain the region of interest matched with the target object, so as to accurately and efficiently set the region of interest for the target object.
And step S204, sending the third panoramic video image and the transformation and restoration information to a panoramic video receiving end so that the panoramic video receiving end can obtain a fourth panoramic video image restored by the image center according to the transformation and restoration information after decoding the third panoramic video image.
This step is a transmission step of the third panoramic video image, which is the second panoramic video image after the region of interest is encoded, and the conversion restoration information. The panoramic video receiving end can perform image center reduction processing on the decoded third panoramic video image according to the conversion reduction information to obtain a fourth panoramic video image reduced by the image center, and at the moment, the image center of the fourth panoramic video image is consistent with the image center of the first panoramic video image, and the panoramic video receiving end performs subsequent rendering flow according to the fourth panoramic video image, so that the panoramic picture seen by a panoramic video user at the panoramic video receiving end is consistent with the panoramic picture seen by the panoramic video user at the panoramic video receiving end.
In the method for encoding and transmitting the panoramic video image of the embodiment, the target object is converted to the image center at the panoramic video transmitting end, and the target object does not have the condition of being stretched in the converted image, so that ROI encoding is performed on the image, compared with the image without conversion, redundant information does not exist, corresponding conversion reduction information and the encoded image are transmitted to the panoramic video receiving end, the panoramic video receiving end obtains the panoramic video image reduced by the image center according to the conversion reduction information, and the image center of the panoramic video receiving end is kept consistent with the panoramic video transmitting end while the encoding and transmitting efficiency is improved.
In one embodiment, as shown in fig. 3, the transforming process of transforming the target object in the image to the center of the image on the first panoramic video image in step S202 may include:
in step S301, a target object in the first panoramic video image is identified.
In this step, after the panoramic video sending end obtains the first panoramic video image, a target object in the first panoramic video image is identified, where the target object may be a certain person or a face of a certain person, and the like.
Step S302, determining a geometric center of a target object in the first panoramic video image.
In this step, after the panoramic video sending end identifies the target object in the first panoramic video image, the relevant information of the target object, such as image pixel information of the target object formed on the first panoramic video image, can be obtained, and the panoramic video sending end can calculate the position of the geometric center of the target object in the first panoramic video image according to the image pixels of the target object formed on the first panoramic video image.
Step S303, transform the first panoramic video image according to the transform relationship of moving the geometric center of the target object to the image center of the first panoramic video image.
In this step, for the first panoramic video image, the panoramic video sending end performs transformation processing on each image pixel in the first panoramic video image according to the transformation relation of moving the position of the geometric center of the target object to the position of the image center of the first panoramic video image. Wherein the transformation relation may be a transformation matrix. And the panoramic video sending end can also calculate a transformation reduction matrix for transforming the second panoramic video image back to the first panoramic video image and takes the transformation reduction matrix as transformation reduction information of the second panoramic video image. The scheme of the embodiment provides an accurate and convenient way for transforming the target object to the image center to obtain the second panoramic video image and the transformation and restoration information thereof.
For the transformation processing of the first panoramic video image, in some embodiments, when the panoramic image is a panoramic image of a target still-like scene, the step S303 specifically includes:
when the first panoramic video image is the first frame image of the panoramic video, acquiring a transformation relation for moving the geometric center of the target object to the image center and taking the transformation relation as the transformation relation of the first frame image, and transforming the first panoramic video image according to the transformation relation of the first frame image; and when the first panoramic video image is a non-first frame image of the panoramic video, carrying out transformation processing on the first panoramic video image according to the transformation relation of the first frame image.
The embodiment provides a transformation processing mode of a panoramic picture aiming at a target static scene so as to save the operation resources. The panoramic picture of the target static scene refers to a panoramic picture shot by a panoramic video sending end, wherein a target object is relatively static, for example, the target object is always sitting or standing at a certain fixed position on a stage in live webcasting, and the shot panoramic picture is the panoramic picture of the target static scene. In this embodiment, the panoramic video sending end only needs to calculate the transformation relationship of the first frame image of the panoramic video, and can apply the transformation relationship to the first frame image and other non-first frame images in the panoramic video, and the region where the target object is located can be always in the originally set region of interest, so as to save computation resources. That is, when the first panoramic video image is the first frame image of the panoramic video, the panoramic video transmitting end obtains the transformation relation of moving the geometric center of the target object to the image center and uses the transformation relation as the transformation relation of the first frame image, and carries out transformation processing on the first panoramic video image according to the transformation relation, and for the non-first frame image of the panoramic video, the panoramic video transmitting end directly carries out transformation processing on the non-first frame image according to the transformation relation of the first frame image, thereby saving the operation resources on the premise of ensuring the whole interested region coding effect of the panoramic video.
For the transformation processing of the first panoramic video image, in some other embodiments, when the panoramic image is a panoramic image of the target motion-based scene, the step S303 specifically includes:
when the first panoramic video image is the first image of the coded image group, acquiring a transformation relation for moving the geometric center of the target object to the image center and taking the transformation relation as the transformation relation of the first image, and transforming the first panoramic video image according to the transformation relation of the first image; when the first panoramic video image is a non-first image of the coded image group, carrying out transformation processing on the first panoramic video image according to the transformation relation of the first image; the size of the coding image group can be set according to the motion characteristics of the target object in the target motion scene.
The embodiment provides a transformation processing mode aiming at the panoramic picture of the target motion scene so as to improve the effect of region-of-interest coding in the scene. The target motion scene is opposite to the target static scene, and the panoramic picture of the target motion scene refers to a panoramic picture shot by a panoramic video sending end, wherein the target object continuously moves to enable the position of the target object to be continuously changed, for example, the position of the target object continuously changes when the target object dances on a stage in live webcast, and the shot panoramic picture is the panoramic picture of the target motion scene. In video image coding, there is a concept of GOP (Group of Pictures, which is referred to as a coded Group of Pictures in this application) representing how long a key frame is. Besides the key frame, the other frames in a coded image group are coded by the key frame, and if a large picture change (such as a large moving range of the target object position, a high speed, etc.) occurs in a coded image group, the coding quality is seriously affected. In order to ensure the encoding effect, in this embodiment, on one hand, the size of the encoded image group is set according to the motion characteristics of the target object in the target motion class scene, and corresponding to the scene with obvious motion of the target object, the motion characteristics such as the motion range size, the speed and the like of the target object can be predicted in advance, and the size of the corresponding encoded image group is set according to the motion characteristics; on the other hand, in the present embodiment, the transformation relationship for performing transformation processing on the first panoramic video image is kept synchronous with the encoded image groups, that is, one transformation relationship corresponds to one encoded image group, that is, the transformation relationship is recalculated once for each encoded image group, that is, the motion range of the target object is large, and the transformation relationship needs to be calculated/updated more frequently, so as to ensure that the target object is always in the region of interest set in the center of the image, and thus the effect of region of interest encoding in this scene is improved. That is, for each encoded image group, when the first panoramic video image is the first image of the encoded image group, the panoramic video transmitting end may obtain the transformation relationship of moving the geometric center of the target object to the image center and use the transformation relationship as the transformation relationship of the first image, and transform the first image according to the transformation relationship, and then for other non-first images of the encoded image group, the panoramic video transmitting end may transform the non-first image according to the transformation relationship of the first image, so as to implement real-time tracking of the target object, always place the region where the target object is located in the region of interest, and improve the effect of encoding the region of interest in the target motion scene.
In one embodiment, as shown in fig. 4, there is provided a method for acquiring a panoramic video image, the method being performed by a panoramic video receiving end, the method including:
and S401, receiving a third panoramic video image and transformation and restoration information from a panoramic video sending end.
In this step, the panoramic video receiving end may receive, from the server, the video stream including the third panoramic video image and the conversion and restoration information, and the panoramic video receiving end needs to analyze the third panoramic video image and the conversion and restoration information. The third panoramic video image and the conversion and restoration information are provided by a panoramic video sending end, and the specific manner of forming and sending the third panoramic video image and the conversion and restoration information by the panoramic video sending end is detailed in relevant contents disclosed in the embodiments of the coding and transmission method of the panoramic video image.
And step S402, decoding the third panoramic video image to obtain a decoded third panoramic video image.
In this step, the panoramic video receiving end decodes the third panoramic video image to obtain a decoded third panoramic video image, an image center of the decoded third panoramic video image is not consistent with an image center of the first panoramic video image initially obtained by the panoramic video transmitting end, and the panoramic video receiving end needs to perform step S403 to perform image center restoration.
And S403, performing image center restoration processing on the decoded third panoramic video image according to the conversion restoration information to obtain a fourth panoramic video image.
In this step, the panoramic video receiving end performs image center restoration processing on the decoded third panoramic video image according to the conversion restoration information, that is, the panoramic video receiving end restores the converted position of each image pixel in the current frame back to the position before conversion according to the conversion restoration information to obtain a fourth panoramic video image with the image center consistent with the first panoramic video image, so as to achieve the effect of keeping consistent with the image center of the first panoramic video image, and then the panoramic video receiving end performs subsequent rendering process according to the fourth panoramic video image, so that the panoramic picture seen by the user at the panoramic video receiving end can be ensured to be consistent with the panoramic picture seen by the user at the panoramic video transmitting end.
According to the scheme of the embodiment, the panoramic video receiving end carries out decoding and position restoration to obtain the fourth panoramic video image with the image center consistent with the first panoramic video image, and the image center of the panoramic video receiving end is consistent with the panoramic video transmitting end while the coding transmission efficiency is improved.
In one embodiment, a panoramic live video transmission system is provided, as shown in fig. 5, comprising a main broadcasting end and a watching end of panoramic live broadcasting. The anchor terminal may be configured to encode a first live panoramic video corresponding to a live panoramic picture according to the encoding and transmitting method for a panoramic video image described in any of the above embodiments and transmit the encoded first live panoramic video to the viewer terminal, and the viewer terminal may be configured to decode and restore the encoded first live panoramic video according to the acquiring method for a panoramic video image described in the above embodiments to obtain a second live panoramic video having an image center that is identical to the first live panoramic video.
Specifically, in this embodiment, a live broadcast end of a panoramic live broadcast can shoot a live broadcast panoramic picture through a panoramic video shooting device configured by the live broadcast end of the panoramic live broadcast, the live broadcast end of the panoramic live broadcast obtains a first live broadcast panoramic video corresponding to the live broadcast panoramic picture, for each frame of video image in the first live broadcast panoramic video, the live broadcast end of the panoramic live broadcast adopts the coding transmission method of the panoramic video image as described in any one of the embodiments, the coded video image and corresponding conversion reduction information are transmitted to a live broadcast watching end of the panoramic live broadcast through a live broadcast server, the live broadcast watching end of the panoramic live broadcast receives the coded video image and corresponding conversion reduction information from the live broadcast server, the coded video image and the corresponding conversion reduction information are decoded and reduced according to the obtaining method of the panoramic video image as described in the embodiments, so that a second live broadcast panoramic video with an image center consistent with the first live broadcast panoramic video is obtained, and then the second live broadcast panoramic video is rendered and displayed, and the live broadcast end of the panoramic video always performs rendering and displaying according to the first live broadcast end, so that the frames of the live broadcast end of the panoramic live broadcast and the live broadcast end are kept consistent.
The method for coding and transmitting the panoramic video image and the method for acquiring the panoramic video image are applied to a panoramic live broadcast system, the region of interest can be effectively selected at the anchor terminal, redundant pixel information is avoided as much as possible, coding and transmitting efficiency of the panoramic live broadcast video is improved, and the panoramic live broadcast effects of code rate reduction, cost saving, high fluency, clear target object and consistent picture are achieved.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiments of the present application further provide a panoramic video image encoding and transmitting apparatus for implementing the above-mentioned panoramic video image encoding and transmitting method, and a panoramic video image acquiring apparatus for implementing the above-mentioned panoramic video image acquiring method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so specific limitations in one or more apparatus embodiments provided below may refer to the limitations in the above corresponding methods, and are not described herein again.
In one embodiment, as shown in fig. 6, there is provided an apparatus for encoding and transmitting a panoramic video image, the apparatus 600 comprising:
an image obtaining module 601, configured to obtain a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders;
a center transformation module 602, configured to perform transformation processing on the first panoramic video image to transform a target object in the image to an image center, so as to obtain a second panoramic video image and obtain transformation and restoration information of the second panoramic video image;
an image encoding module 603, configured to perform region-of-interest encoding on the second panoramic video image according to a region-of-interest that is located in an image center and matches the target object, so as to obtain a third panoramic video image;
an image sending module 604, configured to send the third panoramic video image and the transformed and restored information to a panoramic video receiving end, so that the panoramic video receiving end decodes the third panoramic video image and then obtains a fourth panoramic video image restored by an image center according to the transformed and restored information.
In one embodiment, a center transformation module 602 to identify a target object in the first panoramic video image; determining a geometric center of the target object in the first panoramic video image; and transforming the first panoramic video image according to the transformation relation of moving the geometric center of the target object to the image center of the first panoramic video image.
In one embodiment, the panoramic picture is a panoramic picture of a target still-like scene; a center transformation module 602, configured to, when the first panoramic video image is a first frame image of a panoramic video, obtain a transformation relationship that moves a geometric center of the target object to the image center and use the transformation relationship as the first frame image, and transform the first panoramic video image according to the transformation relationship of the first frame image; and when the first panoramic video image is a non-first frame image of the panoramic video, carrying out transformation processing on the first panoramic video image according to the transformation relation of the first frame image.
In one embodiment, the panoramic picture is a panoramic picture of a target motion-like scene; a center transformation module 602, configured to, when the first panoramic video image is a first image of a coded image group, obtain a transformation relationship that moves a geometric center of the target object to the image center, and use the transformation relationship as a transformation relationship of the first image, and transform the first panoramic video image according to the transformation relationship of the first image; when the first panoramic video image is a non-first image of the coded image group, carrying out transformation processing on the first panoramic video image according to the transformation relation of the first image; and the size of the coding image group is set according to the motion characteristics of the target object in the target motion scene.
In one embodiment, the apparatus 600 comprises: the area setting module is used for determining a circumscribed rectangle of the target object with the geometric center positioned in the image center of the second panoramic video image; and setting the region of interest in the center of the image according to the circumscribed rectangle.
In one embodiment, as shown in fig. 7, there is provided an apparatus for acquiring a panoramic video image, the apparatus 700 comprising:
an image receiving module 701, configured to receive a third panoramic video image and transformation and restoration information from a panoramic video sending end;
an image decoding module 702, configured to decode the third panoramic video image to obtain a decoded third panoramic video image;
a center restoration module 703, configured to perform image center restoration processing on the decoded third panoramic video image according to the conversion restoration information to obtain a fourth panoramic video image;
the third panoramic video image is obtained by the panoramic video sending end through region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the image center and matched with the target object; the second panoramic video image and the conversion and restoration information are obtained by converting the target object in the image to the image center of the first panoramic video image by the panoramic video sending end; the first panoramic video image is obtained by projecting the panoramic picture according to an equidistant cylinder.
The various modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules may be embedded in a hardware form or may be independent of a processor in the electronic device, or may be stored in a memory in the electronic device in a software form, so that the processor calls and executes operations corresponding to the modules.
In one embodiment, an electronic device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 8. The electronic device comprises a processor, a memory, a communication interface, a display screen and an input device which are connected through a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic equipment comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the electronic device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a method of coded transmission of a panoramic video image or a method of acquisition of a panoramic video image. The display screen of the electronic equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the electronic equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the structure shown in fig. 8 is a block diagram of only a portion of the structure relevant to the present disclosure, and does not constitute a limitation on the electronic device to which the present disclosure may be applied, and that a particular electronic device may include more or less components than those shown, or combine certain components, or have a different arrangement of components.
In one embodiment, an electronic device is further provided, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, databases, or other media used in the embodiments provided herein can include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), magnetic Random Access Memory (MRAM), ferroelectric Random Access Memory (FRAM), phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), for example. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.
All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (11)
1. A method for coded transmission of panoramic video images, the method comprising:
acquiring a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders;
performing transformation processing on the first panoramic video image to transform a target object in the image to an image center to obtain a second panoramic video image and obtain transformation reduction information of the second panoramic video image;
carrying out region-of-interest coding on the second panoramic video image according to a region-of-interest which is arranged in the center of the image and matched with the target object to obtain a third panoramic video image;
and sending the third panoramic video image and the transformation and restoration information to a panoramic video receiving end so that the panoramic video receiving end can obtain a fourth panoramic video image restored by an image center according to the transformation and restoration information after decoding the third panoramic video image.
2. The method according to claim 1, wherein the performing, on the first panoramic video image, a transformation process for transforming a target object in an image to an image center comprises:
identifying a target object in the first panoramic video image;
determining a geometric center of the target object in the first panoramic video image;
and performing transformation processing on the first panoramic video image according to a transformation relation of moving the geometric center of the target object to the image center of the first panoramic video image.
3. The method of claim 2, wherein the panoramic picture is a panoramic picture of a target still-like scene; the transforming the first panoramic video image according to the transformation relation of moving the geometric center of the target object to the image center of the first panoramic video image includes:
when the first panoramic video image is a first frame image of a panoramic video, acquiring a transformation relation for moving the geometric center of the target object to the image center and taking the transformation relation as the transformation relation of the first frame image, and transforming the first panoramic video image according to the transformation relation of the first frame image;
and when the first panoramic video image is a non-head frame image of the panoramic video, carrying out transformation processing on the first panoramic video image according to the transformation relation of the head frame image.
4. The method of claim 2, wherein the panoramic picture is a panoramic picture of a target motion-like scene; the transforming the first panoramic video image according to the transformation relation of moving the geometric center of the target object to the image center of the first panoramic video image includes:
when the first panoramic video image is the first image of the coded image group, acquiring a transformation relation for moving the geometric center of the target object to the image center and using the transformation relation as the transformation relation of the first image, and transforming the first panoramic video image according to the transformation relation of the first image;
when the first panoramic video image is a non-first image of an encoded image group, carrying out transformation processing on the first panoramic video image according to the transformation relation of the first image;
and the size of the coding image group is set according to the motion characteristics of the target object in the target motion scene.
5. The method according to any one of claims 1 to 4, wherein before said region-of-interest encoding said second panoramic video image according to a region-of-interest set at the center of the image and matching said target object, said method further comprises:
determining a circumscribed rectangle of the target object with a geometric center located at the image center of the second panoramic video image;
and setting the region of interest in the center of the image according to the circumscribed rectangle.
6. A method for acquiring a panoramic video image, the method comprising:
receiving a third panoramic video image and transformation and restoration information from a panoramic video sending end;
decoding the third panoramic video image to obtain a decoded third panoramic video image;
performing image center restoration processing on the decoded third panoramic video image according to the conversion restoration information to obtain a fourth panoramic video image;
the third panoramic video image is obtained by the panoramic video sending end by coding the region of interest of the second panoramic video image according to the region of interest which is arranged in the image center and matched with the target object; the second panoramic video image and the conversion and restoration information are obtained by converting the target object in the image to the image center of the first panoramic video image by the panoramic video sending end; the first panoramic video image is obtained by projecting the panoramic picture according to equidistant cylinders.
7. A panoramic live video transmission system, comprising: a main broadcasting end and a watching end of panoramic live broadcasting; wherein the content of the first and second substances,
the anchor terminal is used for encoding a first panoramic live video corresponding to a panoramic live picture according to the method of any one of claims 1 to 5 and then transmitting the encoded first panoramic live video to the watching terminal;
the viewing terminal is used for decoding and restoring according to the method of claim 6 to obtain a second panoramic live video with the image center consistent with the first panoramic live video.
8. An apparatus for encoding and transmitting a panoramic video image, the apparatus comprising:
the image acquisition module is used for acquiring a first panoramic video image obtained by projecting a panoramic picture according to equidistant cylinders;
the center transformation module is used for transforming a target object in the image to an image center of the first panoramic video image to obtain a second panoramic video image and transformation and restoration information of the second panoramic video image;
the image coding module is used for coding the region of interest of the second panoramic video image according to the region of interest which is arranged in the center of the image and matched with the target object to obtain a third panoramic video image;
and the image sending module is used for sending the third panoramic video image and the transformation and restoration information to a panoramic video receiving end so that the panoramic video receiving end can obtain a fourth panoramic video image restored by an image center according to the transformation and restoration information after decoding the third panoramic video image.
9. An apparatus for acquiring a panoramic video image, the apparatus comprising:
the image receiving module is used for receiving a third panoramic video image and transformation and restoration information from the panoramic video sending end;
the image decoding module is used for decoding the third panoramic video image to obtain a decoded third panoramic video image;
the center restoration module is used for carrying out image center restoration processing on the decoded third panoramic video image according to the transformation restoration information to obtain a fourth panoramic video image;
the third panoramic video image is obtained by the panoramic video sending end by coding the region of interest of the second panoramic video image according to the region of interest which is arranged in the image center and matched with the target object; the second panoramic video image and the conversion and restoration information are obtained by converting the target object in the image to the image center of the first panoramic video image by the panoramic video sending end; the first panoramic video image is obtained by projecting the panoramic picture according to an equidistant cylinder.
10. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 5 or 6.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 5 or 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210905773.9A CN115243054A (en) | 2022-07-29 | 2022-07-29 | Panoramic video image coding transmission method, panoramic video image acquisition method, related device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210905773.9A CN115243054A (en) | 2022-07-29 | 2022-07-29 | Panoramic video image coding transmission method, panoramic video image acquisition method, related device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115243054A true CN115243054A (en) | 2022-10-25 |
Family
ID=83677804
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210905773.9A Pending CN115243054A (en) | 2022-07-29 | 2022-07-29 | Panoramic video image coding transmission method, panoramic video image acquisition method, related device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115243054A (en) |
-
2022
- 2022-07-29 CN CN202210905773.9A patent/CN115243054A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11128888B2 (en) | Method and apparatus for video decoding | |
| US20180270496A1 (en) | Composite video streaming using stateless compression | |
| CN108063976B (en) | Video processing method and device | |
| US11922599B2 (en) | Video super-resolution processing method and apparatus | |
| US20140086309A1 (en) | Method and device for encoding and decoding an image | |
| JP7177034B2 (en) | Method, apparatus and stream for formatting immersive video for legacy and immersive rendering devices | |
| CN108650460B (en) | Server, panoramic video storage and transmission method and computer storage medium | |
| CN109672897B (en) | Panoramic video coding method and device | |
| EP2171682B1 (en) | Method and system for processing of images | |
| CN115243054A (en) | Panoramic video image coding transmission method, panoramic video image acquisition method, related device and system | |
| CN116170636A (en) | Live video playing method and device, equipment and medium thereof | |
| WO2019127100A1 (en) | Video coding method, device, and computer system | |
| CN112367521A (en) | Display screen content sharing method and device, computer equipment and storage medium | |
| CN118138784A (en) | Video segmentation compression method, device, equipment and medium | |
| CN116708793B (en) | Video transmission method, device, equipment and storage medium | |
| WO2024051299A1 (en) | Encoding method and apparatus, and decoding method and apparatus | |
| WO2024078403A1 (en) | Image processing method and apparatus, and device | |
| CN118283298A (en) | Video transmission method, processing method, apparatus, device, medium, and program product | |
| CN115412727A (en) | Encoding method, decoding method and device thereof | |
| CN117041685A (en) | Road disease display method, system, device, equipment and readable storage medium | |
| CN116828208A (en) | Video compression and enhancement method and device, electronic equipment and storage medium | |
| Li et al. | VertexShuffle-Based Spherical Super-Resolution for 360-Degree Videos | |
| CN116886984A (en) | Image processing method, apparatus, electronic device, and computer-readable storage medium | |
| CN117435146A (en) | Code stream generation method, system and device, electronic equipment and storage medium | |
| CN112422974A (en) | Video encoding method, video encoding device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |