CN113630622A - Panoramic video image processing method, server, target device, apparatus and system - Google Patents
Panoramic video image processing method, server, target device, apparatus and system Download PDFInfo
- Publication number
- CN113630622A CN113630622A CN202110678889.9A CN202110678889A CN113630622A CN 113630622 A CN113630622 A CN 113630622A CN 202110678889 A CN202110678889 A CN 202110678889A CN 113630622 A CN113630622 A CN 113630622A
- Authority
- CN
- China
- Prior art keywords
- panoramic video
- video image
- image
- processing
- hot spot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 76
- 238000012545 processing Methods 0.000 claims abstract description 56
- 238000007781 pre-processing Methods 0.000 claims abstract description 35
- 230000008569 process Effects 0.000 claims description 16
- 238000003860 storage Methods 0.000 claims description 16
- 230000000007 visual effect Effects 0.000 claims description 9
- 238000005520 cutting process Methods 0.000 claims description 2
- 238000002203 pretreatment Methods 0.000 claims 1
- 230000005540 biological transmission Effects 0.000 abstract description 7
- 230000002829 reductive effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 22
- 230000006870 function Effects 0.000 description 9
- 230000008901 benefit Effects 0.000 description 7
- 238000004590 computer program Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 4
- YBJHBAHKTGYVGT-ZKWXMUAHSA-N (+)-Biotin Chemical compound N1C(=O)N[C@@H]2[C@H](CCCCC(=O)O)SC[C@@H]21 YBJHBAHKTGYVGT-ZKWXMUAHSA-N 0.000 description 3
- FEPMHVLSLDOMQC-UHFFFAOYSA-N virginiamycin-S1 Natural products CC1OC(=O)C(C=2C=CC=CC=2)NC(=O)C2CC(=O)CCN2C(=O)C(CC=2C=CC=CC=2)N(C)C(=O)C2CCCN2C(=O)C(CC)NC(=O)C1NC(=O)C1=NC=CC=C1O FEPMHVLSLDOMQC-UHFFFAOYSA-N 0.000 description 3
- 230000007423 decrease Effects 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
Abstract
The invention provides a panoramic video image processing method, a server, target equipment, a device and a system, wherein the method is applied to the server and comprises the following steps: acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area; preprocessing a panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises dividing the panoramic video image into a plurality of image blocks and respectively carrying out scaling processing on each image block according to the distance between each image block and a hot spot area; coding the preprocessed image to obtain coded data; the encoded data is transmitted to the target terminal. By using the method, the bandwidth pressure caused by video data transmission can be reduced without additionally modifying a conventional encoder on the premise of ensuring the picture resolution of the hot spot area which is interested by the user.
Description
Technical Field
The invention belongs to the field of video processing, and particularly relates to a panoramic video image processing method, a server, target equipment, a device and a system.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
The appearance of panoramic video images provides a better visual experience for consumers. The panoramic video playing device mainly has 4K resolution, and the common devices are head displays, mobile phones and the like. At present, the video decoding and rendering capability of panoramic video playing equipment and the bandwidth in the transmission process are limited, and the panoramic video on demand/live broadcast mostly uses 4K resolution videos with 8M-10M code rate. The video code rate has great influence on the quality of the video, and within a certain range, the higher the video code rate is, the better the video picture quality is. However, due to the bandwidth environment used by users and the requirement of bandwidth saving of partial platforms, the bitrate of the panoramic video is usually compressed to a bitrate level that cannot support 4K resolution. In addition, when a user watches the panoramic video playing device, the view range is limited, the picture watched by the user only occupies a part of the whole video picture, and the user feels quite fuzzy and cannot embody the advantage of 4K resolution. Therefore, in the case of insufficient bitrate and resolution, the experience of the user watching the panoramic video by using the panoramic video playing device needs to be improved.
Disclosure of Invention
In view of the above problems in the prior art, a method, a server, a target device, a system, an apparatus, and a computer-readable storage medium for processing a panoramic video image are provided.
The present invention provides the following.
In a first aspect, a method for processing a panoramic video image is provided, which is applied to a server and includes: acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area; preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out zooming processing on each image block according to the distance between each image block and the hot spot area; coding the preprocessed image to obtain coded data; and transmitting the coded data to a target terminal.
In one possible embodiment, the pre-processing further comprises: determining a target watching direction based on the hot spot area, and dividing the panoramic video image into a plurality of image blocks according to the target watching direction and the range of the human eye visual field angle; and carrying out zooming processing on each image block according to the angle difference between the gaze direction corresponding to each image block and the target gaze direction.
In one possible embodiment, the pre-processing further comprises: the method comprises the steps that a first type of division is carried out on a panoramic video image perpendicular to a central longitudinal section of a spherical panoramic video image to obtain a plurality of first image blocks, projections of a plurality of first type of division lines adopted by the first type of division on the central longitudinal section are gathered at a central point and are separated from each other according to a first preset angle; respectively carrying out height scaling processing on the height dimension of each first image block according to the angle difference between the first class dividing line corresponding to each first image block and the target watching direction; wherein the target gaze direction points to an intersection of a central longitudinal section and a central transverse section of the panoramic video image.
In one possible embodiment, the pre-processing further comprises: performing second type division on the panoramic video image perpendicular to the central transverse section of the spherical panoramic video image to obtain a plurality of second image blocks, wherein projections of a plurality of second type division lines adopted by the second type division on the central transverse section of the panoramic video image are gathered at a central point and are separated from each other according to a second preset angle; and respectively carrying out scaling processing on the width size of each first image block according to the angle difference between the second type dividing line corresponding to each second image block and the hot spot area, wherein the target watching direction points to the intersection point of the central longitudinal section and the central transverse section of the panoramic video image.
In one possible embodiment, the viewpoint to which the target gaze direction points is located at 0 latitude of the longitude and latitude map of the panoramic video image; the pretreatment further comprises: transversely dividing the panoramic video image according to a preset latitude interval to obtain a plurality of transverse image blocks, determining the latitude difference of each transverse image block and a viewpoint on the panoramic video image, and scaling the height size of each transverse image block according to the cosine value of the latitude difference; and/or longitudinally dividing the panoramic video image according to a preset longitude interval to obtain a plurality of longitudinal image blocks, determining the longitude difference of each longitudinal image block and a viewpoint on the panoramic video image, and scaling the width of each longitudinal image block according to the cosine value of the longitude difference.
In one possible embodiment, the pre-processing further comprises: and before division, the panoramic video image is cut according to the target gazing direction and the human eye visual field angle range.
In a possible implementation, the panoramic video image to be processed is a live video image obtained in real time or a recorded video image obtained in advance.
In one possible embodiment, the method further comprises: and preprocessing the panoramic video image according to the resolution level supported by the target terminal so that the preprocessed image meets the resolution level supported by the target terminal.
In a second aspect, a method for processing a panoramic video image is provided, which is applied to a target terminal and includes: receiving encoded data processed by the method of the first aspect; decoding the coded data to obtain decoded data; the decoded data is subjected to a restoration process to at least partially restore the panoramic video image, wherein the restoration process is the inverse of the scaling process of the first aspect.
In one possible embodiment, the method further comprises: and displaying the restored panoramic video image, wherein a hot spot area of the panoramic video image is positioned in a display central area.
In a third aspect, there is provided a server configured to perform the method according to the first aspect, the server comprising: the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a panoramic video image to be processed, and the panoramic video image comprises a hot spot area; a preprocessing unit, configured to preprocess the panoramic video image to obtain a preprocessed image, where the preprocessing includes: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out zooming processing on each image block according to the distance between each image block and the hot spot area; the encoding unit is used for encoding the preprocessed image to obtain encoded data; and a transmitting unit which transmits the encoded data to a target terminal.
In a fourth aspect, there is provided a target terminal configured to perform the method of the second aspect, the target terminal comprising: a receiving unit that receives the encoded data processed as in the first aspect; a decoding unit configured to decode the encoded data to obtain decoded data; a restoration unit that performs restoration processing on the decoded data to at least partially restore the panoramic video image, wherein the restoration processing is inverse to the scaling processing of the first aspect.
In a fifth aspect, a panoramic video image processing system is provided, including: a server configured to perform the method of the first aspect; a target terminal configured to perform the method as in the second aspect.
A sixth aspect provides a panoramic video image processing apparatus comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform: such as the method of the first aspect or such as the method of the second aspect.
A seventh aspect provides a computer-readable storage medium storing a program that, when executed by a multicore processor, causes the multicore processor to perform: the method of the first aspect, or the method of the second aspect.
The embodiment of the application adopts at least one technical scheme which can achieve the following beneficial effects: in this embodiment, by using the above technical solution, on the premise of ensuring the picture definition of the hot spot area in which the user is interested, the bandwidth pressure caused by video data transmission can be reduced without additionally modifying the conventional codec.
It should be understood that the above description is only an overview of the technical solutions of the present invention, so as to clearly understand the technical means of the present invention, and thus can be implemented according to the content of the description. In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.
Drawings
The advantages and benefits described herein, as well as other advantages and benefits, will be apparent to those of ordinary skill in the art upon reading the following detailed description of the exemplary embodiments. The drawings are only for purposes of illustrating exemplary embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like elements throughout. In the drawings:
fig. 1 is a flowchart illustrating a panoramic video image processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a panoramic video image according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating division of a panoramic video image according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a panoramic video image according to another embodiment of the present invention;
FIG. 5 is a schematic diagram of a central longitudinal cross-section of a panoramic video image in accordance with an embodiment of the present invention;
FIG. 6 is a schematic diagram of a central transverse cross-section of a panoramic video image in accordance with an embodiment of the present invention;
FIG. 7 is a schematic diagram illustrating division of a panoramic video image according to another embodiment of the present invention;
FIG. 8 is a schematic structural diagram of a panoramic video image processing apparatus according to a further embodiment of the present invention;
FIG. 9 is a flowchart illustrating a panoramic video image processing method according to another embodiment of the present invention;
fig. 10 is a schematic structural diagram of a panoramic video image processing apparatus according to yet another embodiment of the present invention;
FIG. 11 is a schematic diagram of a panoramic video image processing system according to a further embodiment of the present invention;
fig. 12 is a schematic structural diagram of a panoramic video image processing apparatus according to yet another embodiment of the present invention.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In the description of the embodiments of the present application, it is to be understood that terms such as "including" or "having" are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof disclosed in the specification, and are not intended to preclude the presence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
Unless otherwise stated, "/" indicates an OR meaning, e.g., A/B may indicate A or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone.
The terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first," "second," etc. may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present application, "a plurality" means two or more unless otherwise specified.
All code in this application is exemplary and variations will occur to those skilled in the art based upon the programming language used, the specific needs and personal habits without departing from the spirit of the application.
For clarity of explanation of the embodiments of the present application, some concepts that may appear in subsequent embodiments will first be described.
Resolution ratio: the size or dimension of the video pictures, e.g. 4K resolution, 8K resolution.
dpi: the number of pixel points sampled, displayed, or output per inch of length. The larger the dpi of the display device, the smaller the color point of the image output and the finer the image effect of the output.
Frame rate: the number of frames displayed per unit time.
Code rate: the amount of data of binary data is transmitted or processed per unit time.
Definition: when evaluating a video, the information carried by a unit area of a picture is used to examine the sharpness of the video when the frame rate (the number of pictures displayed per second) and the resolution are fixed.
Longitude and latitude map: namely an isometric bar Projection (ERP). One of the panoramic image projection methods is a two-dimensional panoramic image that can be used for storage and transmission and is obtained by uniformly sampling and mapping spherical signals at equal longitude intervals and equal latitude intervals. The horizontal and vertical coordinates of the image can be expressed by longitude and latitude, the width direction can be expressed by longitude, and the span is 360 degrees; the altitude direction can be expressed by latitude and the span is 180 degrees.
Video decoding (video decoding): and restoring the video code stream into a reconstructed image according to a specific grammar rule and a specific processing method.
Video encoding (video encoding): and compressing the image sequence into a code stream.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
The embodiment of the invention provides a panoramic video processing method, and the inventive concept of the method is introduced firstly.
The content of a panoramic video used in a scene such as a theater, a concert, a ball game, etc. has certain characteristics, such as that the watching attention of a user is focused on a hot spot area (such as a stage, a court, etc.) and the attention of other non-hot spot areas (such as an auditorium) in the panoramic video is not high. Based on this, an embodiment of the present invention provides a panoramic video image processing method, which performs a series of preprocessing on an original 360-degree panoramic video, so as to retain a picture with a certain resolution centered on a hot spot area, and zoom or remove content with low importance. Therefore, the scheme can reduce the resolution of the panoramic video image on the premise of not losing the image quality and ensuring the image definition of a hot spot area which is used by a user to watch habits, thereby reducing the bandwidth pressure caused by video data transmission on the one hand and being compatible with panoramic video playing equipment (such as a common television, 4K VR glasses and the like) with insufficient video decoding capacity on the other hand.
Those skilled in the art will appreciate that the described application scenario is only one example in which an embodiment of the present invention may be implemented. The scope of applicability of the embodiments of the present invention is not limited in any way. Having described the general principles of the invention, various non-limiting embodiments of the invention are described in detail below.
Fig. 1 is a schematic flowchart of a panoramic video image processing method according to an embodiment of the present application, configured to perform data processing on a panoramic video image to be played, where from a device perspective, an execution subject may be one or more electronic devices, and more specifically, may be a processing module of a server; from the program perspective, the execution main body may accordingly be a program loaded on these electronic devices. In this embodiment, the execution subject of the method may be the server in the embodiment shown in fig. 1.
As shown in fig. 1, the method provided by the present embodiment may include the following steps S11-S14.
S11, acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area;
the panoramic video image refers to a frame image of the panoramic video. A panoramic video may also be referred to as a 360-degree panoramic video or a 360-degree video, and is a video that is photographed in all directions at 360 degrees by a plurality of cameras. As will be appreciated by those skilled in the art, the panoramic video image may be. Referring to fig. 3, the panoramic video image may be generally displayed on the spherical surface of a sphere, and the image area viewable by the user is the position on the spherical surface. The panoramic video mentioned in this embodiment may be a live video or a recorded video, and this application is not limited to this specifically.
The hot spot area refers to a content area with high interest level in the panoramic video image. For example, in a scene such as a stage drama, a ball game, etc., a stage area and a field area of a panoramic video image may be defined as a hot spot area, an auditorium area, a walkway area, etc. of the panoramic video image may be defined as a non-hot spot area, and the user's interest level in the hot spot area may be significantly higher than that in the non-hot spot area.
In an example, referring to fig. 2, a spherical panoramic video image 3 is shown, in the panoramic video image, a hot spot area 31 is included, a user may watch the hot spot area 31 when watching the panoramic video image 3, a center position or a vicinity of the hot spot area 31 may be used as a viewpoint 32, and a direction pointing from a spherical center point O of the panoramic video image to the viewpoint is a target watching direction, that is, an optimal watching direction for the user to watch the panoramic video image.
S12, preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out scaling processing on each image block according to the distance between each image block and the hot spot area.
In this embodiment, in order to reduce the resolution of the panoramic video image, before the panoramic video image is encoded, scaling processing needs to be performed on the panoramic video image, where the scaling processing is differential scaling processing performed on a plurality of image blocks divided from the panoramic video image, that is, in this embodiment, the panoramic video image is not entirely scaled to a certain size, but different image blocks may be differentially scaled to different scaling degrees. For example, the panoramic video image may be sequentially divided into N regions according to the distance from the hot spot region, where the 1 st region is closest to the hot spot region and the nth region is farthest from the hot spot region, so that the divided N regions may be subjected to differential scaling processing, and the scaling degree is higher as the distance is longer. Preferably, for one or more image patches including the hot spot region, no scaling or scaling to a lesser extent may be performed in order to guarantee the resolution of the hot spot region. Of course, since there may be two or more image areas with equal or similar distances from the hot spot region, the scaling degrees of the two or more image areas may be the same. The spherical panoramic video image and each divided image block can be projected into a 2-dimensional panoramic video image, and then the width and/or height of each image block on the 2-dimensional panoramic video image are/is zoomed in and out to different degrees.
The panoramic video image can be divided into a plurality of image blocks in any dividing mode. For example, the spherical surface of the panoramic video image can be divided horizontally and longitudinally according to the longitude and latitude to obtain a plurality of image blocks. For another example, the panoramic video image may be divided in a single direction to obtain a plurality of image blocks, such as only horizontal division or only vertical division. For another example, the panoramic video image may be divided into a plurality of image blocks with equal size based on an inscribed regular polyhedron with a spherical surface. And so, this is not particularly limited by this application.
The distance between each image block and the hot spot region is an index for evaluating the distance between the image block and the hot spot region, and may be a spherical distance between each image block and the hot spot region, such as a minimum/maximum/average spherical distance, a linear distance between each image block and the hot spot region, such as a minimum/maximum/average linear distance, or an angular distance (also referred to as an angular interval) between a gaze direction corresponding to each image block and a target gaze viewpoint corresponding to the hot spot region, which is not specifically limited in the present application.
After the panoramic video image is preprocessed, a preprocessed image can be obtained, the overall resolution of the preprocessed image is lower than that of the original panoramic video image, and the preprocessed image can still retain the picture resolution at the hot spot area to a certain extent.
S13, coding the preprocessed image to obtain coded data;
and S14, transmitting the coded data to the target terminal.
In order to transmit the preprocessed panoramic video image to the target terminal for displaying the video image, the preprocessed image needs to be encoded by using a preset video encoding format, for example, the preprocessed image may be encoded by using a conventional video encoder such as x264, x265, etc. It can be understood by those skilled in the art that in the design of the official x264, x265 and other encoders, the importance of each pixel of the video picture is the same, the code rate assignment for each pixel has no difference of the region weight, and if different region pixels need to be encoded differently, the conventional encoder needs to be modified additionally to implement the difference of the code rate weight for different region pixels. However, in this embodiment, since the encoding object of the video encoder has already been subjected to the preprocessing as described in S12, the differential resolution representation of different regions of the original panoramic video image can be implemented in the output encoded data without modifying the conventional encoder used, so as to achieve the purposes of reducing the data amount of the panoramic video, lowering the video resolution, and compressing the video bitrate.
In this embodiment, by using the above technical solution, on the premise of ensuring the picture resolution of the hot spot area in which the user is interested, the bandwidth pressure caused by video data transmission can be reduced without additionally modifying the conventional encoder.
In one possible embodiment, due to the limited resolution level supported by the target terminal, there may be a case of incompatible panoramic video images, based on which the panoramic video images may be preprocessed as described above according to the resolution level supported by the target terminal, so that the preprocessed images satisfy the resolution level supported by the target terminal.
In one example, assuming that the original panoramic video image is an 8K video image, the widths and heights of the 8K video images are W, H respectively, and the target terminal only supports a 4K video image, in order to enable the target terminal to support playing the panoramic video image, the panoramic video image may be preprocessed so that the video size of the preprocessed image obtained after preprocessing is 0.5W, 0.5H. In other words, the 8K video is processed into the 4K video, so that the target terminal which cannot decode the 8K video can be compatible, and the user experience is improved. The application may convert any high-resolution video into any low-resolution video through the above preprocessing, and the above 8K video into 4K video is only an example, and the application is not limited thereto.
In one possible embodiment, in S12, the preprocessing further includes: determining a target watching direction based on the hot spot area, and dividing the panoramic video image into a plurality of image blocks according to the target watching direction and the range of the human eye visual field angle; and carrying out zooming processing on each image block according to the angle difference between the gaze direction corresponding to each image block and the target gaze direction.
In an example, referring to fig. 3, the center position of the hotspot region may be taken as a viewpoint 32, a direction in which a spherical center point of the panoramic video image points to the viewpoint 32 is taken as a target gazing direction, N-1 annular division lines may be determined on the surface of the sphere with the viewpoint 32 as a vertex, and the panoramic video image may be divided into 1 st to N th regions from right to left. The minimum angle interval between the target gaze direction and the gaze direction corresponding to each image block may be adopted, wherein the minimum angle interval between the target gaze direction and the 1 st to nth regions is 0 degree, i degree, 2i degree, …, (N-1) i degree, respectively. The cosine value of the included angle may be used as a distance index for evaluating each image block and the hot spot region, and the cosine value of the included angle is distributed between (-1, 1), wherein the scaling factor S of the 1 st region may be the cosine value 1, the picture may be substantially unchanged, and the scaling factors S of the remaining regions may gradually decrease until approaching 0 as the cosine value decreases.
In this embodiment, the scaling factor S is a size ratio of each image block after scaling processing to the original image block. Preferably, the scaling factor S may include a width scaling factor sw in the image width direction and a height scaling factor sh in the image height direction, and let S ═ sw × sh, the scaling factors may be uniformly or non-uniformly distributed over sw and sh.
If the resolution of the original panoramic video image is W, H and the original panoramic video image is divided into n image blocks, the resolution of the video after scaling processing is (sw1+ sw2+ sw3+ sw4+. + swn) × W, (sh1+ sh2+ sh3+ sh4+. + shn) × H. Wherein, (sw1, sh1) are the width scaling factor and the height scaling factor of the 1 st image block, respectively, and so on.
In one possible embodiment, in S12, the preprocessing may include: performing first-class division on the panoramic video image perpendicular to a central longitudinal section of the spherical panoramic video image to obtain a plurality of first image blocks, wherein projections of a plurality of first-class division lines adopted by the first-class division on the central longitudinal section are gathered at a central point and are separated from each other according to a first preset angle alpha; respectively carrying out height scaling processing on the height dimension of each first image block according to the angle difference between the first class dividing line corresponding to each first image block and the target watching direction; wherein the target gaze direction points to an intersection of a central longitudinal section and a central transverse section of the panoramic video image.
Referring to fig. 4, a central longitudinal section 41 and a central transverse section 42 of the panoramic video image of fig. 2 are shown, wherein the viewpoint 32 is located at the intersection of the central longitudinal section 41 and the central transverse section.
Referring to fig. 5, a schematic diagram of a central longitudinal section 41 of a panoramic video image is shown, wherein on the central longitudinal section 41, the panoramic video image is divided into a plurality of image blocks in the longitudinal direction by rotating around a central point O according to a first preset angle α with a target gazing direction as an angle of 0 degree, wherein the image blocks with larger angle difference from the target gazing direction have larger height scaling degree. Likewise, the cosine value of the angular difference may be used to determine the height scaling factor sh for each first image partition.
For example, in fig. 5, the central longitudinal section 41 is divided into 12 parts, and when the height of each first image block is scaled, it is assumed that the size of each first image block is w × h, and the scaled size is w × h (h × sh), where sh is the height scaling factor (sh < 1). In one example, the height scaling factor of each first image block may be calculated according to cosine values of angles between the projection of the first class dividing line of each first image block on the central longitudinal section 41 and the target gaze direction, and the minimum angular intervals between the H1 th region to the H12 th region and the target gaze direction are 0 degrees, 30 degrees, …, 150 degrees, 210 degrees …, 360 degrees, respectively. The cosine values of the included angles can be used as distance indexes for evaluating each image block and a hot spot region, the cosine values of the included angles are distributed between (-1, 1), for H1, H2, … and H6 regions, the cosine values of the included angles are distributed between (-1, 1), sh6< … < sh1, the compressed degree of the region H6 is the highest after zooming, the region H1 is almost not compressed, the original image quality is kept, and the like, the region H7 is the region H12, and the sh7< … < sh 12.
It should be noted that the first preset angle may also be a set of a plurality of preset angles, and thus, the plurality of first-class dividing lines may perform non-uniform first-class division according to the plurality of preset angles.
In one possible embodiment, in S12, the preprocessing may include: performing second type division on the panoramic video image perpendicular to the central transverse section of the spherical panoramic video image to obtain a plurality of second image blocks, wherein projections of a plurality of second type division lines adopted by the second type division on the central transverse section of the panoramic video image are gathered at a central point and are separated from each other according to a second preset angle beta; and respectively carrying out scaling processing on the width size of each first image block according to the angle difference between the second type dividing line corresponding to each second image block and the hot spot area, wherein the target watching direction points to the intersection point of the central longitudinal section and the central transverse section of the panoramic video image.
It should be noted that the second preset angle may also be a set of a plurality of preset angles, and thus, the plurality of second-class dividing lines may perform non-uniform second-class division according to the plurality of preset angles.
Still referring to fig. 4, a center longitudinal section 41 and a center transverse section 42 of the panoramic video image of fig. 2 are shown, wherein the viewpoint 32 is located at the intersection of the center longitudinal section 41 and the center transverse section 42.
Referring to fig. 6, a schematic diagram of a central transverse section 42 is shown, on which a rotation around a central point according to a second preset angle β can be performed starting from a target gaze direction, so as to divide the panoramic video image into M parts in the transverse direction, wherein the larger the angular difference from the target gaze direction, the larger the scaling degree of the width of the region. Likewise, the cosine values of the angular differences may be used to determine the width scaling factor sw for each second image tile.
In one possible implementation, the above-described combination of the first type of division with the height scaling process and the second type of division with the width scaling process may of course be performed on the panoramic video image.
For example, if the resolution of the original panoramic video image is W, H, and n image blocks are obtained through the first-class division and the second-class division, the resolution of the video after scaling processing is (sw1+ sw2+ sw3+ sw4+.. + swn) × W, (sh1+ sh2+ sh3+ sh4+.. + shn) × H. Wherein, (swn, shn) are the width scaling factor and the height scaling factor of the nth image block, and so on.
In one possible embodiment, the preprocessing in S12 may further include: transversely dividing the panoramic video image according to a preset latitude interval gamma to obtain a plurality of transverse image blocks, determining the latitude difference of each transverse image block and a viewpoint on the panoramic video image, and scaling the height size of each transverse image block according to the cosine value of the latitude difference; and/or longitudinally dividing the panoramic video image according to a preset longitude interval delta to obtain a plurality of longitudinal image blocks, determining the longitude difference of each longitudinal image block and a viewpoint on the panoramic video image, and scaling the width of each longitudinal image block according to the cosine value of the longitude difference.
In an implementation manner, referring to fig. 7, a longitude and latitude map of a panoramic video image is shown, wherein a viewpoint 32 pointed by a target gaze direction is located at 0 latitude of the longitude and latitude map, a plurality of division lines adopted for transverse division are a plurality of preset dimension lines of a spherical panoramic video image, the preset dimension lines are separated according to a preset latitude interval γ, the higher the latitude is, the larger the zoom degree of a transverse image block is, and a zoom factor of high zoom can be determined according to a cosine value of the latitude of the image block. The plurality of division lines adopted by the longitudinal division are a plurality of preset longitude lines of the spherical panoramic video image, the preset longitude lines are separated according to a preset longitude interval delta, the larger the longitude difference of the viewpoint on the panoramic video image is, the larger the scaling degree of a longitudinal image block is, and the scaling factor of the width scaling can be determined according to the cosine value of the longitude difference.
Alternatively, when dividing transversely, the plurality of preset latitudinal lines may also be arranged at different latitudinal intervals, for example, the transverse division interval may be larger as the latitude is higher. Alternatively, the plurality of preset longitude lines may be arranged at different longitude intervals when divided longitudinally, for example, the longitude interval may be larger as the distance from the target gaze direction is farther.
Alternatively, in S12, the preprocessing may include only the above transverse division or only the above longitudinal division or a combination of both.
In some possible embodiments, in order to further reduce the resolution of the panoramic video image, in S12, the preprocessing may further include: and cutting the panoramic video image according to the target watching direction and the view angle range.
As will be understood by those skilled in the art, limited to the visual field angle range of human eyes, the user can only see the contents of a part of the area of the panoramic video image, and therefore, the picture area which is too much beyond the visual field angle range of human eyes compared with the target gaze direction can be preprocessed in a clipping manner.
In one example, assuming that the human eye visual field angle range is 90 degrees at maximum in the longitudinal direction and 180 degrees at maximum in the lateral direction, a screen region on the panoramic video image having a difference in the longitudinal angle from the target gaze direction of more than 45 degrees (or more) may be truncated, and a screen region on the panoramic video image having a difference in the lateral angle from the target gaze direction of more than 90 degrees (or more) may be truncated.
In the description of the present specification, reference to the description of the terms "some possible implementations," "some embodiments," "examples," "specific examples," or "some examples," or the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
With regard to the method flow diagrams of embodiments of the present application, certain operations are described as different steps performed in a certain order. Such flow diagrams are illustrative and not restrictive. Certain steps described herein may be grouped together and performed in a single operation, may be divided into multiple sub-steps, and may be performed in an order different than that shown herein. The various steps shown in the flowcharts may be implemented in any way by any circuit structure and/or tangible mechanism (e.g., by software running on a computer device, hardware (e.g., logical functions implemented by a processor or chip), etc., and/or any combination thereof).
Based on the same technical concept, an embodiment of the present invention further provides a server configured to execute the method according to the above embodiment, and referring to fig. 8, the server 80 includes: an obtaining unit 801, configured to obtain a panoramic video image to be processed, where the panoramic video image includes a hot spot area; the preprocessing unit 802 performs preprocessing on the panoramic video image to obtain a preprocessed image, where the preprocessing includes: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out zooming processing on each image block according to the distance between each image block and the hot spot area; an encoding unit 803 that performs encoding processing on the preprocessed image to obtain encoded data; transmitting section 804 transmits encoded data to target terminal
It should be noted that the apparatus in the embodiment of the present application may implement each process of the foregoing method embodiment, and achieve the same effect and function, which are not described herein again.
Based on the same technical concept, an embodiment of the present invention further provides a panoramic video image processing method, which is applied to a target terminal such as a VR device, and fig. 9 shows a flowchart of the panoramic video image processing method.
As shown in fig. 9, the method includes:
s91, receiving coded data; the encoded data is processed using the methods described in the above embodiments.
S92, decoding the coded data to obtain decoded data;
s93, performing a restoration process on the decoded data to at least partially restore the panoramic video image, wherein the restoration process is the inverse of the scaling process described in the above embodiments.
At least the zoomed panoramic video image can be restored to obtain the size before zooming.
In one possible implementation, the method may further include: and displaying the restored panoramic video image, wherein a hot spot area of the panoramic video image is positioned in a display central area.
Based on the same technical concept, an embodiment of the present invention further provides a target terminal configured to perform the panoramic video image processing method as described above, as shown in fig. 10, the target terminal includes: a receiving unit 101 that receives encoded data processed as in the first aspect; a decoding unit 102 that decodes the encoded data to obtain decoded data; a restoration unit 103 that performs restoration processing on the decoded data to at least partially restore the panoramic video image, wherein the restoration processing is the inverse of the scaling processing described in the above embodiments.
Based on the same technical concept, an embodiment of the present invention further provides a panoramic video image processing system 110, as shown in fig. 11, including: a server 80 configured to perform the method of fig. 1; a target terminal 100 configured to perform the method as in fig. 9.
It should be noted that the apparatus and system in the embodiment of the present application may implement each process of the foregoing embodiment of the method, and achieve the same effect and function, which are not described herein again.
Fig. 12 is a panoramic video image processing apparatus according to an embodiment of the present application, configured to execute the panoramic video image processing method shown in fig. 1, and the apparatus includes: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the above embodiments.
According to some embodiments of the present application, there is provided a non-transitory computer storage medium of a panoramic video image processing method having stored thereon computer-executable instructions configured to, when executed by a processor, perform: the method as described in the above example.
The embodiments in the present application are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, device, and computer-readable storage medium embodiments, the description is simplified because they are substantially similar to the method embodiments, and reference may be made to some descriptions of the method embodiments for their relevance.
The apparatus, the device, and the computer-readable storage medium provided in the embodiment of the present application correspond to the method one to one, and therefore, the apparatus, the device, and the computer-readable storage medium also have advantageous technical effects similar to those of the corresponding method.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. Moreover, while the operations of the method of the invention are depicted in the drawings in a particular order, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
While the spirit and principles of the invention have been described with reference to several particular embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, nor is the division of aspects, which is for convenience only as the features in such aspects may not be combined to benefit. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (15)
1. A panoramic video image processing method is applied to a server and comprises the following steps:
acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area;
preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out zooming processing on each image block according to the distance between each image block and the hot spot area;
coding the preprocessed image to obtain coded data;
and transmitting the coded data to a target terminal.
2. The method of claim 1, wherein the pre-processing further comprises:
determining a target gazing direction based on the hot spot area, and dividing the panoramic video image into a plurality of image blocks according to the target gazing direction and the human eye visual field angle range; and the number of the first and second groups,
and zooming each image block according to the angle difference between the gazing direction corresponding to each image block and the target gazing direction.
3. The method of claim 2, wherein the pre-processing further comprises:
performing first-class division on the panoramic video image perpendicular to a central longitudinal section of the spherical panoramic video image to obtain a plurality of first image blocks, wherein projections of a plurality of first-class division lines adopted by the first-class division on the central longitudinal section are gathered at a central point and are separated from each other according to a first preset angle alpha;
respectively carrying out height scaling processing on the height dimension of each first image block according to the angle difference between the first class dividing line corresponding to each first image block and the target watching direction; wherein the target gaze direction points to an intersection of a central longitudinal section and a central transverse section of the panoramic video image.
4. The method of claim 1 or 3, wherein the pre-processing further comprises:
performing second type division on the panoramic video image perpendicular to the central transverse section of the spherical panoramic video image to obtain a plurality of second image blocks, wherein projections of a plurality of second type division lines adopted by the second type division on the central transverse section of the panoramic video image are gathered at a central point and are separated from each other according to a second preset angle beta;
and respectively carrying out scaling processing on the width size of each first image block according to the angle difference between the second type dividing line corresponding to each second image block and the hot spot area, wherein the target watching direction points to the intersection point of the central longitudinal section and the central transverse section of the panoramic video image.
5. The method of claim 2, wherein the viewpoint to which the target gaze direction is directed is located at 0 latitude of a longitude and latitude map of the panoramic video image; the pre-treatment further comprises:
transversely dividing the panoramic video image according to a preset latitude interval to obtain a plurality of transverse image blocks, determining the latitude difference of each transverse image block and the viewpoint on the panoramic video image, and scaling the height size of each transverse image block according to the cosine value of the latitude difference; and/or the presence of a gas in the gas,
and longitudinally dividing the panoramic video image according to a preset longitudinal interval to obtain a plurality of longitudinal image blocks, determining the longitudinal difference between each longitudinal image block and the viewpoint on the panoramic video image, and scaling the width of each longitudinal image block according to the cosine value of the longitudinal difference.
6. The method of claim 1, wherein the pre-processing further comprises:
and cutting the panoramic video image according to the target gazing direction and the human eye visual field angle range before the division.
7. The method of claim 1, wherein the panoramic video image to be processed is a live video image obtained in real time or a recorded video image obtained in advance.
8. The method of claim 1, further comprising:
and performing the preprocessing on the panoramic video image according to the resolution level supported by the target terminal so that the preprocessed image meets the resolution level supported by the target terminal.
9. A panoramic video image processing method is applied to a target terminal, and comprises the following steps:
receiving encoded data processed according to the method of any one of claims 1-8;
decoding the coded data to obtain decoded data;
a restoration process of the decoded data to at least partially restore the panoramic video image, wherein the restoration process is inverse to the scaling process in the method of any one of claims 1-8.
10. The method of claim 9, further comprising:
and displaying the restored panoramic video image, wherein a hot spot area of the panoramic video image is positioned in a display central area.
11. A server configured to perform the method of any one of claims 1-8, the server comprising:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a panoramic video image to be processed, and the panoramic video image comprises a hot spot area;
a preprocessing unit, configured to preprocess the panoramic video image to obtain a preprocessed image, where the preprocessing includes: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out zooming processing on each image block according to the distance between each image block and the hot spot area;
the encoding unit is used for encoding the preprocessed image to obtain encoded data;
and a transmitting unit which transmits the encoded data to a target terminal.
12. A target terminal configured to perform the method of any one of claims 9-10, the target terminal comprising:
a receiving unit, receiving the encoded data processed by the method of any one of claims 1-8;
a decoding unit configured to decode the encoded data to obtain decoded data;
a restoration unit that performs restoration processing on the decoded data to at least partially restore the panoramic video image, wherein the restoration processing is inverse to the scaling processing of any one of claims 1 to 8.
13. A panoramic video image processing system, comprising:
a server configured to perform the method of any one of claims 1-8;
a target terminal configured to perform the method of any one of claims 9-10.
14. A panoramic video image processing apparatus, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform: the method of any one of claims 1-8, or the method of any one of claims 9-10.
15. A computer-readable storage medium storing a program that, when executed by a multi-core processor, causes the multi-core processor to perform: the method of any one of claims 1-8, or the method of any one of claims 9-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110678889.9A CN113630622B (en) | 2021-06-18 | 2021-06-18 | Panoramic video image processing method, server, target equipment, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110678889.9A CN113630622B (en) | 2021-06-18 | 2021-06-18 | Panoramic video image processing method, server, target equipment, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113630622A true CN113630622A (en) | 2021-11-09 |
| CN113630622B CN113630622B (en) | 2024-04-26 |
Family
ID=78378152
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110678889.9A Active CN113630622B (en) | 2021-06-18 | 2021-06-18 | Panoramic video image processing method, server, target equipment, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113630622B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114143602A (en) * | 2021-12-06 | 2022-03-04 | 深圳创维数字技术有限公司 | Panoramic picture playing method and device, panoramic playing server and storage medium |
Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104567708A (en) * | 2015-01-06 | 2015-04-29 | 浙江工业大学 | Tunnel full-section high-speed dynamic health detection device and method based on active panoramic vision |
| US20150172545A1 (en) * | 2013-10-03 | 2015-06-18 | Flir Systems, Inc. | Situational awareness by compressed display of panoramic views |
| CN104735464A (en) * | 2015-03-31 | 2015-06-24 | 华为技术有限公司 | Panorama video interactive transmission method, server and client end |
| US20160260196A1 (en) * | 2015-03-05 | 2016-09-08 | Nokia Technologies Oy | Video streaming method |
| CN106412582A (en) * | 2016-10-21 | 2017-02-15 | 北京大学深圳研究生院 | Panoramic video region of interest description method and coding method |
| CN106791854A (en) * | 2016-11-22 | 2017-05-31 | 北京疯景科技有限公司 | Image Coding, coding/decoding method and device |
| CN107622474A (en) * | 2017-09-26 | 2018-01-23 | 北京大学深圳研究生院 | Panoramic video mapping method based on main view point |
| CN108063946A (en) * | 2017-11-16 | 2018-05-22 | 腾讯科技(成都)有限公司 | Method for encoding images and device, storage medium and electronic device |
| WO2018107467A1 (en) * | 2016-12-16 | 2018-06-21 | Thomson Licensing | Apparatus, method, and storage medium for providing preview of restricted video asset |
| CN108322727A (en) * | 2018-02-28 | 2018-07-24 | 北京搜狐新媒体信息技术有限公司 | A kind of panoramic video transmission method and device |
| CN108513119A (en) * | 2017-02-27 | 2018-09-07 | 阿里巴巴集团控股有限公司 | Mapping, processing method, device and the machine readable media of image |
| US20180310010A1 (en) * | 2017-04-20 | 2018-10-25 | Nokia Technologies Oy | Method and apparatus for delivery of streamed panoramic images |
| CN109194923A (en) * | 2018-10-18 | 2019-01-11 | 眸芯科技（上海）有限公司 | Video image processing apparatus, system and method based on non-uniformed resolution ratio |
| CN109327699A (en) * | 2017-07-31 | 2019-02-12 | 华为技术有限公司 | A kind of processing method of image, terminal and server |
| CN109362242A (en) * | 2016-10-10 | 2019-02-19 | 华为技术有限公司 | A kind of processing method and processing device of video data |
| CN109429062A (en) * | 2017-08-22 | 2019-03-05 | 阿里巴巴集团控股有限公司 | Treating method and apparatus, image encoding method and the device of pyramid model |
| CN110572656A (en) * | 2019-09-19 | 2019-12-13 | 北京视博云科技有限公司 | coding method, image processing method, device and system |
| CN110956583A (en) * | 2018-09-26 | 2020-04-03 | 华为技术有限公司 | Spherical image processing method and device and server |
| CN111726640A (en) * | 2020-07-03 | 2020-09-29 | 中图云创智能科技(北京)有限公司 | Live broadcast method with 0-360 degree dynamic viewing angle |
| CN111885310A (en) * | 2020-08-31 | 2020-11-03 | 深圳市圆周率软件科技有限责任公司 | Panoramic data processing method, processing equipment and playing equipment |
| CN111903118A (en) * | 2018-02-22 | 2020-11-06 | 弗劳恩霍夫应用研究促进协会 | Generating panoramic images |
| US10893043B1 (en) * | 2018-09-12 | 2021-01-12 | Massachusetts Mutual Life Insurance Company | Systems and methods for secure display of data on computing devices |
-
2021
- 2021-06-18 CN CN202110678889.9A patent/CN113630622B/en active Active
Patent Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150172545A1 (en) * | 2013-10-03 | 2015-06-18 | Flir Systems, Inc. | Situational awareness by compressed display of panoramic views |
| CN104567708A (en) * | 2015-01-06 | 2015-04-29 | 浙江工业大学 | Tunnel full-section high-speed dynamic health detection device and method based on active panoramic vision |
| US20160260196A1 (en) * | 2015-03-05 | 2016-09-08 | Nokia Technologies Oy | Video streaming method |
| CN104735464A (en) * | 2015-03-31 | 2015-06-24 | 华为技术有限公司 | Panorama video interactive transmission method, server and client end |
| CN109362242A (en) * | 2016-10-10 | 2019-02-19 | 华为技术有限公司 | A kind of processing method and processing device of video data |
| CN106412582A (en) * | 2016-10-21 | 2017-02-15 | 北京大学深圳研究生院 | Panoramic video region of interest description method and coding method |
| CN106791854A (en) * | 2016-11-22 | 2017-05-31 | 北京疯景科技有限公司 | Image Coding, coding/decoding method and device |
| WO2018107467A1 (en) * | 2016-12-16 | 2018-06-21 | Thomson Licensing | Apparatus, method, and storage medium for providing preview of restricted video asset |
| CN108513119A (en) * | 2017-02-27 | 2018-09-07 | 阿里巴巴集团控股有限公司 | Mapping, processing method, device and the machine readable media of image |
| US20180310010A1 (en) * | 2017-04-20 | 2018-10-25 | Nokia Technologies Oy | Method and apparatus for delivery of streamed panoramic images |
| CN109327699A (en) * | 2017-07-31 | 2019-02-12 | 华为技术有限公司 | A kind of processing method of image, terminal and server |
| CN109429062A (en) * | 2017-08-22 | 2019-03-05 | 阿里巴巴集团控股有限公司 | Treating method and apparatus, image encoding method and the device of pyramid model |
| CN107622474A (en) * | 2017-09-26 | 2018-01-23 | 北京大学深圳研究生院 | Panoramic video mapping method based on main view point |
| CN108063946A (en) * | 2017-11-16 | 2018-05-22 | 腾讯科技(成都)有限公司 | Method for encoding images and device, storage medium and electronic device |
| CN111903118A (en) * | 2018-02-22 | 2020-11-06 | 弗劳恩霍夫应用研究促进协会 | Generating panoramic images |
| CN108322727A (en) * | 2018-02-28 | 2018-07-24 | 北京搜狐新媒体信息技术有限公司 | A kind of panoramic video transmission method and device |
| US10893043B1 (en) * | 2018-09-12 | 2021-01-12 | Massachusetts Mutual Life Insurance Company | Systems and methods for secure display of data on computing devices |
| CN110956583A (en) * | 2018-09-26 | 2020-04-03 | 华为技术有限公司 | Spherical image processing method and device and server |
| CN109194923A (en) * | 2018-10-18 | 2019-01-11 | 眸芯科技（上海）有限公司 | Video image processing apparatus, system and method based on non-uniformed resolution ratio |
| CN110572656A (en) * | 2019-09-19 | 2019-12-13 | 北京视博云科技有限公司 | coding method, image processing method, device and system |
| CN111726640A (en) * | 2020-07-03 | 2020-09-29 | 中图云创智能科技(北京)有限公司 | Live broadcast method with 0-360 degree dynamic viewing angle |
| CN111885310A (en) * | 2020-08-31 | 2020-11-03 | 深圳市圆周率软件科技有限责任公司 | Panoramic data processing method, processing equipment and playing equipment |
Non-Patent Citations (2)
| Title |
|---|
| CHI-CHENG JU: "A panoramic video system by direct manipulation video navigation", 《 2014 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO WORKSHOPS (ICMEW)》, 8 September 2014 (2014-09-08) * |
| 张垚垚: "全景图拼接技术的研究", 《 中国优秀硕士论文电子期刊网》, 15 April 2006 (2006-04-15) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114143602A (en) * | 2021-12-06 | 2022-03-04 | 深圳创维数字技术有限公司 | Panoramic picture playing method and device, panoramic playing server and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113630622B (en) | 2024-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112204993B (en) | Adaptive panoramic video streaming using overlapping partitioned segments | |
| US10244200B2 (en) | View-dependent operations during playback of panoramic video | |
| US11483475B2 (en) | Adaptive panoramic video streaming using composite pictures | |
| US10574955B2 (en) | Re-projecting flat projections of pictures of panoramic video for rendering by application | |
| JP6501904B2 (en) | Spherical video streaming | |
| US6559846B1 (en) | System and process for viewing panoramic video | |
| CN108063976B (en) | Video processing method and device | |
| US11138460B2 (en) | Image processing method and apparatus | |
| CN112188303A (en) | VR (virtual reality) streaming media playing method and device based on visual angle | |
| US20190104330A1 (en) | Method, apparatus and stream of formatting an immersive video for legacy and immersive rendering devices | |
| US20210258590A1 (en) | Switchable scalable and multiple description immersive video codec | |
| US11270413B2 (en) | Playback apparatus and method, and generation apparatus and method | |
| CN114339412B (en) | Video quality enhancement method, mobile terminal, storage medium and device | |
| CN113630622B (en) | Panoramic video image processing method, server, target equipment, device and system | |
| CN110933461B (en) | Image processing method, device, system, network equipment, terminal and storage medium | |
| CN112533005A (en) | Interaction method and system for VR video slow live broadcast | |
| Řeřábek et al. | JPEG backward compatible coding of omnidirectional images | |
| CN112130667A (en) | Interaction method and system for ultra-high definition VR (virtual reality) video | |
| CN111212267A (en) | Partitioning method of panoramic image and server | |
| US20230196700A1 (en) | Image processing apparatus and image processing method | |
| CN116916071A (en) | Video picture display method, system, device, electronic equipment and storage medium | |
| US20210203995A1 (en) | Video decoding method for decoding bitstream to generate projection-based frame with guard band type specified by syntax element signaling | |
| WO2023129214A1 (en) | Methods and system of multiview video rendering, preparing a multiview cache, and real-time multiview video conversion | |
| Groth et al. | Wavelet-Based Fast Decoding of 360-Degree Videos | |
| Sauer | Efficient coding tools for 360° video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |