CN113630622B - Panoramic video image processing method, server, target equipment, device and system - Google Patents
Panoramic video image processing method, server, target equipment, device and system Download PDFInfo
- Publication number
- CN113630622B CN113630622B CN202110678889.9A CN202110678889A CN113630622B CN 113630622 B CN113630622 B CN 113630622B CN 202110678889 A CN202110678889 A CN 202110678889A CN 113630622 B CN113630622 B CN 113630622B
- Authority
- CN
- China
- Prior art keywords
- panoramic video
- video image
- image
- hot spot
- preprocessing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 78
- 238000007781 pre-processing Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 33
- 230000008569 process Effects 0.000 claims description 21
- 238000003860 storage Methods 0.000 claims description 17
- 230000000007 visual effect Effects 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 7
- 230000002829 reductive effect Effects 0.000 abstract description 5
- 230000004048 modification Effects 0.000 abstract description 4
- 238000012986 modification Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 230000008901 benefit Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- YBJHBAHKTGYVGT-ZKWXMUAHSA-N (+)-Biotin Chemical compound N1C(=O)N[C@@H]2[C@H](CCCCC(=O)O)SC[C@@H]21 YBJHBAHKTGYVGT-ZKWXMUAHSA-N 0.000 description 3
- 230000000670 limiting effect Effects 0.000 description 3
- FEPMHVLSLDOMQC-UHFFFAOYSA-N virginiamycin-S1 Natural products CC1OC(=O)C(C=2C=CC=CC=2)NC(=O)C2CC(=O)CCN2C(=O)C(CC=2C=CC=CC=2)N(C)C(=O)C2CCCN2C(=O)C(CC)NC(=O)C1NC(=O)C1=NC=CC=C1O FEPMHVLSLDOMQC-UHFFFAOYSA-N 0.000 description 3
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention provides a panoramic video image processing method, a server, target equipment, a device and a system, wherein the method is applied to the server and comprises the following steps: acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area; preprocessing a panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises dividing the panoramic video image into a plurality of image blocks and respectively performing scaling processing on each image block according to the distance between each image block and a hot spot area; encoding the preprocessed image to obtain encoded data; the encoded data is transmitted to the target terminal. By using the method, on the premise of ensuring the picture resolution of the hot spot area of interest to the user, the bandwidth pressure caused by video data transmission can be reduced without additional modification to a conventional encoder.
Description
Technical Field
The invention belongs to the field of video processing, and particularly relates to a panoramic video image processing method, a server, target equipment, a device and a system.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
The appearance of panoramic video images gives consumers a better visual experience. Panoramic video playing equipment mainly comprises 4K resolution, and common equipment comprises a head display, a mobile phone and the like. Currently, panoramic video on demand/live broadcasting is mostly video with 4K resolution using 8M-10M code rate, which is limited by video decoding and rendering capabilities of panoramic video playing equipment and bandwidth in the transmission process. The video code rate has great influence on the quality of the video, and the higher the code rate of the video is, the better the video picture quality is within a certain range. However, due to the bandwidth environment used by the user and the requirement of saving bandwidth for part of the platform, the code rate of the panoramic video is generally compressed to a code rate level that cannot support 4K resolution. In addition, when the user watches the panoramic video playing device, the user watches the picture only accounting for a part of the whole video picture due to the limited visual field range, and the user can feel quite fuzzy and cannot embody the advantage of 4K resolution. Therefore, in the case of insufficient code rate and resolution, an experience that a user views panoramic video using a panoramic video playback apparatus needs to be improved.
Disclosure of Invention
In order to solve the problems in the prior art, a panoramic video image processing method, a server, a target device, a system, a device and a computer readable storage medium are provided.
The present invention provides the following.
In a first aspect, a panoramic video image processing method is provided, applied to a server, and includes: acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area; preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out scaling treatment on each image block according to the distance between each image block and the hot spot area; coding the preprocessed image to obtain coded data; the encoded data is transmitted to a target terminal.
In one possible embodiment, the preprocessing further comprises: determining a target gazing direction based on the hot spot area, and dividing the panoramic video image into a plurality of image blocks according to the target gazing direction and the human eye view angle range; and scaling each image block according to the angle difference between the gazing direction corresponding to each image block and the target gazing direction.
In one possible embodiment, the preprocessing further comprises: dividing the panoramic video image in a first type perpendicular to the central longitudinal section of the spherical panoramic video image to obtain a plurality of first image blocks, wherein projections of a plurality of first division lines adopted by the first type division on the central longitudinal section are summarized at a central point and are separated from each other according to a first preset angle; respectively carrying out height scaling treatment on the height dimension of each first image block according to the angle difference between the first dividing line corresponding to each first image block and the target gazing direction; wherein the target gaze direction is directed at an intersection of a central longitudinal section and a central transverse section of the panoramic video image.
In one possible embodiment, the preprocessing further comprises: performing second class division on the panoramic video image perpendicular to the central transverse section of the spherical panoramic video image to obtain a plurality of second image blocks, wherein projections of a plurality of second class division lines adopted by the second class division on the central transverse section of the panoramic video image are summarized at a central point and are separated from each other according to a second preset angle; and scaling the width dimension of each first image block according to the angle difference between the second class dividing line corresponding to each second image block and the hot spot region, wherein the target gazing direction points to the intersection point of the central longitudinal section and the central transverse section of the panoramic video image.
In one possible implementation, the viewpoint pointed by the target gaze direction is located at 0 latitude of the longitude and latitude map of the panoramic video image; the pretreatment further comprises: transversely dividing the panoramic video image according to a preset latitude interval to obtain a plurality of transverse image blocks, determining the latitude difference of each transverse image block and the view point on the panoramic video image, and scaling the height dimension of each transverse image block according to the cosine value of the latitude difference; and/or longitudinally dividing the panoramic video image according to the preset longitude interval to obtain a plurality of longitudinal image blocks, determining the longitude differences of each longitudinal image block and the view point on the panoramic video image, and scaling the width size of each longitudinal image block according to the cosine value of the longitude differences.
In one possible embodiment, the preprocessing further comprises: before division, the panoramic video image is cropped according to the target gaze direction and the angular range of the human eye.
In one possible embodiment, the panoramic video image to be processed is a live video image obtained in real time or a recorded video image obtained in advance.
In one possible embodiment, the method further comprises: and preprocessing the panoramic video image according to the resolution level supported by the target terminal so that the preprocessed image meets the resolution level supported by the target terminal.
In a second aspect, a panoramic video image processing method is provided, applied to a target terminal, including: receiving encoded data processed as in the method of the first aspect; decoding the encoded data to obtain decoded data; performing a restoration process on the decoded data to at least partially restore the panoramic video image, wherein the restoration process is inverse to the scaling process of the first aspect.
In one possible embodiment, the method further comprises: and displaying the restored panoramic video image, wherein a hot spot area of the panoramic video image is positioned in a display center area.
In a third aspect, a server is provided, configured to perform the method as in the first aspect, the server comprising: the method comprises the steps of acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area; a preprocessing unit for preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out scaling treatment on each image block according to the distance between each image block and the hot spot area; the encoding unit is used for encoding the preprocessed image to obtain encoded data; and a transmitting unit for transmitting the encoded data to a target terminal.
In a fourth aspect, there is provided a target terminal configured to perform the method as in the second aspect, the target terminal comprising: a receiving unit that receives the encoded data processed as in the first aspect; the decoding unit is used for decoding the encoded data to obtain decoded data; and a restoration unit that performs restoration processing on the decoded data to at least partially restore the panoramic video image, wherein the restoration processing is inverse to the scaling processing of the first aspect.
In a fifth aspect, there is provided a panoramic video image processing system comprising: a server configured to perform the method as in the first aspect; a target terminal configured to perform the method as in the second aspect.
In a sixth aspect, there is provided a panoramic video image processing apparatus comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform: a method as in the first aspect or a method as in the second aspect.
In a seventh aspect, there is provided a computer-readable storage medium storing a program that, when executed by a multi-core processor, causes the multi-core processor to perform: the method as in the first aspect, or the method as in the second aspect.
The above at least one technical scheme adopted by the embodiment of the application can achieve the following beneficial effects: in this embodiment, by using the above technical solution, on the premise of ensuring the definition of the picture in the hot spot area of interest to the user, the bandwidth pressure caused by video data transmission can be reduced without additional modification to the conventional codec.
It should be understood that the foregoing description is only an overview of the technical solutions of the present invention, so that the technical means of the present invention may be more clearly understood and implemented in accordance with the content of the specification. The following specific embodiments of the present invention are described in order to make the above and other objects, features and advantages of the present invention more comprehensible.
Drawings
The advantages and benefits described herein, as well as other advantages and benefits, will become apparent to those of ordinary skill in the art upon reading the following detailed description of the exemplary embodiments. The drawings are only for purposes of illustrating exemplary embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
Fig. 1 is a flowchart illustrating a panoramic video image processing method according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a panoramic video image in accordance with one embodiment of the present invention;
FIG. 3 is a schematic view illustrating the division of a panoramic video image according to an embodiment of the present invention;
Fig. 4 is a schematic view of a panoramic video image according to another embodiment of the present invention;
FIG. 5 is a schematic diagram of a central longitudinal section of a panoramic video image in accordance with one embodiment of the invention;
FIG. 6 is a schematic diagram of a central transverse cross-section of a panoramic video image in accordance with one embodiment of the invention;
fig. 7 is a schematic view illustrating division of a panoramic video image according to another embodiment of the present invention;
fig. 8 is a schematic structural view of a panoramic video image processing apparatus according to still another embodiment of the present invention;
fig. 9 is a flowchart illustrating a panoramic video image processing method according to another embodiment of the present invention;
fig. 10 is a schematic structural view of a panoramic video image processing apparatus according to still another embodiment of the present invention;
fig. 11 is a schematic structural view of a panoramic video image processing system according to yet another embodiment of the present invention;
fig. 12 is a schematic structural view of a panoramic video image processing device according to still another embodiment of the present invention.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In describing embodiments of the present application, it will be understood that terms, such as "comprises" or "comprising," and the like, are intended to indicate the presence of features, numbers, steps, acts, components, portions, or combinations thereof disclosed in the specification, and are not intended to exclude the possibility of one or more other features, numbers, steps, acts, components, portions, or combinations thereof being present.
Unless otherwise indicated, "/" means or, e.g., A/B may represent A or B; "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone.
The terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first", "a second", etc. may explicitly or implicitly include one or more such feature. In the description of the embodiments of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.
All code in the present application is exemplary and variations will occur to those skilled in the art depending upon the programming language used, the specific needs and personal habits, etc., without departing from the spirit of the application.
In order to clearly illustrate embodiments of the present application, concepts that may appear in some of the following embodiments will first be described.
Resolution ratio: the size or dimension of the video picture is, for example, 4K resolution, 8K resolution.
Dpi: the number of pixels sampled, displayed, or output per inch of length. The larger the dpi of the display device, the smaller the color point of the image output, and the finer the image effect of the output.
Frame rate: the number of frames displayed per unit time.
Code rate: the amount of data of binary data is transmitted or processed per unit time.
Definition: when the frame rate (the number of pictures displayed per second) and the resolution are fixed, the information carried by the pictures in a unit area is examined for the definition of the video when the video is evaluated.
Warp and weft drawings: i.e. equidistant columnar projection (Equirectangular Projection, ERP). A two-dimensional panoramic image which is obtained by uniformly sampling and mapping spherical signals according to equal longitude intervals and equal latitude intervals and can be used for storage and transmission is provided. The abscissa of the image can be expressed by longitude and latitude, the width direction can be expressed by longitude, and the span is 360 degrees; the height direction can be expressed in terms of latitude, with a span of 180 °.
Video decoding (video decoding): and restoring the video code stream into a processing procedure of reconstructing the image according to a specific grammar rule and a processing method.
Video encoding (video encoding): a process of compressing a sequence of images into a code stream.
In addition, it should be noted that, without conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other. The invention will be described in detail below with reference to the drawings in connection with embodiments.
The embodiment of the invention provides a panoramic video processing method, and the invention concept of the method is firstly described below.
The content of panoramic video used in scenes such as theatres, concerts, ball games, etc. has certain characteristics, such as that the viewing attention of the user is focused on hot spot areas (such as areas of stages, courts, etc.), while the attention of other non-hot spot areas (such as audience seats) in the panoramic video is not high. Based on this, the embodiment of the invention provides a panoramic video image processing method, which performs a series of preprocessing on an original 360-degree panoramic video, so as to keep a picture with a certain resolution centered on a hot spot area, and scale or remove content with low importance. Therefore, the scheme can reduce the resolution of the panoramic video image on the premise of not losing the image quality and ensuring the image definition of a hot spot area in the watching habit of a user, so that on one hand, the bandwidth pressure caused by video data transmission is reduced, and on the other hand, the scheme can be compatible with panoramic video playing equipment (such as a common television, 4K VR glasses and the like) with insufficient video decoding capability.
Those skilled in the art will appreciate that the application scenario described is but one example in which embodiments of the present invention may be implemented. The application scope of the embodiments of the present invention is not limited in any way. Having described the basic principles of the present invention, various non-limiting embodiments of the invention are described in detail below.
Fig. 1 is a schematic flow chart of a panoramic video image processing method according to an embodiment of the present application, which is used for performing data processing on a panoramic video image to be played, in the flow, from the perspective of a device, an execution subject may be one or more electronic devices, and more specifically may be a processing module of a server; from the program perspective, the execution subject may be a program mounted on these electronic devices, accordingly. In this embodiment, the execution body of the method may be the server in the embodiment shown in fig. 1.
As shown in fig. 1, the method provided in this embodiment may include the following steps S11 to S14.
S11, acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area;
Panoramic video images refer to frame images of panoramic video. Panoramic video may also be referred to as 360-degree panoramic video or 360-degree video, which is a video that is photographed in 360 degrees in all directions with a plurality of cameras. Those skilled in the art will appreciate that the panoramic video image may be. Referring to fig. 3, a panoramic video image may be generally presented on the surface of a sphere, where the image area viewable by the user is a location on the surface of the sphere. The panoramic video mentioned in this embodiment may be a live video or a recorded video, which is not particularly limited in the present application.
The hot spot region refers to a content region in the panoramic video image where the user is interested in a high degree. For example, in scenes such as a stage play, a ball game, etc., a stage area and a course area of a panoramic video image may be defined as hot spot areas, an audience area, a walkway area, etc. of the panoramic video image may be defined as non-hot spot areas, and the user's interest level in the hot spot areas may be significantly higher than in the non-hot spot areas.
In one example, referring to fig. 2, a dome-shaped panoramic video image 3 is shown, in which a hot spot area 31 is included in the panoramic video image, a user may look at the hot spot area 31 when viewing the panoramic video image 3, a center position or a vicinity of the hot spot area 31 may be regarded as a viewpoint 32, and a direction pointing from a dome-shaped center point O of the panoramic video image to the viewpoint is a target gazing direction, that is, an optimal gazing direction in which the user views the panoramic video image.
S12, preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises the following steps: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out scaling processing on each image block according to the distance between each image block and the hot spot area.
In this embodiment, in order to reduce the resolution of the panoramic video image, a scaling process is required to be performed on the panoramic video image before it is encoded, which is a differential scaling process performed on a plurality of image blocks divided from the panoramic video image, that is, the present embodiment does not scale the panoramic video image as a whole to a certain size, but may differentially scale different image blocks to different scaling degrees. For example, the panoramic video image may be sequentially divided into N regions according to the distance between the N regions and the hot spot region, where the 1 st region is closest to the hot spot region and the N region is farthest from the hot spot region, so that differentiated scaling processing may be performed on the N divided regions, and the scaling degree is higher as the distance is further. Preferably, for one or more image tiles comprising the hot spot area, no scaling or a lesser degree of scaling may be performed in order to guarantee the resolution of the hot spot area. Of course, since there may be two or more image blocks whose distances from the hot spot area are equal or similar, the respective zoom levels of the two or more image blocks may be the same. The spherical panoramic video image and the divided image blocks can be projected into a 2-dimensional panoramic video image, and then the width and/or height of each image block on the 2-dimensional panoramic video image are scaled to different degrees.
The panoramic video image may be divided into a plurality of image blocks in any division manner. For example, the spherical surface of the panoramic video image can be divided transversely and longitudinally according to longitude and latitude to obtain a plurality of image blocks. For another example, the panoramic video image may also be unidirectionally divided to obtain a number of image blocks, such as only transversely divided or only longitudinally divided. For another example, a panoramic video image may also be divided into several equal-sized image tiles based on inscribed regular polygons of spherical surfaces. As such, the present application is not particularly limited thereto.
The distance between each image block and the hot spot area is an index for evaluating the distance between the two, and may be a spherical distance between each image block and the hot spot area, such as a minimum/maximum/average spherical distance, a linear distance between each image block and the hot spot area, such as a minimum/maximum/average linear distance, or an angular distance (may also be referred to as an angular interval) between a gaze direction corresponding to each image block and a target gaze viewpoint corresponding to the hot spot area.
A pre-processed image may be obtained after pre-processing the panoramic video image, the overall resolution of which is lower than the original panoramic video image, and which still retains the picture resolution at the hot spot area to some extent.
S13, carrying out coding treatment on the preprocessed image to obtain coded data;
s14, transmitting the coded data to the target terminal.
In order to transmit the preprocessed panoramic video image to the target terminal for displaying the video image, the preprocessed image needs to be encoded by using a preset video encoding format, for example, the preprocessed image may be encoded by using a conventional video encoder such as x264, x265, etc. It will be appreciated by those skilled in the art that in the design of the official x264, x265 isocoder, each pixel of the video picture is of equal importance, the code rate allocation for each pixel is free of differences in regional weights, and if different regional pixels need to be differentially encoded, the conventional encoder needs to be additionally modified to achieve the code rate weight differentiation for the different regional pixels. However, in the present embodiment, since the encoding object of the video encoder has been subjected to preprocessing as described in S12, the differential resolution representation of different regions of the original panoramic video image can be realized in the output encoded data without modifying the conventional encoder used, thereby achieving the purposes of reducing the data amount of the panoramic video, reducing the video resolution, and compressing the video code rate.
In this embodiment, by using the above technical solution, on the premise of ensuring the picture resolution of the hot spot area of interest to the user, the bandwidth pressure caused by video data transmission can be reduced without additional modification to the conventional encoder.
In one possible implementation, due to the limited resolution level supported by the target terminal, there may be a case of incompatible panoramic video images, based on which the panoramic video images may be preprocessed as above according to the resolution level supported by the target terminal such that the preprocessed images meet the resolution level supported by the target terminal.
In one example, assuming that the original panoramic video image is an 8K video image, the width and height of the 8K video image are W, H, respectively, and the target terminal supports only 4K video images, in order for the target terminal to support playing the panoramic video image, the panoramic video image may be preprocessed such that the video size of the preprocessed image obtained after the preprocessing is 0.5w,0.5h. In other words, the 8K video is processed into the 4K video, so that the target terminal which cannot decode the 8K video can be compatible, and the user experience is improved. The present application can convert any high resolution video into any low resolution video through the above preprocessing, and the above conversion of 8K video into 4K video is merely an example, and the present application is not limited thereto.
In one possible implementation, in S12, the preprocessing further includes: determining a target gazing direction based on the hot spot area, and dividing the panoramic video image into a plurality of image blocks according to the target gazing direction and the human eye view angle range; and scaling each image block according to the angle difference between the gazing direction corresponding to each image block and the target gazing direction.
In one example, referring to fig. 3, a center position of a hot spot area may be taken as a viewpoint 32, a direction in which a spherical center point of a panoramic video image points to the viewpoint 32 may be taken as a target gazing direction, N-1 ring division lines may be determined on a sphere surface with the viewpoint 32 as an apex, and the panoramic video image may be divided into 1 st to nth regions from right to left. Minimum angle intervals between the gazing direction and the target gazing direction corresponding to the image blocks can be adopted, wherein the minimum angle intervals between the 1 st region and the nth region and the target gazing direction are respectively 0 degree, i degree, 2i degree, …, (N-1) i degree. The cosine value of the included angle can be used as a distance index for evaluating each image block and each hot spot region, the cosine value of the included angle is distributed between (-1, 1), wherein the scaling factor S of the 1 st region can be cosine value 1, the picture can be basically unchanged, and the scaling factors S of the other regions can be gradually reduced along with the reduction of the cosine value until the cosine value is close to 0.
In this embodiment, the scaling factor S refers to the size ratio of each image block after the scaling process to the original image block. Preferably, the scaling factor S may include a width scaling factor sw in the image width direction and a height scaling factor sh in the image height direction, and the scaling factors may be uniformly or unevenly distributed over sw and sh, with s=sw.
If the resolution of the original panoramic video image is W, H and divided into n image blocks, the resolution of the scaled video is (sw1+sw2+sw3+sw4+ + swn) ×w, (sh1+sh2+sh3+sh4+ + shn) ×h. Wherein (sw 1, sh 1) are the width scaling factor and the height scaling factor of the 1 st image block, respectively, and so on.
In one possible implementation, in S12, the preprocessing may include: dividing the panoramic video image in a first type perpendicular to the central longitudinal section of the spherical panoramic video image to obtain a plurality of first image blocks, wherein projections of a plurality of first division lines adopted by the first type division on the central longitudinal section are summarized at a central point and are separated from each other according to a first preset angle alpha; respectively carrying out height scaling treatment on the height dimension of each first image block according to the angle difference between the first dividing line corresponding to each first image block and the target gazing direction; wherein the target gaze direction is directed at an intersection of a central longitudinal section and a central transverse section of the panoramic video image.
Referring to fig. 4, a central longitudinal section 41 and a central transverse section 42 of the panoramic video image of fig. 2 are shown, wherein the viewpoint 32 is located at the intersection of the central longitudinal section 41 and the central transverse section.
Referring to fig. 5, there is shown a schematic diagram of a central longitudinal section 41 of a panoramic video image, in which the panoramic video image is longitudinally divided into a plurality of image blocks by rotating about a central point O according to a first preset angle α at an angle of 0 degree with respect to a target gazing direction on the central longitudinal section 41, wherein the greater the angle difference from the target gazing direction, the greater the degree of height scaling of the image blocks. Likewise, the cosine value of the angle difference may be used to determine the height scaling factor sh for each first image block.
For example, in fig. 5, the center vertical section 41 is divided into 12 parts, and when each first image block is highly scaled, it is assumed that the size of each first image block is w×h, and the scaled size is w×sh, where sh is a high scaling factor (sh < 1). In one example, the height scaling factor of each first image block may be calculated according to the cosine value of the included angle between the projection of the first division line of each first image block on the central longitudinal section 41 and the target gazing direction, and the minimum angle intervals between the H1 st region to the H12 th region and the target gazing direction are 0 degrees, 30 degrees, … degrees, 150 degrees, 210 degrees … degrees and 360 degrees, respectively. For the areas H1, H2, … and H6, the cosine values of the included angles are distributed between (-1, 1), sh6< … < sh1, the compression degree of the area H6 after scaling is highest, the area H1 is almost not compressed, and the original image quality is kept, and so on, the area H7 to the area H12, and sh7< … < sh12.
It should be noted that the first preset angle may be a set of a plurality of preset angles, so that the plurality of first class dividing lines may perform non-uniform first class division according to the plurality of preset angles.
In one possible implementation, in S12, the preprocessing may include: performing second class division on the panoramic video image perpendicular to the central transverse section of the spherical panoramic video image to obtain a plurality of second image blocks, wherein projections of a plurality of second class division lines adopted by the second class division on the central transverse section of the panoramic video image are summarized at a central point and are separated from each other according to a second preset angle beta; and scaling the width dimension of each first image block according to the angle difference between the second class dividing line corresponding to each second image block and the hot spot region, wherein the target gazing direction points to the intersection point of the central longitudinal section and the central transverse section of the panoramic video image.
It should be noted that the second preset angle may be a set of a plurality of preset angles, so that the plurality of second class dividing lines may perform non-uniform second class division according to the plurality of preset angles.
Still referring to fig. 4, a central longitudinal section 41 and a central transverse section 42 of the panoramic video image of fig. 2 are shown, wherein the viewpoint 32 is located at the intersection of the central longitudinal section 41 and the central transverse section 42.
Referring to fig. 6, there is shown a schematic view of a central transverse section 42, on which a panoramic video image may be divided into M parts in a transverse direction by rotating around a central point according to a second preset angle β starting from a target gazing direction, wherein the greater the angle difference from the target gazing direction, the greater the degree of width scaling of the region. Likewise, the cosine value of the angle difference may be used to determine the width scaling factor sw for each second image block.
In one possible implementation, the panoramic video image may of course be subjected to a combination of the first type of division and the height scaling process described above, and the second type of division and the width scaling process.
For example, if the resolution of the original panoramic video image is W, H and n image blocks are obtained by the first class division and the second class division, the resolution of the scaled video is (sw1+sw2+sw3+sw4+.+ swn) ×w and (sh1+sh2+sh3+sh4+.+ shn) ×h. Where (swn, shn) are the width scaling factor and the height scaling factor of the nth image block, respectively, and so on.
In one possible embodiment, the preprocessing in S12 may further include: transversely dividing the panoramic video image according to a preset latitude interval gamma to obtain a plurality of transverse image blocks, determining the latitude difference of each transverse image block and the view point on the panoramic video image, and scaling the height dimension of each transverse image block according to the cosine value of the latitude difference; and/or longitudinally dividing the panoramic video image according to the preset longitude interval delta to obtain a plurality of longitudinal image blocks, determining the longitude differences of each longitudinal image block and the view point on the panoramic video image, and scaling the width sizes of each longitudinal image block according to the cosine value of the longitude differences.
In one implementation, referring to fig. 7, a longitude and latitude diagram of a panoramic video image is shown, where a viewpoint 32 pointed by a target gaze direction is located at 0 latitude of the longitude and latitude diagram, a plurality of division lines used for transverse division are a plurality of preset dimension lines of the spherical panoramic video image, and the plurality of preset dimension lines are separated according to a preset latitude interval γ, the higher the latitude, the greater the scaling degree of a transverse image block, and a scaling factor of high scaling can be determined according to a cosine value of the latitude of the image block. The longitudinal division adopts a plurality of dividing lines which are a plurality of preset longitude lines of the spherical panoramic video image, the plurality of preset longitude lines are separated according to a preset longitude interval delta, and the scaling degree of a longitudinal image block with larger longitude difference of the viewpoint on the panoramic video image is larger, and the scaling factor of width scaling can be determined according to the cosine value of the longitude difference.
Alternatively, the plurality of preset latitudinal lines may be set at different latitudinal intervals during the transverse division, for example, the higher the latitude, the larger the transverse division interval may be. Alternatively, the plurality of preset longitude lines may be set at different longitude intervals when dividing longitudinally, for example, the longer the distance from the target gaze direction, the larger the longitude interval.
Alternatively, in S12, the preprocessing may include only the above lateral division or only the above longitudinal division or a combination of both.
In some possible embodiments, in order to further reduce the resolution of the panoramic video image, in S12, the preprocessing may further include: and cutting the panoramic video image according to the target gazing direction and the view angle range.
It will be appreciated by those skilled in the art that the user can only see the contents of a partial region of the panoramic video image, subject to the range of the eye's field of view, and therefore, the pretreatment can be performed in a clipping manner for a region of the picture that is too much beyond the range of the eye's field of view as compared with the target gaze direction.
In one example, assuming that the human eye viewing angle range is at most 90 degrees in the longitudinal direction and at most 180 degrees in the lateral direction, a picture region on the panoramic video image having a longitudinal angle difference from the target gazing direction of more than 45 degrees (or more) may be truncated, and a picture region on the panoramic video image having a lateral angle difference from the target gazing direction of more than 90 degrees (or more) may be truncated.
In the description of the present specification, reference to the terms "some possible implementations", "some embodiments", "examples", "specific examples", or "some examples" etc. means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
With respect to the method flow diagrams of embodiments of the application, certain operations are described as distinct steps performed in a certain order. Such a flowchart is illustrative and not limiting. Some steps described herein may be grouped together and performed in a single operation, may be partitioned into multiple sub-steps, and may be performed in an order different than that shown herein. The various steps illustrated in the flowcharts may be implemented in any manner by any circuit structure and/or tangible mechanism (e.g., by software running on a computer device, hardware (e.g., processor or chip implemented logic functions), etc., and/or any combination thereof).
Based on the same technical concept, the embodiment of the present invention further provides a server configured to perform the method as the above embodiment, referring to fig. 8, the server 80 includes: an acquiring unit 801, which acquires a panoramic video image to be processed, the panoramic video image including a hot spot area; a preprocessing unit 802, which preprocesses the panoramic video image to obtain a preprocessed image, wherein the preprocessing includes: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out scaling treatment on each image block according to the distance between each image block and the hot spot area; an encoding unit 803 that performs encoding processing on the preprocessed image to obtain encoded data; transmitting unit 804 for transmitting the encoded data to the target terminal
It should be noted that, the device in the embodiment of the present application may implement each process of the foregoing method embodiment and achieve the same effects and functions, which are not described herein again.
Based on the same technical concept, the embodiment of the present invention also provides a panoramic video image processing method, which is applied to a target terminal such as a VR device, and fig. 9 shows a flowchart of the panoramic video image processing method.
As shown in fig. 9, the method includes:
s91, receiving coded data; the encoded data is processed using the method described in the above embodiments.
S92, decoding the coded data to obtain decoded data;
S93, performing restoration processing on the decoded data to restore the panoramic video image at least partially, wherein the restoration processing is inverse to the scaling processing described in the above embodiments.
At least the panoramic video image subjected to the scaling process may be restored to obtain the pre-scaling size.
In one possible embodiment, the method may further include: and displaying the restored panoramic video image, wherein the hot spot area of the panoramic video image is positioned in the display center area.
Based on the same technical concept, the embodiment of the present invention further provides a target terminal configured to perform the panoramic video image processing method as above, as shown in fig. 10, the target terminal including: a receiving unit 101 that receives the encoded data obtained by the processing as in the first aspect; a decoding unit 102 that performs decoding processing on the encoded data to obtain decoded data; and a restoration unit 103 that performs restoration processing on the decoded data to restore the panoramic video image at least partially, wherein the restoration processing is inverse to the scaling processing described in the above embodiments.
Based on the same technical concept, the embodiment of the present invention further provides a panoramic video image processing system 110, as shown in fig. 11, including: a server 80 configured to perform the method of fig. 1; the target terminal 100 is configured to perform the method as in fig. 9.
It should be noted that, the device and the system in the embodiments of the present application may implement each process of the embodiments of the foregoing method and achieve the same effects and functions, which are not described herein again.
Fig. 12 is a panoramic video image processing apparatus for performing the panoramic video image processing method shown in fig. 1 according to one embodiment of the application, the apparatus comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described in the above embodiments.
According to some embodiments of the present application, there is provided a non-volatile computer storage medium having stored thereon computer executable instructions configured to, when executed by a processor, perform: the method of the above embodiment.
The embodiments of the present application are described in a progressive manner, and the same and similar parts of the embodiments are all referred to each other, and each embodiment is mainly described in the differences from the other embodiments. In particular, for apparatus, devices and computer readable storage medium embodiments, the description thereof is simplified as it is substantially similar to the method embodiments, as relevant points may be found in part in the description of the method embodiments.
The apparatus, the device, and the computer readable storage medium provided in the embodiments of the present application are in one-to-one correspondence with the methods, so that the apparatus, the device, and the computer readable storage medium also have similar beneficial technical effects as the corresponding methods, and since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the apparatus, the device, and the computer readable storage medium are not repeated herein.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Furthermore, although the operations of the methods of the present invention are depicted in the drawings in a particular order, this is not required or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
While the spirit and principles of the present invention have been described with reference to several particular embodiments, it is to be understood that the invention is not limited to the disclosed embodiments nor does it imply that features of the various aspects are not useful in combination, nor are they useful in any combination, such as for convenience of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (12)
1. The panoramic video image processing method is characterized by being applied to a server and comprising the following steps:
Acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area, and the hot spot area is determined by the content of the panoramic video;
Preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out scaling treatment on each image block according to the distance between each image block and the hot spot area;
The preprocessing further comprises:
Determining a target gazing direction based on the hot spot area, and dividing the panoramic video image into a plurality of image blocks according to the target gazing direction and a human eye view angle range; and
Scaling each image block according to the angle difference between the corresponding gazing direction of each image block and the target gazing direction;
before the division, clipping a picture area which is beyond the visual field angle range of human eyes compared with the target gazing direction in the panoramic video image;
The preprocessing further comprises:
Dividing the panoramic video image into a plurality of first image blocks by a first class perpendicular to the central longitudinal section of the spherical panoramic video image, wherein projections of a plurality of first class dividing lines adopted by the first class division on the central longitudinal section are summarized at a central point and are separated from each other according to a first preset angle alpha;
respectively carrying out height scaling treatment on the height dimension of each first image block according to the angle difference between the first dividing line corresponding to each first image block and the target gazing direction; wherein the target gaze direction is directed at an intersection of a central longitudinal section and a central transverse section of the panoramic video image;
Coding the preprocessed image to obtain coded data;
the encoded data is transmitted to a target terminal.
2. The method of claim 1, wherein the preprocessing further comprises:
Performing second class division on the panoramic video image perpendicular to the central transverse section of the spherical panoramic video image to obtain a plurality of second image blocks, wherein projections of a plurality of second class division lines adopted by the second class division on the central transverse section of the panoramic video image are summarized at a central point and are separated from each other according to a second preset angle beta;
and scaling the width dimension of each first image block according to the angle difference between the second class dividing line corresponding to each second image block and the hot spot region, wherein the target gazing direction points to the intersection point of the central longitudinal section and the central transverse section of the panoramic video image.
3. The method of claim 1, wherein the point of view pointed by the target gaze direction is located at 0 latitude of the longitude and latitude map of the panoramic video image; the preprocessing further comprises:
Transversely dividing the panoramic video image according to a preset latitude interval to obtain a plurality of transverse image blocks, determining the latitude differences of the transverse image blocks and the view points on the panoramic video image, and scaling the height dimension of each transverse image block according to the cosine value of the latitude differences; and/or the number of the groups of groups,
And longitudinally dividing the panoramic video image according to a preset longitude interval to obtain a plurality of longitudinal image blocks, determining longitude differences of the longitudinal image blocks and the viewpoints on the panoramic video image, and scaling the width size of each longitudinal image block according to cosine values of the longitude differences.
4. The method according to claim 1, wherein the panoramic video image to be processed is a live video image obtained in real time or a recorded video image obtained in advance.
5. The method as recited in claim 1, further comprising:
and preprocessing the panoramic video image according to the resolution level supported by the target terminal so that the preprocessed image meets the resolution level supported by the target terminal.
6. A panoramic video image processing method, characterized by being applied to a target terminal, the method comprising:
receiving encoded data processed by the method of any one of claims 1-5;
Decoding the encoded data to obtain decoded data;
performing a restoration process on the decoded data to at least partially restore the panoramic video image, wherein the restoration process is inverse to the scaling process in the method of any one of claims 1-5.
7. The method as recited in claim 6, further comprising:
And displaying the restored panoramic video image, wherein a hot spot area of the panoramic video image is positioned in a display center area.
8. A server configured to perform the method of any of claims 1-5, the server comprising:
The method comprises the steps of acquiring a panoramic video image to be processed, wherein the panoramic video image comprises a hot spot area;
A preprocessing unit for preprocessing the panoramic video image to obtain a preprocessed image, wherein the preprocessing comprises: dividing the panoramic video image into a plurality of image blocks, and respectively carrying out scaling treatment on each image block according to the distance between each image block and the hot spot area;
The encoding unit is used for encoding the preprocessed image to obtain encoded data;
and a transmitting unit for transmitting the encoded data to a target terminal.
9. A target terminal configured for performing the method of any of claims 6-7, the target terminal comprising:
A receiving unit for receiving encoded data processed by the method of any one of claims 1 to 5;
the decoding unit is used for decoding the encoded data to obtain decoded data;
A restoration unit that performs a restoration process on the decoded data to at least partially restore the panoramic video image, wherein the restoration process is inverse to the scaling process of any one of claims 1 to 5.
10. A panoramic video image processing system, comprising:
a server configured to perform the method of any one of claims 1-5;
A target terminal configured to perform the method of any of claims 6-7.
11. Panoramic video image processing device, characterized in that it comprises:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform: the method of any one of claims 1-5, or the method of any one of claims 6-7.
12. A computer-readable storage medium storing a program that, when executed by a multi-core processor, causes the multi-core processor to perform: the method of any one of claims 1-5, or the method of any one of claims 6-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110678889.9A CN113630622B (en) | 2021-06-18 | 2021-06-18 | Panoramic video image processing method, server, target equipment, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110678889.9A CN113630622B (en) | 2021-06-18 | 2021-06-18 | Panoramic video image processing method, server, target equipment, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113630622A CN113630622A (en) | 2021-11-09 |
| CN113630622B true CN113630622B (en) | 2024-04-26 |
Family
ID=78378152
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110678889.9A Active CN113630622B (en) | 2021-06-18 | 2021-06-18 | Panoramic video image processing method, server, target equipment, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113630622B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114143602B (en) * | 2021-12-06 | 2023-07-07 | 深圳创维数字技术有限公司 | Panoramic picture playing method and device, panoramic playing server and storage medium |
Citations (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104567708A (en) * | 2015-01-06 | 2015-04-29 | 浙江工业大学 | Tunnel full-section high-speed dynamic health detection device and method based on active panoramic vision |
| CN104735464A (en) * | 2015-03-31 | 2015-06-24 | 华为技术有限公司 | Panorama video interactive transmission method, server and client end |
| CN106412582A (en) * | 2016-10-21 | 2017-02-15 | 北京大学深圳研究生院 | Panoramic video region of interest description method and coding method |
| CN106791854A (en) * | 2016-11-22 | 2017-05-31 | 北京疯景科技有限公司 | Image Coding, coding/decoding method and device |
| CN107622474A (en) * | 2017-09-26 | 2018-01-23 | 北京大学深圳研究生院 | Panoramic video mapping method based on main view point |
| CN108063946A (en) * | 2017-11-16 | 2018-05-22 | 腾讯科技(成都)有限公司 | Method for encoding images and device, storage medium and electronic device |
| WO2018107467A1 (en) * | 2016-12-16 | 2018-06-21 | Thomson Licensing | Apparatus, method, and storage medium for providing preview of restricted video asset |
| CN108322727A (en) * | 2018-02-28 | 2018-07-24 | 北京搜狐新媒体信息技术有限公司 | A kind of panoramic video transmission method and device |
| CN108513119A (en) * | 2017-02-27 | 2018-09-07 | 阿里巴巴集团控股有限公司 | Mapping, processing method, device and the machine readable media of image |
| CN109194923A (en) * | 2018-10-18 | 2019-01-11 | 眸芯科技（上海）有限公司 | Video image processing apparatus, system and method based on non-uniformed resolution ratio |
| CN109327699A (en) * | 2017-07-31 | 2019-02-12 | 华为技术有限公司 | A kind of processing method of image, terminal and server |
| CN109362242A (en) * | 2016-10-10 | 2019-02-19 | 华为技术有限公司 | A kind of processing method and processing device of video data |
| CN109429062A (en) * | 2017-08-22 | 2019-03-05 | 阿里巴巴集团控股有限公司 | Treating method and apparatus, image encoding method and the device of pyramid model |
| CN110572656A (en) * | 2019-09-19 | 2019-12-13 | 北京视博云科技有限公司 | coding method, image processing method, device and system |
| CN110956583A (en) * | 2018-09-26 | 2020-04-03 | 华为技术有限公司 | Spherical image processing method and device and server |
| CN111726640A (en) * | 2020-07-03 | 2020-09-29 | 中图云创智能科技(北京)有限公司 | Live broadcast method with 0-360 degree dynamic viewing angle |
| CN111885310A (en) * | 2020-08-31 | 2020-11-03 | 深圳市圆周率软件科技有限责任公司 | Panoramic data processing method, processing equipment and playing equipment |
| CN111903118A (en) * | 2018-02-22 | 2020-11-06 | 弗劳恩霍夫应用研究促进协会 | Generating panoramic images |
| US10893043B1 (en) * | 2018-09-12 | 2021-01-12 | Massachusetts Mutual Life Insurance Company | Systems and methods for secure display of data on computing devices |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9973692B2 (en) * | 2013-10-03 | 2018-05-15 | Flir Systems, Inc. | Situational awareness by compressed display of panoramic views |
| GB2536025B (en) * | 2015-03-05 | 2021-03-03 | Nokia Technologies Oy | Video streaming method |
| WO2018193330A1 (en) * | 2017-04-20 | 2018-10-25 | Nokia Technologies Oy | Method and apparatus for delivery of streamed panoramic images |
-
2021
- 2021-06-18 CN CN202110678889.9A patent/CN113630622B/en active Active
Patent Citations (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104567708A (en) * | 2015-01-06 | 2015-04-29 | 浙江工业大学 | Tunnel full-section high-speed dynamic health detection device and method based on active panoramic vision |
| CN104735464A (en) * | 2015-03-31 | 2015-06-24 | 华为技术有限公司 | Panorama video interactive transmission method, server and client end |
| CN109362242A (en) * | 2016-10-10 | 2019-02-19 | 华为技术有限公司 | A kind of processing method and processing device of video data |
| CN106412582A (en) * | 2016-10-21 | 2017-02-15 | 北京大学深圳研究生院 | Panoramic video region of interest description method and coding method |
| CN106791854A (en) * | 2016-11-22 | 2017-05-31 | 北京疯景科技有限公司 | Image Coding, coding/decoding method and device |
| WO2018107467A1 (en) * | 2016-12-16 | 2018-06-21 | Thomson Licensing | Apparatus, method, and storage medium for providing preview of restricted video asset |
| CN108513119A (en) * | 2017-02-27 | 2018-09-07 | 阿里巴巴集团控股有限公司 | Mapping, processing method, device and the machine readable media of image |
| CN109327699A (en) * | 2017-07-31 | 2019-02-12 | 华为技术有限公司 | A kind of processing method of image, terminal and server |
| CN109429062A (en) * | 2017-08-22 | 2019-03-05 | 阿里巴巴集团控股有限公司 | Treating method and apparatus, image encoding method and the device of pyramid model |
| CN107622474A (en) * | 2017-09-26 | 2018-01-23 | 北京大学深圳研究生院 | Panoramic video mapping method based on main view point |
| CN108063946A (en) * | 2017-11-16 | 2018-05-22 | 腾讯科技(成都)有限公司 | Method for encoding images and device, storage medium and electronic device |
| CN111903118A (en) * | 2018-02-22 | 2020-11-06 | 弗劳恩霍夫应用研究促进协会 | Generating panoramic images |
| CN108322727A (en) * | 2018-02-28 | 2018-07-24 | 北京搜狐新媒体信息技术有限公司 | A kind of panoramic video transmission method and device |
| US10893043B1 (en) * | 2018-09-12 | 2021-01-12 | Massachusetts Mutual Life Insurance Company | Systems and methods for secure display of data on computing devices |
| CN110956583A (en) * | 2018-09-26 | 2020-04-03 | 华为技术有限公司 | Spherical image processing method and device and server |
| CN109194923A (en) * | 2018-10-18 | 2019-01-11 | 眸芯科技（上海）有限公司 | Video image processing apparatus, system and method based on non-uniformed resolution ratio |
| CN110572656A (en) * | 2019-09-19 | 2019-12-13 | 北京视博云科技有限公司 | coding method, image processing method, device and system |
| CN111726640A (en) * | 2020-07-03 | 2020-09-29 | 中图云创智能科技(北京)有限公司 | Live broadcast method with 0-360 degree dynamic viewing angle |
| CN111885310A (en) * | 2020-08-31 | 2020-11-03 | 深圳市圆周率软件科技有限责任公司 | Panoramic data processing method, processing equipment and playing equipment |
Non-Patent Citations (2)
| Title |
|---|
| A panoramic video system by direct manipulation video navigation;Chi-Cheng Ju;《 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)》;20140908;全文 * |
| 全景图拼接技术的研究;张垚垚;《 中国优秀硕士论文电子期刊网》;20060415;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113630622A (en) | 2021-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112204993B (en) | Adaptive panoramic video streaming using overlapping partitioned segments | |
| US6559846B1 (en) | System and process for viewing panoramic video | |
| US20190191170A1 (en) | System and method for improving efficiency in encoding/decoding a curved view video | |
| US11483475B2 (en) | Adaptive panoramic video streaming using composite pictures | |
| US11138460B2 (en) | Image processing method and apparatus | |
| US11528538B2 (en) | Streaming volumetric and non-volumetric video | |
| CN110663068B (en) | Coordinate mapping for rendering panoramic scenes | |
| CA3018600C (en) | Method, apparatus and stream of formatting an immersive video for legacy and immersive rendering devices | |
| CN112954398B (en) | Encoding method, decoding method, device, storage medium and electronic equipment | |
| Romero-Rondón et al. | Foveated streaming of virtual reality videos | |
| CN110933461B (en) | Image processing method, device, system, network equipment, terminal and storage medium | |
| CN113630622B (en) | Panoramic video image processing method, server, target equipment, device and system | |
| CN112533005B (en) | Interaction method and system for VR video slow live broadcast | |
| US10891711B2 (en) | Image processing method and apparatus | |
| US20210337264A1 (en) | Image processing method, video playback method and apparatuses thereof | |
| CN111212267A (en) | Partitioning method of panoramic image and server | |
| CN112130667A (en) | Interaction method and system for ultra-high definition VR (virtual reality) video | |
| CN109792490A (en) | The improved pseudo- cylindrical projection of spherical video for stream picture compression | |
| Sauer | Efficient coding tools for 360° video | |
| Groth et al. | Wavelet-Based Fast Decoding of 360-Degree Videos | |
| Groth et al. | Wavelet-Based Fast Decoding of 360 Videos | |
| US20220360795A1 (en) | Method and device for coding and decoding | |
| CN116916071A (en) | Video picture display method, system, device, electronic equipment and storage medium | |
| WO2023129214A1 (en) | Methods and system of multiview video rendering, preparing a multiview cache, and real-time multiview video conversion | |
| KR20220067771A (en) | Image processing device and image playing device for high resolution image streaming and operaing method of thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |