WO2023280266A1 - 鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法 - Google Patents

鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法 Download PDF

Info

Publication number
WO2023280266A1
WO2023280266A1 PCT/CN2022/104346 CN2022104346W WO2023280266A1 WO 2023280266 A1 WO2023280266 A1 WO 2023280266A1 CN 2022104346 W CN2022104346 W CN 2022104346W WO 2023280266 A1 WO2023280266 A1 WO 2023280266A1
Authority
WO
WIPO (PCT)
Prior art keywords
fisheye
image
rendering area
compressed
point set
Prior art date
Application number
PCT/CN2022/104346
Other languages
English (en)
French (fr)
Inventor
王果
姜文杰
Original Assignee
影石创新科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 影石创新科技股份有限公司 filed Critical 影石创新科技股份有限公司
Publication of WO2023280266A1 publication Critical patent/WO2023280266A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display

Definitions

  • the present application relates to the technical field of video compression, in particular to a fisheye image compression method, a fisheye video stream compression method, and a panoramic video generation method.
  • a fisheye lens is a lens with a focal length of 16mm or less and a viewing angle close to, equal to or greater than 180°. It is an extreme wide-angle lens, "fisheye lens” is its common name. In order to maximize the photographic angle of view of the lens, the diameter of the front lens of this photographic lens is very short and protrudes toward the front of the lens in a parabolic shape, which is quite similar to the eyes of a fish, hence the name "fisheye lens”.
  • panoramic video stitching cameras usually use fisheye lenses as the acquisition device for panoramic video images, which are favored by the market for their large viewing angle and high resolution.
  • panoramic video images are not conducive to network transmission due to their high resolution. Therefore, video images need to be compressed.
  • the panoramic video image is generally directly compressed, that is, the obtained fisheye image is first stitched to obtain a panoramic image, and then the panoramic image is compressed, but the current method still has the following problems: 1.
  • the panoramic stitching takes a long time; 2.
  • the original fisheye image will be interpolated and sampled to lose part of the information, resulting in lower resolution after compression; 3.
  • the generated panorama stitching images are generally very large, which requires high hardware performance at the compression end .
  • a fisheye video stream compression method comprising:
  • Compressing the fisheye image to obtain a compressed image wherein, the compression ratio of the fisheye rendering area in the compressed image is smaller than the compression ratio of the fisheye non-rendering area, and/or the fisheye rendering area No compression.
  • a fisheye video stream compression method comprising:
  • each frame of the fisheye image of the fisheye video stream is compressed to obtain a compressed image of each frame of the fisheye image
  • a method for generating a panoramic video comprising:
  • the compressed fisheye video stream is obtained by processing the fisheye video stream compression method described in the above embodiments;
  • the compressed image corresponding to the multi-frame original fisheye image is obtained
  • a fisheye video stream compression device comprising:
  • An information transmission module configured to obtain positioning information of the rendering area of the decoding end
  • a compressed area determination module configured to determine a fisheye rendering area on the corresponding fisheye image according to the positioning information, and the area on the fisheye image other than the fisheye rendering area is a fisheye non-rendering area;
  • a compression module configured to compress the fisheye image to obtain a compressed image; wherein, the compression ratio of the fisheye rendering area in the compressed image is smaller than the compression ratio of the fisheye non-rendering area, and/or the The fisheye rendering area described above is not compressed.
  • a device for compressing fisheye video streams comprising:
  • the video stream acquisition module is used to acquire the fisheye video stream
  • An image compression module configured to compress each frame of the fisheye image of the fisheye video stream by using the fisheye image compression method described in the above embodiments, to obtain a compressed image of each frame of the fisheye image;
  • the video stream compression module is configured to perform video stream compression based on the compressed images of each frame of the fisheye video stream to obtain a compressed fisheye video stream.
  • a panoramic video generating device comprising:
  • a video stream acquisition module configured to acquire a compressed fisheye video stream; wherein, the compressed fisheye video stream is obtained by processing the fisheye video stream compression method as in the above-mentioned embodiments;
  • the video stream decompression module is used to obtain the compressed image corresponding to the original fish-eye image of multiple frames according to the compressed fish-eye video stream;
  • a restoration module configured to restore the compressed image to obtain the original fisheye image
  • the splicing module is used to splice each of the original fisheye images to obtain a panoramic video.
  • a computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above-mentioned methods are realized.
  • the fisheye rendering area on the fisheye image is determined based on the positioning information of the rendering area at the decoding end.
  • the compression ratio of the fisheye rendering area in the compressed image is smaller than the compression ratio of the fisheye non-rendering area, and /or the fisheye rendering area is not compressed, so that the purpose of compression can be achieved, and at the same time, the image transmission bandwidth is saved, and the compression of the compressed fisheye rendering area is relatively small, or lossless compression, so that the final rendered picture can not be Too much compression leads to a decrease in image quality, that is, it can ensure the clarity of the image in the rendering area.
  • Fig. 1 is the application environment diagram of fisheye video stream compression method in an embodiment
  • Fig. 2 is a schematic flow chart of a fisheye video stream compression method in an embodiment
  • Fig. 3 is a schematic flow chart of the step of determining the fisheye rendering area on the corresponding fisheye image according to the positioning information in one embodiment
  • Fig. 4 is a schematic diagram of a second two-dimensional point set distributed on two fisheyes in one embodiment
  • Fig. 5 is a schematic diagram of the distribution of the second two-dimensional point set on a fisheye in an embodiment
  • Fig. 6 is a rendering diagram corresponding to Fig. 4 in an embodiment
  • Fig. 7 is a rendering diagram corresponding to Fig. 5 in an embodiment
  • Fig. 8 is a structural block diagram of a fisheye video stream compression device in an embodiment
  • Figure 9 is an internal block diagram of a computer device in one embodiment.
  • the fisheye video stream compression method provided in this application can be applied to the application environment shown in FIG. 1 .
  • the decoding end 102 communicates with the encoding end 104 through the network.
  • the encoding end 104 is specifically an image acquisition device with a fisheye lens arranged at the acquisition site, and the decoding end is a processing device that receives fisheye video streams and stitches them into panoramic images, which can be VR glasses, cameras, mobile phones, computers, IPads, etc. This is not limited in the present invention.
  • the decoding end is VR glasses, and in this embodiment, there are at least two fisheye lenses, which are arranged at the collection site, and can splicing out a 360-degree panoramic view image of the collection site.
  • the VR glasses acquire the user's head movement, determine the current user's viewpoint, and determine the rendering area according to the current user's viewpoint.
  • the encoding end 104 acquires the positioning information of the rendering area of the decoding end 102; according to the positioning information, the fish-eye rendering area on the corresponding fish-eye image is determined, and the area on the fish-eye image except the fish-eye rendering area is a fish-eye non-rendering area ; Compressing the fisheye image to obtain a compressed image; wherein, the compression ratio of the fisheye rendering area in the compressed image is smaller than the compression ratio of the fisheye non-rendering area, and/or the fisheye rendering area is not compressed.
  • a method for compressing a fisheye video stream is provided.
  • the method is applied to the encoding end in FIG. 1 as an example, including the following steps:
  • Step S202 acquiring positioning information of the rendering area of the decoding end.
  • the angle of view of the fisheye lens can generally reach 220° or 230°.
  • the fisheye images collected by multiple fisheye lenses are used, and then a panoramic image can be obtained by splicing multiple frames of fisheye images.
  • the decoding end is a device for decoding.
  • the decoding end can use VR glasses to follow the change of the VR viewpoint and render the image corresponding to the viewpoint. That is to say, the rendering area is the area where the fisheye image or the stitched panoramic image appears in the rendering image of the decoding end.
  • Positioning information is information used to position the rendering area, such as Euler angles yaw and pitch indicating the direction of the viewpoint, which are used to identify the horizontal field of view (hFOV) and vertical field of view (vFOV) of the rendering area, respectively.
  • the positioning information can be understood as the area range, that is, the information that defines the boundary of a region, and the positioning information in this embodiment defines the information of the area rendered by the decoder.
  • the positioning information of the rendering area of the current frame at the decoding end is obtained.
  • the positioning information of the rendering area of the decoding end may be communicated between the decoding end and the encoding end, and sent from the decoding end to the encoding end. It may also be that the encoding end communicates with a third-party device to obtain the positioning information of the rendering area of the decoding end.
  • a set of positioning information may correspond to a single frame of fisheye images, or may correspond to multiple frames of fisheye images. The positioning information can be sent every frame or every other frame.
  • Step S204 determine the fisheye rendering area on the corresponding fisheye image according to the positioning information, and the area on the fisheye image other than the fisheye rendering area is the fisheye non-rendering area.
  • the fisheye image is the image collected by the fisheye lens.
  • the biggest feature of the fisheye lens is the large viewing angle range, which can generally reach 220° or 230°. Therefore, the obtained fisheye image is an image with a large viewing angle range at the collection site.
  • the fisheye video stream can be collected by using the fisheye lens, and each frame in the fisheye video stream is a fisheye image.
  • the rendering area is projected correspondingly to the fisheye of the frame corresponding to the positioning information. on the image.
  • the area projected on the fisheye image is the rendering area determined by this compression, that is, the fisheye rendering area.
  • the area other than the fish-eye rendering area on the fish-eye image is the fish-eye non-rendering area.
  • Step S206 compressing the fisheye image to obtain a compressed image; wherein the compression ratio of the fisheye rendering area in the compressed image is smaller than the compression ratio of the fisheye non-rendering area, and/or the fisheye rendering area is not compressed.
  • the fisheye image is compressed, and the fisheye rendering area is the content requested by the current viewpoint.
  • Compression or compressing the fisheye rendering area in a way that is smaller than the compression ratio of the fisheye non-rendering area, can achieve the purpose of compression, save the image transmission bandwidth, and at the same time make the compression ratio of the compressed fisheye rendering area smaller, Or no compression, so that the final rendered image will not be overly compressed and cause a decrease in image quality, that is, the clarity of the image in the rendering area can be guaranteed.
  • the above fisheye video stream compression method determines the fisheye rendering area on the fisheye image based on the location information of the rendering area at the decoding end.
  • the compression ratio of the fisheye rendering area in the compressed image is smaller than the compression ratio of the fisheye non-rendering area.
  • the fisheye rendering area is not compressed, so that the purpose of compression can be achieved, and at the same time, the image transmission bandwidth is saved, and the compression of the compressed fisheye rendering area is relatively small, or lossless compression, so that the final rendered picture can not If the image quality is degraded due to excessive compression, the sharpness of the image in the rendering area can be guaranteed.
  • determining the fisheye rendering area on the corresponding fisheye image according to the positioning information includes:
  • Step S302 collecting points at equal intervals on the boundary of the rendering area to obtain a first two-dimensional point set.
  • multiple points are collected at equal intervals on the boundary of the rendering area of the decoding end, for example, 100 points may be collected. Then, according to a specific order, the collected points are stored clockwise or counterclockwise to form a first two-dimensional point set P. Since the rendering area of the decoding end is usually a square (rectangle), in order to ensure that each edge on the boundary can be collected, at least four points are collected, that is, each of the four vertices of the square is a sampling point.
  • the collection interval can be determined according to the size of the rendering area and the determined number of collections. In this embodiment, at least 12 points are preferably collected, because the more points are sampled, the more accurate the boundary of the rendering area is positioned on the fisheye image.
  • Step S304 according to the positioning information of the rendering area, project the first two-dimensional point set onto a spherical coordinate system to obtain a three-dimensional point set.
  • the first two-dimensional point set P is projected onto the spherical coordinate system according to the positioning information of the rendering area, and each projection on the spherical coordinate system is obtained.
  • the three-dimensional points form a three-dimensional point set Ps.
  • the projection method adopted may be any one of existing projection methods, such as spherical perspective projection, spherical equidistant projection, and the like.
  • the spherical perspective projection method is used to project the first two-dimensional point set P onto the spherical coordinate system to obtain the three-dimensional point set Ps.
  • Step S306 projecting the 3D point set onto the fisheye image corresponding to the positioning information to obtain a second 2D point set.
  • the encoder After the encoder obtains the three-dimensional point set Ps, it projects the three-dimensional point set Ps onto the fisheye image corresponding to the positioning information, and the point set formed by the two-dimensional points obtained by this projection is used as the second two-dimensional point set Pf .
  • the spherical equidistant projection method is preferably used to project the three-dimensional point set Ps onto each fisheye image to obtain the second two-dimensional point set Pf.
  • the second two-dimensional point set Pf projected onto the fisheye image may be distributed on one fisheye image, or may be distributed on two or more fisheye images.
  • the distribution diagram shown in Figure 4 shows the positioning of the rendering area (white area) on the fisheye image after projection in a direction of a certain viewpoint when the horizontal field of view of the rendering area is 100 degrees. on a fisheye image.
  • the distribution diagram shown in FIG. 5 shows the positioning of the rendering area on the fisheye image after projection at 60 degrees, that is, it is only distributed on one fisheye image.
  • Figure 6 and Figure 7 are the rendered panorama (left) and rendered area (right) corresponding to Figure 4 and Figure 5, respectively.
  • Step S308 determining a fisheye rendering area of the fisheye image according to the second two-dimensional point set.
  • the encoding end determines the fisheye rendering area R according to the second two-dimensional point set Pf finally obtained by projection.
  • the step of determining the fisheye rendering area of the fisheye image according to the second two-dimensional point set includes: judging whether the second two-dimensional point set is A closed point set; when the second two-dimensional point set is a closed point set, the internal area defined by the closed boundary obtained by connecting the points in the second two-dimensional point set in sequence is used as the fisheye rendering area of the fisheye image; When the second two-dimensional point set is not a closed point set, construct a closed second two-dimensional point set, and connect the points in the constructed second two-dimensional point set in order to obtain the internal area defined by the closed boundary as a fisheye The fisheye rendered area of the image.
  • the second two-dimensional point set Pf may or may not be closed due to the difference in the direction of the viewpoint at the decoding end. In the case of not closing, it is necessary to use a specific method to find additional sampling points on the fisheye image to make it close. Therefore, before determining the fisheye rendering region R through the second two-dimensional point set Pf, it must first be determined whether the second two-dimensional point set Pf is a closed point set. When it is determined as a closed point set, the points in the second two-dimensional point set Pf are directly connected in sequence, and the area occupied by the polygon formed by connecting all points is the fisheye rendering area. And when it is determined that it is not a closed point set, after the closed point set is constructed, the points in the constructed closed point set are connected in order to obtain a fisheye rendering area.
  • constructing a closed second two-dimensional point set includes: collecting points at equal intervals on the field angle boundary of the fisheye lens in the fisheye image to obtain an additional point set; combining the additional point set with the second two-dimensional point set Dimensional point sets are merged to obtain a closed second two-dimensional point set.
  • the closed second two-dimensional point set Pf it is realized by collecting points at equal intervals on the field angle boundary of the fisheye lens in the fisheye image.
  • the subset Pf0 distributed on the left fisheye image in the second two-dimensional point set Pf is not a closed point set, on the field angle boundary of a certain field of view angle of the fisheye lens in the left fisheye image , collecting multiple points at equal intervals (such as 500 points) to form an additional point set Pfe.
  • the selected viewing angle can be larger than 180 degrees and smaller than the maximum FOV (field of view boundary) of the fisheye lens.
  • Field of view boundary means that the area covered by a certain FOV in the fisheye image can be idealized as: a circular area determined by a point C in the central area of the fisheye lens, R is the radius, and R is calculated by the FOV Then, C is obtained by calibration, and the boundary of the circular area is the "field angle boundary". Then, the extra point set Pfe is back-projected.
  • the back-projection is to first project the extra point set Pfe onto the set spherical coordinate system to form a three-dimensional point set by means of spherical equidistant projection, and then use the spherical perspective to transform the three-dimensional point
  • the set is projected onto the plane where the rendering area is located, and the points that the projection falls in the rendering area are put into the set to form the point set Pr. That is, the point set Pr is projected from the subset pfe0 of the extra point set Pfe, so the union of the point sets Pfe0 and Pf0 can form a closed point set.
  • points are collected at equal intervals on the field of view boundary of the fisheye lens to obtain an extra point set Pfe, and the subset Pfe0 of the extra point set can be combined with the subset Pf0 to form a closed point set.
  • the fisheye rendering area corresponding to the decoding end rendering area is determined through projection and back-projection mapping, and the compressed area corresponding to the decoding end rendering area can be obtained to ensure the clarity of the compressed rendering area.
  • compressing the fisheye image to obtain the compressed image includes: determining the area of the compressed image according to the preset compression ratio and the resolution of the fisheye image; downsampling the fisheye image according to the first compression ratio, Get the fisheye thumbnail; when the sum of the pixels of the fisheye thumbnail and the fisheye rendering area is less than or equal to the sum of the pixels of the compressed image, the pixels in the fisheye rendering area and the pixels of the fisheye thumbnail Stored in a compressed map.
  • the preset compression ratio is a preset value, which is determined by the transmission performance of the fisheye video stream. Generally, an appropriate compression ratio is determined to enable low-latency transmission of the video stream. Compressed images are used to store fisheye thumbnails and fisheye rendered regions.
  • the size of the compressed image is calculated through the preset compression ratio and the resolution of the fisheye image.
  • the preset compression ratio is K:1 and the resolution is Wf*Hf
  • a corresponding compressed image is generated according to the determined area S of the compressed image.
  • the fisheye image is down-sampled according to a first compression ratio, such as 500:1, to obtain a fisheye thumbnail image. That is to say, the preset compression ratio is related to the first compression ratio, and usually the first compression ratio is greater than the preset compression ratio.
  • the compressed image Before storing, first judge the size relationship between the fisheye thumbnail and the sum of the pixels of the fisheye rendering area and the sum of the pixels of the compressed image, and determine whether to compress the fisheye rendering area according to the size relationship. Specifically, when the sum of the pixel points of the fisheye thumbnail image and the fisheye rendering area is less than or equal to the sum of the pixel points of the compressed image, that is, the total area Sr of each fisheye rendering area ⁇ (area S of the compressed image-fisheye When the area of the thumbnail is w*h), it means that the compressed image can store the pixels of the fisheye thumbnail and the fisheye rendering area at the same time, and store the pixels of the fisheye thumbnail and the fisheye rendering area in the compressed image directly by row , at this time, the pixels in the fisheye rendering area are not compressed, and the lossless compression of the fisheye rendering area is realized.
  • the positioning information of the rendering area can also be stored in the compressed image.
  • the positioning information of the rendering area may also be stored in other ways, which only needs to be able to realize the transmission of the positioning information of the encoding end and the decoding end.
  • the second compression ratio is used to compress the fisheye rendering area, and the compressed fish The pixels in the eye rendering area and the pixels of the fisheye thumbnail are stored in the compressed image; wherein, the second compression ratio is smaller than the first compression ratio.
  • the first compression ratio and the second compression ratio are related to the preset compression ratio, the first compression ratio is greater than the preset compression ratio, and the second compression ratio is smaller than the preset compression ratio.
  • the method of storing the pixels in the fisheye rendering area into the compressed image includes: sequentially extracting pixels in the fisheye rendering area according to a preset direction, and storing the extracted pixels in sequence according to the extraction order into a compressed image; preset orientations include rows or columns.
  • the pixels in the fisheye rendering area corresponding to the fisheye image are stored in the compressed image in a dense storage manner, so as to complete the compression.
  • the arrangement of each pixel stored in the compressed image can be arbitrary, as long as the principle of easy storage and easy decoding is followed.
  • pixel points are sequentially extracted from the fisheye rendering area according to a preset direction, for example, pixel points are sequentially extracted in units of rows or columns, and the extracted pixel points are sequentially stored in the compressed image according to the extraction order.
  • the pixel information can be densely stored in the compressed image, where the last pixel in the nth row/column of the fisheye rendering area in the compressed image is immediately followed by the first pixel in the n+1th row/column point.
  • the fisheye image is compressed to obtain a compressed image, including:
  • each downsampling mapping table records the mapping relationship between the fisheye non-rendering area and the compressed image, and the mapping relationship between the fisheye rendering area and the compressed image; Image remapping is performed on the rendering area and the fisheye rendering area to obtain a compressed image.
  • a mapping table with a downsampling function in a corresponding area is generated, that is, the downsampling mapping table in this embodiment.
  • Each downsampling mapping table records the mapping relationship between the fisheye non-rendering area and the compressed image, and the mapping relationship between the fisheye rendering area and the compressed image. Since it is a mapping table with a downsampling function, it can be understood that the downsampling of the region has been completed during the process of generating the mapping table.
  • the downsampling mapping table includes the original location information of each pixel in the fisheye image in the corresponding area.
  • mapping table image remapping is performed on the fisheye non-rendering area and fisheye rendering area to complete the compression to obtain a compressed image, and the picture saved in the compressed image is the mapping result of image remapping .
  • multiple mapping tables can also be generated at the same time. The difference between each mapping table is that its corresponding area is different, that is, each mapping result obtained by performing image remapping on multiple mapping tables is a part of the compressed image, and All mapping results are stored in the same compressed image to obtain a complete compressed image.
  • Multi-resolution downsampling refers to downsampling with a lower compression ratio or no downsampling of the fisheye rendering area determined in the fisheye (that is, when the compressed image can be placed in the complete rendering area, you can choose not to downsample Sampling) to down-sample the fisheye non-rendering area in the fisheye with a higher compression ratio.
  • Maintaining the local continuity of the fisheye picture means that the relative positional relationship between any two pixels in a certain part of the fisheye does not change in the compressed image, which is beneficial to the video stream when the compressed image is converted into a video stream Encoding compression.
  • mapping table is used for compression. Whether it is a fisheye rendering area or a non-rendering area, the downsampling step can be included in the process of generating the mapping table, directly through one or more mapping tables. Get the whole compressed image in one step or step by step.
  • a method for compressing a fisheye video stream is also provided, the method is applied to the encoding end as shown in Figure 1, and the method includes: obtaining a fisheye video stream; using the fisheye image compression method of each embodiment, Perform compression processing on each frame of the fisheye image of the fisheye video stream to obtain a compressed image of each frame of the fisheye image; perform video stream compression based on the compressed images of each frame of the fisheye video stream to obtain a compressed fisheye image eye video stream.
  • the fisheye video stream refers to a video image collected by a fisheye lens. It can be understood that each frame of the fisheye video stream is the fisheye image mentioned in the foregoing embodiments.
  • the fisheye image compression method of each embodiment has been described in the previous embodiments, and will not be further described here. It can be understood that, for the compressed image of each frame of fisheye image, by determining the fisheye rendering area of the fisheye image according to the positioning information of the rendering area, the fisheye rendering area is compressed with a smaller compression ratio or Lossless compression achieves the purpose of compression, and at the same time prevents the image rendered by fisheye from being overly compressed to cause a decrease in image quality, that is, it can ensure the clarity of the image in the fisheye rendered area.
  • the actual required fisheye rendering area can be determined in real time according to the change of the viewpoint, so that the fisheye video stream can be dynamically compressed in real time according to the change of the viewpoint of the decoding end.
  • the video stream compression method For the compressed image of each frame of the fisheye video stream, the video stream compression method is used to compress again.
  • the video stream compression method is such as H.264 and the like.
  • the video stream compression process results in a compressed fisheye video stream.
  • the above-mentioned fisheye video stream compression can dynamically adjust the fisheye rendering area according to the position of the real-time preview area of the rendering end in the panorama, and can compress the fisheye video stream in real time according to the change of the viewpoint of the decoding end, which not only ensures the rendering
  • the high definition of the area and the high compression rate can greatly save the transmission bandwidth of the video stream.
  • a panoramic video generation method the method is applied to the decoding end as shown in Figure 1, comprising: obtaining a compressed fisheye video stream; wherein, the compressed fisheye video stream adopts the previous The recorded fisheye video stream compression method is obtained; according to the compressed fisheye video stream, a compressed image corresponding to multiple frames of the original fisheye image is obtained; the compressed image is restored to obtain the original fisheye image; Stitching each of the original fisheye images to obtain a panoramic video.
  • the decoding end obtains the fisheye video stream processed by the encoding end.
  • the method for processing and obtaining the fisheye video stream at the encoding end has been described in the previous specification, and will not be expanded here.
  • a decompression method corresponding to the video stream compression method is used to decompress the fisheye video stream to obtain compressed images corresponding to multi-frame original fisheye images.
  • a restoration method corresponding to the compression method is used to restore the compressed image to obtain the original fisheye image.
  • searching for matching points is performed, and splicing is performed based on the matching points to obtain a panoramic video.
  • the video stream is decompressed, the compressed image is restored, and splicing is performed to obtain a panoramic video.
  • the rendering area of the fisheye can be dynamically adjusted according to the position of the real-time preview area of the rendering end in the panorama, and can be adjusted in real time according to the change of the viewpoint of the decoding end.
  • the fisheye video stream is compressed, which not only ensures the high definition of the rendering area, but also achieves a high compression ratio, which greatly saves the transmission bandwidth of the video stream.
  • the encoding end only compresses the fisheye images and video streams without splicing, but decompresses them at the decoding end before splicing, which reduces the high requirements for the hardware performance of the encoding end.
  • the compressed image can be packaged into a data stream and transmitted to the decoding end for decoding and display. Wherein, restoring the display needs to utilize the positioning information of the rendering area.
  • the positioning information of the rendering area can be stored in the compressed image and transmitted to the decoding end. It can also be stored in other ways, as long as it can realize the transmission of the positioning information of the encoding end and the decoding end.
  • the decoding end After acquiring the positioning information of the rendering area, decoding end decodes the compressed image according to the original positioning information, and recovers the original fisheye image.
  • restoring the compressed image to obtain the original fisheye image includes: analyzing the compressed image to obtain each pixel in the fisheye rendering area Original positioning information in the original fisheye image; decoding the compressed image according to the original positioning information, and recovering to obtain the original fisheye image.
  • the compressed image at the encoding end is obtained through dense storage
  • the compressed image when the compressed image is transmitted to the decoding end, the original positioning information of each pixel stored in the compressed image on the original fisheye image needs to be sent to the decoding end. end.
  • the subsequent decoder copies the number of pixels indicated by the positioning information in the compressed image to the position indicated by the positioning information in the fisheye according to the original positioning information to complete the decoding and recovery of the rendering area in the original fish-eye image.
  • the non-rendering area can be compressed
  • the fisheye thumbnail stored in the image is obtained by upsampling, and the rendered area and the non-rendered area together constitute the restored original fisheye image.
  • the original positioning information of each pixel can also be recorded by row, that is, the original positioning information of each row of pixels can be represented by three data: the row index of the first pixel in the original fisheye image, The column index and the total number of pixels in the row.
  • restoring the compressed image to obtain the original fisheye image includes: obtaining a reverse mapping table generated according to the down-sampling mapping table; reproducing the compressed image according to the reverse mapping table Mapping recovery obtains the original fisheye image.
  • the inverse mapping table corresponding to the mapping table or the parameter information that can construct the inverse mapping table needs to be transmitted to the decoding end.
  • the subsequent decoding end performs image remapping on the pixels in the compressed image according to the inverse mapping table to decode and restore the original fisheye image. Then, the decoding end performs panoramic stitching and rendering display on the decompressed original fisheye image.
  • corresponding decoding information is transmitted to the decoding end according to different compression methods, so as to ensure that the decoding end can efficiently complete the decoding and restoration of the fisheye diagram.
  • a fisheye video stream compression device including: an information transmission module 802, a compression area determination module 804, and a compression module 806, wherein:
  • the compressed area determination module 804 is configured to determine a fisheye rendering area on the corresponding fisheye image according to the positioning information, and the area on the fisheye image other than the fisheye rendering area is a fisheye non-rendering area.
  • a compression module 806, configured to compress the fisheye image to obtain a compressed image; wherein, the compression ratio of the fisheye rendering area in the compressed image is smaller than the compression ratio of the fisheye non-rendering area, and/or The fisheye render area is not compressed.
  • the compression area determination module 804 is further configured to collect points at equal intervals on the boundary of the rendering area to obtain a first two-dimensional point set; according to the positioning information of the rendering area, the first two The three-dimensional point set is projected onto a spherical coordinate system to obtain a three-dimensional point set; the three-dimensional point set is projected onto the fisheye image corresponding to the positioning information to obtain a second two-dimensional point set; according to the second two-dimensional point set The set determines the fisheye rendering area of the fisheye image.
  • the compressed area determination module 804 is further configured to judge whether the second two-dimensional point set is a closed point set according to the Euclidean distance between the first and last points of the second two-dimensional point set; when the second two-dimensional point set When the two-dimensional point set is a closed point set, the internal area defined by the closed boundary obtained by connecting the points in the second two-dimensional point set in sequence is used as the fisheye rendering area of the fisheye image; when the When the second two-dimensional point set is not a closed point set, construct a closed second two-dimensional point set, and connect the points in the constructed second two-dimensional point set in order to obtain the internal area defined by the closed boundary as the fish The fisheye rendered area of the eye image.
  • the compressed area determination module 804 is also used to collect points at equal intervals on the field angle boundary of the fisheye lens in the fisheye image to obtain an additional point set; combine the additional point set with the second two Dimensional point sets are merged to obtain a closed second two-dimensional point set.
  • the compression module 806 is further configured to determine the area of the compressed image according to the preset compression ratio and the resolution of the fisheye image; downsample the fisheye image according to the first compression ratio to obtain Fisheye thumbnail; when the sum of the pixels of the fisheye thumbnail and the fisheye rendering area is less than or equal to the sum of the pixels of the compressed image, the pixels in the fisheye rendering area and The pixels of the fisheye thumbnail are stored in the compressed image.
  • the compression module 806 is further configured to use the second compression ratio when the sum of the pixels of the fisheye thumbnail image and the fisheye rendering area is greater than the sum of the pixels of the compressed image.
  • the fisheye rendering area is compressed, and the compressed pixels in the fisheye rendering area and the pixels of the fisheye thumbnail are stored in the compressed image; wherein, the second compression ratio is smaller than the first compression ratio A compression ratio; the preset compression ratio is related to the first compression ratio and the second compression ratio.
  • the compression module is further configured to sequentially extract pixels from the fisheye rendering area according to a preset direction, and store the extracted pixels into the compressed image in sequence according to the extraction order, and the preset Direction includes row or column.
  • the last pixel in the nth row/column of the fisheye rendering area is immediately followed by the first pixel in the n+1th row/column.
  • the compression module 806 is further configured to generate at least one downsampling mapping table, and each downsampling mapping table records the mapping relationship between the fisheye non-rendering area and the compressed image, and the fisheye rendering The mapping relationship between the region and the compressed image: performing image remapping on the fisheye non-rendering region and the fisheye rendering region according to each of the downsampling mapping tables to obtain a compressed image.
  • Each module in the above-mentioned fisheye video stream compression device can be fully or partially realized by software, hardware and combinations thereof.
  • the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.
  • a device for compressing fisheye video streams including:
  • the video stream acquisition module is used to acquire the fisheye video stream
  • An image compression module configured to compress each frame of the fisheye image of the fisheye video stream by using the fisheye image compression method described in the above embodiments, to obtain a compressed image of each frame of the fisheye image;
  • the video stream compression module is configured to perform video stream compression based on the compressed images of each frame of the fisheye video stream to obtain a compressed fisheye video stream.
  • a device for generating a panoramic video includes:
  • a video stream acquisition module configured to acquire a compressed fisheye video stream; wherein, the compressed fisheye video stream is obtained by processing the fisheye video stream compression method as in the above-mentioned embodiments;
  • the video stream decompression module is used to obtain the compressed image corresponding to the original fish-eye image of multiple frames according to the compressed fish-eye video stream;
  • a restoration module configured to restore the compressed image to obtain the original fisheye image
  • the splicing module is used to splice each of the original fisheye images to obtain a panoramic video.
  • the restoration module is used to analyze the compressed image, obtain the original location information of each pixel in the fisheye rendering area in the original fisheye image; decode the compressed image according to the original location information, and restore Obtain the original fisheye image.
  • the restoration module is used to obtain a reverse mapping table generated according to the down-sampling mapping table; remap and recover the compressed image according to the reverse mapping table to obtain the original fisheye image.
  • a computer device is provided.
  • the computer device may be a terminal, and its internal structure may be as shown in FIG. 9 .
  • the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system and computer programs.
  • the internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium.
  • the communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, an operator network, NFC (Near Field Communication) or other technologies.
  • a fisheye video stream compression method, a fisheye video stream compression method or a panoramic video generation method are realized.
  • the display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad, or mouse.
  • FIG. 9 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer equipment on which the solution of this application is applied.
  • the specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.
  • a computer device including a memory and a processor, and a computer program is stored in the memory, and when the processor executes the computer program, the fisheye image compression method and the fisheye video stream compression method of the above-mentioned embodiments are realized. method or panoramic video generation method.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the fisheye image compression method, the fisheye video stream compression method or the panoramic view of the above-mentioned embodiments are implemented.
  • Video generation method is provided, on which a computer program is stored, and when the computer program is executed by a processor, the fisheye image compression method, the fisheye video stream compression method or the panoramic view of the above-mentioned embodiments are implemented.
  • Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc.
  • Volatile memory can include Random Access Memory (RAM) or external cache memory.
  • RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

Abstract

本申请涉及一种鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法。其中，鱼眼图像压缩方法包括：获取解码端渲染区域的定位信息；根据定位信息确定对应的鱼眼图像上的鱼眼渲染区域，鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域；对鱼眼图像进行压缩，得到压缩图像；其中，压缩图像中鱼眼渲染区域的压缩比小于鱼眼非渲染区域的压缩比，和/或鱼眼渲染区域不压缩。该方法能够达到压缩的目的，节省了图像传输宽带，同时压缩后的鱼眼渲染区域的压缩比较小，或是无损压缩，而使得最终渲染的画面能够不被过多压缩而导致画质下降，即能够保证渲染区域画面的清晰度。

Description

鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法 技术领域
本申请涉及视频压缩技术领域，特别是涉及一种鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法。
背景技术
鱼眼镜头是一种焦距为16mm或更短的并且视角接近、等于或大于180°的镜头。它是一种极端的广角镜头，“鱼眼镜头”是它的俗称。为使镜头达到最大的摄影视角，这种摄影镜头的前镜片直径很短且呈抛物状向镜头前部凸出，与鱼的眼睛颇为相似，“鱼眼镜头”因此而得名。
目前，全景视频拼接摄像机通常采用鱼眼镜头作为全景视频图像的采集装置，以其大视角、高分辨率的特性，备受市场青睐。但全景视频图像由于其分辨率较高，不利于网络传输。因此，需要对视频图像进行压缩。目前一般直接对全景视频图像进行压缩，即先对获得的鱼眼图像进行拼接得到全景图像，然后再对全景图像进行压缩，但目前的方法还存在如下问题：1、全景拼接耗时较长；2、在全景拼接过程中会对原鱼眼图像进行插值采样而丢失部分信息，导致压缩后的清晰度变低；3、生成的全景拼接图像一般都很大，对压缩端的硬件性能要求很高。
鉴于此，有必要提供一种减少耗时、确保高清晰度且对压缩端硬件性能依赖较低的图像压缩方法。
发明内容
基于此，有必要针对上述技术问题，提供一种能够减少耗时、确保高清晰度且对压缩端硬件性能依赖低的鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法及其装置，以及计算机设备和存储介质。
一种鱼眼视频流压缩方法，所述方法包括：
获取解码端渲染区域的定位信息；
根据所述定位信息确定对应的鱼眼图像上的鱼眼渲染区域，所述鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域；
对所述鱼眼图像进行压缩，得到压缩图像；其中，所述压缩图像中所述鱼眼渲染区域的压缩比小于所述鱼眼非渲染区域的压缩比，和/或所述鱼眼渲染区域不压缩。
一种鱼眼视频流压缩方法，包括：
获取鱼眼视频流；
采用如上述各实施例所述的鱼眼图像压缩方法，对所述鱼眼视频流的每一帧鱼眼图像进行压缩处理，得到每一帧鱼眼图像的压缩图像；
基于所述鱼眼视频流各帧的压缩图像进行视频流压缩，得到压缩的鱼眼视频流。
一种全景视频生成方法，包括：
获取压缩的鱼眼视频流；其中，所述压缩的鱼眼视频流采用上述各实施例所述的鱼眼视频流压缩方法处理得到；
根据所述压缩的鱼眼视频流，得到多帧原鱼眼图像对应的压缩图像；
对所述压缩图像进行还原，得到所述原鱼眼图像；
拼接各所述原鱼眼图像，得到全景视频。
一种鱼眼视频流压缩装置，所述装置包括：
信息传输模块，用于获取解码端渲染区域的定位信息；
压缩区域确定模块，用于根据所述定位信息确定对应的鱼眼图像上的鱼眼渲染区域，所述鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域；
压缩模块，用于对所述鱼眼图像进行压缩，得到压缩图像；其中，所述压缩图像中所述鱼眼渲染区域的压缩比小于所述鱼眼非渲染区域的压缩比，和/或所述鱼眼渲染区域不压缩。
一种鱼眼视频流压缩装置，包括：
视频流获取模块，用于获取鱼眼视频流；
图像压缩模块，用于采用上述各实施例所述的鱼眼图像压缩方法，对所述鱼眼视频流的每一帧鱼眼图像进行压缩处理，得到每一帧鱼眼图像的压缩图像；
视频流压缩模块，用于基于所述鱼眼视频流各帧的压缩图像进行视频流压缩，得到压缩的鱼眼视频流。
一种全景视频生成装置，包括：
视频流获取模块，用于获取压缩的鱼眼视频流；其中，所述压缩的鱼眼视频流采用如上述各实施例的鱼眼视频流压缩方法处理得到；
视频流解压缩模块，用于根据所述压缩的鱼眼视频流，得到多帧原鱼眼图像对应的压缩图像；
还原模块，用于对所述压缩图像进行还原，得到所述原鱼眼图像；
拼接模块，用于拼接各所述原鱼眼图像，得到全景视频。
一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现上述任一项的方法的步骤。
一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述任一项的方法的步骤。
上述鱼眼图像压缩方法，基于解码端渲染区域的定位信息确定鱼眼图像上的鱼眼渲染区域，压缩时，压缩图像中鱼眼渲染区域的压缩比小于鱼眼非渲染区域的压缩比，和/或鱼眼渲染区域不压缩，从而能够达到压缩的目的，同时节省了图像传输宽带，且压缩后的鱼眼渲染区域的压缩比较小，或是无损压缩，从而使得最终渲染的画面能够不被过多压缩而导致画质下降，即能够保证渲染区域画面的清晰度。
附图说明
图1为一个实施例中鱼眼视频流压缩方法的应用环境图；
图2为一个实施例中鱼眼视频流压缩方法的流程示意图；
图3为一个实施例中根据定位信息确定对应的鱼眼图像上的鱼眼渲染区域步骤的流程示意图；
图4为一个实施例中第二二维点集分布在两个鱼眼上的示意图；
图5为一个实施例中第二二维点集分布在一个鱼眼上示意图；
图6为一个实施例中图4所对应的渲染示意图；
图7为一个实施例中图5所对应的渲染示意图；
图8为一个实施例中鱼眼视频流压缩装置的结构框图；
图9为一个实施例中计算机设备的内部结构图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请提供的鱼眼视频流压缩方法，可以应用于如图1所示的应用环境中。其中，解码端102通过网络与编码端104进行通信。编码端104具体为设置在采集现场的带鱼眼镜头的图像采集设备，解码端为接收鱼眼视频流并拼接为全景图像的处理设备，可以为VR眼镜、相机、手机、电脑、IPad等，在本发明中对此不作限定。一个实施例中，解码端为VR眼镜，在该实施例中，鱼眼镜头至少为两个，设置在采集现 场，能够拼接出采集现场360度的全景环视图像。VR眼镜获取用户头部动作，确定当前用户视点，根据当前用户视角确定渲染区域。具体地，编码端104获取解码端102渲染区域的定位信息；根据定位信息确定对应的鱼眼图像上的鱼眼渲染区域，鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域；对鱼眼图像进行压缩，得到压缩图像；其中，压缩图像中鱼眼渲染区域的压缩比小于鱼眼非渲染区域的压缩比，和/或鱼眼渲染区域不压缩。
在一个实施例中，如图2所示，提供了一种鱼眼视频流压缩方法，以该方法应用于图1中的编码端为例进行说明，包括以下步骤：
步骤S202，获取解码端渲染区域的定位信息。
鱼眼镜头的视角一般可达到220°或230°，采用多个鱼眼镜头采集的鱼眼图像，然后通过拼接多帧鱼眼图像可以得到全景图像。解码端是用于解码的设备，解码端可采用VR眼镜，跟随VR视点变化，渲染视点对应的图像画面。也就是说，渲染区域是鱼眼图像或是拼接后的全景图像出现在解码端渲染画面中的区域。定位信息则是用于定位渲染区域的信息，比如表示视点方向的欧拉角yaw和pitch，分别用于标识渲染区域范围的水平视野(hFOV)和竖直视野(vFOV)。定位信息可以理解为区域范围，即可限定一块区域边界的信息，本实施例的定位信息则限定解码端渲染区域的信息。
具体地，当编码端在对采集的鱼眼图像进行压缩之前，获取解码端当前帧渲染区域的定位信息。解码端渲染区域的定位信息可以是解码端与编码端通信，由解码端发送给编码端。也可以是编码端与第三方设备通信得到解码端渲染区域的定位信息。应当理解的是，一组定位信息可以对应单帧鱼眼图像，也可以对应多帧鱼眼图像。定位信息的发送可以每一帧都发送，也可以隔帧发送。
步骤S204，根据定位信息确定对应的鱼眼图像上的鱼眼渲染区域，鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域。
鱼眼图像是鱼眼镜头采集的图像，鱼眼镜头最大的特点是视角范围大，视角一般可达到220°或230°，因此得到的鱼眼图像为采集现场的超大视角范围图像。利用鱼眼镜头能够采集鱼眼视频流，鱼眼视频流中的每一帧为鱼眼图像。
具体地，每当编码端得到鱼眼视频流中当前帧的渲染区域的定位信息之后，由于渲染区域的定位信息按照帧依次发送，则将渲染区域对应投影到该定位信息所对应帧的鱼眼图像上。在鱼眼图像上投影所得到的区域即为本次压缩所确定的渲染区域，即鱼眼渲染区域。在鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域。可以理解的是，随着VR眼镜视点的变化，定位信息发生变化，进而所请求的渲染区域发生变化，最终每一帧鱼眼图像的鱼眼渲染区域不同。
步骤S206，对鱼眼图像进行压缩，得到压缩图像；其中，压缩图像中鱼眼渲染区域的压缩比小于鱼眼非渲染区域的压缩比，和/或鱼眼渲染区域不压缩。
具体地，当编码端通过定位信息投影确定鱼眼图像上的鱼眼渲染区域后，对鱼眼图像进行压缩，而鱼眼渲染区域为当前视点所请求的内容，通过对鱼眼渲染区域不进行压缩，或是对鱼眼渲染区域以小于鱼眼非渲染区域压缩比的方式进行压缩，能够达到压缩的目的，节省了图像传输宽带，而同时使压缩后的鱼眼渲染区域的压缩比较小，或是不压缩，而使得最终渲染的画面能够不被过多压缩而导致画质下降，即能够保证渲染区域画面的清晰度。
上述鱼眼视频流压缩方法，基于解码端渲染区域的定位信息确定鱼眼图像上的鱼眼渲染区域，压缩时，压缩图像中鱼眼渲染区域的压缩比小于鱼眼非渲染区域的压缩比，和/或鱼眼渲染区域不压缩，从而能够达到压缩的目的，同时节省了图像传输宽带，且压缩后的鱼眼渲染区域的压缩比较小，或是无损压缩，从而使得最终渲染的画面能够不被过多压缩而导致画质下降，即能够保证渲染区域画面的清晰度。
在一个实施例中，如图3所示，根据定位信息确定对应的鱼眼图像上的鱼眼渲染区域的包括：
步骤S302，在渲染区域的边界上等间隔采集点，得到第一二维点集。
具体地，在解码端的渲染区域的边界上等间隔采集多个点，比如可以采集100个点。然后，按照特定的顺序，顺时针或者逆时针将采集的点存储，形成第一二维点集P。由于解码端的渲染区域通常为方形(矩形)，为了确保边界上每个边都能被采集到，采集的点数至少为四个，即方形的四个顶点各为一个采样点。采集的间隔则根据渲染区域的大小和确定的采集数确定即可。本实施例优选采集至少12个点，因为采样的点数越多，在鱼眼图像上定位的渲染区域边界越准确。
步骤S304，根据渲染区域的定位信息，将第一二维点集投影到球面坐标系上，得到三维点集。
具体地，当得到等间隔采集得到的第一二维点集P之后，根据渲染区域的定位信息将第一二维点集P投影到球面坐标系上，得到投影后在球面坐标系上的各个三维点，形成三维点集Ps。其中，采用的投影方式可以是现有投影方式中的任意一种，比如球面透视投影、球面等距投影等。本实施例采用球面透视投影方式将第一二维点集P投影到球面坐标系上得到三维点集Ps。
步骤S306，将三维点集投影到定位信息对应的鱼眼图像上，得到第二二维点集。
具体地，编码端得到三维点集Ps之后，再将该三维点集Ps投影到定位信息对应的鱼眼图像上，本次投影得到的二维点所形成点集作为第二二维点集Pf。其中，本实施例优选采用球面等距投影方式将三维点集Ps投影到各鱼眼图像上得到第二二维点集Pf。
另外，根据渲染端(解码端)视点的不同，投影到鱼眼图像上的第二二维点集Pf可能分布在一个鱼眼 图像上，也可能分布在两个或多个鱼眼图像上。可以参考图4-7，图4所示的分布图为渲染区域水平视野为100度时，某个视点方向下渲染区域(白色区域)投影后在鱼眼图像上的定位情况，即分布在两个鱼眼图像上。而图5所示的分布图为60度时渲染区域投影后在鱼眼图像上的定位情况，即只分布在一个鱼眼图像上。因此，当第二二维点集Pf分布在两个鱼眼图像上时，左鱼眼图像的第二二维点集可以记为Pf0，右鱼眼图像的第二二维点集可以记为Pf1，Pf＝{Pf0，Pf1}。图6和图7则分别是图4和图5对应的渲染全景图(图左)和渲染区域画面(图右)。
步骤S308，根据第二二维点集确定鱼眼图像的鱼眼渲染区域。
具体地，编码端根据投影最终得到的第二二维点集Pf确定鱼眼渲染区域R。
在一个实施例中，根据第二二维点集确定鱼眼图像的鱼眼渲染区域的步骤，包括：根据第二二维点集中首尾点之间的欧式距离判断第二二维点集是否为闭合的点集；当第二二维点集是闭合的点集时，将第二二维点集中的点按顺序连接得到的闭合边界所限定的内部区域作为鱼眼图像的鱼眼渲染区域；当第二二维点集不是闭合的点集时，构造闭合的第二二维点集，将构造的第二二维点集中的点按顺序连接得到的闭合边界所限定的内部区域作为鱼眼图像的鱼眼渲染区域。
具体地，同样由于解码端视点方向的不同，第二二维点集Pf可能是闭合的也可能不是闭合的。而在不闭合的情况下，需要使用特定方法在鱼眼图像上额外寻找采样点来使其闭合。因此，在通过第二二维点集Pf确定鱼眼渲染区域R之前，首先要确定第二二维点集Pf是否为闭合点集。当确定为闭合的点集时，直接按照顺序将第二二维点集Pf中的点连接起来，所有点连接起来形成的多边形所占据的区域即为鱼眼渲染区域。而当确定不是闭合的点集时，则在构造得到闭合的点集之后将构造的闭合的点集中的点按照顺序连接得到鱼眼渲染区域。
在一个实施例中，构造闭合的第二二维点集，包括：在鱼眼图像中鱼眼镜头的视场角边界上等间隔采集点，得到额外点集；将额外点集与第二二维点集合并，得到闭合的第二二维点集。
具体地，当构造闭合的第二二维点集Pf时，通过在鱼眼图像中鱼眼镜头的视场角边界上等间隔采集点实现。比如，当第二二维点集Pf中分布在左鱼眼图像上的子集Pf0不是闭合的点集时，在该左鱼眼图像中鱼眼镜头的某个视野角度的视场角边界上，等间隔采集多个点(如500个点)形成额外点集Pfe。其中，所选的视野角度大于180度且小于鱼眼镜头最大的FOV(视场角边界)均可。“视场角边界”是指鱼眼图像中某个FOV所覆盖的区域可以理想化为：以鱼眼镜头中心区域某点C为圆心，R为半径所确定的圆形区域，R由FOV计算而得，C由标定而得，该圆形区域的边界为“视场角边界”。然后，将额外点集Pfe进行反投影，反投影即为首先采用球面等距投影方式先将额外点集Pfe投影到设定的球面坐标系上形成三 维点集，再利用球面透视将该三维点集投影到渲染区域所在的平面上，将投影落在渲染区域内的点放入集合中组成点集Pr。即，点集Pr是由额外点集Pfe的子集pfe0投影所得，所以点集Pfe0与Pf0的并集是可以构成一个闭合的点集的。因此，在鱼眼图像中鱼眼镜头的视野边界上等间隔采集点，得到额外点集Pfe，额外点集的子集Pfe0与子集Pf0合并可以构成闭合的点集。
本实施例中，通过投影和反投影的映射方式确定解码端渲染区域对应的鱼眼渲染区域，能够得到与解码端渲染区域相对应的压缩区域，确保压缩后渲染区域的清晰度。
在一个实施例中，对鱼眼图像进行压缩，得到压缩图像，包括：根据预设压缩比以及鱼眼图像的分辨率确定压缩图的面积；根据第一压缩比对鱼眼图像进行降采样，得到鱼眼缩略图；当鱼眼缩略图与鱼眼渲染区域的像素点数量总和小于或等于压缩图的像素点数量总和时，将鱼眼渲染区域中的像素点以及鱼眼缩略图的像素点存储到压缩图中。
其中，预设压缩比为预设的值，由鱼眼视频流的传输性能决定，一般确定一个合适的压缩比来使得视频流可以低延时传输。压缩图像则用于来存储鱼眼缩略图和鱼眼渲染区域。
具体地，压缩图像的大小通过预设压缩比和鱼眼图像的分辨率计算得到。比如，若预设压缩比为K:1，而分辨率为Wf*Hf，即可得到压缩图像的压缩图像面积S＝Wf*Hf/K。按照确定的压缩图像面积S生成对应的压缩图像。按照第一压缩比，比如500:1对鱼眼图像进行降采样得到鱼眼缩略图。也就是说，预设压缩比与第一压缩比有关，通常第一压缩比大于预设压缩比。
在进行存储之前，首先判断鱼眼缩略图与鱼眼渲染区域的像素点数量总和与压缩图像的像素点数量总和的大小关系，根据大小关系确定对鱼眼渲染区域是否压缩。具体地，当鱼眼缩略图与鱼眼渲染区域的像素点数量总和小于或等于压缩图像的像素点数量总和时，即各鱼眼渲染区域的总面积Sr≤(压缩图像的面积S-鱼眼缩略图面积w*h)时，表示压缩图像可以同时将鱼眼缩略图与鱼眼渲染区域的像素点存储，则直接按行将鱼眼缩略图与鱼眼渲染区域的像素点存储到压缩图像中，此时，对鱼眼渲染区域的像素点不进行压缩，实现对鱼眼渲染区域的无损压缩。在实际应用中，还可以将渲染区域的定位信息存储到压缩图像中。其中，渲染区域的定位信息还可以采用其它方式存储，能够实现编码端和解码端的定位信息的传输即可。
在另一个实施例中，当鱼眼缩略图与鱼眼渲染区域的像素点数量总和大于压缩图像的像素点数量总和时，采用第二压缩比对鱼眼渲染区域进行压缩，将压缩后的鱼眼渲染区域中的像素点以及所述鱼眼缩略图的像素点存储到压缩图像中；其中，第二压缩比小于第一压缩比。
具体地，当鱼眼缩略图与鱼眼渲染区域的像素点数量总和大于压缩图像的像素点数量总和时，即Sr＞S-w*h时，表示压缩图像无法同时存储鱼眼缩略图与鱼眼渲染区域的像素点，则以第二压缩比对鱼眼渲 染区域进行降采样，再按行将降采样后的各鱼眼渲染区域以及鱼眼缩略图的像素点存储到压缩图像中。其中，第二压缩比小于第一压缩比，例如，第二压缩比K'为K'＝Sr/(S-w*h)。本实施例中，存储到压缩图像之前通过判断大小关系确定是否再次降采样后存储，防止超过所生成压缩图像能够存储的量。第一压缩比和第二压缩比与预设压缩比相关，第一压缩比大于预设压缩比，第二压缩比小于预设压缩比。
在另一个实施例中，将鱼眼渲染区域中的像素点存储到压缩图像中的方式，包括：将鱼眼渲染区域按预设方向依次提取像素点，按提取顺序依次将提取的像素点存储至压缩图像中；预设方向包括行或列。
本实施例中，将鱼眼图像对应的鱼眼渲染区域中的像素按照密集存储的方式存储到压缩图像中，从而完成压缩。各像素存储到压缩图像中的排列方式可以任意，只要遵循便于存储便于解码的原则即可。具体地，将鱼眼渲染区域按预设方向依次提取像素点，如，按以行或列为单位依次取出像素点，按提取顺序依次将提取的像素点存储至压缩图像中。采用该方式，可将像素点信息密集存储于压缩图像中，其中，在压缩图像中鱼眼渲染区域的第n行/列最后一个像素点紧接着第n+1行/列的第一个像素点。
可以理解的是，将鱼眼缩略图的像素点存储至压缩图像中的方式，与将鱼眼渲染区域中的像素点存储到压缩图像中的方式相同，此处不再赘述。
密集存储和普通存储的区别是，破坏了原图像，但保持图像的位置关系。普通存储按方块的方式进行存储，密集存储破坏了方块的概念，没有行和列的概念，如即在压缩图像的内存中，第n行最后一个像素后紧接着存储第n+1行的第一个像素。采用该方式，能够减小图像大小，便于存储，利于还原。
在另一个实施例中，对鱼眼图像进行压缩，得到压缩图像，包括：
生成至少一张降采样映射表，各降采样映射表记录了鱼眼非渲染区域与压缩图像的映射关系，以及鱼眼渲染区域与压缩图像的映射关系；根据各降采样映射表对鱼眼非渲染区域以及鱼眼渲染区域进行图像重映射，得到压缩图像。
具体的，生成对应区域具有降采样功能的映射表，即本实施例的降采样映射表。各降采样映射表记录了鱼眼非渲染区域与压缩图像的映射关系，以及鱼眼渲染区域与压缩图像的映射关系。由于是具有降采样功能的映射表，所以可以理解为在生成映射表的过程中已经完成了对区域的降采样。该降采样映射表中包括了其对应区域中各像素在鱼眼图像中的原定位信息。然后，根据所生成的映射表中的原定位信息，对鱼眼非渲染区域和鱼眼渲染区域进行图像重映射完成压缩得到压缩图像，压缩图像中所保存的画面即为图像重映射的映射结果。另外，也可以同时生成多张映射表，各张映射表的区别在于其所对应的区域不同，也就是多张映射表分别进行图像重映射所得到各映射结果分别是压缩图像中的一部分，而所有的映射结果保存在同一张压缩图像中能够得到完整的压缩图像。
其中，生成映射表的方式可以采用任意一种，主要遵循能够确保多分辨率降采样和保持鱼眼画面的局部连续性的目的即可。多分辨率降采样是指对鱼眼中确定的鱼眼渲染区域进行较低的压缩比的降采样或不进行降采样(即当压缩图像可以放的下完整的渲染区域的时候可以选择不进行降采样)，对鱼眼中的鱼眼非渲染区域进行较高压缩比的降采样。保持鱼眼画面的局部连续性是指鱼眼中某个局部中任意两个像素之间的相对位置关系在压缩图像中不发生变化，该目的有利于压缩图像在转成视频流的时候的视频流编码压缩。
本实施例中，采用映射表的方式进行压缩，无论是鱼眼渲染区域还是非渲染区域，其降采样步骤均可包含在生成映射表的过程中，直接通过一张或多张映射表就可以一步或分步得到整个压缩图像。
在一个实施例中，还提供一种鱼眼视频流压缩方法，该方法应用于如图1所示的编码端，该方法包括：获取鱼眼视频流；采用各实施例的鱼眼图像压缩方法，对所述鱼眼视频流的每一帧鱼眼图像进行压缩处理，得到每一帧鱼眼图像的压缩图像；基于所述鱼眼视频流各帧的压缩图像进行视频流压缩，得到压缩的鱼眼视频流。
其中，鱼眼视频流是指利用鱼眼镜头采集的视频图像。可以理解的是，鱼眼视频流的每一帧即为前面实施例中所提到的鱼眼图像。
针对各实施例的鱼眼图像压缩方法，已记载在前面的实施例中，此处不再展开说明。可以理解的是，针对每一帧鱼眼图像的压缩图像而言，通过根据渲染区域的定位信息，确定鱼眼图像的鱼眼渲染区域，对鱼眼渲染区域进行较小的压缩比压缩或是无损压缩，达到了压缩的目的，同时使得鱼眼渲染的画面能够不被过多压缩而导致画质下降，即能够保证鱼眼渲染区域画面的清晰度。
也就是说，在动态的鱼眼视频流中，能够根据视点的变化实时确定实际所需的鱼眼渲染区域，从而能够实时根据解码端视点的变化对鱼眼视频流进行动态压缩。
对于鱼眼视频流的各帧的压缩图像，采用视频流压缩方法，再次进行压缩。其中，视频流压缩方法如H.264等。视频流压缩处理得到压缩的鱼眼视频流。
上述的鱼眼视频流压缩，可以根据渲染端的实时预览区域在全景中所在的位置，动态调整鱼眼的渲染区域，能够实时根据解码端视点的变化对鱼眼视频流进行压缩，既保证了渲染区域的高清晰度，又达到很高的压缩比率，大大节省了视频流的传输带宽。
在一个实施例中，还提供一种全景视频生成方法，该方法应用于如图1所示的解码端，包括：获取压 缩的鱼眼视频流；其中，所述压缩的鱼眼视频流采用前面所记载的鱼眼视频流压缩方法处理得到；根据所述压缩的鱼眼视频流，得到多帧原鱼眼图像对应的压缩图像；对所述压缩图像进行还原，得到所述原鱼眼图像；拼接各所述原鱼眼图像，得到全景视频。
具体地，解码端获得编码端处理得到的鱼眼视频流。其中，编码端处理得到鱼眼视频流的方法已记载在前面的说明书中，此处不再展开。
可以理解的是，解码端获得的压缩的鱼眼视频流可能有多路，具体与设置在采集现场的鱼眼镜头的数量有关。若在鱼眼现场设置有两个鱼眼镜头，则解码端获得两路鱼眼视频流。
对于接收到的压缩的鱼眼视频流，采用与视频流压缩方法对应的解压缩方法，对鱼眼视频流进行解压缩，得到多帧原鱼眼图像对应的压缩图像。
具体地，采用与压缩方法对应的还原方法，对压缩图像进行还原，得到原鱼眼图像。
具体地，对于原鱼眼图像，进行匹配点查找，基于匹配点进行拼接，得到一个全景视频。
上述的全景视频生成方法，对于压缩的鱼眼视频流，进行视频流解压缩，压缩图像还原，以及拼接等处理，得到全景视频。一方面，由于全景视频是对压缩的鱼眼视频流进行处理得到的，可以根据渲染端的实时预览区域在全景中所在的位置，动态调整鱼眼的渲染区域，能够实时根据解码端视点的变化对鱼眼视频流进行压缩，既保证了渲染区域的高清晰度，又达到很高的压缩比率，大大节省了视频流的传输带宽。另一方面，编码端只对鱼眼图像和视频流进行压缩，不进行拼接，而是由解码端进行解压后再拼接，降低了对编码端的硬件性能要求很高。
在编码端完成压缩之后，即可将压缩后的压缩图像封装成数据流传输给解码端解码恢复显示。其中，恢复显示需要利用到渲染区域的定位信息。如前面所提到的，可以将渲染区域的定位信息存储到压缩图像中，传输至解码端。还可以采用其它方式存储，能够实现编码端和解码端的定位信息的传输即可。解码端在获取到渲染区域的定位信息后，根据所述原定位信息对所述压缩图像进行解码，恢复得到所述原鱼眼图像。
其中，针对渲染区域的定位信息存储在压缩图像的方案而言，对所述压缩图像进行还原，得到所述原鱼眼图像，包括：解析所述压缩图像，获得鱼眼渲染区域中各像素点在所述原始鱼眼图像中的原定位信息；根据所述原定位信息对所述压缩图像进行解码，恢复得到所述原鱼眼图像。
具体地，当编码端的压缩图像是通过密集存储得到的，则在将压缩图像传输给解码端的同时，需要将存储在压缩图像中各像素在原鱼眼图上的原定位信息一并发送给到解码端。后续解码端根据原定位信息将压缩图像中该定位信息所指个数的像素拷贝到鱼眼中该定位信息所指的位置来完成原鱼眼图像中渲染区 域的解码恢复，非渲染区域可以通过压缩图像中存储的鱼眼缩略图升采样获得，渲染区域和非渲染区域共同构成恢复的原鱼眼图像。其中，若像素是按行存储，则各像素的原定位信息也可以按行来记录，即每一行像素的原定位信息可以有三个数据表示为：行首像素在原鱼眼图像中的行索引、列索引和该行像素总个数。
在另一种实施方式中，对所述压缩图像进行还原，得到所述原鱼眼图像，包括：获得根据降采样映射表生成的逆向映射表；根据所述逆向映射表对所述压缩图像重映射恢复得到所述原鱼眼图像。
具体地，当压缩图像是通过映射表进行图像重映射得到，则将压缩图像传输给解码端的同时需要将映射表对应的逆映射表或可以构造逆向映射表的参数信息一并传输给到解码端。后续解码端根据逆映射表对压缩图像中的像素进行图像重映射来解码恢复得到原鱼眼图。然后，解码端对解压缩得到的原鱼眼图进行全景拼接和渲染显示。
本实施例中，根据不同的压缩方式相解码端传输对应的解码信息，确保解码端能够高效完成鱼眼图的解码恢复。
应该理解的是，虽然图2-3的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2-3中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。
在一个实施例中，如图8所示，提供了一种鱼眼视频流压缩装置，包括：信息传输模块802、压缩区域确定模块804和压缩模块806，其中：
信息传输模块802，用于获取解码端渲染区域的定位信息；
压缩区域确定模块804，用于根据所述定位信息确定对应的鱼眼图像上的鱼眼渲染区域，所述鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域。
压缩模块806，用于对所述鱼眼图像进行压缩，得到压缩图像；其中，所述压缩图像中所述鱼眼渲染区域的压缩比小于所述鱼眼非渲染区域的压缩比，和/或所述鱼眼渲染区域不压缩。
在一个实施例中，压缩区域确定模块804还用于在所述渲染区域的边界上等间隔采集点，得到第一二维点集；根据所述渲染区域的定位信息，将所述第一二维点集投影到球面坐标系上，得到三维点集；将所 述三维点集投影到所述定位信息对应的鱼眼图像上，得到第二二维点集；根据所述第二二维点集确定所述鱼眼图像的鱼眼渲染区域。
在一个实施例中，压缩区域确定模块804还用于根据所述第二二维点集中首尾点之间的欧式距离判断所述第二二维点集是否为闭合的点集；当所述第二二维点集是闭合的点集时，将所述第二二维点集中的点按顺序连接得到的闭合边界所限定的内部区域作为所述鱼眼图像的鱼眼渲染区域；当所述第二二维点集不是闭合的点集时，构造闭合的第二二维点集，将构造的第二二维点集中的点按顺序连接得到的闭合边界所限定的内部区域作为所述鱼眼图像的鱼眼渲染区域。
在一个实施例中，压缩区域确定模块804还用于在鱼眼图像中鱼眼镜头的视场角边界上等间隔采集点，得到额外点集；将所述额外点集与所述第二二维点集合并，得到闭合的第二二维点集。
在一个实施例中，压缩模块806还用于根据预设压缩比以及所述鱼眼图像的分辨率确定压缩图像的面积；根据所述第一压缩比对所述鱼眼图像进行降采样，得到鱼眼缩略图；当所述鱼眼缩略图与所述鱼眼渲染区域的像素点数量总和小于或等于所述压缩图像的像素点数量总和时，将所述鱼眼渲染区域中的像素点以及所述鱼眼缩略图的像素点存储到压缩图像中。
在一个实施例中，压缩模块806还用于当所述鱼眼缩略图与所述鱼眼渲染区域的像素点数量总和大于所述压缩图像的像素点数量总和时，采用第二压缩比对所述鱼眼渲染区域进行压缩，将压缩后的所述鱼眼渲染区域中的像素点以及所述鱼眼缩略图的像素点存储到压缩图像中；其中，所述第二压缩比小于所述第一压缩比；所述预设压缩比与所述第一压缩比和所述第二压缩比相关。
在另一个实施例中，压缩模块，还用于将所述鱼眼渲染区域按预设方向依次提取像素点，按提取顺序依次将提取的像素点存储至所述压缩图像中，所述预设方向包括行或列。
其中，在所述压缩图像中所述鱼眼渲染区域的第n行/列最后一个像素点紧接着第n+1行/列的第一个像素点。
在一个实施例中，压缩模块806还用于生成至少一张降采样映射表，各所述降采样映射表记录了所述鱼眼非渲染区域与压缩图像的映射关系，以及所述鱼眼渲染区域与所述压缩图像的映射关系；根据各所述降采样映射表对所述鱼眼非渲染区域以及所述鱼眼渲染区域进行图像重映射，得到压缩图像。
关于鱼眼视频流压缩装置的具体限定可以参见上文中对于鱼眼视频流压缩方法的限定，在此不再赘述。上述鱼眼视频流压缩装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在另一个实施例中，还提供一种鱼眼视频流压缩装置，包括：
视频流获取模块，用于获取鱼眼视频流；
图像压缩模块，用于采用上述各实施例所述的鱼眼图像压缩方法，对所述鱼眼视频流的每一帧鱼眼图像进行压缩处理，得到每一帧鱼眼图像的压缩图像；
视频流压缩模块，用于基于所述鱼眼视频流各帧的压缩图像进行视频流压缩，得到压缩的鱼眼视频流。
在另一处实施例中，一种全景视频生成装置，包括：
视频流获取模块，用于获取压缩的鱼眼视频流；其中，所述压缩的鱼眼视频流采用如上述各实施例的鱼眼视频流压缩方法处理得到；
视频流解压缩模块，用于根据所述压缩的鱼眼视频流，得到多帧原鱼眼图像对应的压缩图像；
还原模块，用于对所述压缩图像进行还原，得到所述原鱼眼图像；
拼接模块，用于拼接各所述原鱼眼图像，得到全景视频。
其中，还原模块，用于解析所述压缩图像，获得鱼眼渲染区域中各像素点在所述原始鱼眼图像中的原定位信息；根据所述原定位信息对所述压缩图像进行解码，恢复得到所述原鱼眼图像。
其中，还原模块，用于获得根据降采样映射表生成的逆向映射表；根据所述逆向映射表对所述压缩图像重映射恢复得到所述原鱼眼图像。
在一个实施例中，提供了一种计算机设备，该计算机设备可以是终端，其内部结构图可以如图9所示。该计算机设备包括通过系统总线连接的处理器、存储器、通信接口、显示屏和输入装置。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机程序。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的通信接口用于与外部的终端进行有线或无线方式的通信，无线方式可通过WIFI、运营商网络、NFC(近场通信)或其他技术实现。该计算机程序被处理器执行时以实现一种鱼眼视频流压缩方法、鱼眼视频流压缩方法或全景视频生成方法。该计算机设备的显示屏可以是液晶显示屏或者电子墨水显示屏，该计算机设备的输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。
本领域技术人员可以理解，图9中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的 部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，提供了一种计算机设备，包括存储器和处理器，存储器中存储有计算机程序，该处理器执行计算机程序时实现上述各实施例的鱼眼图像压缩方法、鱼眼视频流压缩方法或全景视频生成方法。
在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现上述各实施例的鱼眼图像压缩方法、鱼眼视频流压缩方法或全景视频生成方法。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和易失性存储器中的至少一种。非易失性存储器可包括只读存储器(Read-Only Memory，ROM)、磁带、软盘、闪存或光存储器等。易失性存储器可包括随机存取存储器(Random Access Memory，RAM)或外部高速缓冲存储器。作为说明而非局限，RAM可以是多种形式，比如静态随机存取存储器(Static Random Access Memory，SRAM)或动态随机存取存储器(Dynamic Random Access Memory，DRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (16)

  1. 一种鱼眼图像压缩方法，其特征在于，所述方法包括：
    获取解码端渲染区域的定位信息；
    根据所述定位信息确定对应的鱼眼图像上的鱼眼渲染区域，所述鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域；
    对所述鱼眼图像进行压缩，得到压缩图像；其中，所述压缩图像中所述鱼眼渲染区域的压缩比小于所述鱼眼非渲染区域的压缩比，和/或所述鱼眼渲染区域不压缩。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述定位信息确定对应的鱼眼图像上的鱼眼渲染区域，包括：
    在所述渲染区域的边界上等间隔采集点，得到第一二维点集；
    根据所述渲染区域的定位信息，将所述第一二维点集投影到球面坐标系上，得到三维点集；
    将所述三维点集投影到所述定位信息对应的鱼眼图像上，得到第二二维点集；
    根据所述第二二维点集确定所述鱼眼图像的鱼眼渲染区域。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述第二二维点集确定所述鱼眼图像的鱼眼渲染区域，包括：
    根据所述第二二维点集中首尾点之间的欧式距离判断所述第二二维点集是否为闭合的点集；
    当所述第二二维点集是闭合的点集时，将所述第二二维点集中的点按顺序连接得到的闭合边界所限定的内部区域作为所述鱼眼图像的鱼眼渲染区域；
    当所述第二二维点集不是闭合的点集时，构造闭合的第二二维点集，将构造的所述第二二维点集中的点按顺序连接得到的闭合边界所限定的内部区域作为所述鱼眼图像的鱼眼渲染区域。
  4. 根据权利要求3所述的方法，其特征在于，所述构造闭合的第二二维点集，包括：
    在所述鱼眼图像中鱼眼镜头的视场角边界上等间隔采集点，得到额外点集；
    将所述额外点集与所述第二二维点集合并，得到闭合的第二二维点集。
  5. 根据权利要求1所述的方法，其特征在于，所述对所述鱼眼图像进行压缩，得到压缩图像，包括：
    根据预设压缩比以及所述鱼眼图像的分辨率确定压缩图像的面积；
    根据第一压缩比对所述鱼眼图像进行降采样，得到鱼眼缩略图；
    当所述鱼眼缩略图与所述鱼眼渲染区域的像素点数量总和小于或等于所述压缩图像的像素点数量总和时，将所述鱼眼渲染区域中的像素点以及所述鱼眼缩略图的像素点存储到压缩图像中。
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：
    当所述鱼眼缩略图与所述鱼眼渲染区域的像素点数量总和大于所述压缩图像的像素点数量总和时，采用第二压缩比对所述鱼眼渲染区域进行压缩，将压缩后的所述鱼眼渲染区域中的像素点以及所述鱼眼缩略图的像素点存储到压缩图像中；其中，所述第二压缩比小于所述第一压缩比；所述预设压缩比与所述第一压缩比和所述第二压缩比相关。
  7. 根据权利要求5或6所述的方法，其特征在于，将所述鱼眼渲染区域中的像素点存储到压缩图像中的方式，包括：
    将所述鱼眼渲染区域按预设方向依次提取像素点，按提取顺序依次将提取的像素点存储至所述压缩图像中；所述预设方向包括行或列。
  8. 根据权利要求7所述的方法，其特征在于，在所述压缩图像中所述鱼眼渲染区域的第n行/列最后一个像素点紧接着第n+1行/列的第一个像素点。
  9. 根据权利要求1所述的方法，其特征在于，所述对所述鱼眼图像进行压缩，得到压缩图像，包括：
    生成至少一张降采样映射表，各所述降采样映射表记录了所述鱼眼非渲染区域与压缩图像的映射关系，以及所述鱼眼渲染区域与所述压缩图像的映射关系；
    根据各所述降采样映射表对所述鱼眼非渲染区域以及所述鱼眼渲染区域进行图像重映射，得到压缩图像。
  10. 一种鱼眼视频流压缩方法，其特征在于，包括：
    获取鱼眼视频流；
    采用如权利要求1-9的任一项所述的鱼眼图像压缩方法，对所述鱼眼视频流的每一帧鱼眼图像进行压缩处理，得到每一帧鱼眼图像的压缩图像；
    基于所述鱼眼视频流各帧的压缩图像进行视频流压缩，得到压缩的鱼眼视频流。
  11. 一种全景视频生成方法，其特征在于，包括：
    获取压缩的鱼眼视频流；其中，所述压缩的鱼眼视频流采用如权利要求10所述的鱼眼视频流压缩方法处理得到；
    根据所述压缩的鱼眼视频流，得到多帧原鱼眼图像对应的压缩图像；
    对所述压缩图像进行还原，得到所述原鱼眼图像；
    拼接各所述原鱼眼图像，得到全景视频。
  12. 根据权利要求11所述的方法，其特征在于，所述对所述压缩图像进行还原，得到所述原鱼眼图像，包括：
    解析所述压缩图像，获得鱼眼渲染区域中各像素点在所述原始鱼眼图像中的原定位信息；
    根据所述原定位信息对所述压缩图像进行解码，恢复得到所述原鱼眼图像。
  13. 根据权利要求11所述的方法，其特征在于，所述对所述压缩图像进行还原，得到所述原鱼眼图像，包括：
    获得根据降采样映射表生成的逆向映射表；
    根据所述逆向映射表对所述压缩图像重映射恢复得到所述原鱼眼图像。
  14. 一种鱼眼视频流压缩装置，其特征在于，所述装置包括：
    信息传输模块，用于获取解码端渲染区域的定位信息；
    压缩区域确定模块，用于根据所述定位信息确定对应的鱼眼图像上的鱼眼渲染区域，所述鱼眼图像上除鱼眼渲染区域以外的区域为鱼眼非渲染区域；
    压缩模块，用于对所述鱼眼图像进行压缩，得到压缩图像；其中，所述压缩图像中所述鱼眼渲染区域的压缩比小于所述鱼眼非渲染区域的压缩比，和/或所述鱼眼渲染区域不压缩。
  15. 一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至13中任一项所述的方法的步骤。
  16. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至13中任一项所述的方法的步骤。
PCT/CN2022/104346 2021-07-09 2022-07-07 鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法 WO2023280266A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110780685.6 2021-07-09
CN202110780685.6A CN115604528A (zh) 2021-07-09 2021-07-09 鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法

Publications (1)

Publication Number Publication Date
WO2023280266A1 true WO2023280266A1 (zh) 2023-01-12

Family

ID=84800329

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/104346 WO2023280266A1 (zh) 2021-07-09 2022-07-07 鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法

Country Status (2)

Country Link
CN (1) CN115604528A (zh)
WO (1) WO2023280266A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116909407A (zh) * 2023-09-12 2023-10-20 深圳康荣电子有限公司 基于虚拟现实的触控显示屏全景交互方法及控制系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107065197A (zh) * 2017-06-20 2017-08-18 合肥工业大学 面向vr眼镜的人眼跟踪远程渲染实时显示方法及系统
US20180061084A1 (en) * 2016-08-24 2018-03-01 Disney Enterprises, Inc. System and method of bandwidth-sensitive rendering of a focal area of an animation
CN107820012A (zh) * 2017-11-21 2018-03-20 暴风集团股份有限公司 一种鱼眼图像处理方法、装置、服务器及系统
CN108391133A (zh) * 2018-03-01 2018-08-10 京东方科技集团股份有限公司 显示数据的处理方法、处理设备和显示设备
CN108665521A (zh) * 2018-05-16 2018-10-16 京东方科技集团股份有限公司 图像渲染方法、装置、系统、计算机可读存储介质及设备
CN109792490A (zh) * 2018-06-07 2019-05-21 香港应用科技研究院有限公司 用于流图像压缩的球面视频的改进的伪圆柱映射
CN111757090A (zh) * 2019-03-27 2020-10-09 北京传送科技有限公司 基于注视点信息的实时vr图像过滤方法、系统和存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180061084A1 (en) * 2016-08-24 2018-03-01 Disney Enterprises, Inc. System and method of bandwidth-sensitive rendering of a focal area of an animation
CN107065197A (zh) * 2017-06-20 2017-08-18 合肥工业大学 面向vr眼镜的人眼跟踪远程渲染实时显示方法及系统
CN107820012A (zh) * 2017-11-21 2018-03-20 暴风集团股份有限公司 一种鱼眼图像处理方法、装置、服务器及系统
CN108391133A (zh) * 2018-03-01 2018-08-10 京东方科技集团股份有限公司 显示数据的处理方法、处理设备和显示设备
CN108665521A (zh) * 2018-05-16 2018-10-16 京东方科技集团股份有限公司 图像渲染方法、装置、系统、计算机可读存储介质及设备
CN109792490A (zh) * 2018-06-07 2019-05-21 香港应用科技研究院有限公司 用于流图像压缩的球面视频的改进的伪圆柱映射
CN111757090A (zh) * 2019-03-27 2020-10-09 北京传送科技有限公司 基于注视点信息的实时vr图像过滤方法、系统和存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116909407A (zh) * 2023-09-12 2023-10-20 深圳康荣电子有限公司 基于虚拟现实的触控显示屏全景交互方法及控制系统
CN116909407B (zh) * 2023-09-12 2024-01-12 深圳康荣电子有限公司 基于虚拟现实的触控显示屏全景交互方法及控制系统

Also Published As

Publication number Publication date
CN115604528A (zh) 2023-01-13

Similar Documents

Publication Publication Date Title
WO2022057837A1 (zh) 图像处理和人像超分辨率重建及模型训练方法、装置、电子设备及存储介质
KR102458339B1 (ko) 360도 3d 입체 영상을 생성하는 전자 장치 및 이의 방법
CN112204993B (zh) 使用重叠的被分区的分段的自适应全景视频流式传输
US10694102B2 (en) Method to capture, store, distribute, share, stream and display panoramic image or video
CN107454468B (zh) 对沉浸式视频进行格式化的方法、装置和流
CN109194923B (zh) 基于局部非均匀分辨率的视频图像处理设备、系统及方法
WO2017215587A1 (zh) 一种视频图像的编解码方法和装置
CN101689292B (zh) Banana多媒体数字信号编解码器
KR102208773B1 (ko) 파노라마 영상 압축 방법 및 장치
US11483475B2 (en) Adaptive panoramic video streaming using composite pictures
US10235795B2 (en) Methods of compressing a texture image and image data processing system and methods of generating a 360 degree panoramic video thereof
US11941748B2 (en) Lightweight view dependent rendering system for mobile devices
KR102308604B1 (ko) 레거시 및 몰입형 렌더링 디바이스들에 대한 몰입형 비디오를 포맷하는 방법, 장치 및 스트림
WO2018068680A1 (zh) 图像处理方法和装置
US20200413017A1 (en) Methods and apparatus for projection conversion decoding for applications eco-systems
CN111669564B (zh) 图像重建方法、系统、设备及计算机可读存储介质
WO2023280266A1 (zh) 鱼眼图像压缩、鱼眼视频流压缩以及全景视频生成方法
KR20220035229A (ko) 볼류메트릭 비디오 콘텐츠를 전달하기 위한 방법 및 장치
WO2019127100A1 (zh) 视频编码的方法、装置和计算机系统
CN111034187A (zh) 动态图像的生成方法、装置、可移动平台和存储介质
TW202116063A (zh) 用以編碼、傳輸及解碼容積視訊之方法及裝置
WO2022199584A1 (zh) 全景视频压缩方法、装置、计算机设备和存储介质
CN112911186A (zh) 图像存储方法及装置、电子设备、存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22837011

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE