WO2024060281A1

WO2024060281A1 - Video splitting method and apparatus based on panoramic image, and device and storage medium

Info

Publication number: WO2024060281A1
Application number: PCT/CN2022/122388
Authority: WO
Inventors: 邢冬逢; 李国强
Original assignee: 深圳看到科技有限公司
Priority date: 2022-09-20
Filing date: 2022-09-29
Publication date: 2024-03-28
Also published as: CN115484417A

Abstract

The present application relates to image processing technology. Provided in the present application are a video splitting method and apparatus based on a panoramic image, and a device and a storage medium. In the method, the size of a split image is determined according to the resolution size of a panoramic image, starting coordinates of the split image in the panoramic image are set, and starting coordinates of all split images in the panoramic image can then be sequentially determined according to the size of the split image and the size of a preset overlapped image; and image data of the panoramic image is read by means of a row stride encoder, and according to the starting coordinates of each split image in the panoramic image and the size of the split image, data of the split images can be directly output from the data of the panoramic image, such that the split images are directly output without the need to perform copying, cropping and splicing on the panoramic image, thereby reducing the memory pressure and data processing amount of a processing device and improving the split output efficiency of the panoramic image.

Description

Panoramic image-based video offloading method, device, equipment and storage medium

Technical field

The present invention relates to the technical field of image processing, and in particular to a video streaming method, device, equipment and storage medium based on panoramic images.

Background technique

The existing security system uses multiple monitoring devices to monitor different directions of the same area, thereby achieving coverage of the same monitoring area. However, multiple devices need to be equipped with corresponding processing equipment to process the images of each device, resulting in increased system deployment costs; therefore, existing security systems have gradually begun to use panoramic camera equipment to cover the monitoring area.

technical problem

However, the panoramic images output by panoramic camera equipment have a large resolution. Loading and outputting large-resolution images requires powerful data processing capabilities. Existing monitoring equipment generally copies panoramic images in segments and then encodes the copied partial images. The image data processing volume is large and the efficiency of the diversion output of panoramic images is low. Therefore, how to solve the low efficiency of the existing diversion output of panoramic images has become a technical problem that needs to be solved urgently.

Technical solutions

The main purpose of the present invention is to provide a video streaming method, device, equipment and storage medium based on panoramic images, aiming to solve the technical problem of low efficiency in the streaming output of existing panoramic images.

In order to achieve the above object, the present invention provides a panoramic image-based video offloading method. The panoramic image-based video offloading method includes: setting the offloading image size according to the effective resolution of the panoramic image; Assume that the overlapping area size and the split image are at the preset coordinate position of the panoramic image, and set the starting coordinates of each split image; set the split image size, the row pixel value of the panoramic image, and each split image The starting coordinates are input to the row stride encoder to output the shunt image.

In addition, to achieve the above object, the present invention also provides a panoramic image-based video splitting device. The panoramic image-based video splitting device includes: a splitting image size setting module for setting the splitting according to the effective resolution of the panoramic image. Image size; starting coordinate setting module, used to set the starting coordinates of each shunt image according to the shunt image size, the preset overlap area size and the preset coordinate position of the shunt image in the panoramic image; shunt image An output module is configured to input the split image size, the row pixel value of the panoramic image, and the starting coordinates of each split image to a row stride encoder to output the split image.

In addition, to achieve the above object, the present invention also provides a panoramic image-based video streaming device. The panoramic image-based video streaming device includes a processor, a memory, and a device that is stored on the memory and can be used by the processor. A panoramic image-based video offloading program is executed, wherein when the panoramic image-based video offloading program is executed by the processor, the above panoramic image-based video offloading steps are implemented.

In addition, to achieve the above object, the present invention also provides a computer-readable storage medium, which stores a panoramic image-based video streaming program, wherein the panoramic image-based video streaming program is processed by a processor. When executed, the steps of the above panoramic image-based video offloading method are implemented.

beneficial effects

The present invention provides a video shunting method based on a panoramic image, wherein the method sets the size of the shunting image according to the effective resolution of the panoramic image; sets the starting coordinates of each shunting image according to the size of the shunting image, the size of the preset overlapping area, and the preset coordinate position of the shunting image in the panoramic image; and inputs the size of the shunting image, the row pixel value of the panoramic image, and the starting coordinates of each shunting image into a row stride encoder to output the shunting image. In the above manner, the present invention determines the size of the shunting image according to the resolution of the panoramic image, and sets the starting coordinates of the shunting image in the panoramic image, and then according to the preset overlapping image, the starting coordinates of all shunting images in the panoramic image can be determined in sequence; the image data of the panoramic image is read through the row stride encoder, and according to the starting coordinates of the shunting image and the size of the shunting image, the shunting image data can be directly output from the panoramic image data, thereby outputting the shunting image, without the need to copy, crop, and then splice the panoramic image, thereby reducing the amount of data processing, improving the shunting output efficiency of the panoramic image, and solving the technical problem of low shunting output efficiency of the existing panoramic image.

Description of the drawings

Figure 1 is a schematic diagram of the hardware structure of a panoramic image-based video streaming device involved in the embodiment of the present invention.

FIG. 2 is a flow chart of a first embodiment of a video splitting method based on panoramic images according to the present invention.

Figure 3 is a schematic flowchart of the second embodiment of the panoramic image-based video offloading method of the present invention.

Figure 4 is a schematic diagram of image offloading of the panoramic image-based video offloading method of the present invention.

FIG. 5 is a schematic flowchart of the third embodiment of the panoramic image-based video offloading method of the present invention.

Figure 6 is a schematic flowchart of the fourth embodiment of the panoramic image-based video offloading method of the present invention.

Figure 7 is a schematic diagram of the functional modules of the first embodiment of the panoramic image-based video streaming device of the present invention.

Best Mode of Carrying Out the Invention

It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

The panoramic image-based video offloading method involved in the embodiment of the present invention is mainly applied to the panoramic image-based video offloading device. The panoramic image-based video offloading device can be a PC, a portable computer, a mobile terminal and other devices with display and processing functions.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of the hardware structure of a panoramic image-based video streaming device involved in the embodiment of the present invention. In the embodiment of the present invention, the panoramic image-based video distribution device may include a processor 1001 (such as a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize connection and communication between these components; the user interface 1003 can include a display screen (Display) and an input unit such as a keyboard (Keyboard); the network interface 1004 can optionally include a standard wired interface and a wireless interface. (such as WI-FI interface); the memory 1005 can be a high-speed RAM memory or a stable memory (non-volatile memory), such as disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

Those skilled in the art can understand that the hardware structure shown in Figure 1 does not constitute a limitation on the video streaming device based on panoramic images, and may include more or less components than shown, or some components may be combined or different. component layout.

Continuing to refer to FIG. 1 , the memory 1005 as a computer-readable storage medium in FIG. 1 may include an operating system, a network communication module, and a panoramic image-based video streaming program.

In Figure 1, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the panoramic image-based video streaming program stored in the memory 1005, and execute the panoramic image-based video streaming program provided by the embodiment of the present invention. video offloading method.

The embodiment of the present invention provides a video streaming method based on panoramic images.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of the first embodiment of the panoramic image-based video offloading method of the present invention.

In this embodiment, the panoramic image-based video offloading method includes the following steps:

Step S10, set the size of the shunt image according to the effective resolution of the panoramic image;

In this embodiment, the panoramic image is captured by a panoramic camera. The panoramic camera collects images through at least two fisheye lenses and splices them into a 360-degree panoramic video stream. However, due to the wide-angle characteristics of the fisheye lens, there is large distortion at the top and bottom of the captured image, which has no reference value. Therefore, it is necessary to crop the part with small distortion in the panoramic image as an effective image. For example, the resolution of the panoramic image is 7680×3840, the effective resolution after cropping is 7680×1440, and a total of 2000 rows of pixel values are cropped at the top and bottom.

Specifically, according to the characteristics of the fisheye lens, the panoramic image only has distortion at the upper and lower ends, and the pixels in the middle row will not be distorted. When the distorted pixels at the upper and lower ends of the full panoramic image are cropped, the pixel row values of the panoramic image decrease, and The column values have not changed, so the effective resolution of the panorama image is the product of the original column values of the panorama image and the cropped row values.

Understandably, the effective row pixel value of the panoramic image refers to the column value of the effective resolution, that is, how many pixels a row of the effective resolution has represents how many columns the effective resolution has; the effective column pixel value refers to the row of the effective resolution. Value, that is, how many pixels a column of effective resolution has, indicating how many rows there are of effective resolution.

Further, the step S10 specifically includes:

The row value of the split image resolution is determined based on the effective resolution of the panoramic image.

According to the preset splitting angle, the proportion of the splitting image in the panoramic image is calculated, and the column value of the splitting image resolution is determined.

In this embodiment, in the effective resolution of the panoramic image, because the pixels with large distortion in the panoramic image have been excluded before determining the effective resolution, if the split image wants to cover the effective resolution, the rows in the effective resolution are value (i.e. width) as the resolution line value of the split image.

It can be understood that if the row value (i.e., width) of the effective resolution of the panoramic image is too large, the row value can also be divided, and multiple diversion images can be set to cover the effective resolution of the panoramic image; for example, if the effective resolution row value is 1000, the diversion image resolution row value can be set to 100, and there is a certain overlapping area between the two diversion images to ensure that the panoramic image can be fully covered to avoid image missing.

In this embodiment, according to the preset number of diverted videos and based on the 360-degree panoramic image, a preset diversion angle is set. The overlapping coverage area needs to be considered here. According to the ratio of the preset diversion angle to 360 degrees, the area of the panoramic image that can be covered by the preset diversion angle is calculated. The panoramic image resolution column value (i.e., length) contained in the coverage area is the column value of the diverted image resolution.

Furthermore, by combining the row values and column values, the size of the splitting resolution can be determined.

It can be understood that offloading is to intercept pictures of corresponding angles from the panoramic 360-degree picture and send them to the receiving end.

In specific embodiments, the angle is configured in the background (or monitoring), and any angle can be set for each stream. For example, if a panoramic video stream is divided into 4 ordinary video streams (non-panoramic), the operating terminal can pull 4 video streams at the same time. The horizontal FOV of each video stream is 120 degrees. To ensure complete coverage of the panoramic image, adjacent The two video streams overlap by 30 degrees. Taking the panoramic image 7680×1440 as an example, after cutting into 4 streams horizontally, the image resolution of each stream is 2560×1440, and the overlap between two adjacent streams is 640× 1440.

Understandably, the angle of each stream of the pulled 4-channel video stream can be set independently. It can be configured to the camera by calling the interface of the camera side through the operation terminal. The camera intercepts the stream of the corresponding angle from the panoramic stream according to this configuration and sends it to the camera. monitor.

Step S20: Set the starting coordinates of each split image according to the split image size, the preset overlap area size and the preset coordinate position of the split image in the panoramic image.

In this embodiment, when the size of the split image is determined, the encoder needs to be given an encoding starting coordinate so that the encoder can read the split image in the panoramic image; in order to cover all effective areas of the panoramic image and avoid the problem of missing pictures, There needs to be a certain range of preset overlapping areas between shunt images. A reasonable overlap area size can be set based on the size of the shunt images and the number of shunts.

Specifically, the pixels in the first column of the panoramic image can be used as the starting column coordinates of the first split image, and the coordinates of the first row of pixels in the panoramic image at the effective resolution of the panoramic image can be used as the starting row of the first split image. coordinates, thereby determining the starting coordinates of the first shunt image in the panoramic image, and then deriving the starting coordinates of other shunt images in the panoramic image based on the overlapping area; the starting coordinates of each shunt image can also be artificially set according to the actual needs of the user. coordinate.

Understandably, the size of the preset overlapping area cannot be too large, otherwise it will easily lead to excessive picture duplication, data redundancy, and a waste of computing power; it cannot be too small, otherwise the coverage effect will not be achieved. If the edge area of the display device is incompletely displayed, It is easy to cause the edge area of the displayed image to be missing.

Specifically, the size of the overlapping area can be set according to reasonable rules, such as setting the overlapping area to account for 15% of the split image, or when setting the angle of the split image, specifying a certain angle range as the overlapping area, etc.

Further, the step S20 specifically includes:

According to the effective resolution of the panoramic image, the readable area of the panoramic image by the row stride encoder is determined.

The starting coordinates of the diversion image in the panoramic image are set according to the readable area and the size of the diversion image.

The starting coordinates of adjacent split images of the split image are determined according to the preset overlapping image size and the starting coordinates.

In this embodiment, first determine the row value (width) of the panoramic image resolution, and then determine the position of the effective resolution in the panoramic image resolution, because the column value (length) of the effective resolution is equal to the column value of the panoramic image resolution. , so it is only necessary to determine that the first row of pixels of the effective resolution is located at the row coordinates of the panoramic image resolution, and the row coordinates of the first row of pixels of the effective image resolution are used as the starting row coordinates of the first split image.

Specifically, first determine the row coordinates of the first row of pixels of the effective resolution in the panoramic image resolution, and then determine the row coordinates of the last row of the effective resolution in the panoramic image resolution based on the row value (width) of the effective resolution; Combined with the effective resolution size, the coordinates of the row stride encoder in the readable area of the panoramic image can be determined.

Optionally, according to the split image resolution, the starting line coordinates of the split image resolution are set in the readable area; for example, the starting row coordinates of the effective resolution can be used as the starting line coordinates of the first split image resolution.

Specifically, the starting column coordinates of the split image resolution can be set through the operation terminal. The first column of the effective image resolution can be defaulted as the starting column coordinate of the first split image resolution, or any of the effective image resolutions can be set. A column that serves as the starting column coordinates of the first shunt image resolution.

Specifically, according to the starting row coordinates and starting column coordinates of the first split image resolution, the resolution coordinates of the first split image in the panoramic image can be determined; and then according to the preset overlapping resolution, each step can be determined in turn. The resolution coordinate of a shunt image in the panoramic image.

Step S30: The split image size, the row pixel value of the panoramic image, and the starting coordinates of the split image are input to a row stride encoder to output the split image.

Generally, to capture a small picture from a large picture, you usually need to apply for a memory from the system, then copy the pixels of the small picture to this memory, and then give this memory to the encoder to encode and generate a video stream; Since the panoramic video is relatively large, doing so consumes a lot of CPU and memory resources, which affects performance on embedded systems.

In this embodiment, the present application utilizes the row stride of the encoder (the row span of the memory image) to directly send the large picture to the encoder. For example, the encoding area is set to 2560×1440, the stride is 7680, and the memory copy part is removed to achieve the required performance. The encoder reads data according to the resolution of the large panoramic image, and reads the pixel data of the panoramic image in sequence according to the row order of the panoramic image resolution; for ease of understanding, each row of pixels of the panoramic image resolution is regarded as a group of data, assuming that the panoramic image is 7680×1440, which means that the stride encoder will read 1440 groups of pixel data; if the starting coordinates of the first shunt image in the panoramic image are (0, 0), then the stride encoder actually needs to encode the 1st data to the 2560th data in each group of pixel data, and the data segment from the 2561st data to the 7680th data of each row does not need to be processed or is directly deleted; that is, when the stride encoder completes the encoding of the 1st data to the 2560th data in the previous row of pixel data, the subsequent data of the previous row is no longer encoded, but continues to be encoded from the 1st data of the next row to the 2560th data of the row of pixel data, and the same processing is performed on each row of pixel data in sequence; finally, the encoded pixel data is output in sequence, thereby outputting the first shunt image.

In order to facilitate understanding by those skilled in the art, this embodiment is explained by exemplifying the generation process of the first shunt image.

Specifically, the generation process of the first split image based on the panoramic image is as follows:

In a specific embodiment, the panoramic image with row pixel values, the split image resolution size (ie split image size) and the starting coordinates of the first split image to be intercepted are input into the encoder, and the encoder calls the memory image row The stride (row stride) plugin processes panoramic images.

Optionally, the encoder reads the row pixel data of the panoramic image in row order, and determines the pixels that need to be encoded in the read panoramic image data according to the starting coordinates of the split image and the row value in the resolution size of the split image. Split image data, process the split image data by calling the stride plug-in, and directly output the split image.

In this embodiment, the resolution size covered by the overlapping area can be calculated based on the preset splitting angle, and the starting point of the second splitting image in the panoramic image can be determined based on the calculation based on the edge pixel column of the first splitting image. Column coordinates.

Specifically, the resolution of each branch image is the same. When the starting column of the second branch image is determined, the end column of the second branch image can be determined; and by analogy, the panoramic position of each branch image can be determined. The starting column and ending column in the image can determine the resolution coordinates of each split image in the panoramic image.

Specifically, the resolution coordinates of each split image to be intercepted are input into the encoder in sequence, the encoding starting coordinates of each split image are determined, and combined with the split image size, the video stream of each split image can be output in sequence.

In this embodiment, according to the preset overlapping area size and the starting coordinates of the first split image, the resolution coordinates of each channel can be determined, and the corresponding image data is read in the panoramic image through the row stride encoder, and then output as By shunting image data, multiple channels of video can be generated sequentially.

In this embodiment, the data of the panoramic image is read through the row stride encoder, and then the image data corresponding to the shunt image is directly output according to the coordinates of the shunt image in the panoramic image, and the invalid data that is not the shunt image data is not processed or Direct deletion reduces memory pressure and data processing volume, thereby improving the output efficiency of split images.

The present invention provides a video splitting method based on panoramic images. The method sets the splitting image size according to the effective resolution of the panoramic image; sets the starting point of the splitting image according to the splitting image size and the preset overlapping image size. Coordinates; input the split image size, the row pixel value of the panoramic image, and the starting coordinate of the split image to the row stride encoder to output the split image. Through the above method, the present invention determines the size of the shunt image according to the resolution of the panoramic image, and sets the starting coordinates of the shunt image in the panoramic image, and then based on the preset overlapping image, can sequentially determine the position of all shunt images in the panoramic image. The starting coordinates in; read the image data of the panoramic image through the row stride encoder, and according to the starting coordinates of the shunt image and the size of the shunt image, the shunt image data can be output directly from the panoramic image data, thereby outputting the shunt image, without Panoramic images need to be copied, cropped and then spliced, which reduces the amount of data processing, improves the efficiency of the split output of panoramic images, and solves the technical problem of low efficiency of split output of existing panoramic images.

Referring to FIG. 3 , FIG. 3 is a schematic flowchart of a video offloading method based on panoramic images according to a second embodiment of the present invention.

Based on the above embodiment shown in Figure 2, in this embodiment, step S30 includes:

Step S31: According to the starting coordinates of the split image, run the row stride encoder to read the image data of the panoramic image.

Step S32: Determine valid data in the image data read by the row stride encoder based on the split image size and the row pixel values of the panoramic image.

In this embodiment, the row alignment feature of stride can be used to allow the encoder to output the data of the first 1/4 coding area (2560×1440) in each data cycle in the stride as the data of the split image, and the following 3/4 If the data is deleted as invalid data, the shunt image can be directly output and the processing of redundant data can be avoided.

Further, the step S32 specifically includes:

Panoramic image row pixels are sequentially read through the row stride encoder to obtain panoramic image encoded data.

According to the starting coordinates of the split image and the split image size, the effective row pixels of each row of the split image in the corresponding row of the panoramic image are determined.

According to the valid row of pixels, valid data of the corresponding split image is determined in the panoramic image encoding data.

In this embodiment, after distortion screening, although the effective image resolution of the panoramic image is determined, the invalid pixels are not cropped, so it is necessary to determine the position of the effective resolution in the panoramic image, that is, the first effective resolution The coordinates of the row pixel and the last row pixel in the panoramic image.

Specifically, the encoder operates on the entire panoramic image, while the row stride encoder "screens" the shunt image data directly from the panoramic image data, so to read the shunt image correctly and completely, you need to determine the shunt image The position in the panoramic image is the starting coordinate of the shunt image.

Specifically, the effective resolution starting row can be determined to be located at the row coordinate of the panoramic image according to the distortion filtering operation. The starting column of the first split image can be any column of the effective resolution, and can start from the first column, or it can be Start with other columns.

Understandably, during the encoding process, the row stride encoder reads data sequentially according to the rows of the panoramic image. For example, the encoder reads the data of the first row of pixels in the complete panoramic image before reading the second row. The data of pixels, and the data length of each row of pixels is the same. The coordinates of the split image and the resolution size of the split image "tell" the row stride encoder how to "distinguish" valid data and invalid data.

Specifically, assuming that the effective resolution of the panoramic image is 10000×5000, the starting interception coordinates of the shunt image are (1000, 1000), and the resolution of the shunt image is 3000×1000; then the row stride encoder will read all the effective resolution data of the panoramic image, which is arranged in row order according to the effective resolution of the panoramic image, that is, 5000 rows of data, each row of data has 10000 pixel data; assuming that these data are numbered, with 10000 pixel data as a cycle data; when the shunt image data needs to be output, the row stride encoder will start from the 1000th cycle data, starting from the 1000th pixel data in each row of data, and read 3000 pixel data, a total of 1000 corresponding pixel data in the cycle data as valid data, that is, shunt image data, output the valid data, and then convert it into a shunt image; and the other panoramic image data is regarded as invalid data and can be directly deleted.

Specifically, refer to FIG. 4 , which is a schematic diagram of image offloading based on the panoramic image-based video offloading method of the present invention. Input the starting coordinates of each shunt image, the shunt image resolution (width value and height value) and the width value of the panoramic image into the stride encoder, and divide the panoramic image into shunt areas corresponding to multiple shunt images; and adjacent There are overlapping areas between the shunt images to ensure that the output shunt image can fully cover the panoramic image.

In a specific embodiment, the stride encoder reads each row of pixel data of the panoramic image in sequence according to the row order of the panoramic image. In the row corresponding to the row coordinates of the panoramic image of the split image, according to the starting coordinates of each split image, intercept the split image row corresponding to the width value of the split image in the corresponding row pixel data, and then skip the remaining rows of pixel data, Continue to read the corresponding shunt image row in the next row of the panoramic image until the reading of pixel data of all rows of the shunt image is completed.

Among them, when the split image has cross-rows, since the stride encoder reads pixel data sequentially in line order, there is a row of pixel misalignment in the edge split image across rows. At this time, you can choose to perform the panoramic image before splitting. Pre-processing, intercept the part of the panoramic image that has cross-lines and splice it to the other side of the panoramic image, thereby obtaining a panoramic image with a larger resolution and avoiding the stride encoder's cross-line reading of the edge shunt image.

Step S33: Output the valid data through the row stride encoder as split image data to obtain the split image.

In this embodiment, through the above implementation, there is no need to preprocess the panoramic image (such as segmentation, splicing, etc.). It is only necessary to directly read the corresponding pixel data from the panoramic image through the row stride encoder as the data of the split image. , thereby directly outputting the shunt image, and the data that needs to be processed is only the data corresponding to the shunt image, which reduces the amount of data processing and thereby improves the output efficiency of the shunt image.

Referring to FIG. 5 , FIG. 5 is a schematic flowchart of a third embodiment of a video offloading method based on panoramic images of the present invention.

Based on the above embodiment shown in Figure 2, in this embodiment, before step S10, it also includes:

Step S01: Screen all pixels of the panoramic image according to a preset distortion threshold, and determine the area where the distortion amount of the pixel is within the distortion threshold as a valid area;

Step S02: Determine the effective resolution of the panoramic image according to the pixel values contained in the effective area.

In this embodiment, the distortion threshold parameter is set, and the distortion amount of each pixel is calculated by measuring the distortion of all pixels of the panoramic image. For more than half of the pixel rows whose pixel distortion is greater than the distortion threshold, it is determined to be an invalid pixel row; For pixel rows in which more than half of the pixel distortion is less than the distortion threshold, it is determined to be a valid pixel row, and distortion correction is performed.

Specifically, distortion, also known as distortion, is an aberration that can turn straight lines into curves. Wide-angle lenses, zoom lenses, and telephoto lenses can easily cause severe distortion. Tangents are particularly noticeable at the edges of the image, but no distortion is visible for radial lines along the radius. Due to the inherent characteristics of optical lenses, distortion (distortion) is caused in the captured pictures, which is detrimental to the imaging quality.

Furthermore, after distortion correction, the pixel area determined to be the effective pixel row is regarded as the effective area, and all pixels contained in this area are effective pixels. Thus, the effective resolution of the panoramic image can be determined.

Referring to Figure 6, Figure 6 is a schematic flowchart of a fourth embodiment of a video offloading method based on panoramic images of the present invention.

Based on the above embodiment shown in Figure 2, in this embodiment, before step S30, it also includes:

Step S031, determining the missing image size of the edge diversion image according to the diversion image size, the preset overlapping image size and the starting coordinates of the diversion image;

Step S032: According to the missing image size, starting from the first column on the other side of the panoramic image, intercept a complementary image with the same size as the missing image;

Step S033: Align and splice the supplementary image to the edge position of the panoramic image, and update the panoramic image so that the edge shunt image is read completely.

In this embodiment, because there is an overlapping area between two adjacent shunt images, the edge shunt image on one side of the panoramic image is repeated with the shunt image on the other side of the panoramic image, and the encoder reads sequentially according to the image pixel rows. After completing the reading of a row of pixel data, it will jump to the first pixel of the next row of pixels and continue to read backward; when the edge shunt image is repeated with the shunt image on the other side, when the encoder reads the data, the front data and the rear data of each row of pixel data of the edge shunt image correspond to two rows of pixel data of the panoramic image, so each row of pixel data of the edge shunt image is misaligned by one row. Therefore, in order to make the edge shunt image read completely and the pixel data arranged in row order, the panoramic image needs to be preprocessed before reading the shunt image; that is, the complementary image is intercepted for splicing, and the panoramic image is updated to obtain a panoramic image with a higher resolution and the same screen at both ends.

Optionally, take the 4-channel split image as an example: in the 4-channel split image, part of it is on the left side of the panoramic image, and the other part is on the right side of the panoramic image. It happens that the leftmost and rightmost parts of the panoramic image can be connected. The image data is also contiguous in the encoder memory (the first pixel of a row follows the last pixel of the previous row).

Optionally, you can use the stride method to encode the fourth stream, but there will be a misalignment of one line (the left side of the first line and the right side of the second line). The misalignment of one line is not obvious in the picture.

Optionally, if resource consumption is not considered, since the right side of the panoramic screen of the fourth video stream spans to the left side of the panoramic screen, there may be an offset of row pixels. Therefore, if a plane crossing the boundary of the panoramic screen is generated, Before splitting the panoramic image, the left image of the panoramic image can be directly spliced to the right image in advance to ensure that the flat image does not span the spliced panoramic image. The size of the missing area can be determined based on the position of the edge shunt image in the panoramic image, and then based on the size of the missing area, an image area of the same size is intercepted on the other side of the panoramic image and spliced to the edge of the panoramic image to form a larger Panoramic image, such as changing the panoramic image from 360 degrees to 390 degrees; then resetting the image parameter value of the modified panoramic image The stride value (the image format and the length and width dimensions of the target image do not need to be modified).

Referring to Figure 6, Figure 6 is a schematic diagram of the functional modules of the first embodiment of the panoramic image-based video streaming device of the present invention.

In this embodiment, the panoramic image-based video streaming device includes:

A split image size setting module 10, used to set the split image size according to the effective resolution of the panoramic image;

The starting coordinate setting module 20 is used to set the starting coordinates of each shunt image according to the shunt image size, the preset overlap area size and the preset coordinate position of the shunt image in the panoramic image;

The split image output module 30 is configured to input the split image size, the row pixel value of the panoramic image, and the starting coordinates of each split image to a row stride encoder to output the split image.

Further, the shunt image output module 30 specifically includes:

An image data reading unit, configured to run the row stride encoder according to the starting coordinates of the shunt image and read the image data of the panoramic image;

A valid data determination unit, configured to determine valid data in the image data read by the row stride encoder according to the split image size and the row pixel value of the panoramic image;

A split image obtaining unit is configured to output the valid data through the row stride encoder as split image data to obtain the split image.

Further, the valid data determination unit specifically includes:

The encoding data obtaining subunit is used to sequentially read the panoramic image row pixels through the row stride encoder to obtain the panoramic image encoding data;

An effective row pixel determination subunit, configured to determine the effective row pixels of each row of the shunt image in the corresponding row of the panoramic image according to the starting coordinates of the shunt image and the size of the shunt image;

A valid data determination subunit is configured to determine valid data corresponding to the split image in the panoramic image encoded data according to the valid row pixels.

Further, the panoramic image-based video streaming device further includes an effective resolution determination module, which specifically includes:

An effective area determination unit, configured to screen all pixels of the panoramic image according to a preset distortion threshold, and determine the area in which the distortion amount of the pixel is within the distortion threshold as an effective area;

An effective resolution determination unit is configured to determine the effective resolution of the panoramic image according to the pixel values contained in the effective area.

Furthermore, the offload image size setting module 10 specifically includes:

A row value determination unit configured to determine the row value of the split image resolution according to the effective resolution of the panoramic image;

A column value determination unit, configured to calculate the proportion of the split image in the panoramic image according to the preset split angle, and determine the column value of the split image resolution.

Furthermore, the starting coordinate setting module 20 specifically includes:

A readable area determination unit configured to determine the readable area of the row stride encoder in the panoramic image according to the effective resolution of the panoramic image;

A starting coordinate setting unit, configured to set the starting coordinates of the split image in the panoramic image according to the readable area and the split image size;

An adjacent starting coordinate determining unit is configured to determine the starting coordinates of adjacent split images of the split image according to the preset overlapping image size and the starting coordinates.

Further, the panoramic image-based video streaming device also includes a panoramic image splicing module, and the panoramic image splicing module specifically includes:

A missing image size determination unit configured to determine the missing image size of the edge shunt image according to the shunt image size, the preset overlapping image size and the starting coordinates of the shunt image;

A supplementary image interception unit, configured to intercept a supplementary image of the same size as the missing image, starting from the first column on the other side of the panoramic image according to the missing image size;

An image splicing unit, configured to align and splice the complementary image to the edge position of the panoramic image, and update the panoramic image so that the edge shunt image is read completely.

Among them, each module in the above-mentioned video splitting device based on panoramic images corresponds to each step in the above-mentioned video splitting method embodiment based on panoramic images, and their functions and implementation processes are no longer described one by one here.

In addition, embodiments of the present invention also provide a computer-readable storage medium.

The computer-readable storage medium of the present invention stores a panoramic image-based video streaming program, wherein when the panoramic image-based video streaming program is executed by a processor, the steps of the above panoramic image-based video streaming method are implemented.

For the method implemented when the panoramic image-based video streaming program is executed, reference may be made to various embodiments of the panoramic image-based video streaming method of the present invention, which will not be described again here.

It should be noted that, as used herein, the terms "include", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or system that includes a list of elements not only includes those elements, but It also includes other elements not expressly listed or that are inherent to the process, method, article or system. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

The application may be used in a variety of general or special purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, including Distributed computing environment for any of the above systems or devices, etc. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product that is essentially or contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM) as mentioned above. , magnetic disk, optical disk), including several instructions to cause a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the method described in various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the description and drawings of the present invention may be directly or indirectly applied in other related technical fields. , are all similarly included in the scope of patent protection of the present invention.

Claims

A video streaming method based on panoramic images, which includes:

Screen all pixels of the panoramic image according to the preset distortion threshold, and determine the area where the distortion amount of the pixel is within the distortion threshold as a valid area;

Determine the effective resolution of the panoramic image according to the pixel values contained in the effective area;

Set the split image size according to the effective resolution of the panoramic image;

Determine the readable area of the row stride encoder in the panoramic image according to the effective resolution of the panoramic image;

Set the starting coordinates of the shunt image in the panoramic image according to the readable area and the shunt image size;

Determining the starting coordinates of the adjacent shunt images of the shunt image according to the preset overlapping image size and the starting coordinates;

According to the starting coordinates of the shunt image, run the row stride encoder and read the image data of the panoramic image;

The panoramic image row pixels are sequentially read through the row stride encoder to obtain panoramic image encoded data;

According to the starting coordinates of the split image and the size of the split image, determine the effective row pixels of each row of the split image in the corresponding row of the panoramic image;

According to the effective row pixels, the effective data corresponding to the shunt image is determined in the panoramic image encoded data;

The effective data is output by the row stride encoder as split image data, and the split image is obtained.
The panoramic image-based video splitting method according to claim 1, wherein the step of setting the split image size according to the effective resolution of the panoramic image includes:

Determine the row value of the split image resolution according to the effective resolution of the panoramic image;

According to the preset splitting angle, the proportion of the splitting image in the panoramic image is calculated, and the column value of the splitting image resolution is determined.
The video splitting method based on panoramic images according to claim 1, wherein the split image size, the row pixel value of the panoramic image and the starting coordinates of each split image are input to a row stride encoder, The steps before outputting the shunt image also include:

Determine the missing image size of the edge shunt image according to the shunt image size, the preset overlapping image size and the starting coordinates of the shunt image;

According to the missing image size, starting from the first column on the other side of the panoramic image, intercept a complementary image with the same size as the missing image;

The complementary image is aligned and spliced to the edge position of the panoramic image, and the panoramic image is updated so that the edge shunt image is read completely.
A video streaming method based on panoramic images, which includes:

Set the split image size according to the effective resolution of the panoramic image;

Set the starting coordinates of each shunt image according to the shunt image size, the preset overlap area size and the preset coordinate position of the shunt image in the panoramic image;

The split image size, the row pixel value of the panoramic image, and the starting coordinates of each split image are input to a row stride encoder to output the split image.
The video splitting method based on panoramic images according to claim 4, wherein the split image size, the row pixel value of the panoramic image and the starting coordinates of each split image are input to a row stride encoder, To output the shunt image, include:

According to the starting coordinates of the shunt image, run the row stride encoder and read the image data of the panoramic image;

Determine the valid data in the image data read by the row stride encoder according to the split image size and the row pixel value of the panoramic image;

The effective data is output by the row stride encoder as split image data, and the split image is obtained.
According to the video shunting method based on panoramic images according to claim 5, wherein determining the valid data in the image data read by the row stride encoder according to the shunted image size and the row pixel value of the panoramic image comprises:

The panoramic image row pixels are sequentially read through the row stride encoder to obtain panoramic image encoded data;

According to the starting coordinates of the split image and the size of the split image, determine the effective row pixels of each row of the split image in the corresponding row of the panoramic image;

According to the effective row pixels, effective data corresponding to the split image is determined in the panoramic image encoded data.
The panoramic image-based video splitting method according to claim 4, wherein before setting the split image size according to the effective resolution of the panoramic image, it further includes:

According to a preset distortion threshold, all pixels of the panoramic image are screened, and a region where the distortion amount of the pixels is within the distortion threshold is determined as a valid region;

The effective resolution of the panoramic image is determined according to the pixel values contained in the effective area.
The panoramic image-based video splitting method according to claim 7, wherein the step of setting the splitting image size according to the effective resolution of the panoramic image includes:

Determine the row value of the split image resolution according to the effective resolution of the panoramic image;

According to the preset splitting angle, the proportion of the splitting image in the panoramic image is calculated, and the column value of the splitting image resolution is determined.
The video splitting method based on panoramic images according to claim 8, wherein each split image is set according to the split image size, the preset overlap area size and the preset coordinate position of the split image in the panoramic image. The starting coordinates include:

Determine the readable area of the row stride encoder in the panoramic image according to the effective resolution of the panoramic image;

Set the starting coordinates of the shunt image in the panoramic image according to the readable area and the shunt image size;

The starting coordinates of adjacent split images of the split image are determined according to the preset overlapping image size and the starting coordinates.
The video splitting method based on panoramic images according to claim 4, wherein the split image size, the row pixel value of the panoramic image and the starting coordinates of each split image are input to a row stride encoder, Before outputting the shunt image, also include:

Determine the missing image size of the edge shunt image according to the shunt image size, the preset overlapping image size and the starting coordinates of the shunt image;

According to the missing image size, starting from the first column on the other side of the panoramic image, intercept a complementary image with the same size as the missing image;

The complementary image is aligned and spliced to the edge position of the panoramic image, and the panoramic image is updated so that the edge shunt image is read completely.
A video streaming device based on panoramic images, which includes:

The split image size setting module is used to set the split image size according to the effective resolution of the panoramic image; wherein the effective resolution includes the effective row pixel values and the effective column pixel values of the panoramic image;

A starting coordinate setting module, configured to set the starting coordinates of each shunt image according to the size of the shunt image and the preset coordinate position of the shunt image in the panoramic image;

A split image output module is configured to input the split image size, the row pixel value of the panoramic image, and the starting coordinates of each split image to a row stride encoder to output the split image.
The video streaming device according to claim 11, wherein the streaming image output module specifically includes:

An image data reading unit, configured to run the row stride encoder according to the starting coordinates of the shunt image and read the image data of the panoramic image;

A valid data determination unit, configured to determine valid data in the image data read by the row stride encoder according to the split image size and the row pixel value of the panoramic image;

A split image obtaining unit is configured to output the valid data through the row stride encoder as split image data to obtain the split image.
The video streaming device according to claim 12, wherein the valid data determining unit specifically includes:

The coded data acquisition subunit is used to sequentially read the panoramic image row pixels through the row stride encoder to obtain the panoramic image coded data;

An effective row pixel determination subunit, used to determine effective row pixels of each row of the diversion image in a corresponding row of the panoramic image according to the starting coordinates of the diversion image and the size of the diversion image;

A valid data determination subunit is configured to determine valid data corresponding to the split image in the panoramic image encoded data according to the valid row pixels.
The video streaming device according to claim 11, wherein the panoramic image-based video streaming device further includes an effective resolution determination module, and the effective resolution determination module specifically includes:

An effective area determination unit, configured to screen all pixels of the panoramic image according to a preset distortion threshold, and determine the area in which the distortion amount of the pixel is within the distortion threshold as an effective area;

An effective resolution determination unit is configured to determine the effective resolution of the panoramic image according to the pixel values contained in the effective area.
The video streaming device according to claim 14, wherein the streaming image size setting module specifically includes:

A row value determination unit configured to determine the row value of the split image resolution according to the effective resolution of the panoramic image;

A column value determination unit, configured to calculate the proportion of the split image in the panoramic image according to the preset split angle, and determine the column value of the split image resolution.
The video streaming device according to claim 15, wherein the starting coordinate setting module 20 specifically includes:

a readable area determining unit, configured to determine a readable area of the row stride encoder in the panoramic image according to the effective resolution of the panoramic image;

A starting coordinate setting unit, configured to set the starting coordinates of the split image in the panoramic image according to the readable area and the split image size;

An adjacent starting coordinate determining unit is configured to determine the starting coordinates of adjacent split images of the split image according to the preset overlapping image size and the starting coordinates.
The video streaming device according to claim 11, wherein the panoramic image-based video streaming device further includes a panoramic image splicing module, and the panoramic image splicing module specifically includes:

a missing image size determining unit, configured to determine the missing image size of the edge split image according to the split image size, the preset overlapping image size and the starting coordinates of the split image;

A supplementary image interception unit, configured to intercept a supplementary image of the same size as the missing image, starting from the first column on the other side of the panoramic image according to the missing image size;

An image splicing unit, configured to align and splice the complementary image to the edge position of the panoramic image, and update the panoramic image so that the edge shunt image is read completely.