CN107592549B

CN107592549B - Panoramic video playing and photographing system based on two-way communication

Info

Publication number: CN107592549B
Application number: CN201711061964.7A
Authority: CN
Inventors: 朱磊; 李建英; 韩琦
Original assignee: Harbin Yishe Technology Co ltd
Current assignee: QINGDAO YISPACE TECHNOLOGY Co.,Ltd.
Priority date: 2017-10-27
Filing date: 2017-10-27
Publication date: 2020-06-09
Anticipated expiration: 2037-10-27
Also published as: CN107592549A

Abstract

The utility model provides a panorama video broadcast system of shooing based on two-way communication. The system comprises an image acquisition end, a streaming media server end and one or more user ends; the method comprises the steps that an image acquisition end carries out first-time panoramic stitching on multi-path picture data in real time to form a mapping relation between a panoramic image sequence and the multi-path picture data, and a video stream is pushed to a streaming media server end through a first transmission module to be transmitted to a user for playing; the first transmission module adopts a duplex working mode; when the first transmission module receives a picture request instruction from the streaming media server, corresponding multi-path picture data are sent to the streaming media server, so that the pictures required by the user are obtained through secondary panoramic stitching and sent to the user. The panoramic video playing and photographing system based on the two-way communication has the advantages of small occupied bandwidth and high transmission and playing speed, and can quickly provide high-resolution pictures for users when the users need to take pictures.

Description

Panoramic video playing and photographing system based on two-way communication

Technical Field

The present disclosure relates to communication technologies, and in particular, to a panoramic video playing and photographing system based on bidirectional communication.

Background

In the current era, video playing is more and more popular with the public, the development history of video playing is evolved on the basis of video on demand, and the video is played by the internet and a streaming media technology, so that the video content can be comprehensively transmitted in real time. In early video playing, the visible area of the video content is due to shooting and other reasons, such as: the fixed and unadjustable viewing angle and range of the camera lens can cause that a user can only watch the current area, the video picture can only change along with the movement of the lens, and the user can not watch stereoscopic impression and can not feel personally on the scene.

With the development of the panoramic video technology, a 360-degree panoramic video appears, so that a user can freely watch the panoramic video in 360 degrees in an all-around manner, and the panoramic video can be watched from top to bottom, from left to right, from front to back, and even at a far and near distance. The method is generally realized by shooting objects in the same space at multiple angles by a plurality of cameras at the same time, and a user is allowed to freely switch viewing angles at a playing end under the condition of not pausing video playing; the panoramic video has stereoscopic impression and sense of reality, and can provide visual experience well; the experience of being personally on the scene and the surprise effect of being more striking and seeing can be realized.

With the continuous improvement of the demand of people, people hope to see not only real, comprehensive and visual videos but also obtain high-quality images in real time for commemoration or sharing with friends when watching panoramic videos. However, the transmission speed of the image data is limited by the transmission bandwidth, and the transmission stability is also affected by the transmission bandwidth, i.e. the transmission bandwidth affects the smoothness of video playing and the immediacy of image data acquisition.

Disclosure of Invention

The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. It should be understood that this summary is not an exhaustive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure or to delineate the scope of the disclosure. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.

In view of the foregoing defects of the prior art, it is an object of the present disclosure to provide a panoramic video playing and photographing system based on two-way communication, so as to solve at least the problem of how to quickly obtain high resolution pictures in video playing.

According to one aspect of the present disclosure, a panoramic video playing and photographing system based on two-way communication is provided, the system includes an image acquisition end, a streaming media server end and one or more user ends; the image acquisition end comprises a camera module, a multimedia processing module and a first transmission module; the camera module is suitable for acquiring multi-path picture data in the visual field of the camera module; the multimedia processing module is adapted to: performing panoramic stitching on multiple paths of picture data acquired by the camera module in real time to form a panoramic image sequence, forming a mapping relation between the multiple paths of picture data and the panoramic image sequence according to a time corresponding relation, storing the mapping relation and at least part of the multiple paths of picture data, performing compression coding on the panoramic image sequence to obtain a video stream after compression coding, and pushing the video stream after compression coding to a streaming media server end through the first transmission module, wherein the first transmission module adopts a duplex working mode, and when a request picture instruction from the streaming media server end is received through the first transmission module, searching the multiple paths of picture data corresponding to the searched object information in the stored at least part of the multiple paths of picture data according to the mapping relation between the multiple paths of picture data and the panoramic image sequence based on the searched object information contained in the request picture instruction, sending the result of the search to the streaming media server through the first transmission module; the search object information comprises related information of a panoramic image frame to be searched; the streaming media server is suitable for receiving the compressed and coded video stream from the image acquisition end in real time and sending the video stream to the corresponding user end, after receiving a search result from the image acquisition end, performing secondary panoramic stitching on the multi-path picture data in the search result to obtain a secondary panoramic stitched image, and sending a local or whole image of the secondary panoramic stitched image to the corresponding user end as an output result.

The panoramic video playing and photographing system based on bidirectional communication comprises an image acquisition end, a streaming media server end and one or more user ends; the image acquisition end comprises a camera module, a multimedia processing module and a first transmission module. The multimedia processing module carries out panoramic stitching on the multi-path picture data collected by the camera module in real time for the first time to form a panoramic image sequence, a mapping relation between the multi-path picture data and the panoramic image sequence is formed, and the video stream is pushed to a streaming media server end through the first transmission module; when receiving a picture request instruction from a streaming media server, sending corresponding multi-path picture data to the streaming media server; and the streaming media server receives the compressed and coded video stream in real time and sends the video stream to the user side, and when the user requests to take a picture, secondary panoramic stitching is carried out on the multi-path picture data acquired from the image acquisition side so as to obtain a picture required by the user. Therefore, through the bidirectional communication between the image acquisition end and the streaming media server end, a large number of high-resolution pictures do not need to be stored in the streaming media server end, and the system burden is reduced.

Further, on one hand, a fast stitching algorithm may be used to perform a first panorama stitching on the multiple paths of picture data collected by the camera module 11 to form a panoramic image sequence, and the compressed video stream occupies a smaller memory through compression coding and can be quickly transmitted to the server side with a smaller bandwidth (i.e., the "compressed video stream of the panoramic image sequence formed through the fast stitching algorithm" transmitted from the collection side to the server side occupies a smaller bandwidth and has a faster transmission speed), so that the server side can quickly provide the compressed video stream to the user for playing with a smaller bandwidth (i.e., the "compressed video stream of the panoramic image sequence formed through the fast stitching algorithm" transmitted from the server side to the user side occupies a smaller bandwidth and has a faster transmission speed).

In other words, in the video playing mode, the panoramic image sequence is obtained through the fast splicing algorithm and is transmitted to the server after being compressed, and then the panoramic image sequence is transmitted to the client by the server, so that the occupied bandwidth is smaller, the transmission speed is higher, the transmission burden of the server is smaller, the playing of the user is smoother, and the viewing experience is better.

On the other hand, under the condition that the (first transmission module of the) acquisition end adopts the duplex working module, the acquisition end stores the multi-channel picture data, so that when the user needs to take a picture or self-shoot, the user sends a picture taking request to the server end, and then the server end sends the picture request to the acquisition end, so that the acquisition end searches the multi-channel picture data corresponding to the frame which the user requests to take the picture or self-shoot as a searching result and sends the searching result to the server end; and performing secondary panorama stitching on the multi-path picture data in the search result by using an accurate stitching algorithm at the server side to obtain a secondary panorama stitched image, and obtaining the part of the image which is required to be photographed or self-photographed by the user by using the secondary panorama stitched image (for example, corresponding clipping can be performed according to the posture sent by the user side), so as to fulfill the aim of realizing high-definition photographing or high-definition self-photographing by the user.

In other words, in the "user request photographing mode", the multi-path picture data of the corresponding frame is searched from the multi-path picture data stored in the acquisition end and returned to the server end, and then the server end obtains the secondary panorama mosaic image by using the precise mosaic algorithm, and obtains (for example, cuts) the picture required by the user to send to the user, so that the user can obtain the high-definition picture (higher than the resolution of the user in the video to be watched), without affecting the transmission speed between the acquisition end and the server end and between the server end and the user end, and the playing speed of the user end is higher.

In summary, by adopting the above-mentioned manner that the compressed video stream obtained by obtaining the panoramic image sequence by using the fast stitching algorithm in the "video playing mode" is transmitted to the user, and the multi-path picture data of the corresponding frame is obtained from the acquisition end in the "user request photographing mode" and the high-definition panoramic image sequence (i.e. the secondary panoramic stitched image) obtained by using the precise stitching algorithm is transmitted to the user, the high-definition photos can be provided to the user under the conditions of small bandwidth occupation and high transmission speed; meanwhile, due to the adoption of a duplex working mode, a large amount of high-definition picture data does not need to be stored at the server end, the burden of the server is reduced, and meanwhile, the transmission bandwidth between the acquisition end and the server end is further reduced.

The panoramic video playing and photographing system based on bidirectional communication according to the embodiment of the present disclosure can further have at least one of the following beneficial effects:

1. the user can use the system to take pictures at a unique visual angle to obtain a high-resolution picture;

2. multiple users can simultaneously share the system to watch video playing and take pictures in real time;

3. by carrying the multimedia processing module and the two-way communication module on the image acquisition end, the image acquisition end only needs to transmit the compressed video stream in real time and transmit the required original picture data to the streaming media server end after receiving the request instruction, and does not need to transmit all the original picture data in real time, so that the operation pressure of the streaming media server end is reduced, the transmission bandwidth of the image data is effectively saved, and the transmission efficiency of the image data is improved.

For example, an image is captured by the camera module and all original picture data are directly transmitted to the streaming media server through the transmission module, the streaming media server realizes video playing and photographing functions, and if the frame rate is 25, the size of each picture in a single path is about 5M, and the total number of pictures is 8, the bandwidth required by the transmission module per second is 5 ＊ 8 ＊ 25-1000M.

By mounting the multimedia processing module and the bidirectional communication module at the image acquisition end, the image acquisition end only needs to transmit the compressed video stream in real time and transmit the required original picture data to the streaming media server end after receiving the request instruction, if the frame rate is 25, the size of each picture in a single path is about 5M, and the total number of the pictures is 8, taking the video playing time 1min as an example, the required bandwidth of the transmission module per second is about 47M (video stream obtained by experiment) +5 ＊ 8 (required original picture) > 87M.

These and other advantages of the present disclosure will become more apparent from the following detailed description of the preferred embodiments of the present disclosure when taken in conjunction with the accompanying drawings.

Drawings

The disclosure may be better understood by reference to the following description taken in conjunction with the accompanying drawings, in which like or similar reference numerals identify like or similar parts throughout the figures. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate preferred embodiments of the present disclosure and, together with the detailed description, serve to explain the principles and advantages of the disclosure. Wherein:

fig. 1 is a schematic diagram illustrating an example structure of a panoramic video playback photographing system based on bidirectional communication according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram showing a possible configuration of the image capturing end of FIG. 1;

fig. 3 is a schematic diagram showing a possible structure of the streaming server side in fig. 1;

fig. 4 is a schematic diagram illustrating a possible structure of a corresponding device at the user end in fig. 1;

fig. 5 is a schematic diagram illustrating another example structure of a panoramic video playback photographing system based on bidirectional communication according to an embodiment of the present disclosure.

Skilled artisans appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help improve understanding of embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. In the interest of clarity and conciseness, not all features of an actual implementation are described in the specification. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.

Here, it should be further noted that, in order to avoid obscuring the present disclosure with unnecessary details, only device structures and/or processing steps closely related to the scheme according to the present disclosure are shown in the drawings, and other details not so relevant to the present disclosure are omitted.

The embodiment of the disclosure provides a panoramic video playing and photographing system based on bidirectional communication, which comprises an image acquisition end, a streaming media server end and one or more user ends; the image acquisition end comprises a camera module, a multimedia processing module and a first transmission module; the camera module is suitable for acquiring multi-path picture data in the visual field of the camera module; the multimedia processing module is adapted to: performing panoramic stitching on multiple paths of picture data acquired by the camera module in real time to form a panoramic image sequence, forming a mapping relation between the multiple paths of picture data and the panoramic image sequence according to a time corresponding relation, storing the mapping relation and at least part of the multiple paths of picture data, performing compression coding on the panoramic image sequence to obtain a video stream after compression coding, and pushing the video stream after compression coding to a streaming media server end through the first transmission module, wherein the first transmission module adopts a duplex working mode, and when a request picture instruction from the streaming media server end is received through the first transmission module, searching multiple paths of picture data corresponding to the search object information in the stored at least part of the multiple paths of picture data according to the mapping relation between the multiple paths of picture data and the panoramic image sequence based on the search object information contained in the request picture instruction, sending the result of the search to the streaming media server through the first transmission module; the search object information comprises related information of a panoramic image frame to be searched; the streaming media server is suitable for receiving the compressed and coded video stream from the image acquisition end in real time and sending the video stream to the corresponding user end, after receiving a search result from the image acquisition end, performing secondary panoramic stitching on the multi-path picture data in the search result to obtain a secondary panoramic stitched image, and sending a local or whole image of the secondary panoramic stitched image to the corresponding user end as an output result.

The panoramic video playing and photographing system based on bidirectional communication can be used for on-demand (recording or playback and the like) or live broadcasting of panoramic videos.

Fig. 1 schematically shows an example structure of a panoramic video playing and photographing system based on bidirectional communication according to an embodiment of the present disclosure.

As shown in fig. 1, a panoramic video playing and photographing system based on bidirectional communication according to an embodiment of the present disclosure includes an image capturing end 1, a streaming server end 2, and one or more user ends 3, where the image capturing end 1 includes a camera module 11, a multimedia processing module 12, and a first transmission module 13.

The camera module 11 is adapted to acquire multiple paths of picture data within its field of view.

According to an embodiment of the present disclosure, the camera module 11 may include, for example, a plurality of camera modules, and the plurality of camera modules are adapted to acquire corresponding images of the plurality of viewing angles as multi-path image data. Wherein, a plurality of camera modules for example can include at least one in fisheye camera and wide-angle camera.

In one example, the plurality of camera modules includes a plurality of fisheye cameras, for example, 2 (or other numbers of) fisheye cameras may be used to acquire two-way picture data covering a 360-degree view angle range (as an example of multi-way picture data).

In another example, the plurality of camera modules includes, for example, a plurality of wide-angle cameras, for example, 8 (or other numbers of) wide-angle cameras may be used to acquire 8-way picture data covering a 360-degree view angle range (as an example of multi-way picture data).

The multimedia processing module 12 is adapted to perform the following processes: performing panoramic stitching on the multi-path picture data acquired by the camera module 11 in real time for the first time to form a panoramic image sequence, forming a mapping relation between the multi-path picture data and the panoramic image sequence according to the time correspondence, and storing the mapping relation and at least part of the multi-path picture data; compressing and coding the panoramic image sequence to obtain a compressed and coded video stream, and pushing the compressed and coded video stream to a streaming media server end through a transmission module 13; when a request picture instruction from the streaming media server 2 is received through the first transmission module 13, based on the search object information contained in the request picture instruction, according to the mapping relationship between the multi-path picture data and the panoramic image sequence, searching the multi-path picture data corresponding to the search object information in at least part of the stored multi-path picture data, and sending the multi-path picture data to the streaming media server 2 through the first transmission module 13 as a search result; the search object information comprises related information of the panoramic image frame to be searched. The multimedia processing module 12 may be implemented, for example, using any of the following: developing a board; a CPU; a microprocessor; a microcomputer; and so on.

The first transmission module adopts a duplex working mode, that is, the first transmission module is not only used for sending data to the streaming media server 2 (such as the second transmission module of the streaming media server 2), but also used for receiving data from the streaming media server 2.

When receiving a photographing request instruction (i.e., a photographing request described below) from the user end 3, the streaming media server end 2 sends a corresponding image request instruction to the acquisition end 1, where the image request instruction includes "object finding information" described above, and "the information related to the panoramic image frame to be found" in the object finding information "may be, for example, a watermark included in the photographing request instruction from the user end 3, for example, information obtained by decoding a watermark (e.g., a one-dimensional code or a two-dimensional code watermark) (e.g., a watermark generation parameter Iri mentioned below) is used as the information related to the panoramic image frame to be found.

According to an embodiment of the present disclosure, at least a portion of the multi-path picture data saved by the multimedia processing module 12 may be multi-path picture data saved within a last preset time period, for example, multi-path picture data within a last 1 minute (or 3 minutes, 10 minutes, 1 hour, 1 day, etc.). The preset time period may be set according to an empirical value, for example.

According to an embodiment of the present disclosure, the panoramic image sequence includes, for example, a plurality of frame sequences, Ir represents an r-th frame sequence of the plurality of frame sequences, r is a positive integer, Irn represents an n-th frame image of the r-th frame sequence, and n is also a positive integer. n may for example be equal to the frame rate, i.e. the number of frames per second, such as 25.

The multi-path picture data may be expressed as, for example, Drn ═ D₁，D₂......，D_mM represents the number of camera modules, which is a positive integer, Drn represents the multi-path picture data corresponding to the nth frame image in the r frame sequence, and the mapping relationship between the multi-path picture data and the panoramic image sequence can be represented as Irn → Drn.

The streaming media server 2 is adapted to receive the compressed and encoded video stream from the image acquisition end 1 in real time and send the video stream to a corresponding user side, perform secondary panorama stitching on the multiple paths of picture data in the search result after receiving the search result from the image acquisition end 1 to obtain a secondary panorama stitched image, and send a local or whole image of the secondary panorama stitched image as an output result to the corresponding user side.

Fig. 2 shows a schematic diagram of a possible configuration of the image acquisition terminal 1 in fig. 1.

As shown in fig. 2, the multimedia processing module 12 in the image capturing end 1 may include any one or more of a picture input sub-module 121, a picture stitching sub-module 122, a picture caching sub-module 123, a picture output sub-module 124, a video encoding sub-module 125, and a video output sub-module 126. The processing and functions performed by the above respective sub-modules will be described below by way of example, respectively.

It should be noted that, in fig. 2, each of the sub-modules 121 to 126 is represented by a dashed box, which is intended to indicate that these sub-modules are not indispensable, but are individually selectable.

According to the embodiment of the present disclosure, the multimedia processing module 12 includes, for example, a picture stitching sub-module 122, and the picture stitching sub-module 122 may be adapted to perform a first panorama stitching on the multiple paths of picture data acquired by the camera module 11 by using a fast stitching algorithm to form a panorama image sequence. The fast splicing algorithm may use, for example, an existing fast splicing method, which is not described in detail herein.

According to one implementation, the sequence of panoramic images formed by the picture stitching sub-module 122 is, for example, a sequence of tagged panoramic images. Each frame of panoramic image in the panoramic image sequence is, for example, provided with a mark corresponding to the frame, where the mark may be, for example, a watermark, such as a one-dimensional code watermark or a two-dimensional code watermark.

Wherein the marked panoramic image sequence may be obtained, for example, by a video encoding sub-module 125. According to one implementation, the video encoding sub-module 125 may obtain the sequence of tagged panoramic images, for example, as follows:

setting a frame rate f, dividing the video stream into a plurality of frame sequences in the order of playing time, where f is a positive integer (e.g., f may be 25), Ir represents an r-th frame sequence, r is a positive integer, r is 1,2, such as I1 represents a first frame sequence, I2 represents a second frame sequence, and so on. Ir1, Ir2,...... Irf denotes f frame images in the r frame sequence, and "Iri" is used as a watermark generation parameter of the i frame image in the r frame sequence, wherein i ∈ {1, 2...., f }, a corresponding mark is obtained according to the watermark generation parameter Iri (for example, a one-dimensional code or two-dimensional code watermark is generated according to Iri), and the mark is added to the corresponding image frame (i.e., the i frame image in the r frame sequence), so that marked frame images are obtained, and a marked panoramic image sequence is formed.

In addition, the picture input sub-module 121 may receive multiple paths of picture data collected by the camera module in real time, and transmit the multiple paths of picture data to the picture stitching sub-module 122 and the picture caching sub-module 123 at the same time.

The picture buffer sub-module 123 may store multiple paths of picture data within a certain time (for example, within a latest preset time period), and store a mapping relationship between the multiple paths of picture data formed according to the time correspondence and a panoramic image sequence (for example, a marked panoramic image sequence).

When receiving the picture request command from the streaming media server 2 through the first transmission module 13, the picture output sub-module 124 searches the multi-path picture data Drn corresponding to the current frame in the picture cache sub-module 123 according to the mapping relationship Irn → Drn between the multi-path picture data and the panoramic image sequence (e.g., the panoramic image sequence with the mark), and outputs the multi-path picture data Drn to the streaming media server 2 through the first transmission module 13.

The video encoding sub-module 125 may compressively encode a panoramic image sequence (e.g., a tagged panoramic image sequence).

The video output sub-module 126 may push the compressed and encoded video stream to the streaming server 2 through the first transmission module 13 by using a video transmission protocol.

Fig. 3 shows a schematic diagram of a possible structure of the streaming server side 2 in fig. 1.

As shown in fig. 3, the streaming server 2 may include any one or more of a video processing module 21, a virtual photographing module 22, a user management module 23, and a second transmission module 24, for example. The processes and functions performed by the above respective modules will be described below by way of example, respectively.

It should be noted that, in fig. 3, each of the modules 21 to 24 is represented by a dashed box, and it is intended to indicate that these modules are not indispensable, but are individually selectable.

According to one implementation, the streaming media server 2 includes, for example, a video processing module 21, and the video processing module 21 may be adapted to receive, in real time, the compression-coded video stream transmitted by the image capturing end 1 through a second transmission module 24, and obtain, based on the compression-coded video stream, a video stream to be delivered to the user end.

In one example, the video processing module 21 may directly transmit the compression-coded video stream to the user end as the video stream to be transmitted.

In another example, the video processing module 21 may also issue the video stream to be issued to the user terminal according to a predetermined manner, for example, the video processing module 21 may re-compress and encode the compressed and encoded video stream into a plurality of video streams, so as to send at least part of the plurality of video streams to the corresponding user terminals, where resolutions of the plurality of video streams are different from each other. The "at least part of the plurality of video streams" may be, for example, a part of the video streams (such as one or several video streams), or may be all of the video streams.

For example, the video processing module 21 may re-compression encode the compression-encoded video stream into N (N is a positive integer, N may be 1 or greater than 1, such as 3 or 5, etc.) resolution (or definition) video streams, such as ultra-fast, standard definition, normal definition, high definition, VIP super definition, and so on. According to the user level (for example, the user is a common user), one or more selectable resolution video streams can be provided for the user, and the video stream with the corresponding resolution is transmitted to the user according to the selection of the user; if the user is a VIP user, when the user selects a VIP super-definition video stream, the video stream of that definition (i.e., resolution) is transmitted thereto.

According to one implementation, the streaming server 2 includes, for example, a virtual camera module 22, and the virtual camera module 22 may be adapted to receive a camera request instruction from the user side and execute corresponding processing.

In addition, the virtual photographing module 22 may also be adapted to perform secondary panorama stitching on the multiple paths of picture data in the search result by using an accurate stitching algorithm, so as to obtain a secondary panorama stitched image. The precise stitching algorithm may adopt, for example, an existing precise stitching algorithm, such as an optical flow stitching algorithm.

The virtual photographing module 22 may perform, for example, the following processing: when at least one photographing request instruction is received from one or more user terminals, the steps a 1-a 3 described below are respectively executed for each photographing request instruction.

In step a1, the related information of the panoramic image frame to be searched is determined according to the video frame data contained in the request photographing instruction, and a request picture instruction containing the related information of the panoramic image frame to be searched is generated, and then step a2 is executed.

In step a2, the request picture command is sent to the image capturing end 1 through the second transmission module 24, and the multiple paths of picture data sent back by the image capturing end 1 in response to the request picture command are received from the image capturing end 1, and the multiple paths of picture data are subjected to secondary panorama stitching by using a precise stitching algorithm, and then step a3 is executed.

In step a3, a picture corresponding to the sending gesture is cut out from the obtained secondary panoramic stitched image according to the sending gesture contained in the shooting request command, and is sent to the corresponding user end as an output result through the second transmission module 24.

According to one implementation, the virtual camera module 22 may obtain the relevant information of the corresponding panoramic image frame to be searched, for example, by decoding a watermark contained in the video frame data.

The relevant information of the panoramic image frame to be searched is represented as Iri, wherein i is a positive integer from 1 to m.

The watermark is, for example, a one-dimensional code watermark or a two-dimensional code watermark.

The virtual photographing module 22 may store the obtained output result in the corresponding user account of the user management module 23, for example, for the user to preview and/or pay for downloading.

According to one implementation, the virtual photographing module 22 may, for example, remap the secondary panoramic stitched image onto a spherical surface, where the spherical surface is used as a panoramic mapping spherical surface, determine an angle range of a user field of view according to a sending gesture of a device at a user end, determine a spherical image to be clipped on the panoramic mapping spherical surface according to the angle range of the user field of view, clip the spherical image to be clipped, and map the spherical image to be clipped into a target rectangular image, where a size of the target rectangular image is a preset value. The preset value may be set based on an empirical value or may be set based on a screen size parameter included in a photographing instruction transmitted from the user equipment.

According to one implementation, the virtual photographing module 22 may generate a corresponding panoramic mapping spherical surface by taking the side of the planar rectangular picture corresponding to the secondary panoramic mosaic image as a spherical semicircular arc, and map the planar rectangular picture onto the panoramic mapping spherical surface, so as to complete mapping the secondary panoramic mosaic image onto the spherical surface. The side of the planar rectangular picture is, for example, the shorter side of the planar rectangular picture.

According to one implementation, the streaming media server 2 includes, for example, a user management module 23, and the user management module 23 may be adapted to receive an identity authentication request from each user terminal, perform identity authentication on corresponding user equipment based on the identity authentication request, and send authentication success information to the corresponding user equipment after the authentication is successful so as to establish a data connection with the user equipment.

According to one implementation, the streaming media server 2 includes, for example, a second transmission module 24, and the second transmission module 24 may be configured to send the whole image of the secondary panorama stitching image to a corresponding user side.

According to another implementation manner, the virtual photographing module 22 in the streaming server 2 may also determine a sending pose of the user end (the sending pose is received from the user end, for example) first, so as to cut out a local image corresponding to the pose in the secondary panoramic stitched image according to the sending pose of the user end, and send the local image as an output result to the user end.

According to embodiments of the present disclosure, the first transmission module 13 may, for example, include one or more transmission components. The first transmission module 13 may employ at least one of wired transmission and wireless transmission, for example.

According to embodiments of the present disclosure, the second transmission module 24 may include, for example, one or more transmission components. The second transmission module 24 may adopt at least one of wired transmission and wireless transmission, for example.

According to embodiments of the present disclosure, the third transmission module 34 may include, for example, one or more transmission components. The third transmission module 34 may adopt at least one of wired transmission and wireless transmission, for example.

Fig. 4 shows a possible structure of a client device corresponding to the client 3. As shown in fig. 4, each customer premises device may comprise, for example, any one or more of a user access module 31, a display module 32, a user interaction module 33 and a third transmission module 34. The processes and functions performed by the above respective modules will be described below by way of example, respectively. In fig. 4, each of the modules 31 to 34 is indicated by a dashed box, which is intended to indicate that these modules are not optional but are individually optional.

Each user end 3 may be provided with a user interaction module 33 for sending various requests, for example, and the third transmission module 34 in the user end 3 may be used for implementing data transmission and the like with the second transmission module 24 of the streaming server end 2, for example.

According to one implementation, each user terminal 3 may send an identity authentication request to the streaming media server terminal 2 in the panoramic video playing and photographing system, for example, and establish a data connection with the streaming media server terminal 2 after receiving the authentication success information from the streaming media server terminal 2. For example, after the authentication is successful, a connection with the streaming server 2 may be established through the user access module 31.

According to one implementation, each user terminal 3 may, for example, send a video playing request to the streaming media server terminal 2 after establishing a data connection with the streaming media server terminal 2, and decode video data distributed from the streaming media server terminal 2 in real time.

According to one implementation, each user terminal 3 may select one of one or more resolution viewing modes to view the panoramic video playback, for example, according to user input.

According to one implementation, each user terminal 3 may send a photographing request according to a user input during the playing of the panoramic video, for example, to obtain a high resolution picture. The high resolution picture is, for example, a picture with a resolution higher than or equal to (for example, 4K) a preset threshold, and the preset threshold may be set according to an empirical value or determined by an experimental method, which is not described herein again.

For example, the resolution of the images in the first panoramic image sequence obtained by first panoramic stitching is 2880 ＊ 1440, and the resolution of the secondary panoramic stitched image obtained by secondary panoramic stitching is 4000 ＊ 3000. it should be understood that, in practical applications, the resolution of the images in the first panoramic image sequence is not limited to 2880 ＊ 1440, and the resolution of the secondary panoramic stitched image is not limited to 4000 ＊ 3000, and the above resolutions can be changed and set to different values according to practical situations and needs, but the resolution of the secondary panoramic stitched image is required to be higher than the resolution of the images in the first panoramic image sequence.

According to one implementation, each client 3 may send a preview request to the streaming server 2 according to a user input, for example, and when the client 3 is allowed to preview to the streaming server 2, preview a corresponding photo.

According to one implementation, each user terminal 3 may send a payment download request to the streaming server 2 according to a user input, for example, and download the corresponding photo to the local when the user terminal 3 is allowed to download to the streaming server 2.

The photographing request sent by the user end 3 includes, for example, video frame data and a sending gesture of a device corresponding to the user end 3. The video frame data contains, for example, a watermark, such as a one-dimensional code or a two-dimensional code watermark.

For example, the user end 3 may obtain the normal direction of the screen of the display module 32 in the device corresponding to the user end 3 or the normal direction of the back of the screen thereof as the sending posture when determining to send the photographing request.

For another example, the user end 3 may set the sending gesture by detecting a touch operation of the corresponding device (for example, the display module 32 may be a touch screen, and detect a touch operation on the touch screen), or obtain the sending gesture by using a gesture sensor (for example, a gyroscope, etc.) in the corresponding device.

PREFERRED EMBODIMENTS

A preferred embodiment of the panoramic video playback photographing system based on two-way communication of the present disclosure is described below with reference to fig. 5. It should be noted that the image capturing end and the streaming media server end in the figure are also applicable to the panoramic video playing and photographing system based on bidirectional communication described above, and are not described herein again.

As shown in fig. 5, the image capturing end includes a camera module, a multimedia processing module, and a first transmission module. The camera module comprises a camera module consisting of a plurality of fisheyes or wide-angle cameras; the multimedia processing module comprises an embedded development board or a microcomputer and comprises a picture input sub-module, a picture splicing sub-module, a picture caching sub-module, a picture output sub-module, a video coding sub-module and a video output sub-module; the first transmission module is in real-time two-way communication with the streaming media server, and comprises one or more transmission modules, and wired transmission or wireless transmission can be adopted. The image acquisition end pushes the video stream and sends the picture data through the first transmission module, and meanwhile receives a picture request instruction of the streaming media server end.

The streaming media server comprises a video processing module, a virtual photographing module, a user management module and a second transmission module.

The user terminal (i.e. the user terminal) comprises one or more user equipments, each of which comprises a user access module, a user interaction module, a display module and a third transmission module. The user equipment can be terminal equipment such as a smart phone, a tablet computer or a head-mounted VR system.

The image acquisition end work flow comprises the following steps:

the method comprises the following steps: the camera module receives the acquisition instruction through the first transmission module, acquires multi-path picture data in the visual field of the camera module, and stops acquisition when receiving the acquisition stopping instruction. The camera module comprises a camera module consisting of a plurality of fisheyes or wide-angle cameras, and multi-path picture data are multi-view pictures collected by the camera module consisting of the fisheyes or the wide-angle cameras, wherein the multi-path picture data can be represented by D (D1, D2 … … and Dm), wherein m represents the number of the camera modules and is a positive integer.

Step two: and the picture input submodule in the multimedia processing module receives the multi-path picture data acquired by the camera module in real time and simultaneously transmits the multi-path picture data to the picture splicing submodule and the picture caching submodule.

Step three: and the picture splicing submodule in the multimedia processing module splices multiple paths of picture data in real time by using a quick splicing algorithm to complete panorama splicing and form a marked panorama image sequence, and then transmits the marked panorama image sequence to the video coding submodule.

The sequence of marked panoramic images may be formed by: if the frame rate is set to be 25, the video stream takes 25 frames as a frame sequence, the video is divided into a plurality of frame sequences according to the playing time sequence, the r-th frame sequence is represented by Ir, and the 25-frame images in the r-th frame sequence are represented by Ir1 and Ir 2.

Step four: and a video coding sub-module in the multimedia processing module compresses and codes the marked panoramic image sequence.

Step five: and the video output submodule in the multimedia processing module utilizes a video transmission protocol to push the compressed and coded video stream to a streaming media server side through the first transmission module.

Step six: and the picture cache submodule in the multimedia processing module stores multi-path picture data within a plurality of times and stores the mapping relation between the multi-path picture data formed according to the time corresponding relation and the marked panoramic image sequence.

For example, if the frame rate is set to 25, the sequence of marked panoramic images is denoted by Irn, and the multi-path picture data is denoted by Drn ═ { D1, D2 … …, Dm }, where m denotes the number of imaging modules and is a positive integer, and Drn denotes the multi-path picture data corresponding to the nth frame image in the r-th frame sequence, the mapping relationship between the multi-path picture data and the sequence of marked panoramic images can be denoted by Irn → Drn.

Step seven: when the picture output submodule receives a picture request instruction of the streaming media server through the first transmission module, the picture output submodule searches the multi-path picture data Drn corresponding to the current frame in the picture cache submodule according to the mapping relation Irn → Drn between the multi-path picture data and the marked panoramic image sequence. And outputs multiple paths of picture data Drn to the streaming media server through the first transmission module.

The user management module is used for receiving the identity authentication request from each user device, authenticating the identity of the corresponding user device based on the identity authentication request, and sending authentication success information to the corresponding user device after the authentication is successful so as to establish data connection with the user device. The video processing module receives the video stream transmitted by the image acquisition end in real time through the second transmission module and distributes the video stream according to a certain rule; or the video stream can be recompressed and coded into a plurality of video streams with different resolutions, and the video streams are distributed according to a certain rule. The virtual photographing module is used for processing a command of the user terminal for requesting photographing.

The step of the streaming media server terminal processing the user terminal request to take a picture:

when the streaming media server receives at least one instruction of requesting to take a picture from one or more user equipments of the user terminal, (step 101) firstly, a current frame is searched according to video frame data contained in the instruction of requesting to take a picture sent by the user equipment, and the method comprises the following steps: the video frame data comprises a one-dimensional code or a two-dimensional code watermark, and the decoded one-dimensional code or two-dimensional code searches the current frame according to Irn; (step 102) returning the current frame Irn to the image acquisition end through the second transmission module and requesting a plurality of paths of pictures Drn; (step 103) receiving the multiple paths of pictures Drn transmitted by the image acquisition end, and then finishing panorama stitching by using a second accurate stitching algorithm; (step 104) cutting a picture corresponding to the gesture in the panoramic picture according to the sending gesture of the user terminal; (step 105) finally saving the cut picture to the user account number in the user management module for final preview and payment downloading of the user terminal.

The clipping step in step 104 is as follows:

(step 201) the panoramic picture (the plane rectangular picture) is mapped to a spherical surface again, namely the side edge of the plane rectangular picture is taken as a spherical surface semi-circular arc to generate a corresponding panoramic mapping spherical surface, and the plane rectangular picture is mapped to the panoramic mapping spherical surface; (step 202) determining an angle range of the user's field of view according to the transmission gesture of the user device; (step 203) searching spherical graphics to be cut on the panoramic mapping spherical surface according to the angle range of the user visual field; (step 204) mapping the clipped spherical figure into a target rectangular picture, wherein the size of the target rectangular picture is a preset value, and the preset value can be set according to an empirical value or can be set according to a screen size parameter contained in a photographing instruction sent by user equipment.

It should be noted that the second transmission module may be one transmission module, may also be multiple transmission modules, and may also adopt wired transmission or wireless transmission. The first quick splicing algorithm and the second accurate splicing algorithm are different in that the algorithm of the first quick splicing algorithm needs to finish splicing panoramic pictures in real time quickly so as to meet the smoothness of video playing; and the algorithm of the second accurate splicing algorithm mainly focuses on accurate seamless high-quality panoramic picture splicing so as to meet the requirement that a user terminal acquires a high-resolution picture in the process of watching a video.

User terminal workflow:

the method comprises the following steps: the user terminal sends an identity authentication request to the streaming media server terminal, and establishes data connection with the streaming media server terminal after receiving authentication success information from the streaming media server terminal.

Step two: and the user terminal is accessed into the system and then sends a video watching request to the streaming media server, decodes the video data distributed by the streaming media server in real time and selects watching modes with different resolutions to watch the panoramic video.

Step three: the user terminal can send a photographing request in the process of watching the panoramic video so as to obtain a high-resolution picture.

Step four: the user terminal can send a preview request to preview the photo taken by the user terminal; or sending a payment downloading request to download the photo taken by the user to the local.

Wherein, the user terminal sends the request through the user interaction module. The shooting request sent by the user terminal equipment comprises video frame data and a sending gesture, wherein the sending gesture refers to the gesture of the user equipment when sending the request, and can be the normal direction of a screen of a display module in the user equipment or the normal direction of the back of the screen of the display module; the sending gesture can be set by a user through a touch screen operation mode, and gesture sensing sub-modules such as a gyroscope and the like can be integrated in the user interaction module, so that the current gesture of the user equipment can be acquired.

In summary, in the embodiments according to the present disclosure, the present disclosure provides the following solutions, but is not limited thereto:

the panoramic video playing and photographing system based on the bidirectional communication is characterized by comprising an image acquisition end 1, a streaming media server end 2 and one or more user ends 3; the image acquisition terminal 1 comprises a camera module 11, a multimedia processing module 12 and a first transmission module 13;

the camera module 11 is suitable for acquiring multi-path picture data in the field of view;

the multimedia processing module 12 is adapted to:

performing panoramic stitching on the multi-path picture data acquired by the camera module 11 in real time for the first time to form a panoramic image sequence, forming a mapping relation between the multi-path picture data and the panoramic image sequence according to the time corresponding relation, storing the mapping relation and at least part of the multi-path picture data,

performing compression coding on the panoramic image sequence to obtain a compression-coded video stream, and pushing the compression-coded video stream to a streaming media server end through the first transmission module 13, wherein the first transmission module adopts a duplex working mode, and

when a request picture instruction from the streaming media server 2 is received through the first transmission module 13, based on search object information included in the request picture instruction, according to a mapping relationship between the multi-path picture data and a panoramic image sequence, searching the multi-path picture data corresponding to the search object information in the stored at least part of the multi-path picture data, and sending the multi-path picture data to the streaming media server 2 through the first transmission module 13 as a search result; the search object information comprises related information of a panoramic image frame to be searched;

the streaming media server 2 is adapted to receive the compressed and encoded video stream from the image acquisition terminal 1 in real time and send the video stream to the corresponding user terminal 3, perform secondary panorama stitching on the multiple paths of picture data in the search result after receiving the search result from the image acquisition terminal 1 to obtain a secondary panorama stitched image, and send the local or whole image of the secondary panorama stitched image as an output result to the corresponding user terminal 3.

The panoramic video playing and photographing system according to the scheme 2 and the scheme 1 is characterized in that the streaming media server 2 comprises a second transmission module 24,

the second transmission module 24 is adapted to transmit the whole image of the secondary panoramic stitched image to the corresponding user terminal 3.

The panoramic video playing and photographing system according to the scheme 3 and the scheme 1 is characterized in that the streaming media server 2 comprises a virtual photographing module 22,

the virtual photographing module 22 is adapted to determine the sending gesture of the user terminal 3, so as to cut out a local image corresponding to the gesture from the secondary panoramic stitched image according to the sending gesture of the user terminal 3, and send the local image as an output result to the user terminal 3.

Scheme 4, the panoramic video playing and photographing system according to any of the schemes 1 to 3, wherein the multimedia processing module 12 includes a picture stitching sub-module 122,

the picture stitching sub-module 122 is adapted to perform a first panorama stitching on the multiple paths of picture data collected by the camera module 11 by using a fast stitching algorithm to form a panorama image sequence.

The panoramic video playing and photographing system according to the scheme 5 and 3 is characterized in that the virtual photographing module 22 is adapted to perform secondary panoramic stitching on the multiple paths of picture data in the search result by using an accurate stitching algorithm to obtain a secondary panoramic stitching image.

The panoramic video playing and photographing system according to claim 6 or any one of claims 1 to 5, wherein the at least part of the multiple paths of picture data stored in the multimedia processing module 12 are multiple paths of picture data stored in a latest preset time period.

Scheme 7, according to any one of schemes 1 to 6, the panoramic video playing and photographing system is characterized in that the camera module 11 includes a plurality of camera modules, and the plurality of camera modules are adapted to acquire corresponding pictures of view angles based on a plurality of view angles as the multi-path picture data.

Scheme 8, the panoramic video playing and photographing system according to scheme 7, characterized in that,

the panoramic image sequence comprises a plurality of frame sequences, Ir represents an r frame sequence in the plurality of frame sequences, r is a positive integer, and Irn represents an n frame image in the r frame sequence;

the multi-path picture data is represented by Drn ═ D₁，D₂......，D_mWhere m represents the number of camera modules,which is a positive integer, Drn indicates the multi-path picture data corresponding to the nth frame image in the nth frame sequence, and the mapping relationship between the multi-path picture data and the panoramic image sequence is indicated as Irn → Drn.

Scheme 9, according to scheme 7 or 8 panorama video broadcast system of shooing, its characterized in that, a plurality of modules of making a video recording include at least one in fisheye camera and the wide angle camera.

The panoramic video playing and photographing system according to the scheme 10 and 4 is characterized in that the panoramic image sequence formed by the picture stitching sub-module 122 is a marked panoramic image sequence.

Scheme 11, the panoramic video playing and photographing system according to scheme 10, wherein the picture stitching sub-module 122 is adapted to:

setting a frame rate f, wherein each f frame of a video stream is a frame sequence, dividing the video into a plurality of frame sequences according to the playing time sequence, and using Ir to represent an r-th frame sequence, wherein r is a positive integer and is 1, 2; ir1, Ir2,...... Irf is used to represent f frame images in the r frame sequence, Iri is used as a watermark generation parameter of the i frame image in the r frame sequence, wherein i ∈ {1,2,..., f }, a corresponding mark is obtained according to the watermark generation parameter Iri, and the mark is added to the corresponding image frame to obtain each frame image with the mark, so as to form the panoramic image sequence with the mark.

The panoramic video playing and photographing system according to claim 12 and any one of claims 1 to 11, wherein the streaming server 2 includes a user management module 23,

the user management module 23 is adapted to receive an identity authentication request from each user terminal, perform identity authentication on the corresponding user equipment based on the identity authentication request, and send authentication success information to the corresponding user equipment after the authentication is successful so as to establish data connection with the user equipment.

Scheme 13 and the panoramic video playing and photographing system according to any one of schemes 1 to 12, wherein the streaming media server 2 includes a video processing module 21

The video processing module 21 is adapted to receive the compressed and encoded video stream transmitted by the image capturing end 1 in real time through the second transmission module 24, and obtain a video stream to be delivered based on the compressed and encoded video stream to be delivered to the user end 3.

The panoramic video playing and photographing system according to the claim 14 and the claim 13 is characterized in that the video processing module 21 is adapted to issue the video stream to be issued to the user terminal 3 according to a predetermined manner.

The panoramic video playing and photographing system according to claim 15 and 14, wherein the video processing module 21 is adapted to re-compression-encode the compression-encoded video streams into a plurality of video streams, so as to transmit at least part of the plurality of video streams to the corresponding users 3, wherein resolutions of the plurality of video streams are different from each other.

The panoramic video playing and photographing system according to claim 16 and 5 is characterized in that the virtual photographing module 22 is adapted to receive a photographing request command from the user side 3 and execute corresponding processing.

The panoramic video playing and photographing system according to the claim 17 and the claim 16 is characterized in that the virtual photographing module 22 is suitable for

When receiving at least one photographing request instruction from one or more user terminals 3, for each photographing request instruction,

determining the related information of the panoramic image frame to be searched according to the video frame data contained in the request photographing instruction, generating a request picture instruction containing the related information of the panoramic image frame to be searched,

the request picture instruction is sent to the image acquisition terminal 1 through the second transmission module 24, and the multi-path picture data sent back by the image acquisition terminal 1 aiming at the request picture instruction is received from the image acquisition terminal 1, and the multi-path picture data is subjected to secondary panoramic stitching by adopting an accurate stitching algorithm,

and cutting a picture corresponding to the sending gesture in the obtained secondary panoramic mosaic image according to the sending gesture contained in the shooting request instruction, and sending the picture as an output result to the corresponding user terminal 3 through the second transmission module 24.

The panoramic video playing and photographing system according to claim 18 and 17, wherein the virtual photographing module 22 is adapted to obtain the related information of the corresponding panoramic image frame to be searched by decoding the watermark contained in the video frame data.

The panoramic video playing and photographing system according to claim 19 and 18, wherein the information related to the panoramic image frame to be searched is represented as Iri, where i is a positive integer from 1 to m.

The panoramic video playing and photographing system according to the scheme 20 or the scheme 18 or 19 is characterized in that the watermark is a one-dimensional code watermark or a two-dimensional code watermark.

The panoramic video playing and photographing system according to any one of the schemes 21 and 17-20 is characterized in that the virtual photographing module 22 is adapted to store the obtained output result in the user management module 23 under the corresponding user account for user preview and/or pay downloading.

Scheme 22, the panoramic video playing and photographing system according to any one of the schemes 17 to 21, wherein the virtual photographing module 22 is adapted to:

remapping the secondary panorama stitching image onto a spherical surface as a panorama mapping spherical surface,

the angular range of the user's field of view is determined according to the sending pose of the device of the user terminal 3,

and determining a spherical figure to be cut on the panoramic mapping spherical surface according to the angle range of the user visual field, cutting off the spherical figure to be cut, and mapping the spherical figure to be cut into a target rectangular picture, wherein the size of the target rectangular picture is a preset value.

Scheme 23, the panoramic video playing and photographing system according to scheme 22, wherein the virtual photographing module 22 is adapted to:

and generating a corresponding panoramic mapping spherical surface by taking the side edge of the planar rectangular picture corresponding to the secondary panoramic mosaic image as a spherical semicircular arc, and mapping the planar rectangular picture onto the panoramic mapping spherical surface to complete the mapping of the secondary panoramic mosaic image onto the spherical surface.

Scheme 24, the panoramic video playing and photographing system according to

scheme

2 or 3, wherein each of the first transmission module 13, the second transmission module 24 and the third transmission module 34 respectively includes one or more transmission components.

Scheme 25 and the panoramic video playing and photographing system according to scheme 24 are characterized in that each of the first transmission module 13, the second transmission module 24 and the third transmission module 34 adopts at least one transmission mode of wired transmission and wireless transmission.

Scheme 26, the panoramic video playing and photographing system according to any one of the schemes 1 to 25, wherein the user terminal 3 is adapted to send an identity authentication request to the streaming media server terminal 2 in the panoramic video playing and photographing system, and establish a data connection with the streaming media server terminal 2 after receiving authentication success information from the streaming media server terminal 2.

The panoramic video playing and photographing system according to claim 27 and claim 26, wherein the user terminal 3 is adapted to send a video playing request to the streaming media server terminal 2 after establishing a data connection with the streaming media server terminal 2, and decode video data distributed from the streaming media server terminal 2 in real time.

Scenario 28, the panoramic video playing and photographing system according to any of scenarios 1 to 27, wherein the user terminal 3 is adapted to select one of one or more resolution viewing modes for panoramic video playing according to user input.

Scheme 29, the panoramic video playing and photographing system according to any one of the schemes 1 to 28, wherein the user terminal 3 is adapted to send a photographing request according to a user input during the panoramic video playing process to obtain a high resolution picture.

The panoramic video playing and photographing system according to claim 30 and any one of claims 1 to 29, wherein the user terminal 3 is adapted to send a preview request to the streaming server terminal 2 according to a user input, and when the user terminal 3 is allowed to preview to the streaming server terminal 2, preview a corresponding photo.

Scheme 31 and the panoramic video playing and photographing system according to any one of the schemes 1 to 30, wherein the user terminal 3 is adapted to send a payment downloading request to the streaming server terminal 2 according to a user input, and download the corresponding photo to the local area when the user terminal 3 is allowed to download to the streaming server terminal 2.

The panoramic video playing and photographing system according to the scheme 32 and the scheme 29 is characterized in that the photographing request sent by the user side 3 includes video frame data and a sending gesture of a device corresponding to the user side 3.

Scheme 33, the panoramic video playing and photographing system according to scheme 32, wherein the user end 3 is adapted to obtain a screen normal direction of a display module in a device corresponding to the user end 3 or a screen back normal direction thereof as a sending gesture when determining that a photographing request is to be sent.

Scheme 34, the panoramic video playing and photographing system according to

scheme

32 or 33, wherein the user end 3 is adapted to set a sending gesture by detecting a touch operation of a corresponding device, or acquire the sending gesture by a gesture sensor in the corresponding device.

Scheme 35 and the panoramic video playing and photographing system according to any one of schemes 1 to 34, wherein a user interaction module for sending various requests is provided in the user side 3.

Scheme 36 and the panoramic video playing and photographing system according to any one of schemes 1 to 35, wherein the panoramic video playing and photographing system is used for live panoramic video.

Finally, it is also noted that, in the present disclosure, relational terms such as left and right, first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "comprising", without further limitation, means that the element so defined is not excluded from the group consisting of additional identical elements in the process, method, article, or apparatus that comprises the element.

While the disclosure has been disclosed by the description of specific embodiments thereof, it will be appreciated that those skilled in the art will be able to devise various modifications, improvements, or equivalents of the disclosure within the spirit and scope of the appended claims. Such modifications, improvements and equivalents are intended to be included within the scope of the present disclosure as claimed.

Claims

1. The panoramic video playing and photographing system based on bidirectional communication is characterized by comprising an image acquisition end (1), a streaming media server end (2) and one or more user ends (3); the image acquisition terminal (1) comprises a camera module (11), a multimedia processing module (12) and a first transmission module (13);

the camera module (11) is suitable for acquiring multi-path picture data in the visual field of the camera module;

the multimedia processing module (12) is adapted to:

performing panoramic stitching on the multi-path picture data acquired by the camera module (11) in real time for the first time to form a panoramic image sequence, forming a mapping relation between the multi-path picture data and the panoramic image sequence according to a time corresponding relation, and storing the mapping relation and at least part of the multi-path picture data,

compressing and coding the panoramic image sequence to obtain a compressed and coded video stream, and pushing the compressed and coded video stream to a streaming media server end through the first transmission module (13), wherein the first transmission module adopts a duplex working mode, and

when a request picture instruction from a streaming media server (2) is received through the first transmission module (13), based on search object information contained in the request picture instruction, according to a mapping relation between the multi-path picture data and a panoramic image sequence, searching multi-path picture data corresponding to the search object information in at least part of the stored multi-path picture data, and sending the multi-path picture data to the streaming media server (2) through the first transmission module (13) as a search result; the search object information comprises related information of a panoramic image frame to be searched;

the streaming media server (2) is suitable for receiving the compressed and coded video stream from the image acquisition end (1) in real time and sending the video stream to a corresponding user end (3), when receiving a search result from the image acquisition end (1), performing secondary panoramic stitching on multiple paths of picture data in the search result to obtain a secondary panoramic stitched image, and sending a local or whole image of the secondary panoramic stitched image as an output result to the corresponding user end (3);

the multimedia processing module (12) comprises a picture splicing sub-module (122), and the picture splicing sub-module (122) is suitable for carrying out first-time panoramic splicing on the multi-channel picture data acquired by the camera module (11) by adopting a quick splicing algorithm to form a panoramic image sequence;

the streaming media server (2) comprises a virtual photographing module (22), wherein the virtual photographing module (22) is suitable for performing secondary panoramic stitching on the multi-path picture data in the search result by adopting an accurate stitching algorithm to obtain a secondary panoramic stitching image.

2. The panoramic video playback photographing system of claim 1,

the virtual photographing module (22) is suitable for determining the sending gesture of the user side, so that a local image corresponding to the gesture is cut out from the secondary panoramic mosaic image according to the sending gesture of the user side and is sent to the user side as an output result.

3. The panoramic video playback photographing system of claim 1 or 2,

the multi-path picture data is represented by Drn ═ D₁，D₂……，D_mM represents the number of camera modules, which is a positive integer, Drn represents the multi-path picture data corresponding to the nth frame image in the r frame sequence, and the mapping relationship between the multi-path picture data and the panoramic image sequence is represented as Irn → Drn.

4. The panoramic video playback photographing system of claim 1, wherein the panoramic image sequence is a marked panoramic image sequence;

the video encoding submodule (125) is adapted to:

setting a frame rate f, dividing a video into a plurality of frame sequences according to the playing time sequence by taking each f frame of the video stream as one frame sequence, and expressing an r-th frame sequence by using Ir, wherein r is a positive integer and r is 1,2 and …; the f frame images in the r frame sequence are represented by Ir1, Ir2 and … … Irf, Iri is used as watermark generation parameters of the i frame image in the r frame sequence, wherein i belongs to {1,2, … and f }, corresponding marks are obtained according to the watermark generation parameters Iri, and the marks are added to the corresponding image frames to obtain marked frame images so as to form a marked panoramic image sequence.

5. The panoramic video playing and photographing system according to claim 1, wherein the streaming media server (2) comprises a second transmission module (24); the virtual photography module (22) is adapted to:

when at least one photographing request instruction is received from one or more user terminals, for each photographing request instruction,

sending the request picture instruction to the image acquisition end (1) through the second transmission module (24), receiving multi-path picture data sent back by the image acquisition end (1) aiming at the request picture instruction from the image acquisition end, and carrying out secondary panoramic stitching on the multi-path picture data by adopting a precise stitching algorithm,

and cutting a picture corresponding to the sending gesture in the obtained secondary panoramic mosaic image according to the sending gesture contained in the shooting request instruction, and sending the picture as an output result to a corresponding user side through the second transmission module (24).

6. The panoramic video playback photographing system of claim 5, wherein the virtual photographing module (22) is adapted to:

determining the angle range of the user visual field according to the sending gesture of the device at the user end,

7. The panoramic video playing and photographing system of claim 1, wherein the photographing request sent by the user terminal (3) comprises video frame data and a sending gesture of a device corresponding to the user terminal (3).

8. The system according to claim 1 or 2, wherein said user terminal 3 is adapted to select one of one or more resolution viewing modes for panoramic video playback according to user input.