CN107592549A

CN107592549A - Panoramic video based on two-way communication plays camera system

Info

Publication number: CN107592549A
Application number: CN201711061964.7A
Authority: CN
Inventors: 朱磊; 李建英; 韩琦
Original assignee: Harbin Yishe Technology Co Ltd
Current assignee: QINGDAO YISPACE TECHNOLOGY Co.,Ltd.
Priority date: 2017-10-27
Filing date: 2017-10-27
Publication date: 2018-01-16
Anticipated expiration: 2037-10-27
Also published as: CN107592549B

Abstract

Present disclose provides a kind of panoramic video based on two-way communication to play camera system.Said system includes IMAQ end, streaming media server end and one or more user terminals；IMAQ end carries out panoramic mosaic first to multichannel image data in real time, the mapping relations formed between panoramic image sequence and multichannel image data, is played video flowing plug-flow to streaming media server end to be transferred to user by the first transport module；First transport module uses duplex mode of operation；When the first transport module receives the request picture instruction from streaming media server end, corresponding multichannel image data is sent to streaming media server end, to obtain photo needed for user by secondary panoramic mosaic to be sent to user.The panoramic video based on two-way communication of the present invention plays camera system, and occupied bandwidth is smaller, and transmission and broadcasting speed are very fast, and high-resolution picture can be quickly provided it when user needs to take pictures.

Description

Panoramic video based on two-way communication plays camera system

Technical field

This disclosure relates to the communication technology, more particularly to a kind of panoramic video based on two-way communication play camera system.

Background technology

In this day and age, video playback is increasingly welcome by masses, makes a general survey of the development history of video playback, is by video point Develop on the basis of broadcasting, played out by internet and stream media technology, so as in real time, comprehensively propagate video Content.In the video playback of early stage, the viewing area of video content is because of reasons such as shootings, such as：The angle of finding a view of camera lens Degree, scope, cause its fixations non-adjustable, and user can only watch current region, video pictures can only follow camera lens movement and Change, user are watched without third dimension, can not there is impression on the spot in person.

With the development of panoramic video technology, there are 360 degree of panoramic videos, can allow user comprehensive 360 degree from , can front and rear up and down, even far and near distance viewing by watching.Typically by multiple video cameras simultaneously to same space Under object carry out multiple angle shots to realize, allow user in the case where video playback need not be suspended playing end, from By switching viewing angle；Panoramic video has third dimension and the sense of reality, can provide visual experience well；It can realize that body faces The experience in its border and the surprised effect more to make one see with different eyes.

With the continuous improvement of people's demand, people are more desirable to not only to can see when watching panoramic video true, complete Face, intuitively video, high quality graphic can also be obtained in real time and is kept as a souvenir or shares with friend.But view data Transmission speed can be limited by transmission bandwidth, transmission stability can also be influenceed by transmission bandwidth, i.e. transmission bandwidth meeting Influence the fluency of video playback and the instantaneity of image data acquisition.

The content of the invention

The brief overview on the disclosure has been given below, so as to provide on the disclosure it is some in terms of it is basic Understand.It should be appreciated that this general introduction is not the exhaustive general introduction on the disclosure.It is not intended to determine the key of the disclosure Or pith, also it is not intended to limit the scope of the present disclosure.Its purpose only provides some concepts in simplified form, with this As the preamble in greater detail discussed later.

In view of the drawbacks described above of prior art, the first purpose of the disclosure is to provide a kind of aphorama based on two-way communication Frequency plays camera system, at least to solve the problems, such as how quickly to obtain high-resolution pictures in video playback.

According to an aspect of this disclosure, there is provided a kind of panoramic video based on two-way communication plays camera system, should System includes IMAQ end, streaming media server end and one or more user terminals；Described image collection terminal includes shooting Module, multimedia process module and the first transport module；The photographing module is suitable to gather the multichannel image data in its visual field； The multimedia process module is suitable to：Panoramic mosaic first is carried out to the multichannel image data of photographing module collection in real time To form panoramic image sequence, the mapping formed according to corresponding time relationship between multichannel image data and panoramic image sequence is closed System, and mapping relations and at least part the multichannel image data is preserved, coding is compressed to the panoramic image sequence, The video flowing after compressed encoding is obtained, and the video flowing after compressed encoding is passed through into the first transport module plug-flow to Streaming Media Server end, wherein, first transport module uses duplex mode of operation, and is received by first transport module When being instructed to the request picture from streaming media server end, based on the lookup object letter included in the request picture instruction Breath, according to the multichannel image data and the mapping relations of panoramic image sequence, at least part multichannel picture of preservation Multichannel image data corresponding with the lookup object information is searched in data, passes through the described first transmission mould as lookup result Block and be sent to the streaming media server end；Wherein, the object information of searching includes the phase of panoramic image frame to be found Close information；The streaming media server end be suitable to from the video flowing after compressed encoding described in described image collection terminal real-time reception with Corresponding user terminal is handed down to, after receiving since the lookup result of described image collection terminal, in the lookup result Multichannel image data carries out secondary panoramic mosaic, obtains secondary panoramic mosaic image, and by the office of the secondary panoramic mosaic image Portion or general image are handed down to corresponding user terminal as output result.

The above-mentioned panoramic video based on two-way communication according to the embodiment of the present disclosure plays camera system, including IMAQ End, streaming media server end and one or more user terminals；IMAQ end include photographing module, multimedia process module and First transport module.Multimedia process module carries out panoramic mosaic first to the multichannel image data of photographing module collection in real time To form panoramic image sequence, the mapping relations formed between multichannel image data and panoramic image sequence, pass through the first transmission Module is by video flowing plug-flow to streaming media server end；When receiving the instruction of the request picture from streaming media server end, Corresponding multichannel image data is sent to streaming media server end；Regarding after the real-time reception compressed encoding of streaming media server end Frequency, which flows down, issues user terminal, and when user's request is taken pictures, the multichannel image data obtained from IMAQ end is carried out secondary Panoramic mosaic is to obtain photo needed for user.Thus, by the two-way communication between IMAQ end and streaming media server end, Substantial amounts of high-resolution pictures need not be stored at streaming media server end, reduce system burden.

Further, on the one hand, the multichannel image data that Fast stitching algorithm can be used to gather photographing module 11 enters Panoramic mosaic forms panoramic image sequence to row first, is passing through compressed encoding so that and internal memory shared by the video flowing after compression is smaller, Can be rapidly transferred to less bandwidth server end (that is, from collection terminal to server end transmit " by fast Speed Pinyin Connect the compressed video data of the panoramic image sequence of algorithm formation " shared by bandwidth is small, transmission speed is fast), and then cause server Compressed video data can be quickly provided to user with less bandwidth and played (that is, being passed from server end to user terminal by end Bandwidth shared by defeated " compressed video data of the panoramic image sequence formed by Fast stitching algorithm " is small, transmission speed is fast).

In other words, under " video play mode ", after obtaining panoramic image sequence and compression by Fast stitching algorithm Server end is transferred to, then user terminal is handed down to by server, can ensure that shared bandwidth is smaller, transmission speed is very fast so that The transmission burden of server is smaller, and user plays more smoothness, and viewing experience is preferable.

On the other hand, in the case where collection terminal (the first transport module) uses diplex operation module, collection terminal preserves Multichannel image data, so, needing to take pictures or during self-timer in user, user sends request of taking pictures to server end, then by servicing Device end sends picture request to collection terminal, is taken pictures or that frame of self-timer is corresponding with user's request needs with being searched in collection terminal Multichannel image data is sent to server end as lookup result；In server end using accurate stitching algorithm in lookup result Multichannel image data carry out secondary panoramic mosaic and obtain secondary panoramic mosaic image, utilize secondary panoramic mosaic image (example Corresponding cut out is carried out as posture can be sent according to user terminal) take pictures needed for obtaining user or the part image of self-timer, To meet that user realizes that high definition is taken pictures or the purpose of high definition self-timer.

In other words, under " user ask exposal model ", by being searched in the multichannel image data that is stored in collection terminal The multichannel image data of corresponding frame returns to server end, then obtains secondary panorama using accurate stitching algorithm by server end and spell Map interlinking picture, (such as cutting) photo needed for user is obtained wherein to be sent to user so that user can obtain its needs High definition photo (resolution ratio watched higher than it in video), without influenceing between collection terminal and server end and servicing Transmission speed between device end and user terminal, and then make it that the broadcasting speed of user terminal is very fast.

To sum up, obtain what panoramic image sequence obtained using Fast stitching algorithm under " video play mode " using above-mentioned Compressed video data is transferred to user and obtains the multichannel image data of corresponding frame from collection terminal under " user asks exposal model " And the high definition panorama image sequence (i.e. secondary panoramic mosaic image) obtained using accurate stitching algorithm is transferred to the side of user Formula, it can take in the case of smaller strip is wide, transmission speed is faster, additionally it is possible to provide the user with high definition photo；Simultaneously because Using duplex mode of operation, it is not necessary to a large amount of high definition image datas are stored in server end, while alleviating server burden, Further reduce the transmission bandwidth between collection terminal and server end.

The above-mentioned panoramic video based on two-way communication according to the embodiment of the present disclosure plays camera system, can also have with Lower at least one beneficial effect：

1st, user can use the system to be realized with unique perspective and take pictures, and obtain high-resolution pictures；

2nd, several users can share the system simultaneously and watch video playback in real time and take pictures；

3rd, reality is only needed by carrying multimedia process module and bi-directional communication modules, IMAQ end at IMAQ end When transmission compression after video flowing and after request instruction is received original image data needed for transmission to streaming media server End, without all original image data of real-time Transmission, not only alleviate the computing pressure at streaming media server end, Er Qieyou The transmission bandwidth for saving view data of effect, improve the efficiency of transmission of view data.

For example, image is gathered by photographing module and directly transmitted all original image data to stream by transport module Media server end, video playback and camera function are realized by streaming media server end, frame per second is 25 if assuming, every figure of single channel Piece size is 5M or so, totally 8 road picture, then transport module required bandwidth per second is 5*8*25=1000M.

Only needed in real time by carrying multimedia process module and bi-directional communication modules, IMAQ end at IMAQ end Video flowing after transmission compression and after request instruction is received original image data needed for transmission to streaming media server end, If assuming, frame per second is 25, and single channel is 5M or so per pictures size, and totally 8 road picture, by taking video playback time 1min as an example, is then passed Bandwidth needed for defeated module is per second is approximately equal to 47M (experiment gained video flowing)+5*8 (required original image)=87M.

By excellent below in conjunction with detailed description of the accompanying drawing to the most preferred embodiment of the disclosure, the these and other of the disclosure Point will be apparent from.

Brief description of the drawings

The disclosure can be by reference to being better understood, wherein in institute below in association with the description given by accompanying drawing Have and same or analogous reference has been used in accompanying drawing to represent same or similar part.The accompanying drawing is together with following Describe in detail and include in this manual and formed the part of this specification together, and for this is further illustrated Disclosed preferred embodiment and the principle and advantage for explaining the disclosure.Wherein：

Fig. 1 is that the one kind for showing to play camera system according to the panoramic video based on two-way communication of the embodiment of the present disclosure is shown The schematic diagram of example structure；

Fig. 2 is the schematic diagram for the possibility structure for showing the IMAQ end in Fig. 1；

Fig. 3 is the schematic diagram for the possibility structure for showing the streaming media server end in Fig. 1；

Fig. 4 is the schematic diagram of the possibility structure of the user terminal corresponding device in the Fig. 1 shown；

Fig. 5 is the another kind for showing to play camera system according to the panoramic video based on two-way communication of the embodiment of the present disclosure The schematic diagram of exemplary construction.

It will be appreciated by those skilled in the art that element in accompanying drawing is just for the sake of showing for the sake of simple and clear, And be not necessarily drawn to scale.For example, the size of some elements may be exaggerated relative to other elements in accompanying drawing, with Just it is favorably improved the understanding to the embodiment of the present disclosure.

Embodiment

The one exemplary embodiment of the disclosure is described hereinafter in connection with accompanying drawing.For clarity and conciseness, All features of actual embodiment are not described in the description.It should be understood, however, that developing any this actual implementation It must be made during example much specific to the decision of embodiment, to realize the objectives of developer, for example, symbol Those restrictive conditions related to system and business are closed, and these restrictive conditions may have with the difference of embodiment Changed.In addition, it will also be appreciated that although development is likely to be extremely complex and time-consuming, to having benefited from the disclosure For those skilled in the art of content, this development is only routine task.

Herein, it is also necessary to which explanation is a bit, in order to avoid having obscured the disclosure because of unnecessary details, in the accompanying drawings It illustrate only with according to the closely related apparatus structure of the scheme of the disclosure and/or processing step, and eliminating and the disclosure The little other details of relation.

Embodiment of the disclosure provides a kind of panoramic video based on two-way communication and plays camera system, and the system includes IMAQ end, streaming media server end and one or more user terminals；Described image collection terminal includes photographing module, more matchmakers Body processing module and the first transport module；The photographing module is suitable to gather the multichannel image data in its visual field；More matchmakers Body processing module is suitable to：It is complete to be formed that panoramic mosaic first is carried out to the multichannel image data that the photographing module gathers in real time Scape image sequence, the mapping relations formed according to corresponding time relationship between multichannel image data and panoramic image sequence, and protect Mapping relations and at least part the multichannel image data is deposited, coding is compressed to the panoramic image sequence, pressed The video flowing after code is reduced the staff, and the video flowing after compressed encoding is passed through into the first transport module plug-flow to streaming media server End, wherein, the first transport module uses duplex mode of operation, and is received by first transport module come matchmaker of flowing automatically During the request picture instruction of body server end, based on the lookup object information included in the request picture instruction, according to described The mapping relations of multichannel image data and panoramic image sequence, at least part multichannel image data of preservation search with Multichannel image data corresponding to the lookup object information, as lookup result by first transport module to be sent to State streaming media server end；Wherein, the object information of searching includes the relevant information of panoramic image frame to be found；The stream Media server end is suitable to corresponding to be handed down to from the video flowing after compressed encoding described in described image collection terminal real-time reception User terminal, after receiving since the lookup result of described image collection terminal, to the multichannel image data in the lookup result Secondary panoramic mosaic is carried out, obtains secondary panoramic mosaic image, and by the partly or wholly image of the secondary panoramic mosaic image Corresponding user terminal is handed down to as output result.

Wherein, camera system is played according to the panoramic video based on two-way communication of the embodiment of the present disclosure and can be used for panorama The program request (recorded broadcast or playback etc.) or live of video.

Fig. 1 is schematically shown plays camera system according to the panoramic video based on two-way communication of the embodiment of the present disclosure A kind of exemplary construction.

As shown in figure 1, playing camera system according to the panoramic video based on two-way communication of the embodiment of the present disclosure includes figure As collection terminal 1, streaming media server end 2 and one or more user terminals 3, wherein, IMAQ end 1 includes photographing module 11st, the transport module 13 of multimedia process module 12 and first.

Photographing module 11 is suitable to gather the multichannel image data in its visual field.

In accordance with an embodiment of the present disclosure, photographing module 11 can for example include multiple camera modules, and multiple camera modules are fitted In based on multiple visual angles obtain corresponding to institute's multi-perspective picture, as multichannel image data.Wherein, multiple camera modules for example can be with Including at least one of fish-eye camera and wide-angle camera.

In one example, multiple camera modules are for example including multiple fish-eye cameras, such as, can use 2 (or its His quantity) fish-eye camera obtains the two-way image data of 360 degree of angulars field of view of covering (as showing for multichannel image data Example).

In another example, multiple camera modules for example including multiple wide-angle cameras, such as, can use 8 (or Other quantity) wide-angle camera come obtain covering 360 degree of angulars field of view 8 road image datas (as multichannel image data Example).

Multimedia process module 12 is adapted for carrying out following processing：The multichannel image data gathered in real time to photographing module 11 Carry out panoramic mosaic first and, to form panoramic image sequence, multichannel image data and panoramic picture are formed according to corresponding time relationship Mapping relations between sequence, and preserve mapping relations and at least part multichannel image data；Panoramic image sequence is carried out Compressed encoding, the video flowing after compressed encoding is obtained, and the video flowing after compressed encoding is passed through into the plug-flow of the first transport module 13 To streaming media server end；And the request picture from streaming media server end 2 is being received by the first transport module 13 During instruction, based on the lookup object information that includes in request picture instruction, according to multichannel image data and panoramic image sequence Mapping relations, multichannel image data corresponding with searching object information is searched at least part multichannel image data of preservation, Streaming media server end 2 is sent to by the first transport module 13 as lookup result；Wherein, searching object information includes treating The relevant information of the panoramic image frame of lookup.Multimedia process module 12 can for example use following any realization：Development board； CPU；Microprocessor；Microcomputer；Etc..

Wherein, the first transport module uses duplex mode of operation, that is to say, that the first transport module is applied not only to stream matchmaker Body server end 2 (second transport module at such as streaming media server end 2) sends data, additionally it is possible to from streaming media server end 2 Receive data.

For example in the request photographing instruction for receiving user terminal 3, (i.e. described below take pictures please at streaming media server end 2 Ask) when, corresponding request picture instruction is sent to collection terminal 1, " lookup pair described above is included in request picture instruction Image information ", " relevant information of panoramic image frame to be found " in " lookup object information " for example may come from user terminal 3 Request photographing instruction included in watermark, such as, pass through and obtained after being decoded to watermark (one-dimension code or two-dimentional code watermark) Information (the watermark generation parameter Iri mentioned in following article) is as above-mentioned " relevant information of panoramic image frame to be found ".

In accordance with an embodiment of the present disclosure, at least part multichannel image data that multimedia process module 12 preserves can be most The multichannel image data preserved in nearly preset time period, for example, in nearest 1 minute (or 3 minutes, 10 minutes, 1 hour, 1 day etc.) Multichannel image data.Wherein, preset time period can for example be set based on experience value.

In accordance with an embodiment of the present disclosure, panoramic image sequence for example represents the plurality of frame sequence including multiple frame sequences, Ir In r-th of frame sequence, r is positive integer, and Irn represents the n-th frame image in r-th of frame sequence, and n is also positive integer.N for example may be used With equal to frame per second, i.e. number of pictures per second, such as 25.

Multichannel image data can for example be expressed as Drn={ D₁, D₂..., D_m, wherein m represents the number of camera module Amount, it is positive integer, and Drn represents the multichannel image data corresponding to the n-th frame image in r-th of frame sequence, multichannel picture number Irn → Drn can be expressed as according to the mapping relations between panoramic image sequence.

Streaming media server end 2 is suitable to from the video flowing after the real-time reception compressed encoding of IMAQ end 1 to be handed down to pair The user terminal answered, after receiving since the lookup result at IMAQ end 1, to the multichannel image data in the lookup result Secondary panoramic mosaic is carried out, obtains secondary panoramic mosaic image, and by the partly or wholly image of the secondary panoramic mosaic image Corresponding user terminal is handed down to as output result.

Fig. 2 shows the schematic diagram of the possibility structure at the IMAQ end 1 in Fig. 1.

As shown in Fig. 2 the multimedia process module 12 in IMAQ end 1 can include picture input submodule 121, figure Piece splicing submodule 122, image cache submodule 123, picture output sub-module 124, Video coding submodule 125 and video are defeated Any one or more submodule gone out in submodule 126.Hereinafter by respectively more than citing description performed by each submodule Processing and function.

It should be noted that in fig. 2, each submodule 121~126, which is adopted, to be indicated by the dashed box, it is intended that indicates these It is essential that submodule, which is not, but optional respectively.

In accordance with an embodiment of the present disclosure, multimedia process module 12 for example splices submodule 122, picture splicing including picture The multichannel image data that submodule 122 may be adapted to gather photographing module 11 using Fast stitching algorithm carries out panorama first and spelled Connect, to form panoramic image sequence.Wherein, Fast stitching algorithm can for example use existing quick joining method, here no longer It is described in detail.

According to an implementation, the panoramic image sequence that picture splicing submodule 122 is formed is, for example, tape label Panoramic image sequence.Wherein, each frame panoramic picture in panoramic image sequence is for example respectively equipped with corresponding to the frame and marked, mark Note for example can be watermark, such as, one-dimensional code watermark or two-dimentional code watermark etc..

Wherein, the panoramic image sequence of tape label can for example be obtained by Video coding submodule 125.According to a reality Existing mode, Video coding submodule 125 can for example obtain the panoramic image sequence of tape label as follows：

Set frame per second f, video flowing using every f frames be a frame sequence (f is positive integer, for example, f can for 25), by video according to The time sequencing of broadcasting is divided into multiple frame sequences, and r-th of frame sequence is represented with Ir, and r is positive integer, r=1, and 2 ..., such as I1 represents first frame sequence, and I2 represents second frame sequence, etc..With Ir1, Ir2 ... Irf represents r-th of frame sequence In f two field pictures, by " Iri " as the i-th two field picture in r-th of frame sequence watermark generate parameter, wherein, i ∈ 1, 2 ..., f }, parameter Iri is generated according to watermark obtain corresponding mark and (for example an one-dimension code or Quick Response Code are generated according to Iri Watermark), the mark is added on corresponding picture frame (namely i-th two field picture in r-th of frame sequence), is derived from band mark Each two field picture of note, to form the panoramic image sequence of tape label.

In addition, the multichannel image data that picture input submodule 121 can be gathered with real-time reception photographing module, and pass simultaneously Defeated multichannel image data to picture splices submodule 122 and image cache submodule 123.

Image cache submodule 123 can preserve in some time multichannel picture number of (in such as nearest preset time period) According to, and preserve the multichannel image data formed according to corresponding time relationship and panoramic image sequence (such as the panorama of tape label Image sequence) between mapping relations.

Picture output sub-module 124 can receive the request figure at streaming media server end 2 by the first transport module 13 When piece instructs, according to multichannel image data and the mapping relations of panoramic image sequence (such as panoramic image sequence of tape label) Irn → Drn is searched in image cache submodule 123 and is corresponded to the multichannel image data Drn of present frame, and passes through the first transport module 13 output multi-channel image data Drn are to streaming media server end 2.

Video coding submodule 125 can be pressed panoramic image sequence (such as panoramic image sequence of tape label) Reduce the staff code.

Video frequency output submodule 126 can utilize transmission of video agreement that the video flowing after compressed encoding is passed through into the first transmission The plug-flow of module 13 is to streaming media server end 2.

Fig. 3 shows the schematic diagram of the possibility structure at the streaming media server end 2 in Fig. 1.

As shown in figure 3, streaming media server end 2 can for example include video processing module 21, virtual photograph module 22, use Any of transport module 24 of family management module 23 and second or multiple module.It is hereinafter that citing description respectively is each above Processing and function performed by module.

It should be noted that in figure 3, modules 21~24, which are adopted, to be indicated by the dashed box, it is intended that indicates these modules It is not essential, but it is optional respectively.

According to an implementation, streaming media server end 2 is for example including video processing module 21, video processing module 21 The video flowing that may be adapted to after the compressed encoding by the transmission of the real-time reception IMAQ end 1 of the second transport module 24, based on pressure Reduce the staff the video flowing after code and obtain video flowing to be issued to be issued to user terminal.

In one example, video processing module 21 can be by the video flowing after compressed encoding directly as video to be issued Flow and transmit to user terminal.

In another example, video flowing to be issued can also be issued to use by video processing module 21 according to predetermined way Family end, for example, video processing module 21 can be by the video flowing re-compression and encoding after compressed encoding into multiple video flowings, will At least part of the plurality of video flowing is sent to corresponding user terminal, wherein, the mutual not phase of resolution ratio between the plurality of video flowing Together.Wherein, " at least part of the plurality of video flowing " be, for example, which part video flowing (such as one or several) or its In whole.

For example, video processing module 21 can into N kinds, (N be by the video flowing re-compression and encoding after compressed encoding Positive integer, N can be 1 or more than 1, such as 3 or 5 etc.) video flowing of resolution ratio (or definition), such as very fast, SD, general Clearly, high definition, VIP are super clear, etc..According to user gradation (such as user is domestic consumer), one or more can be provided the user Optional resolution video stream, the video flowing of resolution ratio is corresponded to its transmission according to the selection of user；If user is that VIP is used Family, when the super clear video flowings of its selection VIP, then the video flowing of the definition (namely resolution ratio) is transmitted to it.

According to an implementation, streaming media server end 2 is for example including virtual photograph module 22, virtual photograph module 22 It may be adapted to the request photographing instruction for receiving user terminal, and perform alignment processing.

In addition, virtual photograph module 22 is for example also suitable for using accurate stitching algorithm to the multichannel picture in lookup result Data carry out secondary panoramic mosaic, to obtain secondary panoramic mosaic image.Wherein, accurate stitching algorithm can for example use existing Accurate stitching algorithm, such as light stream stitching algorithm.

Virtual photograph module 22 can for example perform following processing：When receiving from one or more user terminals at least During one request photographing instruction, for each request photographing instruction, steps as described below a1~a3 is performed respectively.

In step a1, panoramic image frame to be found is determined according to the video requency frame data included in the request photographing instruction Relevant information, and generate include panoramic image frame to be found relevant information request picture instruction, then perform step Rapid a2.

In step a2, the request picture is sent to IMAQ end 1 by the second transport module 24 and instructed, and from image Collection terminal 1 receives its multichannel image data for instructing and sending back for the request picture, and using accurate stitching algorithm to this Multichannel image data carries out secondary panoramic mosaic, then performs step a3.

In step a3, the secondary panoramic mosaic figure that is obtained according to transmission posture contained in the request photographing instruction at this The picture corresponding to the transmission posture is cut as in, corresponding user is handed down to by the second transport module 24 as output result End.

According to an implementation, virtual photograph module 22 for example can be by the watermark that is included in decoding video frame data And the relevant information of panoramic image frame to be found corresponding to obtaining.

Wherein, the relevant information of panoramic image frame to be found is for example expressed as Iri, wherein, i is just whole in 1~m Number.

Watermark is for example with one-dimensional code watermark or two-dimentional code watermark etc..

Obtained output result can for example be stored in user management module 23 and correspond to user by virtual photograph module 22 Under account, so that user terminal preview and/or payment are downloaded.

According to an implementation, secondary panoramic mosaic image can be for example remapped to one by virtual photograph module 22 On individual sphere, the sphere maps sphere as panorama, and the angle in the user visual field is determined according to the transmission posture of the equipment of user terminal Scope, spherical figure to be cut is determined on panorama mapping sphere according to the angular range in the user visual field, and cut out lower this and wait to cut Target rectangle picture is mapped to after spherical figure, wherein, the size of the target rectangle picture is preset value.The preset value can root Set, or can be set according to the screen size parameter included in the photographing instruction of user equipment transmission according to empirical value.

According to an implementation, virtual photograph module 22 for example can be with plane square corresponding to secondary panoramic mosaic image The side of shape picture generates corresponding panorama mapping sphere as sphere semi arch, and the planar rectangular picture is mapped to entirely On scape mapping sphere, to complete secondary panoramic mosaic image being mapped on sphere.Wherein, the side of planar rectangular picture is for example It is the shorter side of planar rectangular picture.

According to an implementation, streaming media server end 2 is for example including user management module 23, user management module 23 It may be adapted to receive the ID authentication request from each user terminal, and based on the ID authentication request to corresponding user equipment Authentication is carried out, sends authentication success message to the corresponding user equipment after the authentication has been successful to be established with the user equipment Data connect.

According to an implementation, streaming media server end 2 is for example including the second transport module 24, the second transport module 24 It can be used for the general image of secondary panoramic mosaic image being sent to corresponding user terminal.

According to another implementation, the virtual photograph module 22 in streaming media server end 2 can also determine to use first Family end transmission posture (send posture be, for example, from user terminal receive), with according to the transmission posture of user terminal in secondary panorama Cut out in stitching image to should posture topography, be handed down to user terminal as output result.

In accordance with an embodiment of the present disclosure, the first transport module 13 can for example include one or more transmission assemblies.First Transport module 13 can for example use wire transmission and at least one of be wirelessly transferred transmission means.

In accordance with an embodiment of the present disclosure, the second transport module 24 can for example include one or more transmission assemblies.Second Transport module 24 can for example use wire transmission and at least one of be wirelessly transferred transmission means.

In accordance with an embodiment of the present disclosure, the 3rd transport module 34 can for example include one or more transmission assemblies.3rd Transport module 34 can for example use wire transmission and at least one of be wirelessly transferred transmission means.

Fig. 4 shows a kind of possible structure of ustomer premises access equipment corresponding to user terminal 3.As shown in figure 4, each user terminal is set It is standby for example to include appointing in User Access Module 31, display module 32, the transport module 34 of user interactive module 33 and the 3rd One or more modules.Hereinafter by the processing performed by citing description above modules respectively and function.Need what is illustrated It is that in Fig. 4, modules 31~34, which are adopted, to be indicated by the dashed box, it is intended that it is essential to indicate that these modules are not, but point It is not optional.

Wherein, it can be for example provided with each user terminal 3 and be used to sending the user interactive modules 33 of various requests, and user The 3rd transport module 34 in end 3 for example can be used for realizing between second transport module 24 at streaming media server end 2 Data transfer etc..

According to an implementation, each user terminal 3 can for example play the Streaming Media in camera system to panoramic video Server end 2 sends ID authentication request, and is built therewith after the authentication success message from streaming media server end 2 is received Vertical data connection.For example, after the authentication has been successful, it can be established by User Access Module 31 between streaming media server end 2 Connection.

According to an implementation, each user terminal 3 can be for example connected establishing data with streaming media server end 2 Afterwards, viewing video playback request is sent to streaming media server end 2, and next regard is distributed at real-time decoding streaming media server end 2 Frequency evidence.

According to an implementation, each user terminal 3 for example can be according to user's input come in one or more resolution ratio Watching mode in select one kind come watch panoramic video play.

According to an implementation, each user terminal 3 can for example input in panoramic video playing process according to user To send request of taking pictures, to obtain high-resolution pictures.Wherein, high-resolution pictures are, for example, that resolution ratio is greater than or equal to (example Such as 4K) picture of predetermined threshold value, predetermined threshold value can for example be set based on experience value, or be determined by the method for experiment, here Repeat no more.

Wherein, high-resolution pictures are, for example, greater than or equal to the picture of default photo resolution, preset photo resolution Such as set based on experience value, or obtained by way of experiment.For example, the panoramic picture sequence first that panoramic mosaic obtains first Image resolution ratio in row is 2880*1440, and the resolution ratio for the secondary panoramic mosaic image that secondary panoramic mosaic obtains is 4000*3000.It should be appreciated that in practical application, the image resolution ratio in panoramic image sequence is not limited to first 2880*1440, the resolution ratio of secondary panoramic mosaic image are also not necessarily limited to 4000*3000, and above resolution ratio can be according to reality Situation changes the different numerical value of setting with needing, but it is required that the high resolution of secondary panoramic mosaic image in panorama first Image resolution ratio in image sequence.

According to an implementation, each user terminal 3 can for example be inputted according to user and sent out to streaming media server end 2 Preview request is sent, when allowing 3 preview of user terminal to streaming media server end 2, photo corresponding to preview.

According to an implementation, each user terminal 3 can for example be inputted according to user and sent out to streaming media server end 2 Payment download request is sent, when allowing the user terminal 3 to download to streaming media server end 2, corresponding photo is downloaded to local.

Wherein, for example comprising equipment corresponding to video requency frame data and the user terminal 3 in the request of taking pictures of the transmission of user terminal 3 Send posture.Video requency frame data is for example comprising watermark, such as one-dimension code or two-dimentional code watermark.

For example, user terminal 3 can it is determined that to send take pictures request when, obtain in the corresponding device of user terminal 3 and show mould The screen normal direction of block 32 or its screen back side normal direction, it is used as it and sends posture.

And for example, user terminal 3 can by detect corresponding device touch operation (such as display module 32 can be touch-screen, Touch operation on detection touch-screen) transmission posture is set, or pass through attitude transducer (such as gyroscope in corresponding device Deng) obtain transmission posture.

Preferred embodiment

Play camera system with reference to 5 panoramic videos based on two-way communication for describing the disclosure one is preferable to carry out Example.It should be noted that the IMAQ end and streaming media server end in the figure be equally applicable to it is described above based on double Camera system is played to the panoramic video of communication, is repeated no more here.

As shown in figure 5, IMAQ end includes photographing module, multimedia process module and the first transport module.Image mould Block includes the camera module that multiple flakes or wide-angle camera form；Multimedia process module include one piece of embedded board or One microcomputer of person, it includes picture input submodule, picture splicing submodule, image cache submodule, picture output Submodule, Video coding submodule and video frequency output submodule；First transport module and streaming media server end real time bidirectional lead to Letter, it includes one or more transport modules, can use wire transmission, can also use and be wirelessly transferred.IMAQ end leads to Cross the first transport module pushing video stream and send image data, while the request picture instruction of receiving stream media server end.

Streaming media server end includes video processing module, virtual photograph module, user management module and the second transmission mould Block.

User terminal (i.e. user terminal) includes one or more user equipmenies, and each user equipment includes user and accesses mould Block, user interactive module, display module and the 3rd transport module.Wherein, user equipment can be smart mobile phone, tablet personal computer or The terminal devices such as wear-type VR systems.

IMAQ end workflow：

Step 1：Photographing module receives acquisition instructions by the first transport module, gathers multichannel image data in its visual field, When receiving stopping acquisition instructions, photographing module stops collection.Photographing module includes multiple flakes or wide-angle camera forms Camera module, multichannel image data is the various visual angles picture of the camera module collection of multiple flakes or wide-angle camera composition, Multichannel image data can use D={ D1, D2 ..., Dm } to represent, wherein m represents the quantity of camera module, and it is positive integer.

Step 2：The multichannel picture number that picture input submodule real-time reception photographing module gathers in multimedia process module According to, and simultaneous transmission multichannel image data to picture splices submodule and image cache submodule.

Step 3：Picture splicing submodule splices multichannel picture in real time with Fast stitching algorithm in multimedia process module Data complete panoramic mosaic and form the panoramic image sequence of tape label, then the panoramic image sequence of transmission tape label to video Encoding submodule.

The panoramic image sequence of tape label may be formed so that：If it is 25 to set frame per second, video flowing is using 25 frames as one Frame sequence, video is divided into several frame sequences according to the time sequencing of broadcasting, r-th of frame sequence is represented with Ir, Ir1, Ir2 ... Ir25 represents 25 two field pictures in r-th of frame sequence, and parameter, i.e. r-th of frame sequence are generated using Irn as watermark In n-th frame image, one-dimension code or two-dimentional code watermark are added on each two field picture of each frame sequence.

Step 4：Video coding submodule is compressed volume to the panoramic image sequence of tape label in multimedia process module Code.

Step 5：Video frequency output submodule utilizes transmission of video agreement by regarding after compressed encoding in multimedia process module Frequency stream passes through the first transport module plug-flow to streaming media server end.

Step 6：Image cache submodule preserves the multichannel image data in some time in multimedia process module, and Preserve the mapping relations between the multichannel image data formed according to corresponding time relationship and the panoramic image sequence of tape label.

For example, setting frame per second is 25, the panoramic image sequence of tape label is represented with Irn, multichannel image data Drn= { D1, D2 ..., Dm } is represented, wherein m represents the quantity of camera module, and it is positive integer, and Drn is represented in r-th of frame sequence Multichannel image data corresponding to n-th frame image, then the mapping between multichannel image data and tape label panoramic image sequence Relation can be expressed as Irn → Drn.

Step 7：Picture is asked to refer to when picture output sub-module receives streaming media server end by the first transport module When making, image cache submodule is searched according to multichannel image data and mapping relations Irn → Drn of tape label panoramic image sequence The multichannel image data Drn of middle corresponding present frame.And taken by the first transport module output multi-channel image data Drn to Streaming Media Business device end.

User management module is used to receive the ID authentication request from each user equipment, and please based on the authentication Ask and authentication is carried out to corresponding user equipment, send authentication success message to the corresponding user equipment after the authentication has been successful It is connected with establishing data with the user equipment.Video processing module is transmitted by the second transport module real-time reception IMAQ end Video flowing, and by video flowing according to it is certain rule distribute；Video flowing re-compression and encoding can also be differentiated into some differences The video flowing of rate, and video flowing is distributed according to certain rule.Virtual photograph module is used to handle what user terminal requests were taken pictures Instruction.

The step of streaming media server end processing user terminal requests are taken pictures：

At least one request for carrying out user terminal one or more user equipment bat is received when streaming media server terminates According to instruction when, (step 101) first according to user equipment send request photographing instruction in include video requency frame data lookup Present frame, method：Video requency frame data includes one-dimension code or two-dimentional code watermark, decodes one-dimension code or Quick Response Code and is searched according to Irn and is worked as Previous frame；(step 102) and then present frame Irn is returned to IMAQ end by the second transport module and asks multichannel picture Drn； (step 103) uses accurate stitching algorithm two to complete panoramic mosaic after receiving the multichannel picture Drn of IMAQ end transmission； (step 104) then according to the transmission posture of user terminal cut in the panoramic pictures to should posture picture；(step 105) finally the picture cut is saved in user management module under the user account, for final pre- of user terminal Look at and pay download.

Cutting picture step in step 104 is as follows：

The panoramic pictures (planar rectangular picture) are remapped on a sphere by (step 201), i.e., with planar rectangular The side of picture is mapped to panorama mapping as panorama mapping sphere corresponding to the generation of sphere semi arch, and by planar rectangular picture On sphere；(step 202) determines the angular range in the user visual field according to the transmission posture of user equipment；(step 203) according to The angular range in the family visual field searches spherical figure to be cut on panorama mapping sphere；This has been cut spherical diagram by (step 204) Shape is mapped to target rectangle picture, and the size of the target rectangle picture is preset value, and the preset value can be set based on experience value, Or it can be set according to the screen size parameter included in the photographing instruction of user equipment transmission.

It should be noted that the second transport module can be a transport module or multiple, can use wired Transmission, it can also use and be wirelessly transferred.Fast stitching algorithm one and accurate stitching algorithm two, which are distinguished, to be, Fast stitching algorithm one Algorithm need real-time complete panoramic pictures splicing, to meet the fluency of video playback；And accurate stitching algorithm two Algorithm primarily focuses on the panoramic pictures splicing of accurate seamless high quality, to meet user terminal in video display process is watched Obtain the demand of high-resolution pictures.

User terminal workflow：

Step 1：User terminal sends ID authentication request to streaming media server end, and comes from Streaming Media receiving Data are established after the authentication success message of server end with streaming media server end to be connected.

Step 2：Viewing video playback request is sent to streaming media server end after user terminal access system, in real time solution The video data that the distribution of code stream media server end comes, selects the watching mode of different resolution to be broadcast to watch panoramic video Put.

Step 3：User terminal can send request of taking pictures during panoramic video is watched, to obtain high resolution graphics Piece.

Step 4：User terminal can send preview request, the photo that preview oneself is clapped；Payment download can also be sent please Ask, the photo that oneself is clapped is downloaded to local.

Wherein, user terminal is sent by user interactive module and asked.Wrapped in the request of taking pictures of subscriber terminal equipment transmission Containing video requency frame data and posture is sent, posture is sent and refers to posture of the user equipment when sending the request, can be that user sets The screen normal direction or its screen back side normal direction of standby middle display module；Can by user by way of contact action come Set and send posture, the postures such as gyroscope can also integrate in user interactive module and sense submodule, user is obtained with this The current pose of equipment.

To sum up, in accordance with an embodiment of the present disclosure, present disclose provides following scheme, but not limited to this：

Scheme 1, the panoramic video based on two-way communication play camera system, it is characterised in that including IMAQ end 1, stream Media server end 2 and one or more user terminals 3；Described image collection terminal 1 includes photographing module 11, multi-media processing mould The transport module 13 of block 12 and first；

The photographing module 11 is suitable to gather the multichannel image data in its visual field；

The multimedia process module 12 is suitable to：

The multichannel image data of the photographing module 11 collection is carried out in real time first panoramic mosaic to form panorama sketch As sequence, the mapping relations formed according to corresponding time relationship between multichannel image data and panoramic image sequence, and preserve institute Mapping relations and at least part multichannel image data are stated,

Coding is compressed to the panoramic image sequence, obtains the video flowing after compressed encoding, and by after compressed encoding Video flowing by the plug-flow of the first transport module 13 to streaming media server end, wherein, first transport module uses Duplex mode of operation, and

When receiving the instruction of the request picture from streaming media server end 2 by first transport module 13, base The lookup object information that includes in the request picture instruction, according to reflecting for the multichannel image data and panoramic image sequence Relation is penetrated, multichannel picture corresponding with the lookup object information is searched at least part multichannel image data of preservation Data, the streaming media server end 2 is sent to by first transport module 13 as lookup result；Wherein, it is described Searching object information includes the relevant information of panoramic image frame to be found；

The streaming media server end 2 is suitable to from the video after compressed encoding described in the real-time reception of described image collection terminal 1 Stream is to be handed down to corresponding user terminal 3, after receiving since the lookup result of described image collection terminal 1, to the lookup knot Multichannel image data in fruit carries out secondary panoramic mosaic, obtains secondary panoramic mosaic image, and by the secondary panoramic mosaic figure The partly or wholly image of picture is handed down to corresponding user terminal 3 as output result.

Scheme 2, the panoramic video according to scheme 1 play camera system, it is characterised in that the streaming media server End 2 includes the second transport module 24,

Second transport module 24 is suitable to the general image of the secondary panoramic mosaic image being sent to corresponding use Family end 3.

Scheme 3, the panoramic video according to scheme 1 play camera system, it is characterised in that the streaming media server End 2 includes virtual photograph module 22,

The virtual photograph module 22 is adapted to determine that the transmission posture of the user terminal 3, with the hair according to the user terminal 3 Send posture cut out in the secondary panoramic mosaic image to should posture topography, be handed down to institute as output result State user terminal 3.

Scheme 4, the panoramic video according to any one of scheme 1-3 play camera system, it is characterised in that described more Medium process module 12 includes picture splicing submodule 122,

The picture splicing submodule 122 is suitable to the multichannel figure gathered using Fast stitching algorithm to the photographing module 11 Sheet data carries out panoramic mosaic first, to form panoramic image sequence.

Scheme 5, the panoramic video according to scheme 3 play camera system, it is characterised in that the virtual photograph module 22 are suitable to carry out secondary panoramic mosaic to the multichannel image data in the lookup result using accurate stitching algorithm, to obtain two Secondary panoramic mosaic image.

Scheme 6, the panoramic video according to any one of scheme 1-5 play camera system, it is characterised in that described more At least part multichannel image data that medium process module 12 preserves is the interior multichannel picture preserved of nearest preset time period Data.

Scheme 7, the panoramic video according to any one of scheme 1-6 play camera system, it is characterised in that described to take the photograph Picture module 11 includes multiple camera modules, and the multiple camera module is suitable to based on institute's visual angle figure corresponding to the acquisition of multiple visual angles Piece, as the multichannel image data.

Scheme 8, the panoramic video according to scheme 7 play camera system, it is characterised in that

The panoramic image sequence includes multiple frame sequences, and Ir represents r-th of frame sequence in the plurality of frame sequence, and r is Positive integer, Irn represent the n-th frame image in r-th of frame sequence；

Multichannel image data is expressed as Drn={ D₁, D₂..., D_m, wherein m represents the quantity of camera module, and it is just Integer, Drn represent the multichannel image data corresponding to the n-th frame image in r-th of frame sequence, multichannel image data and panorama sketch As the mapping relations between sequence are expressed as Irn → Drn.

Scheme 9, the panoramic video according to scheme 7 or 8 play camera system, it is characterised in that the multiple shooting Module includes at least one of fish-eye camera and wide-angle camera.

Scheme 10, the panoramic video according to scheme 4 play camera system, it is characterised in that picture splicing The panoramic image sequence that module 122 is formed is the panoramic image sequence of tape label.

Scheme 11, the panoramic video according to scheme 10 play camera system, it is characterised in that picture splicing Module 122 is suitable to：

Frame per second f is set, video flowing is divided into according to the time sequencing of broadcasting more using every f frames as a frame sequence, by video Individual frame sequence, r-th of frame sequence being represented with Ir, r is positive integer, r=1,2 ...；With Ir1, Ir2 ... Irf represents r F two field pictures in individual frame sequence, parameter is generated using Iri as the watermark of the i-th two field picture in r-th of frame sequence, wherein, i ∈ { 1,2 ..., f }, parameter Iri is generated according to watermark and obtains corresponding mark, the mark is added on corresponding picture frame, obtained Each two field picture of tape label is obtained, to form the panoramic image sequence of tape label.

Scheme 12, the panoramic video according to any one of scheme 1-11 play camera system, it is characterised in that described Streaming media server end 2 includes user management module 23,

The user management module 23 is suitable to receive the ID authentication request from each user terminal, and is recognized based on the identity Card request carries out authentication to corresponding user equipment, sends certification success to the corresponding user equipment after the authentication has been successful Information is connected with establishing data with the user equipment.

Scheme 13, the panoramic video according to any one of scheme 1-12 play camera system, it is characterised in that described Streaming media server end 2 includes video processing module 21

The video processing module 21 is suitable to pass by the real-time reception described image collection terminal 1 of the second transport module 24 Video flowing after defeated compressed encoding, video flowing to be issued is obtained based on the video flowing after the compressed encoding to be issued to user End 3.

Scheme 14, the panoramic video according to scheme 13 play camera system, it is characterised in that the Video processing mould Block 21 is suitable to video flowing to be issued is issued into user terminal 3 according to predetermined way.

Scheme 15, the panoramic video according to scheme 14 play camera system, it is characterised in that the Video processing mould Block 21 is suitable to the video flowing re-compression and encoding after the compressed encoding into multiple video flowings, by the plurality of video flowing extremely Small part is sent to corresponding user 3, wherein, the resolution ratio between the plurality of video flowing is different.

Scheme 16, the panoramic video according to scheme 5 play camera system, it is characterised in that the virtual photograph mould Block 22 is suitable to the request photographing instruction for receiving user terminal 3, and performs alignment processing.

Scheme 17, the panoramic video according to scheme 16 play camera system, it is characterised in that the virtual photograph mould Block 22 is suitable to

When receiving at least one request photographing instruction from one or more user terminals 3, clapped for each request According to instruction,

Video requency frame data according to being included in the request photographing instruction determines the relevant information of panoramic image frame to be found, And the request picture instruction for the relevant information for including the panoramic image frame to be found is generated,

The request picture is sent by second transport module 24 to described image collection terminal 1 to instruct, and from the figure The multichannel image data that its instructs and sent back for the request picture as the reception of collection terminal 1, and use accurate stitching algorithm pair The multichannel image data carries out secondary panoramic mosaic,

Cut according to transmission posture contained in the request photographing instruction in the obtained secondary panoramic mosaic image pair Corresponding user terminal 3 should be handed down to by second transport module 24 as output result in the picture of the transmission posture.

Scheme 18, the panoramic video according to scheme 17 play camera system, it is characterised in that the virtual photograph mould Block 22 is suitable to the phase that corresponding panoramic image frame to be found is obtained by decoding the watermark included in the video requency frame data Close information.

Scheme 19, panoramic video according to scheme 18 play camera system, it is characterised in that described to be found complete The relevant information of scape picture frame is expressed as Iri, wherein, i is the positive integer in 1~m.

Scheme 20, the panoramic video according to scheme 18 or 19 play camera system, it is characterised in that the watermark is adopted With one-dimensional code watermark or two-dimentional code watermark.

Scheme 21, the panoramic video according to any one of scheme 17-20 play camera system, it is characterised in that institute Virtual photograph module 22 is stated to be suitable to obtained output result being stored in the user management module 23 to correspond under user account, So that user terminal preview and/or payment are downloaded.

Scheme 22, the panoramic video according to any one of scheme 17-21 play camera system, it is characterised in that institute Virtual photograph module 22 is stated to be suitable to：

The secondary panoramic mosaic image is remapped on a sphere, the sphere maps sphere as panorama,

The angular range in the user visual field is determined according to the transmission posture of the equipment of user terminal 3,

Spherical figure to be cut is determined on panorama mapping sphere according to the angular range in the user visual field, and cuts out lower be somebody's turn to do Target rectangle picture is mapped to after spherical figure is cut, wherein, the size of the target rectangle picture is preset value.

Scheme 23, the panoramic video according to scheme 22 play camera system, it is characterised in that the virtual photograph mould Block 22 is suitable to：

Generated using the side of planar rectangular picture corresponding to the secondary panoramic mosaic image as sphere semi arch pair The panorama mapping sphere answered, and the planar rectangular picture is mapped on panorama mapping sphere, to complete the secondary panorama Stitching image is mapped on the sphere.

Scheme 24, the panoramic video according to scheme 2 or 3 play camera system, it is characterised in that first transmission Each transport module in module 13, second transport module 24 and the 3rd transport module 34 include respectively one or Multiple transmission assemblies.

Scheme 25, the panoramic video according to scheme 24 play camera system, it is characterised in that the first transmission mould Each transport module in block 13, second transport module 24 and the 3rd transport module 34 uses wire transmission and nothing At least one of line transmission transmission means.

Scheme 26, the panoramic video according to any one of scheme 1-25 play camera system, it is characterised in that described User terminal 3 is suitable to play the transmission of streaming media server end 2 ID authentication request in camera system to the panoramic video, and Data connection is established therewith after receiving the authentication success message from streaming media server end 2.

Scheme 27, the panoramic video according to scheme 26 play camera system, it is characterised in that the user terminal 3 is suitable After data connection is established at streaming media server end 2 described in Yu Yu, sending viewing video playback to streaming media server end 2 please Ask, and the video data that the distribution of streaming media server end 2 described in real-time decoding comes.

Scheme 28, the panoramic video according to any one of scheme 1-27 play camera system, it is characterised in that described User terminal 3 is suitable to select one kind in the watching mode of one or more resolution ratio according to user's input to carry out panoramic video Play.

Scheme 29, the panoramic video according to any one of scheme 1-28 play camera system, it is characterised in that described User terminal 3 is suitable to send request of taking pictures according to user's input in panoramic video playing process, to obtain high-resolution pictures.

Scheme 30, the panoramic video according to any one of scheme 1-29 play camera system, it is characterised in that described User terminal 3 is suitable to input according to user and send preview request to streaming media server end 2, permits to streaming media server end 2 Perhaps during 3 preview of user terminal, photo corresponding to preview.

Scheme 31, the panoramic video according to any one of scheme 1-30 play camera system, it is characterised in that described User terminal 3 is suitable to input according to user and send payment download request to streaming media server end 2, to streaming media server end 2 allow the user terminal 3 download when, corresponding photo is downloaded to local.

Scheme 32, the panoramic video according to scheme 29 play camera system, it is characterised in that the user terminal 3 is sent out The transmission posture for including equipment corresponding to video requency frame data and the user terminal 3 in the request of taking pictures sent.

Scheme 33, the panoramic video according to scheme 32 play camera system, it is characterised in that the user terminal 3 is suitable In it is determined that to send take pictures request when, obtain the screen normal direction or its screen of display module in the corresponding device of user terminal 3 Curtain back side normal direction, is used as it and sends posture.

Scheme 34, the panoramic video according to scheme 32 or 33 play camera system, it is characterised in that the user terminal 3 are suitable to set transmission posture by detecting the touch operation of corresponding device, or by the attitude transducer in corresponding device come Obtain and send posture.

Scheme 35, the panoramic video according to any one of scheme 1-34 play camera system, it is characterised in that described The user interactive module for being used for sending various requests is provided with user terminal 3.

Scheme 36, the panoramic video according to any one of scheme 1-35 play camera system, it is characterised in that panorama It is live that video playback camera system is used for panoramic video.

Finally, it is to be noted that, in the disclosure, such as left and right, first and second or the like relational terms It is used merely to make a distinction an entity or operation with another entity or operation, and not necessarily requires or imply these Any this actual relation or order be present between entity or operation.Moreover, term " comprising ", "comprising" or its is any Other variants are intended to including for nonexcludability, so that process, method, article or equipment including a series of elements Not only include those key elements, but also the other element including being not expressly set out, or also include for this process, side Method, article or the intrinsic key element of equipment.In the absence of more restrictions, limited by sentence " including one ... " Key element, it is not excluded that other identical element in the process including the key element, method, article or equipment also be present.

Although being had been disclosed above by the description of the specific embodiment of the disclosure to the disclosure, however, it should Understand, those skilled in the art can design various modifications, the improvement to the disclosure in the spirit and scope of the appended claims Or equivalent.These modifications, improvement or equivalent should also be as being to be considered as included in the claimed scope of the disclosure.

Claims

1. the panoramic video based on two-way communication plays camera system, it is characterised in that is taken including IMAQ end (1), Streaming Media Business device end (2) and one or more user terminals (3)；Described image collection terminal (1) includes photographing module (11), multi-media processing Module (12) and the first transport module (13)；

The photographing module (11) is suitable to gather the multichannel image data in its visual field；

The multimedia process module (12) is suitable to：

The multichannel image data of the photographing module (11) collection is carried out in real time first panoramic mosaic to form panoramic picture Sequence, the mapping relations formed according to corresponding time relationship between multichannel image data and panoramic image sequence, and described in preservation Mapping relations and at least part multichannel image data,

Coding is compressed to the panoramic image sequence, obtains the video flowing after compressed encoding, and by regarding after compressed encoding Frequency stream passes through the first transport module (13) plug-flow to streaming media server end, wherein, first transport module is using double Work mode of operation, and

When receiving the instruction of the request picture from streaming media server end (2) by first transport module (13), base The lookup object information that includes in the request picture instruction, according to reflecting for the multichannel image data and panoramic image sequence Relation is penetrated, multichannel picture corresponding with the lookup object information is searched at least part multichannel image data of preservation Data, the streaming media server end (2) is sent to by first transport module (13) as lookup result；Wherein, The object information of searching includes the relevant information of panoramic image frame to be found；

The streaming media server end (2) is suitable to from the video after compressed encoding described in described image collection terminal (1) real-time reception Stream after receiving since the lookup result of described image collection terminal (1), is looked into this with being handed down to corresponding user terminal (3) The multichannel image data looked in result carries out secondary panoramic mosaic, obtains secondary panoramic mosaic image, and the secondary panorama is spelled The partly or wholly image of map interlinking picture is handed down to corresponding user terminal (3) as output result.

2. panoramic video according to claim 1 plays camera system, it is characterised in that the streaming media server end (2) virtual photograph module (22) is included,

The virtual photograph module (22) is adapted to determine that the transmission posture of the user terminal, with the transmission appearance according to the user terminal State cut out in the secondary panoramic mosaic image to should posture topography, be handed down to the use as output result Family end.

3. panoramic video according to claim 2 plays camera system, it is characterised in that the virtual photograph module (22) It is secondary to obtain suitable for carrying out secondary panoramic mosaic to the multichannel image data in the lookup result using accurate stitching algorithm Panoramic mosaic image.

4. panoramic video according to claim 1 or 2 plays camera system, it is characterised in that

The panoramic image sequence includes multiple frame sequences, and Ir represents r-th of frame sequence in the plurality of frame sequence, and r is just whole Number, Irn represent the n-th frame image in r-th of frame sequence；

Multichannel image data is expressed as Drn={ D₁, D₂..., D_m, wherein m represents the quantity of camera module, and it is just whole Number, Drn represent the multichannel image data corresponding to the n-th frame image in r-th of frame sequence, multichannel image data and panoramic picture Mapping relations between sequence are expressed as Irn → Drn.

5. panoramic video according to claim 4 plays camera system, it is characterised in that the multimedia process module (12) picture splicing submodule (122) is included, the picture splicing submodule (122) is suitable to using Fast stitching algorithm to described The multichannel image data of photographing module (11) collection carries out panoramic mosaic first, to form panoramic image sequence.

6. panoramic video according to claim 5 plays camera system, it is characterised in that the panoramic image sequence is band The panoramic image sequence of mark；

The Video coding submodule (125) is suitable to：

Frame per second f is set, video flowing is divided into multiple frames using every f frames as a frame sequence, by video according to the time sequencing of broadcasting Sequence, r-th of frame sequence being represented with Ir, r is positive integer, r=1,2 ...；With Ir1, Ir2 ... Irf represents r-th of frame F two field pictures in sequence, parameter is generated using Iri as the watermark of the i-th two field picture in r-th of frame sequence, wherein, i ∈ 1, 2 ..., f }, parameter Iri is generated according to watermark and obtains corresponding mark, the mark is added on corresponding picture frame, obtained Each two field picture of tape label, to form the panoramic image sequence of tape label.

7. panoramic video according to claim 3 plays camera system, it is characterised in that the virtual photograph module (22) It is suitable to：

When receiving at least one request photographing instruction from one or more user terminals, finger of taking pictures is asked for each Order,

Video requency frame data according to being included in the request photographing instruction determines the relevant information of panoramic image frame to be found, and raw Request picture into the relevant information for including the panoramic image frame to be found instructs,

The request picture is sent by second transport module (24) to described image collection terminal (1) to instruct, and from the figure Picture collection terminal (1) receives its multichannel image data for instructing and sending back for the request picture, and uses accurate stitching algorithm Secondary panoramic mosaic is carried out to the multichannel image data,

Cut and corresponded in the obtained secondary panoramic mosaic image according to transmission posture contained in the request photographing instruction The picture of the transmission posture, corresponding user terminal is handed down to by second transport module (24) as output result.

8. panoramic video according to claim 7 plays camera system, it is characterised in that the virtual photograph module (22) It is suitable to：

The angular range in the user visual field is determined according to the transmission posture of the equipment of user terminal,

Spherical figure to be cut is determined on panorama mapping sphere according to the angular range in the user visual field, and this is to be cut under sanction Target rectangle picture is mapped to after cutting spherical figure, wherein, the size of the target rectangle picture is preset value.

9. panoramic video according to claim 1 or 2 plays camera system, it is characterised in that the user terminal (3) sends Request of taking pictures in include the transmission posture of equipment corresponding to video requency frame data and the user terminal (3).

10. system according to claim 1 or 2, it is characterised in that the user terminal 3 be suitable to according to user's input come One kind is selected to carry out panoramic video broadcasting in the watching mode of one or more resolution ratio.