CN108810567A

CN108810567A - A kind of matched method in audio & video visual angle, client and server

Info

Publication number: CN108810567A
Application number: CN201710289042.5A
Authority: CN
Inventors: 高莹; 顾迎节; 张尧烨
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2017-04-27
Filing date: 2017-04-27
Publication date: 2018-11-13
Anticipated expiration: 2037-04-27
Also published as: CN108810567B

Abstract

This application discloses a kind of matched method in audio & video visual angle, client and servers, to solve client present in the scheme of existing client terminal playing panoramic video when current visual angle changes, the problem of matching audio file can not be selected to play out, lead to poor user experience.This method is that user end to server sends the first request message of mark for carrying the MPD file of the MPD file for obtaining panoramic video；Receive the MPD file of the server according to the identification feedback of the MPD file, the MPD file includes the mark of at least one audio fragment and its corresponding spatial description information, and the audio space description information is used to describe the associated region of at least one of described MPD file audio fragment；According to the current visual angle range of user and at least one audio space description information, the first audio fragment with the current visual angle commensurate in scope is determined.

Description

A kind of matched method in audio & video visual angle, client and server

Technical field

This application involves a kind of matched method of multimedia technology field more particularly to audio & video visual angle, clients And server.

Background technology

Panoramic video is also referred to as 360 degree of panoramic videos, is to carry out 360 degree entirely to surrounding by the camera positioned at center Scape is shot, and by technologies such as synchronization, splicing, projections, and the image of multiple angle shots is synthesized panoramic picture, and by multiple frames Panoramic picture form panoramic video.

User can arbitrarily change the angle of viewing up and down when watching panoramic video, obtain preferably experience.Panorama Video and the one very big difference of traditional ordinary video are：What user watched at a certain moment is not complete video pictures, only It is a part of region of complete video picture.Usually the content of the currently practical viewing of user residing for panoramic video coordinate system Region is known as current visual angle, and the video pictures that user watches in current visual angle are known as video visual angle in the application.User sees By sliding screen or rotation head (helmet) when seeing, different video visual angles is watched to convert current visual angle.

In current panoramic video application, it is different with the change of user's current visual angle to only considered video visual angle, and Other media component such as audio, subtitle are not accounted for.And in application scenes, when user's current visual angle changes, If it will be that user brings better viewing experience that audio can synchronize matching with video visual angle.For example, when we watch Such as《Father go where》Etc. entertainments when, when multigroup family gathers together, if user's current visual angle be family 1, Indicate that user is interested in family 1, matching at this time can be the relevant audio of 1 member of family.And when user works as When preceding visual angle is switched to family 2, matching should be the relevant audio of 2 member of family.When the family that user does not pay special attention to When in front yard or video pictures including multiple families, matching can be default audio, and still, current panoramic video is answered In, when the current video visual angle of user changes, matching audio file can not be selected to play out, cause to use Family experience is poor.

Invention content

A kind of matched method in audio & video visual angle of the embodiment of the present application offer, client and server, it is existing to solve Client present in the scheme of some client terminal playing panoramic videos can not select therewith when current visual angle changes The problem of audio file matched plays out, leads to poor user experience.

Specific technical solution provided by the embodiments of the present application is as follows：

In a first aspect, the embodiment of the present application provides a kind of matched method in audio & video visual angle, including：

Server receives the first request that the display advertising for obtaining panoramic video that client is sent describes MPD file Message carries the mark of the MPD file in first request message；

The server returns to the MPD file, the MPD texts according to the mark of the MPD file to the client Part includes the mark of at least one audio fragment and its corresponding audio space description information, the audio space description information Associated region for describing at least one audio fragment.

Using the above method, user end to server acquisition request includes the mark and its corresponding audio sky of audio fragment Between description information MPD file so that client can current visual angle range determination after, according to audio space description information meter Calculate associated region of each audio in full-view video image.When some corresponding associated region of audio fragment and user are current When angular field of view matches, so that client is got, precisely matched audio file plays out with video image, to realize The simultaneously match of audio & video image promotes the viewing experience of user.It can be existing to solve by the embodiment of the present application Client terminal playing panoramic video scheme present in client when current visual angle changes, can not select matching Audio file the problem of playing out, leading to poor user experience.

With reference to first aspect, in a kind of possible design, further include in the MPD file in the MPD file at least The Region Matching condition of one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

In this design, when the MPD file includes Region Matching condition, the associated region when audio fragment and user When meeting Region Matching condition between current visual angle range, that is, thinks the audio fragment and to your money visual angle be mostly matched. When MPD file includes Multi-audio-frequency matching strategy, when there are the associated regions of at least two audio fragments and user to work as forward sight When meeting Region Matching condition between angular region, determined and the audio of current visual angle commensurate in scope point according to Multi-audio-frequency matching strategy Piece provides more flexible video matching effect to the user.

With reference to first aspect, in a kind of possible design, the method further includes：

The server receives the second request message for obtaining video slicing that the client is sent, and described second The mark of the video slicing is carried in request message；

The server sends the video slicing according to the mark of the video slicing to the client.

The server receives the acquisition that is used for that the client is sent and divides with matched first audio of the video slicing The third request message of piece carries the mark of the first audio fragment in the third request message；

The server sends first audio point according to the mark of the first audio fragment to the client Piece.

Second aspect, the embodiment of the present application provide a kind of matched method in audio & video visual angle, including：

The first request that user end to server transmission describes MPD file for obtaining the display advertising of panoramic video disappears It ceases, the mark of the MPD file is carried in first request message；

The client receives the MPD file of the server according to the identification feedback of the MPD file, described MPD file includes the mark of at least one audio fragment and its corresponding spatial description information, the audio space description letter Cease the associated region for describing at least one of described MPD file audio fragment；

Current visual angle range and at least one audio space description information of the client according to user, determine with First audio fragment of the current visual angle commensurate in scope.

In the above method, user end to server acquisition request includes the mark of audio fragment and its corresponding audio space The MPD file of description information so that client can calculate after the determination of current visual angle range according to audio space description information Go out associated region of each audio in full-view video image.When some corresponding associated region of audio fragment and user work as forward sight When angular region matches, so that client is got, precisely matched audio file plays out with video image, to realize sound The simultaneously match of frequency and video image, promotes the viewing experience of user.It can be existing to solve by the embodiment of the present application Client present in the scheme of client terminal playing panoramic video can not select matching when current visual angle changes Audio file plays out, the problem of leading to poor user experience.

In conjunction with second aspect, in a kind of possible design, further include in the MPD file in the MPD file at least The Region Matching condition of one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

In conjunction with second aspect, in a kind of possible design, the client is according to the current visual angle range of user and described At least one audio space description information determines the first audio fragment with the current visual angle commensurate in scope, including：

The client obtains at least one of described MPD file according at least one audio space description information Audio fragment is at least one of panoramic video associated region；

The client by least one associated region with the association area that matches within the scope of the current visual angle The corresponding audio fragment in domain is determined as alternative audio fragment；

If only exist an alternative audio fragment, the alternative audio fragment is determined as the first audio fragment；

If there are when at least two alternative audio fragments, according to the matching strategy of the Multi-audio-frequency fragment, the first sound is determined Frequency division piece；

If there is no when alternative audio fragment, the default audio fragment of pre-configuration is set to the first audio fragment.

In this design, by the way that Multi-audio-frequency matching strategy is arranged in MPD file, when multiple associated regions and user are current When angular field of view matches, according to Multi-audio-frequency matching strategy, client can select best audio to carry out matching broadcasting.

In conjunction with second aspect, in a kind of possible design, at least one associated region with the current visual angle model The associated region to match in enclosing is associated region identical with the current visual angle range；Or,

Meet the associated region of the Region Matching condition with the current visual angle range.

In this design, it is associated with what is matched within the scope of the current visual angle at least one associated region Region is arranged different conditions, user can specifically determine according to actual needs at least one associated region whether with work as Preceding angular field of view matching, mode is flexible, improves user experience.

It is described to meet the Region Matching with the current visual angle range in a kind of possible design in conjunction with second aspect The associated region of condition, including：

Fall into the associated region of the current visual angle range；Or,

With the current visual angle range associated region that match degree is greater than the preset threshold.

In this design, by the way that the Region Matching condition of audio fragment is arranged in MPD file, the pass of audio may be implemented Join condition coupling different between region and user's current visual angle, to provide between more flexible audio & video image With effect, further,

In conjunction with second aspect, in a kind of possible design, the method further includes：

At least one audio fragment that the MPD file includes is downloaded to the client local by the client, The client according to user current visual angle range and at least one audio space description information, determination work as with described After the preceding matched first audio fragment of angular field of view, described first is obtained from being downloaded in local at least one audio fragment Audio fragment is decoded broadcasting.

In this design, due to audio fragment data amount and little, multiple audios are downloaded to local by client in advance, Locally the audio fragment is being directly acquired after determining the audio fragment that the region with the current visual angle range of user matches in the middle It is decoded broadcasting, improves the acquisition efficiency of audio, further increases matching efficiency, promotes user experience.

The third aspect, the embodiment of the present application provide a kind of server, including：

Receiving unit, the display advertising for obtaining panoramic video for receiving client transmission describe MPD file First request message carries the mark of the MPD file in first request message；

Processing unit returns to the MPD file for the mark according to the MPD file to the client, described MPD file includes that the mark of at least one audio fragment and its corresponding audio space description information, the audio space are retouched State associated region of the information for describing at least one audio fragment.

In conjunction with the third aspect, in a kind of possible design, further include in the MPD file in the MPD file at least The Region Matching condition of one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

In conjunction with the third aspect, in a kind of possible design, the server further includes transmission unit,

The receiving unit, the second request for obtaining video slicing for being additionally operable to receive the client transmission disappear It ceases, the mark of the video slicing is carried in second request message；

The transmission unit sends the video slicing for the mark according to the video slicing to the client.

In conjunction with the third aspect, in a kind of possible design, the receiving unit is additionally operable to receive what the client was sent For obtain with the third request message of the matched first audio fragment of the video slicing, carry in the third request message There is the mark of the first audio fragment；

The transmission unit is additionally operable to the mark according to the first audio fragment, and described the is sent to the client One audio fragment.

Fourth aspect, the embodiment of the present application provide a kind of client, including：

Transmission unit, for sending describe MPD file for obtaining the display advertising of panoramic video first to server Request message carries the mark of the MPD file in first request message；

Receiving unit, the MPD file for receiving the server according to the identification feedback of the MPD file, institute State the mark and its corresponding spatial description information that MPD file includes at least one audio fragment, the audio space description Information is used to describe the associated region of at least one of described MPD file audio fragment；

Processing unit is used for the current visual angle range according to user and at least one audio space description information, really Fixed the first audio fragment with the current visual angle commensurate in scope.

In conjunction with fourth aspect, in a kind of possible design, further include in the MPD file in the MPD file at least The Region Matching condition of one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

In conjunction with fourth aspect, in a kind of possible design, the processing unit according to the current visual angle range of user and At least one audio space description information, when determining the first audio fragment with the current visual angle commensurate in scope, specifically For：

At least one of described MPD file audio fragment is obtained according at least one audio space description information to exist At least one of panoramic video associated region；

By sound corresponding with the associated region to match within the scope of the current visual angle at least one associated region Frequency division piece is determined as alternative audio fragment；

In conjunction with fourth aspect, in a kind of possible design, at least one associated region with the current visual angle model The associated region to match in enclosing is associated region identical with the current visual angle range；Or,

It is described to meet the Region Matching with the current visual angle range in a kind of possible design in conjunction with fourth aspect The associated region of condition, including：

Fall into the associated region of the current visual angle range；Or,

In conjunction with fourth aspect, in a kind of possible design, the processing unit is additionally operable to：

At least one audio fragment that the MPD file includes is downloaded to the client local, the client In the current visual angle range and at least one audio space description information, determination and the current visual angle range according to user After matched first audio fragment, from be downloaded in local at least one audio fragment obtain the first audio fragment into Row decoding plays.

5th aspect, a kind of server provided by the embodiments of the present application, including memory, processor and communication interface； Wherein,

The memory is for storing computer-readable program；

The processor by running the program in the memory, with complete it is any in first aspect and first aspect can The method that the realization method of energy provides；

The communication interface under the control of the processor for sending and receiving data.

6th aspect, a kind of client provided by the embodiments of the present application, including memory, processor and communication interface； Wherein,

The memory is for storing computer-readable program；

The processor by running the program in the memory, with complete it is any in second aspect and second aspect can The method that the realization method of energy provides；

7th aspect, the embodiment of the present application provide a kind of computer storage media, and the storage medium is computer-readable Storage medium, it includes instruction that the computer-readable recording medium storage, which has program, program, and described instruction is when by with processor The network equipment so that the network equipment is executed what each possible realization method of above-mentioned first aspect and one side provided Method.

Eighth aspect, the embodiment of the present application provide a kind of computer storage media, and the storage medium is computer-readable Storage medium, it includes instruction that the computer-readable recording medium storage, which has program, program, and described instruction is when by with processor Electronic equipment each possible realization method for making the electronic equipment execute above-mentioned second aspect and second aspect when executing provide Method.

Description of the drawings

Fig. 1 is a kind of network architecture schematic diagram provided by the embodiments of the present application；

Fig. 2 is the content structure schematic diagram of MPD file in the prior art；

Fig. 3 A are the video schematic diagram of full width transmission mode；

Fig. 3 B are the video schematic diagram of block transmission mode；

Fig. 4 is that video pictures switch schematic diagram in the prior art；

Fig. 5 is a kind of structural schematic diagram of server provided by the embodiments of the present application；

Fig. 6 is a kind of structural schematic diagram of client provided by the embodiments of the present application；

Fig. 7 is a kind of matched method flow schematic diagram in audio & video visual angle provided by the embodiments of the present application；

Fig. 8 is the matched method flow schematic diagram in another audio & video visual angle provided by the embodiments of the present application；

Fig. 9 A, Fig. 9 B and Fig. 9 C are the matching process schematic diagram of the associated region and current visual angle of audio fragment；

Figure 10 A, Figure 10 B and the quantity that Figure 10 C are associated region are more than schematic diagram at one；

Figure 11 is the structural schematic diagram of another server provided by the embodiments of the present application；

Figure 12 is the structural schematic diagram of another client provided by the embodiments of the present application.

Specific implementation mode

Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete Site preparation describes.

Wherein, method and apparatus be based on same inventive concept, since the principle that method and device solves the problems, such as is similar, Therefore the implementation of apparatus and method can be with cross-reference, and overlaps will not be repeated.

The network architecture that technical solution provided by the embodiments of the present application is related to is as shown in Figure 1, include server 101 and client End 102.Server is corresponding with client, provides the program of local service to the user, the invention relates to client Have the function of for user's playing panoramic video, panoramic video player is run in client, which can be mounted in visitor An application on the end of family, can also be a page on browser.Client can be wireless terminal device, can also be Line termination unit.Wireless terminal device can be had the portable equipment of wireless connecting function or be connected to wireless-modulated Other processing equipments of demodulator.Wireless terminal device can through wireless access network (Radio Access Network, RAN) with One or more core nets are communicated, and wireless terminal device can be mobile terminal device, as mobile phone (or be " bee Nest " phone) and computer with mobile terminal device, for example, it may be portable, pocket, hand-held, built-in computer Or vehicle-mounted mobile device, they exchange language and/or data with wireless access network.Line termination unit can be wired Television set, hard wired computer etc..Server refers to the equipment for providing the service of calculating, server can the service at customer in response end ask Ask, server has the function of undertaking service and ensures service, the invention relates to server have be client The function of panoramic video is provided.The composition of server is similar with general computer architecture, generally include to blow afloat, hard disk, memory, System bus etc., example more demanding in processing capacity, reliability, stability, safety, scalability, manageability etc. Such as, server can be PC (Personal Computer, PC) server.Communication between client and server Support the media transmission protocol of general panoramic video, such as real-time transport protocol (Real-Time Protocol, RTP), reality When stream protocol (Real-Time Streaming Protocol, RTSP), hypertext transfer protocol (HyperText Transfer Protocol, HTTP), HTTP dynamic self-adaptings stream (Dynamic Adaptive Streaming over HTTP, DASH) matchmaker Body agreement, HTTP live TV streams stream (HTTP Live Streaming, HLS) media protocol etc..

The invention relates to server and client side can be based on DASH technologies, other technologies can also be based on. For being based on DASH technologies, DASH technologies use different HTTP Streaming Media skills primarily to solving different video distributor Lengthy and tedious problem caused by art in deployment and reception mechanism.The client that is mainly characterized by of DASH technologies can be according to Network status Such as speed of download, caching are how many, select the media slicing of suitable code check, distribution of media quotient passes through according to the selection of client Media slicing is sent to client by http protocol, to ensure the viewing experience of user.

The media exhibition description of existing DASH standards Main Specification (Media Presentation Description, MPD) the format of file and media slicing (Segment).The content structure of existing MPD file is as shown in Fig. 2, MPD file point For the period (Period), adapt to collection (Adaptation Set), description (Representation), fragment (Segment) totally 4 A level.One MPD file is made of one or more continuous period, and a Period indicates a media time section, There are initial time and end time；One period includes one or more Adaptation Set, each Adaptation Set generally corresponds to a kind of Media component, such as audio, video, subtitle.By taking the MPD file of video as an example, video Adaptation Set generally include multiple Representation, and different Representation correspond to different code checks, divide The other features such as resolution, between multiple Representation that the same Adaptation Set include can into Mobile state from Adapt to switching；Each Representation is made of one or more media slicings, and it is the basic unit of MPD to divide media piece, Client can by the uniform resource locator of the media slicing in MPD file (Uniform Resource Locator, URL media slicing) is obtained and handled to server to realize streaming media service.

The invention relates to panoramic video transmitting scenes, and in particular to asks transmission panorama in user end to server The scene of the forward direction server acquisition request MPD file of the video slicing of video.

Panoramic video is also referred to as 360 degree of panoramic videos, panoramic video be by the camera positioned at center to surrounding into Row 360o pan-shots change observation visual angle by sliding screen or the rotation head drive helmet when user watches, play complete The picture of scape video can switch therewith automatically, and user is as being in true environment.

In panoramic video transmitting scene, client obtains the MPD file of panoramic video to server first, it is one Meta data file provides the information how client accesses the media slicing of panoramic video.

Since the data volume of panoramic video is much larger compared with ordinary video, the mode for transmitting panoramic video at present mainly may be used To be divided into two classes：

1) full width is transmitted：It is consistent with ordinary video transmission method, by whole picture panoramic picture using H.264, the videos such as H.265 Coding form carries out coding transmission, and what client received is complete panoramic video content, as shown in Figure 3A.

2) block transmission：Panoramic picture is cut into multiple pieces (tile), every block diagram picture is encoded, is corresponded to per block diagram picture One video slicing is transmitted by the corresponding piecemeal content priority transmission of the current visual angle of user or with high-resolution when transmission. As shown in Figure 3B, entire panoramic picture is divided into 16 blocks, and a video slicing is corresponded to per block diagram picture.

Client can go that corresponding video slicing, the current video of user is asked to regard according to the current video visual angle of user Angle may be fallen on one or more blocks, therefore what client received is the corresponding video slicing of one or more blocks.Assuming that Client needs to ask the corresponding video slicing of four piecemeals in the left side institute diagram such as Fig. 4 respectively according to user's current visual angle. Client is decoded splicing to obtaining 4 video slicings back, renders and plays, and the video pictures of end user's viewing are as schemed Shown in 4 right side.

Motion Picture Experts Group's (Moving Picture Experts Group, MPEG) DASH standards are in MPD texts at present Visual angle (viewpoint) descriptor defined in part, the video and audio content with identical viewpoint values can be broadcast simultaneously It puts.Client can find video and audio fragment list with identical viewpoint values in MPD file, and according to working as Preceding bandwidth obtains the video and audio fragment of suitable code check respectively.Such as the video row for the MPD examples 1 of property illustrated below provided Include altogether 4 AdaptationSet in table, may determine that the first two AdaptationSet corresponds to video from mineType, after Two AdaptationSet correspond to audio, wherein the corresponding video slicings of Representation and id that id is 11 or 12 be The 31 or 32 corresponding audio fragments of Representation can play together, because their viewpoint values are equal to vp1.And the corresponding video slicings of Representation that id is 21 or 22 and Representation pairs that id is 41 or 42 The audio fragment answered can play together, because their viewpoint values are equal to vp2.

MPD examples one

It follows that the visual angle matching relationship between video slicing and audio fragment can only be indicated using the prior art, but When being panoramic video transmission, video slicing and video visual angle are not one-to-one, cannot represent audio well and regard Matching relationship between frequency visual angle.Such as when block transmission, video visual angle may be made of multiple video slicings, according to existing skill Identical viewpoint values should be arranged in the matched audio of these video slicings of art and the visual angle.But the same video point Piece may belong in different video visual angles, especially when the matched audio difference in the two video visual angles, using existing skill Art can not represent the matching relationship between the video slicing and multiple audios at composition different video visual angle.

And when panoramic video full width is transmitted, full image corresponds to a video slicing, wherein may include multiple videos If visual angle can not be represented using the prior art in the same video slicing when the corresponding audio difference in these video visual angles Matching relationship between video visual angle and different audios.

And increasing audio space description information in the embodiment of the present application in MPD file, client can utilize audio empty Between description information calculate the associated region of the audio fragment corresponding to the audio space description information, when user's current visual angle is true After fixed, client can obtain the audio fragment of associated region and user's current visual angle commensurate in scope and play, realize audio and The effect of video visual angle simultaneously match.

Based on the above problem of the existing technology, the embodiment of the present application provides a kind of matched side in audio & video visual angle Method, client and server.Technical solution provided by the embodiments of the present application is described in detail below by specific embodiment, needs Bright, the displaying sequence of embodiment only represents the sequencing of embodiment, does not represent the technical solution that embodiment is provided Quality.

Embodiment one

The embodiment of the present application provides a kind of server, as shown in fig.5, the host 500 where the server includes：Extremely A few processor 501, memory 502 and communication interface 503；At least one processor 501,502 and of the memory The communication interface 503 is connected by bus 504；

The memory 502, for storing computer executed instructions.

At least one processor 501, the computer executed instructions for executing the storage of the memory 502 so that The host 500 carries out data interaction by the host where the communication interface 503 and client to be implemented to execute the application A kind of matched method in audio & video visual angle that example provides.Wherein,

At least one processor 501 reads the program in memory 502, executes following process：

At least one processor 501 is used to obtain for what is sent by the reception client of the communication interface 503 First request message of the MPD file of panoramic video carries the mark of the MPD file in first request message；Root According to the mark of the MPD file, the MPD file is returned to the client, the MPD file includes at least one audio The mark of fragment and its corresponding audio space description information, the audio space description information are described at least one for describing The associated region of audio fragment.

In one possible implementation, in the MPD file further include at least one of MPD file audio The Region Matching condition of fragment and/or the matching strategy of Multi-audio-frequency fragment.

At least one processor 501, is additionally operable to：Receive what the client was sent by the communication interface 503 The second request message for obtaining video slicing carries the mark of the video slicing in second request message；Root According to the mark of the video slicing, the video slicing is sent to the client by the communication interface 503.

At least one processor 501, is additionally operable to：Receive what the client was sent by the communication interface 503 For obtain with the third request message of the matched first audio fragment of the video slicing, carry in the third request message There is the mark of the first audio fragment；According to the mark of the first audio fragment, by the communication interface 503 to described Client sends the first audio fragment.

In the present embodiment, at least one processor 501 may include different types of processor 501, or including The processor 501 of same type；Processor 501 can be below any：Central processing unit (Central Processing Unit, CPU), it is microprocessor, field programmable gate array (Field Programmable Gate Array, FPGA), special Processor etc. has the device of calculation processing ability.A kind of optional embodiment, at least one processor 501 can also collect As many-core processor.

The memory 502 can be below any or any combination：Random access memory (Random Access Memory, RAM), read-only memory (read only memory, ROM), nonvolatile memory (non- Volatile memory, NVM), solid state disk (Solid State Drives, SSD), mechanical hard disk, disk, disk permutation Equal storage mediums.

The communication interface 503 carries out data friendship for host 500 and other equipment (such as host where client) Mutually.Communication interface 503 can be below any or any combination：Network interface (such as Ethernet interface), wireless network The device with network access facility such as card.

The bus 504 may include address bus, data/address bus, controlling bus etc., and for ease of indicating, Fig. 5 is with one Thick line indicates the bus.The bus 504 can be below any or any combination：Industry standard architecture (Industry Standard Architecture, ISA) bus, peripheral component interconnection (Peripheral Component Interconnect, PCI) bus, expanding the industrial standard structure (Extended Industry Standard Architecture, EISA) wired data transfers such as bus device.

An embodiment of the present invention provides a kind of clients, as shown in fig.6, the host 600 where the client includes：Extremely A few processor 601, memory 602 and communication interface 603；At least one processor 601,602 and of the memory The communication interface 603 is connected by bus 604；

The memory 602, for storing computer executed instructions.

At least one processor 601, the computer executed instructions for executing the storage of the memory 602 so that The host 600 carries out data interaction by the host where the communication interface 603 and client to be implemented to execute the application A kind of matched method in audio & video visual angle that example provides.Wherein,

At least one processor 601 reads the program in memory 602, executes following process：

At least one processor 601, for being sent to server for obtaining panorama by the communication interface 603 First request message of the MPD file of video carries the mark of the MPD file in first request message；Pass through institute It states communication interface 603 and receives the server according to the MPD file of the identification feedback of the MPD file, the MPD file Include the mark of at least one audio fragment and its corresponding spatial description information, the audio space description information is for retouching State the associated region of at least one of described MPD file audio fragment；According to the current visual angle range of user and it is described at least One audio space description information determines the first audio fragment with the current visual angle commensurate in scope.

At least one processor 601 is retouched according to the current visual angle range and at least one audio space of user Information is stated, when determining the first audio fragment with the current visual angle commensurate in scope, is specifically used for：

At least one of described MPD file audio fragment is obtained according at least one audio space description information to exist At least one of panoramic video associated region；By at least one associated region within the scope of the current visual angle The corresponding audio fragment of associated region to match is determined as alternative audio fragment；If only exist an alternative audio fragment, The alternative audio fragment is determined as the first audio fragment；If there are when at least two alternative audio fragments, according to described more The matching strategy of audio fragment determines the first audio fragment；If there is no when alternative audio fragment, by the default audio of pre-configuration Fragment is set to the first audio fragment.

In one possible implementation, at least one associated region with phase within the scope of the current visual angle The associated region matched is associated region identical with the current visual angle range；Or, with described in current visual angle range satisfaction The associated region of Region Matching condition.

In one possible implementation, the pass that the Region Matching condition is met with the current visual angle range Join region, including：Fall into the associated region of the current visual angle range；Or, being more than with the matching degree of the current visual angle range The associated region of predetermined threshold value.

At least one processor 601 is additionally operable at least one audio fragment for including by the MPD file and downloads Local to the client, the client is described according to the current visual angle range and at least one audio space of user Information, determine with after the first audio fragment of the current visual angle commensurate in scope, from being downloaded to local at least one audio The first audio fragment is obtained in fragment is decoded broadcasting.

In the present embodiment, at least one processor 601 may include different types of processor 601, or including The processor 601 of same type；Processor 601 can be below any：CPU, arm processor, FPGA, application specific processor Deng the device with calculation processing ability.A kind of optional embodiment, at least one processor 601 can also be integrated into crowd Core processor.

The memory 602 can be below any or any combination：RAM, ROM, NVM, SSD, mechanical hard disk, The storage mediums such as disk, disk permutation.

The communication interface 603 carries out data friendship for host 600 and other equipment (such as host where server) Mutually.Communication interface 603 can be below any or any combination：Network interface (such as Ethernet interface), wireless network The device with network access facility such as card.

The bus 604 may include address bus, data/address bus, controlling bus etc., and for ease of indicating, Fig. 6 is with one Thick line indicates the bus.The bus 604 can be below any or any combination：Isa bus, pci bus, EISA The device of the wired data transfers such as bus.

It includes sound that user end to server acquisition request, which may be implemented, in server and client side provided by the embodiments of the present application The mark of frequency division piece and its MPD file of corresponding audio space description information so that client can be in current visual angle range After determination, associated region of each audio in full-view video image is calculated according to audio space description information.When some sound When the corresponding associated region of frequency division piece and user's current visual angle range match, client is made to get and accurate of video image The audio file matched plays out, and to realize the simultaneously match of audio & video image, promotes the viewing experience of user.Pass through The embodiment of the present application can work as forward sight to solve client present in the scheme of existing client terminal playing panoramic video When angle changes, the problem of matching audio file can not be selected to play out, lead to poor user experience.Further , in the embodiment of the present application, by the way that the Region Matching condition of audio fragment is arranged in MPD file, the pass of audio may be implemented Join condition coupling different between region and user's current visual angle, to provide between more flexible audio & video image With effect, further, by the way that Multi-audio-frequency matching strategy is arranged in MPD file, when multiple associated regions and user are current When angular field of view matches, according to Multi-audio-frequency matching strategy, client can select best audio to carry out matching broadcasting.

Embodiment two

The embodiment of the present application provides a kind of matched method in audio & video visual angle, as shown in fig. 7, being serviced in this method Device and the interaction flow of client are as follows：

S701：User end to server sends the first request message of MPD file for obtaining panoramic video, and described the The mark of the MPD file is carried in one request message.

In S701, the mark of MPD file obtains the MPD file of the mark instruction of the MPD file for server.MPD texts The mark of part can be uniform resource identifier (Uniform Resource Idetifier, URI), be http with URI:// For example.com/mpd, the first request message includes following content：

GET http://example.com/mpd HTTP/1.1

Connection:keep-alive

It should be noted that above-mentioned first request message is merely illustrative, the first request message in the present embodiment In addition to the mark including MPD file, it can also include other parameters, no longer repeat one by one herein.

S702：Server obtains MPD file according to the mark of MPD file.

In S702, MPD file includes the mark of at least one audio fragment and its corresponding audio space description information, The audio space description information is used to describe the associated region of at least one audio fragment.

Illustratively, including the content of the MPD file of audio space description information is as follows：

Adhering to separately property is described as follows shown in table one in the middle part of the above-mentioned MPD file including spatial description information：

Table one

In above-mentioned table one, adaptationSet@mimeType indicate medium type, from AdaptationSet (mimeType=" video/mp4 "), should it is found that include the video file of a mp4 type in above-mentioned MPD file The video slicing of 3 kinds of different code checks is contained in AdaptationSet, they correspond to different video heights and width respectively, Such as：When code check is " 1024000 " bandwidth=, the width of video image is width=" 2560 ", is highly Height=" 720 ", because video is by the way of full width transmission in the present embodiment, the width of panoramic picture in panoramic video Degree and height are 2560 and 720.In addition, also including 3 audio fragments, AdaptationSet in the MPD file (mimeType=" audio/mp4 ") comprising the corresponding audio fragment of a main audio fragment and 2 specific regions, SchemeIdUri=" urn:mpeg:dash:asrd:2016 " audio space description information, the definition of key (value) value are indicated As shown in Table 2, wherein M is indicated essential, and O indicates optional.

Table two

@value	Usage	Description
			object_x	M	Abscissa of the upper left corner of audio fragment corresponding region in full-view video image
object_y	M	Ordinate of the upper left corner of audio fragment corresponding region in full-view video image
			object_width	M	The width or horizontal direction size of audio fragment corresponding region
object_height	M	The height or vertical direction size of audio fragment corresponding region
			total_width	O	The width of full-view video image
total_height	O	The height of full-view video image

Therefore 1 corresponding audio space description information of audio fragment<SupplementalProperty schemeIdUri =" urn:mpeg:dash:asrd:2016 " value=" 480,390,810,300,3840,1080 "/>Indicate the audio fragment Associated region be in width be 3840, highly in 1080 full-view video image with (480,390) for the upper left corner, width is The region that 810 height are 300.Because providing the width of full-view video image in 1 corresponding spatial description information of audio fragment Degree and height, thus it is corresponding in audio fragment 2<SupplementalProperty schemeIdUri=" urn:mpeg: dash:asrd:2016 " value=" 3072,285,480,510 "/>In can no longer provide full-view video image width and Highly, it indicates that the associated region of audio fragment 2 be in width is 3840, highly in 1080 full-view video image with (3072,285) it is the upper left corner, width is the region that 480 height are 510.

In the present embodiment, the audio fragment for not providing audio relation description information is considered as main audio, also may be used With referred to as default audio, in addition to there is no the audio for providing audio relation description information that can be used as default audio, if audio point When piece includes precedence information, the audio fragment of highest priority is not limited in the application it is also assumed that be default audio The method for determining default audio fragment.

It should be noted that audio space description information in addition to can such as table two in other than the description method that provides, may be used also To be described by each apex coordinate position of the corresponding associated region of audio fragment, application scheme does not limit area of space Description method.It therefore, can also be by providing the relative scale with full-view video image other than above-mentioned absolute value description To describe.

S703：Server returns to the MPD file to client, and the MPD file includes at least one audio fragment Mark and its corresponding audio space description information, the audio space description information is for describing at least one audio The associated region of fragment.

In the present embodiment, server may be implemented by the above method and send MPD file to client, client is made to be based on The MPD file realizes the matching one by one of video slicing and audio fragment.The above method can also include the following steps, to realize Audio fragment of the server to client transmissions panoramic video：

S704：User end to server sends the second request message for obtaining video slicing, second request message It include the mark of the video slicing.

Client is according to current bandwidth situation to the video slicing of the suitable code check of server request selecting, it is assumed here that client It is bandwidth=" 1024000 " to hold the code check selected, and corresponding representation is as follows：

</Representation>

Therefore, the URL of video slicing is http://cdn1.example.com/562465736.mp4, the second request disappear It is as follows to cease format：

GET http://cdn1.example.com/562465736.mp4HTTP/1.1

Connection:keep-alive

S705：Server sends the video slicing according to the mark of the video slicing to the client.

S706：Client describes to believe according at least one of the current visual angle range of user and MPD file audio space Breath,

Determine the first audio fragment with the current visual angle commensurate in scope.

It is highly 720 because the corresponding panoramic picture width of video slicing that client obtains in S705 is 2560, it is false If it in width is 2560 that the current visual angle range areas of user, which is, highly to be with (320,260) in 720 full-view video image The upper left corner, width are the region that 540 height are 200.Due to corresponding in the audio space description information of the MPD file in table one The width of full-view video image is 3840, is highly 1080, therefore client is needed the value in audio space description information Value converts：

Object_x '=object_x*width '/total_width

Object_y '=object_y*height '/total_height

Object_width '=object_width*width '/total_width

Object_height '=object_height*height '/total_height

Wherein, object_x, object_y, object_width, object_height, total_width, total_ Height is the original value values in the corresponding audio space description information of MPD file sound intermediate frequency fragment, and width, height are The width and height for the corresponding full-view video image of video slicing that client obtains, object_x ', object_y ', Object_width ', object_height ', width, height are that the video slicing that audio fragment is obtained in client corresponds to Full-view video image in spatial description information.After calculating, audio fragment 1 width be 2560, highly for 720 it is complete Associated region is for the upper left corner, width is the region that 540 height are 200, audio fragment with (320,260) in scape video image 2 width be 2560, highly for associated region in 720 full-view video image be with (2030,190) for the upper left corner, width The region for being 340 for 320 height, therefore client determines that with the matched audio fragment in current visual angle range areas of user be sound Frequency division piece 1, i.e. the first audio fragment are audio fragment 1.

S707：The user end to server is sent for obtaining and the matched first audio fragment of the video slicing Third request message carries the mark of the first audio fragment in the third request message.

1 corresponding AdaptationSet of audio fragment is as follows, includes the audio fragment of two different code checks, it is assumed that client End selects code check for the audio fragment of bandwidth=" 64000 " according to current bandwidth determination

Therefore, the URL of the audio fragment selected is http://cdn1.example.com/3463275477.mp4, third Request message format is as follows：

GET http://cdn1.example.com/3463275477.mp4 HTTP/1.1

Connection:keep-alive

S708：The server sends first sound according to the mark of the first audio fragment to the client Frequency division piece.

Server sends corresponding audio fragment to client, client is to this according to the third solicited message of client Audio fragment is decoded broadcasting.

It should be noted that due to audio fragment data amount and little, client can also in advance by multiple audios all under It is downloaded to local, is directly obtained locally after determining the audio fragment that the region with the current visual angle range of user matches in S706 The audio fragment is taken to be decoded broadcasting.

Further, after user converts current visual angle, client obtains the audio to match with newest current visual angle Fragment is decoded broadcasting.

Assuming that it is 2560 that the region watched of the transformed current visual angle of user, which is in width, highly for 720 aphorama With (2030,190) for the upper left corner in frequency image, width is the region that 320 height are 340, therefore client is according to step S706 It determines that the audio fragment that the current visual angle range areas with user matches is audio fragment 2, then, executes S707 and S708 and obtain It is decoded after taking the audio fragment that code check is bandwidth=" 64000 " in 2 corresponding AdaptationSet of audio fragment It plays.

Sequence is executed it should be noted that being not intended to limit in the application between S704-S705 and S706-S708.

Further include at least one of MPD file audio point in the MPD file in a kind of possible embodiment The Region Matching condition of piece and/or the matching strategy of Multi-audio-frequency fragment.For this embodiment, following example three is come to this It is described in detail.

Fig. 8 shows a kind of matched method in audio & video visual angle, is retouched by executive agent of client in Fig. 8 It states, at this point, the implementation procedure of server is identical with Fig. 7, details are not described herein.

As shown in figure 8, client determines that the method with the matched audio fragment in current video visual angle comprises the steps of：

800：User end to server sends the first request message of MPD file for obtaining panoramic video, and described the The mark of the MPD file is carried in one request message.Specific implementation process sees the S701 in Fig. 7, no longer superfluous herein It states.

801：Client receives the MPD file that server is sent, and the MPD file includes at least one audio fragment Mark and its corresponding audio space description information, the audio space description information is for describing at least one audio point The associated region of piece.

The mode of transmission panoramic video can be mainly divided into full width transmission and two class of block transmission at present, be passed when using full width When defeated panoramic video, the content of the MPD file can be as shown in embodiment two.The present embodiment three will focus on block transmission For illustrate, at this point, including audio space description information MPD file content it is as follows.

Include the corresponding audio fragment of a main audio fragment and 2 specific regions in above-mentioned MPD file, SchemeIdUri=" urn:mpeg:dash:asrd:2016 " audio space description information, the audio space description letter are indicated Breath may be used as other than the representation method of the definition of table one in embodiment two, described using a kind of audio space in the present embodiment three The relative value representation method of information, the definition of value values are as shown in Table 3：

Table three

802：Client selects video slicing, determines the width and height of the corresponding full-view video image of the video slicing Degree.

Client selects the video slicing of suitable code check according to current bandwidth, when using such as the full width transmission in embodiment two When panoramic video, the corresponding width of selected video slicing and height are the width and height of full-view video image.When Using in the present embodiment three when the video slicing of block transmission panorama, it is assumed that client according to current bandwidth select code check for The video slicing of bandwidth=" 128000 ", width=" 960 " height=" 270 " indicate the corresponding video of video slicing Picture traverse is 960, is highly 270, is illustrated with above-mentioned exemplary MPD file, video AdaptationSet (mimeType=" video/mp4's ")<SupplementalProperty schemeIdUri=" urn:mpeg:dash: srd:2014 " value=" 0,0,0,1,1,4,4 "/>Indicate that the full-view video image width and height are respectively divided into 4 parts, entirely Full-view video image is divided into 4*4=16 blocks (Tile), that is to say, that the width of each block of video slicing image and height are respectively The width of full-view video image and a quarter of height, therefore client selects code check for bandwidth='s " 128000 " The width of the corresponding full-view video image of video slicing is 960*4=3840, is highly 270*4=1080.

It should be noted that it is existing to determine that the width of the corresponding full-view video image of video slicing and height are referred to The prior art is merely given as one kind in the present embodiment three and illustrates, and is not especially limited.

803：Client calculates each audio fragment in the video slicing according to the audio space description information in MPD file Associated region in corresponding full-view video image.

It, can be according in embodiment two when audio space description information uses absolute value representation mode as shown in Table 2 Method described in S706 calculates associated region of each audio fragment in the corresponding full-view video image of the video slicing.This Indicate that calculate each audio fragment when audio space description information regards described for relative scale shown in table three in embodiment three The method of associated region is described in detail in the corresponding full-view video image of frequency division piece.

It is respectively 3840 and 1080 by the overall width and total height of the full-view video image determined in 802, according in table three The audio space description information value value attributes (relative scale representation) provided can determine：1 corresponding audio of audio fragment Spatial description information<SupplementalProperty schemeIdUri=" urn:mpeg:dash:asrd:2016"value =" 0.125,0.361,0.211,0.278 "/>Indicate that the associated region of audio fragment to be 3840 in width, is highly With (0.125*3840=480,0.361*1080=390) for the upper left corner in 1080 full-view video image, width 0.211* 3840=810 height is the region of 0.278*1080=300.2 corresponding audio space description information of audio fragment< SupplementalProperty schemeIdUri=" urn:mpeg:dash:asrd:2016 " value=" 0.8,0.264, 0.125,0.472"/>It in width is 3840 to indicate that the associated region of audio fragment is, highly for 1080 full-view video image In with (0.8*3840=3072,0.264*1080=285) be the upper left corner, width is that 0.125*3840=480 height is The region of 0.472*1080=510.

804：When matching there are the associated region of alternative audio fragment and current visual angle range, 805 are executed；Otherwise, Execute 807.

Wherein, client by least one associated region with the associated region pair that matches within the scope of the current visual angle The audio fragment answered is determined as alternative audio fragment.

Specifically, determining whether there is the associated region of audio fragment and current visual angle range matches.Can by with Under type determines：

Mode one, if the associated region of an audio fragment is associated region identical with current visual angle range, it is determined that with Current visual angle range matches.

After calculating the associated region of an audio fragment according to the above method, if the associated region and use of general audio fragment When the current visual angle range areas at family is identical, then it is assumed that the audio fragment matches with current visual angle range.Such as assume user Current visual angle range areas be in width be 3840, highly in 1080 full-view video image with (480,390) for upper left When width is the region that 810 height are 300, the associated region of audio fragment 1 can be determined according to the result of calculation in 803 for angle It is identical as the current visual angle range areas of user, i.e. audio fragment 1 and current visual angle commensurate in scope, as shown in Figure 9 A.

Mode two：If the associated region of an audio fragment is the pass for meeting Region Matching condition with the current visual angle range Join region, it is determined that match with current visual angle range.

Specifically, meet the associated region of the Region Matching condition with the current visual angle range, including：

Fall into the associated region of the current visual angle range；Or, being more than with the matching degree of the current visual angle range default The associated region of threshold value.

Specifically, in MPD file can with setting area matching condition, when audio fragment associated region and user it is current When meeting the Region Matching condition between angular field of view region, it is determined that the audio fragment matches with current visual angle range.

For example, 1) Region Matching condition is the condition of inclusion relation, when the current visual angle range areas of user includes audio When the associated region of fragment, it is believed that the audio fragment matches with current visual angle range, as shown in figs. 9 a and 9b；2) region Matching condition is the condition of smallest match ratio, and smallest match ratio is preset ratio value.When the current visual angle range of user The ratio that the lap of the associated region of region and audio fragment accounts for the associated region of audio fragment is more than smallest match ratio When, it is believed that the audio fragment matches with current visual angle range, as shown in Figure 9 C.

It should be noted that being not intended to limit the match party of the associated region and current visual angle range of audio fragment in the application Method.

805：When matching there are the associated region of at least two alternative audio fragments and current visual angle range, execute 806；Otherwise, 808 are executed.

The quantity of the associated region of the audio fragment to match with the current visual angle range determined according to the method described above More than one, specifically see shown in Figure 10 A, Figure 10 B and Figure 10 C.

806：When in MPD file including Multi-audio-frequency matching strategy, 809 are executed；Otherwise, 807 are executed.

807：Client selects default audio fragment to be decoded broadcasting as the first audio fragment.

Default audio fragment can be the audio fragment of no any associated region or be not provided with audio space description The audio fragment of information can also be the audio fragment provided with highest priority.

808：Selection is decoded broadcasting with the first audio fragment that current visual angle range matches.

809：The the first audio fragment to match with current visual angle range to be obtained according to Multi-audio-frequency matching strategy determination It is decoded broadcasting.

Multi-audio-frequency matching strategy is used to indicate when the associated region of multiple audio fragments can be with current visual angle range phase The strategy of selection and the audio fragment of current visual angle commensurate in scope when matching.For example, priority match strategy can be used as multitone A kind of embodiment of frequency matching strategy.At this time, it may be necessary to the priority of each audio fragment be preset in MPD file, according to pre- If each audio fragment priority, select the audio fragment of highest priority as first with current visual angle commensurate in scope Audio fragment；In another example the matching strategy of matching degree can be as a kind of embodiment of Multi-audio-frequency matching strategy.At this point, can To calculate the overlapping region of each associated region and current visual angle range areas, using the maximum associated region in overlapping region as With the maximum associated region of degree；Alternatively, the ratio value of overlapping region and associated region is calculated, by the maximum associated region of ratio value As the maximum associated region of matching degree, so that it is determined that the first audio corresponding with the associated region of current visual angle commensurate in scope point Piece.

It should be noted that Multi-audio-frequency matching strategy is not limited in the application specifically, it is any to can be used for working as multiple sounds The associated region of frequency division piece can be with selection when current visual angle commensurate in scope and the audio fragment of current visual angle commensurate in scope Method all can serve as Multi-audio-frequency matching strategy.

If Multi-audio-frequency matching strategy is priority match strategy, audio adapts to wrap in collection (AdaptationSet) Containing priority attribute, it is used to indicate the priority of the audio fragment.When multiple audio fragments associated region can with it is current When angular field of view matches, by comparing the priority of these audio fragments, determine that the satisfactory audio fragment of priority is The audio fragment to match with current visual angle range.

The MPD file of embodiment three increases Region Matching condition on the basis of embodiment two, can be with more flexible earth's surface Show the matching relationship between the associated region of audio fragment and current visual angle range areas, plan is further matched by Multi-audio-frequency Can slightly solve the problems, such as how to select optimal audio fragment when multiple audio fragments match with current visual angle range, so as to To bring the viewing experience of more accurately audio & video visual angle simultaneously match to user.

Embodiment three

Based on above example, the embodiment of the present invention additionally provides a kind of server, the server can be with shown in Fig. 5 The identical equipment of server, the method that server side in embodiment two executes may be used.Refering to fig. 1 shown in 1, the present invention is real Applying a kind of server 1100 that example provides includes：Receiving unit 1101, processing unit 1102.Wherein,

Receiving unit 1101, the display advertising for obtaining panoramic video for receiving client transmission describe MPD texts First request message of part carries the mark of the MPD file in first request message；

Processing unit 1102 returns to the MPD file, institute for the mark according to the MPD file to the client State the mark and its corresponding audio space description information that MPD file includes at least one audio fragment, the audio space Description information is used to describe the associated region of at least one audio fragment.

In one possible implementation, the server further includes transmission unit 1103,

The receiving unit 1101 is additionally operable to receive the second request for obtaining video slicing that the client is sent Message carries the mark of the video slicing in second request message；

The transmission unit 1103 sends the video for the mark according to the video slicing to the client Fragment.

In one possible implementation, the receiving unit 1101 is additionally operable to receive the use that the client is sent In obtaining the third request message with the matched first audio fragment of the video slicing, carried in the third request message The mark of the first audio fragment；

The transmission unit 1103 is additionally operable to the mark according to the first audio fragment, and institute is sent to the client State the first audio fragment.

The method that the function of above-mentioned each unit can be found in the execution of two server side of embodiment, details are not described herein again.

It should be noted that being schematical, only a kind of logic function to the division of unit in the embodiment of the present invention It divides, formula that in actual implementation, there may be another division manner.In addition, each functional unit in each embodiment of the application can be with It is integrated in a processing unit, can also be that each unit physically exists alone, it can also two or more unit collection At in a unit.The form that hardware had both may be used in above-mentioned integrated unit is realized, SFU software functional unit can also be used Form realize.

Based on above example, the embodiment of the present invention additionally provides a kind of client, the client can be with shown in Fig. 6 The identical equipment of client, the method that client-side in embodiment two executes may be used.Refering to fig. 1 shown in 2, the present invention is real Applying a kind of client 1200 that example provides includes：Receiving unit 1201, processing unit 1202 and transmission unit 1203.Wherein,

Transmission unit 1203 describes MPD file for being sent to server for obtaining the display advertising of panoramic video First request message carries the mark of the MPD file in first request message；

Receiving unit 1201, for receiving MPD text of the server according to the identification feedback of the MPD file Part, the MPD file include the mark of at least one audio fragment and its corresponding spatial description information, the audio space Description information is used to describe the associated region of at least one of described MPD file audio fragment；

Processing unit 1202, for being believed according to the current visual angle range and at least one audio space description of user Breath determines the first audio fragment with the current visual angle commensurate in scope.

In one possible implementation, the processing unit 1202 is according to the current visual angle range of user and described At least one audio space description information is specifically used for when determining the first audio fragment with the current visual angle commensurate in scope：

In one possible implementation, at least one associated region with phase within the scope of the current visual angle The associated region matched is associated region identical with the current visual angle range；Or,

In one possible implementation, the pass that the Region Matching condition is met with the current visual angle range Join region, including：

Fall into the associated region of the current visual angle range；Or,

In one possible implementation, the processing unit 1202 is additionally operable to：

The method that the function of above-mentioned each unit can be found in the execution of two client-side of embodiment, details are not described herein again.

It should be understood by those skilled in the art that, the embodiment of the present application can be provided as method, system or computer program production Product.Therefore, in terms of the embodiment of the present application can be used complete hardware embodiment, complete software embodiment or combine software and hardware Embodiment form.Moreover, it wherein includes computer available programs generation that the embodiment of the present application, which can be used in one or more, The meter implemented in the computer-usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) of code The form of calculation machine program product.

The embodiment of the present application is with reference to the method, equipment (system) and computer program product according to the embodiment of the present application Flowchart and/or the block diagram describe.It should be understood that can be realized by computer program instructions in flowchart and/or the block diagram The combination of flow and/or box in each flow and/or block and flowchart and/or the block diagram.These calculating can be provided Processing of the machine program instruction to all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices Device is to generate a machine so that the instruction executed by computer or the processor of other programmable data processing devices generates For realizing the function of being specified in one flow of flow chart or multiple flows and/or one box of block diagram or multiple boxes Device.

These computer program instructions, which may also be stored in, can guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works so that instruction generation stored in the computer readable memory includes referring to Enable the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one box of block diagram or The function of being specified in multiple boxes.

These computer program instructions also can be loaded onto a computer or other programmable data processing device so that count Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, in computer or The instruction executed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one The step of function of being specified in a box or multiple boxes.

Obviously, those skilled in the art can carry out the embodiment of the present application various modification and variations without departing from this Shen Spirit and scope please.In this way, if these modifications and variations of the embodiment of the present application belong to the application claim and its wait Within the scope of technology, then the application is also intended to include these modifications and variations.

Claims

1. a kind of matched method in audio & video visual angle, which is characterized in that including：

The first request that the display advertising for obtaining panoramic video that server reception client is sent describes MPD file disappears It ceases, the mark of the MPD file is carried in first request message；

The server returns to the MPD file according to the mark of the MPD file, to the client, in the MPD file Mark including at least one audio fragment and its corresponding audio space description information, the audio space description information are used for The associated region of at least one audio fragment is described.

2. the method as described in claim 1, which is characterized in that further include in the MPD file in the MPD file at least The Region Matching condition of one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

3. method as claimed in claim 1 or 2, which is characterized in that the method further includes：

The server receives the second request message for obtaining video slicing that the client is sent, second request The mark of the video slicing is carried in message；

4. method as claimed in claim 3, which is characterized in that the method further includes：

The server receive that the client sends for obtaining and the matched first audio fragment of the video slicing Third request message carries the mark of the first audio fragment in the third request message；

The server sends the first audio fragment according to the mark of the first audio fragment to the client.

5. a kind of matched method in audio & video visual angle, which is characterized in that including：

User end to server sends the first request message that MPD file is described for obtaining the display advertising of panoramic video, institute State the mark that the MPD file is carried in the first request message；

The client receives the MPD file of the server according to the identification feedback of the MPD file, the MPD texts Part includes the mark of at least one audio fragment and its corresponding spatial description information, and the audio space description information is used for The associated region of at least one of described MPD file audio fragment is described；

Current visual angle range and at least one audio space description information of the client according to user, determine with it is described First audio fragment of current visual angle commensurate in scope.

6. method as claimed in claim 5, which is characterized in that further include in the MPD file in the MPD file at least The Region Matching condition of one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

7. such as method described in claim 5 or 6, which is characterized in that the client according to the current visual angle range of user and At least one audio space description information determines the first audio fragment with the current visual angle commensurate in scope, including：

The client obtains at least one of MPD file audio according at least one audio space description information Fragment is at least one of panoramic video associated region；

The client by least one associated region with the associated region pair that matches within the scope of the current visual angle The audio fragment answered is determined as alternative audio fragment；

If there are when at least two alternative audio fragments, according to the matching strategy of the Multi-audio-frequency fragment, the first audio point is determined Piece；

8. the method for claim 7, which is characterized in that at least one associated region with the current visual angle model The associated region to match in enclosing is associated region identical with the current visual angle range；Or,

9. method as claimed in claim 8, which is characterized in that described to meet the Region Matching with the current visual angle range The associated region of condition, including：

Fall into the associated region of the current visual angle range；Or,

10. method as claimed in claim 5, which is characterized in that the method further includes：

At least one audio fragment that the MPD file includes is downloaded to the client local by the client, described Client according to user current visual angle range and at least one audio space description information, determine and described work as forward sight After the matched first audio fragment of angular region, first audio is obtained from being downloaded in local at least one audio fragment Fragment is decoded broadcasting.

11. a kind of server, which is characterized in that including：

Receiving unit, the display advertising for obtaining panoramic video for receiving client transmission describe the first of MPD file Request message carries the mark of the MPD file in first request message；

Processing unit returns to the MPD file, the MPD texts for the mark according to the MPD file to the client Part includes the mark of at least one audio fragment and its corresponding audio space description information, the audio space description information Associated region for describing at least one audio fragment.

12. server as claimed in claim 11, which is characterized in that further include in the MPD file in the MPD file The Region Matching condition of at least one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

13. the server as described in claim 11 or 12, which is characterized in that the server further includes transmission unit,

The receiving unit is additionally operable to receive the second request message for obtaining video slicing that the client is sent, institute State the mark that the video slicing is carried in the second request message；

14. server as claimed in claim 13, which is characterized in that the receiving unit is additionally operable to receive the client What is sent is used to obtain and the third request message of the matched first audio fragment of the video slicing, the third request message In carry the mark of the first audio fragment；

The transmission unit is additionally operable to the mark according to the first audio fragment, and first sound is sent to the client Frequency division piece.

15. a kind of client, which is characterized in that including：

Transmission unit, for sending the first request for describing MPD file for obtaining the display advertising of panoramic video to server Message carries the mark of the MPD file in first request message；

Receiving unit, the MPD file for receiving the server according to the identification feedback of the MPD file, the MPD File includes the mark of at least one audio fragment and its corresponding spatial description information, and the audio space description information is used In the associated region for describing at least one of MPD file audio fragment；

Processing unit, for according to user current visual angle range and at least one audio space description information, determine with First audio fragment of the current visual angle commensurate in scope.

16. client as claimed in claim 15, which is characterized in that further include in the MPD file in the MPD file The Region Matching condition of at least one audio fragment and/or the matching strategy of Multi-audio-frequency fragment.

17. the client as described in claim 15 or 16, which is characterized in that the processing unit is working as forward sight according to user Angular region and at least one audio space description information determine the first audio fragment with the current visual angle commensurate in scope When, it is specifically used for：

At least one of described MPD file audio fragment is obtained described according at least one audio space description information At least one of panoramic video associated region；

By audio corresponding with the associated region to match within the scope of the current visual angle at least one associated region point Piece is determined as alternative audio fragment；

18. client as claimed in claim 17, which is characterized in that work as forward sight with described at least one associated region The associated region to match in angular region is associated region identical with the current visual angle range；Or,

19. client as claimed in claim 18, which is characterized in that described to meet the region with the current visual angle range The associated region of matching condition, including：

Fall into the associated region of the current visual angle range；Or,

20. client as claimed in claim 15, which is characterized in that the processing unit is additionally operable to：

At least one audio fragment that the MPD file includes is downloaded to the client local, the client is in root Current visual angle range according to user and at least one audio space description information determine and the current visual angle commensurate in scope The first audio fragment after, obtain the first audio fragment from being downloaded in local at least one audio fragment and solved Code plays.

21. a kind of server, which is characterized in that including memory, processor and communication interface；Wherein,

The memory is for storing computer-readable program；

The processor is by running the program in the memory, to complete the method as described in Claims 1-4 is any；

22. a kind of client, which is characterized in that including memory, processor and communication interface；Wherein,

The memory is for storing computer-readable program；

The processor is by running the program in the memory, to complete the method as described in claim 5 to 10 is any；