CN105894581A

CN105894581A - Method and device for displaying multimedia information

Info

Publication number: CN105894581A
Application number: CN201610044190.6A
Authority: CN
Inventors: 冯歆鹏; 周骥
Original assignee: 冯歆鹏; 周骥
Current assignee: Kunshan Zhaoguan Electronic Technology Co., Ltd.; Shaoxing Zhaoguan Electronic Technology Co., Ltd.
Priority date: 2016-01-22
Filing date: 2016-01-22
Publication date: 2016-08-24
Anticipated expiration: 2036-01-22
Also published as: CN105894581B

Abstract

The invention discloses a method and device for displaying multimedia information. The method comprises the following steps: obtaining presentation information, wherein the presentation information comprises electromagnetic field spectral information for showing an object and capable of being observed by naked eyes and/ or capable of being collected by a device; establishing a four-dimensional space-time model for showing the presentation information according to the obtained presentation information, wherein the four-dimensional space-time model has attributes capable of showing the time-dependent presentation information in a digital manner; and displaying the presentation information shown by the four-dimensional space-time model. In the scheme, the four-dimensional space-time model has the attributes capable of showing the time-dependent presentation information in a digital manner, so that the problem of delay of the shown presentation information is solved to some extent. The method and device solve the defect of time delay in the prior art to some extent.

Description

A kind of method and device presenting multimedia messages

Technical field

The present invention relates to communication technical field, particularly relate to a kind of method presenting multimedia messages and dress Put.

Background technology

Along with the development of communication technology, it is right that communication has been changed into by people by initial single voice demand Video and the communication requirement of audio frequency, therefore, with the video communication transmitting voice, data, video are integrated Business is more and more important, and in video conferencing, long-distance video medical treatment, long-distance video religion The more and more extensive of aspect application such as educate.

VR (Virtual Reality, virtual reality) technology be one can simulating human vision, audition, The human-computer interaction technology of the high fidelity of the perception behaviors such as sense of touch, utilizes this technology that people can be made to be in and uses tricks In the environment that calculation machine generates, by sense organ language, gesture can be to carry out " handing over by natural mode wherein Mutually, dialogue ", and freely move, arbitrarily observe things around, see thing, hear sound, feel power, People is made to have sense the most on the spot in person.

But the method at present multimedia messages gathered being used VR technical finesse, owing to not processing The multimedia messages of Real-time Collection, therefore, presents represented by time and the multimedia messages of multimedia messages The real scene time there is time delay.

In sum, present the method for multimedia messages at present to have and there is the defect that time delay is longer.

Summary of the invention

In view of the above problems, it is proposed that the present invention, in order to provide one to overcome the problems referred to above or at least portion Ground is divided to solve a kind of method and apparatus presenting multimedia messages of the problems referred to above.

According to a first aspect of the present invention, it is provided that a kind of method presenting multimedia messages, including:

Obtain presentation information, that described presentation information includes observing with the naked eye and/or can be with setting The standby electromagnetic field spectral information for characterizing object collected；

The four-dimensional spacetime model for characterizing described presentation information is set up according to the presentation information got, Described four-dimensional spacetime model has can characterize time dependent described presentation letter in digitized form The attribute of breath；

The presentation information that described four-dimensional spacetime model characterizes is presented.

Alternatively, in the method presenting multimedia messages according to the abovementioned embodiments of the present invention, described Presentation information also includes sound field information that is that can feel and/or that can collect with equipment with ear；

Before setting up described four-dimensional spacetime model according to the presentation information got, described method is also wrapped Include:

The sound field information of the object corresponding with described presentation information is calculated according to described presentation information；

The four-dimensional spacetime model for characterizing described presentation information is set up, specifically according to described presentation information Including:

According to described presentation information and described sound field information, set up and be used for characterizing described presentation information and institute State the four-dimensional spacetime model of sound field information.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, Set up the four-dimensional spacetime model for characterizing described presentation information according to the presentation information got, specifically wrap Include:

Described presentation information is processed, obtains the first markup information；

According to described first markup information and described presentation information, obtain including first cloud of geological information Information, include the second point cloud information of texture information；

Described first cloud information and described second point cloud information are merged, obtains impact point cloud letter Breath；

Visual information is obtained according to described impact point cloud information；

Obtain spatial model according to described visual information, will melt for spatial model the most in the same time Close；

Space module, described first markup information and described second markup information obtained according to fusion, To described four-dimensional spacetime model.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, Described presentation information also includes sound field that is that can feel and/or that can collect with equipment with ear Information；

Before obtaining spatial model according to described visual information, described method also includes:

Obtain spatial model according to described visual information, specifically include:

Described visual information and described sound field information are merged, obtains described spatial model.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, Described first cloud information and described second point cloud information are merged, after obtaining impact point cloud information, Before obtaining described visual information, described method also includes:

Described impact point cloud information is processed, obtains the second markup information；

According to described impact point cloud information, obtain visual information, specifically include:

According to described second markup information and described impact point cloud information, obtain described visual information.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, According to described second markup information and described impact point cloud information, obtain described visual information, specifically include:

Described impact point cloud information is carried out geometry vertex position optimization and normal direction calculates, obtain the first knot Really；

Described first result is carried out surface fitting and triangle gridding processes, obtain the second result；

According to described second result, obtain described visual information.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, Described presentation information is processed, obtains the first markup information, specifically include:

Described presentation information is carried out Digital Image Processing analysis, obtains described first markup information.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, According to described first markup information and described presentation information, obtain including first cloud letter of geological information Breath, specifically includes:

According to described first markup information, described presentation information is processed, obtain and described presentation information The coordinate information of corresponding object；

According to described coordinate information, generate first the cloud information including described geological information.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, According to described first markup information and described presentation information, obtain including the second point cloud letter of texture information Breath, specifically includes:

Use pointwise and/or the mode of image synthesis, according to described first markup information, to described presentation Information carries out extracting information processing, obtains including the second point cloud information of texture information.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, According to described second markup information and described impact point cloud information, obtain visual information, including:

According to described second markup information and described impact point cloud information, calculate the surface normal letter of object Breath；

Visual information is obtained according to described surface normal information.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, Before the presentation information that described four-dimensional spacetime model characterizes being presented, described method also includes:

Described four-dimensional spacetime model and the first space-time model are merged, obtains target four-dimensional spacetime mould Type, described first space-time model is for characterizing the presentation letter of the object presenting place residing for multimedia messages Breath；

The presentation information that described four-dimensional spacetime model characterizes is presented, specifically includes:

According to described target four-dimensional spacetime model, the presentation information that described four-dimensional spacetime model is characterized The presentation information characterized with described first space-time model presents.

First space-time model, second space-time model of described four-dimensional spacetime model with local terminal are melted Closing, obtain target four-dimensional spacetime model, described first space-time model presents multimedia messages for sign The presentation information of the object in residing place, described second space-time model is for characterizing the presentation of dummy object Information；

According to described target four-dimensional spacetime model, the presentation letter that described four-dimensional spacetime model is characterized The table that breath, the presentation information that characterized of described first space-time model and described second space-time model are characterized Image information presents.

Described four-dimensional spacetime model is additionally operable to characterize the sound field letter of the object corresponding with described presentation information Breath；

Described method also includes:

The sound field information that described four-dimensional spacetime model characterizes is played out.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, Before the presentation information that described four-dimensional spacetime model characterizes is presented, also include:

Determine present multimedia messages device just facing to；

The presentation information that described four-dimensional spacetime model characterizes is presented, including:

According to described just facing to, by described four-dimensional spacetime model characterize presentation information present.

Alternatively, in the method presenting multimedia messages according to any of the above-described embodiment of the present invention, Described method also includes:

Determine present multimedia messages device just facing to destination multimedia information；

Just feed back to send setting of described presentation information facing to described destination multimedia information by described Standby.

According to a second aspect of the present invention, it is provided that a kind of device presenting multimedia messages, including:

Acquiring unit, is used for obtaining presentation information, and described presentation information includes observing with the naked eye And/or the electromagnetic field spectral information for characterizing object that can collect with equipment；

Unit set up by model, for setting up for characterizing described presentation letter according to the presentation information got The four-dimensional spacetime model of breath, described four-dimensional spacetime model has and can characterize in time in digitized form The attribute of the described presentation information of change；

Display unit, for presenting the presentation information that described four-dimensional spacetime model characterizes.

Alternatively, in the device presenting multimedia messages according to the abovementioned embodiments of the present invention, described Presentation information also includes sound field information that is that can feel and/or that can collect with equipment with ear；

Described device also includes sound field information computing unit, for calculating and institute according to described presentation information State the sound field information of object corresponding to presentation information；

Described model is set up unit and is set up for characterizing described presentation information according to described presentation information During four-dimensional spacetime model, particularly as follows:

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described model is set up unit and is included the first markup information signal generating unit, some cloud information generating unit, some cloud letter Breath integrated unit, visual information signal generating unit, four-dimensional spacetime model generation unit, wherein:

Described first markup information signal generating unit, for processing described presentation information, obtains first Markup information；

Described some cloud information generating unit, is used for according to described first markup information and described presentation information, Obtain including first cloud information of geological information, including the second point cloud information of texture information；

Described some cloud information fusion unit, for by described first cloud information and described second point cloud information Merge, obtain impact point cloud information；

Described visual information signal generating unit, for obtaining visual information according to described impact point cloud information；

Described four-dimensional spacetime model generation unit, for obtaining spatial model according to described visual information, will Merge for spatial model the most in the same time, according to merging the space module obtained, described first mark Note information and described second markup information, obtain described four-dimensional spacetime model.

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described presentation information also includes sound field that is that can feel and/or that can collect with equipment with ear Information；

Described device also includes sound field information computing unit, for calculating described table according to described presentation information The sound field information of the object that image information is corresponding；

When described four-dimensional spacetime model generation unit obtains spatial model according to described visual information, specifically For:

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described some cloud information generating unit includes the second markup information signal generating unit, for believing described impact point cloud Breath processes, and obtains the second markup information；

Described visual information signal generating unit is according to described impact point cloud information, when obtaining visual information, specifically For:

According to described second markup information and described impact point cloud information, obtain described visual information

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described visual information signal generating unit is additionally operable to:

According to described second result, obtain described visual information.

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described presentation information is processed by described first markup information signal generating unit, obtains the first markup information Time, particularly as follows:

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described some cloud information generating unit according to described first markup information and described presentation information, including During first cloud information of geological information, particularly as follows:

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described some cloud information generating unit according to described first markup information and described presentation information, including During the second point cloud information of texture information, particularly as follows:

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described visual information signal generating unit according to described second markup information and described impact point cloud information, depending on During visual information, particularly as follows:

According to described second markup information and described impact point cloud information, calculate body surface normal direction information；

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described device also includes Model Fusion unit, for by described four-dimensional spacetime model and the first space-time model Merging, obtain target four-dimensional spacetime model, described first space-time model presents many matchmakers for sign The presentation information of the object in place residing for body information；

The presentation information that described four-dimensional spacetime model characterizes is being carried out in now by described display unit, Particularly as follows:

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described device also includes Model Fusion unit, for during by the first of described four-dimensional spacetime model and local terminal Empty model, the second space-time model merge, and obtain target four-dimensional spacetime model, described first space-time Model is for characterizing the presentation information of the object presenting place residing for multimedia messages, described second space-time Model is for characterizing the presentation information of dummy object；

Described device also includes broadcast unit, for the sound field letter characterized by described four-dimensional spacetime model Breath plays out.

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described device also includes processing unit, for determine present multimedia messages device just facing to；

According to described forward towards, by described four-dimensional spacetime model characterize presentation information present.

Alternatively, in the device presenting multimedia messages according to any of the above-described embodiment of the present invention, Described device also includes processing unit, determine present multimedia messages device just facing to many with target Media information；

Described device also includes feedback unit, for by described just facing to described destination multimedia information Feed back to send the equipment of described presentation information.

In the embodiment of the present invention, disclose a kind of method and apparatus presenting multimedia messages: obtain table Image information is that described presentation information includes observing with the naked eye and/or can collect with equipment For characterizing the electromagnetic field spectral information of object；Set up for characterizing according to the presentation information got Stating the four-dimensional spacetime model of presentation information, described four-dimensional spacetime model has can table in digitized form Levy the attribute of time dependent described presentation information；The presentation that described four-dimensional spacetime model is characterized Information presents；In this scenario, four-dimensional spacetime model have can characterize in digitized form with The attribute of the presentation information of time change, so, to some extent solves the presentation information presented The problem that there is time delay, therefore, this programme to some extent solves tool present in prior art There is the defect of time delay.

Described above is only the general introduction of technical solution of the present invention, in order to better understand the skill of the present invention Art means, and can being practiced according to the content of description, and in order to allow, the present invention's is above and other Objects, features and advantages can become apparent, below especially exemplified by the detailed description of the invention of the present invention.

Accompanying drawing explanation

By reading the detailed description of hereafter preferred implementation, various other advantage and benefit for this Field those of ordinary skill will be clear from understanding.Accompanying drawing is only used for illustrating the purpose of preferred implementation, And it is not considered as limitation of the present invention.And in whole accompanying drawing, be denoted by the same reference numerals Identical parts.In the accompanying drawings:

Figure 1A is the method flow diagram presenting multimedia messages according to an embodiment of the invention；

Figure 1B is the another kind of schematic flow sheet setting up four-dimensional spacetime model according to an embodiment of the invention；

Fig. 1 C is that the another kind of flow process of the method presenting multimedia messages according to an embodiment of the invention is shown It is intended to；

Fig. 2 A is the schematic diagram of the device presenting multimedia messages according to an embodiment of the invention；

Fig. 2 B is a kind of schematic diagram of acquiring unit according to an embodiment of the invention；

Fig. 2 C is the another kind of schematic diagram of acquiring unit according to an embodiment of the invention；

Fig. 2 D is the top view of acquiring unit according to an embodiment of the invention；

Fig. 2 E is the side view of acquiring unit according to an embodiment of the invention；

Fig. 3 A is a kind of scene schematic diagram provided according to embodiments of the invention；

Fig. 3 B is the another kind of scene schematic diagram provided according to embodiments of the invention；

Fig. 3 C is the another kind of scene schematic diagram provided according to embodiments of the invention；

Fig. 3 D is the another kind of scene schematic diagram provided according to embodiments of the invention；

Fig. 3 E is the another kind of scene schematic diagram provided according to embodiments of the invention；

Fig. 3 F is the another kind of scene schematic diagram provided according to embodiments of the invention.

Detailed description of the invention

In order to make the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing to this Invention is described in further detail, it is clear that described embodiment is only that a part of the present invention is implemented Example rather than whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art exist Do not make all other embodiments obtained under creative work premise, broadly fall into present invention protection Scope.

The proposed by the invention method and apparatus processing multimedia messages can be applied in following scene, But it is not limited to following scene:

Real-time communication scene: such as, first collects real-time Transmission to second self and surrounding, Second roams and and first interaction wherein；The most such as, first and second are all self and surrounding collection Real-time Transmission of getting off is to the other side, at the environment or any 3rd including that two people are physically residing alternatively Roaming and interaction in side's environment；

Remote observation and monitoring scene；

Operative scenario: such as, individual or many people telecommuting, the most immersively meeting, remote Journey is immersively cooperated or helps client to solve problem, or, immersively train；

Education scene: for example, it is possible to immersively virtual classroom and and teacher in virtual environment Middle interaction；

Medical treatment scene: such as, tele-medicine and and doctor's interaction in virtual environment；

Business scenario: such as, teleshopping and and businessman interaction, comprehensive examination in virtual environment Clothing mirror；

Sports Scene: such as, individual or many people and dash champion compete dash in virtual environment；

Entertainment Scene: such as, individual or game that many people is in Virtual Space, can be immersively Participating television is live or interactive with film appearances；

Personal lifestyle scene: such as, the record of four-dimensional diary and projection, remote visit museum, far Journey accompanies household or house pet, application of being remotely grown up；

Can also be used for being applied to following scene:

The scene that virtual reality or augmented reality content generate: include film, TV, play, regard Frequently content production；

Place, special time space is done the scene of four-dimensional historical record.

Figure 1A shows the schematic flow sheet of a kind of method presenting multimedia messages according to the present invention, Detailed process is as follows:

Step 100: obtain presentation information, that described presentation information includes observing with the naked eye and/ The electromagnetic field spectral information for characterizing object that maybe can collect with equipment.

In the embodiment of the present invention, when step 100 obtains presentation information, can be self to gather to obtain , it is also possible to it is to send over after other equipment collections, is not specifically limited at this.

In the embodiment of the present invention, the presentation information got can be different spaces point and/or different time The presentation information that point gets.

Step 110: set up for the four-dimension characterizing described presentation information according to the presentation information got Space-time model, described four-dimensional spacetime model has can characterize time dependent institute in digitized form State the attribute of presentation information.

In the embodiment of the present invention, four-dimensional spacetime model the most at least includes as properties:

Locus attribute: can refer to that the every bit on any time object does not changes over time at one Coordinate system under coordinate；

Appearance attribute: texture and spectral signature (such as, the face of any time body surface can be referred to Color), the geometrical property (such as, normal direction, curvature, smoothness etc.) of body surface；

Voice attribute；

Movement properties: the movement velocity vector of every bit on any time object, acceleration can be referred to Vector；Or, it is also possible to refer to regard the angular velocity vector of every part of rigid body on object as, angle adds Velocity；

Other attribute: can refer to that the classification of object, identity, material, mutual relation etc. are all can be by table At least one in the information that image information and presentation information are inferred over time.

In form, four-dimensional spacetime model is to be present in storage medium with digitized data mode, this The form of kind can be stored, and presents, retrieval, editor, transmission, encryption, and is used for higher level Intelligent use.

In the embodiment of the present invention, after setting up four-dimensional spacetime model, further, it is also possible to amendment, Strengthen, optimize four-dimensional spacetime model.

In actual applications, presentation information is except that can include observing with the naked eye and/or can be with setting Standby collect outside the electromagnetic field spectral information characterizing object, it is also possible to include sound field information, The most described presentation information also includes sound that is that can feel and/or that can collect with equipment with ear Field information, now, before setting up described four-dimensional spacetime model according to the presentation information got, described Method also includes operating as follows:

Sound field information described in the embodiment of the present invention refers not only to audio-frequency information itself, the most implicit Sound source spatial positional information inside, can include collectable information of acoustic wave and/or ultrasound information.

In the embodiment of the present invention, set up for characterizing described presentation information according to the presentation information got During four-dimensional spacetime model, it is alternatively possible in the following way:

In the embodiment of the present invention, that described presentation information also includes can feeling with ear and/or can The sound field information collected with equipment；

Before obtaining spatial model according to described visual information, described method also includes operating as follows:

In the embodiment of the present invention, when obtaining spatial model according to described visual information, it is alternatively possible to adopt By following manner:

In the embodiment of the present invention, described first cloud information and described second point cloud information are merged, After obtaining impact point cloud information, before obtaining described visual information, described method also includes operating as follows:

In the embodiment of the present invention, according to described second markup information and described impact point cloud information, obtain institute When stating visual information, it is alternatively possible in the following way:

According to described second result, obtain described visual information.

In the embodiment of the present invention, described presentation information is processed, when obtaining the first markup information, It is alternatively possible in the following way:

In the embodiment of the present invention, when presentation information is carried out Digital Image Processing analysis, it is alternatively possible to In the following way:

Split presentation information, detect, follow the tracks of, identification etc. processes.

In the embodiment of the present invention, split, detect, follow the tracks of, identify and there is no clearly between these steps Ordering relation, for example, it is possible to presentation information is first split, then detect；Can also first detect, then Segmentation.In order to improve the accuracy of the first markup information obtained, can circulate execution several times segmentation, Detect, follow the tracks of and identify.Such as, after having performed once to split, detect, follow the tracks of and identify, root According to current results, then perform at least one to take turns segmentation, detect, follow the tracks of and identify, so can improve essence Degree.

In the embodiment of the present invention, segmentation can refer to divide the image into as prospect and background, such as, segmentation Become sky, ground or other, detection can refer to detect pedestrian, detection car plate, tracking can refer to The arm motion of track people, identification can refer to identify vehicle.

In the embodiment of the present invention, according to described first markup information and described presentation information, including During first cloud information of geological information, can be in the following way:

In the embodiment of the present invention, the most in the same time may with the coordinate information of the object corresponding to presentation information Corresponding to different coordinate systems, now, in order to improve the accuracy of first the cloud information obtained, obtain After the coordinate information of the object corresponding to presentation information, it is also possible to different local the most in the same time are sat The coordinate information of the object under mark system is fused under same coordinate system, then according to being fused to same seat Coordinate information under mark system, generates first the cloud information including geological information.

In the embodiment of the present invention, according to described first markup information and described presentation information, including During the second point cloud information of texture information, it is alternatively possible in the following way:

In the embodiment of the present invention, according to described second markup information and described impact point cloud information, depending on During visual information, it is alternatively possible in the following way:

The present invention gives the detailed process setting up four-dimensional spacetime model, refering to shown in Figure 1B, according to table Image information obtains the first markup information and sound field information, and obtains according to presentation information and the first markup information Some cloud information and second point cloud information, obtain target by first cloud information and second point cloud information fusion Point cloud information, obtains the second markup information according to impact point cloud information, and carries out several to impact point cloud information What vertex position optimization and normal direction are calculated the first result, and the first result is carried out surface fitting and triangle Gridding processes, and obtains the second result, obtains visual information according to the second result and the second markup information, By visual information and sound field information fusion, obtain spatial model, after being merged by spatial model Spatial model, spatial model, the first markup information and the second markup information after merging process and obtain four Dimension space-time model.

Step 120: the presentation information that described four-dimensional spacetime model characterizes is presented.

In the embodiment of the present invention, further, it is also possible to this one end of device of multimedia messages will be presented Scene present, therefore, further, by described four-dimensional spacetime model characterize presentation letter Before breath presents, described method also includes operating as follows:

Now, the presentation information that described four-dimensional spacetime model characterizes is carried out in now, alternatively, Can be in the following way:

Such as, the scene that the presentation information of four-dimensional spacetime model sign is corresponding is scene by the sea, the The scene corresponding to presentation information that one space-time model characterizes is the scene in desk office, at this In the case of Zhong, the scene presented can be the scene of this fusion being seashore before desk.

Further, it is also possible to human body and object detection, follow the tracks of and identify: can be by actual physical district Territory is superimposed to virtual region, and such as, observer wears the VR helmet in the face of a grassland, his place in reality There is wall in room, and this time " based on object detection " just can be the information of the wall of actual physical It is superimposed to the grassland of the VR helmet, presents the translucent walls in a grassland；The most such as, staff detects, Can the most then virtual hands superposition be entered in four dimension module by the gestures detection of real hands, say, that Some virtual scenes, therefore, the presentation letter characterized by described four-dimensional spacetime model can also be merged Before breath presents, described method also includes operating as follows:

Such as, the scene that the presentation information of four-dimensional spacetime model sign is corresponding is scene by the sea, the The scene corresponding to presentation information that one space-time model characterizes is the scene in desk office, at this In the case of Zhong, the scene presented can be the scene of this fusion being seashore before desk, enters one Step, want on the desk presented, put a basin flower, but reality is the freshest on desk Colored, at this point it is possible to characterize flower with the second space-time model, and by described four-dimensional spacetime model and basis First space-time model of end, the second space-time model merge, and obtain target four-dimensional spacetime model, this The scene presented in the case of Zhong can be to be this that put flower on seashore, and desk before desk Plant the scene merged.

In the embodiment of the present invention, the scene presented is possible not only to picture, it is also possible to have sound, therefore, Described presentation information also includes sound field that is that can feel and/or that can collect with equipment with ear Information, described four-dimensional spacetime model is additionally operable to characterize the sound field letter of the object corresponding with described presentation information Breath；Now, described method also includes operating as follows:

In the embodiment of the present invention, in order to improve the scene corresponding to the presentation information presented and real scene Similarity, when presenting the presentation information that four-dimensional spacetime model characterizes, be with reference to presenting multimedia messages The front orientation information of device, therefore, the presentation information that described four-dimensional spacetime model characterizes is entered Before row presents, also include operating as follows:

Determine present multimedia messages device just facing to；

In the embodiment of the present invention, when determining the front orientation information of the device presenting multimedia messages, optional , can be in the following way:

The inertial navigation being associated with the device presenting multimedia messages is carried out attitude algorithm, is presented The front orientation information of the device of multimedia messages.

Wherein, inertial navigation can be in gyroscope, magnetometer, accelerometer any one or appoint Meaning combination.

In the embodiment of the present invention, the precision of observer's part interested can be improved selectively, Further, also include operating as follows:

Such as, the scene corresponding to presentation information having seabeach, personage, and sailing boat, presenting if held The eyeball fixes of the user of the device of multimedia messages personage, then using personage as destination multimedia information. So, the device of four-dimensional spacetime model is sent when obtaining presentation information, it is possible to the only presentation of personage Information, can not include the presentation information of sailing boat.

In the embodiment of the present invention, when determining destination multimedia information, can be based on presenting multimedia messages " eyeball " of the photographic head of device determines.

It should be noted that the first space-time model described in the embodiment of the present invention and the second space-time model Can be that the device presenting multimedia messages pre-builds, it is also possible to be to set up in real time, or, Can also be that other devices pre-build, or set up in real time and be sent to present multimedia messages Device, it is not specifically limited at this.

In the embodiment of the present invention, in some scene, can only present what four-dimensional spacetime model characterized Presentation information, such as, in telecommuting or telecommunication scene, present the device of multimedia messages simply Want to experience and send " the most long-range " scene that the device of four-dimensional spacetime model is sent, now, only Present the presentation information that four-dimensional spacetime model characterizes.In some scene, when presenting the four-dimension On the basis of the presentation information that empty model characterizes, it is also possible to present further the first space-time model or The presentation information that second space-time model is characterized, presenting this one end of presentation information, also to add some virtual Stage property, such as, the device presenting multimedia messages not only to experience the device sending four-dimensional spacetime model The scene sent, also to add virtual item in this scenario, such as, wave conveniently aloft Draw a blank, or in order to play, add some virtual item and (e.g., send out " lightning " together on hand to go Hit the stone in scene).

In the embodiment of the present invention, further, it is also possible to by the first markup information and/or the second mark letter Breath presents.

In the embodiment of the present invention, it is also possible to receive the four-dimensional spacetime model that multiple device sends respectively, Such as, the presentation information receiving the first four-dimensional spacetime model sign that the first transmitting terminal sends is corresponding Scene be the Temple of Heaven, receive second transmitting terminal send second four-dimensional spacetime model characterize presentation Scene corresponding to information is Eiffel Tower, in now can by the Temple of Heaven and Eiffel Tower side by side in Existing.

The present invention gives the detailed process presenting four-dimensional spacetime model, refering to shown in Fig. 1 C, by the four-dimension Space-time model, the first space-time model and the second space-time model merge, and obtain target four-dimensional spacetime model, really Surely front orientation information and the destination multimedia information of the device of multimedia messages are presented, and according to just facing The presentation information that four-dimensional spacetime model characterizes is presented to information and target four-dimensional spacetime model, and by front Orientation information and destination multimedia information feed back to send the device of four-dimensional spacetime model.

Refering to shown in Fig. 2 A, the embodiment of the present invention also proposes a kind of device presenting multimedia messages, Including:

Acquiring unit 21, is used for obtaining presentation information, and described presentation information includes detecting by an unaided eye To and/or the electromagnetic field spectral information for characterizing object that can collect with equipment；

Unit 22 set up by model, for setting up for characterizing described presentation according to the presentation information got The four-dimensional spacetime model of information, described four-dimensional spacetime model has and can characterize at any time in digitized form Between the attribute of described presentation information of change；

Display unit 23, for presenting the presentation information that described four-dimensional spacetime model characterizes.

In the embodiment of the present invention, the electromagnetic field spectral information acquired in acquiring unit 21 can be object Launch, it is also possible to be object reflection, or can also be object refraction, do not do at this concrete Limit.

In the embodiment of the present invention, the electromagnetic field spectral information described by acquiring unit 21 can include nothing Line wave information, infrared ray information, visible ray information, ultraviolet information, X-ray information, and gal At least one in horse ray information, wherein, it is seen that optical information can include laser.

In the embodiment of the present invention, object corresponding to presentation information can be indoor and/or outdoor arbitrary regard Wild size and the object of angle.

In the embodiment of the present invention, when acquiring unit 21 obtains presentation information, per second obtain 24 frames extremely 120 frames.

In the embodiment of the present invention, the presentation information accessed by acquiring unit 21 can be different spaces The presentation information that point and different time points get.

Voice attribute；

Movement properties: can refer to that the movement velocity vector of every bit on any time object, acceleration are vowed Amount；Or, it is also possible to refer to regard the angular velocity vector of every part of rigid body on object as, angle is accelerated Degree vector；

Other attribute: can refer to that the classification of object, identity, material, mutual relation etc. are all can be by table At least one in the information that image information and presentation are inferred over time.

In the embodiment of the present invention, model is set up after unit 22 sets up four-dimensional spacetime model, further , it is also possible to revise, strengthen, optimize four-dimensional spacetime model.

In actual applications, presentation information is except that can include observing with the naked eye and/or can be with setting Standby collect outside the electromagnetic field spectral information characterizing object, it is also possible to include sound field information, Now, in the embodiment of the present invention, further, described presentation information also includes to feel with ear To and/or the sound field information that can collect with equipment；

Described device also includes sound field information computing unit 24, for according to described presentation information calculate with The sound field information of the object that described presentation information is corresponding；

Described model is set up unit 22 and is set up for characterizing described presentation information according to described presentation information Four-dimensional spacetime model time, particularly as follows:

In the embodiment of the present invention, alternatively, described model is set up unit 22 and is included that the first markup information is raw Become unit 22A, some cloud information generating unit 22B, some cloud information fusion unit 22C, visual information raw Become unit 22D, four-dimensional spacetime model generation unit 22E, wherein:

Described first markup information signal generating unit 22A, for processing described presentation information, obtains First markup information；

Described some cloud information generating unit 22B, for believing according to described first markup information and described presentation Breath, obtains including first cloud information of geological information, including the second point cloud information of texture information；

Described some cloud information fusion unit 22C, for by described first cloud information and described second point cloud Information merges, and obtains impact point cloud information；

Described visual information signal generating unit 22D, for obtaining visual information according to described impact point cloud information；

Described four-dimensional spacetime model generation unit 22E, for obtaining spatial model according to described visual information, To merge for spatial model the most in the same time, according to fusion obtain space module, described first Markup information and described second markup information, obtain described four-dimensional spacetime model.

Further, described presentation information also include can feeling with ear and/or equipment can be used The sound field information collected；

Described device also includes sound field information computing unit 24, described for calculating according to described presentation information The sound field information of the object that presentation information is corresponding；

When described four-dimensional spacetime model generation unit 22E obtains spatial model according to described visual information, tool Body is:

Further, described some cloud information generating unit 22B includes the second markup information signal generating unit 25, For described impact point cloud information is processed, obtain the second markup information；

Now, described visual information signal generating unit 22D, according to described impact point cloud information, obtains vision letter During breath, particularly as follows:

Further, described visual information signal generating unit 22D is additionally operable to:

According to described second result, obtain described visual information.

Alternatively, described presentation information is processed by described first markup information signal generating unit 22A, When obtaining the first markup information, particularly as follows:

In the embodiment of the present invention, described presentation information is carried out by described first markup information signal generating unit 22A When Digital Image Processing is analyzed, split presentation information, detect, follow the tracks of, identification etc. processes.

In the embodiment of the present invention, described some cloud information generating unit 22B is according to described first markup information With described presentation information, when obtaining first the cloud information including geological information, particularly as follows:

In the embodiment of the present invention, the most in the same time may with the coordinate information of the object corresponding to presentation information Corresponding to different coordinate systems, now, in order to improve the accuracy of first the cloud information obtained, obtain After the coordinate information of the object corresponding to presentation information, some cloud information generating unit 22B can also be by The coordinate information of the object under different local coordinate systems the most in the same time is fused under same coordinate system, so Afterwards according to being fused to the coordinate information under same coordinate system, generate first the cloud letter including geological information Breath.

Alternatively, described some cloud information generating unit 22B is according to described first markup information and described table Image information, when obtaining the second point cloud information including texture information, particularly as follows:

Alternatively, described visual information signal generating unit 22D is according to described second markup information and described target Point cloud information, when obtaining visual information, particularly as follows:

In the embodiment of the present invention, alternatively, described device also includes Model Fusion unit 26, and being used for will Described four-dimensional spacetime model and the first space-time model merge, and obtain target four-dimensional spacetime model, institute State the first space-time model for characterizing the presentation information of the object presenting place residing for multimedia messages；

Now, the presentation information that described four-dimensional spacetime model characterizes is being entered by described display unit 23 Row in now, particularly as follows:

Such as, the scene that the presentation information of four-dimensional spacetime model sign is corresponding is scene by the sea, the The scene corresponding to presentation information that one space-time model characterizes is the scene in desk office, at this In the case of Zhong, the scene that display unit 23 presents can be to be this fusion on seashore before desk Scene.

Further, it is also possible to human body and object detection, follow the tracks of and identify: can be by actual physical district Territory is superimposed to virtual region, and such as, observer wears the VR helmet in the face of a grassland, his place in reality There is wall in room, and this time " based on object detection " just can be the information of the wall of actual physical It is superimposed to the grassland of the VR helmet, presents the translucent walls in a grassland；The most such as, staff detects, Can the most then virtual hands superposition be entered in four dimension module by the gestures detection of real hands, say, that Can also merge some virtual scenes, described device also includes Model Fusion unit 26, for by institute The first space-time model, the second space-time model of stating four-dimensional spacetime model and local terminal merge, and obtain mesh Mark four-dimensional spacetime model, described first space-time model presents place residing for multimedia messages for characterizing The presentation information of object, described second space-time model is for characterizing the presentation information of dummy object.

Such as, the scene that the presentation information of four-dimensional spacetime model sign is corresponding is scene by the sea, the The scene corresponding to presentation information that one space-time model characterizes is the scene in desk office, at this In the case of Zhong, the scene that display unit 23 presents can be to be this fusion on seashore before desk Scene, further, want on the desk presented, put a basin flower, but in reality handle official business Flower is not had, at this point it is possible to characterize flower with the second space-time model, and by the described four-dimension on table Space-time model merges with the first space-time model, second space-time model of local terminal, obtains target four-dimensional Space-time model, the scene that display unit 23 presents in this case can be before desk to be seashore, And on desk, put the scene of this fusion of flower.

In the embodiment of the present invention, the scene presented is possible not only to picture, it is also possible to have sound, therefore, Described presentation information also includes sound field that is that can feel and/or that can collect with equipment with ear Information；

Described device also includes broadcast unit 27, for the sound field characterized by described four-dimensional spacetime model Information plays out.

In the embodiment of the present invention, in order to improve the scene corresponding to the presentation information presented and real scene Similarity, display unit 23, be with reference to presenting when presenting the presentation information that four-dimensional spacetime model characterizes The front orientation information of the device of multimedia messages, therefore, further, described device also includes processing Unit 28, for determine present multimedia messages device just facing to；

Now, the presentation information that described four-dimensional spacetime model characterizes is being carried out by described display unit 23 In now, particularly as follows:

In the embodiment of the present invention, processing unit 28 determine present multimedia messages device just facing to letter During breath, optionally, can be in the following way:

In the embodiment of the present invention, the precision of observer's part interested can be improved selectively, Further, described device also includes processing unit 28, determines and is just presenting the device of multimedia messages Facing to destination multimedia information；

Described device also includes feedback unit 29, for just believing described facing to described destination multimedia Breath feeds back to send the equipment of described presentation information.

In the embodiment of the present invention, when processing unit 28 determines destination multimedia information, can be many based on presenting " eyeball " of the photographic head of the device of media information determines.

In the embodiment of the present invention, in some scene, display unit 23 can only present four-dimensional spacetime mould The presentation information that type is characterized, such as, in telecommuting or telecommunication scene, presents multimedia messages Device just hope and experience " the most long-range " field that sent of device sending four-dimensional spacetime model Scape, now, only presents the presentation information that four-dimensional spacetime model characterizes.In some scene, Display unit 23 is on the basis of presenting the presentation information that four-dimensional spacetime model characterizes, it is also possible to enter The presentation information that one step presents the first space-time model or the second space-time model is characterized, presents presentation letter Ceasing this one end and also to add some virtual item, such as, the device presenting multimedia messages not only wants body Issue after examination and approval the scene sending the device of four-dimensional spacetime model to be sent, also to add virtual road in this scenario Tool, such as, wave conveniently and aloft draws a blank, or in order to play, add some virtual item (e.g., send out " lightning " together on hand and remove the stone hitting in scene).

In the embodiment of the present invention, acquiring unit 21 is when other devices obtain presentation information, it is also possible to Receive the presentation information that multiple device sends respectively, such as, receive the table that the first transmitting terminal sends The scene that image information is corresponding is the Temple of Heaven, receives the scene that the presentation information of the second transmitting terminal transmission is corresponding For Eiffel Tower, in the Temple of Heaven and Eiffel Tower being presented side by side now.

In the embodiment of the present invention, acquiring unit 21 can be to include column type, cuboid, prismatic, ring Type, spherical, and any one shape in hemispherical, including at least one photographic head, photographic head is permissible For color photographic head, depth camera or infrared camera.

Further, acquiring unit 21 can also include at least one mike, as shown in Fig. 2 B, 2C, Wherein, Fig. 2 D is Fig. 2 B or the top view of Fig. 2 C, and Fig. 2 E is Fig. 2 B or the side-looking of Fig. 2 C Figure.

Alternatively, acquiring unit 21 includes 8 pairs of color photographic head and 8 mikes, wherein: top has 1 pair of color photographic head, visual angle are respectively 180 degree；Side has 6 pairs of color photographic head, visual angles to be respectively 70 degree； Bottom has 1 pair of color photographic head, visual angle to be respectively 180 degree；1 Mike is had in the middle of every 1 pair of photographic head Wind.

Alternatively, acquiring unit 21 can also be following form:

1 or 1 pair of color photographic head are arranged at top, and visual angle is 45～180 degree；Side have 2 or Person 8 is to color photographic head, and visual angle is respectively 45～180 degree；1 or 1 pair of color shooting are arranged at bottom Head, visual angle is 45～180 degree；There is 1 mike, or, have 1 in the middle of every 1 pair of photographic head Individual mike, alternatively, the quantity of mike is between 1～8.

In the embodiment of the present invention, alternatively, the photographic head at top can also be three-dimensional camera, multifocal One in photographic head, structure light video camera head, flight time (ToF) photographic head, light field photographic head group or Person's combination in any.

In the embodiment of the present invention, alternatively, the photographic head of side can be that three-dimensional camera, many focal lengths are taken the photograph As the one in head, structure light video camera head, flight time (ToF) photographic head, light field photographic head group or appoint Meaning combination.

Such as, acquiring unit 21 is column type, has six pairs of binocular camera at cylindrical side surface, each Camera view is 70 degree；It is respectively arranged with a pair binocular camera, each shooting at cylinder end face and bottom surface The head visual field is 180 degree, it is possible to achieve the visual field of stereoscopic full views covers, and all photographic head are all through thing First calibration has also obtained parameter matrix.Acquiring unit 21 can be with built-in eight mikes.

In the embodiment of the present invention, color photographic head can by optical lens, sensor devices (Image Sensor), ISP (Image Signal Processing Unit, image signal processing chip) forms.

VPU (Vision Processing Unit, vision processor) can include that unit 22 set up by model With processing unit 23, wherein, photographic head can be by MIPI (Mobile Industry Processor Interface, mobile Industry Processor Interface) it is connected to VPU chip, it is right that VPU chip processes two The data that photographic head is transmitted through, are four VPU chips inside such a cylinder.

In the embodiment of the present invention, model is set up unit 22 and can be included processor, video card, internal memory, show Deposit, flash memory, hard disk, be wirelessly transferred, wire transmission and multiple Bus Interface Chip.

Shown below is the scene that the embodiment of the present invention is suitable for.

Scene shown in Fig. 3 A is: first is in the first scene, and second is in the second scene, and first and second are passed through The method that present example provides makes the surrounding of first and first " remotely present " face in second in real time Before, second can carry out interaction with first.

Further, four-dimensional spacetime model can also be first stored in storage and sets by device for processing muti-medium information In Bei, the device that can receive and process four-dimensional spacetime model that second is held can obtain from storage device Take four-dimensional spacetime model, as shown in Figure 3 B, in this case, scene that second institute " sees " and figure The scene seen in the case of shown in 3A can be the same.

When four-dimensional spacetime model is stored in storage device by device for processing muti-medium information, first can also Hold the device that can receive and process four-dimensional spacetime model, from storage device, obtain four-dimensional spacetime model, Perceive oneself in the first scene residing for one time point of past, as shown in Figure 3 C.

Scene shown in Fig. 3 D is: first is in the first scene, and second is in the second scene, and first and second are passed through Present example one makes the surrounding of first and first " remotely present " in face of second in real time, and second can To carry out interaction with first；First and second realize two-way " remotely reality " in real time by present example one and " mix Close reality ", what first was perceived is the first scene and the superposition of second at first place, and what second was perceived is First and first scene at first place；It should be noted that first and second are all right for the scene of wanted perception Having multiple choices, both sides can select the first scene seeing first place can also select to see second place Second scene, or see the 3rd scene at other side place；First and second it can be seen that same true or Person's virtual scene is it can also be seen that different true or virtual scene.

Scene shown in Fig. 3 E is: first realizes telecommuting by embodiment provided by the present invention.

Scene shown in Fig. 3 F is: first and second can realize sense by embodiment provided by the present invention To virtual environment, further, it is also possible to realize interaction, as " on the spot in person ".

Method and apparatus is not solid with any certain computer, virtual system or miscellaneous equipment provided herein Have relevant.Various general-purpose systems can also be used together with based on example in this.As described above, Construct the structure required by this kind of device to be apparent from.Additionally, the present invention is also not for any specific Programming language.It is understood that, it is possible to use various programming languages realize the content of invention described herein, And the description done language-specific above is the preferred forms in order to disclose the present invention.

In description mentioned herein, illustrate a large amount of detail.It is to be appreciated, however, that this Inventive embodiment can be put into practice in the case of not having these details.In some instances, not It is shown specifically known method, structure and technology, in order to do not obscure the understanding of this description.

Similarly, it will be appreciated that in order to simplify the disclosure and help understand in each inventive aspect one Or multiple, above in the description of the exemplary embodiment of the present invention, each feature of the present invention is sometimes It is grouped together in single embodiment, figure or descriptions thereof.But, should be by the disclosure Method be construed to reflect an intention that i.e. the present invention for required protection require ratio in each claim The middle more feature of feature be expressly recited.More precisely, as the following claims reflect, Inventive aspect is all features less than single embodiment disclosed above.Therefore, it then follows be embodied as Claims of mode are thus expressly incorporated in this detailed description of the invention, the most each claim itself All as the independent embodiment of the present invention.

Those skilled in the art are appreciated that and can carry out the module in the device in embodiment certainly Change adaptively and they are arranged in one or more devices different from this embodiment.Permissible Some block combiner in embodiment are become a module or unit or assembly, and in addition can be them It is divided into multiple submodule or subelement or sub-component.Except in such feature and/or process or module Outside at least some excludes each other, can use any combination that (this specification is included adjoint right Require, summary and accompanying drawing) disclosed in all features and so disclosed any method or equipment All processes or unit are combined.Unless expressly stated otherwise, this specification (includes adjoint right Require, summary and accompanying drawing) disclosed in each feature can by provide identical, equivalent or similar purpose replace Replace for feature.

Although additionally, it will be appreciated by those of skill in the art that embodiments more described herein include it Some feature included in its embodiment rather than further feature, but the group of the feature of different embodiment Close and mean to be within the scope of the present invention and formed different embodiments.Such as, in claim In book, one of arbitrarily can mode using in any combination of embodiment required for protection.

Each device embodiment of the present invention can realize with hardware, or to process at one or more The software module run on device realizes, or realizes with combinations thereof.Those skilled in the art should Understand, microprocessor or digital signal processor (DSP) can be used in practice to realize basis The some or all functions of the some or all modules in the device of the embodiment of the present invention.The present invention is also Can be implemented as part or all the device program for performing method as described herein (such as, Computer program and computer program).The program of such present invention of realization can be stored in calculating On machine computer-readable recording medium, or can be to have the form of one or more signal.Such signal can be from Download on internet website and obtain, or provide on carrier signal, or provide with any other form.

The present invention will be described rather than limits the invention to it should be noted above-described embodiment, And those skilled in the art can design replacement in fact without departing from the scope of the appended claims Execute example.In the claims, should not will be located in any reference marks between bracket to be configured to right is wanted The restriction asked.Word " comprises " and does not excludes the presence of the element or step not arranged in the claims.It is positioned at unit Word "a" or "an" before part does not excludes the presence of multiple such element.The present invention can be by means of Include the hardware of some different elements and realize by means of properly programmed computer.Listing If in the unit claim of equipment for drying, several in these devices can be by same hardware branch Specifically embody.Word first, second and third use do not indicate that any order.Can be by this A little word explanations are title.

Claims

1. the method presenting multimedia messages, including:

2. the method for claim 1, described presentation information also includes to feel with ear And/or the sound field information that can collect with equipment；

3. the method for claim 1, sets up for characterizing described according to the presentation information got The four-dimensional spacetime model of presentation information, specifically includes:

4. method as claimed in claim 3, described presentation information also includes to feel with ear And/or the sound field information that can collect with equipment；

5. method as claimed in claim 3, by described first cloud information and described second point cloud information Merging, after obtaining impact point cloud information, before obtaining described visual information, described method is also wrapped Include:

6. method as claimed in claim 5, believes according to described second markup information and described impact point cloud Breath, obtains described visual information, specifically includes:

According to described second result, obtain described visual information.

7. method as claimed in claim 3, processes described presentation information, obtains the first mark Note information, specifically includes:

8. the method as described in claim 3 or 7, according to described first markup information and described presentation Information, obtains including first cloud information of geological information, specifically includes:

9. the method as described in claim 3 or 7, according to described first markup information and described presentation Information, obtains including the second point cloud information of texture information, specifically includes:

10. present a device for multimedia messages, including: