CN109074680A - Realtime graphic and signal processing method and system in augmented reality based on communication - Google Patents

Abstract

Present specification discloses the method and system of realtime graphic and signal processing in augmented reality environment, for example, the video conference in the selected virtual environment of video conference participants.Particularly, by the way that real-life object and its practical environment separation to be extracted to the image information of real-life object in one or more pictures, one or more of images are captured in real time using the multi-mechanism based on characteristic.Then, by method integrated pixel-by-pixel, according to the images relations between the respective pixel of each image of each pixel of the image information of real-life object and the multiple images of virtual environment, by the real-life object integration of extraction into virtual environment.Described image relationship includes at least depth relationship or transparency relationship.

Description

Realtime graphic and signal processing method and system in augmented reality based on communication

Cross reference

This application claims application No. is 62/317554, it is filed on April 4th, 2016, it is entitled " existing based on enhancing The U.S. Provisional Patent Application of the realtime graphic and signal processing system of real communication ", and application No. is 62/322358, submit On April 14th, 2016, the U.S. of entitled " using the real-time immersive ad system of the communication based on augmented reality " The priority of temporary patent application, the two are collectively referred to herein in present specification.

Technical field

The invention mainly relates to the system of image and signal processing for the communication based on augmented reality (AR) system, set Standby and method.

Background technique

Real-time communication system or method based on AR are known.For example, apple Apple Photobooth^TMAllow user Created in virtual environment themselves photo or video (Photobooth be Apple exploitation operate in Mac An applet on OS or iOS, major function are to carry out digital photographing by camera iSight).Sometimes, virtually Environment can be a mobile environment.Google Google Hangout^TMIt is an audio and videoconference platform, it allows user Required individual scene is selected in videoconference session, and it is replaceable to allow user to wear cap, glasses and beard etc. Virtual objects.However, such existing system is rough, original from the perspective of vision.In real-life object (Google Hangout^TMUser) with the junction of virtual objects (for example, virtual background and wearable decorative article), often In the presence of the obvious and notch and flaw of highly significant sometimes.In the case where virtual environment includes movement elements, these defects are more Add obvious.

In some cases, professional photographer or film-maker are using green curtain shooting initial pictures or video, with as far as possible Reduce defect.Later period needs are largely edited, and dull background is replaced with virtual environment.This is one very time-consuming Expensive process again.In video conference (such as Google Hangout^TM) in, puzzlement can be also fallen into using the people of different language, and It is invalid to cause to link up.

This field is what is desired is that for the real-time image and other signals (such as audio signal) for handling the communication based on AR Improved system and method.

Summary of the invention

Present specification disclose for based on augmented reality (AR) system communication image and signal processing system, The method and system of device and method.

On the one hand, present specification discloses a kind of method for integrating real-life object and virtual environment.This method The following steps are included: the multidimensional image information of real-life object is provided in real time in computer equipment, wherein by real-time Real-life object is separated from its actual environment to extract image information in one or more images of capture, it is one Or multiple images are captured using based on the multi-mechanism of characteristic；It is received in real time by multiple images on a computing device The selection of the virtual environment of building；Each pixel of the image information of real-life object is provided in real time on a computing device Images relations between the respective pixel of each image of the multiple images of virtual environment, wherein images relations include at least deep Degree relationship or transparency relationship；And multiple integrated images are presented in real time on a computing device, wherein in integrated image Each pixel is divided into multiple layers, and uses the respective pixel in the image information of real-life object based on images relations Each layer is determined with each image in the multiple images of virtual environment.

In some embodiments, the method step occurs on user equipment or server or any computer equipment.

In some embodiments, this method further include identical computer equipment or another different computer equipment in real time The multidimensional image information of real-life object is extracted from one or more images of the real-life object in actual environment Step.

In some embodiments, the multi-mechanism based on characteristic is based between real-life object and actual environment one One or more differences in a characteristic, which includes the visual characteristic of real-life object or captures from video clip Actual environment visual characteristic, the real-time learning characteristic in real-life object or actual environment, or with real-life object Or the related preparatory learning characteristic of actual environment.

In some embodiments, visual characteristic includes spatial character, dimensional characteristic, shape feature, kinetic characteristic, color spy Property, illuminance and reflection characteristic, time response or transparency characteristic, depth characteristic, material property or their combination.

In some embodiments, spatial character includes three-dimensional space characteristic.

In some embodiments, real-time learning characteristic include color, shape, edge, light reflectivity, illuminance, exposure, Brightness, shade, highlighted, contrast, stain (blackpoint), movement, depth, material, contrast or their combination.

In some embodiments, preparatory learning characteristic include color, shape, edge, light reflectivity, illuminance, exposure, Brightness, shade, highlighted, contrast, stain, movement, depth, material, contrast or their combination.

In some embodiments, described extract includes that a wheel coarse extraction and a wheel finely extract.

In some embodiments, this method further includes the computer equipment or another different computer equipments by multiple figures As the step of building virtual environment.

In some embodiments, when for when constructing virtual environment, at least one image in multiple images to be modified.

In some embodiments, size, shape, picture quality, color, light, perspective, the visual effect of at least one image Or their combination is modified.

In some embodiments, this method further includes that the image information and void of real-life object are integrated based on images relations Each image of the multiple images in near-ring border, to create multiple integrated images.It is described it is integrated be in the computer equipment or It carries out in real time and pixel by pixel in another different computer equipment.

In some embodiments, multiple layers of pixel include foreground layer, middle layer and background layer.

In some embodiments, foreground layer is determined before determining middle layer or background layer.

In some embodiments, one or more ad elements are embedded in multiple integrated images.

In some embodiments, the image information of the extraction of one or more of ad elements and real-life object, Virtual environment or their combination are associated.

On the one hand, present specification discloses a kind of method for integrating two real-life objects and virtual environment.It should Method the following steps are included: provide the multidimensional image information of the first real-life object in computer equipment, wherein leading in real time It crosses and separates the first real-life object from the first actual environment in one or more pictures to extract image information, described one A or multiple images are captured in real time using the multi-mechanism based on characteristic；It is received in real time by multiple images on a computing device The selection of the virtual environment of building；Each pixel of the image information of real-life object is provided in real time on a computing device Images relations between the respective pixel of each image of the multiple images of virtual environment, wherein described image relationship is at least wrapped Include depth relationship or transparency relationship；The multi-dimensional map of second real-life object of extraction is provided in real time on a computing device As each image of the multiple images of each pixel and virtual environment of information and the image information of the second real-life object Respective pixel between new images relationship, wherein new images relationship include at least depth relationship or transparency relationship；It is calculating Multiple integrated images are presented on machine equipment in real time, wherein each pixel in integrated image is divided into multiple layers, wherein base In images relations and new images relationship, using the first real-life object, the second real-life object image information in pair Each image of the multiple images of pixel and virtual environment is answered to determine each layer.

On the one hand, present specification discloses a kind of computer system.The system includes one or more processors；And The addressable memory of one or more processors.The instruction that memory storage can be performed by one or more processors, to hold Row following steps: providing the multidimensional image information of real-life object in computer equipment in real time, wherein by one or Real-life object is separated from its actual environment to extract image information in multiple images, one or more of images are It is captured in real time using the multi-mechanism based on characteristic；It is received in real time on a computing device by the virtual of multiple images building The selection of environment；The each pixel and virtual environment of the image information of real-life object are provided in real time on a computing device Multiple images each image respective pixel between images relations, wherein described image relationship include at least depth relationship Or transparency relationship；Multiple integrated images are presented in real time on a computing device, wherein each pixel quilt in integrated image Multiple layers are divided into, wherein being based on images relations, use the respective pixel and virtual ring in the image information of real-life object Each images of the multiple images in border determines each layer.

In some embodiments, the further instruction that memory storage can be performed by one or more processors, by: in terms of It calculates in machine equipment or another different computer equipments in real time from the one or more figure of the real-life object in actual environment The multidimensional image information of real-life object is extracted as in.

In some embodiments, the further instruction that memory storage can be performed by one or more processors, by: in terms of Calculate the virtual environment that multiple images are constructed in machine equipment or another different computer equipments.

In some embodiments, the further instruction that memory storage can be performed by one or more processors, with: it is based on Images relations integrate real-life object in computer equipment or another different computer equipments in real time in a manner of pixel-by-pixel Image information and virtual environment multiple images in each image, to create multiple integrated images.

In some embodiments, the further instruction that memory storage can be performed by one or more processors, by: in terms of Calculate machine equipment provide in real time the second real-life object image information each pixel and multiple images each image New images relationship between respective pixel, wherein new images relationship includes at least depth relationship or transparency relationship.

In some embodiments, the further instruction that memory storage can be performed by one or more processors, with: it is based on Images relations and new images relationship integrate in a manner of pixel-by-pixel in real time in computer equipment or another different computer equipments The multiple images of the image information of real-life object, the new image information of the second real-life object and virtual environment Each image, to create multiple new integrated images.

In some embodiments, one or more ad elements are embedded in multiple integrated images.

In some embodiments, the figure of the extraction of one or more of ad elements and the first real-life object As information, the image information of the extraction of the second real-life object, the virtual environment or their combination are associated.

It is understood that method and system disclosed in present specification is also applied for more than two real-lifes pair As.

On the one hand, present specification discloses a kind of non-transitory computer-readable medium, comprising instruction, when it includes When instruction is subsequently can by computer device execution, these instructions can make computer processor: provide in real time really in computer equipment The multidimensional image information of life object, wherein by one or more pictures by real-life object and its actual environment point From image information is extracted, one or more of images are captured in real time using the multi-mechanism based on characteristic；It is calculating The selection of the virtual environment by multiple images building is received on machine equipment in real time；It provides in real time on a computing device true Figure between the respective pixel of each image of the multiple images of each pixel and virtual environment of the image information of life object As relationship, wherein described image relationship includes at least depth relationship or transparency relationship；It presents in real time on a computing device Multiple integrated images, wherein each pixel in integrated image is divided into multiple layers, wherein images relations are based on, using true Each images of the multiple images of respective pixel and virtual environment in the image information of life object determines each layer.

In some embodiments, the non-transitory computer-readable medium includes and can be performed by one or more processors Further instruction, with: in real time from the true life in actual environment in computer equipment or another different computer equipment The multidimensional image information of real-life object is extracted in one or more images of object living.

In some embodiments, the non-transitory computer-readable medium includes and can be performed by one or more processors Further instruction, with: construct virtual environment according to multiple images in computer equipment or another different computer equipment.

In some embodiments, the non-transitory computer-readable medium includes and can be performed by one or more processors Further instruction, with: be based on images relations, it is real in a manner of pixel-by-pixel in computer equipment or another different computer equipment When integrate each image in the image information of real-life object and the multiple images of virtual environment, to create multiple collection At image.

In some embodiments, the non-transitory computer-readable medium includes and can be performed by one or more processors Further instruction, with: computer equipment provide in real time each pixel of the image information of the second real-life object with New images relationship between the respective pixel of each image of multiple images, wherein new images relationship include at least depth relationship or Transparency relationship.

In some embodiments, the non-transitory computer-readable medium includes and can be performed by one or more processors Further instruction, with: be based on images relations and new images relationship, in computer equipment or another different computer equipment with Pixel-by-pixel mode integrate in real time the image information of real-life object, the new image information of the second real-life object and Each image of the multiple images of virtual environment, to create multiple new integrated images.

It is understood that any embodiment disclosed in present specification can be in where applicable with independent or any combination Mode be applied to any aspect of the invention.

One or more embodiments are illustrated in the accompanying drawings and the description below.Other features, aspect and potential excellent It will definitely be emerged from specification, drawings and the claims.

Detailed description of the invention

It will be understood by those skilled in the art that the purpose that attached drawing described below is merely to illustrate.Attached drawing is not intended to appoint Where formula limits the range of this introduction.

Figure 1A shows the block diagram of the example system for executing realtime graphic and signal processing.

Figure 1B shows the instantiation procedure for realtime graphic and signal processing.

Fig. 1 C, which is shown, provides the instantiation procedure of auxiliary data.

Fig. 1 D shows the example system for realtime graphic and signal processing.

Fig. 2 shows the exemplary steps and system of realtime graphic and signal processing on user equipment.

Fig. 3 A shows the exemplary step and system of realtime graphic and signal processing between user equipment and server.

Fig. 3 B shows the exemplary step for realtime graphic and signal processing between user equipment and server and is System.

Fig. 3 C shows the exemplary step for realtime graphic and signal processing between user equipment and server and is System.

Fig. 4 A is shown for carrying out realtime graphics and signal processing between user equipment and intermediate server at two Exemplary step and system.

Fig. 4 B is shown for carrying out realtime graphics and signal processing between user equipment and intermediate server at two Exemplary step and system.

Fig. 4 C is shown for carrying out realtime graphics and signal processing between user equipment and intermediate server at two Exemplary step and system.

Fig. 5 A shows the exemplary step for carrying out realtime graphic and signal processing between two user equipmenies and is System.

Fig. 5 B shows the exemplary step for carrying out realtime graphic and signal processing between two user equipmenies and is System.

Fig. 5 C shows the exemplary step for carrying out realtime graphic and signal processing between two user equipmenies and is System.

Fig. 6 shows the block diagram of the example system architecture of the feature and process for realizing Fig. 1-5.

Fig. 7 shows the block diagram of the example system architecture of the feature and process for realizing Fig. 1-5.

Same reference numeral indicates identical element in each diagram.

Specific embodiment

It summarizes

Figure 1A shows the element present in the exemplary realtime graphic based on AR and signal processing system 10.When true When life object is visibly located in actual environment (for example, box 12), the signal of such as image of real-life object is captured, Then institute's captured image is handled in real time to extract the image information (for example, box 14) of real-life object.Virtual environment (example Such as, box 16) it can be constructed before or while real-life object is extracted.Then by extracted real-life object with Selected virtual environment is integrated.Optional auxiliary signal, such as non-visual signal or ad elements, can be added to system 10 In any element.Although certain virtual environment elements in some embodiments, can be constructed first, real-life is being captured Whole process can carry out in real time while the image of object.

As disclosed in present specification, term " real-time " is referred in the case where seldom with and without time delay Execute function.For example, after image is captured soon or just when performing image processing, carrying out image zooming-out in real time.One In a little embodiments, delay can be in one minute or in several seconds.In some embodiments, delay may be by data transmission or Caused by the hardware limitations such as the speed of image procossing.

As disclosed in present specification, term " real-life object " and " user " are used interchangeably sometimes.Some In embodiment, user is real-life object.In some embodiments, user is the operator of computer equipment, and user Control includes one or more of the function of image capture.

As disclosed in present specification, term " image " can be used to refer to discontinuous time point shooting or in video The single photo intercepted in picture frame.Term " image " and " video " may be used interchangeably, unless otherwise indicated.Video is actually It is the set of the multiple images continuously captured.

As disclosed in present specification, term " signal " and " data " be may be used interchangeably.For example, they may include Image, audio, video, text, space, geography or any other information associated with real-life object.They further include Reflect the metadata or other embedding informations of real-life Obj State.They can also include closing indirectly with real-life object The data of connection, such as the image or the information such as video of reflection real-life object geographic location.As disclosed in present specification, " signal " and " data " may include internally and/or externally data.In present specification, internal data refers in actual environment In capture real-life object in real time during the data collected, including video, audio and other kinds of information.External number It include but is not limited to the data having stored in local subscriber apparatus, from another according to the data beyond real-time collecting are referred to One user equipment data (by be connected to the network may have access to), storage on the server data (e.g., including be stored in advertisement Ad elements on provisioning server) or using network collection real-time retrieval data.Big disclosed in present specification Most examples refer to image, however, anyway, they should not all be limited the scope of the invention.

As disclosed in present specification, " auxiliary signal " be except reflection real-life object itself signal or data it Outer signal or data.Auxiliary signal can also include internal or external data.In some embodiments, auxiliary signal includes non- Optical signal, such as audio tracks or external audio file.In some embodiments, auxiliary signal include can be true with extraction The ad elements that life object, virtual environment or final integrated image or video combine.

Real-life pair is captured using such as image/audio capturing equipment (such as camera) user equipment in box 12 The signal (for example, image or audio) of elephant.In some embodiments, camera is the component part of user equipment.In some implementations In example, camera can be connected to the external hardware component of user equipment.In some embodiments, user equipment is to support network Camera.Preferably, camera is depth camera.In some embodiments, image/audio capturing equipment includes one group of photograph Machine.As described herein, user equipment should be equipped with CPU/GPU processor, camera, microphone, display, loudspeaker, logical Believe unit and memory.It includes but is not limited to that desktop computer, laptop computer, smart phone device, individual digital help Reason, network camera, tablet computer, AR glasses, the AR helmet, VR glasses, smart television etc..Camera can be three-dimensional (3D) and shine Camera, conventional RGB camera, infrared (IR) camera, multispectral camera, bloom spectrum camera, 360 degree of cameras etc..

In some embodiments, real-life object is people.In some embodiments, real-life object is animal or object Body.In some embodiments, multiple images are shot to real-life object.In some embodiments, continuously shot images and shape At video.It in all embodiments, can be by real-life object for capturing the computer equipment of the image of real-life object Or user's access of computer equipment.

As disclosed in present specification, when capturing the image, real-life object be can reside in any environment.It is right The environment of image capture does not have particular/special requirement, for example, rear projection screen color is not needed uniform or approached uniform.Implement most of In example, the image of real-life object is captured, just as being in practical tangible environment real-life object.Some In embodiment, the image of real-life object is shooting when real-life object carries out conventional activity.

At block 14), while capturing image, the image information of real-life object is extracted.In some embodiments In, based on one or more differences in the characteristic between real-life object and actual environment, by by real-life object Extraction is executed with its practical environment separation.In some embodiments, which can be visual characteristic, including but not limited to empty Between characteristic, dimensional characteristic, shape feature, kinetic characteristic, color characteristics, illuminance and reflection characteristic, time response or transparency Characteristic, depth characteristic, material property or their combination.

In some embodiments, visual characteristic includes three-dimensional space characteristic.

In some embodiments, characteristic can be study acquire in real time or preparatory with real-life object or actual rings The relevant feature in border.Real-time learning or the characteristic learnt in advance include but is not limited to, such as the reflection of color, shape, edge, light Rate, illuminance, movement, depth, material, contrast or their combination.

In some embodiments, the extraction of real-life object is carried out on a image-by-image basis.This is also suitable In video extraction, it is carried out on a frame-by-frame basis.In some embodiments, while parallel processing multiple images.

In box 16, virtual environment is provided.In some embodiments, it can even be caught in image before image zooming-out Virtual environment is constructed before obtaining.In some embodiments, virtual environment can be constructed simultaneously in image zooming-out or image capture.

In some embodiments, virtual environment is different from actual environment.In some embodiments, virtual environment is actual rings The revision in border.In any embodiment, the image for constructing virtual environment can be the image modified, characteristic packet Include but be not limited to size, shape, picture quality, color, perspective, light, visual effect or their combination.

In some embodiments, virtual environment can be identical as actual environment, but has one or more features can be by Change.For example, can handle actual environment, to enhance certain elements to which virtual environment be presented.Furthermore, it is possible to modify Actual environment is to enhance picture quality to which virtual environment be presented.In some embodiments, one or more of actual environment Element visual characteristic such as color, shape, size, in terms of change.

In some embodiments, virtual environment can be identical as actual environment, but the relationship of itself and real-life object is sent out It is raw to change.In some embodiments, become smaller or larger for environment of the real-life object locating for it, such as electricity Shadow " Alice in Wonderland ".In another example the people having tea in the image of shooting may be displayed on the tea in final integrated image In cup, and the other elements in environment remain unchanged.

In some embodiments, server can provide a user the set of virtual environment.Virtual environment can be according to it Present in theme be divided into different classifications.Exemplary subject includes but is not limited to nature, animal, space, film, builds Build, is cultural, travelling etc..User can store in user preference the selection of particular type of subject matter.

In some embodiments, virtual environment includes image.In some embodiments, virtual environment includes video.One It may include multiple virtual environments, each virtual environment is considered as individual during image or video integration in a little embodiments Element.

In box 18, optional auxiliary signal is provided.Auxiliary signal may include internal or external data, including image, Audio, video, text, space, geography or any other type information.In some embodiments, auxiliary signal includes non-view Signal, such as audio signal are felt, for example, the acoustic information collected when capturing the image information of real-life object.Some In embodiment, audio signal includes internal or existing audio signal or external audio signal.In some embodiments, from capture Image or video obtain internal or existing audio signal, and can be further processed (for example, speech recognition and subsequent language Translation).In some embodiments, it can handle the audio signal of video to enhance sound effect.For example, can remove ambient noise with Enhance the sound of real-life object (for example, loudspeaker during videoconference).It in some embodiments, can will be special Sound effect is added in desired sound.For example, sound can be rendered as having three-dimensional hollow effect to imitate in echo environment Sound.

In some embodiments, external audio signal can be added in existing audio signal.For example, user can be with Selection plays background music during conference call.In some embodiments, external audio signal is used to replace existing Audio signal (for example, the audio signal collected when capturing the image of real-life object).Such audio signal (it is external or It is internal) it can be with any element in system (for example, box 12,14,16 and 20) being associated.In some embodiments, audio Signal is associated with virtual environment.In some embodiments, audio signal can be added to integrated image.It describes in fig. 1 c For realizing the example process of audio signal (for example, as real time translation).

In some embodiments, auxiliary signal includes one or more ad elements.Ad elements can be with product or clothes Business is associated.In some embodiments, ad elements include internal or existing audio signal or external audio signal.In some realities It applies in example, ad elements can be associated with real-life object (for example, element 12 or 14).For example, real-life object can To wear or hold the product comprising ad elements.In some embodiments, ad elements are added to virtual environment 16.Example Such as, ad elements can be shown as a part of virtual environment.In some embodiments, ad elements may be implemented as collecting At image or video.For example, other than existing virtual environment, ad elements can be considered as another in integrating process A virtual environment.In certain environments, ad elements can add in rear integrate, such as when user checks integrated image or view When frequency.

In some embodiments, ad elements are provided based on user preference, these user preferences or is stored in local On user equipment, or storage is on the server (such as a part as user profile).In some embodiments, it uses Family preference is determined by user's shopping history.In some embodiments, user can specifically request product, service, product type Or service type.In some embodiments, general users' information such as age, gender can be used as reference.In some implementations In example, the tendency information being generally available can be used.

Advantageously, moreover, ad elements are provided based on scene information.Scene information includes but is not limited to communication scenes, wide Accuse content scene, performance scene etc..For example, advertisement may include having with company if advertisement is presented in the session of company The content of the relevant products & services of body business.When advertisement is by as in online advertisement insertion forum (such as website or webpage) When displaying, it may be considered that the content of website and/or webpage.If the content on website and/or webpage is considered improper, no Advertisement is provided.Inappropriate content includes but is not limited to religious frenzy, terrorism, pornographic etc..The standard can be by offer advertisement The server settings of content.In some embodiments, advertiser can be not intended to associated curriculum offering standard for it.

In some embodiments, when multiple advertisers can provide the same or similar appropriate content, it is possible to implement competing Valence process selects ad content.

In box 20, extracted real-life object and virtual environment are integrated or combined, to be in virtual environment The image or video of existing real-life object.It is integrated in order to realize, will definition extract real-life object and virtual environment it Between relationship.In some embodiments, these relationships are defined simultaneously as image zooming-out and/or virtual environment building.Some In embodiment, once being extracted the image information of real-life object and constructing virtual environment, these relationships are just defined.

In some embodiments, system or user provide general predefined guide for relationship.In some embodiments, it is based on The real-life object of extraction and the information of virtual environment, system automatically define these relationships.In some embodiments, user Real-time adjustment can be provided, with redefine or modify in integrating process extracted real-life object and virtual environment it Between relationship.

In some embodiments, the relationship between extracted real-life object and virtual environment includes depth relationship. For example, extracted real-life object can be located partially or entirely at an element in virtual environment " front " or " back ".In some embodiments, extracted real-life object can be partially or even wholly in the " preceding of element Face ", but partially or even wholly in another element " back " of same virtual environment.In some embodiments, extracted Transient state occurs in the time course of depth relationship between images between all or part of of real-life object to become Change.In some embodiments, the depth relationship between all or part of of extracted real-life object is in same image In different elements on change spatially occurs.

In some embodiments, the relationship between extracted real-life object and virtual environment includes that transparency is closed System.For example, extracted real-life object can partly or completely all-transparent relative to the element of virtual environment.In some implementations In example, the time course of transparency relationship between images between all or part of extracted real-life object Middle generation temporal variations.In some embodiments, the transparency between all or part of of extracted real-life object Change on space occurs on the different elements in same image for relationship.

Image set is at explanation

In some embodiments, it integrates and is carried out in a manner of pixel-by-pixel.In some aspects, integrating process can be from integrated image Start, may be better understood in mode opposite in this way.In some embodiments, each pixel in integrated image can be by It is divided into multiple layers, such as foreground layer, middle layer and background layer.Each layer is corresponded to accordingly based on the real-life object of extraction The image information and/or virtual environment of pixel is filled.

Figure 1B provides the explanation to the exemplary embodiment of integrating process 30 pixel-by-pixel.Here, the extraction simplified is true Life object 32 indicates that simplified virtual environment is identified as 34 with triangle, and there are also the rectangles with lattice, wherein each Grid represents a pixel.By the way that real-life object 32 and the integrated image obtained of virtual environment 34 are illustrated as integrated image 36.As shown, certain pixels of real-life object 32 become invisible (being located at behind virtual environment 34), for example, with reference to 36 (x of pixel_l, y₁).Some pixels of real-life object 32, which are that partially visible, part is sightless, (is partially located at virtual ring Before border 34, partially it is located at behind virtual environment 34), for example, with reference to 36 (x of pixel₂, y₂).The one of real-life object 32 A little pixels are fully visible (before virtual environments 34), for example, with reference to 36 (x of pixel₃, y₃).In integrated image 36 Some pixels lack any information from real-life object 32 completely, for example, with reference to 36 (x of pixel₄, y₄)。

By using three layer methods, 36 (x of pixel_l, y₁) with the foreground layer by the pixel definition in virtual environment 34, and And wherein interbed or background layer can be filled with the information of the respective pixel from real-life object 32.However, due to prospect Layer transparent value be set as 0, therefore, the pixel in virtual environment 34 block completely foreground layer it is subsequent anything.Cause This, can be by the Pixel Information for self-virtualizing environment 34 to define foreground layer, while the value of middle layer or background layer being set It is set to zero and obtains 36 (x of pixel_l, y₁)。

36 (x of pixel₂, y₂) show the information from real-life object 32 and virtual environment 34.The pixel can pass through Foreground layer is defined with the partial information from real-life object 32, is determined for the information of the respective pixel of self-virtualizing environment 34 Adopted middle layer and background layer is defined with the partial information from real-life object 32 and is obtained.Equally, each layer saturating Lightness degree is set as 0, therefore middle layer is only partially shown and background layer is blocked completely.When the transparency quilt of different layers When being set as nonzero value, the information from real-life object 32 and virtual environment 34 will be mixed with each other in entire pixel.

36 (x of pixel₃, y₃) and 36 (x of pixel_l, y₁) opposite.It has by the pixel definition in real-life object 32 Foreground layer, and wherein interbed or background layer can be used to the information of the respective pixel of self-virtualizing environment 34 to fill.However, by Be set as 0 in the transparence value of foreground layer, no matter in the subsequent anything of foreground layer all by the picture in real-life object 32 Element blocks completely.Therefore, can be by defining foreground layer with the Pixel Information from real-life object 32, while it will be intermediate The value of layer or background layer is set as zero and obtains 36 (x of pixel₃, y₃)。

36 (x of pixel₄, y₄) be located in real-life object 32 and the nonoverlapping region of virtual environment 34.In such case Under, real-life object 32 completely disappears.Three can be defined by the information of the respective pixel for self-virtualizing environment 34 Any one in layer, while setting zero for the value of remainder layer and obtaining 36 (x of pixel₄, y₄)。

In some embodiments, as set forth above, it is possible to execute parallel computation to multiple pixels simultaneously.In some embodiments In, it can be calculated by the pixel in identification Non-overlapping Domain to simplify.It in some embodiments, can also be by only complete Nontransparent solid pixel calculates to define top layer to simplify.

In some embodiments, extracted real-life object includes three-dimensional image information, these three-dimensional image informations Can it is preceding, in divided in any combination between one or more of background layer.In some embodiments, virtual ring Border includes three-dimensional image information, these three-dimensional image informations can it is preceding, between one or more of background layer to appoint Meaning combination is divided.

In some embodiments, the pass between the real-life object (for example, 32) and virtual environment (for example, 34) of extraction System dynamically changes.In some embodiments, such dynamic change occurs between integrated image at any time.For example, true Life object 32 can pass in and out virtual environment 34.In some embodiments, at least part and virtual ring of real-life object One or more element interactions in border.For example, this can occur in gaming, wherein real-life object (user) can be in void Using gesture from popping up ball on wall or ball is passed to the player in virtual environment in near-ring border.

It should be noted that function associated with the box in Figure 1A and 1B can in same computer equipment or It is carried out in different computer equipments.When function is executed by different computer equipments, the combination of any equipment can be used. It should be noted that image capture is always by user or the addressable computer equipment of real-life object in any combination It executes.

Handle auxiliary signal

Fig. 1 C shows the process for handling the auxiliary signals such as non-optical signal.Process 60 is shown for from capture The example of video data acquisition non-visual signal.All steps executed in present specification can be by signal shown in Fig. 1 D Processing module 128 is realized.

In step 62, the video of user equipment capture is handled to extract audio-frequency information, which includes but is not limited to, For example, ambient sound, the voice of real-life object or noise and any other audio-frequency information.

In step 64, it is further processed non-optical signal.For example, ambient sound and other noises can be reduced or eliminated, To enhance the sound or noise of real-life object.Then it is special that the audio signal enhanced can carry out speech recognition program etc. The processing of industry program, to detect language and identify the content of audio signal.

In step 66, during videoconference, if there is one or more participants are ignorant of language used in a people, Then the language can be automatically translated into the language that other participants can understand.In some embodiments, it can mention over the display For subtitle to help to understand.

In step 68, synthesis speech production method can be used to create the audio of the audio signal obtained in step 62 and turn over It translates.In some embodiments, user can choose the preference that certain type of voice and accent is arranged.In some embodiments, Audio translation file can be optionally enhanced for additional sound effect, such as three-dimensional echo sound effect.

In some embodiments, as shown in process 70, external audio signal can be added.External sound in present specification Frequency signal is the signal that those are not belonging to a part of the video file of capture.For example, in step 72, user can choose he/ Background music of the music of her preference as the video using the creation of method disclosed in present specification.In some embodiments In, user can select the snatch of music of preference as background sound during Conference Calling.

In step 76, selected audio fragment can be edited or handled to be preferably bonded expected purpose.For example, only one The audio that section snatch of music can be selected as video is dubbed in background music.In addition, selected snatch of music can be used as video conference Volume is turned down before the background audio of calling.The example that present specification provides is related to the conference system there are two participant. However, method and system disclosed in present specification also can be applied to more than two participants, including three or more, Five or more, ten or more, 20 or more, 50 or more or hundreds of or more, as long as conference system energy Enough support the quantity of participant.

In step 80, external audio signal and internal audio signal are combined to produce to the sound of enhancing.

Exemplary system

As shown in the exemplary embodiment in Figure 1B, information flow can be divided into three blocks: the real-life object 32 of extraction, Virtual environment 34 and integrated image 36.Associated functions of block of information different from these can be by one or more computer equipments It (is carried out further for example, the data handled on one device can be sent to another equipment with any combination to execute Or additional treatments).

Fig. 1 D shows the exemplary system 100 for executing function disclosed in present specification.Here, multiple users set Standby (for example, 120 and 140) are connected to server apparatus 170 by network 160.Information processing is set in the one or more of diagram Standby upper progress.

User equipment 120 describes the local device equipped with multiple functions and (sets for example, real-life object is addressable It is standby).Particularly, user equipment 120 includes that user outputs and inputs (I/O) module 122, local data base 124 and for capturing The image of real-life object all functional modules (for example, module 126,128,130 and 132), extracts real-life object, Building virtual environment simultaneously integrates real-life object with virtual environment.

In some embodiments, user equipment 120 includes that user outputs and inputs module (I/O module) 122.For example, I/O Module 122 can receive input of the user to user equipment 120, and be exported using text, audio, video, movement and/or tactile The output from user equipment 102 is presented in mechanism.For example, I/O module 122 can receive gesture input, body kinematics or user Voice input.In some embodiments, I/O module 122 is also used for detection and tracking eye movement, facial expression etc..Again Such as, I/O module 122 receives touch input from the touch-sensitive display of user equipment 120.In other embodiments, I/O module 122 may be coupled to other input equipments, such as keyboard, mouse, monitor, touching sensitive screen curtain, loudspeaker etc., and receive User's input from these equipment.In some embodiments, I/O module 122 includes keyboard, mouse, monitor, touches sensitivity Screen, loudspeaker etc. are as I/O module 122 or a component part of user equipment 120.In some embodiments, Yong Hushe Standby 120 can receive a plurality of types of user's inputs.

In some embodiments, audio, vision, movement and/or tactile output can also be presented in I/O module 122 to user. For example, I/O module 122 shows integrated image or video on the monitor of user equipment 120.In some embodiments, I/O mould Block 122 can be presented graphic user interface (GUI), and GUI is enabled or supports the one or more of present specification description figure in real time Picture and signal processing method/system function, functional module include but are not limited to signal capture module 126, signal processing module 128, virtual environment module 130 and integration module 132.In some embodiments, it includes multiple dishes that user, which outputs and inputs module, Single command, each menucommand correspond to the function of one or more query processing modules.

In some embodiments, information needed for I/O module 122 allows user to position image procossing.In embodiment, may be used To provide menu option, so that user can choose one or more options to start one or more functions.For example, user can Start image capture process to click icon.In addition, for example, the virtual environment of multiple classifications can be provided by menu option Image.

In some embodiments, it is available in local subscriber apparatus 120 to request that I/O module 122 can be used in user Information, or information can be obtained by being connected to the network from service equipment 170 or another user equipment 140.For example, I/O module 122, which can permit user, requests certain type of virtual environment (for example, the Dark Ages castle image) using voice command. Once image is transmitted (locally or through network connection), user can request processing image to construct virtual environment.

In some embodiments, I/O module 122 can be used to manage various functional modules in user.For example, user can be with It requests to change virtual environment during real time communication by using I/O module 122.User can be the case where not interrupting communication Under by being individually chosen menu option or inputting a command for carrying out such operation.

When method/system disclosed in present specification is used for image or video processing and editor, user can make The process is instructed and controlled with any kind of input.

In some embodiments, user equipment 120 further includes local data base 124.For example, local data base can store Captured image, the image partially or completely handled (for example, the real-life object extracted and image for virtual environment), Or the image being partially or completely integrated.In some embodiments, local data base 124 stores user specific information；For example, with Family can store the information of the virtual environment of one or more preferences.In some embodiments, local data base 124 is stored from another One equipment or the information of server retrieval.In some embodiments, the storage of local data base 124 is retrieved from Internet search Information.

In some embodiments, local data base 124 transmits data to one or more functions module, and receives and come from The data of one or more functions module, functional module include but be not limited to signal capture module 126, signal processing module 128, Virtual environment module 130 and integration module 132.

In some embodiments, user equipment 120 includes signal capture module 126.For example, signal capture module 126 can To include the image capture apparatus such as camera.In some embodiments, camera is that depth enables.In some embodiments, make With two or more cameras.In some embodiments, built-in microphone or external microphone can be used to carry out audio collection.? In some embodiments, signal capture module 126 captures multiple images.In some embodiments, signal capture module 126 is continuously caught Obtain video.

As disclosed in present specification, signal capture module 126 captures the figure of real-life object in the actual environment Picture, the actual environment, that is, actually located environment of real-life object.In some embodiments, real-life object is user equipment 120 user.In some embodiments, user is not real-life object, on the contrary, user's control user equipment 120 is to capture The image of the addressable object of signal capture module 126.For example, real-life object can be by user's operation signal capture mould The famous person that block 126 is shot.

Method and system disclosed in present specification is advantageous, because they not ask real-life object to be present in Image procossing is helped in certain types of environment.

In some embodiments, user equipment 120 includes signal processing module 128.For example, signal processing module 128 is logical It crosses and separates the image information of real-life object from its actual environment, extracted from 126 captured image of module true Real life object.Multi-mechanism of the extraction and application based on characteristic, including the mechanism based on artificial intelligence.Based on the comprehensive of characteristic Conjunction mechanism identifies the difference of one or more special characteristics between real-world objects and actual environment.For example, feature can be with Including and be not limited to the actual environment captured in the visual characteristic or video clipping of real-life object, real-life object or The real-time learning characteristic of actual environment, or the feature learnt in advance related with real-life object or actual environment.Some In embodiment, visual characteristic can include but is not limited to include spatial character, dimensional characteristic, shape feature, kinetic characteristic, color Characteristic, illuminance and reflection characteristic, time response or transparency characteristic, depth characteristic, material property or their combination.? In some embodiments, space characteristics include three-dimensional space characteristic.

In some embodiments, the characteristic of real-time learning includes but is not limited to color, shape, edge, light reflectivity, illumination Degree, exposure, brightness, shade, highlighted, contrast, stain (blackpoint), movement, depth, material or their combination. In some embodiments, the feature learnt in advance further includes but is not limited to color, shape, edge, light reflectivity, illuminance, exposure Luminosity, brightness, shade, highlighted, contrast, stain (blackpoint), movement, depth, material or their combination.

Extraction process disclosed in present specification can be completed in a wheel or more wheels.In some embodiments, at signal A wheel coarse extraction is first carried out in reason module 128, then executes the fine extraction of one or more additional wheels again.For example, coarse extraction The profile of real-life object can be tracked, and finely extracts the side that can refine segmentation real-life object and actual environment Edge.In some embodiments, the region in the profile of real-life object can also be identified by a wheel or the fine extraction of more wheels, The region is actually a part of environment, and the region is then removed from real object.

In some embodiments, other than image zooming-out, image can also be assessed and be modified to signal processing module 128.

In some embodiments, signal processing module can handle the non-optical signal such as audio signal, as shown in Figure 1 C.

In some embodiments, user equipment 120 includes virtual environment module 130.After virtual environment module 130 can be The integrated creation of continuous image and modification virtual environment.For example, virtual environment module 130 can picture construction based on one or more Virtual environment.Virtual environment can be two-dimensional or three-dimensional.Virtual environment may include the image being based in virtual environment In the feature that is not present.For example, virtual environment can be altered or modified by modifying corresponding image in virtual environment module 130 In one or more elements.In some embodiments, the one or more features based on real-life object are made such Modification or adjustment, so as to more effectively integrate extracted real-life object and virtual environment.Illustrative modification or Adjustment includes but is not limited to that scaling, direction change, change in shape, color change, picture quality adjust (for example, exposure, bright Degree, shade, highlighted, contrast or stain) etc..Local individual element can be modified or be adjusted in virtual environment, or Entire virtual environment is modified or adjusted.In some embodiments, virtual environment can be different from actual environment.Some In embodiment, virtual environment can be identical as actual environment, and wherein one or more elements in virtual environment are modified to use It is integrated in subsequent image.

There are many similarities for virtual environment module 130 and the function of signal processing module 128, in some embodiments, It can be bonded to.

In some embodiments, user equipment 120 includes integration module 130.Integration module 130 is by the real-life of extraction Object is combined with virtual environment, generates integrated image.It, can be on a pixel-by-pixel basis as Figure 1B is explained in detail It is integrated, to improve efficiency and precision.

In some embodiments, exemplary system 100 further includes user equipment 140.In some embodiments, user equipment 140 can have functional module identical with user equipment 120；For example, user outputs and inputs module 142, local data base 144, signal capture module 146, signal processing module 148, virtual environment module 150 and integration module 152.When there are function moulds When block, it can be similar to user equipment 140 or be realized according to any applicable known technology.

In some embodiments, user equipment can have less functional module, and provide one dependent on server 170 A or multiple functions.As shown in figure iD, other than signal capture module 146, other images integrate correlation module (including signal Processing module 148, virtual environment module 150 and integration module 152) user equipment 140 can be optionally.These functions can Effectively, to be divided between user equipment 140 and server 170 with any combination.For example, user equipment 140 can be with It sends the captured image to server 170 and carries out image procossing (for example, image zooming-out).In some embodiments, extraction Virtual environment on real-life object and server 170 is integrated.In some embodiments, the real-life object of extraction can be with User equipment is transferred back to integrated with virtual environment.In some embodiments, user, which can choose, provides the virtual ring of customization Border.For example, user equipment can send server 170 for the virtual environment of selection, then, image is integrated in server 170 Upper progress.

In some embodiments, user, which can choose, opens or closes function.For example, user can choose in user's reality It is communicated in locating actual environment with another user.For example, user can choose aobvious when communicating with outgoing kinsfolk Show true home environment, does not make any change to it.This can simultaneously close off it by only opening signal trapping module 146 His functional module is realized.In addition, user, which can choose, opens any one of remaining functional module.

In some embodiments, exemplary system 100 further includes server 170.Server 170 and one or more users Equipment communication, and including server database 172, signal processing module 174, virtual environment module 176, integration module 178, The functional modules such as ad serving module 180 or their variant.In some embodiments, signal processing module 174, virtual ring Border module 176 and integration module 178 are similar to related to user equipment 120 or user equipment 140 disclosed in present specification Module.In some embodiments, since compared with user equipment, the computing capability and memory space of server have enhancing, Therefore these modules can execute different operations on server 170.For example, with the integrated phase that can allow on user equipment Than the integrated of greater number pixel-parallel can be carried out on server.

Ad serving module 180 determines whether ad content will provide together with certain integrated images and/or video.Such as figure Shown in 1A, ad content is a type of auxiliary signal, can each point in integrating process be integrated into final collection At in image and/or video, such as in image zooming-out, the building of virtual environment or final integrating process.In addition, in integrated figure When picture or video are watched, middle addition ad elements can be integrated after in real time.

Exemplary embodiment

System and method disclosed in present specification have many applications.For example, they can be used for creating or modifying Image is for demonstration, for example, see Fig. 2 and 3A to 3C.Alternatively, they can be used to realize more effective real time communication, such as scheme Shown in 4A to 4C and Fig. 5 A to 5C.

Fig. 2 shows execute the functional sample processes of institute on a user device.In some embodiments, this can by with The equipment such as family equipment 120 enable.Here, user can start image capture by opening depth enabling camera, depth is enabled Camera is connected to a component part of user equipment or user equipment.Here real-life object can be the use of equipment Family or the addressable object of equipment.For example, photographer or cameraman can operate user equipment to collect real-life object (example Such as, people, animal or combinations thereof) image and video.

The extraction of real-life object follows after image capture, it can automatically be carried out with image capture simultaneously. In some embodiments, when user starts image zooming-out if can manually controlling.For example, for image or video editing, user Can choose image zooming-out when and since where, to only handle desired image or video section.

In some embodiments, user can choose virtual environment.Selection can carry out after image zooming-out, Huo Zheyu Image zooming-out carries out simultaneously.In some embodiments, existing virtual environment is selected.It in some embodiments, can be in user Interim building virtual environment after selection.In some embodiments, when constructing virtual environment, it may be considered that real-life object One or more features.For example, when real-life object intends the element interactions with virtual environment, adjustable virtual ring The size or shape in border are to carry out Seamless integration-.

Once the data of the real-life object and virtual environment that extract are available, can also automatically integrate extracted true Real life object and virtual environment.In some embodiments, user can manually control when should to start image integrated.Example Such as, for image or video editing, user can choose when and wherein to start image integrated, only to handle desired figure Picture or video section.

As shown, show and colour can be added in one or more steps on a user device.In some embodiments In, user equipment can send ad elements to another user equipment, or receive ad elements, example from another user equipment Such as, more facility communication systems as shown in Fig. 5 A to 5C.

Fig. 3 A to 3C shows the sample embodiment of the partition functionality between user equipment and remote server.Such as Fig. 3 A institute Show, other than carrying out image capture on a user device, other processing steps executed on the remote server include image Extract, the offer of virtual unit or building and the two it is integrated.Then integrated image is sent back into user equipment to watch. User, which can choose, modifies integrated image by providing user's input after watching integrated image.

In figure 3b, other than image capture step, image extracting step also carries out on a user device.Virtual unit Offer or collection Chengdu of building and the two execute on the remote server.Then integrated image user equipment is sent back to see It sees.

In figure 3b, other than image capture step, the offer and building of image zooming-out and virtual environment are also in user It is carried out in equipment.It only carries out on the remote server integrated.Then integrated image is sent back into user equipment for viewing.

Process shown in Fig. 2 and 3A to 3C can be used for non-communicating purpose, such as image and/or video editing, be used for Create the presentation material of real-time or non real-time/delay, including but not limited to individual video, documentary film, news material, education material Material, show and colour.Such process can be interactive mode, and order or selection can be altered or modified in different points in user, Including but not limited to start the time of image capture and the time of selection virtual environment.For example, can shift to an earlier date or create in real time Virtual education video about ancient Roman.When teacher shows video on classroom, she can require student to participate in as guide. Using system disclosed in present specification, the virtual guide that student can be used as ancient Roman occurs in video.

Embodiment shown in Fig. 2 can be used independently to create image or video, without from external clothes in user The help of business device.Image or video can be used for it is personal appreciate or as speech a kind of form (e.g., including be used for advertisement).Such as Disclosed in present specification, ad elements can be added in any one or more method and steps.

For example, realtor can be created by the way that the realtime graphic of oneself to be inserted into existing house property photo Leaflet, without re-shooting photo.In addition, method permission herein is enhanced when generating integrated image by real time modifying Existing house property photo.

For another example user can create small budget advertisement in local subscriber apparatus, regardless of whether having from server It helps.For example, the storekeeper of a florist's shop can most like the image integration that the characteristic of oneself is arranged flowers to her in video clipping Scene in.

User can also be created with the help of one or more external servers using embodiment shown in Fig. 3 A to 3C Build image or video.Image or video can be used for it is personal appreciate or as speech a kind of form (e.g., including be used for advertisement). As disclosed in present specification, advertisement member can be added in the method and step that any one or more are executed on the server Element.

Process shown in Fig. 2 and 3A to 3C also can be applied to real time communication.For example, integrated image will be direct or indirect Another user equipment is sent to via server.Here, virtual ring can be carried out by sending the user equipment of integrated image The selection in border.The user equipment for receiving integrated image will be seen that the virtual environment of sending device selection.In some embodiments, it connects Debit's user equipment can choose the virtual environment closed and selected by transmission user equipment.

In some embodiments, one or more ad elements can be added in communication process.The content of ad elements It can be determined based on Multiple factors such as user preference or scene informations.In some embodiments, selected ad elements can To be integrated into virtual environment.For example, when the trip person to go on business to San Francisco contacts his household, the virtual environment of his selection It may include the advertisement of local gift shop.However, when he carries out business telephony meeting, the virtual environment that he selects may include Advertisement relevant to his running of company or industry.In these examples, ad content is determined by communication scenes.

Fig. 4 and Fig. 5 provides the further explanation of the sample embodiment for real time communication, and wherein integrated image passes through net Network set, intermediary service are directly sent to another user equipment from a user equipment.

Fig. 4 A shows the sample embodiment that two user equipmenies are in communication with each other by intermediate server.In addition to being set in user Except standby upper progress image capture, other processing steps executed on intermediate server further include image zooming-out, virtual unit Offer or building and the two it is integrated.Here, the first user equipment and second user equipment are all by respective real-life pair The capture image of elephant is sent to intermediate server, and the first and second real-lifes are then extracted from their own actual environment Object.

First user equipment sends the selection of first environment to server, and second user equipment sends the second ring to server The selection in border.In some embodiments, server provides institute from the virtual environment library in the database stored on the server The virtual environment of selection.In some embodiments, server is provided based on user equipment image or from the data on server The respective virtual environment of the picture construction selected in library.During integrated, extracted first real-life object will with by the The virtual environment of two user equipmenies selection is integrated.Similarly, extracted second real-life object will be set with by the first user The virtual environment of alternative is integrated.Although the virtual environment that the real-life object of extraction is selected with oneself can be integrated, It is to depict preferred embodiment in Fig. 4 A, because the image of the first real-life object will be seen in second user equipment It sees.

Fig. 4 B shows sample processes similar with Fig. 4 A, in addition to image zooming-out on a user device rather than intermediary service It is carried out on device.Then intermediate server is sent by the image of extraction.Virtual environment is still provided by intermediate server or is constructed. During integrated, extracted first real-life object will be integrated with the virtual environment selected by second user equipment.It is similar Ground, extracted second real-life object will be integrated with the virtual environment selected by the first user equipment.Although can will mention The real-life object taken and the virtual environment of oneself selection are integrated, but depict preferred embodiment in Fig. 4 A, because The image of first real-life object will be watched in second user equipment.

Fig. 4 C shows the sample processes similar with Fig. 4 A and 4B.Here, carrying out additional processing step on a user device Suddenly.For example, the offer and building of image zooming-out and virtual environment also carry out on a user device other than image capture.So Intermediate server is sent by the image of extraction and virtual environment afterwards.During integrated, extracted first real object will be with It is integrated by the virtual environment that second user equipment selects.Similarly, extracted second real object will be set with by the first user The virtual environment of alternative is integrated.Although the virtual environment that the real-life object of extraction is selected with oneself can be integrated, It is to depict preferred embodiment in Fig. 4 A, because the image of the first real-life object will be seen in second user equipment It sees.

Sample embodiment described in Fig. 4 A to 4C allows server attracting participation other movable (such as video conferences) User provide ad content simultaneously.In some embodiments, one or more ad elements can be added in communication process. The content of ad elements can be determined based on Multiple factors such as user preference or scene informations.

In the embodiment shown in Fig. 4 A to 4C, user does not find advertisement actively, therefore is believed based on user preference and scene Breath, by non-intruding and it is relevant in a manner of to provide advertisement critically important.In some embodiments, selected ad elements can be by It is merged into virtual environment.For example, when the trip person to go on business to San Francisco contacts his household, virtual environment that he selects can be with Advertisement including local gift shop.In addition, advertisement can also be presented to his household during videoconference, and content can be with Selection (such as based on he household geographical location).

However, when he carries out business telephony meeting, the virtual environment that he selects may include and his company or industry The relevant advertisement of running.Equally, during videoconference, advertisement can also be presented to other callers, and can be such as Content is selected based on the running of the geo-location of its position and their company or industry.In some embodiments, phase is presented Same advertisement.In some embodiments, different advertisements is presented to different participants.In these examples, ad content It is determined by communication scenes.

Fig. 5 A to 5C shows no intermediate server, the sample embodiment that user equipment directly communicates with each other.

In fig. 5, each user equipment capture image in two user equipmenies, extraction image, offer/building are virtual Environment simultaneously integrates the real-life object of extraction with virtual environment.Integrated image for example by bluetooth or is wirelessly connected transmission It is shown to another user equipment.Equally, this may not be preferred environment, because of the user on a user equipment It will be forced to check the virtual environment selected by another equipment.

Fig. 5 B shows an exemplary embodiment, the virtual ring that the user in one of equipment can select in their own Integrated image is checked in border.For example, the first user equipment captures the image of the first real-life object in the first actual environment. Then it extracts the first real-life object concurrency and is sent to second user equipment, be then integrated into second user equipment and select The second virtual environment in.Similarly, second user equipment captures the figure of the second real-life object in the first actual environment Picture.Then it extracts the second real object and is sent to the first user equipment, be then integrated into second user equipment and select The first virtual environment in.In such embodiments, each user can check that they communicate in the virtual environment of preference A side.

Fig. 5 C shows an exemplary embodiment, and the user in one of equipment can select virtual in their own Integrated image is checked in environment.For example, the first user equipment captures the figure of the first real-life object in the first actual environment Picture.Then it extracts the first real-life object concurrency and is sent to second user equipment, be then integrated into second user equipment In second virtual environment of selection.Similarly, second user equipment captures the second real-life object in the first actual environment Image.Then it extracts the second real-life object concurrency and gives the first user equipment, be then integrated into second user and set In first virtual environment of standby middle selection.In such embodiments, each user can check them in the virtual ring of preference The side communicated in border.

Although not shown in Fig. 5 A to 5C, it is to be understood that, user can choose in one or more processing steps Add ad elements.In some embodiments, example process shown in Fig. 5 A to 5C can be applied to joint or cooperative Create demo content, including but not limited to ad content.

Fig. 4 and process shown in fig. 5 can be used for real time communication purpose.It is such communication include but is not limited to video conference, Interactive entertainment, interactive news material, interactive educational material or Interactive Advertising material.For example, can be in advance or real-time Ground creates the virtual education video about ancient Roman.When teacher shows video on classroom, she can ask student as guide It participates in.Using system disclosed in present specification, the virtual guide that student can be used as ancient Roman occurs in video.

Although example provided in Fig. 4 and Fig. 5 is all referring to two side's communication systems, it is not by the scope of the present invention Limited to this.As long as supporting such communication pattern, the method and system disclosed in present specification can be used for any number The participant of amount.

System and method shown in the above-mentioned Fig. 1-5 of present specification provide many advantages, especially with the figure based on AR The realtime graphic and signal processing of picture and video editing and the real time communication based on AR are associated.For example, present specification institute is public The system and method opened ensure that accuracy, efficiency, convenience and flexibility, and can realize in numerous applications.

Method and system disclosed in present specification further merges the music for being suitable for scene in integrated image/video And ambient sound.In addition, interpreter language voice appropriate is used to further enhance sound and experience.In short, present specification Disclosed method and system will provide the user with the vision and audio experience of depth immersion, feel them in identical In environment, even if they are actually located far away from one another.

Multiple examples that described method and system can be implemented have been summarized below.This method and system can be used for based on AR's Video conference.This method and system help to protect privacy of user, and allow user that the environment of preference is selected to check meeting Other participants.

This method and system can be with embedded marketing applications together, for the video conference based on AR.User or enterprise can Being inserted into marketing message, advertisement in the video conference based on AR, so that the user for participating in video conference be enable to touch Product or service.

This method and system can also be used for videograph and editor based on AR.User can create themselves in universe In it is any it is contemplated that environment in video, it makes green curtain video capture seem out-of-date.It allows user to carry out the time Travelling, such as the Rome in access epoch in flourishing age.It can be used in education or be intended merely to entertain and use.

Described method and system creates the environment of a seamless immersion, allow one to across it is huge practically Distance is managed, is felt but as in same place.

This method and system allow to say that the user of different language communicates freely of each other.

It is understood that any applicable method and/or system can be used to improve material production and demonstration in user Experience, and actively and passively advertisement experience.For example, this method and system includes mentioning application No. is 15/367124 On December 1st, 2016 is met at, in the U. S. application of entitled " personalized, interactive and intelligent search method and system " Disclosure is included in present specification by whole reference herein.

Instance system structure

Fig. 6 can be achieved on the block diagram of the feature of Fig. 1-6 and the example calculation equipment 600 of process.Calculating equipment 600 can be with It is connect including memory interface 602, one or more data processors, image processor and/or central processing unit 604 and periphery Mouth 606.Memory interface 602, one or more processors 604 and/or peripheral device interface 506 can be individual component, Or it can integrate in one or more integrated circuits.The various components calculated in equipment 600 can pass through one or more Communication bus or signal wire coupling.

Sensor, equipment and subsystem may be coupled to peripheral interface 606, in order to realize multiple functions.For example, movement Sensor 610, optical sensor 612 and proximity sensor 614 may be coupled to peripheral interface 606, in order to orient, illuminate and connect The realization of nearly function.Other sensors 616 also may be coupled to peripheral interface 606, such as Global Navigation Satellite System (GNSS) (such as GPS receiver), accelerometer, gyro sensor, temperature sensor, biometric sensor, magnetometer or other Sensing device, to realize correlation function.

Camera subsystem 620 and optical sensor 622, for example, charge-coupled device (CCD) or complementary metal oxide Semiconductor (CMOS) optical sensor can be used to convenient for the realization of camera function, such as record photos and video clips.Camera The image of subsystem 620 and optical sensor 622 user to be used during can be used for collecting user authentication, for example, by holding Row Face recognition analysis.For example, camera subsystem 620 includes but is not limited to 2D camera, 3D camera, 2D/3D camera, infrared phase The combination of machine, near infrared camera, ultraviolet camera, multispectral camera, EO-1 hyperion camera and a variety of cameras.

It can promote communication function by one or more radio communication subsystems 624, which can To include radio frequency receiver and transmitter and/or optics (for example, infrared) receiver and transmitter.The tool of communication subsystem 624 Body, which designs and implements, can depend on calculating the communication network that the plan of equipment 600 is run on it.It can for example, calculating equipment 600 To include being designed in GSM network, GPRS network, enhanced data rates for gsm evolution (Enhanced Data Rate for GSM Evolution, referred to as " EDGE ") the upper communication subsystem to work such as network, Wi-Fi or WiMAX network and blueteeth network 624.Especially, radio communication subsystem 624 may include host protocol, so that equipment 120 can be used as other wireless devices Base station.Exemplary wireless system 624 include but is not limited to wireless Internet (WLAN, WiFi etc.), wired internet, bluetooth, 2G/3G/4G/5G is wireless etc..

Audio subsystem 626 may be coupled to loudspeaker 628 and microphone 630, in order to the realization of phonetic function, such as Speaker Identification, speech reproduction, digital record and telephony feature.For example, audio subsystem 626 can be used for facilitating processing voice Order, voice printing and voice authentication.Audio subsystem 626 includes but is not limited to regular microphones, high definition microphone, stereo The combination of microphone and multiple microphones.

I/O subsystem 640 may include display/touch surface controller 642 and/or other input controllers 644.Display Controller 642 may be coupled to touch-surface etc. and shows equipment 646.For example, display/touch-surface equipment 646 and relevant aobvious Show that any one of multiple touch sensitive technologies can be used to detect contact and movement or disconnection in controller 642, including but It is not limited to capacitor, resistance, infrared and surface acoustic wave technique and other proximity sensor arrays or other is used to determine and show Show/touch-surface 646 contact one or more contact points element.They can be AR glasses display, be also possible to often It advises display (mobile telephone display, television indicator etc.).

Another input controller (S) 644 may be coupled to other input/control devicess 648, such as one or more is pressed Button, rocker switch, thumb wheel, infrared port, USB port and/or indicator device, such as writing pencil.One or more of buttons (not shown) may include the up down button for the volume control of loudspeaker 628 and/or microphone 630.

In one embodiment, the locking of display device 646 can be released by lower button with the first duration, and The power supply for calculating equipment 600 can be switched on or off by lower button with second long duration of than first duration. Voice control can be activated by lower button with the third duration, voice command, or activation is activated to allow users to wheat It is issued in gram wind 630 and orders the module so that equipment executes verbal order.User can customize the function of one or more buttons. For example, display equipment 646 can be used for realizing virtual or soft buttons and/or keyboards.

User can pass through the input and output mould of I/O subsystem 640, other input controllers 644 or any other type One or more components in block provide Real-time Feedback.

In some embodiments, the audio and/or video files of record can be presented by calculating equipment 600, such as MP3, AAC and mpeg file.In some embodiments, calculating equipment 500 may include smart phone, tablet device, meter on knee Calculation machine or wearable device (for example, smartwatch, intelligent glasses etc.), game system, the equipment based on AR and/or based on MR's System.Exemplary AR equipment include AR glass goggles (for example, Microsoft holographic glasses Microsoft Hololens, Mo Fuliao glassesMoverio etc.) and the AR helmet (for example, Oakley radio wave Oakley ariwave Deng).Exemplary MR system includes the Microsoft Kinect combined with Xbox and display；The Intel combined with computer is real Feel (realsense) camera, etc..Exemplary smart phone based on AR system may include substantially any equipped with phase The smart phone of machine/microphone and other sensors.

Memory interface 602 may be coupled to memory 650.Memory 650 may include high-speed random access memory And/or nonvolatile memory, such as one or more disk storage equipments, one or more light storage devices and/or flash memory (for example, NAND, NOR).Memory 650 can store an operating system 652, for example, Darwin (Darwin), RTXC, LINUX, UNIX, OS X, WINDOWS or embedded OS such as VxWorks.

Operating system 652 may include the instruction for handling the task of basic system services and execution dependence hardware.? In some embodiments, operating system 652 can be kernel (for example, UNIX kernel).In some embodiments, operating system 652 may include the instruction for executing voice authentication.For example, the image as described in Fig. 1-5 can be implemented in operating system 652 Processing feature.

Memory 650 can also store communication instruction 654, in order to one or more optional equipments, one or more Computer and/or one or more server communications.Memory 650 may include graphical user interface instructions 656, in order to scheme Shape user interface process；Sensor processing instruction 658, in order to processing and functions relevant to sensor；Telephone order 660, In order to the relevant process of phone and function；Electronic messaging instruction 662, in order to electronic message delivery correlated process and function Energy；Web browsing instruction 664, in order to process relevant to web-browsing and function；Media processing instruction 666, in order to media Handle relevant process and function；GNSS/ navigation instruction 668, in order to GNSS and navigate relevant process and instruction；And/or Camera commands 670, in order to the relevant process of video camera and function.

Memory 650 can store image and signal processing instruction 672, in order to other processing and functions, such as Fig. 1-6 institute The image processing process and function shown.

Memory 650 can also store other software instructions 674, such as Web video instructions, in order to Web video phase The process and function of pass；And/or Web shopping instruction, in order to process relevant to Web shopping and function.In some embodiment party In formula, media processing instruction 666 is divided into audio processing instruction and video processing instruction, to be convenient for and audio processing phase respectively The processing and functions of pass and processing and functions relevant to video processing.

Each of above-mentioned identification instruction and application program can correspond to execute the one of said one or multiple functions Group instruction.These instructions do not need to implement as individual software program, process, or module.Memory 650 may include attached Add instruction or less instruction.In addition, calculating the various functions of equipment 600 can realize in hardware and/or software, it is included in In one or more signal processings and/or application specific integrated circuit.

Fig. 7 shows the block diagram of the feature of implementation Fig. 1-5 and the example remote server system architecture of process.

In some embodiments, remote data server 700 may include central processing unit 710, power supply 712, user Interface 720, telecommunication circuit 716, bus 714, controller 726, optional non-volatile memories 728 and at least one processor 730。

Memory 730 may include volatile and non-volatile storage unit, such as random access memory (RAM), only Read memory (ROM), flash memory etc..In preferred embodiments, memory 730 includes high-speed RAM, controls journey for storage system Sequence, data, application program, such as the program and data that are loaded from non-volatile memories 728.It is appreciated that any given The all or part of time, any module or data structure in memory 730 actually may be stored in memory 728.

User interface 720 may include one or more input equipments 724, such as keyboard, keyseat, mouse, idler wheel etc., with And display 722 or other output equipments.Network interface card or other telecommunication circuits 716 can provide be connected to it is any wired or Cordless communication network may include internet and/or any other wide area network, and in certain embodiments include movement Telephone network is expected someone's call network.Internal bus 714 provides the interconnection of the above-mentioned element of centralized data server 700.

In some embodiments, the operation of centralized data server 700 is mainly controlled by operating system 732, operation system System 732 is executed by central processing unit 710.Operating system 732 can store in system storage 730.In addition to operating system Except 732, system storage 730 is typically implemented include for controlling to various file sum numbers used in the present invention According to the file system 734 of the access of structure, one or more application module 736 and one or more databases or data module 750。

In some embodiments in accordance with the present invention, application module 736 may include that one or more is described below simultaneously The module being shown in FIG. 7.

Signal processing applications 738.In some embodiments in accordance with the present invention, signal processing applications 738 receive and process The signal of server, including but not limited to image, audio or video are sent to from user equipment.

By computation technology (for example, hash function), signal processing applications 738 by input data, sensing data, Network derivative data is converted into numerical data to construct one or more databases 752 (for example, 754-774).

Content management tools 740.In some embodiments, content management tools 640 are used for various forms of databases 752 are organized into multiple databases, such as treatment mechanism database 754, geolocation data library 756, network-based trend number According to library 758, virtual environment data library 760, user profile and authentication database 762 and additional database 764, in advertisement Hold.In some embodiments in accordance with the present invention, content management tools 740 are for searching for and comparing trustship in remote date transmission Any database on device 700.Content according to the present invention can be text message, network address (URL), Web link, annotation and disappear Breath, postal message, sensing data, file, image, audio file, video file, flash file, media file, lantern slide text Part or any printable or visual file.

The database being stored on remote data server 700 includes any type of data-storage system, including but not It is limited to flat file, relational database (SQL) and on-line analytical processing (OLAP) database (MDX and/or its variant).Some In specific embodiment, database is layering OLAP cube.In some embodiments, each database have one not It is stored as cube but there is the star schema for the dimension table for defining hierarchical structure.In addition, in some embodiments, number Have the hierarchical structure of the non-Explicit Factorization in basic database or database schema (for example, dimension table is not layering row according to library Column).In some embodiments, database is not in fact trustship on remote data server 700, and is actually concentrated Formula data server is accessed by secure network interface.In such embodiments, the safety measures such as encryption is taken to deposit to protect Store up the sensitive information in such database.

System administration and adviser tool 742.In some embodiments in accordance with the present invention, system administration and adviser tool All application programs and data file of 742 management and monitoring remote data server.Due to individuation datas such as biological characteristics It is stored on remote data server 700, therefore the file for accessing those strict controls and monitoring is critically important.System pipes Reason and adviser tool 742 determine the accessible remote data server 700 of which user equipment based on stringent user authentication. In some embodiments, it may be desired to mostly wheel certification.In some embodiments, system administration and adviser tool 742 use more than One safety measure protects the data being stored on remote data server 700.It in some embodiments, can be using random Spin safe system (random rotational security system) is stored in remote data server 700 to protect On data.

Network application 746.In some embodiments, network application 746 connects teledata clothes by multiple network services Business device 700.Remote data server 600 is connected to a plurality of types of user equipmenies, this requires remote data server to be adapted to Based on the communication of different types of network interface, for example, the computer network interface based on router, the phone based on interchanger Formula network interface and mobile phone wireless network interface based on launching tower, for example, 802.11 networks or blueteeth network.

Customer support tool 748.Customer support tool 748 helps user to understand in relation to their account, technical support, hidden Information or the problems such as private or advertisement experience feedback.In certain embodiments, user can request (for example, by real-time voice, text Sheet or gesture command) it is experienced to open and close advertisement.

In some embodiments, each data structure being stored on remote data server 700 is single data structure. In other embodiments, any or all such data structure may include multiple data structures (for example, database, file And files), these data structures may or be not necessarily stored on remote data server 700.One or more numbers It may include any number of database 752 according to module 750, these databases are organized into different knots by content management tools 740 Structure (or data structure of other forms).

Other than above-mentioned identification module, data 750 can also be stored on remote data server 700.Exemplary number It include treatment mechanism database 754, scene database 756, advertiser database 758, virtual environment data library 760 according to library 752 With user profile and authentication data collection 762, these databases will be described in more detail below.

Treatment mechanism database 754.Any applicable mechanism can store in treatment mechanism database 754.It is such Mechanism is not limited to the mechanism of capture signal data.Additional mechanism (such as virtual environment for handling integrated image or video or The additional mechanism of post production process) it also can store in treatment mechanism database 754.

Scene database 756.In some embodiments, remote data server 700 carries scene database 756.Scene The storage of database 756 and tissue data relevant to scene, such as time, weather, position dependent event, history, map, news Deng date and time information, communication scenes, content scene, advertiser's scene, geo-location scene, present scene or they Combination.When server is that user selects suitable ad content, information here can provide scene information.In some implementations In example, the trend data including nearest popular inquiry, event, domain knowledge and other relevant informations may be used as scene letter Breath.It, can be with real-time update information to provide scene information when server is that user selects suitable ad content.

Advertiser database 758.As disclosed in application documents, the information for being associated by advertiser or being specified is stored in extensively It accuses in quotient data library 758.In some embodiments, advertiser can propose the preference to particular platform.In some embodiments, The pricing information for participating in bid process can be set in advertiser.

Virtual environment data library 760.In some embodiments, for constructing the virtual environment or figure of this virtual environment Picture/video is stored in virtual environment data library 760.In addition, user's past can to the preference of the modification of one or more types With the virtual environment following for guidance and help modification.In some embodiments, the elder generation of virtual environment or its modifications and variations Preceding preference can be stored and be organized together with corresponding user profile (for example, coming from subscriber profile database 762), To provide the specific ad content of user.

Subscriber profile database 762.In some embodiments in accordance with the present invention, subscriber profile database 762 can be created and stored on remote data server 700, and 700 storage and management of remote data server are related with user Password and authentication data.In some embodiments, user has an opportunity to select security setting.In some embodiments, user is inclined Good information is also included in subscriber profile database.Example user preference information includes but is not limited to the specified letter of user Breath, user's biographic information, user behavior information, User Activity, user psychology, user social contact state, user's Real time request Information or their combination.

The present invention may be embodied as computer system and/or computer program product comprising be embedded into computer-readable Computer program mechanism in storage medium.In addition, any method of the invention can be in one or more computers or meter Implement in calculation machine system.In addition, any method of the invention can be implemented in one or more computer program products.This Invention some embodiments provide a kind of computer system or computer program product, coding or have for executing this Shen Please any or all method disclosed in file instruction.Such method/instruction can store deposits in CD-ROM, DVD, disk In storage product, flash drive or any other mechanized data or program storage product.Such method can also be embedding Enter in permanent memory, such as ROM, one or more programmable chips or one or more application specific integrated circuit (ASIC).It is this permanently store can be positioned in server, 802.11 access points, 802.11 wireless bridges/stand, repeater, In router, mobile phone or other electronic equipments.These methods encoded in computer program product can also be by mutual Networking or other modes by computer data signal (which is embedded software modules) with number or carrier format transmit come into Row electronic distribution.

Some embodiments of the present invention provide a kind of computer system or computer program product, and it includes the application texts Any or all of program module disclosed in part.These program modules can store in CD-ROM, DVD, flash drive, disk It stores in product or any other mechanized data or program storage product.Program module can also be embedded in permanent memory In, such as ROM, one or more programmable chips or one or more application specific integrated circuit (ASIC).It is this permanent Storage can be positioned in server, 802.11 access points, 802.11 wireless bridges/stand, repeater, router, mobile phone Or in other electronic equipments.Software module in computer program product can also pass through calculating by internet or other modes Machine data-signal (which is embedded software modules) transmits to carry out electronic distribution with number or carrier format.

Above-mentioned various methods and techniques provide multiple implementations method of the invention.Of course, it is to be understood that not described All targets or advantage can be realized according to any specific embodiment described in present specification.Thus, for example, this Field it will be recognized that this method can be to realize or optimize the advantage or one group that present specification is instructed The mode of advantage executes, without realizing such as present specification taught in or other targets or advantage of suggestion.The application File refers to various advantageous and unfavorable alternative solutions.It should be understood that some preferred embodiments specifically include one, it is another One or several favorable characteristics, and other specifically eliminate one, another or several unfavorable features, pass through there are also some Clearly reduce the unfavorable property of current unfavorable feature including one, another or several favorable characteristics.

In addition, the applicability of the various features of different embodiments will be recognized in technical staff.Similarly, discussed above each Other known equivalents ingredients of kind element, feature and step and each such element, feature or step can be by this field Those of ordinary skill be mixed and matched, execute method according to principle described in present specification.Various elements, feature and In step, it is some will be expressly included in, and other will be explicitly excluded in various specific embodiments.

Although the present invention discloses in the scene of some embodiments and example, it will be apparent to one skilled in the art that The embodiment of the present invention has exceeded specifically disclosed embodiment, can extend to other alternative embodiments and/or use, modification and In equivalent component.

Many versions and substitute element have been disclosed in an embodiment of the present invention.Further version It will be readily apparent to persons skilled in the art with substitute element.

The grouping of substitute element or embodiment of the invention disclosed in present specification is not necessarily to be construed as limitation item Part.In present specification, each single item in group can individually be quoted and prescription, can also be with its in the group He or other elements carry out any combination to be cited and prescription.It, can for the reason of the convenient and/or patentability To include to be deleted in the group or from group by projects one or more in group.When carry out it is any it is such include or delete when, Application documents specification is considered the group comprising modification, to meet all marlcush groups used in appended claims (Markushgroups) written description.

Closing, it is to be understood that the embodiment of the present invention disclosed in present specification illustrates the principle of the present invention.Other can be with The modification of use may belong to the scope of the present invention.Therefore, it by example, rather than limits, alternative configuration of the invention can be with It is used according to the introduction of present specification.Therefore, the embodiment of the present invention is not restricted to shown and described content.

Claims (31)

1. a kind of method for integrating real-life object and virtual environment, comprising:

The multidimensional image information of real-life object is provided in real time in computer equipment, wherein by one captured in real time Or separate the real-life object to extract described image information from actual environment in multiple images, it is one or more A image is captured using based on the multi-mechanism of characteristic；

Receive the selection of the virtual environment by multiple images building in real time in the computer equipment；

Each pixel and the institute of the described image information of the real-life object are provided in real time in the computer equipment The images relations between the respective pixel of each image of the described multiple images of virtual environment are stated, wherein described image relationship is extremely It less include depth relationship or transparency relationship；

And multiple integrated images are presented in real time in the computer equipment, wherein each pixel quilt in integrated image Multiple layers are divided into, and the correspondence picture in the described image information of the real-life object is used based on described image relationship Each image in the described multiple images of the plain and described virtual environment determines each layer.

2. method described in claim 1, further comprises:

In the computer equipment or another different computer equipments in real time from the real-life object in actual environment One or more of images in extract the multidimensional image information of the real-life object.

3. method described in claim 1, wherein the multi-mechanism based on characteristic be based on the real-life object with One or more differences between the actual environment in a characteristic, the characteristic includes the real-life object The visual characteristic of visual characteristic or the actual environment captured from video clip, the real-life object or the reality Real-time learning characteristic in environment, or preparatory learning characteristic related with the real-life object or the actual environment.

4. method as claimed in claim 3, wherein the visual characteristic includes spatial character, dimensional characteristic, shape feature, movement Characteristic, color characteristics, illuminance and reflection characteristic, time response or transparency characteristic, depth characteristic, material property or they Combination.

5. method as claimed in claim 4, wherein the spatial character includes three-dimensional space characteristic.

6. method as claimed in claim 3, wherein the real-time learning characteristic includes color, shape, edge, light reflectivity, light Illumination, exposure, brightness, shade, highlighted, contrast, stain, movement, depth, material, contrast or their combination.

7. method as claimed in claim 3, wherein the preparatory learning characteristic includes color, shape, edge, light reflectivity, light Illumination, exposure, brightness, shade, highlighted, contrast, stain, movement, depth, material, contrast or their combination.

8. method as claimed in claim 2, wherein described extract includes that a wheel coarse extraction and a wheel finely extract.

9. method described in claim 1, further comprises:

The virtual environment is constructed by described multiple images in the computer equipment or another different computer equipments.

10. method described in claim 1, wherein when being used to construct the virtual environment, in described multiple images at least One image is modified.

11. method described in any one of claim 10, wherein the size of at least one image, shape, picture quality, color, light, Perspective, visual effect or their combination are modified.

12. method described in claim 1, further comprises: integrating the real-life object based on described image relationship Each image of the described multiple images of described image information and the virtual environment, so that the multiple integrated image is created, Described integrate is real-time in the computer equipment or another different computer equipment and carries out pixel by pixel.

13. method described in claim 1, wherein the multiple layer of pixel includes foreground layer, middle layer and background layer.

14. method described in claim 1, wherein determining the prospect before determining the middle layer or the background layer Layer.

15. a kind of method for integrating two real-life objects and virtual environment, comprising:

The multidimensional image information of the first real-life object is provided in real time in computer equipment, wherein by one or more The first real-life object is separated in image from the first actual environment to extract described image information, it is one or more A image is captured in real time using the multi-mechanism based on characteristic；

Receive the selection of the virtual environment by multiple images building in real time in the computer equipment；

Each pixel and the institute of the described image information of the real-life object are provided in real time in the computer equipment The images relations between the respective pixel of each image of the described multiple images of virtual environment are stated, wherein described image relationship is extremely It less include depth relationship or transparency relationship；

The multidimensional image information of the extraction of second real-life object and described is provided in real time in the computer equipment Each figure of the described multiple images of each pixel and virtual environment of the described image information of second real-life object New images relationship between the respective pixel of picture, wherein the new images relationship includes at least depth relationship or transparency relationship；

Multiple integrated images are presented in real time in the computer equipment, wherein each pixel in integrated image is divided into Multiple layers use the first real-life object and described the wherein being based on described image relationship and the new images relationship Each figure of the described multiple images of respective pixel and the virtual environment in the described image information of two real-life objects As determining each layer.

16. a kind of computer system, comprising:

One or more processors；With

The addressable memory of one or more processors, the memory storage can be executed by one or more of processors Instruction, with:

The multidimensional image information of real-life object is provided in real time in computer equipment, wherein by one or more images Middle that the real-life object is extracted described image information from its practical environment separation, one or more of images are to make It is captured in real time with the multi-mechanism based on characteristic；

Receive the selection of the virtual environment by multiple images building in real time in the computer equipment；

Each pixel and the institute of the described image information of the real-life object are provided in real time in the computer equipment The images relations between the respective pixel of each image of the described multiple images of virtual environment are stated, wherein described image relationship is extremely It less include depth relationship or transparency relationship；And

Multiple integrated images are presented in real time in the computer equipment, wherein each pixel in integrated image is divided into Multiple layers, wherein be based on described image relationship, using in the described image information of the real-life object respective pixel and Each images of the described multiple images of the virtual environment determines each layer.

17. computer system described in claim 16, wherein the memory storage can be by one or more of processors The further instruction executed, with:

In real time from the real-life pair in actual environment in the computer equipment or another different computer equipments The multidimensional image information of the real-life object is extracted in one or more of images of elephant.

18. computer system described in claim 16, wherein the memory storage can be by one or more of processors The further instruction executed, with: described multiple images are constructed in the computer equipment or another different computer equipments The virtual environment.

19. computer system described in claim 16, wherein the memory storage can be by one or more of processors The further instruction executed, with:

Based on described image relationship, in the computer equipment or another different computer equipments in a manner of pixel-by-pixel in real time Each image in the described image information of the real-life object and the described multiple images of the virtual environment is integrated, from And create the multiple integrated image.

20. computer system described in claim 16, wherein the memory storage can be by one or more of processors The further instruction executed, with:

The computer equipment provide in real time each pixel of the described image information of the second real-life object with it is described New images relationship between the respective pixel of each image of multiple images, wherein the new images relationship is closed including at least depth System or transparency relationship.

21. computer system described in claim 16, wherein the memory storage can be by one or more of processors The further instruction executed, with:

Based on described image relationship and the new images relationship, in the computer equipment or another different computer equipments with Pixel-by-pixel mode integrate in real time the described image information of the real-life object, the second real-life object it is described Each image of the described multiple images of new image information and the virtual environment, to create multiple new integrated images.

22. a kind of non-transitory computer-readable medium, comprising instruction, when it includes instruction be subsequently can by computer device and execute When, these instructions can make the computer processor:

The multidimensional image information of real-life object is provided in real time in computer equipment, wherein by one or more images Middle that the real-life object is extracted described image information with its practical environment separation, one or more of images are to make It is captured in real time with the multi-mechanism based on characteristic；

Receive the selection of the virtual environment by multiple images building in real time in the computer equipment；

Each pixel and the institute of the described image information of the real-life object are provided in real time in the computer equipment The images relations between the respective pixel of each image of the described multiple images of virtual environment are stated, wherein described image relationship is extremely It less include depth relationship or transparency relationship；

Multiple integrated images are presented in real time in the computer equipment, wherein each pixel in integrated image is divided into Multiple layers, wherein be based on described image relationship, using in the described image information of the real-life object respective pixel and Each images of the described multiple images of the virtual environment determines each layer.

23. non-transitory computer-readable medium described in claim 22, wherein the non-transitory computer-readable medium packet Containing the further instruction that can be executed by one or more of processors, with:

In real time from the real-life pair in actual environment in the computer equipment or another different computer equipments The multidimensional image information of the real-life object is extracted in one or more of images of elephant.

24. non-transitory computer-readable medium described in claim 22, wherein the non-transitory computer-readable medium packet Containing the further instruction that can be executed by one or more of processors, with:

The virtual environment of described multiple images is constructed in the computer equipment or another different computer equipments.

25. non-transitory computer-readable medium described in claim 22, wherein the non-transitory computer-readable medium packet Containing the further instruction that can be executed by one or more of processors, with:

Based on described image relationship, in the computer equipment or another different computer equipments in a manner of pixel-by-pixel in real time Each image in the described image information of the real-life object and the described multiple images of the virtual environment is integrated, from And create the multiple integrated image.

26. non-transitory computer-readable medium described in claim 22, wherein the non-transitory computer-readable medium packet Containing the further instruction that can be executed by one or more of processors, with:

The computer equipment provide in real time each pixel of the described image information of the second real-life object with it is described New images relationship between the respective pixel of each image of multiple images, wherein the new images relationship is closed including at least depth System or transparency relationship.

27. non-transitory computer-readable medium described in claim 22, wherein the non-transitory computer-readable medium packet Containing the further instruction that can be executed by one or more of processors, with:

Based on described image relationship and the new images relationship, in the computer equipment or another different computer equipments with Pixel-by-pixel mode integrate in real time the described image information of the real-life object, the two real-lifes object it is described Each image of the described multiple images of new image information and the virtual environment, to create multiple new integrated images.

28. method described in claim 1, wherein being embedded in one or more ad elements in the multiple integrated image.

29. method described in claim 28, wherein the institute of one or more of ad elements and the real-life object The image information, the virtual environment or their combination for stating extraction are associated.

30. method of claim 15, wherein being embedded in one or more ad elements in the multiple integrated image.

31. method described in claim 30, wherein one or more of ad elements and the first real-life object The image information of the extraction, the image information of the extraction of the second real-life object, the virtual environment or Their combination is associated.