WO2019137006A1 - 虚实对象合成方法及装置 - Google Patents
虚实对象合成方法及装置 Download PDFInfo
- Publication number
- WO2019137006A1 WO2019137006A1 PCT/CN2018/101241 CN2018101241W WO2019137006A1 WO 2019137006 A1 WO2019137006 A1 WO 2019137006A1 CN 2018101241 W CN2018101241 W CN 2018101241W WO 2019137006 A1 WO2019137006 A1 WO 2019137006A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- depth information
- virtual object
- virtual
- target object
- terminal device
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Definitions
- the present disclosure relates to the field of image processing technologies, and in particular, to a method and apparatus for synthesizing virtual and real objects.
- AR Augmented Reality
- AR Augmented Reality
- the virtual and real object synthesis method in the related art can correctly handle the occlusion relationship of the virtual real object in some cases, but has a large amount of calculation, and is not suitable for the terminal device such as a mobile phone or a tablet computer.
- the present disclosure provides a virtual and real object synthesis method and apparatus, which can correctly display a virtual object in an image captured by a terminal device in real time.
- a method for synthesizing a virtual object comprising: acquiring a target object in an image captured by the terminal device; in the depth information fusion mode Obtaining depth information of the target object by using an imaging device of the terminal device, where the depth information fusion mode is used to superimpose a virtual object generated by computer modeling into the image, and depth information of the target object is used for Representing a distance between the target object and the terminal device; acquiring depth information of the virtual object; and superimposing the virtual object into the image based on depth information of the target object and depth information of the virtual object.
- the superimposing the virtual object into the image based on the depth information of the target object and the depth information of the virtual object includes: if the depth information of the target object is smaller than Depicting the depth information of the virtual object, the portion of the virtual object that is not obscured by the target object is superimposed on the image.
- the superimposing the virtual object into the image based on the depth information of the target object and the depth information of the virtual object includes: if the depth information of the target object is greater than or Equal to the depth information of the virtual object, the virtual object is superimposed on the image.
- the camera device of the terminal device includes at least two cameras, and acquiring the depth information of the target object by using the camera device of the terminal device, including: according to the target object and the at least The positional relationship of the two cameras and the distance between the at least two cameras determine depth information of the target object.
- acquiring the depth information of the virtual object includes: acquiring location information of a location of the virtual object; and determining depth information of the virtual object according to the location information.
- a virtual object synthesizing device configured to apply to a terminal device, and the device includes: an object acquiring module, configured to acquire a target object in an image captured by the terminal device; An information acquiring module, configured to acquire depth information of the target object by using an imaging device of the terminal device in the depth information fusion mode, where the depth information fusion mode is used to superimpose a virtual object generated by computer modeling to In the image, the depth information of the target object is used to represent the distance between the target object and the terminal device; the second information acquiring module is configured to acquire depth information of the virtual object; and the overlay module is configured to be based on the target object The depth information and the depth information of the virtual object, the virtual object is superimposed into the image.
- the superimposing module includes: a first superposition sub-module, configured to superimpose the virtual image on the image when depth information of the target object is smaller than depth information of the virtual object The portion of the object that is not obscured by the target object.
- the superimposing module further includes: a second superposition sub-module, configured to superimpose on the image when the depth information of the target object is greater than or equal to the depth information of the virtual object The virtual object.
- the first obtaining module includes: a first determining submodule, configured to: according to a positional relationship between the target object and the at least two cameras, and between the at least two cameras The distance determines depth information of the target object.
- the second obtaining module includes: an acquiring submodule, configured to acquire location information of a location of the virtual object; and a second determining submodule, configured to determine, according to the location information, Describe the depth information of the virtual object.
- the technical solution provided by the embodiment of the present disclosure may include the following effects: the occlusion relationship of the object of different depth information is different.
- the terminal device after acquiring the target object, the terminal device is based on the depth information of the target object and the depth information of the virtual object.
- the virtual object is superimposed on the image captured by the terminal device, and the superimposing manner can be determined according to the comparison result of the depth information when the target object is acquired, instead of directly superimposing or comparing the depth information of all the objects and superimposing, the superposition can be reduced.
- the amount of calculation in the process can be better applied to the AR application in the terminal device.
- the embodiment of the present disclosure utilizes the camera device of the terminal device to acquire the depth information of the target object, which is advantageous for the application of the embodiment on the terminal device. Therefore, the virtual object synthesis method and apparatus according to aspects of the present disclosure can correctly display a virtual object in an image captured by the terminal device in real time.
- FIG. 1 is a flowchart of a method for synthesizing virtual objects according to an exemplary embodiment.
- FIG. 2 is a flowchart of a method for synthesizing virtual objects according to an exemplary embodiment.
- FIG. 3 is a block diagram of a virtual real object synthesizing apparatus according to an exemplary embodiment.
- FIG. 4 is a block diagram of a virtual real object synthesizing apparatus according to an exemplary embodiment.
- FIG. 5 is a block diagram of a virtual real object synthesizing apparatus according to an exemplary embodiment.
- FIG. 6 is a block diagram of a virtual real object synthesizing apparatus according to an exemplary embodiment.
- FIG. 1 is a flowchart of a method for synthesizing a virtual object according to an exemplary embodiment. As shown in FIG. 1 , the method for synthesizing a virtual object is used in a terminal device, such as a mobile phone, a tablet computer, etc. Includes the following steps:
- step S11 a target object in the image captured by the terminal device is acquired.
- the image captured by the terminal device may be an image corresponding to the real scene, and the terminal device may capture the image by using the camera device of the camera.
- the terminal device can capture the image in real time, and the terminal device can also continuously capture multiple images. It can be understood that a plurality of real-time images continuously captured by the terminal device can constitute a video.
- the target object can be a real object in a real scene, for example, it can be a real person or an object.
- the target object can be a specific object in the real scene, or it can be a certain type of object in the real scene. For example, there are multiple people in a real scene, and the target object may be one of a plurality of people, or may be all people in a real scene.
- the target object can be an object that has an occlusion relationship with the virtual object. The present disclosure does not limit the target object.
- the terminal device can identify and acquire the target object in the image by using an image recognition technology in the related art.
- the target object can be moved in an image captured by the terminal device, for example, a person can walk around in a real scene.
- the target image may enter or disappear from the image, such as a person walking into a real scene or leaving a real scene.
- step S12 in the depth information fusion mode, the depth information of the target object is acquired by the camera device of the terminal device, and the depth information fusion mode is used to superimpose a virtual object generated by computer modeling into the image.
- the depth information of the target object is used to represent the distance between the target object and the terminal device.
- AR technology can superimpose virtual objects into real scenes.
- virtual objects can be used to represent objects generated by computer modeling, such as virtual people, objects, and the like.
- the virtual object may be a computer-created mathematical model, and the mathematical model may include data representing features such as outlines, textures, and the like of the virtual object, and the data may be extracted based on real objects such as real people, objects, or the like, or may be manually created as needed.
- Image rendering can be based on this data, and the virtual object has been displayed in the real scene.
- the terminal device can change the position of the virtual object so that the virtual object moves in the image corresponding to the real scene.
- the present disclosure does not limit the manner in which virtual objects are generated.
- the terminal device can generate virtual objects locally, and can also acquire virtual objects from other devices. After the terminal device obtains the virtual object from other devices, the virtual object can be locally adjusted.
- the terminal device may directly display the virtual object before the captured image when the target object is not acquired.
- the terminal device can display the virtual object on a layer above the layer in which the image is located. In this way, the terminal device can quickly and conveniently superimpose the virtual object into the real scene.
- the present disclosure does not limit the operation when the target object is not acquired.
- the terminal device may initiate the depth information fusion mode when the target object is acquired.
- the depth information fusion mode can be used to superimpose a virtual object into the image.
- the depth information fusion mode may determine an occlusion relationship of the virtual object and the target object based on the depth information, and determine a manner of superimposing the virtual object into the image according to the occlusion relationship.
- the fusion of virtual objects in the terminal device is more realistic.
- the target object is obtained, the object captured by the terminal device may have an occlusion relationship with the virtual object. In this case, incorrect occlusion may cause poor fusion effect, and some scenarios cannot be implemented, so the terminal device can be started. Deep information fusion mode for more realistic integration.
- the target object is a person and the virtual object is a stool.
- the deep information fusion mode can be activated, thereby realizing the scene where the person stands in front of the stool and sits on the stool.
- the depth information of the target object is used to indicate the distance between the target object and the terminal device.
- the realistic objective world is a three-dimensional space that becomes a two-dimensional image that loses the third-dimensional depth information after being imaged by the camera.
- the terminal device can acquire depth information of the target object through the camera device.
- the camera device of the terminal device includes at least two cameras, and the terminal device may be configured according to a positional relationship between the target object and the at least two cameras, and a distance between the at least two cameras. Determining depth information of the target object.
- the terminal device can determine the angle between the connection between the target object and the two cameras and the connection between the two cameras, and the distance between the two cameras. Depth information of the target object. The above is only one example of determining the depth information of the target object, and the terminal device may also determine the depth information of the target object according to other manners in the related art.
- step S13 depth information of the virtual object is acquired.
- the terminal device can place the virtual model corresponding to the virtual object at different positions of the captured image, and can move the position of the virtual model.
- the virtual object is not a real object in the image captured by the terminal device, and corresponds to a virtual model.
- a virtual model can be called a virtual object when it is displayed on an image.
- the depth information of the virtual object may be fixed.
- the terminal device can record the correspondence between the virtual model and the depth information in the process of generating the virtual model.
- the terminal device may obtain the depth information corresponding to the virtual object according to the correspondence between the virtual model and the depth information.
- the depth information of the virtual object may change as the virtual model moves in the image.
- the terminal device may record the depth information in the location information of the virtual model in the process of generating the virtual model, and may use the location information of the virtual model as the location information of the virtual object.
- the terminal device may acquire location information of a location of the virtual object, and determine depth information of the virtual object according to the location information.
- step S14 the virtual object is superimposed into the image based on the depth information of the target object and the depth information of the virtual object.
- An object with a large depth information may be occluded by an object having a small depth information, and only a part of an object having a large depth information may be displayed in the captured image of the terminal device.
- FIG. 2 is a flowchart of a method for synthesizing virtual objects according to an exemplary embodiment. As shown in FIG. 2, step S14 includes the following steps:
- step S141 it is determined whether the depth information of the target object is smaller than the depth information of the virtual object. If the depth information of the target object is smaller than the depth information of the virtual object, step S142 is performed; otherwise, step S143 is performed.
- step S142 a portion of the virtual object that is not occluded by the target object is superimposed on the image.
- a pixel location of each pixel of the target object and a pixel location of each pixel of the virtual object may be acquired.
- the pixel points of the virtual object may not be drawn at the position.
- the pixel point of the virtual object can be drawn at the position.
- the target object is a person
- the virtual object is a stool
- the depth information of the person is less than the depth information of the stool.
- the terminal device may superimpose a portion of the stool that is not obscured by the person on the image. In this way, it is possible to present a scene in which a person stands in front of a stool and sits on a stool.
- step S143 the virtual object is superimposed on the image.
- the target object is a person
- the virtual object is a stool
- the depth information of the person is greater than or equal to the depth information of the stool.
- the terminal device can superimpose the stool directly on the image. In this way, it is possible to present a scene in which a person stands behind a stool.
- the occlusion relationship of the object of the different depth information is different.
- the terminal device when the target object is acquired, the terminal device superimposes the virtual object on the image captured by the terminal device based on the depth information of the target object and the depth information of the virtual object.
- the superposition method is determined according to the comparison result of the depth information, instead of directly superimposing or comparing the depth information of all the objects and superimposing, the calculation amount in the superimposition process can be reduced, so that the terminal device can be better applied to the terminal device.
- the embodiment of the present disclosure utilizes the camera device of the terminal device to acquire the depth information of the target object, which is advantageous for the application of the embodiment on the terminal device. Therefore, according to the virtual object synthesis method of the embodiment, the virtual object can be correctly displayed in the image captured by the terminal device in real time.
- FIG. 3 is a block diagram of a virtual real object synthesizing apparatus according to an exemplary embodiment.
- the virtual object synthesizing device 30 is applied to a terminal device, and the device 30 includes an object acquiring module 31, a first information acquiring module 32, a second information acquiring module 33, and a superimposing module 34.
- the object acquisition module 31 is configured to acquire a target object in an image captured by the terminal device
- the first information acquiring module 32 is configured to acquire depth information of the target object by using an imaging device of the terminal device in the depth information fusion mode, where the depth information fusion mode is used to model a computer a virtual object is superimposed into the image, and the depth information of the target object is used to indicate a distance between the target object and the terminal device;
- the second information acquiring module 33 is configured to acquire depth information of the virtual object
- the overlay module 34 is configured to overlay the virtual object into the image based on depth information of the target object and depth information of the virtual object.
- FIG. 4 is a block diagram of a virtual real object synthesizing apparatus according to an exemplary embodiment.
- the overlay module 34 includes a first overlay sub-module 341.
- the first superposition sub-module 341 is configured to superimpose, on the image, a portion of the virtual object that is not occluded by the target object when depth information of the target object is smaller than depth information of the virtual object.
- the overlay module further includes a second overlay sub-module 352.
- the second superposition sub-module 342 is configured to superimpose the virtual object on the image when the depth information of the target object is greater than or equal to the depth information of the virtual object.
- the camera device of the terminal device includes at least two cameras, and the first information acquiring module 32 includes a first determining sub-module 321 .
- the first determining sub-module 321 is configured to determine depth information of the target object according to a positional relationship between the target object and the at least two cameras, and a distance between the at least two cameras.
- the second information acquiring module 33 includes an obtaining submodule 331 and a second determining submodule 332.
- the obtaining submodule 331 is configured to acquire location information of the virtual object in the image
- the second determining submodule 332 is configured to determine depth information of the virtual object according to location information of the virtual object.
- FIG. 5 is a block diagram of an apparatus 800 for virtual reality object composition, according to an exemplary embodiment.
- device 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a gaming console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.
- apparatus 800 can include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, And a communication component 816.
- Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations.
- Processing component 802 can include one or more processors 820 to execute instructions to perform all or part of the steps of the above described methods.
- processing component 802 can include one or more modules to facilitate interaction between component 802 and other components.
- processing component 802 can include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
- Memory 804 is configured to store various types of data to support operation at device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phone book data, messages, pictures, videos, and the like.
- the memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable.
- SRAM static random access memory
- EEPROM electrically erasable programmable read only memory
- EPROM Electrically erasable programmable read only memory
- PROM Programmable Read Only Memory
- ROM Read Only Memory
- Magnetic Memory Flash Memory
- Disk Disk or Optical Disk.
- Power component 806 provides power to various components of device 800.
- Power component 806 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for device 800.
- the multimedia component 808 includes a screen between the device 800 and the user that provides an output interface.
- the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user.
- the touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may sense not only the boundary of the touch or sliding action, but also the duration and pressure associated with the touch or slide operation.
- the multimedia component 808 includes a front camera and/or a rear camera. When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.
- the audio component 810 is configured to output and/or input an audio signal.
- the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode.
- the received audio signal may be further stored in memory 804 or transmitted via communication component 816.
- the audio component 810 also includes a speaker for outputting an audio signal.
- the I/O interface 812 provides an interface between the processing component 802 and the peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button.
- Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of device 800.
- sensor assembly 814 can detect an open/closed state of device 800, relative positioning of components, such as the display and keypad of device 800, and sensor component 814 can also detect a change in position of one component of device 800 or device 800. The presence or absence of user contact with device 800, device 800 orientation or acceleration/deceleration, and temperature variation of device 800.
- Sensor assembly 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact.
- Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications.
- the sensor assembly 814 can also include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
- Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices.
- the device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof.
- communication component 816 receives broadcast signals or broadcast associated information from an external broadcast management system via a broadcast channel.
- the communication component 816 also includes a near field communication (NFC) module to facilitate short range communication.
- NFC near field communication
- the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
- RFID radio frequency identification
- IrDA infrared data association
- UWB ultra-wideband
- Bluetooth Bluetooth
- device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor, or other electronic component implementation for performing the above methods.
- ASICs application specific integrated circuits
- DSPs digital signal processors
- DSPDs digital signal processing devices
- PLDs programmable logic devices
- FPGA field programmable A gate array
- controller microcontroller, microprocessor, or other electronic component implementation for performing the above methods.
- non-transitory computer readable storage medium comprising instructions, such as a memory 804 comprising instructions executable by processor 820 of apparatus 800 to perform the above method.
- the non-transitory computer readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.
- FIG. 6 is a block diagram of an apparatus 1900 for virtual and real object composition, according to an exemplary embodiment.
- device 1900 can be provided as a server.
- apparatus 1900 includes a processing component 1922 that further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by processing component 1922, such as an application.
- An application stored in memory 1932 can include one or more modules each corresponding to a set of instructions.
- processing component 1922 is configured to execute instructions to perform the methods described above.
- Apparatus 1900 can also include a power supply component 1926 configured to perform power management of apparatus 1900, a wired or wireless network interface 1950 configured to connect apparatus 1900 to the network, and an input/output (I/O) interface 1958.
- Device 1900 can operate based on an operating system stored in memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.
- non-transitory computer readable storage medium comprising instructions, such as a memory 1932 comprising instructions executable by processing component 1922 of apparatus 1900 to perform the above method.
- the non-transitory computer readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.
Abstract
Description
Claims (12)
- 一种虚实对象合成方法,其特征在于,所述方法应用于终端设备,所述方法包括:获取所述终端设备捕捉的图像中的目标对象;在深度信息融合模式下,通过所述终端设备的摄像装置获取所述目标对象的深度信息,所述深度信息融合模式用于将计算机建模产生的虚拟对象叠加到所述图像中,所述目标对象的深度信息用于表示目标对象与终端设备之间的距离;获取虚拟对象的深度信息;基于所述目标对象的深度信息和所述虚拟对象的深度信息,将所述虚拟对象叠加到所述图像中。
- 根据权利要求1所述的虚实对象合成方法,其特征在于,基于所述目标对象的深度信息和所述虚拟对象的深度信息,将所述虚拟对象叠加到所述图像中,包括:若所述目标对象的深度信息小于所述虚拟对象的深度信息,则在所述图像上叠加所述虚拟对象未被所述目标对象遮挡的部分。
- 根据权利要求1所述的虚实对象合成方法,其特征在于,基于所述目标对象的深度信息和所述虚拟对象的深度信息,将所述虚拟对象叠加到所述图像中,包括:若所述目标对象的深度信息大于或等于所述虚拟对象的深度信息,则在所述图像上叠加所述虚拟对象。
- 根据权利要求1至3任一项所述的虚实对象合成方法,其特征在于,所述终端设备的摄像装置包括至少两个摄像头,通过所述终端设备的摄像装置获取所述目标对象的深度信息,包括:根据所述目标对象与所述至少两个摄像头的位置关系,以及所述至少两个摄像头之间的距离确定所述目标对象的深度信息。
- 根据权利要求1至3任一项所述的虚实对象合成方法,其特征在于,获取虚拟对象的深度信息,包括:获取放置所述虚拟对象的位置的位置信息;根据所述位置信息,确定所述虚拟对象的深度信息。
- 一种虚实对象合成装置,其特征在于,所述装置应用于终端设备,所述装置包括:对象获取模块,用于获取所述终端设备捕捉的图像中的目标对象;第一信息获取模块,用于在所述深度信息融合模式下,通过所述终端设备的摄像装置获取所述目标对象的深度信息,所述深度信息融合模式用于将计算机建模产生的虚拟对象叠加到所述图像中,所述目标对象的深度信息用于表示目标对象与终端设备之间的距离;第二信息获取模块,用于获取虚拟对象的深度信息;叠加模块,用于基于所述目标对象的深度信息和所述虚拟对象的深度信息,将所述虚拟对象叠加到所述图像中。
- 根据权利要求6所述的虚实对象合成装置,其特征在于,所述叠加模块包括:第一叠加子模块,用于当所述目标对象的深度信息小于所述虚拟对象的深度信息时,在所述图像上叠加所述虚拟对象未被所述目标对象遮挡的部分。
- 根据权利要求6所述的虚实对象合成装置,其特征在于,所述叠加模块还包括:第二叠加子模块,用于当所述目标对象的深度信息大于或等于所述虚拟对象的深度信息时,在所述图像上叠加所述虚拟对象。
- 根据权利要求6至8任一项所述的虚实对象合成装置,其特征在于,所述终端设备的摄像装置包括至少两个摄像头,所述第一信息获取模块包括:第一确定子模块,用于根据所述目标对象与所述至少两个摄像头的位置关系,以及所述至少两个摄像头之间的距离确定所述目标对象的深度信息。
- 根据权利要求6至8任一项所述的虚实对象合成装置,其特征在于,所述第二信息获取模块包括:获取子模块,用于获取放置所述虚拟对象的位置的位置信息;第二确定子模块,用于根据所述位置信息,确定所述虚拟对象的深 度信息。
- 一种虚实对象合成装置,其特征在于,包括:处理器;用于存储处理器可执行指令的存储器;其中,所述处理器被配置为执行如权利要求1至5中任一项所述的方法。
- 一种非临时性计算机可读存储介质,当所述存储介质中的指令由处理器执行时实现权利要求1至5中任一项所述的方法。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
RU2018143709A RU2715797C1 (ru) | 2018-01-12 | 2018-08-20 | Способ и устройство для синтеза объектов виртуальной реальности |
JP2018567578A JP2020507136A (ja) | 2018-01-12 | 2018-08-20 | Vrオブジェクトの合成方法、装置、プログラム及び記録媒体 |
KR1020197004558A KR102194094B1 (ko) | 2018-01-12 | 2018-08-20 | 가상과 실제 물체의 합성 방법, 장치, 프로그램 및 기록매체 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810029995.2 | 2018-01-12 | ||
CN201810029995.2A CN108182730B (zh) | 2018-01-12 | 2018-01-12 | 虚实对象合成方法及装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019137006A1 true WO2019137006A1 (zh) | 2019-07-18 |
Family
ID=62550371
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2018/101241 WO2019137006A1 (zh) | 2018-01-12 | 2018-08-20 | 虚实对象合成方法及装置 |
Country Status (7)
Country | Link |
---|---|
US (1) | US11636653B2 (zh) |
EP (1) | EP3511864A1 (zh) |
JP (1) | JP2020507136A (zh) |
KR (1) | KR102194094B1 (zh) |
CN (1) | CN108182730B (zh) |
RU (1) | RU2715797C1 (zh) |
WO (1) | WO2019137006A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11636653B2 (en) | 2018-01-12 | 2023-04-25 | Beijing Xiaomi Mobile Software Co., Ltd. | Method and apparatus for synthesizing virtual and real objects |
Families Citing this family (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108961197A (zh) * | 2018-06-27 | 2018-12-07 | 联想(北京)有限公司 | 一种对象合成方法及装置 |
CN110716685B (zh) * | 2018-07-11 | 2023-07-18 | 广东虚拟现实科技有限公司 | 图像显示方法,图像显示装置、系统及其实体对象 |
US10785413B2 (en) * | 2018-09-29 | 2020-09-22 | Apple Inc. | Devices, methods, and graphical user interfaces for depth-based annotation |
CN109474816B (zh) * | 2018-12-28 | 2024-04-05 | 上海北冕信息科技有限公司 | 增强现实的虚实融合装置及其虚实融合方法、设备、介质 |
CN109922331B (zh) * | 2019-01-15 | 2021-12-07 | 浙江舜宇光学有限公司 | 一种图像处理方法及装置 |
CN111651031B (zh) * | 2019-03-04 | 2023-08-29 | 广东虚拟现实科技有限公司 | 虚拟内容的显示方法、装置、终端设备及存储介质 |
JP2021064320A (ja) * | 2019-10-17 | 2021-04-22 | ソニー株式会社 | 情報処理装置、情報処理方法、及びプログラム |
CN110889890B (zh) * | 2019-11-29 | 2023-07-28 | 深圳市商汤科技有限公司 | 图像处理方法及装置、处理器、电子设备及存储介质 |
CN111223192B (zh) * | 2020-01-09 | 2023-10-03 | 北京华捷艾米科技有限公司 | 一种图像处理方法及其应用方法、装置及设备 |
US11727650B2 (en) | 2020-03-17 | 2023-08-15 | Apple Inc. | Systems, methods, and graphical user interfaces for displaying and manipulating virtual objects in augmented reality environments |
CN111416938B (zh) * | 2020-03-27 | 2021-11-02 | 咪咕文化科技有限公司 | 一种增强现实合拍方法、设备及计算机可读存储介质 |
CN111638798A (zh) * | 2020-06-07 | 2020-09-08 | 上海商汤智能科技有限公司 | 一种ar合影方法、装置、计算机设备及存储介质 |
CN111640203B (zh) * | 2020-06-12 | 2024-04-12 | 上海商汤智能科技有限公司 | 一种图像处理方法及装置 |
CN111898640B (zh) * | 2020-06-28 | 2023-10-31 | 武汉旷视金智科技有限公司 | 模拟抓拍机推送图片的方法、装置、测试系统及电子设备 |
CN112422945A (zh) * | 2020-09-01 | 2021-02-26 | 华为技术有限公司 | 图像处理方法、移动终端及存储介质 |
CN112394815A (zh) * | 2020-11-20 | 2021-02-23 | 歌尔科技有限公司 | 基于增强现实的绘画辅助方法、增强现实设备和存储介质 |
CN112906467A (zh) * | 2021-01-15 | 2021-06-04 | 深圳市慧鲤科技有限公司 | 合影图像生成方法及装置、电子设备和存储介质 |
US11941764B2 (en) | 2021-04-18 | 2024-03-26 | Apple Inc. | Systems, methods, and graphical user interfaces for adding effects in augmented reality environments |
CN113269782B (zh) * | 2021-04-21 | 2023-01-03 | 青岛小鸟看看科技有限公司 | 数据生成方法、装置及电子设备 |
CN113538318B (zh) * | 2021-08-24 | 2023-12-15 | 北京奇艺世纪科技有限公司 | 图像处理方法、装置、终端设备以及可读存储介质 |
CN114564108A (zh) * | 2022-03-03 | 2022-05-31 | 北京小米移动软件有限公司 | 图像展示的方法、装置和存储介质 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102473324A (zh) * | 2009-08-18 | 2012-05-23 | Metaio有限公司 | 用于在真实环境中表示虚拟信息的方法 |
CN103489214A (zh) * | 2013-09-10 | 2014-01-01 | 北京邮电大学 | 增强现实系统中基于虚拟模型预处理的虚实遮挡处理方法 |
CN104021590A (zh) * | 2013-02-28 | 2014-09-03 | 北京三星通信技术研究有限公司 | 虚拟试穿试戴系统和虚拟试穿试戴方法 |
CN105931289A (zh) * | 2016-04-14 | 2016-09-07 | 大连新锐天地传媒有限公司 | 实现真实模型遮盖虚拟物体的系统及方法 |
CN106030335A (zh) * | 2014-03-28 | 2016-10-12 | 英特尔公司 | 使用微功率脉冲雷达确定移动显示位置和取向 |
CN108182730A (zh) * | 2018-01-12 | 2018-06-19 | 北京小米移动软件有限公司 | 虚实对象合成方法及装置 |
Family Cites Families (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6166744A (en) * | 1997-11-26 | 2000-12-26 | Pathfinder Systems, Inc. | System for combining virtual images with real-world scenes |
JP2002157607A (ja) * | 2000-11-17 | 2002-05-31 | Canon Inc | 画像生成システム、画像生成方法および記憶媒体 |
JP2004145448A (ja) * | 2002-10-22 | 2004-05-20 | Toshiba Corp | 端末装置、サーバ装置および画像加工方法 |
JP4522129B2 (ja) * | 2004-03-31 | 2010-08-11 | キヤノン株式会社 | 画像処理方法および画像処理装置 |
DE102008060301B4 (de) | 2008-12-03 | 2012-05-03 | Grenzebach Maschinenbau Gmbh | Verfahren und Vorrichtung zum kraftschlüssigen Verbinden von glasartigen Bauteilen mit Metallen sowie Computerprogramm und maschinenlesbarer Träger zur Durchführung des Verfahrens |
JP2012058968A (ja) * | 2010-09-08 | 2012-03-22 | Namco Bandai Games Inc | プログラム、情報記憶媒体及び画像生成システム |
US8884984B2 (en) * | 2010-10-15 | 2014-11-11 | Microsoft Corporation | Fusing virtual content into real content |
JP5791434B2 (ja) * | 2011-08-31 | 2015-10-07 | 任天堂株式会社 | 情報処理プログラム、情報処理システム、情報処理装置および情報処理方法 |
US9224237B2 (en) * | 2013-09-27 | 2015-12-29 | Amazon Technologies, Inc. | Simulating three-dimensional views using planes of content |
JP2015114307A (ja) * | 2013-12-16 | 2015-06-22 | ソニー株式会社 | 画像処理装置と画像処理方法および撮像装置 |
US9690370B2 (en) * | 2014-05-05 | 2017-06-27 | Immersion Corporation | Systems and methods for viewport-based augmented reality haptic effects |
JP5762600B1 (ja) * | 2014-06-06 | 2015-08-12 | キヤノン株式会社 | 情報処理装置及び情報処理方法 |
US20160019718A1 (en) * | 2014-07-16 | 2016-01-21 | Wipro Limited | Method and system for providing visual feedback in a virtual reality environment |
US10726593B2 (en) * | 2015-09-22 | 2020-07-28 | Fyusion, Inc. | Artificially rendering images using viewpoint interpolation and extrapolation |
RU2606874C1 (ru) * | 2015-12-02 | 2017-01-10 | Виталий Витальевич Аверьянов | Способ управления устройством формирования среды дополненной реальности |
GB2551396B (en) * | 2016-06-17 | 2018-10-10 | Imagination Tech Ltd | Augmented reality occlusion |
JP2018097141A (ja) * | 2016-12-13 | 2018-06-21 | 富士ゼロックス株式会社 | 頭部装着型表示装置、仮想物体表示システム |
-
2018
- 2018-01-12 CN CN201810029995.2A patent/CN108182730B/zh active Active
- 2018-08-20 KR KR1020197004558A patent/KR102194094B1/ko active IP Right Grant
- 2018-08-20 RU RU2018143709A patent/RU2715797C1/ru active
- 2018-08-20 WO PCT/CN2018/101241 patent/WO2019137006A1/zh active Application Filing
- 2018-08-20 JP JP2018567578A patent/JP2020507136A/ja active Pending
- 2018-12-17 US US16/222,546 patent/US11636653B2/en active Active
-
2019
- 2019-01-11 EP EP19151454.6A patent/EP3511864A1/en active Pending
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102473324A (zh) * | 2009-08-18 | 2012-05-23 | Metaio有限公司 | 用于在真实环境中表示虚拟信息的方法 |
CN104021590A (zh) * | 2013-02-28 | 2014-09-03 | 北京三星通信技术研究有限公司 | 虚拟试穿试戴系统和虚拟试穿试戴方法 |
CN103489214A (zh) * | 2013-09-10 | 2014-01-01 | 北京邮电大学 | 增强现实系统中基于虚拟模型预处理的虚实遮挡处理方法 |
CN106030335A (zh) * | 2014-03-28 | 2016-10-12 | 英特尔公司 | 使用微功率脉冲雷达确定移动显示位置和取向 |
CN105931289A (zh) * | 2016-04-14 | 2016-09-07 | 大连新锐天地传媒有限公司 | 实现真实模型遮盖虚拟物体的系统及方法 |
CN108182730A (zh) * | 2018-01-12 | 2018-06-19 | 北京小米移动软件有限公司 | 虚实对象合成方法及装置 |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11636653B2 (en) | 2018-01-12 | 2023-04-25 | Beijing Xiaomi Mobile Software Co., Ltd. | Method and apparatus for synthesizing virtual and real objects |
Also Published As
Publication number | Publication date |
---|---|
KR20190086658A (ko) | 2019-07-23 |
EP3511864A1 (en) | 2019-07-17 |
JP2020507136A (ja) | 2020-03-05 |
RU2715797C1 (ru) | 2020-03-03 |
CN108182730B (zh) | 2022-08-12 |
CN108182730A (zh) | 2018-06-19 |
US20190221041A1 (en) | 2019-07-18 |
US11636653B2 (en) | 2023-04-25 |
KR102194094B1 (ko) | 2020-12-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2019137006A1 (zh) | 虚实对象合成方法及装置 | |
US9674395B2 (en) | Methods and apparatuses for generating photograph | |
US20170304735A1 (en) | Method and Apparatus for Performing Live Broadcast on Game | |
CN106408603B (zh) | 拍摄方法及装置 | |
EP3179408A2 (en) | Picture processing method and apparatus, computer program and recording medium | |
WO2017088266A1 (zh) | 图片处理方法及装置 | |
CN106210496B (zh) | 照片拍摄方法及装置 | |
CN108154465B (zh) | 图像处理方法及装置 | |
WO2016127671A1 (zh) | 图像滤镜生成方法及装置 | |
WO2016192325A1 (zh) | 视频文件的标识处理方法及装置 | |
CN110853095B (zh) | 相机定位方法、装置、电子设备及存储介质 | |
CN109922252B (zh) | 短视频的生成方法及装置、电子设备 | |
CN109218709B (zh) | 全息内容的调整方法及装置和计算机可读存储介质 | |
CN116939275A (zh) | 直播虚拟资源展示方法、装置、电子设备、服务器及介质 | |
KR102557592B1 (ko) | 이미지 표시 방법 및 장치, 전자 기기, 컴퓨터 판독 가능 저장 매체 | |
CN111586296B (zh) | 图像拍摄方法、图像拍摄装置及存储介质 | |
CN113315903B (zh) | 图像获取方法和装置、电子设备、存储介质 | |
US9619016B2 (en) | Method and device for displaying wallpaper image on screen | |
CN113989424A (zh) | 三维虚拟形象的生成方法、装置及电子设备 | |
CN113286073A (zh) | 拍摄方法、拍摄装置及存储介质 | |
CN108159686B (zh) | 投影设备投影的方法、装置及存储介质 | |
CN106598217B (zh) | 显示方法、显示装置和电子设备 | |
CN109447929B (zh) | 图像合成方法及装置 | |
CN110458962B (zh) | 一种图像处理方法、装置、电子设备及存储介质 | |
CN106713748B (zh) | 发送图片的方法及装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 2018567578 Country of ref document: JP Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 20197004558 Country of ref document: KR Kind code of ref document: A |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18899877 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18899877 Country of ref document: EP Kind code of ref document: A1 |