WO2020007182A1 - 个性化场景图像的处理方法、装置及存储介质 - Google Patents

个性化场景图像的处理方法、装置及存储介质 Download PDF

Info

Publication number
WO2020007182A1
WO2020007182A1 PCT/CN2019/091303 CN2019091303W WO2020007182A1 WO 2020007182 A1 WO2020007182 A1 WO 2020007182A1 CN 2019091303 W CN2019091303 W CN 2019091303W WO 2020007182 A1 WO2020007182 A1 WO 2020007182A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal device
touch event
coordinate system
scene image
view
Prior art date
Application number
PCT/CN2019/091303
Other languages
English (en)
French (fr)
Inventor
柯灵杰
崔凌睿
赵桢阳
戴威
陆旭彬
刘小龙
钟琦琦
吴歆婉
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP19829990.1A priority Critical patent/EP3819752A4/en
Publication of WO2020007182A1 publication Critical patent/WO2020007182A1/zh
Priority to US16/994,516 priority patent/US11880999B2/en
Priority to US18/507,178 priority patent/US20240078703A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2021—Shape modification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2024—Style variation

Definitions

  • the present application relates to the field of image processing technologies, and in particular, to a method and device for processing a personalized scene image, an electronic device, and a computer-readable storage medium.
  • Augmented Reality is one of the research hotspots of many well-known foreign universities and research institutions in recent years.
  • AR technology is an emerging technology developed in recent years. It is a technology that calculates the position and angle of a camera image in real time and adds corresponding images, videos, and 3D models.
  • the core is to fuse real-time content with virtual content. Form virtual and real interactions to create completely new experiences.
  • Software apps that can currently capture short videos are mostly based on the capabilities of ARkit (augmented reality tools). They use animation, pets, etc as prototypes for complex 3D modeling, and project virtual models on a plane in real space. , Such as the desktop, the ground, etc., the fusion of the captured real scene and the virtual model projected in space to form a personalized video.
  • ARkit augmented reality tools
  • An embodiment of the present application provides a method for processing a personalized scene image, which is executed by a terminal device, and includes:
  • the model view is superimposed on a scene image collected by the terminal device in the posture to obtain a personalized scene image.
  • An embodiment of the present application provides a device for processing a personalized scene image, where the device includes:
  • a trajectory acquisition module configured to acquire a trajectory of a touch event in the screen area according to a touch event triggered in a screen area of the terminal device
  • a model construction module configured to generate a virtual model according to the projection of the touch event trajectory in a spatial coordinate system
  • a view reconstruction module configured to reconstruct a model view of the virtual model mapped in the field of view of the terminal device according to the position of the terminal device in a spatial coordinate system
  • a scene overlay module is configured to superimpose the model view and a scene image collected by the terminal device in the posture to obtain a personalized scene image.
  • an embodiment of the present application further provides an electronic device.
  • the electronic device includes:
  • Memory for storing processor-executable instructions
  • the processor is configured to execute the processing method of the above-mentioned personalized scene image.
  • an embodiment of the present application further provides a storage medium, where the storage medium stores a computer program, and the computer program can be executed by a processor to complete the processing method of the personalized scene image.
  • FIG. 1 is a schematic diagram of an implementation environment involved in this application
  • Fig. 2 is a block diagram of a computing device according to an exemplary embodiment
  • Fig. 3 is a flow chart showing a method for processing a personalized scene image according to an exemplary embodiment
  • FIG. 4 is an effect diagram of displaying a 3D model by a related technology
  • Fig. 5 is a flowchart illustrating a method for processing a personalized scene image according to an exemplary embodiment
  • FIG. 6 is a schematic diagram showing display effects of three track material styles
  • step 310 is a detailed flowchart of step 310 in the embodiment corresponding to FIG. 3;
  • FIG. 8 is a schematic diagram illustrating an effect of switching a display interface according to an exemplary embodiment of the present application.
  • Fig. 9 is a diagram showing a display effect of performing brush switching according to an exemplary embodiment of the present application.
  • step 330 is a detailed flowchart of step 330 in the embodiment corresponding to FIG. 3;
  • step 350 is a detailed flowchart of step 350 in the embodiment corresponding to FIG. 3;
  • FIG. 12 is a schematic diagram of the architecture of the ARKit framework
  • FIG. 13 is a schematic flowchart of implementing the present application based on the ARkit framework
  • Fig. 14 is a block diagram showing a device for processing a personalized scene image according to an exemplary embodiment
  • FIG. 15 is a detailed block diagram of a trajectory acquisition module in the embodiment corresponding to FIG. 14;
  • FIG. 16 is a detailed block diagram of a model building module in the embodiment corresponding to FIG. 14;
  • FIG. 17 is a detailed block diagram of a view reconstruction module in the embodiment corresponding to FIG. 14.
  • FIG. 1 is a schematic diagram of an implementation environment according to the present application.
  • the implementation environment includes: a terminal device 110 with a camera 111, and a software client 112 is installed in the terminal device 110.
  • the terminal device 110 can adopt the processing method of the personalized scene image provided in this application. , Call the camera to collect the scene image and superimpose a custom virtual model on the scene image.
  • Fig. 2 is a block diagram of a computing device 200 according to an exemplary embodiment.
  • the computing device 200 may be a terminal device 110 in the implementation environment shown in FIG. 1.
  • the terminal device 110 may be a mobile terminal such as a smart phone or a tablet computer; the terminal device 110 may also be a smart home device such as a smart camera.
  • the computing device 200 may include one or more of the following components: a processing component 202, a memory 204, a power component 206, a multimedia component 208, an audio component 210, a sensor component 214, and a communication component 216.
  • the processing component 202 generally controls overall operations of the computing device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations.
  • the processing component 202 may include one or more processors 218 to execute instructions to complete all or part of the steps of the method described below.
  • the processing component 202 may include one or more modules to facilitate the interaction between the processing component 202 and other components.
  • the processing component 202 may include a multimedia module to facilitate the interaction between the multimedia component 208 and the processing component 202.
  • the memory 204 is configured to store various types of data to support operation at the computing device 200. Examples of such data include instructions for any application program or method operating on the computing device 200.
  • the memory 204 may be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (Electrically Erasable Programmable Read-Only Memory (referred to as EEPROM), Erasable Programmable Read-Only Memory (referred to as EPROM), Programmable Read-Only Memory (referred to as PROM), read-only memory (referred to as PROM Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
  • SRAM Static Random Access Memory
  • EEPROM Electrically Erasable Programmable Read Only Memory
  • EPROM Erasable Programmable Read-Only Memory
  • PROM Programmable Read-Only Memory
  • PROM Read-Only Memory ROM
  • magnetic memory flash memory
  • flash memory magnetic disk or optical disk.
  • One or more modules are also stored in the memory 204, and the one or more modules are configured to be executed by the one or more processors 218 to complete any one of FIG. 3, FIG. 7, FIG. 10, and FIG. 11 described below. Shows all or part of the steps in the method.
  • the power component 206 provides power to various components of the computing device 200.
  • the power component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the computing device 200.
  • the multimedia component 208 includes a screen that provides an output interface between the computing device 200 and a user.
  • the screen may include a liquid crystal display (Liquid Crystal Display, LCD for short) and a touch panel. If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user.
  • the touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation.
  • the screen may further include an organic electroluminescent display (Organic Light Emitting Display, OLED for short).
  • the audio component 210 is configured to output and / or input audio signals.
  • the audio component 210 includes a microphone (Microphone, MIC for short).
  • the microphone is configured to receive an external audio signal.
  • the received audio signal may be further stored in the memory 204 or transmitted via the communication component 216.
  • the audio component 210 further includes a speaker for outputting audio signals.
  • the sensor component 214 includes one or more sensors for providing a status assessment of various aspects of the computing device 200.
  • the sensor component 214 can detect the on / off state of the computing device 200, the relative positioning of the components, and the sensor component 214 can also detect a change in the position of the computing device 200 or a component of the computing device 200 and a temperature change of the computing device 200.
  • the sensor component 214 may further include a magnetic sensor, a pressure sensor, or a temperature sensor.
  • the communication component 216 is configured to facilitate wired or wireless communication between the computing device 200 and other devices.
  • the computing device 200 may access a wireless network based on a communication standard, such as WiFi (WIreless-Fidelity).
  • the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel.
  • the communication component 216 further includes a Near Field Communication (NFC) module to facilitate short-range communication.
  • NFC Near Field Communication
  • the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wideband (UWB) technology, Bluetooth technology, and other technologies. .
  • RFID radio frequency identification
  • IrDA infrared data association
  • UWB ultra wideband
  • Bluetooth and other technologies.
  • the computing device 200 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital signal processors, digital signal processing devices, programmable logic devices, and field programmable gate arrays. , Controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described below.
  • ASICs Application Specific Integrated Circuits
  • DSPs digital signal processors
  • digital signal processing devices programmable logic devices
  • field programmable gate arrays field programmable gate arrays.
  • the virtual models in related technologies are pre-configured, and users can only choose among the pre-configured virtual models, so the types of virtual models available for users to choose are limited. If you increase the type of virtual model configured, it will increase the footprint of the software app, and the cost of downloading the software app, which will increase the labor cost of virtual model design.
  • Fig. 3 is a flow chart showing a method for processing a personalized scene image according to an exemplary embodiment.
  • This method for processing a personalized scene image may be executed by the terminal device 110 in the implementation environment shown in FIG. 1, and may also be understood as being executed by a software client 112 running on the terminal device 110, for example, a short video software app. As shown in FIG. 3, the method may include the following steps.
  • step 310 a touch event track in the screen area is acquired according to a touch event triggered in a screen area of the terminal device.
  • the terminal device may be a mobile terminal with an image acquisition function, such as a smart phone or a tablet computer, or may be another intelligent device with an image acquisition function, such as a smart camera.
  • the screen area refers to an area where a pattern can be drawn by touch in a display interface of a terminal device. The user can draw a pattern with a finger or a stylus in the screen area of the terminal device.
  • a touch event refers to an event that occurs when a user touches the screen with a finger or a stylus in the screen area.
  • a touch event track is a track of a finger or a stylus touching in a screen area, and includes a trigger position of a touch event.
  • the software client receives a touch event triggered in the screen area of the terminal device, and forms a touch event track with the continuous change of the trigger position of the touch event.
  • a virtual model is generated according to the projection of the touch event trajectory in a spatial coordinate system.
  • the spatial coordinate system is an absolute coordinate system, which can be a world coordinate system (with a point in space as the origin), and all objects have corresponding position coordinates in the spatial coordinate system, so the terminal device also has a corresponding in the spatial coordinate system.
  • Position coordinates The projection of the touch event trajectory refers to converting the pixel point position (u, v) in the display interface of the terminal device into spatial coordinates according to the conversion rules between the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system. Corresponding positions in the system (x, y, z are preset constants).
  • Generating a virtual model refers to forming a three-dimensional three-dimensional figure at a position point obtained by mapping in a spatial coordinate system.
  • step 350 a model view of the virtual model mapped in the field of view of the terminal device is reconstructed according to the posture of the terminal device in a spatial coordinate system.
  • the terminal device comes with a gyroscope and an acceleration sensor, which can collect the posture change of the terminal device in real time.
  • the pose includes position and attitude data.
  • the position is used to represent the position of the camera in the spatial coordinate system of the terminal device
  • the attitude is used to represent the orientation of the terminal device, that is, the orientation of the camera in the terminal device.
  • the terminal device converts the virtual model in the spatial coordinate system (that is, the world coordinate system) according to its real-time position and orientation in the spatial coordinate system, and according to the conversion rules between the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system.
  • the location point mapping is used to obtain the pixel location in the display interface of the terminal device.
  • the model view can be regarded as a picture obtained by the terminal device in the current posture and acquiring the virtual model through the camera configured by itself, and the area other than the virtual model can be regarded as transparent.
  • step 370 the model view and the scene image collected by the terminal device in the pose are superimposed to obtain a personalized scene image.
  • the scene image refers to a real-world image collected by a camera configured by the terminal device itself.
  • the overlay method can use the scene image as the background and overlay the model view on the scene image.
  • the virtual model can be regarded as an object fixed in the real world.
  • the construction is captured at the same time.
  • Virtual model. Personalized scene image refers to the superimposed model view on the real scene image.
  • a touch event trajectory is projected into a spatial coordinate system to construct a virtual model according to the triggered touch event trajectory, and then, in the process of moving the terminal device, according to the real-time posture of the terminal device , The model view captured in this pose and the scene image can be superimposed to form a personalized scene image. Therefore, the virtual model of the present application is no longer configured in advance, and a custom virtual model can be formed according to user-drawn graphics, thereby saving the traffic of downloading the virtual model and the space for storing the virtual model, and improving the terminal device's The operating efficiency reduces the labor cost of designing virtual models and enriches the types of virtual models.
  • the method for processing a personalized scene image further includes the following steps:
  • step 301 multiple trajectory material styles are displayed according to the received trajectory drawing request
  • the software client displays a virtual model drawing button on the display interface of the terminal device, receives a trajectory drawing request triggered by the user to trigger the virtual model drawing button, and then responds to the virtual model drawing request triggered by the user to obtain multiple trajectory material styles to be selected To display the icon corresponding to each track material style.
  • the track material style is used to indicate the brush style of the virtual model drawn by the user through touch.
  • a target trajectory style is obtained according to the selection of multiple trajectory material style triggers; the virtual model is configured according to the target trajectory style.
  • the user can trigger the selection of one of the multiple track material styles displayed.
  • the target track style refers to a style selected by the user among various track material styles.
  • the software client receives the trigger instruction of the user to click one of the icons after displaying the icons of various track material styles.
  • the software client obtains the target trajectory style selected by the user according to the user's selection of multiple trajectory material style triggers instructed by the trigger instruction.
  • subsequent virtual models can be generated according to the brush style configured by the target trajectory style.
  • Figure 6 is a schematic diagram of the effects of three track material styles.
  • the three track material styles shown in FIG. 6 are the track material style 601 (shown in the left part of Figure 6) and "smoke".
  • Track material style 602 (as shown in the middle part of Figure 6)
  • Colorful brush “track material style 603 (as shown in the right part of Figure 6)
  • the user can choose any track material style, and receive the user After the selection is made, the selected track material style is used as the target track style.
  • the selected track material style is used as the target track style. For example, when the user selects a track material style 602 of “smoke”, a brush pattern configured with a smoke style is used to create a track pattern, as shown in the middle part in FIG. 6.
  • step 310 specifically includes:
  • step 311 the image acquisition device of the terminal device is started to collect the current scene image
  • the image acquisition device refers to a camera configured by the terminal device itself.
  • the current scene image refers to an image of a real scene collected by the camera of the terminal device at the current moment.
  • the software client After receiving the trigger instruction of the user to select the target track style, the software client calls the camera of the terminal device to collect an image of the real scene.
  • step 312 a touch event triggered in the current scene image display interface is received, and a trigger position of the touch event is recorded, and the trigger position of the touch event constitutes the touch event track.
  • the software client displays the current scene image on the display interface of the terminal device, receives the touch event triggered in the current scene image display interface, and records the trigger position of the touch event.
  • the connection of one or more trigger positions constitutes the touch. Control event track.
  • Fig. 8 is a schematic diagram showing an effect of switching a display interface according to an exemplary embodiment of the present application.
  • the terminal device opens a software client, and the software client receives an instruction of the user to click a “shoot” button, and enters a front camera shooting interface 810.
  • the software client receives the instruction of the user to click the "magic” button 801, and displays the operation panel 802 corresponding to the "magic” button on the display interface.
  • the software client receives the instruction of the user to click the "magic brush” button 803 in the operation panel 802, and enters the magic brush selection interface 820.
  • the magic brush selection interface 820 displays a variety of track material styles (that is, the types of brushes), such as "money more ",” Smoke “,” colorful brushes “and other types of brushes, users can choose any brush type, and show users the” drawing in three-dimensional space “prompts to guide users to create.
  • the software client can receive the touch event triggered on the screen to obtain the touch event track, and the touch event track is projected in space to generate a three-dimensional graphic.
  • the stereo graphics are fixed based on space. The user adjusts the position and angle of the camera of the terminal device, so as to display the pictures of the stereo graphic at different angles on the terminal device display interface (see interfaces 830 and 840).
  • the user can draw freely in conjunction with the captured scene images in the space.
  • interface 910 uses the “money-to-more” track material style (that is, the type of brush).
  • a prompt box 923 pops up to prompt the user to clear the track pattern 911 created by the “better money” brush type 921.
  • the above step 330 specifically includes:
  • step 331 according to the position of the terminal device in the spatial coordinate system, the touch event trajectory is projected onto a plane, the plane is located in the direction of the field of view of the terminal device, and is at The distance is a preset distance, and the direction of the field of view of the terminal device is the acquisition direction of the camera in the terminal device;
  • the touch event occurs when the user's finger or stylus touches the screen.
  • the field of view direction of the terminal device refers to the direction in which the camera configured by the terminal device performs image acquisition.
  • the pose includes position and attitude data, and the position is used to represent the position of the camera in the spatial coordinate system in the terminal device.
  • the attitude is used to characterize the acquisition direction of the camera in the terminal device.
  • the software client changes the position coordinates of the pixel point where the touch event occurred in the display interface from the pixel coordinates according to the pose (including the position and posture) of the camera in the spatial coordinate system in the terminal device.
  • the system is converted into a position point in the world coordinate system (the corresponding point in the pixel coordinate system to find a corresponding point in the world coordinate system belongs to the prior art, and is not repeated here).
  • a coordinate point in a world coordinate system can find a corresponding pixel point in the image, but the other way around If a pixel point in the image is used to find its corresponding point in the world coordinate system, a depth value is missing. Therefore, in this application, by setting a depth value (ie, a preset distance) in advance, the touch event trajectory is projected into a plane in the spatial coordinate system at a preset distance from the current position, and according to the attitude of the camera, the plane needs to be located Collection direction of the camera of the terminal device. Therefore, according to the coordinate position of the pixel point of the touch event track in the pixel coordinate system, the corresponding position point in the world coordinate system can be found.
  • a depth value ie, a preset distance
  • step 332 a virtual model is generated at the projected position of the touch event track according to the configured target track style.
  • the plane refers to a plane in the spatial coordinate system that is preset at a distance from the camera and in the camera image acquisition direction.
  • the software client projects the acquired touch event trajectory onto the above-mentioned plane in the spatial coordinate system, and obtains the above-mentioned plane in the spatial coordinate system by the conversion rule between the pixel coordinate system and the world coordinate system.
  • the touch event track includes a plurality of pixel point position coordinates, and thus the projection position includes a plurality of position points in a space corresponding to the pixel point position coordinates of the touch event track.
  • the target track style is one of the styles selected by the user from a variety of track material styles, that is, the type of brush.
  • the software client forms the touch event trajectory at the projection position of the spatial coordinate system, and forms a three-dimensional graphic indicated by the target trajectory pattern according to the target trajectory style selected by the user.
  • the three-dimensional graphic generated at the projection position can be considered as a virtual model.
  • the virtual model can be regarded as fixed in space. When the terminal device adjusts the pose, it can capture images of the virtual model at different angles.
  • the above step 350 specifically includes:
  • step 351 according to the position of the terminal device in the spatial coordinate system and the acquisition direction of the camera in the terminal device, locate the field of view range of the terminal device in the spatial coordinate system;
  • the gyroscope and acceleration sensor configured by the terminal device can collect the position of the terminal device in the space coordinate system in real time, and then obtain the camera configured by the terminal device in the space coordinate system.
  • Position and posture The field of view of the positioning terminal device in the spatial coordinate system refers to re-establishing the camera coordinate system according to the position of the camera and the shooting direction.
  • the origin of the camera coordinate system is the optical center of the camera, the x-axis, y-axis, and X of the image,
  • the Y axis is parallel
  • the z axis is the optical axis of the camera, which is perpendicular to the image plane (ie, the screen display interface).
  • the intersection of the optical axis and the image plane is the origin of the image coordinate system, and the rectangular coordinate system formed is the camera coordinate system.
  • the origin of the pixel coordinate system is the upper left corner of the image (screen display interface).
  • step 352 the virtual model within the field of view is mapped to the display area of the terminal device to form a model view.
  • the spatial position points of the virtual model constructed in the world coordinate system are converted into the pixel point positions in the pixel coordinate system. coordinate. Since the display area of the terminal device is limited, the pixel values corresponding to the corresponding position points of the virtual model are filled under the corresponding pixel point position coordinates of the terminal device display area according to the converted pixel point position coordinates, and thus obtained in the terminal device display area.
  • the pattern mapped by the virtual model is the model view.
  • the foregoing step 370 specifically includes:
  • the scene image collected by the terminal device in the pose is used as a background, and the model view is overlaid on the scene image to obtain a personalized scene image.
  • the software client uses the scene image as a background, and overlays the pattern mapped by the virtual model on the scene image. Get personalized scene images.
  • the foregoing step 370 specifically includes:
  • the model view is superimposed frame by frame on the image sequence collected by the terminal device in the pose to obtain a personalized scene video.
  • the video recording function can be started to collect image sequences. While performing image sequence acquisition, for the posture of the terminal device when acquiring each frame of image, the model view in this posture is superimposed on the acquired image, and the current position is superimposed on the acquired scene image frame by frame The model view under the pose, so that you can get personalized scene video.
  • the method for processing a personalized scene image provided in this application may be implemented based on an ARKit framework.
  • ARKit provides two types of virtual augmented reality views, including scene images with 3D effects and scene images with 2D effects.
  • Figure 12 is a schematic diagram of the architecture of the ARKit framework.
  • UIView user interface view
  • ARSCNView augmented reality view
  • ARSCNView just a view container, its role is to manage an ARSession (augmented reality session) 1203.
  • ARKit framework is only responsible for transforming the real-world picture into a 3D scene.
  • This transformation process is mainly divided into two links: ARCamera (augmented reality camera) 1204 is responsible for capturing the camera picture as the background View, ARSession is responsible for building the 3D scene, and finally ARSCNView displays the 3D scene.
  • ARCamera augmented reality camera
  • ARSession serves as a communication bridge between ARCamera and ARSCNView.
  • the main participants in building a communication bridge are two ARSessionConfiguration (session tracking configuration) 1205 and ARFrame (augmented reality architecture) 1206.
  • the main purpose of ARSessionConfiguration is to track the position and orientation of the camera in the 3D world and capture some feature scenes (such as plane capture), that is, to detect the surface of the real world seen by the device camera. It mainly detects the movement, rotation and even rolling of the terminal equipment by calling the sensors equipped on the terminal equipment.
  • ARFrame contains relevant information such as the pictures captured by the camera and the camera position.
  • the node (SCNNode) 1207 obtains a view of the virtual model drawn by the user
  • the scene class (SCNScene class) 1208 superimposes this view with the image taken by the camera to form a personalized scene image.
  • FIG. 13 is a schematic flowchart of implementing the present application based on the ARkit framework.
  • the software client receives a “magic brush” label triggered by the user, and displays a panel corresponding to the “magic brush” label. This panel shows a variety of brush styles.
  • step 1302 the user may select one of a plurality of brush styles displayed, and the software client receives the user's request to select a specified brush style in the panel.
  • step 1303 the software client responds to the request to run ARSCNView to load the scene SCNScene.
  • step 1304 SCNScene starts the camera ARCamera to start capturing scene images.
  • step 1305 the software client runs ARSCNView and passes the captured scene data to ARsession.
  • step 1306 the software client runs ARsession.
  • ARsession tracks the pose of the camera in real time by managing ARsessionConfiguration, and returns ARFame (including scene image, pose, and other information).
  • a touch event trace drawn by a user is acquired and projected in space to form a virtual model.
  • ARsessionConfiguration tracks the pose of the camera in real time in order to determine the real position of the virtual model relative to the camera in space.
  • step 1308 the software client runs SCNNode to capture a view of the virtual model in the current pose.
  • step 1309 the software client runs SCNScene to superimpose the view with the image captured by the camera in the current posture to form a personalized scene image, and displays the personalized scene image on the terminal device through UIView.
  • a pop-up window is displayed to remind the user whether the current effect is clear.
  • the following is a device embodiment of the present application, which can be used to execute an embodiment of a method for processing a personalized scene image performed by the software client 110 described above.
  • a device embodiment of the present application can be used to execute an embodiment of a method for processing a personalized scene image performed by the software client 110 described above.
  • the embodiment of a method for processing a personalized scene image in the present application please refer to the embodiment of a method for processing a personalized scene image in the present application.
  • Fig. 14 is a block diagram of a device for processing a personalized scene image according to an exemplary embodiment.
  • the device for processing a personalized scene image may be used in the terminal device 110 of the implementation environment shown in Fig. 1 to execute Fig. 3, All or part of the steps of the method for processing a personalized scene image shown in any one of FIG. 7, FIG. 10, and FIG.
  • the device includes, but is not limited to, a trajectory acquisition module 1410, a model construction module 1430, a view reconstruction module 1450, and a scene overlay module 1470.
  • the trajectory acquisition module 1410 is configured to acquire a trajectory of a touch event in the screen area according to a touch event triggered in a screen area of the terminal device;
  • a model building module 1430 configured to generate a virtual model according to the projection of the touch event trajectory in a spatial coordinate system
  • a view reconstruction module 1450 configured to reconstruct a model view of the virtual model mapped in the field of view of the terminal device according to the posture of the terminal device in a spatial coordinate system;
  • a scene superimposing module 1470 is configured to superimpose the model view and a scene image collected by the terminal device in the posture to obtain a personalized scene image.
  • the trajectory acquisition module 1410 may be, for example, a certain physical structure sensor component 214 in FIG. 2.
  • the model construction module 1430, the view reconstruction module 1450, and the scene overlay module 1470 may also be functional modules for performing corresponding steps in the processing method of the personalized scene image described above. It can be understood that these modules can be implemented by hardware, software, or a combination of both. When implemented in hardware, these modules may be implemented as one or more hardware modules, such as one or more application specific integrated circuits. When implemented in software, these modules may be implemented as one or more computer programs executing on one or more processors, such as programs stored in the memory 204 executed by the processor 218 of FIG. 2.
  • the apparatus for processing a personalized scene image further includes:
  • Style acquisition module used to display a variety of track material styles according to the received track drawing request
  • a style selection module is configured to obtain the selected target trajectory style according to the selection of the multiple trajectory material style triggers; the virtual model is configured according to the target trajectory style.
  • the trajectory acquisition module 1410 includes:
  • An image acquisition unit 1411 configured to start an image acquisition device of the terminal device, and acquire an image of a current scene
  • the event receiving unit 1412 is configured to receive a touch event triggered in the current scene image display interface, record a trigger position of the touch event, and the trigger position of the touch event constitutes the touch event track.
  • the model building module 1430 includes:
  • a trajectory projection unit 1431 is configured to project the trajectory of the touch event onto a plane according to a posture of the terminal device in a spatial coordinate system, where the plane is located in a direction of a field of view of the terminal device, and The distance of the current position of the terminal device is a preset distance, the pose includes the position of the terminal device in the spatial coordinate system and the acquisition direction of the camera in the terminal device, and the field of view direction is the Collection direction of the camera in the terminal device;
  • a model generating unit 1432 is configured to generate the virtual model at the projection position of the touch event track according to the configured target track style.
  • the view reconstruction module 1450 includes:
  • a field of view positioning unit 1451 is configured to locate a field of view range of the terminal device in the spatial coordinate system according to a position of the terminal device in a spatial coordinate system and a collection direction of a camera in the terminal device;
  • a model mapping unit 1452 is configured to map a virtual model within the field of view to a display area of the terminal device to form a model view.
  • the scene overlay module 1470 includes:
  • a view overlay unit is configured to use a scene image collected by the terminal device in the pose as a background, and overlay the model view on the scene image to obtain the personalized scene image.
  • the scene overlay module 1470 includes:
  • the model view is superimposed frame by frame on an image sequence collected by the terminal device in the pose to obtain a personalized scene video.
  • the present application further provides an electronic device, which can be used in the terminal device 110 of the implementation environment shown in FIG. 1 to execute any one of FIG. 3, FIG. 7, FIG. All or part of the steps of a method for personalizing a scene image.
  • the electronic device includes:
  • Memory for storing processor-executable instructions
  • the processor is configured to execute the method for processing a personalized scene image according to the foregoing exemplary embodiment.
  • a storage medium is also provided, and the storage medium is a computer-readable storage medium, and may be, for example, temporary and non-transitory computer-readable storage media including instructions.
  • the storage medium includes, for example, a memory 204 of instructions, and the storage medium stores a computer program that can be executed by the processor 218 of the computing device 200 to complete the processing method of the personalized scene image described above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Human Computer Interaction (AREA)
  • Architecture (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本申请揭示了一种个性化场景图像的处理方法及装置、电子设备、计算机可读存储介质，所述方法由终端设备执行，包括：根据在终端设备的屏幕区域内触发的触控事件，获取屏幕区域内的触控事件轨迹；根据触控事件轨迹在空间坐标系中的投射，生成虚拟模型；按照终端设备在空间坐标系中的位姿，重构虚拟模型在终端设备视场范围内映射的模型视图；将模型视图与终端设备在位姿下采集的场景图像进行叠加，获得个性化场景图像。本申请

Description

个性化场景图像的处理方法、装置及存储介质
本申请要求于2018年7月3日提交中国专利局、申请号为201810717653.X、名称为“个性化场景图像的处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，特别涉及一种个性化场景图像的处理方法及装置、电子设备、计算机可读存储介质。
背景
增强现实(Augmented Reality，简称AR)是近年来国外众多知名大学和研究机构的研究热点之一。AR技术是近年来发展起来的新兴技术，是一种实时计算摄影机影像的位置以及角度并加上相应图像、视频、3D模型的技术，其核心是将虚拟内容和真实存在的内容进行实时融合，形成虚拟、现实之间的互动，从而创造出全新的体验。
目前可以采集短视频的软件APP(应用程序)，多数基于ARkit(增强现实工具)的能力，以动漫、宠物等为原型进行复杂的3D建模，将虚拟模型投射在真实空间的某一平面上，如桌面、地面等，将拍摄的现实场景与空间中投射的虚拟模型进行融合，形成个性化视频。
技术内容
本申请实施例提供了一种个性化场景图像的处理方法，由终端设备执行，包括：
根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹；
根据所述触控事件轨迹在空间坐标系中的投射，生成虚拟模型；
按照所述终端设备在空间坐标系中的位姿，重构所述虚拟模型在所述终端设 备视场范围内映射的模型视图；
将所述模型视图与所述终端设备在所述位姿下采集的场景图像进行叠加，获得个性化场景图像。
本申请实施例提供了一种个性化场景图像的处理装置，所述装置包括：
轨迹获取模块，用于根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹；
模型构建模块，用于根据所述触控事件轨迹在空间坐标系中的投射，生成虚拟模型；
视图重构模块，用于按照所述终端设备在空间坐标系中的位姿，重构所述虚拟模型在所述终端设备视场范围内映射的模型视图；
场景叠加模块，用于将所述模型视图与所述终端设备在所述位姿下采集的场景图像进行叠加，获得个性化场景图像。
进一步的，本申请实施例还提供了一种电子设备，所述电子设备包括：
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器被配置执行上述个性化场景图像的处理方法。
进一步的，本申请实施例还提供了一种存储介质，所述存储介质存储有计算机程序，所述计算机程序可由处理器执行完成上述个性化场景图像的处理方法。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。
图1是根据本申请所涉及的实施环境的示意图；
图2是根据一示例性实施例示出的一种计算设备的框图；
图3是根据一示例性实施例示出的一种个性化场景图像的处理方法的流程图；
图4是相关技术对3D模型进行显示的效果图；
图5是一示例性实施例示出的一种个性化场景图像的处理方法的流程图；
图6是三种轨迹素材样式的显示效果示意图；
图7是图3对应实施例中步骤310的细节流程图；
图8是本申请示例性实施例示出的显示界面切换的效果示意图；
图9是本申请示例性实施例示出的进行画笔切换的显示效果图。
图10是图3对应实施例中步骤330的细节流程图；
图11是图3对应实施例中步骤350的细节流程图；
图12是ARKit框架的架构示意图；
图13是基于ARkit框架实现本申请的流程示意图；
图14是根据一示例性实施例示出的一种个性化场景图像的处理装置的框图；
图15是图14对应实施例中轨迹获取模块的细节框图；
图16是图14对应实施例中模型构建模块的细节框图；
图17是图14对应实施例中视图重构模块的细节框图。
实施方式
这里将详细地对示例性实施例执行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本申请相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本申请的一些方面相一致的装置和方法的例子。
图1是根据本申请所涉及的实施环境的示意图。该实施环境包括：带有摄像头111的终端设备110，终端设备110中安装有软件客户端112，终端设备110通过运行该软件客户端112，从而可以采用本申请提供的个性化场景图像的处理方法，调用摄像头采集场景图像并在场景图像上叠加自定义的虚拟模型。
图2是根据一示例性实施例示出的一种计算设备200的框图。例如，计算设备200可以是图1所示实施环境中的终端设备110。终端设备110可以是移动终端如智能手机、平板电脑等；终端设备110还可以是智能家居设备，如智能摄像头等。
参照图2，计算设备200可以包括以下一个或多个组件：处理组件202，存储器204，电源组件206，多媒体组件208，音频组件210，传感器组件214以及 通信组件216。
处理组件202通常控制计算设备200的整体操作，诸如与显示，电话呼叫，数据通信，相机操作以及记录操作相关联的操作等。处理组件202可以包括一个或多个处理器218来执行指令，以完成下述的方法的全部或部分步骤。此外，处理组件202可以包括一个或多个模块，便于处理组件202和其他组件之间的交互。例如，处理组件202可以包括多媒体模块，以方便多媒体组件208和处理组件202之间的交互。
存储器204被配置为存储各种类型的数据以支持在计算设备200的操作。这些数据的示例包括用于在计算设备200上操作的任何应用程序或方法的指令。存储器204可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(Static Random Access Memory，简称SRAM)，电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，简称EEPROM)，可擦除可编程只读存储器(Erasable Programmable Read Only Memory，简称EPROM)，可编程只读存储器(Programmable Red-Only Memory，简称PROM)，只读存储器(Read-Only Memory，简称ROM)，磁存储器，快闪存储器，磁盘或光盘。存储器204中还存储有一个或多个模块，该一个或多个模块被配置成由该一个或多个处理器218执行，以完成下述图3、图7、图10、图11任一所示方法中的全部或者部分步骤。
电源组件206为计算设备200的各种组件提供电力。电源组件206可以包括电源管理系统，一个或多个电源，及其他与为计算设备200生成、管理和分配电力相关联的组件。
多媒体组件208包括在所述计算设备200和用户之间的提供一个输出接口的屏幕。在一些实施例中，屏幕可以包括液晶显示器(Liquid Crystal Display，简称LCD)和触摸面板。如果屏幕包括触摸面板，屏幕可以被实现为触摸屏，以接收来自用户的输入信号。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。所述触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与所述触摸或滑动操作相关的持续时间和压力。屏幕还可以包括有机电致发光显示器(Organic Light Emitting Display，简称OLED)。
音频组件210被配置为输出和/或输入音频信号。例如，音频组件210包括 一个麦克风(Microphone，简称MIC)，当计算设备200处于操作模式，如呼叫模式、记录模式和语音识别模式时，麦克风被配置为接收外部音频信号。所接收的音频信号可以被进一步存储在存储器204或经由通信组件216发送。在一些实施例中，音频组件210还包括一个扬声器，用于输出音频信号。
传感器组件214包括一个或多个传感器，用于为计算设备200提供各个方面的状态评估。例如，传感器组件214可以检测到计算设备200的打开/关闭状态，组件的相对定位，传感器组件214还可以检测计算设备200或计算设备200一个组件的位置改变以及计算设备200的温度变化。在一些实施例中，该传感器组件214还可以包括磁传感器，压力传感器或温度传感器。
通信组件216被配置为便于计算设备200和其他设备之间有线或无线方式的通信。计算设备200可以接入基于通信标准的无线网络，如WiFi(WIreless-Fidelity，无线保真)。在一个示例性实施例中，通信组件216经由广播信道接收来自外部广播管理系统的广播信号或广播相关信息。在一个示例性实施例中，所述通信组件216还包括近场通信(Near Field Communication，简称NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(Radio Frequency Identification，简称RFID)技术，红外数据协会(Infrared Data Association，简称IrDA)技术，超宽带(Ultra Wideband，简称UWB)技术，蓝牙技术和其他技术来实现。
在示例性实施例中，计算设备200可以被一个或多个应用专用集成电路(Application Specific Integrated Circuit，简称ASIC)、数字信号处理器、数字信号处理设备、可编程逻辑器件、现场可编程门阵列、控制器、微控制器、微处理器或其他电子元件实现，用于执行下述方法。
通常情况下，相关技术中的虚拟模型均是预先配置好的，用户只能在预先配置的虚拟模型中进行选择，由此可供用户选择的虚拟模型种类有限。如果增加配置的虚拟模型种类，则会增大软件APP的占用空间，以及下载软件APP的流量成本，更加增加了虚拟模型设计的人力成本。
本申请实施例提供了一种个性化场景图像的处理方法。图3是根据一示例性实施例示出的一种个性化场景图像的处理方法的流程图。该种个性化场景图像的处理方法可以由图1所示实施环境中的终端设备110执行，也可以理解为是由终端设备110所运行的软件客户端112执行，例如，短视频类软件APP。如图3所 示，该方法可以包括以下步骤。
在步骤310中，根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹。
其中，终端设备可以是具有图像采集功能的移动终端，如智能手机或平板电脑，还可以是其他具有图像采集功能的智能设备，如智能摄像头。屏幕区域是指在终端设备的显示界面中，可以通过触控绘制图案的区域。用户可以在终端设备的屏幕区域内通过手指或触控笔绘制图案。触控事件是指用户在屏幕区域内通过手指或触控笔接触屏幕产生的事件。触控事件轨迹是手指或触控笔在屏幕区域内触控的轨迹，包含触控事件的触发位置。软件客户端接收终端设备的屏幕区域内触发的触控事件，随着触控事件触发位置的连续变化形成触控事件轨迹。
在步骤330中，根据所述触控事件轨迹在空间坐标系中的投射，生成虚拟模型。
其中，空间坐标系是一种绝对坐标系，可以是世界坐标系(以空间中一点为原点)，所有物体在空间坐标系中具有相应的位置坐标，由此终端设备在空间坐标系中也有对应的位置坐标。其中，触控事件轨迹的投射是指依据世界坐标系、相机坐标系、图像坐标系以及像素坐标系之间的转换规则，将终端设备显示界面中像素点位置(u，v)转换成空间坐标系中对应位置(x，y，z为预设常数)。即将显示界面中屏幕区域内触控事件触发的像素点位置映射得到空间坐标系中的位置点，由此触控事件轨迹可以在空间坐标系中映射得到一个或多个位置点。生成虚拟模型是指在空间坐标系中映射得到的位置点处形成三维立体图形。
在步骤350中，按照所述终端设备在空间坐标系中的位姿，重构所述虚拟模型在所述终端设备视场范围内映射的模型视图。
需要说明的是，终端设备自带陀螺仪、加速度传感器，可以实时采集终端设备的位姿变化。位姿包括位置和姿态数据，位置用于表征终端设备中摄像头在空间坐标系中的位置，姿态用于表征终端设备的朝向，也即用于表征终端设备中摄像头的采集方向。终端设备根据自身在空间坐标系中实时的位置和朝向，依据世界坐标系、相机坐标系、图像坐标系以及像素坐标系之间的转换规则，将空间坐标系(即世界坐标系)中虚拟模型的位置点映射得到在终端设备显示界面中的像素点位置。将虚拟模型在终端设备映射的像素点位置填充对应像素值，得到模型 视图。需要解释的是，模型视图可以看成是终端设备在当前位姿下，通过自身配置的摄像头对虚拟模型进行图像采集所获得的画面，其中除虚拟模型以外的区域可以看成是透明的。
在步骤370中，将模型视图与终端设备在位姿下采集的场景图像进行叠加，获得个性化场景图像。
其中，场景图像是指终端设备自身配置的摄像头采集的真实世界的图像。终端设备在移动过程中，将当前位姿下重构的模型视图与当前位姿下摄像头采集的场景图像进行叠加。叠加方式可以是以场景图像为背景，将模型视图覆盖在场景图像之上，由此虚拟模型可以看成是固定在现实世界中的一个物体，在对现实世界进行图像采集时，同时捕捉到了构建的虚拟模型。个性化场景图像是指在真实采集的场景图像上叠加了模型视图。
需要说明的是，相关技术中，虚拟模型是事先配置好存储在终端设备中的，下载和存储大量的虚拟模型，占用较大的空间和下载流量；并且大量虚拟模型的构建需要较多的人力成本；进一步的，可供用户选择的虚拟模型种类有限，由此不能满足所有用户的需求。如图4所示，相关技术以动漫、宠物为原型进行3D建模，一个复杂物体的3D模型构建需要大量的人力物力，且模型较为单一，在用户使用过程中，互动性较差。
本申请上述示例性实施例提供的技术方案，根据触发的触控事件轨迹，将触控事件轨迹投射到空间坐标系中构建虚拟模型，进而在终端设备移动过程中，按照终端设备实时的位姿，可以将该位姿下捕捉的模型视图和场景图像进行叠加，形成个性化场景图像。由此，本申请虚拟模型不再是预先配置好的，可以根据用户自定义绘制的图形，形成自定义虚拟模型，从而节省了下载虚拟模型的流量以及存储虚拟模型的空间，提高了终端设备的运行效率，降低了设计虚拟模型的人力成本，丰富了虚拟模型的种类。
在一种示例性实施例中，如图5所示，在上述步骤310之前，本申请提供的个性化场景图像的处理方法还包括以下步骤：
在步骤301中，根据接收到的轨迹绘制请求，展示多种轨迹素材样式；
具体的，软件客户端在终端设备显示界面中展示虚拟模型绘制按钮，接收用 户触发虚拟模型绘制按钮的轨迹绘制请求，进而响应用户触发进行的虚拟模型绘制请求，获取待选择的多种轨迹素材样式，将每种轨迹素材样式所对应的图标进行展示。其中，轨迹素材样式用于指示用户通过触控所绘制虚拟模型的画笔样式。
在步骤302中，根据对多种轨迹素材样式触发进行的选择，获得目标轨迹样式；虚拟模型按照目标轨迹样式进行配置。
用户可以从展示的多种轨迹素材样式中触发选择其中一种样式。其中，目标轨迹样式是指多种轨迹素材样式中用户选择的样式。软件客户端在展示多种轨迹素材样式的图标后，接收用户点击其中一张图标的触发指令。软件客户端根据触发指令指示的用户对多种轨迹素材样式触发进行的选择，获得用户所选择的目标轨迹样式。进而后续虚拟模型可以按照目标轨迹样式所配置的画笔样式生成。图6为三种轨迹素材样式的效果示意图，图6所示的三种轨迹素材样式，分别为“钱多多”的轨迹素材样式601(如图6的左边部分所示)、“smoke(烟雾)”的轨迹素材样式602(如图6的中间部分所示)、“炫彩画笔”的轨迹素材样式603(如图6的右边部分所示)，用户可以选择任一轨迹素材样式，接收到用户的选择以后，将选择的轨迹素材样式作为目标轨迹样式。例如，当用户选择“smoke(烟雾)”的轨迹素材样式602时，采用烟雾的样式所配置的画笔创作轨迹图案，如图6中的中间部分所示。
在一种示例性实施例中，如图7所示，上述步骤310具体包括：
在步骤311中，启动终端设备的图像采集装置，采集当前场景图像；
具体的，图像采集装置是指终端设备自身配置的相机。当前场景图像是指当前时刻终端设备的相机采集的真实场景的图像。软件客户端在接收到用户触发选择目标轨迹样式的触发指令后，调用终端设备的相机采集真实场景的图像。
在步骤312中，接收所述当前场景图像显示界面内触发的触控事件，记录所述触控事件的触发位置，所述触控事件的触发位置构成所述触控事件轨迹。
软件客户端在终端设备的显示界面中进行当前场景图像的显示，接收当前场景图像显示界面内触发的触控事件，并记录触控事件的触发位置，一个或多个触发位置的连线构成触控事件轨迹。
图8是本申请示例性实施例示出的显示界面切换的效果示意图。
如图8所示，终端设备打开软件客户端，软件客户端接收到用户点击“拍摄”按钮的指令，进入前置摄像头拍摄界面810。
软件客户端接收到用户点击“魔法”按钮801的指令，在显示界面展示“魔法”按钮对应的操作面板802。软件客户端接收到用户点击操作面板802中“神奇画笔”按钮803的指令，进入神奇画笔选择界面820，神奇画笔选择界面820中展示了多种轨迹素材样式(即画笔种类)，例如“钱多多”、“smoke”、“炫彩画笔”等多种画笔种类，用户可以选择任意画笔种类，并向用户展示“在立体空间中画画”的提示，引导用户创作。
软件客户端接收到用户选择任意一种画笔种类的指令后，可以接收屏幕上触发的触控事件，获得触控事件轨迹，触控事件轨迹在空间中投射生成立体图形。该立体图形基于空间固定。用户调整终端设备摄像头的位置和角度，从而在终端设备显示界面中展示立体图形在不同角度下的画面(参见界面830和840)。
在一种实施例中，如图9所示，用户可以在空间中配合采集的场景图像自由绘制，参见界面910，可以看出界面910中采用“钱多多”的轨迹素材样式(即画笔种类)绘制轨迹图案911。由于多种画笔不可同时使用，当用户切换画笔时，可以提示用户清空当前创作的轨迹图案，参见界面920，当用户由“钱多多”的画笔种类921切换到“炫彩画笔”的画笔种类922时，弹出提示框923来提示用户清空由“钱多多”的画笔种类921创作的轨迹图案911。
在上述示例性实施例的基础上，如图10所示，上述步骤330具体包括：
在步骤331中，根据所述终端设备在空间坐标系中的位姿，将所述触控事件轨迹投射到一个平面上，所述平面位于所述终端设备视场方向上，且与当前位置的距离为预设距离，所述终端设备视场方向为所述终端设备中摄像头的采集方向；
其中，触控事件发生时是指用户手指或触控笔接触屏幕时。终端设备的视场方向是指终端设备自身配置的摄像头进行图像采集的方向。位姿包括位置和姿态数据，位置用于表征终端设备中摄像头在空间坐标系中的位置。姿态用于表征终端设备中摄像头的采集方向。
具体的，在触控事件发生时，软件客户端根据终端设备中摄像头在空间坐标 系中的位姿(包括位置和姿态)，将显示界面中触控事件发生的像素点位置坐标，从像素坐标系转换成世界坐标系下的位置点(像素坐标系中指定点找到在世界坐标系中的对应点属于现有技术，在此不再赘述)。需要解释的是，根据世界坐标系、相机坐标系、图像坐标系和像素坐标系之间的转换规则，一个世界坐标系中的坐标点，可以在图像中找到一个对应的像素点，但是反过来，通过图像中的一个像素点找到其在世界坐标系中对应的点，则缺少一个深度值。由此，本申请通过预先设置一个深度值(即预设距离)，将触控事件轨迹投射到空间坐标系中距离当前位置预设距离的一个平面内，并且根据摄像头的姿态，该平面需位于终端设备摄像头的采集方向。由此，根据像素坐标系中触控事件轨迹的像素点坐标位置，可以找到对应的世界坐标系下的位置点。
在步骤332中，按照配置的目标轨迹样式，将触控事件轨迹在平面的投射位置处生成虚拟模型。
具体的，该平面是指在空间坐标系中，距离摄像头预设距离且在摄像头图像采集方向的一个平面。软件客户端将获取的触控事件轨迹投射到空间坐标系中的上述平面，通过像素坐标系与世界坐标系之间的转换规则，得到像素坐标系中触控事件轨迹在空间坐标系中上述平面内的投射位置点。在一种实施例中，触控事件轨迹包括多个像素点位置坐标，由此投射位置包括与触控事件轨迹的像素点位置坐标对应的空间中多个位置点。
目标轨迹样式是用户从多种轨迹素材样式中选择的其中一个样式，即画笔种类。软件客户端将触控事件轨迹在空间坐标系的投射位置处，按照用户选择的目标轨迹样式，形成目标轨迹样式指示的立体图形，在投射位置处生成的立体图形可以认为是虚拟模型。虚拟模型可以看成在空间中固定，当终端设备调整位姿时，可以捕捉到虚拟模型在不同角度下的图像。
在一种示例性实施例中，如图11所示，上述步骤350具体包括：
在步骤351中，根据终端设备在空间坐标系中的位置以及所述终端设备中摄像头的采集方向，定位终端设备在空间坐标系中的视场范围；
需要说明的是，随着终端设备的移动，终端设备自身配置的陀螺仪和加速度传感器可以实时采集终端设备在空间坐标系中的位姿，进而得到终端设备自身配 置的摄像头在空间坐标系中的位置和姿态。定位终端设备在空间坐标系中的视场范围是指根据摄像头所处位置和拍摄方向，重新建立相机坐标系，相机坐标系的原点为相机的光心，x轴、y轴与图像的X,Y轴平行，z轴为相机光轴，它与图像平面(即屏幕显示界面)垂直。光轴与图像平面的交点，即为图像坐标系的原点，构成的直角坐标系为相机坐标系。像素坐标系的原点为图像(屏幕显示界面)的左上角。
在步骤352中，将处于视场范围内的虚拟模型映射到终端设备的显示区域，形成模型视图。
具体的，根据世界坐标系、相机坐标系、图像坐标系以及像素坐标系之间的转换规则，将世界坐标系中已构建的虚拟模型的空间位置点，转换成像素坐标系下的像素点位置坐标。由于终端设备的显示区域有限，按照转换得到的像素点位置坐标，在终端设备显示区域的对应像素点位置坐标下，填充与虚拟模型对应位置点一致的像素值，由此在终端设备显示区域得到虚拟模型映射的图案，即为模型视图。
在一种示例性实施例中，上述步骤370具体包括：
将终端设备在位姿下采集的场景图像作为背景，将模型视图覆盖在场景图像上，得到个性化场景图像。
具体的，对于终端设备在某一位姿下采集的场景图像与该位姿下重构的模型视图，软件客户端将该场景图像作为背景，将虚拟模型映射的图案覆盖在场景图像之上，得到个性化场景图像。
在一种示例性实施例中，上述步骤370具体包括：
将模型视图逐帧叠加至终端设备在位姿下采集的图像序列，获得个性化场景视频。
需要说明的是，可以启动视频录制功能，进行图像序列的采集。在进行图像序列采集的同时，针对采集每一帧图像时终端设备的位姿，将该位姿下的模型视图，叠加至所采集的图像上，通过逐帧在采集的场景图像上叠加当前位姿下的模型视图，从而可以得到个性化场景视频。
在一种实施例中，本申请提供的个性化场景图像的处理方法可以基于ARKit 框架实现。ARKit提供两种虚拟增强现实视图，包括3D效果的场景图像，以及2D效果的场景图像。图12是ARKit框架的架构示意图。如图12所示，UIView(用户界面视图)1201的作用是将视图显示在终端设备的显示界面中，ARSCNView(增强现实视图)1202的作用也是显示一个3D场景，只不过这个3D场景是由摄像头捕捉到的现实世界图像构成的。ARSCNView只是一个视图容器，它的作用是管理一个ARSession(增强现实会话)1203。在一个完整的虚拟增强现实体验中，ARKit框架只负责将真实世界画面转变为一个3D场景，这一个转变的过程主要分为两个环节：由ARCamera(增强现实相机)1204负责捕捉摄像头画面作为背景视图，由ARSession负责搭建3D场景，最后由ARSCNView进行3D场景的显示。
其中，ARSession作为ARCamera和ARSCNView之间的沟通桥梁，搭建沟通桥梁的主要参与者有两个ARSessionConfiguration(会话追踪配置)1205与ARFrame(增强现实架构)1206。ARSessionConfiguration的主要目的就是负责追踪相机在3D世界中的位置和方向以及一些特征场景的捕捉(例如平面捕捉)，即检测设备相机看到的现实世界的表面。主要通过调用终端设备配备的传感器来检测终端设备的移动及旋转甚至是翻滚。ARFrame包含了相机捕捉的图片、相机位置等相关信息。其中，节点(SCNNode)1207获取用户绘制的虚拟模型的一个视图,场景类(SCNScene类)1208将该视图与相机拍摄的图像叠加形成个性化场景图像。
图13是基于ARkit框架实现本申请的流程示意图。如图13所示，在步骤1301中，软件客户端接收用户触发“神奇画笔”标签，展示“神奇画笔”标签对应的面板。该面板展示了多种画笔样式。
在步骤1302中，用户可以在展示的多种画笔样式中选择其中一种，软件客户端接收用户在面板中选择指定画笔样式的请求。
在步骤1303中，软件客户端响应请求运行ARSCNView加载场景SCNScene。
在步骤1304中，SCNScene启动相机ARCamera开始捕捉场景图像。
在步骤1305中，软件客户端运行ARSCNView，将捕捉的场景数据传递给ARsession。
在步骤1306中，软件客户端运行ARsession，ARsession通过管理 ARsessionConfiguration实时追踪相机的位姿，返回ARFame(包含场景图像、位姿等信息)。
在步骤1307中，获取用户绘制的触控事件轨迹，在空间中投射形成虚拟模型。其中，ARsessionConfiguration实时追踪相机的位姿是为了确定虚拟模型在空间中相对于相机的真实位置。
在步骤1308中，软件客户端运行SCNNode，捕捉虚拟模型在当前位姿下的视图。
在步骤1309中，软件客户端运行SCNScene将该视图与当前位姿下相机拍摄的图像叠加形成个性化场景图像，并通过UIView在终端设备进行个性化场景图像的显示。
其中，如果用户选择切换画笔种类，则展示弹窗提醒用户是否清楚当前效果。
下述为本申请装置实施例，可以用于执行本申请上述软件客户端110执行的个性化场景图像的处理方法实施例。对于本申请装置实施例中未披露的细节，请参照本申请个性化场景图像的处理方法实施例。
图14是根据一示例性实施例示出的一种个性化场景图像的处理装置的框图，该个性化场景图像的处理装置可以用于图1所示实施环境的终端设备110中，执行图3、图7、图10、图11任一所示的个性化场景图像的处理方法的全部或者部分步骤。如图14所示，该装置包括但不限于：轨迹获取模块1410、模型构建模块1430、视图重构模块1450以及场景叠加模块1470。
其中，轨迹获取模块1410，用于根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹；
模型构建模块1430，用于根据所述触控事件轨迹在空间坐标系中的投射，生成虚拟模型；
视图重构模块1450，用于按照所述终端设备在空间坐标系中的位姿，重构所述虚拟模型在所述终端设备视场范围内映射的模型视图；
场景叠加模块1470，用于将所述模型视图与所述终端设备在所述位姿下采集的场景图像进行叠加，获得个性化场景图像。
上述装置中各个模块的功能和作用的实现过程具体详见上述个性化场景图 像的处理方法中对应步骤的实现过程，在此不再赘述。
轨迹获取模块1410比如可以是图2中的某一个物理结构传感器组件214。
模型构建模块1430、视图重构模块1450以及场景叠加模块1470也可以是功能模块，用于执行上述个性化场景图像的处理方法中的对应步骤。可以理解，这些模块可以通过硬件、软件、或二者结合来实现。当以硬件方式实现时，这些模块可以实施为一个或多个硬件模块，例如一个或多个专用集成电路。当以软件方式实现时，这些模块可以实施为在一个或多个处理器上执行的一个或多个计算机程序，例如图2的处理器218所执行的存储在存储器204中的程序。
在一种示例性实施例中，上述个性化场景图像的处理装置还包括：
样式获取模块，用于根据接收到的轨迹绘制请求，展示多种轨迹素材样式；
样式选择模块，用于根据对所述多种轨迹素材样式触发的选择，获得选择的目标轨迹样式；所述虚拟模型按照所述目标轨迹样式进行配置。
在一种示例性实施例中，如图15所示，所述轨迹获取模块1410包括：
图像采集单元1411，用于启动所述终端设备的图像采集装置，采集当前场景图像；
事件接收单元1412，用于接收所述当前场景图像显示界面内触发的触控事件，记录所述触控事件的触发位置，所述触控事件的触发位置构成所述触控事件轨迹。
在一种示例性实施例中，如图16所示，所述模型构建模块1430包括：
轨迹投射单元1431，用于根据所述终端设备在空间坐标系中的位姿，将所述触控事件轨迹投射到一个平面上，所述平面位于所述终端设备视场方向上，且与所述终端设备的当前位置的距离为预设距离，所述位姿包括所述终端设备在所述空间坐标系中的位置以及所述终端设备中摄像头的采集方向，所述视场方向为所述终端设备中摄像头的采集方向；
模型生成单元1432，用于按照配置的目标轨迹样式，将所述触控事件轨迹在所述平面的投射位置处生成虚拟模型。
在一种示例性实施例中，如图17所示，所述视图重构模块1450包括：
视场定位单元1451，用于根据所述终端设备在空间坐标系中的位置以及述终端设备中摄像头的采集方向，定位所述终端设备在所述空间坐标系中的视场范 围；
模型映射单元1452，用于将处于所述视场范围内的虚拟模型映射到所述终端设备的显示区域，形成模型视图。
在一种示例性实施例中，所述场景叠加模块1470包括：
视图覆盖单元，用于将所述终端设备在所述位姿下采集的场景图像作为背景，将所述模型视图覆盖在所述场景图像上，得到所述个性化场景图像。
在一种示例性实施例中，所述场景叠加模块1470包括：
将所述模型视图逐帧叠加至所述终端设备在所述位姿下采集的图像序列，获得个性化场景视频。
在一些实施例中，本申请还提供一种电子设备，该电子设备可以用于图1所示实施环境的终端设备110中，执行图3、图7、图10、图11任一所示的个性化场景图像的处理方法的全部或者部分步骤。所述电子设备包括：
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器被配置为执行上述示例性实施例所述的个性化场景图像的处理方法。
该实施例中电子设备的处理器执行操作的具体方式已经在有关该个性化场景图像的处理方法的实施例中执行了详细描述，此处将不做详细阐述说明。
在示例性实施例中，还提供了一种存储介质，该存储介质为计算机可读存储介质，例如可以为包括指令的临时性和非临时性计算机可读存储介质。该存储介质例如包括指令的存储器204，该存储介质存储有计算机程序，所述计算机程序可由可由计算设备200的处理器218执行以完成上述个性化场景图像的处理方法。
应当理解的是，本申请并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围执行各种修改和改变。本申请的范围仅由所附的权利要求来限制。

Claims (15)

  1. 一种个性化场景图像的处理方法，由终端设备执行，包括；
    根据在所述终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹；
    根据所述触控事件轨迹在空间坐标系中的投射，生成虚拟模型；
    按照所述终端设备在空间坐标系中的位姿，重构所述虚拟模型在所述终端设备视场范围内映射的模型视图；
    将所述模型视图与所述终端设备在所述位姿下采集的场景图像进行叠加，获得个性化场景图像。
  2. 根据权利要求1所述的方法，其中，所述根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹之前，所述方法还包括：
    根据接收到的轨迹绘制请求，展示多种轨迹素材样式；
    根据对所述多种轨迹素材样式触发的选择操作，获得选择的目标轨迹样式，其中，根据所述触控事件轨迹在所述空间坐标系中的投射，按照所述目标轨迹样式所配置的画笔样式生成所述虚拟模型。
  3. 根据权利要求1所述的方法，其中，在根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹之前，所述方法进一步包括：
    启动所述终端设备的图像采集装置，采集当前场景图像；
    其中，所述根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹，包括：
    接收所述当前场景图像显示界面内触发的触控事件，记录所述触控事件的触发位置，所述触控事件的触发位置的连续变化构成所述触控事件轨迹。
  4. 根据权利要求1所述的方法，其中，所述位姿包括所述终端设备在所述空间坐标系中的位置以及所述终端设备中摄像头的采集方向；
    其中，所述根据所述触控事件轨迹在空间坐标系中的投射，生成虚拟模型，包括：
    根据所述终端设备在空间坐标系中的位置以及述终端设备中摄像头的采集方向，将所述触控事件轨迹投射到一个平面上，所述平面位于所述终端设备中摄 像头的采集方向上，且与所述终端设备的当前位置的距离为预设距离；
    按照配置的目标轨迹样式，将所述触控事件轨迹在所述平面的投射位置处生成虚拟模型。
  5. 根据权利要求4所述的方法，其中，所述按照所述终端设备在空间坐标系中的位姿，重构所述虚拟模型在所述终端设备视场范围内映射的模型视图，包括：
    根据所述终端设备在空间坐标系中的位置以及述终端设备中摄像头的采集方向，重构所述终端设备在所述空间坐标系中的视场范围；
    将处于所述视场范围内的虚拟模型映射到所述终端设备的显示区域，得到模型视图。
  6. 根据权利要求1所述的方法，其中，所述将所述模型视图与所述终端设备在所述位姿下采集的场景图像进行叠加，获得个性化场景图像，包括：
    将所述终端设备在所述位姿下采集的场景图像作为背景，将所述模型视图覆盖在所述场景图像上，得到所述个性化场景图像。
  7. 根据权利要求1所述的方法，其中，所述将所述模型视图与所述终端设备在所述位姿下采集的场景图像进行叠加，获得个性化场景图像，包括：
    将所述模型视图逐帧叠加至所述终端设备在所述位姿下采集的图像序列，获得个性化场景视频。
  8. 一种个性化场景图像的处理装置，包括：
    轨迹获取模块，用于根据在终端设备的屏幕区域内触发的触控事件，获取所述屏幕区域内的触控事件轨迹；
    模型构建模块，用于根据所述触控事件轨迹在空间坐标系中的投射，生成虚拟模型；
    视图重构模块，用于按照所述终端设备在空间坐标系中的位姿，重构所述虚拟模型在所述终端设备视场范围内映射的模型视图；
    场景叠加模块，用于将所述模型视图与所述终端设备在所述位姿下采集的场景图像进行叠加，获得个性化场景图像。
  9. 根据权利要求8所述的装置，其中，所述装置还包括：
    样式获取模块，用于根据接收到的轨迹绘制请求，展示多种轨迹素材样式；
    样式选择模块，用于根据对所述多种轨迹素材样式触发的选择操作，获得选择的目标轨迹样式；其中，根据所述触控事件轨迹在所述空间坐标系中的投射，按照所述目标轨迹样式所配置的画笔样式生成所述虚拟模型。
  10. 根据权利要求8所述的装置，其中，所述轨迹获取模块包括：
    图像采集单元，用于启动所述终端设备的图像采集装置，采集当前场景图像；
    事件接收单元，用于接收所述当前场景图像显示界面内触发的触控事件，记录所述触控事件的触发位置，所述触控事件的触发位置的连续变化构成所述触控事件轨迹。
  11. 根据权利要求8所述的装置，其中，所述位姿包括所述终端设备在所述空间坐标系中的位置以及所述终端设备中摄像头的采集方向；
    其中，所述模型构建模块包括：
    轨迹投射单元，用于根据所述终端设备在空间坐标系中的位置以及述终端设备中摄像头的采集方向，将所述触控事件轨迹投射到一个平面上，所述平面位于所述终端设备中摄像头的采集方向上，且与所述终端设备的当前位置的距离为预设距离；
    模型生成单元，用于按照配置的目标轨迹样式，将所述触控事件轨迹在所述平面的投射位置处生成虚拟模型。
  12. 根据权利要求11所述的装置，其中，所述视图重构模块包括：
    视场定位单元，用于根据所述终端设备在空间坐标系中的位置以及述终端设备中摄像头的采集方向，重构所述终端设备在所述空间坐标系中的视场范围；
    模型映射单元，用于将处于所述视场范围内的虚拟模型映射到所述终端设备的显示区域，得到模型视图。
  13. 根据权利要求8所述的装置，其中，所述场景叠加模块包括：
    视图覆盖单元，用于将所述终端设备在所述位姿下采集的场景图像作为背景，将所述模型视图覆盖在所述场景图像上，得到所述个性化场景图像。
  14. 一种电子设备，包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器被配置执行权利要求1-7任意一项所述的个性化场景图像 的处理方法。
  15. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序可由处理器执行完成如权利要求1-7任意一项所述的个性化场景图像的处理方法。
PCT/CN2019/091303 2018-07-03 2019-06-14 个性化场景图像的处理方法、装置及存储介质 WO2020007182A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP19829990.1A EP3819752A4 (en) 2018-07-03 2019-06-14 CUSTOM SCENE IMAGE PROCESSING METHOD AND APPARATUS, AND STORAGE MEDIA
US16/994,516 US11880999B2 (en) 2018-07-03 2020-08-14 Personalized scene image processing method, apparatus and storage medium
US18/507,178 US20240078703A1 (en) 2018-07-03 2023-11-13 Personalized scene image processing method, apparatus and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810717653.X 2018-07-03
CN201810717653.XA CN110162258A (zh) 2018-07-03 2018-07-03 个性化场景图像的处理方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/994,516 Continuation US11880999B2 (en) 2018-07-03 2020-08-14 Personalized scene image processing method, apparatus and storage medium

Publications (1)

Publication Number Publication Date
WO2020007182A1 true WO2020007182A1 (zh) 2020-01-09

Family

ID=67645009

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/091303 WO2020007182A1 (zh) 2018-07-03 2019-06-14 个性化场景图像的处理方法、装置及存储介质

Country Status (4)

Country Link
US (2) US11880999B2 (zh)
EP (1) EP3819752A4 (zh)
CN (1) CN110162258A (zh)
WO (1) WO2020007182A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114067040A (zh) * 2022-01-05 2022-02-18 北京顶象技术有限公司 一种动态生成验证码背景图的方法及装置

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110941341B (zh) * 2019-11-29 2022-02-01 维沃移动通信有限公司 图像控制方法及电子设备
CN111833459B (zh) * 2020-07-10 2024-04-26 北京字节跳动网络技术有限公司 一种图像处理方法、装置、电子设备及存储介质
CN113157092B (zh) * 2021-04-08 2023-03-24 海信视像科技股份有限公司 可视化方法、终端设备和存储介质
CN115150551A (zh) * 2022-06-20 2022-10-04 湖北星纪时代科技有限公司 用于终端的拍照处理方法、装置、电子设备和存储介质
CN116110080B (zh) * 2023-04-04 2023-07-04 成都新希望金融信息有限公司 一种真人面签和虚拟人面签的切换方法

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103390287A (zh) * 2012-05-11 2013-11-13 索尼电脑娱乐欧洲有限公司 用于增强现实的装置和方法
CN105068653A (zh) * 2015-07-22 2015-11-18 深圳多新哆技术有限责任公司 确定虚拟空间中触摸事件的方法及装置
US20150363965A1 (en) * 2014-06-17 2015-12-17 Chief Architect Inc. Virtual Model Navigation Methods and Apparatus
CN105894566A (zh) * 2015-12-01 2016-08-24 乐视致新电子科技（天津）有限公司 模型渲染方法及装置
CN106648098A (zh) * 2016-12-23 2017-05-10 武汉市马里欧网络有限公司 一种自定义场景的ar投影方法及系统
US20170154468A1 (en) * 2015-12-01 2017-06-01 Le Holdings (Beijing) Co., Ltd. Method and electronic apparatus for constructing virtual reality scene model
CN107204031A (zh) * 2017-04-27 2017-09-26 腾讯科技（深圳）有限公司 信息展示方法及装置

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2813069A4 (en) * 2012-02-08 2016-12-07 Intel Corp CREATING AN ADVANCED REALITY WITH A REAL SCENE
US10281987B1 (en) * 2013-08-09 2019-05-07 Leap Motion, Inc. Systems and methods of free-space gestural interaction
US9235924B2 (en) * 2013-12-17 2016-01-12 Microsoft Technology Licensing, Llc Cubify brush operation for virtual worlds
US20170287215A1 (en) * 2016-03-29 2017-10-05 Google Inc. Pass-through camera user interface elements for virtual reality
CN105912257B (zh) * 2016-04-11 2019-03-05 腾讯科技（深圳）有限公司 图像混合处理方法和装置
US10234964B2 (en) * 2016-12-19 2019-03-19 Wacom Co., Ltd. Image processing apparatus
CN107526443A (zh) * 2017-09-29 2017-12-29 北京金山安全软件有限公司 一种增强现实方法、装置、系统、电子设备及存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103390287A (zh) * 2012-05-11 2013-11-13 索尼电脑娱乐欧洲有限公司 用于增强现实的装置和方法
US20150363965A1 (en) * 2014-06-17 2015-12-17 Chief Architect Inc. Virtual Model Navigation Methods and Apparatus
CN105068653A (zh) * 2015-07-22 2015-11-18 深圳多新哆技术有限责任公司 确定虚拟空间中触摸事件的方法及装置
CN105894566A (zh) * 2015-12-01 2016-08-24 乐视致新电子科技（天津）有限公司 模型渲染方法及装置
US20170154468A1 (en) * 2015-12-01 2017-06-01 Le Holdings (Beijing) Co., Ltd. Method and electronic apparatus for constructing virtual reality scene model
CN106648098A (zh) * 2016-12-23 2017-05-10 武汉市马里欧网络有限公司 一种自定义场景的ar投影方法及系统
CN107204031A (zh) * 2017-04-27 2017-09-26 腾讯科技（深圳）有限公司 信息展示方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3819752A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114067040A (zh) * 2022-01-05 2022-02-18 北京顶象技术有限公司 一种动态生成验证码背景图的方法及装置
CN114067040B (zh) * 2022-01-05 2022-05-13 北京顶象技术有限公司 一种动态生成验证码背景图的方法及装置

Also Published As

Publication number Publication date
US11880999B2 (en) 2024-01-23
EP3819752A1 (en) 2021-05-12
EP3819752A4 (en) 2021-09-29
US20240078703A1 (en) 2024-03-07
CN110162258A (zh) 2019-08-23
US20200380724A1 (en) 2020-12-03

Similar Documents

Publication Publication Date Title
WO2020007182A1 (zh) 个性化场景图像的处理方法、装置及存储介质
CN110147231B (zh) 组合特效生成方法、装置及存储介质
CN110708596A (zh) 生成视频的方法、装置、电子设备及可读存储介质
CN111240673B (zh) 互动图形作品生成方法、装置、终端及存储介质
US20150185825A1 (en) Assigning a virtual user interface to a physical object
CN108038726B (zh) 物品展示方法及装置
US20150130836A1 (en) Adapting content to augmented reality virtual objects
WO2016114930A2 (en) Systems and methods for augmented reality art creation
CN103426202A (zh) 三维全景互动移动终端展示系统及展示方法
US11776209B2 (en) Image processing method and apparatus, electronic device, and storage medium
CN109582122B (zh) 增强现实信息提供方法、装置及电子设备
CN110751707B (zh) 动画显示方法、装置、电子设备及存储介质
CN110782532B (zh) 图像生成方法、生成装置、电子设备及存储介质
CN112199016A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
US20230334789A1 (en) Image Processing Method, Mobile Terminal, and Storage Medium
US20190155465A1 (en) Augmented media
CN114387445A (zh) 对象关键点识别方法及装置、电子设备和存储介质
US20200106967A1 (en) System and method of configuring a virtual camera
KR20150079387A (ko) 카메라 광 데이터로 가상 환경을 조명하는 방법
KR20140102386A (ko) 디스플레이장치 및 그 제어방법
CN109636917B (zh) 三维模型的生成方法、装置、硬件装置
CN116170624A (zh) 一种对象展示方法、装置、电子设备及存储介质
CN109544698A (zh) 图像展示方法、装置及电子设备
CN109472873B (zh) 三维模型的生成方法、装置、硬件装置
US20230405475A1 (en) Shooting method, apparatus, device and medium based on virtual reality space

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19829990

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019829990

Country of ref document: EP

Effective date: 20210203