WO2023072113A1 - 显示方法及电子设备 - Google Patents

显示方法及电子设备 Download PDF

Info

Publication number
WO2023072113A1
WO2023072113A1 PCT/CN2022/127535 CN2022127535W WO2023072113A1 WO 2023072113 A1 WO2023072113 A1 WO 2023072113A1 CN 2022127535 W CN2022127535 W CN 2022127535W WO 2023072113 A1 WO2023072113 A1 WO 2023072113A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional representation
electronic device
dimensional
field
view
Prior art date
Application number
PCT/CN2022/127535
Other languages
English (en)
French (fr)
Inventor
郭洋洋
林涛
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202210222388.4A external-priority patent/CN116071523A/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP22885990.6A priority Critical patent/EP4383191A1/en
Publication of WO2023072113A1 publication Critical patent/WO2023072113A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation

Definitions

  • the present application relates to the field of media technology, in particular to a display method and electronic equipment.
  • Augmented reality (augmented reality, AR) technology is a technology that ingeniously integrates virtual information with the real world.
  • the generated text, images, 3D models, music, video and other virtual information are simulated and applied to the real world, and the two kinds of information complement each other, thereby realizing the "enhancement" of the real world.
  • the present application provides a display method and an electronic device, which can improve the interactivity and immersion of AR applications, so as to improve user experience when users use AR applications.
  • the application adopts the following technical solutions:
  • the present application provides a display method applied to an electronic device including a camera, and the method includes:
  • the electronic device starts the camera and displays the application interface in response to the first operation.
  • the electronic device then performs a third operation in response to the second operation.
  • the application interface includes a viewfinder frame
  • the viewfinder frame of the viewfinder includes a target object
  • the third operation includes generating a first mask (mask) in the first area and projecting the first three-dimensional representation in the first field of view to the first mask
  • the first area is the area where the target object is located in the viewfinder screen
  • the first field of view is the field of view corresponding to the first area in the 3D map
  • the first field of view includes the first 3D representation and the second 3D representation
  • the second 3D representation is the 3D representation of the target object.
  • the first three-dimensional representation is a three-dimensional representation other than the second three-dimensional representation among the three-dimensional representations within the first field of view, and the above three-dimensional representation includes at least one of point cloud, mesh (Mesh) or semantics.
  • the electronic device can display the objects in the real world captured by the camera through the viewfinder screen in the AR application interface, and the user can make the AR application first viewfinder through operations.
  • the area where an object is located in the screen generates a mask to block the object, and then divides the corresponding virtual three-dimensional object (i.e.
  • the second three-dimensional other three-dimensional representations i.e., the first three-dimensional representation
  • the AR application will display "After the object disappears invisible The real world", which increases the interactivity between AR applications and the real world, and enhances the user's immersion in AR applications.
  • the user can make the cabinet invisible in the AR application through the operation to view the background wall behind the cabinet through the AR application, thereby increasing the interactivity of the AR application in the real world and improving the user's immersion in the AR application.
  • the framing frame of the framing frame may include multiple objects, and the target object may be any object in the multiple objects.
  • the electronic device can display multiple objects in the real world captured by the camera through the viewfinder screen in the AR application interface, and the user can first select from these multiple objects Select an object as the target object, and then make the AR application generate a mask to block the object in the area where the target object is located in the viewfinder through operation, and then use the field of view corresponding to the area where the object is located in the 3D map (that is, the first field of view)
  • the AR application displays "the real world after the object disappears invisible", thereby increasing the interaction between the AR application and the real world and improving the user's immersion in the AR application.
  • the framing screen of the framing frame may include editable objects and non-editable objects, and the target object is any editable object.
  • the electronic device can display multiple objects in the real world captured by the camera through the viewfinder screen in the AR application interface, and the user can first select from these multiple objects Select an editable object as the target object, and then make the AR application generate a mask to block the object in the area where the target object is located in the viewfinder through operations, and then use the field of view corresponding to the area where the object is located in the 3D map (that is, the first In addition to the virtual three-dimensional object corresponding to the object (that is, the second three-dimensional representation), other three-dimensional representations (that is, the first three-dimensional representation) are projected onto the above-mentioned mask, so that the user in the AR application in the real world The object disappears invisible with the user's operation, so that the AR application displays "the real world after the object disappears invisible", thereby increasing the interactivity between the AR application and the real world, and improving the user's immersion in the AR application.
  • the method may further include: marking the three-dimensional representations in the three-dimensional map as editable three-dimensional representations and non-editable three-dimensional representations.
  • the display method provided by the embodiment of the present application can mark the three-dimensional representations in the three-dimensional map as editable three-dimensional representations and non-editable three-dimensional representations.
  • the electronic device can display multiple objects in the real world captured by the camera through the viewfinder screen in the AR application interface.
  • the electronic device can determine whether the object can be edited through the three-dimensional mark corresponding to the object.
  • the object selected by the user is You can edit the object, set the object as the target object, and then make the AR application generate a mask to block the object in the area where the target object is located in the viewfinder through operation, and then use the field of view corresponding to the area where the object is located in the 3D map (ie Other three-dimensional representations (that is, the first three-dimensional representation) other than the virtual three-dimensional object corresponding to the object (that is, the second three-dimensional representation) within the first field of view) are projected onto the above mask, so that the user in the AR application is in the real world
  • the object disappears invisible with the user's operation, so that the AR application displays "the real world after the object disappears invisible", thereby increasing the interactivity between the AR application and the real world, and improving the user's immersion in the AR application.
  • the third operation may further include filling an area in the first mask where the three-dimensional representation is not projected according to the third three-dimensional representation.
  • the third three-dimensional representation is a three-dimensional representation within the second viewing range
  • the second viewing range is a viewing range adjacent to the first viewing range.
  • the 3D representation of the background part behind the object in the 3D map may be incomplete, and the electronic device may cause the background behind the invisible object in the framing screen to be incomplete after performing the above third operation.
  • the viewfinder screen is not continuous.
  • the area not projected in the first mask can be filled according to the third three-dimensional representation to complement the missing background in the viewfinder picture to make the viewfinder picture continuous, thereby improving the user experience of the AR application.
  • the method may further include: in response to the fourth operation, the electronic device performs a third operation to generate a second mask in the second area and project the second three-dimensional representation to the second mask.
  • the electronic device can first hide objects in the real world where the user is located in the viewfinder screen in the AR application, and then generate a mask in the user-specified area (ie, the second area) and map the object to The three-dimensional representation of the object is projected onto the mask, so that the object in the real world where the user is located in the AR application moves with the user's operation, so that the AR application displays "the real world after the object moves", thereby further increasing the relationship between the AR application and the real world.
  • the interactivity of the world enhances the user immersion of AR applications
  • the method may further include: in response to the fifth operation, the electronic device performs a third operation, generates a third mask in the third area and projects the fourth three-dimensional representation to the third mask, and the fourth The three-dimensional representation is the deformed second three-dimensional representation.
  • the electronic device can first hide objects in the real world where the user is located in the viewing screen in the AR application, and then generate a mask in the viewing screen and project the deformed second three-dimensional representation to the This mask enables the object in the real world where the user is located in the AR application to deform with the user's operation, so that the AR application displays "the real world after the deformation of the object", thereby further increasing the interactivity between the AR application and the real world.
  • the method may further include: generating a fourth three-dimensional representation according to deformation information and three-dimensional coordinates of the second three-dimensional representation, where the deformation information is used to represent a deformation type and a deformation magnification.
  • the electronic device may first hide objects in the real world where the user is located in the framing screen in the AR application, and then generate a fourth three-dimensional representation (ie, deformed second 3D representation), and then generate a mask in the viewfinder and project the deformed second 3D representation onto the mask, so that the objects in the real world where the user is in the AR application can be deformed with the user's operation , so that the AR application displays "the real world after the deformation of the object", thereby further increasing the interactivity between the AR application and the real world, and improving the user's immersion of the AR application.
  • a fourth three-dimensional representation ie, deformed second 3D representation
  • the fourth three-dimensional representation may be the enlarged, reduced or stretched second three-dimensional representation.
  • the fourth three-dimensional representation may be the second three-dimensional representation enlarged by 5 times.
  • the electronic device can first hide objects in the real world where the user is located in the viewing screen in the AR application, and then generate a mask in the viewing screen and zoom in, shrink or stretch the first object.
  • the 2D and 3D representation is projected onto the mask, so that the objects in the real world where the user is located in the AR application can be enlarged, reduced or stretched according to the user's operation, so that the AR application can display "the real world after the object is enlarged, reduced or stretched.” ", thereby further increasing the interactivity between AR applications and the real world, and improving the user's immersion in AR applications.
  • the anchor points of the second three-dimensional representation and the fourth three-dimensional representation may be the same.
  • the anchor point may be a point with the smallest Z-axis coordinate in the direction of the vertical line of the center of gravity of the three-dimensional representation.
  • the method may further include: determining the third field of view range according to the pose information of the framing picture, and A visual range of the position determination area within the third visual range.
  • the third field of view is the field of view corresponding to the viewfinder picture in the three-dimensional map.
  • the electronic device can display the objects in the real world captured by the camera through the viewfinder screen in the AR application interface, and the user can make the AR application first viewfinder through operations.
  • the area where an object is located in the screen generates a mask to block the object, and then determines the field of view (ie, the first field of view) corresponding to the area where the object is located in the three-dimensional map through the pose information of the viewfinder screen, and then passes the three-dimensional map.
  • the method may further include: according to the first image block, determining a second three-dimensional representation from three-dimensional representations within the first field of view.
  • the first image block is an image block including the target object in the viewfinder picture.
  • the electronic device can display the objects in the real world captured by the camera through the viewfinder screen in the AR application interface, and the user can make the AR application first viewfinder through operations.
  • a mask is generated in the area where an object is located in the screen to block the object, and then the three-dimensional representation corresponding to the object is determined from the field of view corresponding to the area where the object is located in the three-dimensional map (ie, the first field of view) through the image block containing the object and Project other three-dimensional representations (i.e., the first three-dimensional representation) in the field of view corresponding to the area where the object is located in the three-dimensional map to the above mask, so that the user in the AR application is in the real world.
  • the object disappears invisible with the user's operation, so that the AR application displays "the real world after the object disappears invisible", thereby increasing the interactivity between the AR application and the real world, and improving the user's immersion in the AR application.
  • the present application also provides another display method applied to an electronic device including a camera.
  • the method includes: firstly, the electronic device starts the camera and displays an application interface in response to a first operation.
  • the electronic device then performs a third operation in response to the second operation.
  • the application interface includes a viewfinder frame
  • the viewfinder frame of the viewfinder includes the target object
  • the third operation includes generating a mask in the first area and projecting a three-dimensional representation within the first field of view to the first mask, and the first area is the viewfinder image
  • the first field of view is the field of view corresponding to the first area in the 3D map
  • the 3D representation includes at least one of point cloud, Mesh, or semantics.
  • the electronic device can display the objects in the real world captured by the camera through the viewfinder screen in the AR application interface, and the user can make the AR application first viewfinder through operations.
  • a mask is generated in the area where an object is located in the screen to block the object, and then a virtual three-dimensional object (ie, a three-dimensional representation) within the field of view corresponding to the area where the object is located in the three-dimensional map (ie, the first field of view) is projected onto the above mask film, so that the object in the real world where the user is in the AR application disappears invisible with the user's operation, so that the AR application displays "the real world after the object disappears invisible", thereby increasing the interactivity between the AR application and the real world and improving Enhanced user immersion in AR applications.
  • the user can execute the above display method by operating the electronic device, so that the pedestrian disappears from the viewing screen, and the viewing screen displays "the real world after the pedestrian disappears invisible".
  • the electronic device may perform the foregoing third operation when the first three-dimensional representation does not exist within the first field of view.
  • the first three-dimensional representation is a three-dimensional representation of the target object.
  • the absence of the first three-dimensional representation within the first field of view indicates that the three-dimensional representation of the target object is not stored in the three-dimensional map, that is, the scene stored in the three-dimensional map is a scene without the target object. Therefore, the three-dimensional representation of the area where the target object is located in the three-dimensional map can be directly projected into the viewing frame so that the target object disappears invisible from the viewing frame.
  • the method further includes: determining whether a first three-dimensional representation exists from the three-dimensional representations within the first field of view according to the first image block, the first image block being the viewfinder The image block of the target object is included in the frame, and the first three-dimensional representation is a three-dimensional representation of the target object.
  • the first image block can be matched to the corresponding three-dimensional representation within the first field of view, it means that there is a three-dimensional representation of the target object within the first field of view; If the corresponding three-dimensional representation can be matched, it means that there is no three-dimensional representation of the target object within the first field of view.
  • the present application also provides an electronic device, including: at least one processor, a memory, and at least one camera, the memory, the camera and the processor are coupled, the memory is used to store computer program codes, the computer program codes include computer instructions,
  • the processor reads computer instructions from the memory, so that the electronic device performs the following operations: in response to the first operation, start the camera and display the application interface, the application interface includes a viewing frame, and the viewing frame of the viewing frame includes a target object;
  • the second operation is to perform the third operation.
  • the third operation includes generating a first mask mask in the first area and projecting the first three-dimensional representation in the first field of view to the first mask.
  • the first area is the target object in the viewfinder picture
  • the area where the first field of view is the field of view corresponding to the first area in the three-dimensional map
  • the first field of view includes the first three-dimensional representation and the second three-dimensional representation
  • the second three-dimensional representation is the three-dimensional representation of the target object.
  • the first three-dimensional representation is a three-dimensional representation other than the first three-dimensional representation among the three-dimensional representations within the first field of view, and the above three-dimensional representation includes at least one of point cloud, Mesh, or semantics.
  • the framing frame of the framing frame may include multiple objects, and the target object may be any object in the multiple objects.
  • the framing screen of the framing frame may include editable objects and non-editable objects, and the target object is any editable object.
  • the processor is further configured to mark the three-dimensional representations in the three-dimensional map as editable three-dimensional representations and non-editable three-dimensional representations.
  • the third operation may further include filling an area in the first mask where the three-dimensional representation is not projected according to the third three-dimensional representation.
  • the third three-dimensional representation is a three-dimensional representation within the second viewing range
  • the second viewing range is a viewing range adjacent to the first viewing range.
  • the processor is further configured to cause the electronic device to perform the following operations: in response to the fourth operation, perform a third operation to generate a second mask in the second area and project the second three-dimensional representation to the second mask.
  • the processor is further configured to cause the electronic device to perform the following operations: in response to the fifth operation, perform a third operation, generate a third mask in the third area and project the fourth three-dimensional representation to the third mask, the fourth three-dimensional representation is the fourth three-dimensional representation, and the fourth three-dimensional representation is the deformed second three-dimensional representation.
  • the processor is further configured to cause the electronic device to perform the following operations: generate a fourth 3D representation according to deformation information and 3D coordinates of the second 3D representation, where the deformation information is used to represent a deformation type and a deformation magnification.
  • the fourth three-dimensional representation may be the enlarged, reduced or stretched second three-dimensional representation.
  • the fourth three-dimensional representation may be the second three-dimensional representation enlarged by 5 times.
  • the anchor points of the second three-dimensional representation and the fourth three-dimensional representation may be the same.
  • the anchor point may be a point with the smallest Z-axis coordinate in the direction of the vertical line of the center of gravity of the three-dimensional representation.
  • the processor is further configured to cause the electronic device to perform the following operations: determine the third field of view range according to the pose information of the viewfinder picture, ) in the viewfinder frame determines the field of view of the region within the third field of view.
  • the third field of view is the field of view corresponding to the viewfinder screen in the three-dimensional map.
  • the processor is further configured to cause the electronic device to perform the following operation: determine a second three-dimensional representation from three-dimensional representations within the first field of view according to the first image block.
  • the first image block is an image block including the target object in the viewfinder picture.
  • the present application further provides a chip, including: an input interface, an output interface, and at least one processor.
  • the chip also includes a memory.
  • the at least one processor is used to execute the code in the memory, and when the at least one processor executes the code, the chip implements the method described in the above first aspect or any possible implementation thereof.
  • the aforementioned chip may also be an integrated circuit.
  • the present application further provides a terminal, where the terminal includes the above-mentioned electronic device or the above-mentioned chip.
  • the present application further provides a computer-readable storage medium for storing a computer program, and the computer program includes a method for realizing the above-mentioned first aspect or any possible implementation thereof.
  • the present application further provides a computer program product containing instructions, which, when run on a computer, enable the computer to implement the method described in the above first aspect or any possible implementation thereof.
  • the electronic equipment, computer storage medium, computer program product and chip provided in this embodiment are all used to implement the display method provided above, therefore, the beneficial effects that it can achieve can refer to the beneficial effects of the display method provided above effects, which will not be repeated here.
  • FIG. 1 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
  • FIG. 2 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a user interface of an electronic device provided in an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a user interface of another electronic device provided in an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a user interface of another electronic device provided in an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a user interface of another electronic device provided in an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a user interface of another electronic device provided in an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a user interface of another electronic device provided in an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of a display method provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of another display method provided by the embodiment of the present application.
  • Fig. 11 is a schematic structural diagram of a device provided by an embodiment of the present application.
  • Fig. 12 is a schematic structural diagram of another device provided by the embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of a chip provided by an embodiment of the present application.
  • first and second in the specification and drawings of the present application are used to distinguish different objects, or to distinguish different processes for the same object, rather than to describe a specific sequence of objects.
  • the display method provided by the embodiment of the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, AR devices, notebook computers, ultra-mobile personal computers (ultra-mobile personal computers, UMPCs), netbooks, personal digital assistants (personal digital assistant, PDA) and other electronic devices that can implement the AR function, the embodiments of the present application do not impose any restrictions on the specific types of electronic devices.
  • FIG. 1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.
  • the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 .
  • the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the electronic device 100 may also include components such as a vision sensor, an inertial measurement unit (inertial measurement unit, IMU), a laser radar (light detection and ranging, LIDAR), and a magnetometer.
  • the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU) wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit, NPU
  • the controller may be the nerve center and command center of the electronic device 100 .
  • the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus
  • the processor 110 can couple the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100 .
  • the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 .
  • MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc.
  • the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 .
  • the processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .
  • the interface connection relationship between the modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 .
  • the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.
  • the charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger.
  • the power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 .
  • the power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 .
  • the electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor.
  • the GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos and the like.
  • the display screen 194 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.
  • the electronic device 100 can realize the shooting function through the ISP, the camera 193 , the touch sensor, the video codec, the GPU, the display screen 194 and the application processor.
  • the ISP is used for processing the data fed back by the camera 193 .
  • the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.
  • ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be located in the camera 193 .
  • Camera 193 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other image signals. It should be understood that in the description of the embodiment of the application, the image in RGB format is used as an example. The embodiment of the application does not limit the image format. .
  • the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 100 may support one or more video codecs.
  • the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.
  • MPEG moving picture experts group
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100.
  • the internal memory 121 may be used to store computer-executable program codes including instructions.
  • the processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 .
  • the internal memory 121 may include an area for storing programs and an area for storing data.
  • the electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.
  • the pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal.
  • the gyro sensor 180B can be used to determine the motion posture of the electronic device 100 .
  • the air pressure sensor 180C is used to measure air pressure.
  • the magnetic sensor 180D includes a Hall sensor.
  • the electronic device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip leather case.
  • the acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes).
  • the distance sensor 180F is used to measure the distance.
  • the electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.
  • Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes.
  • the ambient light sensor 180L is used for sensing ambient light brightness.
  • the electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.
  • the temperature sensor 180J is used to detect temperature.
  • Touch sensor 180K also known as "touch panel".
  • the touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”.
  • the touch sensor 180K is used to detect a touch operation on or near it.
  • the bone conduction sensor 180M can acquire vibration signals.
  • the audio module 170 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the voice part acquired by the bone conduction sensor 180M, so as to realize the voice function.
  • the keys 190 include a power key, a volume key and the like.
  • the key 190 may be a mechanical key. It can also be a touch button.
  • the electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .
  • the motor 191 can generate a vibrating reminder.
  • the motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects.
  • the motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 .
  • the indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
  • the SIM card interface 195 is used for connecting a SIM card.
  • the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.
  • the embodiment of the present application uses an operating system with a layered architecture as an example to illustrate the software structure of the electronic device 100 .
  • FIG. 2 is a block diagram of the software structure of the electronic device 100 according to the embodiment of the present application.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
  • the operating system is divided into four layers, which are application program layer, application program framework layer, operating system runtime (runtime) and system library, and kernel layer from top to bottom.
  • the application layer can consist of a series of application packages. As shown in FIG. 2, the application package may include application programs such as camera, photo album, music, and settings.
  • the application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
  • the application framework layer includes some predefined functions. As shown in Figure 2, the application framework layer can include window managers, content providers, view systems, resource managers, notification managers, etc.
  • a window manager is used to manage window programs.
  • the window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.
  • Content providers are used to store and retrieve data and make it accessible to applications.
  • Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.
  • the view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on.
  • the view system can be used to build applications.
  • a display interface can consist of one or more views.
  • a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
  • the resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.
  • the notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages.
  • the notification information displayed in the status bar can disappear automatically after a short stay, such as a message reminder to inform the user that the download is complete.
  • the notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, text information is prompted in the status bar, or the notification manager can also emit a prompt sound, such as electronic device vibration, indicator light flashing, and the like.
  • Runtime includes core library and virtual machine. Runtime is responsible for the scheduling and management of the operating system.
  • the core library consists of two parts: one part is the functions that the java language needs to call, and the other part is the core library of the operating system.
  • the application layer and the application framework layer run in virtual machines.
  • the virtual machine executes the java files of the application program layer and the application program framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • a system library can include multiple function modules. For example: surface manager (surface manager), media library (media libraries), 3D graphics processing library (for example: open graphics library (open graphics library, OpenGL) embedded systems (embedded systems, ES)), 2D graphics engine (for example : Scene graph library (scene graph library, SGL)), etc.
  • surface manager surface manager
  • media library media libraries
  • 3D graphics processing library for example: open graphics library (open graphics library, OpenGL) embedded systems (embedded systems, ES)
  • 2D graphics engine for example : Scene graph library (scene graph library, SGL)
  • the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc.
  • the media library can support a variety of audio and video encoding formats, such as: moving pictures experts group (MPEG) 4, H.264, MP3, advanced audio coding (AAC), multi-rate adaptive (adaptibve multi rate, AMR), joint photographic experts group (joint photographic experts group, JPG), portable network graphics (portable network graphics, PNG) and so on.
  • MPEG moving pictures experts group
  • AAC advanced audio coding
  • AMR multi-rate adaptive (adaptibve multi rate, AMR)
  • joint photographic experts group joint photographic experts group
  • JPG joint photographic experts group
  • PNG portable network graphics
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc.
  • 2D graphics engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer may include hardware driver modules, such as display drivers, camera drivers, sensor drivers, etc.
  • the application framework layer may call the hardware driver modules of the kernel layer.
  • FIG. 3 is a schematic diagram of a graphical user interface (graphical user interface, GUI) provided by the embodiment of the present application.
  • FIG. 3 shows that in the unlock mode of the electronic device 100, the screen display system of the electronic device 100 displays the interface content currently output 301.
  • the interface content 301 is the main interface of the electronic device 100.
  • the interface content 301 displays a variety of applications (applications, Apps), such as AR, camera, address book, phone, information, clock and other applications. It can be understood that the interface content 301 may also include more or less other application programs, which is not limited in this embodiment of the present application.
  • the user can instruct the electronic device 100 to start the AR application by touching a specific control on the screen of the electronic device 100 , pressing a specific physical key or combination of keys, inputting voice, gestures in the air, and the like.
  • the electronic device 100 starts the camera and the AR application, and displays the AR application interface.
  • the user's instruction to instruct the electronic device 100 to start the AR application may be referred to as the first operation for short.
  • the user can instruct the electronic device 100 to start the AR application by clicking the "AR" application icon on the main interface, and the electronic device 100 displays the application interface as shown in (a) of FIG. 4 .
  • the user when the electronic device 100 is in the locked screen state, the user can also instruct the electronic device 100 to start the AR application by sliding down on the screen of the electronic device 100, and the electronic device 100 displays the AR application as shown in (a) in FIG. 4 .
  • the user when the electronic device 100 is in the lock screen state, the user can instruct the electronic device 100 to start the AR application by clicking the shortcut icon of the "AR" application on the lock screen interface, and the electronic device 100 displays the AR application as shown in (a) of FIG. 4 . application interface.
  • the user can also enable the electronic device 100 to start the AR application by clicking a corresponding control.
  • the user can also instruct the electronic device 100 to start the AR application by selecting the control of the AR function, and the electronic device 100 displays the application interface as shown in (a) of FIG. 4 .
  • the interface of the AR application includes a viewing frame 401 , which can be used to preview images (or frames) captured by the camera.
  • the viewfinder frame 401 shown in (a) of FIG. 4 can be used to preview the viewfinder pictures captured by the camera including V-shaped objects, rectangular objects, black striped backgrounds and white striped backgrounds.
  • the objects in the viewfinder picture are all objects in the real scene.
  • the electronic device 100 can read the established 3D map (hereinafter referred to as the offline 3D map), and the electronic device 100 can also create a 3D map online (hereinafter referred to as the online 3D map).
  • the 3D map may include 3D representation (including information such as point cloud, Mesh, semantics and image feature points).
  • SLAM simultaneous localization and mapping
  • SFM structure from motion
  • the 3D map generating device or electronic device can save the 3D map offline as a file when generating the 3D map, and different saving methods can be adopted for the 3D map of different types of scenes. For example, for family-level scenes (such as a three-dimensional map of a room or a whole house), the corresponding map can be saved as a file and read directly when used; for scenes of shopping malls and scenic spots, since the maps of these scenes are large, you can The corresponding map is divided into small blocks and stored in blocks. When in use, the electronic device 100 can read or load the three-dimensional map in blocks according to its own positioning information (such as GPS or location based service (LBS)).
  • LBS location based service
  • the framing picture may include multiple objects, and the multiple objects may be collectively referred to as a first object, and the target object may be any object among the multiple first objects.
  • the target object may be a V-shaped object or a rectangular object as shown in (a) of FIG. 4 .
  • the framing screen may include editable objects and non-editable objects.
  • the target object can be any editable object.
  • the 3D representation corresponding to the editable object in the 3D map is an editable 3D representation
  • the 3D representation corresponding to the non-editable object in the 3D map is an uneditable 3D representation or there is no corresponding 3D representation in the 3D map.
  • Non-editable objects may include the background in the framing frame and objects that exist in the framing frame but do not exist in the 3D map (that is, objects that exist in the real scene but do not exist in the 3D map, hereinafter referred to as second objects).
  • the V-shaped object and the rectangular object in the viewfinder picture shown in (a) of FIG. 4 are editable objects in the first object
  • the black striped background and white striped background are non-editable objects in the first object.
  • the objects in the viewfinder picture may be inconsistent with the objects in the 3D map.
  • the 3D map of a certain room there is no object on the table in the room, but after the 3D map is established, someone places a water glass on the table.
  • the viewfinder can see that there is a water cup in the room, but there is no water cup in the 3D map, and the water cup is a non-editable object among the first objects in the scene.
  • the electronic device 100 may determine a three-dimensional representation corresponding to the object from the three-dimensional map. For example, the user may select a target object by long pressing an object in the viewfinder screen. After the user selects an object, the electronic device 100 may first select an image block including the target object (hereinafter referred to as the first image block) from the viewfinder screen through an image segmentation technology. , and then determine the three-dimensional representation corresponding to the target object from the three-dimensional representations in the three-dimensional map according to the first image block.
  • the first image block including the target object (hereinafter referred to as the first image block) from the viewfinder screen through an image segmentation technology.
  • the size of the three-dimensional representation corresponding to the target object may be larger than the portion displayed in the image of the target object in the viewfinder frame. For example, there is only half a table in the viewing screen, and after the user clicks to match the 3D representation of the table in the 3D map, the subsequent editing function can be an operation on the 3D representation of the complete table.
  • the electronic device 100 may determine the field of view corresponding to the viewfinder in the three-dimensional map (hereinafter referred to as the third field of view) according to the pose information of the viewfinder, and then from the three-dimensional representation in the third field of view Determine the three-dimensional representation corresponding to the target object.
  • the third field of view the three-dimensional map
  • the electronic device 100 determines the third field of view through relocation and SLAM real-time tracking technology, and then determines the three-dimensional representation corresponding to the target object from the three-dimensional representations in the third field of view. For example, first establish a local coordinate system with the viewfinder picture as the origin through SLAM, and calculate the 6 degrees of freedom (DoF) of the viewfinder picture of the electronic device 100 in the three-dimensional map based on the sensor data collected by the electronic device 100 through relocation technology Pose and the offset value (delta) between the local coordinate system of the viewfinder screen and the coordinate system of the 3D map, align the local coordinate system to the coordinate system of the 3D map through the delta amount, and then determine the third according to the 6Dof pose of the viewfinder screen Field of view, and then perform two-dimensional (2D)-three-dimensional (3D) matching with the first image block through the three-dimensional representation of the third field of view, when most of the feature points in the first image block match the three-dimensional representations belonging to the same three-dimensional When representing, it is
  • the electronic device 100 may determine the field of view (hereinafter referred to as the third field of view) corresponding to the viewfinder in the three-dimensional map according to the pose information of the viewfinder, and then, according to the first image block in the viewfinder, Determine the field of view corresponding to the first image in the three-dimensional map from the third field of view, and then determine the three-dimensional representation corresponding to the target object from the three-dimensional representations in the field of view corresponding to the first image.
  • the third field of view the field of view
  • the electronic device 100 may determine the field of view (hereinafter referred to as the third field of view) corresponding to the viewfinder in the three-dimensional map according to the pose information of the viewfinder, and then, according to the first image block in the viewfinder, Determine the field of view corresponding to the first image in the three-dimensional map from the third field of view, and then determine the three-dimensional representation corresponding to the target object from the three-dimensional representations in the field of view corresponding to the first image.
  • the third field of view
  • the above-mentioned specific method of relocation may be processed by any method conceivable by those skilled in the art, which is not specifically limited in this embodiment of the present application.
  • matching schemes based on bag of word (BoW) such as ORB-SLAM, VINS, etc.
  • methods based on machine learning patch such as PTAM
  • network local feature aggregation description based on deep learning Symbol net vector of locally aggregated descriptors, NetVlad
  • the electronic device 100 may mark the three-dimensional representations in the three-dimensional map as editable three-dimensional representations and non-editable three-dimensional representations.
  • the electronic device 100 can determine whether the object selected by the user is an editable object, and when the object selected by the user is an uneditable object, it can prompt the user to select the object through text information, image information or sound information are non-editable objects. For example, the electronic device 100 may determine whether the object selected by the user is the background through the marking information, and determine whether the object selected by the user is the second object by determining whether there is a 3D representation corresponding to the object selected by the user in the 3D map. If the object is not the background and the second object, it is determined that the object is an editable object; if the object is the background or the second object, it is determined that the object is an uneditable object.
  • the above tag information is used to identify whether the object is the background.
  • Objects in the 3D map can be marked by 3D semantic segmentation/detection techniques. For example, it is possible to perform semantic segmentation/detection on the point cloud or Mesh of objects in a 3D map (such as a SLAM map), discretize and number them to mark which editable objects (such as tables, stools, cups, computers) in the 3D map etc.), which are backgrounds (such as ground, wall, etc.), for subsequent editing and processing.
  • the above-mentioned three-dimensional semantic segmentation/detection can use any one of the existing technologies, which is not limited in the embodiment of the present application.
  • the electronic device 100 may mark the target object in the framing screen so that the user can select the target object.
  • the electronic device 100 may highlight the editable first object in the framing screen when the user does not select the target object, and stop highlighting after the user selects the target object.
  • the electronic device 100 may highlight the editable first object in the framing screen when the user does not select the target object, and stop highlighting after the user selects the target object.
  • the electronic device 100 may mark the target object in the framing screen so that the user can select the target object.
  • the electronic device 100 may highlight the editable first object in the framing screen when the user does not select the target object, and stop highlighting after the user selects the target object.
  • the electronic device 100 may highlight the editable first object in the framing screen when the user does not select the target object, and stop highlighting after the user selects the target object.
  • the image blocks are marked and highlighted to prompt the user which are editable objects that can be selected.
  • the processing can be turned off.
  • the user may instruct to make the target object in the framing screen invisible by touching a specific control on the screen of the electronic device 100 , pressing a specific physical key or key combination, inputting a voice, gestures in the air, and the like.
  • the electronic device 100 After the user performs the operation of making the target object invisible on the electronic device 100 (hereinafter referred to as the second operation), the electronic device 100 performs the third operation in response to the second operation.
  • the third operation includes generating a first mask (mask) in the first area and projecting the first three-dimensional representation in the first field of view to the first mask, the first area is the area where the target object is located in the viewfinder picture, and the first The field of view is the field of view corresponding to the first region in the three-dimensional map, and the first field of view includes a first three-dimensional representation and a second three-dimensional representation, and the second three-dimensional representation is a three-dimensional representation corresponding to the target object.
  • the first three-dimensional representation is a three-dimensional representation other than the first three-dimensional representation among the three-dimensional representations within the first field of view.
  • the electronic device 100 responds to user operations, as shown in (b) in Figure 4, the electronic device 100 first generates a first mask to block the V-shaped object in the area where the V-shaped object is located in the viewfinder screen, and then as shown in (c) in Figure 4 ) shown in the figure, the electronic device 100 displays other three-dimensional representations (i.e. The first three-dimensional representation) is projected onto the above mask. It can be seen from the (c) figure in FIG. 4 that the V-shaped object has disappeared (disappeared) from the viewfinder frame 401. The background (i.e. black and white stripes) behind the V-shaped object.
  • the background i.e. black and white stripes
  • the third operation may also be projecting the second three-dimensional representation to the first mask, then deleting the displayed content on the first mask, and then projecting the first three-dimensional representation to the first mask.
  • the electronic device can display the objects in the real world captured by the camera through the viewfinder screen in the AR application interface, and the user can make the AR application first viewfinder through operations.
  • the area where an object is located in the screen generates a mask to block the object, and then divides the three-dimensional representation corresponding to the object (that is, the second three-dimensional representation ) other three-dimensional representations (that is, the first three-dimensional representation) are projected onto the above mask, so that the objects in the real world where the user is located in the AR application disappear invisible with the user's operation, so that the AR application displays "the object after the invisible disappears "Real world", which increases the interactivity between AR applications and the real world, and improves the user's immersion in AR applications.
  • the user wants to see the background wall behind the cabinet but does not want to move the cabinet.
  • the user can make the cabinet disappear invisible in the AR application through the operation, so as to view the background wall behind the cabinet through the AR application, thereby increasing the interactivity of the AR application in the real world and improving the user immersion of the AR application.
  • the electronic device 100 may package the above “third operation” as a “stealth operation” for other functions or other operations to call.
  • the first mask may not completely coincide with the target object in the framing frame, that is, the first mask may not be able to block the target object in the framing frame.
  • the electronic device 100 may supplementarily generate a supplementary mask fitted with the target object through an image segmentation technique.
  • the electronic device 100 may also delete the display content on the supplementary mask.
  • the electronic device 100 first generates a first mask to block the V-shaped object in the area where the V-shaped object is located in the viewfinder screen. It can be seen from (a) of FIG. 5 that the first mask does not Finish occluding the V-shaped object. As shown in (b) of FIG. 5 , the electronic device 100 supplementarily generates a supplementary mask. From the diagram (b) of FIG. 5 , it can be seen that the first mask and the supplementary mask have completely blocked the V-shaped object. Then, as shown in FIG.
  • the electronic device 100 projects other three-dimensional representations in the field of view corresponding to the area where the V-shaped object is located in the three-dimensional map, except for the three-dimensional representation corresponding to the V-shaped object, to the first mask and the supplementary mask, It can be seen from the diagram (c) in FIG. 5 that the V-shaped object has disappeared from the viewfinder frame 401, and the user can view the background behind the V-shaped object through the diagram (c) in FIG. 5 .
  • the three-dimensional representation of the background part behind the object in the three-dimensional map may be incomplete, and the electronic device 100 may cause invisible objects in the viewfinder after performing the above invisible operation.
  • the background at the rear is incomplete, and the framing screen is not continuous.
  • the user can instruct the electronic device 100 to complete the background operation to complete the missing information in the framing screen by touching a specific control on the screen of the electronic device 100, pressing a specific physical key or key combination, inputting voice, gestures in the air, etc.
  • the background makes the viewfinder continuous.
  • the electronic device 100 may identify whether the background of the framing picture in the framing frame is complete, and when the background is incomplete, perform a background complement operation to complement the missing background in the framing picture to make the framing picture continuous.
  • a supplementary background operation may be added to the above third operation, and the electronic device 100 automatically executes the supplementary background operation when performing the above third operation.
  • the specific method of supplementing the missing background in the framing picture to make the framing picture continuous can be processed by any method conceivable by those skilled in the art, which is not specifically limited in this embodiment of the present application.
  • artificial intelligence (AI) technology may be used to supplement the missing background in the viewfinder image to make the viewfinder image continuous.
  • the electronic device 100 first generates a first mask in the area where the V-shaped object is located in the viewfinder screen to block the V-shaped object. As shown in (a) of FIG. 6 , the electronic device 100 generates the first mask The three-dimensional representation is projected onto the above mask, and it can be seen from Figure 6 (a) that the V-shaped object has been invisible from the viewfinder frame 401, but the lack of background behind the V-shaped object causes the viewfinder image to be discontinuous. At this time, the electronic device 100 may obtain the framing picture shown in (b) of FIG.
  • the third three-dimensional representation is a three-dimensional representation within the second viewing range
  • the second viewing range is a viewing range adjacent to the first viewing range.
  • the user can also instruct the electronic device 100 to move objects in the framing screen by touching a specific control on the screen of the electronic device 100, pressing a specific physical key or key combination, inputting voice, and gestures in the air.
  • the electronic device 100 After the user performs the operation of moving the target object on the electronic device 100 (hereinafter referred to as the fourth operation), the electronic device 100 responds to the fourth operation, firstly performs the third operation to make the target object in the viewfinder invisible, and then generates a target object in the second area.
  • the second mask projects the second three-dimensional representation (that is, the second three-dimensional representation corresponding to the target object) to the second mask.
  • the second area is the area where the target position is located in the viewfinder picture.
  • a user can select a target location by clicking.
  • the user may move an object in the viewfinder image through a drag operation, and the electronic device 100 may determine the end position of the user's drag operation as the target position.
  • the user performs a drag operation to instruct the electronic device 100 to move the V-shaped object from the lower left to the upper left in the framing screen, and the electronic device 100 responds to the user operation, as shown in FIG.
  • the electronic device 100 shown in Figure 7 (b) first performs a third operation on the V-shaped object to make the V-shaped object invisible from the viewfinder frame 401 and generate a second mask on the upper left of the viewfinder frame 401 .
  • the electronic device 100 projects the three-dimensional representation corresponding to the V-shaped object to the second mask. It can be seen from the diagram (c) in FIG. 7 that the V-shaped object has moved from the lower left to the upper left of the viewing frame of the viewing frame 401 .
  • the electronic device can first hide objects in the real world where the user is located in the viewfinder screen in the AR application, and then generate a mask in the user-specified area (ie, the second area) and map the object to The three-dimensional representation of the object is projected onto the mask, so that the object in the real world where the user is located in the AR application moves with the user's operation, so that the AR application displays "the real world after the object moves", thereby further increasing the relationship between the AR application and the real world.
  • the interactivity of the world improves the user immersion of AR applications.
  • the electronic device 100 may also determine the second target in the three-dimensional map according to the target position in the viewfinder position, and then move the second 3D representation to a second target position. For example, the electronic device 100 may first determine the coordinates of the 3D representation corresponding to the target position on the plane of the 3D map according to the target position in the viewfinder screen, and then move the anchor point of the second 3D representation to the coordinates, and then make the second 3D representation The remaining three-dimensional representations of are moved in the same way, so that the second three-dimensional representation is moved to the second target position.
  • the plane of the three-dimensional map is a plane formed by three-dimensional representations in the three-dimensional map.
  • the anchor point may be a point with the smallest Z-axis coordinate in the direction of the vertical line of the center of gravity of the three-dimensional representation.
  • the user can also instruct the electronic device 100 to deform (such as zoom in, zoom out, stretching, etc.).
  • the AR interface can be set with configuration buttons such as zooming in (such as zooming in 5 times), zooming out (such as zooming out 10 times), horizontal stretching (such as getting 2 times fatter), and vertical stretching (such as growing three times taller).
  • the electronic device 100 may be instructed to deform the object in the viewfinder screen by touching the configuration button.
  • the user may deform objects in the viewfinder screen through operations such as pinching and zooming.
  • the electronic device 100 After the user performs the operation of deforming the target object on the electronic device 100 (hereinafter referred to as the fifth operation), the electronic device 100 responds to the fifth operation, first performs the third operation, and then generates a third mask in the third area and converts the fourth three-dimensional Indicates projection to the third mask.
  • the fourth three-dimensional representation is a fourth three-dimensional representation
  • the fourth three-dimensional representation is a deformed second three-dimensional representation.
  • the fourth three-dimensional representation may be the enlarged, reduced or stretched second three-dimensional representation.
  • the user performs a two-finger zoom-in operation to instruct the electronic device 100 to zoom in on the rectangular object in the framing screen of the preview frame 401, and the electronic device 100 responds to the user operation, as shown in FIG. 8 ( b)
  • the electronic device 100 shown in the figure first performs a third operation on the rectangular object to make the rectangular object invisible from the viewfinder frame of the viewfinder frame 401 and generate a third mask in the viewfinder frame of the viewfinder frame 401, and then the electronic device 100
  • the corresponding three-dimensional representation of the object ie, the second three-dimensional representation
  • the electronic device 100 projects the fourth three-dimensional representation to the third mask as shown in (c) of FIG. 8 . It can be seen from the diagram (c) in FIG. 8 that the rectangular object in the viewfinder frame of the preview frame 401 has been enlarged.
  • the user can also deform the object in the real world where the user is located by operating in the AR application, so that the AR application displays "the deformed real world of the object", thereby further It increases the interactivity between AR applications and the real world, and improves the user immersion of AR applications.
  • the anchor points of the second three-dimensional representation and the fourth three-dimensional representation may be the same.
  • the electronic device 100 may generate the fourth three-dimensional representation according to the deformation information and the three-dimensional coordinates of the second three-dimensional representation.
  • the deformation information is used to characterize the deformation type and deformation magnification.
  • the user instructs the electronic device 100 to magnify the object in the viewfinder screen by 5 times by touching the corresponding configuration button
  • the electronic device 100 responds to the user operation and determines the deformation type as zoom in according to the user operation, and the deformation magnification is 5 times, and then uses
  • the anchor point of the second three-dimensional representation is used as the zoom origin, calculate the coordinate difference between the coordinates of other points (three-dimensional representation) in the three-dimensional coordinate system relative to the anchor point, and then multiply the coordinate difference of each point by 5 and add
  • the coordinates of the upper anchor point can obtain a new three-dimensional representation enlarged by 5 times (ie, the fourth three-dimensional representation).
  • the method includes:
  • the electronic device 100 starts a camera and displays an application interface in response to a first operation.
  • the application interface includes a viewfinder frame, and the viewfinder frame of the viewfinder includes a target object.
  • the viewfinder frame of the viewfinder may include multiple objects, and the target object may be any one of the multiple objects.
  • editable objects and non-editable objects may be included in the viewfinder frame of the viewfinder.
  • the first object can be any object in the editable objects.
  • the 3D representation corresponding to the editable object in the 3D map is an editable 3D representation
  • the 3D representation corresponding to the non-editable object in the 3D map is an uneditable 3D representation or there is no corresponding 3D representation in the 3D map.
  • the electronic device 100 may mark the three-dimensional representations in the three-dimensional map as editable three-dimensional representations and non-editable three-dimensional representations.
  • the electronic device 100 performs a third operation in response to the second operation, where the third operation includes generating a first mask in the first area and projecting the first three-dimensional representation within the first field of view to the first mask.
  • the first area is the area where the target object is located in the viewfinder picture
  • the first field of view is the field of view corresponding to the first area in the three-dimensional map
  • the first field of view includes the first three-dimensional representation and the second three-dimensional representation
  • the second three-dimensional representation is a second three-dimensional representation
  • the second three-dimensional representation is a three-dimensional representation corresponding to the target object.
  • the first three-dimensional representation is a three-dimensional representation other than the first three-dimensional representation among the three-dimensional representations within the first field of view.
  • the third operation may also include filling the area in the first mask where the 3D representation is not projected according to the third 3D representation, the third 3D representation is a 3D representation within the second field of view, and the second field of view is the view range adjacent to the first view range.
  • the electronic device 100 may also determine the second three-dimensional representation from the three-dimensional representations within the first field of view according to the first image block.
  • the first image block is an image block including the target object in the viewfinder picture.
  • the method may also include:
  • the electronic device performs a third operation to generate a second mask in the second area and project the second three-dimensional representation to the second mask.
  • the electronic device performs a third operation to generate a third mask in the third area and project the fourth three-dimensional representation to the third mask.
  • the fourth three-dimensional representation is a fourth three-dimensional representation
  • the fourth three-dimensional representation is a deformed second three-dimensional representation
  • the fourth three-dimensional representation is the enlarged, reduced or stretched second three-dimensional representation.
  • the fourth three-dimensional representation may be the second three-dimensional representation enlarged by 5 times.
  • the electronic device 100 may also generate a fourth three-dimensional representation according to deformation information and three-dimensional coordinates of the second three-dimensional representation, where the deformation information is used to represent a deformation type and a deformation magnification.
  • the anchor points of the second three-dimensional representation and the fourth three-dimensional representation are the same.
  • the anchor point may be a point in the three-dimensional representation that is located in a direction perpendicular to the center of gravity of the three-dimensional representation with the smallest Z-axis coordinate.
  • the electronic device 100 also determines the third field of view according to the pose information of the framing picture, and according to the position of the area (such as the first area, the second area, or the third area, etc.) in the framing picture Determines the field of view of the area within the third field of view.
  • the third field of view is the field of view corresponding to the viewfinder picture in the three-dimensional map.
  • the above pose information may be a 6Dof pose.
  • the embodiment of the present application also provides another display method, which can make objects that exist in the real scene but do not exist in the three-dimensional map disappear from the viewfinder screen of the electronic device, as shown in Figure 10, the method includes:
  • the electronic device 100 starts a camera and displays an application interface in response to a first operation.
  • the application interface includes a viewfinder frame, and the viewfinder frame of the viewfinder includes a target object.
  • the viewfinder frame of the viewfinder may include multiple objects, and the target object may be any one of the multiple objects.
  • the electronic device 100 performs a third operation in response to the second operation.
  • the third operation includes generating a first mask in the first area and projecting a three-dimensional representation within the first field of view to the first mask.
  • the first area is the area where the target object is located in the viewfinder picture
  • the first field of view is the field of view corresponding to the first area in the 3D map
  • the 3D representation includes at least one of point cloud, Mesh, or semantics.
  • the electronic device 100 may perform the third operation above when the first three-dimensional representation does not exist within the first field of view.
  • the first three-dimensional representation may be the above-mentioned three-dimensional representation of the target object.
  • a 3D map stores a 3D representation of a building.
  • the user wants to view the building through the viewing frame of the application interface, but there are "pedestrians" beside the building.
  • the user wants to make the "pedestrian” in the viewfinder disappear from the viewfinder.
  • the user can select the "pedestrian” as the target object through the operation application, so that the electronic device performs the third operation, and the location of the "pedestrian” in the three-dimensional map is displayed.
  • the three-dimensional representation corresponding to the area is projected to the area where the "pedestrian" is located in the viewfinder screen.
  • the method may further include: the electronic device 100 determines whether the first three-dimensional representation exists from the three-dimensional representations within the first field of view according to the first image block.
  • the first image block is an image block including the target object in the viewfinder picture.
  • the first image block can be matched to the corresponding three-dimensional representation within the first field of view, it means that there is a three-dimensional representation of the target object within the first field of view; If the corresponding three-dimensional representation can be matched, it means that there is no three-dimensional representation of the target object within the first field of view.
  • the electronic device includes hardware and/or software modules corresponding to each function.
  • the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.
  • the embodiment of the present application may divide the electronic device into functional modules according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module.
  • the above integrated modules may be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division methods in actual implementation.
  • FIG. 11 shows a possible composition diagram of the electronic device involved in the above embodiment.
  • the apparatus 1100 may include: a transceiver unit 1101 and A processing unit 1102, where the processing unit 1102 may implement the method executed by the electronic device in the foregoing method embodiments, and/or other processes used in the technologies described herein.
  • the apparatus 1100 may include a processing unit, a storage unit and a communication unit.
  • the processing unit may be used to control and manage the actions of the device 1100, for example, may be used to support the device 1100 to execute the steps performed by the above-mentioned units.
  • the storage unit may be used to support the device 1100 to execute stored program codes, and/or data, and the like.
  • the communication unit may be used to support communication between the apparatus 1100 and other devices.
  • the processing unit may be a processor or a controller. It can implement or execute the various illustrative logical blocks, modules and circuits described in connection with the present disclosure.
  • the processor can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (digital signal processing, DSP) and a microprocessor, and the like.
  • the storage unit may be a memory.
  • the communication unit may be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, and a Wi-Fi chip.
  • the electronic device involved in this embodiment of the present application may be an apparatus 1200 having the structure shown in FIG. 12 , where the apparatus 1200 includes a processor 1201 and a transceiver 1202 .
  • the transceiver unit 1101 and the processing unit 1102 in FIG. 11 may be implemented by the processor 1201 .
  • the apparatus 1200 may further include a memory 1203, and the processor 1201 and the memory 1203 communicate with each other through an internal connection path.
  • the relevant functions implemented by the storage unit in FIG. 11 may be implemented by the memory 1203 .
  • the embodiment of the present application also provides a computer storage medium, the computer storage medium stores computer instructions, and when the computer instructions are run on the electronic device, the electronic device executes the steps of the above-mentioned related methods to implement the display method in the above-mentioned embodiments.
  • An embodiment of the present application also provides a computer program product, which, when running on a computer, causes the computer to execute the above-mentioned related steps, so as to realize the display method in the above-mentioned embodiment.
  • the embodiment of the present application also provides an electronic device, and this device may specifically be a chip, an integrated circuit, a component or a module.
  • the device may include a connected processor and a memory for storing instructions, or the device may include at least one processor for fetching instructions from an external memory.
  • the processor can execute instructions, so that the chip executes the display methods in the above method embodiments.
  • FIG. 13 shows a schematic structural diagram of a chip 1300 .
  • the chip 1300 includes one or more processors 1301 and an interface circuit 1302 .
  • the above-mentioned chip 1300 may further include a bus 1303 .
  • the processor 1301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above display method may be completed by an integrated logic circuit of hardware in the processor 1301 or instructions in the form of software.
  • the above-mentioned processor 1301 may be a general-purpose processor, a digital signal processing (digital signal processing, DSP) device, an integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array) , FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP digital signal processing
  • ASIC application specific integrated circuit
  • FPGA field-programmable gate array
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the interface circuit 1302 can be used for sending or receiving data, instructions or information.
  • the processor 1301 can use the data, instructions or other information received by the interface circuit 1302 to process, and can send the processing completion information through the interface circuit 1302 .
  • the chip further includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor.
  • a portion of the memory may also include non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the memory stores executable software modules or data structures
  • the processor can execute corresponding operations by calling operation instructions stored in the memory (the operation instructions can be stored in the operating system).
  • the chip may be used in the electronic device or DOP involved in the embodiment of the present application.
  • the interface circuit 1302 may be used to output an execution result of the processor 1301 .
  • processor 1301 and the interface circuit 1302 can be realized by hardware design, software design, or a combination of software and hardware, which is not limited here.
  • the electronic device, computer storage medium, computer program product or chip provided in this embodiment is all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the corresponding method provided above The beneficial effects in the method will not be repeated here.
  • sequence numbers of the above-mentioned processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not be used in the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the disclosed systems, devices and methods may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the above units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be combined or can be Integrate into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the above-mentioned methods in various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other various media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请公开了显示方法及电子设备，涉及媒体技术领域，能够提升AR应用的交互性和沉浸感。该方法包括：首先电子设备响应于第一操作，启动摄像头并显示应用界面。然后电子设备响应于第二操作，执行第三操作。其中，应用界面包括取景框，取景框的取景画面中包括目标物体，第三操作包括在第一区域生成第一mask并将第一视野范围内的第一三维表示投影至第一mask，第一区域为取景画面中目标物体所在区域，第一视野范围为三维地图中第一区域对应的视野范围，第一视野范围内包括第一三维表示和第二三维表示，第二三维表示为目标物体的三维表示，三维表示包括点云、Mesh或语义中的至少一项。

Description

显示方法及电子设备
本申请要求于2021年11月01日提交中国专利局、申请号为202111283430.5、申请名称为“显示方法及电子设备”的中国专利申请的优先权，以及于2022年03月07日提交中国专利局、申请号为202210222388.4、申请名称为“显示方法及电子设备”的中国专利申请的优先权，它们的全部内容通过引用结合在本申请中。
技术领域
本申请涉及媒体技术领域，尤其涉及显示方法及电子设备。
背景技术
增强现实(augmented reality，AR)技术是一种将虚拟信息与真实世界巧妙融合的技术，广泛运用了多媒体、三维建模、实时跟踪及注册、智能交互、传感等多种技术手段，将计算机生成的文字、图像、三维模型、音乐、视频等虚拟信息模拟仿真后，应用到真实世界中，两种信息互为补充，从而实现对真实世界的“增强”。
然而，现有AR应用功能较为单调，缺乏交互性和沉浸感，用户体验较差。
发明内容
本申请提供了显示方法及电子设备，能够提升AR应用的交互性和沉浸感，以提高用户使用AR应用时的用户体验。为达到上述目的，本申请采用如下技术方案：
第一方面，本申请提供了一种显示方法，应用于包含摄像头的电子设备，该方法包括：
首先电子设备响应于第一操作，启动摄像头并显示应用界面。然后电子设备响应于第二操作，执行第三操作。其中，应用界面包括取景框，取景框的取景画面中包括目标物体，第三操作包括在第一区域生成第一掩膜(mask)并将第一视野范围内的第一三维表示投影至第一mask，第一区域为取景画面中目标物体所在区域，第一视野范围为三维地图中第一区域对应的视野范围，第一视野范围内包括第一三维表示和第二三维表示，第二三维表示为目标物体的三维表示。第一三维表示为第一视野范围内的三维表示中除第二三维表示之外的其他三维表示，上述三维表示包括点云、网格(Mesh)或语义中的至少一项。
可以看出，本申请实施例提供的显示方法中，电子设备通过AR应用界面内的取景画面可以显示摄像头所采集到的用户所处现实世界内的物体，用户可以通过操作使AR应用先在取景画面内的某个物体所在区域生成掩膜遮挡该物体，然后通过将三维地图中该物体所在区域对应的视野范围(即第一视野范围)内除该物体对应的虚拟三维物体(即第二三维表示)外的其他三维表示(即第一三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
例如，用户所处房间内存在一个柜子，用户想要看到柜子后方的背景墙但不想搬动柜子。这时用户就可以通过操作使柜子在AR应用中隐形消失以通过AR应用查看柜子后方 的背景墙，由此增加了AR应用于现实世界的交互性，提升了AR应用的用户沉浸感。
可选地，取景框的取景画面中可以包括多个物体，目标物体可以为所述多个物体中的任一物体。
可以看出，本申请实施例提供的显示方法中，电子设备通过AR应用界面内的取景画面可以显示摄像头所采集到的用户所处现实世界内的多个物体，用户可以先从这多个物体中选择一个物体作为目标物体，然后通过操作使AR应用先在取景画面内的目标物体所在区域生成掩膜遮挡该物体，然后通过将三维地图中该物体所在区域对应的视野范围(即第一视野范围)内除该物体对应的虚拟三维物体(即第二三维表示)外的其他三维表示(即第一三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
可选地，取景框的取景画面中可以包括可编辑物体和不可编辑物体，目标物体为任一可编辑物体。
可以看出，本申请实施例提供的显示方法中，电子设备通过AR应用界面内的取景画面可以显示摄像头所采集到的用户所处现实世界内的多个物体，用户可以先从这多个物体中选择一个可编辑物体作为目标物体，然后通过操作使AR应用先在取景画面内的目标物体所在区域生成掩膜遮挡该物体，然后通过将三维地图中该物体所在区域对应的视野范围(即第一视野范围)内除该物体对应的虚拟三维物体(即第二三维表示)外的其他三维表示(即第一三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
在一种可能的实现方式中，该方法还可以包括：将三维地图中的三维表示标记为可编辑三维表示和不可编辑三维表示。
可以看出，本申请实施例提供的显示方法，可以将三维地图中的三维表示标记为可编辑三维表示和不可编辑三维表示。电子设备通过AR应用界面内的取景画面可以显示摄像头所采集到的用户所处现实世界内的多个物体，电子设备可以通过物体对应的三维标记确定物体是否可编辑物体，当用户选择的物体为可编辑物体，将该物体作为目标物体，然后通过操作使AR应用先在取景画面内的目标物体所在区域生成掩膜遮挡该物体，然后通过将三维地图中该物体所在区域对应的视野范围(即第一视野范围)内除该物体对应的虚拟三维物体(即第二三维表示)外的其他三维表示(即第一三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
在一种可能的实现方式中，第三操作还可以包括根据第三三维表示填充第一mask中未投影三维表示的区域。其中，第三三维表示为第二视野范围内的三维表示，第二视野范围是与第一视野范围相邻的视野范围。
可以理解的是，由于物体之间的遮挡关系，三维地图中物体后方的背景部分的三维表示可能是不完整的，电子设备执行上述第三操作后可能会造成取景画面中隐形物品后方的背景不完整，取景画面不连续。对于上述情况可以根据第三三维表示填充第一mask中未 投影三维表示的区域以补全取景画面中缺失的背景使取景画面连续，从而提升AR应用的用户体验。
在一种可能的实现方式中，该方法还可以包括：电子设备响应于第四操作，执行第三操作，在第二区域生成第二mask并将第二三维表示投影至第二mask。
可以看出，电子设备还可以响应于用户操作，先将AR应用中取景画面中用户所处现实世界内的物体隐形，然后在用户指定区域(即第二区域)生成掩膜并将该物体对应的三维表示投影至该掩膜，以使得AR应用中用户所处现实世界内的物体随用户操作而移动，使AR应用显示“该物体移动后的现实世界”，从而进一步增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感
在一种可能的实现方式中，该方法还可以包括：电子设备响应于第五操作，执行第三操作，在第三区域生成第三mask并将第四三维表示投影至第三mask，第四三维表示为形变后的第二三维表示。
可以看出，电子设备还可以响应于用户操作，先将AR应用中取景画面中用户所处现实世界内的物体隐形，然后在取景画面内生成掩膜并将形变后的第二三维表示投影至该掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而形变，使AR应用显示“该物体形变后的现实世界”，从而进一步增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
在一种可能的实现方式中，该方法还可以包括：根据形变信息和第二三维表示的三维坐标生成第四三维表示，形变信息用于表征形变类型和形变倍率。
可以看出，电子设备还可以响应于用户操作，先将AR应用中取景画面中用户所处现实世界内的物体隐形，之后根据形变信息和第二三维表示的三维坐标生成第四三维表示(即形变后的第二三维表示)，然后在取景画面内生成掩膜并将形变后的第二三维表示投影至该掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而形变，使AR应用显示“该物体形变后的现实世界”，从而进一步增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
可选地，第四三维表示可以为放大、缩小或拉伸后第二三维表示。例如，第四三维表示可以为放大5倍后的第二三维表示。
可以看出，电子设备还可以响应于用户操作，先将AR应用中取景画面中用户所处现实世界内的物体隐形，然后在取景画面内生成掩膜并将放大、缩小或拉伸后的第二三维表示投影至该掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而放大、缩小或拉伸，使AR应用显示“该物体放大、缩小或拉伸后的现实世界”，从而进一步增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
可选地，第二三维表示和第四三维表示的锚点可以相同。其中，锚点可以为位于三维表示的重心垂线方向上Z轴坐标最小的点。
在一种可能的实现方式中，该方法还可以包括：根据取景画面的位姿信息确定第三视野范围，根据区域(如第一区域、第二区域或第三区域等)在取景画面中的位置确定区域在第三视野范围内的视野范围。其中，第三视野范围为三维地图中取景画面对应的视野范围。
可以看出，本申请实施例提供的显示方法中，电子设备通过AR应用界面内的取景画 面可以显示摄像头所采集到的用户所处现实世界内的物体，用户可以通过操作使AR应用先在取景画面内的某个物体所在区域生成掩膜遮挡该物体，之后通过取景画面的位姿信息确定三维地图中该物体所在区域对应的视野范围(即第一视野范围)，然后通过将三维地图中该物体所在区域对应的视野范围内除该物体对应的虚拟三维物体(即第二三维表示)外的其他三维表示(即第一三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
在一种可能的实现方式中，该方法还可以包括：根据第一图像块，从第一视野范围内的三维表示中确定第二三维表示。其中，第一图像块为取景画面中包含目标物体的图像块。
可以看出，本申请实施例提供的显示方法中，电子设备通过AR应用界面内的取景画面可以显示摄像头所采集到的用户所处现实世界内的物体，用户可以通过操作使AR应用先在取景画面内的某个物体所在区域生成掩膜遮挡该物体，然后通过包含该物体的图像块从三维地图中该物体所在区域对应的视野范围(即第一视野范围)确定该物体对应的三维表示并将三维地图中该物体所在区域对应的视野范围内除该物体对应的虚拟三维物体外的其他三维表示(即第一三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
第二方面，本申请还提供了另一种显示方法，应用于包含摄像头的电子设备，该方法包括：首先电子设备响应于第一操作，启动摄像头并显示应用界面。然后电子设备响应于第二操作，执行第三操作。其中，应用界面包括取景框，取景框的取景画面中包括目标物体，第三操作包括在第一区域生成mask并将第一视野范围内的三维表示投影至第一mask，第一区域为取景画面中目标物体所在区域，第一视野范围为三维地图中第一区域对应的视野范围，三维表示包括点云、Mesh或语义中的至少一项。
可以看出，本申请实施例提供的显示方法中，电子设备通过AR应用界面内的取景画面可以显示摄像头所采集到的用户所处现实世界内的物体，用户可以通过操作使AR应用先在取景画面内的某个物体所在区域生成掩膜遮挡该物体，然后通过将三维地图中该物体所在区域对应的视野范围(即第一视野范围)内的虚拟三维物体(即三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
例如，用户在通过AR应用拍摄某一物体时，取景画面中出现了行人。用户则可以通过操作电子设备执行上述显示方法，使行人从取景画面中消失，使取景画面显示“该行人隐形消失后的现实世界”。
在一种可能的实现方式中，电子设备可以在第一视野范围内不存在第一三维表示的情况下，执行上述第三操作。其中，上述第一三维表示为上述目标物体的三维表示。
可以理解的是，第一视野范围内不存在第一三维表示说明目标物体的三维表示并未存储在三维地图中，即三维地图中存储的场景为不存在目标物体的场景。因此可以直接将三维地图中目标物体所在区域的三维表示直接投影至取景画面中以使得目标物体从取景画面中隐形消失。
在一种可能的实现方式中，该方法还包括：根据第一图像块，从所述第一视野范围内的三维表示中确定是否存在第一三维表示，所述第一图像块为所述取景画面中包含所述目标物体的图像块，所述第一三维表示为所述目标物体的三维表示。
可以理解的是，当第一图像块在第一视野范围内能够匹配到对应的三维表示，则说明第一视野范围内存在目标物体的三维表示；当第一图像块在第一视野范围内不能够匹配到对应的三维表示则说明第一视野范围内不存在目标物体的三维表示。
第三方面，本申请还提供了一种电子设备，包括：至少一个处理器、存储器和至少一个摄像头，存储器、摄像头与处理器耦合，存储器用于存储计算机程序代码，计算机程序代码包括计算机指令，当处理器从存储器中读取计算机指令，以使得电子设备执行如下操作：响应于第一操作，启动摄像头并显示应用界面，应用界面包括取景框，取景框的取景画面中包括目标物体；响应于第二操作，执行第三操作，第三操作包括在第一区域生成第一掩膜mask并将第一视野范围内的第一三维表示投影至第一mask，第一区域为取景画面中目标物体所在区域，第一视野范围为三维地图中第一区域对应的视野范围，第一视野范围内包括第一三维表示和第二三维表示，第二三维表示为目标物体的三维表示。第一三维表示为第一视野范围内的三维表示中除第一三维表示之外的其他三维表示，上述三维表示包括点云、Mesh或语义中的至少一项。
可选地，取景框的取景画面中可以包括多个物体，目标物体可以为所述多个物体中的任一物体。
可选地，取景框的取景画面中可以包括可编辑物体和不可编辑物体，目标物体为任一可编辑物体。
在一种可能的实现方式中，处理器还用于将三维地图中的三维表示标记为可编辑三维表示和不可编辑三维表示。
在一种可能的实现方式中，第三操作还可以包括根据第三三维表示填充第一mask中未投影三维表示的区域。其中，第三三维表示为第二视野范围内的三维表示，第二视野范围是与第一视野范围相邻的视野范围。
在一种可能的实现方式中，处理器还用于使得电子设备执行如下操作：响应于第四操作，执行第三操作，在第二区域生成第二mask并将第二三维表示投影至第二mask。
在一种可能的实现方式中，处理器还用于使得电子设备执行如下操作：响应于第五操作，执行第三操作，在第三区域生成第三mask并将第四三维表示投影至第三mask，第四三维表示为第四三维表示，第四三维表示为形变后的第二三维表示。
在一种可能的实现方式中，处理器还用于使得电子设备执行如下操作：根据形变信息和第二三维表示的三维坐标生成第四三维表示，形变信息用于表征形变类型和形变倍率。
可选地，第四三维表示可以为放大、缩小或拉伸后第二三维表示。例如，第四三维表示可以为放大5倍后的第二三维表示。
可选地，第二三维表示和第四三维表示的锚点可以相同。其中，锚点可以为位于三维表示的重心垂线方向上Z轴坐标最小的点。
在一种可能的实现方式中，处理器还用于使得电子设备执行如下操作：根据取景画面的位姿信息确定第三视野范围，根据区域(如第一区域、第二区域或第三区域等)在取景画面中的位置确定区域在第三视野范围内的视野范围。其中，第三视野范围为三维地图中 取景画面对应的视野范围。
在一种可能的实现方式中，处理器还用于使得电子设备执行如下操作：根据第一图像块，从第一视野范围内的三维表示中确定第二三维表示。其中，第一图像块为取景画面中包含目标物体的图像块。
第四方面，本申请还提供一种芯片，包括：输入接口、输出接口、至少一个处理器。可选地，该芯片还包括存储器。该至少一个处理器用于执行该存储器中的代码，当该至少一个处理器执行该代码时，该芯片实现上述第一方面或其任意可能的实现方式中所述的方法。
可选地，上述芯片还可以为集成电路。
第五方面，本申请还提供一种终端，该终端包括上述电子设备或上述芯片。
第六方面，本申请还提供一种计算机可读存储介质，用于存储计算机程序，该计算机程序包括用于实现上述第一方面或其任意可能的实现方式中所述的方法。
第七方面，本申请还提供一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机实现上述第一方面或其任意可能的实现方式中所述的方法。
本实施例提供的电子设备、计算机存储介质、计算机程序产品和芯片均用于执行上文所提供的显示方法，因此，其所能达到的有益效果可参考上文所提供的显示方法中的有益效果，此处不再赘述。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单的介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请实施例提供的一种电子设备的结构示意图；
图2为本申请实施例提供的一种电子设备的软件结构示意图；
图3为本申请实施例提供的一种电子设备的用户界面示意图；
图4为本申请实施例提供的另一种电子设备的用户界面示意图；
图5为本申请实施例提供的又一种电子设备的用户界面示意图；
图6为本申请实施例提供的又一种电子设备的用户界面示意图；
图7为本申请实施例提供的又一种电子设备的用户界面示意图；
图8为本申请实施例提供的又一种电子设备的用户界面示意图；
图9为本申请实施例提供的一种显示方法的流程示意图；
图10为本申请实施例提供的另一种显示方法的流程示意图；
图11为本申请实施例提供的一种装置的结构示意图；
图12为本申请实施例提供的另一种装置的结构示意图；
图13为本申请实施例提供的一种芯片的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地 描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。
本申请的说明书以及附图中的术语“第一”和“第二”等是用于区别不同的对象，或者用于区别对同一对象的不同处理，而不是用于描述对象的特定顺序。
此外，本申请的描述中所提到的术语“包括”和“具有”以及它们的任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元，而是可选的还包括其他没有列出的步骤或单元，或可选的还包括对于这些过程、方法、产品或设备固有的其他步骤或单元。
需要说明的是，本申请实施例的描述中，“示例性地”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性地”或者“例如”的任何实施例或设计方案不应被解释为比其他实施例或设计方案更优选或更具优势。确切而言，使用“示例性地”或者“例如”等词旨在以具体方式呈现相关概念。
在本申请的描述中，除非另有说明，“多个”的含义是指两个或两个以上。
本申请实施例提供的显示方法可以应用于手机、平板电脑、可穿戴设备、车载设备、AR设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)等可以实现AR功能的电子设备上，本申请实施例对电子设备的具体类型不作任何限制。
示例性地，图1是本申请实施例提供的一例电子设备100的结构示意图。电子设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。
可以理解的是，本申请实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。例如，在一种可能的实现方式中，电子设备100还可以包括视觉传感器、惯性测量单元(inertial measurement unit，IMU)、激光雷达(light detection and ranging，LIDAR)、磁力计等部件。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit， GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
其中，控制器可以是电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
其中，I2C接口是一种双向同步串行总线，处理器110可以通过I2C接口耦合触摸传感器180K，使处理器110与触摸传感器180K通过I2C总线接口通信，实现电子设备100的触摸功能。MIPI接口可以被用于连接处理器110与显示屏194，摄像头193等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现电子设备100的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现电子设备100的显示功能。
可以理解的是，本申请实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备100的结构限定。在本申请另一些实施例中，电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，外部存储器，显示屏194，摄像头193，和无线通信模块160等供电。
电子设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备100可以包括1个或N个显示屏194，N为大于1的正整数。
电子设备100可以通过ISP，摄像头193，触摸传感器、视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。
其中，ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号，应理解，在本申请实施例的描述中，以RGB格式的图像为例进行介绍，本申请实施例对图像格式不做限定。在一些实施例中，电子设备100可以包括1个或N个摄像头193，N为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样，电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令，从而执行电子设备100的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。
电子设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。陀螺仪传感器180B可以用于确定电子设备100的运动姿态。气压传感器180C用于测量气压。磁传感器180D包括霍尔传感器。电子设备100可以利用磁传感器180D检测翻盖皮套的开合。加速度传感器180E可检测电子设备100在各个方向上(一般为三轴)加速度的大小。距离传感器180F，用于测量距离。电子设备100可以通过红外或激光测量距离。在一些实施例中，拍摄场景，电子设备100可以利用距离传感器180F测距以实现快速对焦。接近光传感器180G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。环境光传感器180L用于感知环境光亮度。电子设备100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。指纹传感器180H用于采集指纹。电子设备100可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。温度传感器180J用于检测温度。触摸传感器180K，也称“触控面板”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。骨传导传感器180M可以获取振动信号。音频模块170可以基于所述骨传导传感器180M获取 的声部振动骨块的振动信号，解析出语音信号，实现语音功能。
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。电子设备100可以接收按键输入，产生与电子设备100的用户设置以及功能控制有关的键信号输入。马达191可以产生振动提示。马达191可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。作用于显示屏194不同区域的触摸操作，马达191也可对应不同的振动反馈效果。指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。SIM卡接口195用于连接SIM卡。
电子设备100的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本申请实施例以一种分层架构的操作系统为例，示例性说明电子设备100的软件结构。
图2是本申请实施例的电子设备100的软件结构框图。分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将该操作系统分为四层，从上至下分别为应用程序层，应用程序框架层，操作系统运行时(runtime)和系统库，以及内核层。
应用程序层可以包括一系列应用程序包。如图2所示，应用程序包可以包括相机、相册、音乐、设置等应用程序。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。如图2所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，资源管理器，通知管理器等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，状态栏中显示通知信息可以短暂停留后自动消失，例如用于告知用户下载完成的消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，或者通知管理器还可以发出提示音，例如电子设备振动，指示灯闪烁等。
Runtime包括核心库和虚拟机。Runtime负责该操作系统的调度和管理。核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是该操作系统的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(media libraries)，三维图形处理库(例如：开放图形库(open graphics library，OpenGL)嵌入式系统(embedded systems，ES))，2D图形引擎(例如：场景图库(scene graph library，SGL))等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如：动态图像专家组(moving pictures experts group，MPEG)4，H.264，MP3，高级音频编码(advanced audio coding，AAC)，多速率自适应(adaptibve multi rate，AMR)，图像专家联合小组(joint photographic experts group，JPG)，便携式网络图形(portable network graphics，PNG)等。
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。2D图形引擎是2D绘图的绘图引擎。
内核层是硬件和软件之间的层。其中，内核层可以包含硬件驱动模块，例如显示驱动，摄像头驱动、传感器驱动等，应用程序框架层可以调用内核层的硬件驱动模块。
为了便于理解，本申请以下实施例将以具有图1和图2所示结构的电子设备为例，介绍本申请实施例提供的显示方法。
以下实施例中所涉及的技术方案均可以在具有上述硬件架构和软件架构的电子设备100中实现。
下面结合附图对本申请实施例提供的技术方案进行详细说明。
图3是本申请实施例提供的一种图形用户界面(graphical user interface，GUI)示意图，图3示出了电子设备100的解锁模式下，电子设备100的屏幕显示系统显示了当前输出的界面内容301，该界面内容301为电子设备100的主界面。该界面内容301显示了多款应用程序(application，App)，例如AR、相机、通讯录、电话、信息、时钟等应用程序。可以理解的是，界面内容301还可以包括其他更多或更少的应用程序，本申请实施例对此不作限定。
用户可以通过触摸电子设备100屏幕上特定的控件、按压特定的物理按键或按键组合、输入语音、隔空手势等方式，指示电子设备100开启AR应用。响应于接收到用户开启AR应用的指示后，电子设备100启动摄像头和AR应用，显示AR应用界面。其中，用户指示电子设备100开启AR应用的指示可简称为第一操作。
示例性地，如图3所示，用户可以通过在主界面上点击“AR”应用图标，指示电子设备100开启AR应用，电子设备100显示如图4中(a)图所示的应用界面。
又示例性地，在电子设备100处于锁屏状态时，用户也可以通过在电子设备100屏幕上向下滑动的手势，指示电子设备100开启AR应用，电子设备100显示如图4中(a)图所示的应用界面。或者，电子设备100处于锁屏状态时，用户可以通过在锁屏界面上点击“AR”应用的快捷图标，指示电子设备100开启AR应用，电子设备100显示如图4中(a)图所示的应用界面。
又示例性地，在电子设备100运行其他应用时，用户也可以通过点击相应的控件使得电子设备100开启AR应用。例如，用户正在使用即时通信类应用时，用户也可以通过选 择AR功能的控件，指示电子设备100开启AR应用，电子设备100显示如图4中(a)图所示的应用界面。
如图4中(a)图所示，AR应用的界面包括取景框401，取景框401可用于预览摄像头采集到的图像(或画面)。如图4中(a)图所示的取景框401可用于预览摄像头采集到的包括V形物体、矩形物体、黑色条纹背景和白色条纹背景的取景画面。此时，取景画面中的物体均为真实场景中的物体。
在开启AR应用后，电子设备100可以读取已建立好的三维地图(以下简称离线三维地图)，电子设备100也可以在线建立三维地图(以下简称在线三维地图)。其中，三维地图中可以包括三维表示(包括点云、Mesh、语义和图像特征点等信息)。
上述建立三维地图的具体方法可以采用本领域技术人员能够想到的任何一种方法进行处理，本申请实施例对此不作具体限定。例如，可以使用即时定位与地图构建(simultaneous localization and mapping，SLAM)或运动恢复结构(structure from motion，SFM)等算法通过视觉传感器、IMU、LIDAR、磁力计、全球定位系统(global positioning system，GPS)等传感器采集的数据建立三维地图。
三维地图生成设备或电子设备在生成三维地图可以将其离线保存为文件，对于不同类型场景的三维地图可以采用不同的保存方式。例如，对于家庭级的场景(如房间或全屋的三维地图)，可以将对应的地图保存成一个文件，使用时直接读取；对于商场、景区的场景，由于这些场景的地图较大，可以将对应的地图分成小块，分块保存，使用时，电子设备100可以根据自身的定位信息(如GPS或基于位置的服务(location based service，LBS))，分块读取或加载三维地图。
用户可以通过触摸电子设备100屏幕上特定的控件、按压特定的物理按键或按键组合、输入语音、隔空手势等方式选择目标物体和指示电子设备100对取景画面中的目标物体进行编辑(如隐形、移动和形变等)。其中，取景画面中可以包括多个物体，该多个物体可以统称为第一物体，目标物体可以为多个第一物体中的任一物体。例如，目标物体可以为图4中(a)图所示的V形物体或矩形物体。
可选地，取景画面可以包括可编辑物体和不可编辑物体。目标物体可以为任一可编辑物体。其中，可编辑物体在三维地图中对应的三维表示为可编辑三维表示，不可编辑物体在三维地图中对应的三维表示为不可编辑三维表示或在三维地图中不存在对应的三维表示。不可编辑物体可以包括取景画面中的背景以及取景画面中存在但三维地图中不存在的物体(即真实场景中存在但三维地图中不存在的物体，以下简称为第二物体)。例如，图4中(a)图所示的取景画面中V形物体和矩形物体为第一物体中的可编辑物体，黑色条纹背景和白色条纹背景为第一物体中的不可编辑物体。
可以理解的是，当三维地图为离线地图时，由于离线地图的建立时刻早于取景画面的生成时刻，可能会造成取景画面中的物体与三维地图中的物体不一致的情况。例如，在建立某房间的三维地图时该房间内的桌子上无物体，但在建立三维地图后有人在桌子上放置了水杯。用户在该房间使用AR应用时，通过取景画面可以看到该房间内存在水杯，但三维地图中不存在水杯，则水杯为该场景内第一物体中的不可编辑物体。
在用户选择目标物体后，电子设备100可以从三维地图中确定该物体对应的三维表示。例如，用户可以通过长按取景画面中的物体选择目标物体，用户选择物体后，电子设备 100首先可以通过图像分割技术从取景画面中选择包括该目标物体的图像块(以下简称第一图像块)，然后根据第一图像块从三维地图内的三维表示中确定该目标物体对应的三维表示。
值得一提的是，目标物体对应的三维表示的大小可能比取景框画面中目标物体的图像中显示的部分更大。例如，取景画面中只有半张桌子，用户点击匹配到三维地图中桌子的三维表示后，后续的编辑功能可以是对这个完整桌子的三维表示的操作。
在一种可能的实现方式中，电子设备100可以根据取景画面的位姿信息确定三维地图中取景画面对应的视野范围(以下简称为第三视野范围)，然后从第三视野范围中的三维表示中确定该目标物体对应的三维表示。
示例性地，电子设备100通过重定位和SLAM实时跟踪技术确定第三视野范围，然后从第三视野范围中的三维表示中确定该目标物体对应的三维表示。例如，先通过SLAM以取景画面为原点建立局部坐标系，通过重定位技术根据电子设备100采集的传感器数据计算出电子设备100的取景画面在三维地图中的6自由度(degree O freedom，DoF)位姿以及取景画面的局部坐标系与三维地图的坐标系的对冲值(delta)量，通过delta量将局部坐标系对齐到三维地图的坐标系，然后根据取景画面的6Dof位姿确定第三视野范围，之后通过第三视野范围的三维表示与第一图像块进行二维(2D)－三维(3D)匹配，当第一图像块内的大部分特征点匹配到的三维表示属于同一个三维表示时，即确定该三维表示为目标物体对应的三维表示。
在另一种可能的实现方式中，电子设备100可以根据取景画面的位姿信息确定三维地图中取景画面对应的视野范围(以下简称为第三视野范围)，然后根据第一图像块在取景画面中的位置从第三视野范围内确定三维地图中第一图像快对应的视野范围，然后从第一图像快对应的视野范围中的三维表示中确定该目标物体对应的三维表示。
上述重定位的具体方法可以采用本领域技术人员能够想到的任何一种方法进行处理，本申请实施例对此不作具体限定。例如，基于词袋模型(bag of word，BoW)的匹配方案(如ORB-SLAM、VINS等)、基于机器学习匹配补丁(patch)的方法(如PTAM)或基于深度学习的网局部特征聚合描述符(net vector of locally aggregated descriptors，NetVlad)方法等。
在一种可能的实现方式中，电子设备100(或其他设备)可以将三维地图中的三维表示标记为可编辑三维表示和不可编辑三维表示。
在一种可能的实现方式中，电子设备100可以判断用户选择的物体是否为可编辑物体，当用户选择的物体为不可编辑物体时，可以通过文字信息、图像信息或声音信息提示用户选择的物体为不可编辑物体。例如，电子设备100可以通过标记信息判断用户选择的物体是否为背景，通过判断三维地图中是否存在用户选择的物体对应的三维表示判断用户选择的物体是否为第二物体。若该物体非背景且非第二物体，则确定该物体为可编辑物体；若该物体为背景或第二物体，则确定该物体为不可编辑物体。
上述标记信息用于标识物体是否为背景。可以通过三维语义分割/检测技术对三维地图中的物体进行标记。例如，可以通过对三维地图(如SLAM地图)中物体的点云或Mesh进行语义分割/检测，离散化并编号以标记出该三维地图中哪些是可编辑物体(如桌子、凳子、杯子、电脑等)，哪些是背景(如地面、墙面等)，用于后续编辑处理。上述三维 语义分割/检测可采用现有技术中的任意一种，本申请实施例对此不做限定。
在另一种可能的实现方式中，电子设备100可以在取景画面中对目标物体进行标记以便于用户选择目标物体。例如，电子设备100可以用户未选择目标物体时在取景画面中对可编辑的第一物体进行高亮显示，当用户选择目标物体后停止高亮显示。例如，在用户选择目标物体之前，根据当前取景画面的6Dof位姿，在当前取景画面上投影匹配三维地图中当前取景画面对应的视野范围内的三维表示，建立取景画面中的图像块与三维表示的对应关系并根据图像块与三维表示的对应关系对图像块进行区分标记，高亮，以提示用户哪些是可以选择的可编辑物体。用户点击后可编辑物体的图像块后，立即匹配到相应的三维表示上。用户选择后，即可关闭该处理。
在一种可能的实现方式中，用户可以通过触摸电子设备100屏幕上特定的控件、按压特定的物理按键或按键组合、输入语音、隔空手势等方式，指示将取景画面中的目标物体隐形。
用户对电子设备100执行使目标物体隐形的操作(以下简称第二操作)后，电子设备100响应于第二操作，执行第三操作。其中，第三操作包括在第一区域生成第一掩膜(mask)并将第一视野范围内的第一三维表示投影至第一mask，第一区域为取景画面中目标物体所在区域，第一视野范围为三维地图中第一区域对应的视野范围，第一视野范围内包括第一三维表示和第二三维表示，第二三维表示为目标物体对应的三维表示。第一三维表示为第一视野范围内的三维表示中除第一三维表示之外的其他三维表示。
示例性地，用户想要使图4中(a)图所示的取景画面中V形物体(即目标物体)隐形，用户则可以通过长按该V形物体指示电子设备100将该V形物体隐形，电子设备100响应于用户操作，如图4中(b)图所示电子设备100首先在取景画面中V形物体所在区域生成第一mask遮挡该V形物体，然后如图4中(c)图所示电子设备100将三维地图中V形物体所在区域对应的视野范围(即第一视野范围)内除V形物体对应的三维表示(即第二三维表示)外的其他三维表示(即第一三维表示)投影至上述mask，通过图4中(c)图可以看出V形物体已经从取景框401的取景画面中隐形(消失)，用户可以通过图4中(c)图查看到V形物体后方的背景(即黑色条纹和白色条纹)。
在一种可能的实现方式中，第三操作也可以为将第二三维表示投影至第一mask，然后删除第一mask上的显示内容，之后将第一三维表示投影至第一mask。
可以看出，本申请实施例提供的显示方法中，电子设备通过AR应用界面内的取景画面可以显示摄像头所采集到的用户所处现实世界内的物体，用户可以通过操作使AR应用先在取景画面内的某个物体所在区域生成掩膜遮挡该物体，然后通过将三维地图中该物体所在区域对应的视野范围(即第一视野范围)内除该物体对应的三维表示(即第二三维表示)外的其他三维表示(即第一三维表示)投影至上述掩膜，以使AR应用中用户所处现实世界内的物体随用户操作而隐形消失，使AR应用显示“该物体隐形消失后的现实世界”，从而增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
例如，用户所处房间内存在一个柜子，用户想要看到柜子后方的背景墙但不想搬动柜子。这时用户就可以通过操作使柜子在AR应用中隐形消失以通过AR应用查看柜子后方的背景墙，由此增加了AR应用于现实世界的交互性，提升了AR应用的用户沉浸感。
可选地，电子设备100可以将上述“第三操作”封装为“隐形操作”以供其他功能或 其他操作调用。
在一种可能的实现方式中，第一mask可能与取景画面中的目标物体不完全重合，即第一mask可能无法遮挡取景画面中的目标物体。这种情况下，电子设备100可以通过图像分割技术，补充生成与目标物体贴合的补充mask。另外，若补充生成的补充mask存在显示内容，电子设备100还可以删除补充mask上的显示内容。
上述图像分割的具体方法可以采用本领域技术人员能够想到的任何一种方法进行处理，本申请实施例对此不作具体限定。
示例性地，用户想要使图4中(a)图所示的取景画面中V形物体隐形，用户则可以通过语音指令指示电子设备100将该V形物体隐形，电子设备100响应于用户操作，如图5中(a)图所示电子设备100首先在取景画面中V形物体所在区域生成第一mask遮挡该V形物体，通过图5中(a)图可以看出第一mask并未完成遮挡V形物体。如图5中(b)图所示，电子设备100补充生成补充mask，通过图5中(b)图可以看出第一mask和补充mask已完全遮挡V形物体。然后如图5中(c)图所示电子设备100将三维地图中V形物体所在区域对应的视野范围内除V形物体对应的三维表示外的其他三维表示投影至第一mask和补充mask，通过图5中(c)图可以看出V形物体已经从取景框401的取景画面中隐形，用户可以通过图5中(c)图查看到V形物体后方的背景。
在一种可能的实现方式中，由于物体之间的遮挡关系，三维地图中物体后方的背景部分的三维表示可能是不完整的，电子设备100执行上述隐形操作后可能会造成取景画面中隐形物品后方的背景不完整，取景画面不连续。对于上述情况用户可以通过触摸电子设备100屏幕上特定的控件、按压特定的物理按键或按键组合、输入语音、隔空手势等方式，指示电子设备100执行补全背景操作以补全取景画面中缺失的背景使取景画面连续。或者，电子设备100可以识别取景框中的取景画面的背景是否完整，当背景不完整时，执行补全背景操作以补全取景画面中的缺失的背景使取景画面连续。又或者，可以在上述第三操作中添加补全背景操作，电子设备100在执行上述第三操作时自动执行补全背景操作。
上述补全取景画面中缺失的背景使取景画面连续(即补全背景操作)的具体方法可以采用本领域技术人员能够想到的任何一种方法进行处理，本申请实施例对此不做具体限定。例如，可以根据三维表示填充mask中未投影三维表示的区域以补全取景画面中缺失的背景使取景画面连续。又例如，可以通过人工智能(artificial intelligence，AI)技术补全取景画面中缺失的背景使取景画面连续。
示例性地，用户想要使图4中(a)图所示的取景画面中V形物体隐形，用户则可以通过语音指令指示电子设备100将该V形物体隐形，电子设备100响应于用户操作，如图4中(b)图所示电子设备100首先在取景画面中V形物体所在区域生成第一mask遮挡该V形物体，如图6中(a)图所示电子设备100将第一三维表示投影至上述mask，通过图6中(a)图可以看出V形物体已经从取景框401的取景画面中隐形，但是V形物体后方的背景缺失导致取景画面不连续。这时电子设备100可以通过向mask中未投影三维表示的区域投影第三三维表示以得到图6中(b)图中所示的取景画面。通过图6中(b)图可以看出取景画面连续且取景画面中缺失的背景已被补全。其中，第三三维表示为第二视野范围内的三维表示，第二视野范围是与第一视野范围相邻的视野范围。
用户还可以通过触摸电子设备100屏幕上特定的控件、按压特定的物理按键或按键组 合、输入语音、隔空手势等方式，指示电子设备100将取景画面中的物体移动。
用户对电子设备100执行使目标物体移动的操作(以下简称第四操作)后，电子设备100响应于第四操作，首先执行第三操作使取景画面中的目标物体隐形，然后在第二区域生成第二mask并将第二三维表示(即目标物体对应的第二三维表示)投影至第二mask。其中，第二区域为取景画面中目标位置所在区域。例如，用户可以通过点击操作选择目标位置。又例如，用户可以通过拖拽操作使取景画面中的物体移动，电子设备100则可以将用户拖拽操作的终点位置确定为目标位置。
示例性地，如图7中(a)所示，用户执行拖拽操作指示电子设备100将该V形物体由取景画面中的左下方移动至左上方，电子设备100响应于用户操作，如图7中(b)图所示电子设备100首先对V形物体执行第三操作使V形物体从取景框401的取景画面中隐形并在取景框401的取景画面左上方生成第二mask。如图7中(c)图所示电子设备100将V形物体对应的三维表示投影至第二mask。通过图7中(c)图可以看出V形物体已经由取景框401的取景画面的左下方移动到左上方。
可以看出，电子设备还可以响应于用户操作，先将AR应用中取景画面中用户所处现实世界内的物体隐形，然后在用户指定区域(即第二区域)生成掩膜并将该物体对应的三维表示投影至该掩膜，以使得AR应用中用户所处现实世界内的物体随用户操作而移动，使AR应用显示“该物体移动后的现实世界”，从而进一步增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
在一种可能的实现方式中，在第二区域生成第二mask并将第二三维表示投影至第二mask之前，电子设备100还可以根据取景画面中的目标位置确定三维地图中的第二目标位置，然后将第二三维表示移动至第二目标位置。例如，电子设备100可以先根据取景画面中的目标位置在三维地图的平面上确定目标位置对应的三维表示的坐标，然后将第二三维表示的锚点移动至该坐标，之后对第二三维表示的剩余三维表示做同样的移动，从而使第二三维表示移动至第二目标位置。其中，三维地图的平面是由三维地图中的三维表示构成的平面。
可选地，锚点可以为位于三维表示的重心垂线方向上Z轴坐标最小的点。
用户还可以通过触摸电子设备100屏幕上特定的控件、按压特定的物理按键或按键组合、输入语音、隔空手势等方式，指示电子设备100对取景画面中的物体进行形变(如放大、缩小、拉伸等)。例如，AR界面可以设置有放大(如放大5倍)、缩小(如缩小10倍)、横向拉伸(如变胖2倍)、纵向拉伸(如长高三倍)之类的配置按钮，用户可以通过触摸上述配置按钮指示电子设备100对取景画面中的物体进行形变。又例如，用户可以通过手指开合控制缩放等操作对取景画面中的物体进行形变。
用户对电子设备100执行使目标物体形变的操作(以下简称第五操作)后，电子设备100响应于第五操作，先执行第三操作，然后在第三区域生成第三mask并将第四三维表示投影至第三mask。其中，第四三维表示为第四三维表示，第四三维表示为形变后的第二三维表示。
可选地，第四三维表示可以为放大、缩小或拉伸后第二三维表示。
示例性地，如图8中(a)所示，用户执行双指放大操作指示电子设备100将预览框401的取景画面内的矩形物体放大，电子设备100响应于用户操作，如图8中(b)图所 示电子设备100首先对该矩形物体执行第三操作使该矩形物体从取景框401的取景画面中隐形并在取景框401的取景画面中生成第三mask，然后电子设备100根据矩形物体对应的三维表示(即第二三维表示)生成第四三维表示。之后如图8中(c)图所示电子设备100将第四三维表示投影至第三mask。通过图8中(c)图可以看出预览框401的取景画面内的矩形物体已被放大。
可以看出，本申请实施例提供的显示方法中，用户还可以通过操作在AR应用中使用户所处现实世界内的物体形变，使AR应用显示“该物体形变后的现实世界”，从而进一步增加了AR应用与现实世界的交互性，提升了AR应用的用户沉浸感。
可选地，第二三维表示和第四三维表示的锚点可以相同。
在一种可能的实现方式中，电子设备100可以根据形变信息和第二三维表示的三维坐标生成第四三维表示。其中，形变信息用于表征形变类型和形变倍率。
示例性地，用户通过触摸相应的配置按钮指示电子设备100对取景画面中的物体放大5倍，电子设备100响应于用户操作并根据用户操作确定形变类型为放大，形变倍率为5倍，然后以第二三维表示的锚点作为缩放原点，计算其他各点(三维表示)在三维坐标系中的坐标相对于锚点的坐标差值，然后对每个点的坐标差值均乘以5后加上锚点的坐标，即可得到一个放大5倍的新的三维表示(即第四三维表示)。
下面结合图9介绍本申请实施例提供的显示方法，如图9所示，该方法包括：
S901、电子设备100响应于第一操作，启动摄像头并显示应用界面。
其中，应用界面包括取景框，取景框的取景画面中包括目标物体。
可以理解的是，取景框的取景画面中可以包括多个物体，目标物体可以为多个物体中的任一物体。
可选地，取景框的取景画面中可以包括可编辑物体和不可编辑物体。第一物体可以为可编辑物体中的任一物体。其中，可编辑物体在三维地图中对应的三维表示为可编辑三维表示，不可编辑物体在三维地图中对应的三维表示为不可编辑三维表示或在三维地图中不存在对应的三维表示。
在一种可能的实现方式中，电子设备100可以将三维地图中的三维表示标记为可编辑三维表示和不可编辑三维表示。
S902、电子设备100响应于第二操作，执行第三操作，第三操作包括在第一区域生成第一mask并将第一视野范围内的第一三维表示投影至第一mask。
其中，第一区域为取景画面中目标物体所在区域，第一视野范围为三维地图中第一区域对应的视野范围，第一视野范围内包括第一三维表示和第二三维表示，第二三维表示为第二三维表示，第二三维表示为目标物体对应的三维表示。第一三维表示为第一视野范围内的三维表示中除第一三维表示之外的其他三维表示。
在一种可能的实现方式中，第三操作还可以包括根据第三三维表示填充第一mask中未投影三维表示的区域，第三三维表示为第二视野范围内的三维表示，第二视野范围是与第一视野范围相邻的视野范围。
在一种可能的实现方式中，电子设备100还可以根据第一图像块，从第一视野范围内的三维表示中确定第二三维表示。其中，第一图像块为取景画面中包含目标物体的图像块。
如图9所示，该方法还可以包括：
S903、电子设备响应于第四操作，执行第三操作，在第二区域生成第二mask并将第二三维表示投影至第二mask。
S904、电子设备响应于第五操作，执行第三操作，在第三区域生成第三mask并将第四三维表示投影至第三mask。
其中，第四三维表示为第四三维表示，第四三维表示为形变后的第二三维表示。
可选地，第四三维表示为放大、缩小或拉伸后第二三维表示。例如，第四三维表示可以为放大5倍后的第二三维表示。
在一种可能的实现方式中，电子设备100还可以根据形变信息和第二三维表示的三维坐标生成第四三维表示，形变信息用于表征形变类型和形变倍率。
可选地，第二三维表示和第四三维表示的锚点相同。其中，锚点可以为三维表示中位于三维表示的重心垂线方向上Z轴坐标最小的点。
在一种可能的实现方式中，电子设备100还根据可以取景画面的位姿信息确定第三视野范围，根据区域(如第一区域、第二区域或第三区域等)在取景画面中的位置确定区域在第三视野范围内的视野范围。其中，第三视野范围为三维地图中取景画面对应的视野范围。
可选地，上述位姿信息可以为6Dof位姿。
本申请实施例还提供了另一种显示方法，该显示方法可以使真实场景中存在但三维地图中不存在的物体从电子设备的取景画面隐身消息，如图10所示，该方法包括：
S1001、电子设备100响应于第一操作，启动摄像头并显示应用界面。
其中，应用界面包括取景框，取景框的取景画面中包括目标物体。
可以理解的是，取景框的取景画面中可以包括多个物体，目标物体可以为多个物体中的任一物体。
S1002、电子设备100响应于第二操作，执行第三操作，第三操作包括在第一区域生成第一mask并将第一视野范围内的三维表示投影至第一mask。
其中，第一区域为取景画面中目标物体所在区域，第一视野范围为三维地图中第一区域对应的视野范围，三维表示包括点云、Mesh或语义中的至少一项。
在一种可能的实现方式中，电子设备100可以在第一视野范围内不存在第一三维表示的情况下，执行上述第三操作。其中，第一三维表示可以为上述目标物体的三维表示。
例如，三维地图存储了某个建筑物的三维表示。用户想通过应用界面的取景框观看该建筑物，但建筑物旁边有“行人”。用户想要使取景框中的“行人”从取景框中消失，这时用户就可以通过操作应用将“行人”选为目标物体，使电子设备执行第三操作，将三维地图中“行人”所在区域对应的三维表示投影至取景画面中“行人”所在区域。可以理解的是，由于三维地图中并未存储行人的三维表示，即三维地图中存储的场景为不存在“行人”的场景。因此可以直接将三维地图中“行人”所在区域的三维表示直接投影至取景画面中以使得“行人”从取景画面中隐形消失。
在一种可能的实现方式中，该方法还可以包括：电子设备100根据第一图像块，从上述第一视野范围内的三维表示中确定是否存在第一三维表示。其中，第一图像块为上述取景画面中包含上述目标物体的图像块。
可以理解的是，当第一图像块在第一视野范围内能够匹配到对应的三维表示，则说明 第一视野范围内存在目标物体的三维表示；当第一图像块在第一视野范围内不能够匹配到对应的三维表示则说明第一视野范围内不存在目标物体的三维表示。
下面将结合图11和图12介绍用于执行上述显示方法的电子设备。
可以理解的是，电子设备为了实现上述功能，其包含了执行各个功能相应的硬件和/或软件模块。结合本文中所公开的实施例描述的各示例的算法步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。本领域技术人员可以结合实施例对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请实施例可以根据上述方法示例对电子设备进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块可以采用硬件的形式实现。需要说明的是，本实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
在采用对应各个功能划分各个功能模块的情况下，图11示出了上述实施例中涉及的电子设备的一种可能的组成示意图，如图11所示，该装置1100可以包括：收发单元1101和处理单元1102，该处理单元1102可以实现上述方法实施例中由电子设备所执行的方法，和/或用于本文所描述的技术的其他过程。
需要说明的是，上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应功能模块的功能描述，在此不再赘述。
在采用集成的单元的情况下，装置1100可以包括处理单元、存储单元和通信单元。其中，处理单元可以用于对装置1100的动作进行控制管理，例如，可以用于支持装置1100执行上述各个单元执行的步骤。存储单元可以用于支持装置1100执行存储程序代码、和/或数据等。通信单元可以用于支持装置1100与其他设备的通信。
其中，处理单元可以是处理器或控制器。其可以实现或执行结合本申请公开内容所描述的各种示例性的逻辑方框，模块和电路。处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，数字信号处理(digital signal processing，DSP)和微处理器的组合等等。存储单元可以是存储器。通信单元具体可以为射频电路、蓝牙芯片、Wi-Fi芯片等与其他电子设备交互的设备。
在一种可能的实现方式中，本申请实施例所涉及的电子设备可以为具有图12所示结构的装置1200，该装置1200包括处理器1201和收发器1202。图11中的收发单元1101和处理单元1102所实现的相关功能可以由处理器1201来实现。
可选地，该装置1200还可以包括存储器1203，该处理器1201和该存储器1203通过内部连接通路互相通信。图11中的存储单元所实现的相关功能可以由存储器1203来实现。
本申请实施例还提供一种计算机存储介质，该计算机存储介质中存储有计算机指令，当该计算机指令在电子设备上运行时，使得电子设备执行上述相关方法步骤实现上述实施例中的显示方法。
本申请实施例还提供了一种计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述相关步骤，以实现上述实施例中的显示方法。
本申请实施例还提供一种电子设备，这个装置具体可以是芯片、集成电路、组件或模 块。具体的，该装置可包括相连的处理器和用于存储指令的存储器，或者该装置包括至少一个处理器，用于从外部存储器获取指令。当装置运行时，处理器可执行指令，以使芯片执行上述各方法实施例中的显示方法。
图13示出了一种芯片1300的结构示意图。芯片1300包括一个或多个处理器1301以及接口电路1302。可选的，上述芯片1300还可以包含总线1303。
处理器1301可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述显示方法的各步骤可以通过处理器1301中的硬件的集成逻辑电路或者软件形式的指令完成。
可选地，上述的处理器1301可以是通用处理器、数字信号处理(digital signal proce ssing，DSP)器、集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
接口电路1302可以用于数据、指令或者信息的发送或者接收，处理器1301可以利用接口电路1302接收的数据、指令或者其他信息，进行加工，可以将加工完成信息通过接口电路1302发送出去。
可选的，芯片还包括存储器，存储器可以包括只读存储器和随机存取存储器，并向处理器提供操作指令和数据。存储器的一部分还可以包括非易失性随机存取存储器(non-volatile random access memory，NVRAM)。
可选的，存储器存储了可执行软件模块或者数据结构，处理器可以通过调用存储器存储的操作指令(该操作指令可存储在操作系统中)，执行相应的操作。
可选的，芯片可以使用在本申请实施例涉及的电子设备或DOP中。可选的，接口电路1302可用于输出处理器1301的执行结果。关于本申请的一个或多个实施例提供的显示方法可参考前述各个实施例，这里不再赘述。
需要说明的，处理器1301、接口电路1302各自对应的功能既可以通过硬件设计实现，也可以通过软件设计来实现，还可以通过软硬件结合的方式来实现，这里不作限制。
其中，本实施例提供的电子设备、计算机存储介质、计算机程序产品或芯片均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其他的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，上述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其他的形式。
上述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
上述功能如果以软件功能单元的形式实现并作为独立地产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例上述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应所述以权利要求的保护范围为准。

Claims (20)

  1. 一种显示方法，其特征在于，所述方法应用于包含摄像头的电子设备，所述方法包括：
    所述电子设备响应于第一操作，启动所述摄像头并显示应用界面，所述应用界面包括取景框，所述取景框的取景画面中包括目标物体；
    所述电子设备响应于第二操作，执行第三操作，所述第三操作包括在第一区域生成第一掩膜mask并将第一视野范围内的第一三维表示投影至所述第一mask，所述第一区域为所述取景画面中目标物体所在区域，所述第一视野范围为三维地图中所述第一区域对应的视野范围，所述第一视野范围内包括第一三维表示和第二三维表示，所述第二三维表示为所述目标物体的三维表示，所述三维表示包括点云、网格Mesh或语义中的至少一项。
  2. 根据权利要求1所述的方法，其特征在于，所述第三操作还包括根据所述第三三维表示填充所述第一mask中未投影三维表示的区域，所述第三三维表示为第二视野范围内的三维表示，所述第二视野范围是与所述第一视野范围相邻的视野范围。
  3. 根据权利要求1或2所述的方法，其特征在于，所述方法还包括：
    所述电子设备响应于第四操作，执行所述第三操作，在第二区域生成第二mask并将所述第二三维表示投影至所述第二mask。
  4. 根据权利要求1至3中任一项所述的方法，其特征在于，所述方法还包括：
    所述电子设备响应于第五操作，执行所述第三操作，在第三区域生成第三mask并将第四三维表示投影至所述第三mask，所述第四三维表示为形变后的第二三维表示。
  5. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    根据形变信息和所述第二三维表示的三维坐标生成所述第四三维表示，所述形变信息用于表征形变类型和形变倍率。
  6. 根据权利要求4或5所述的方法，其特征在于，所述第四三维表示为放大、缩小或拉伸后第二三维表示。
  7. 根据权利要求4至6中任一项所述的方法，其特征在于，所述第二三维表示和所述第四三维表示的锚点相同，锚点为位于三维表示的重心垂线方向上Z轴坐标最小的点。
  8. 根据权利要求1至7中任一项所述的方法，其特征在于，所述方法还包括：
    根据所述取景画面的位姿信息确定第三视野范围，所述第三视野范围为所述三维地图中所述取景画面对应的视野范围；
    根据区域在取景画面中的位置确定所述区域在所述第三视野范围内的视野范围。
  9. 根据权利要求8所述的方法，其特征在于，所述位姿信息为6自由度Dof位姿。
  10. 根据权利要求1至9中任一项所述的方法，其特征在于，所述方法还包括：
    根据第一图像块，从所述第一视野范围内的三维表示中确定所述第二三维表示，所述第一图像块为所述取景画面中包含所述目标物体的图像块。
  11. 根据权利要求1至10中任一项所述的方法，其特征在于，所述取景框的取景画面中包括多个物体，所述目标物体为所述多个物体中的任一物体。
  12. 根据权利要求1至10中任一项所述的方法，其特征在于，所述取景框的取景画面中包括可编辑物体和不可编辑物体，所述目标物体为任一可编辑物体，所述可编辑物体在 所述三维地图中对应的三维表示为可编辑三维表示，所述不可编辑物体在所述三维地图中对应的三维表示为不可编辑三维表示或在所述三维地图中不存在对应的三维表示。
  13. 根据权利要求1至12中任一项所述的方法，其特征在于，所述方法还包括：将所述三维地图中的三维表示标记为可编辑三维表示和不可编辑三维表示。
  14. 一种显示方法，其特征在于，所述方法应用于包含摄像头的电子设备，所述方法包括：
    所述电子设备响应于第一操作，启动所述摄像头并显示应用界面，所述应用界面包括取景框，所述取景框的取景画面中包括目标物体；
    所述电子设备响应于第二操作，执行第三操作，所述第三操作包括在第一区域生成第一掩膜mask并将第一视野范围内的三维表示投影至所述第一mask，所述第一区域为所述取景画面中目标物体所在区域，所述第一视野范围为三维地图中所述第一区域对应的视野范围，所述三维表示包括点云、Mesh或语义中的至少一项。
  15. 根据权利要求14所述的方法，其特征在于，所述执行第三操作包括：
    在第一视野范围内不存在第一三维表示的情况下，执行所述第三操作，所述第一三维表示为所述目标物体的三维表示。
  16. 根据权利14或15所述的方法，其特征在于，所述方法还包括：
    根据第一图像块，从所述第一视野范围内的三维表示中确定是否存在第一三维表示，所述第一图像块为所述取景画面中包含所述目标物体的图像块，所述第一三维表示为所述目标物体的三维表示。
  17. 一种电子设备，包括至少一个处理器和接口电路，所述至少一个处理器和所述接口电路耦合，其特征在于，所述至少一个处理器执行存储在存储器中的程序或指令，以使得所述数据交换装置实现权利要求1至13中任一项或14至16中任一项所述的方法。
  18. 一种计算机可读存储介质，用于存储计算机程序，其特征在于，所述计算机程序包括用于实现上述权利要1至13中任一项或14至16中任一项所述的方法的指令。
  19. 一种计算机程序产品，所述计算机程序产品中包含指令，其特征在于，当所述指令在计算机或处理器上运行时，使得所述计算机或所述处理器实现上述权利要求1至13中任一项或14至16中任一项所述的方法。
  20. 一种电子设备上的图形用户界面，其特征在于，所述电子设备具有显示屏、存储器、以及处理器，所述处理器用于执行存储在所述存储器中的一个或多个计算机程序，所述图形用户界面包括所述电子设备执行上述权利要求1至13中任一项或14至16中任一项所述的方法时显示的图形用户界面。
PCT/CN2022/127535 2021-11-01 2022-10-26 显示方法及电子设备 WO2023072113A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP22885990.6A EP4383191A1 (en) 2021-11-01 2022-10-26 Display method and electronic device

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202111283430 2021-11-01
CN202111283430.5 2021-11-01
CN202210222388.4 2022-03-07
CN202210222388.4A CN116071523A (zh) 2021-11-01 2022-03-07 显示方法及电子设备

Publications (1)

Publication Number Publication Date
WO2023072113A1 true WO2023072113A1 (zh) 2023-05-04

Family

ID=86159072

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/127535 WO2023072113A1 (zh) 2021-11-01 2022-10-26 显示方法及电子设备

Country Status (2)

Country Link
EP (1) EP4383191A1 (zh)
WO (1) WO2023072113A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107180406A (zh) * 2016-03-09 2017-09-19 腾讯科技（深圳）有限公司 图像处理方法和设备
CN107369204A (zh) * 2017-07-27 2017-11-21 北京航空航天大学 一种基于深度学习从单幅照片恢复出场景基本三维结构的方法
CN110619674A (zh) * 2019-08-15 2019-12-27 重庆特斯联智慧科技股份有限公司 用于事故和警情场景还原的三维增强现实设备及方法
US10682108B1 (en) * 2019-07-16 2020-06-16 The University Of North Carolina At Chapel Hill Methods, systems, and computer readable media for three-dimensional (3D) reconstruction of colonoscopic surfaces for determining missing regions

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107180406A (zh) * 2016-03-09 2017-09-19 腾讯科技（深圳）有限公司 图像处理方法和设备
CN107369204A (zh) * 2017-07-27 2017-11-21 北京航空航天大学 一种基于深度学习从单幅照片恢复出场景基本三维结构的方法
US10682108B1 (en) * 2019-07-16 2020-06-16 The University Of North Carolina At Chapel Hill Methods, systems, and computer readable media for three-dimensional (3D) reconstruction of colonoscopic surfaces for determining missing regions
CN110619674A (zh) * 2019-08-15 2019-12-27 重庆特斯联智慧科技股份有限公司 用于事故和警情场景还原的三维增强现实设备及方法

Also Published As

Publication number Publication date
EP4383191A1 (en) 2024-06-12

Similar Documents

Publication Publication Date Title
WO2021238325A1 (zh) 一种图像处理方法及装置
WO2021147482A1 (zh) 一种长焦拍摄的方法及电子设备
JP2023514631A (ja) インタフェースレイアウト方法、装置、及び、システム
WO2023093169A1 (zh) 拍摄的方法和电子设备
CN115689963B (zh) 一种图像处理方法及电子设备
CN113709355A (zh) 滑动变焦的拍摄方法及电子设备
CN113538227B (zh) 一种基于语义分割的图像处理方法及相关设备
WO2022262550A1 (zh) 一种拍摄视频的方法及电子设备
WO2022057384A1 (zh) 拍摄方法和装置
WO2021254113A1 (zh) 一种三维界面的控制方法和终端
WO2023066165A1 (zh) 动画效果显示方法及电子设备
WO2023005751A1 (zh) 渲染方法及电子设备
WO2023072113A1 (zh) 显示方法及电子设备
WO2023078133A1 (zh) 视频播放方法和装置
CN116071523A (zh) 显示方法及电子设备
WO2023035868A1 (zh) 拍摄方法及电子设备
CN116193243B (zh) 拍摄方法和电子设备
WO2023231696A1 (zh) 一种拍摄方法及相关设备
CN113821153B (zh) 手势导航方法、电子设备及可读存储介质
WO2023066177A1 (zh) 动画效果显示方法及电子设备
WO2024012354A1 (zh) 一种显示方法及电子设备
CN115623318B (zh) 对焦方法及相关装置
WO2024099206A1 (zh) 一种图形界面处理方法以及装置
CN116688494B (zh) 生成游戏预测帧的方法和电子设备
CN117148959B (zh) 眼动追踪的帧率调整方法及相关装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22885990

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022885990

Country of ref document: EP

Effective date: 20240307