WO2021203902A1

WO2021203902A1 - Virtual image realization method and apparatus, and storage medium and terminal device

Info

Publication number: WO2021203902A1
Application number: PCT/CN2021/080040
Authority: WO
Inventors: 季春林
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-04-08
Filing date: 2021-03-10
Publication date: 2021-10-14
Also published as: CN111476911A; CN111476911B

Abstract

A virtual image realization method, a virtual image realization apparatus, a storage medium and a terminal device. The virtual image realization method is applied to a first terminal and comprises: acquiring editing information input by a user for a target object (S310); generating a virtual image of the target object according to the editing information (S320); and uploading the virtual image to an augmented reality scene where the target object is located, wherein the augmented reality scene is stored in a cloud end, so that a second terminal acquires the augmented reality scene from the cloud end to display the virtual image (S330). The diversified experience of augmented reality is improved, and a channel for realizing information transfer and sharing is provided. (FIG. 3)

Description

Virtual image realization method, device, storage medium and terminal equipment

This application claims the priority of a Chinese patent application filed on April 8, 2020 with the application number 202010269991.9 and the title "Virtual image realization method, device, storage medium and terminal equipment". The entire content of the Chinese patent application is approved Reference is incorporated in this article.

Technical field

The present disclosure relates to the technical field of virtual reality and augmented reality, and in particular to virtual image realization methods, virtual image realization devices, computer-readable storage media and terminal equipment.

Background technique

In an AR (Augmented Reality) scene or AR game, the user can see the superimposition effect of the virtual image projected onto the real scene, thereby bringing the user an immersive visual experience.

Summary of the invention

The present disclosure provides a method for realizing a virtual image, a device for realizing a virtual image, a computer-readable storage medium, and terminal equipment.

According to a first aspect of the present disclosure, there is provided a virtual image realization method, which is applied to a first terminal, the method includes: acquiring edit information for a target object input by a user; and generating a virtual image of the target object according to the edit information. Image; upload the virtual image to the augmented reality scene where the target object is located, the augmented reality scene is stored in the cloud, so that the second terminal displays the virtual image by acquiring the augmented reality scene from the cloud .

According to a second aspect of the present disclosure, there is provided a virtual image realization method applied to a second terminal, the method comprising: acquiring an augmented reality scene from the cloud, the augmented reality scene including at least a virtual image of a target object, the The virtual image is uploaded to the augmented reality scene by the first terminal; relocation is performed based on the augmented reality scene, and when the relocation is successful, the virtual image or prompt information of the virtual image is displayed.

According to a third aspect of the present disclosure, there is provided a virtual image realization device, which is provided in a first terminal, the device includes a processor, and the processor is configured to execute the following program modules stored in a memory: an acquisition module for acquiring a user Input editing information for the target object; a generating module for generating a virtual image of the target object according to the editing information; an uploading module for uploading the virtual image to the augmented reality scene where the target object is located , The augmented reality scene is stored in the cloud, so that the second terminal obtains the augmented reality scene from the cloud to display the virtual image.

According to a fourth aspect of the present disclosure, there is provided a virtual image realization device, which is provided in a second terminal, the device includes a processor, and the processor is configured to execute the following program modules stored in a memory: an acquisition module for downloading from the cloud Acquire an augmented reality scene, the augmented reality scene includes at least a virtual image of a target object, the virtual image is uploaded to the augmented reality scene by the first terminal; the relocation module is configured to perform relocation based on the augmented reality scene Positioning, when repositioning is successful, display the virtual image or prompt information of the virtual image.

According to a fifth aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the virtual image realization method described in the first or second aspect and the same Possible implementation.

According to a sixth aspect of the present disclosure, there is provided a terminal device, including: a processor; a memory for storing executable instructions of the processor; and a display; wherein the processor is configured to execute the executable Instructions to execute the virtual image realization method described in the first or second aspect and possible implementations thereof.

Description of the drawings

FIG. 1 shows a schematic diagram of the architecture of an AR system in this exemplary embodiment;

FIG. 2 shows a schematic structural diagram of a mobile terminal in this exemplary embodiment;

Fig. 3 shows a flowchart of a method for implementing virtual images in this exemplary embodiment;

Fig. 4 shows a flowchart of another virtual image realization method in this exemplary embodiment;

Fig. 5 shows a flowchart of a relocation method in this exemplary embodiment;

Fig. 6 shows a flowchart of a method for modifying a virtual image in this exemplary embodiment;

FIG. 7 shows a schematic structural diagram of a virtual image realization device provided in a first terminal in this exemplary embodiment;

FIG. 8 shows a schematic structural diagram of another virtual image realization device provided in the first terminal in this exemplary embodiment;

FIG. 9 shows a schematic structural diagram of a virtual image realization device provided in a second terminal in this exemplary embodiment;

FIG. 10 shows a schematic structural diagram of another virtual image realization device provided in the second terminal in this exemplary embodiment.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms, and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed To those skilled in the art. The described features, structures or characteristics can be combined in one or more embodiments in any suitable way. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, the well-known technical solutions are not shown or described in detail in order to avoid overwhelming the crowd and obscure all aspects of the present disclosure.

In addition, the drawings are only schematic illustrations of the present disclosure, and are not necessarily drawn to scale. The same reference numerals in the figures denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

In related technologies, the AR virtual images that the user can view are generally set in advance by the program, for example, the user walks to a certain place to trigger the appearance of a specific virtual character. This kind of AR experience process is immutable, it is difficult to arouse the user's interest in multiple experiences, and its practicability is low.

In view of the foregoing problems, exemplary embodiments of the present disclosure provide a virtual image realization method and a virtual image realization device.

FIG. 1 shows a schematic diagram of the architecture of an AR system suitable for implementing an exemplary embodiment of the present disclosure.

As shown in FIG. 1, the AR system 100 may include: a first terminal 110, a second terminal 120, and a cloud 130. Among them, the first terminal 110 and the second terminal 120 are used by users, and may be various electronic devices with display functions, including but not limited to smart phones, tablet computers, portable computers, smart wearable devices (such as AR glasses), and so on. Both the first terminal 110 and the second terminal 120 can run AR program clients. The cloud 130 is a collection of hardware devices on the server side where AR programs are deployed, which can include a series of physical architectures such as proxy servers, application servers, databases, etc. This disclosure does not make a special distinction on this, and abstracts them as the cloud 130 to provide AR cloud. service. The first terminal 110 and the second terminal 120 may be connected to the server 130 through a network to perform data exchange.

It should be understood that the number of terminal devices in FIG. 1 is only illustrative, and any number of terminal devices can be provided according to implementation needs, for example, a third terminal, a fourth terminal, etc. can be provided.

In order to realize the above virtual image realization method and realization device, the exemplary embodiment of the present disclosure also provides a terminal device, which may be the above-mentioned first terminal 110 or the second terminal 120, which may be realized in various forms, for example, may include smart Mobile phones, tablet computers, portable computers, smart wearable devices (such as AR glasses), etc. Taking the mobile terminal 200 in FIG. 2 as an example, the structure of the terminal device will be exemplarily described below. In other embodiments, the mobile terminal 200 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically, and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt a different interface connection manner from that shown in FIG. 2 or a combination of multiple interface connection manners.

As shown in FIG. 2, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, Battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, earphone interface 274, sensor module 280, display 290, camera module 291, indicator 292, a motor 293, a button 294, a Subscriber Identification Module (SIM) card interface 295, and so on. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, an air pressure sensor 2804, and the like.

The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (Image Signal Processor, ISP), controller, video codec, digital signal processor (Digital Signal Processor, DSP), baseband processor and/or Neural-Network Processing Unit (NPU), etc. Among them, the different processing units may be independent devices or integrated in one or more processors.

The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of reading instructions and executing instructions.

A memory may also be provided in the processor 210 for storing instructions and data. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the processor 210 controls the execution. In some embodiments, the memory in the processor 210 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 210. If the processor 210 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 210 is reduced, and the efficiency of the system is improved.

In some embodiments, the processor 210 may include one or more interfaces. Interfaces can include integrated circuit (Inter-Integrated Circuit, I2C) interface, integrated circuit built-in audio (Inter-Integrated Circuit Sound, I2S) interface, pulse code modulation (Pulse Code Modulation, PCM) interface, universal asynchronous transceiver (Universal Asynchronous) Receiver/Transmitter, UART) interface, Mobile Industry Processor Interface (MIPI), General-Purpose Input/Output (GPIO) interface, Subscriber Identity Module (SIM) interface and/ Or Universal Serial Bus (Universal Serial Bus, USB) interface, etc. Connections are formed with other components of the mobile terminal 200 through different interfaces.

The USB interface 230 is an interface that complies with the USB standard specification, and specifically may be a MiniUSB interface, a MicroUSB interface, a USBTypeC interface, and so on. The USB interface 230 can be used to connect a charger to charge the mobile terminal 200, can also connect to earphones to play audio through the earphones, and can also be used to connect the mobile terminal 200 to other touch terminals, such as computers, peripherals, and the like.

The charging management module 240 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging implementations, the charging management module 240 may receive the charging input of the wired charger through the USB interface 230. In some implementations of wireless charging, the charging management module 240 may receive the wireless charging input through the wireless charging coil of the mobile terminal 200. While the charging management module 240 charges the battery 242, it can also supply power to the touch terminal through the power management module 241.

The power management module 241 is used to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives the input of the battery 242 and/or the charge management module 240, and supplies power to the processor 210, internal memory 221, display 290, camera module 291, and wireless communication module 260. It can also be used to monitor battery capacity. The number of cycles, battery health status (leakage, impedance) and other parameters.

The wireless communication function of the mobile terminal 200 can be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, and the baseband processor.

The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile terminal 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 250 may provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the mobile terminal 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and so on. The mobile communication module 250 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering and amplifying the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave radiation via the antenna 1.

The wireless communication module 260 can provide applications on the mobile terminal 200, including wireless local area networks (Wireless Local Area Networks, WLAN) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (Bluetooth, BT), and global navigation satellites. System (Global Navigation Satellite System, GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared Technology (Infrared, IR) and other wireless communication solutions. The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 may also receive the signal to be sent from the processor 210, perform frequency modulation, amplify, and convert it into electromagnetic waves and radiate it through the antenna 2.

In some embodiments, the antenna 1 of the mobile terminal 200 is coupled with the mobile communication module 250, and the antenna 2 is coupled with the wireless communication module 260, so that the mobile terminal 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), and broadband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Time Division Code Division Multiple Access (Time Division-Synchronous Code Division Multiple Access, TD-SCDMA), Long Term Evolution (LTE), New Radio (New Radio, NR), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (Quasi -Zenith Satellite System, QZSS) and/or Satellite Based Augmentation Systems (SBAS).

The mobile terminal 200 implements a display function through a GPU, a display 290, an application processor, and the like. The GPU is a microprocessor that connects the display 290 and the application processor. The GPU is used to perform mathematical and geometric calculations and is used for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

The mobile terminal 200 may include one or more displays 290 for displaying images, videos, and the like. The display 290 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), Active-Matrix Organic Light-Emitting Diode, or Active-Matrix Organic Light-Emitting Diode (Active-Matrix Organic Light-Emitting Diode). Emitting Diode, AMOLED), Flexible Light Emitting Diode (Flexlight-Emitting Diode, FLED), Miniled, MicroLed, Micro-oLed, Quantum Dot Light Emitting Diodes (QLED), etc.

The mobile terminal 200 can implement a shooting function through an ISP, a camera module 291, a video codec, a GPU, a display 290, and an application processor.

The ISP is used to process the data fed back from the camera module 291. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye. In some embodiments, the ISP may be provided in the camera module 291.

The mobile terminal 200 may include one or more camera modules 291 for capturing still images or videos. The object generates an optical image through the lens and is projected to the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals.

Video codecs are used to compress or decompress digital video. The mobile terminal 200 may support one or more video codecs. In this way, the mobile terminal 200 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The external memory interface 222 may be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the mobile terminal 200. The external memory card communicates with the processor 210 through the external memory interface 222 to realize the data storage function. For example, save music, video and other files in an external memory card.

The internal memory 221 may be used to store computer executable program code, where the executable program code includes instructions. The internal memory 221 may include a storage program area and a storage data area. Among them, the storage program area can store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required by at least one function, and the like. The data storage area can store data (such as audio data, phone book, etc.) created during the use of the mobile terminal 200. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (Universal Flash Storage, UFS), and the like. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal memory 221 and/or instructions stored in a memory provided in the processor.

The mobile terminal 200 can implement audio functions through an audio module 270, a speaker 271, a receiver 272, a microphone 273, a headset interface 274, an application processor, and the like. For example, music playback, recording, etc.

The audio module 270 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 may be provided in the processor 210, or part of the functional modules of the audio module 270 may be provided in the processor 210.

The speaker 271, also called a "speaker", is used to convert audio electrical signals into sound signals. The mobile terminal 200 can listen to music through the speaker 271, or listen to a hands-free call.

The receiver 272, also called "earpiece", is used to convert audio electrical signals into sound signals. When the mobile terminal 200 answers a call or voice message, it can receive the voice by bringing the receiver 272 close to the human ear.

The microphone 273, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can approach the microphone 273 through the mouth to make a sound, and input the sound signal to the microphone 273. The mobile terminal 200 may be provided with multiple microphones 273 to implement functions such as noise reduction, sound source identification, and directional recording.

The earphone interface 274 is used to connect wired earphones. The earphone interface 274 can be a USB interface 230, or a 3.5mm open mobile terminal platform (Open Mobile Terminal Platform, OMTP) standard interface, and the US Cellular Telecommunications Industry Association (Cellular Telecommunications Industry Association of the USA, CTIA) standard interface .

The depth sensor 2801 is used to obtain depth information of the scene. In some embodiments, the depth sensor may be provided in the camera module 291. The pressure sensor 2802 is used to sense the pressure signal, and can convert the pressure signal into an electrical signal for implementing functions such as pressure touch. The gyroscope sensor 2803 can be used to determine the motion posture of the mobile terminal 200, and can be used to shoot scenes such as anti-shake, navigation, and somatosensory games. The air pressure sensor 2804 is used to measure air pressure, and can assist positioning and navigation by calculating the altitude.

In addition, according to actual needs, sensors with other functions can be set in the sensor module 280, such as magnetic sensors, acceleration sensors, distance sensors, proximity light sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc. Wait.

The indicator 292 can be an indicator light, which can be used to indicate the charging status, power change, or to indicate messages, missed calls, notifications, and so on.

The motor 293 can generate vibration prompts, such as vibration prompts for incoming calls, alarm clocks, received messages, etc., and can also be used for touch vibration feedback and the like.

The button 294 includes a power-on button, a volume button, and so on. The button 294 may be a mechanical button. It can also be a touch button. The mobile terminal 200 may receive key input, and generate key signal input related to user settings and function control of the mobile terminal 200.

The mobile terminal 200 may support one or more SIM card interfaces 295 for connecting to the SIM card, so that the mobile terminal 200 can interact with the network through the SIM card to implement functions such as call and data communication.

The virtual image realization method and virtual image realization device of the exemplary embodiment of the present disclosure will be described in detail below.

The virtual image realization method can be realized in the form of an AR program. FIG. 3 shows a schematic flow of the virtual image realization method executed by the first terminal 110. The first terminal 110 may implement the following steps S310 to S330 through a client running an AR program:

Step S310: Obtain editing information for the target object input by the user.

Among them, the target object can be any object in the real world environment, such as a refrigerator, a pool, etc. in a room. The user can input editing information in any form and content for the target object, which can include any one or more of text, pictures, hand-drawn drawings, animation effects, video, and audio. For example, the user can input a paragraph of text, set the corresponding font, color, etc., and edit the frame of the note paper style for it, as well as the animation effect of the text display, such as gradually appearing in a faded manner, flying in from a certain direction, etc.; Or the user can manually draw the picture and add a certain background texture to it; or the user can edit a virtual icon and add a piece of audio, the virtual icon can trigger the audio to be played, and so on.

In an embodiment, the target object can be determined by scanning with a camera. Specifically, before step S310, the following steps may be performed:

When the target object is scanned from the real scene, the editing interface is triggered to be displayed.

Among them, the real scene refers to the real world environment where the target object is located, such as a real room, coffee shop, shopping mall, or street. When the user wants to edit a virtual image of an object, he can turn on the camera of the first terminal, aim at the object, and the camera scans the collected image. When the object is recognized, the editing interface is triggered, or the message "Is it right?" Edit virtual image", the user selects "Yes" and the editing interface pops up. The editing interface is used to input editing information, and can include editing columns for various information, such as text columns, picture columns, and animation effect columns.

Further, in order to accurately identify the target object, when the user turns on the camera to scan, the client can guide the user to move multiple angles, for example, the scanning interface displays "please aim at the target from another angle", "please aim at the back of the target" "And other guidance information, so as to collect the full picture of the target object.

In another embodiment, the user can click to select the target object on the client. Specifically, the client can download the scene information from the server in advance and open the display. The user clicks on an object in the scene, determines it as the target object, and enters the editing interface.

In step S320, a virtual image of the target object is generated according to the above-mentioned editing information.

After the editing is completed, a virtual image can be generated correspondingly. Generally, the user's editing information is the main content of the virtual image. The client can directly use the editing information as the virtual image, or based on this, add some virtual visual effects, or combine the virtual image with the target object to a certain extent . For example, the user edits a virtual note with the refrigerator in the room as the target object, including the size, shape, and color of the note, and enters text in the note. The client can automatically generate a virtual note for the virtual note. Light and shadow, or set a certain degree of transparency to be placed on a plane of the refrigerator, or deform the virtual label to make it more three-dimensional, or add dynamic effects to the virtual label, such as shaking with the wind, or to it The text generates a corresponding piece of speech and so on.

In an optional implementation manner, the user can also edit the display position of the virtual image to include it in the above-mentioned editing information. For example, the user clicks on a certain position on the target object to edit and generate a virtual note at that position, or the user edits the virtual note and moves it to a certain position, that is, the display position of the virtual image.

In step S330, the virtual image is uploaded to the augmented reality scene where the target object is located, and the augmented reality scene is stored in the cloud, so that the second terminal obtains the augmented reality scene from the cloud to display the virtual image.

In this exemplary embodiment, the storage of AR data in the cloud is implemented in units of augmented reality scenes. Augmented reality scene refers to a data collection established for the real scene where the target object is located, which can include map data of the real scene, cloud anchor point data, 3D point cloud data of each object, and added virtual image data (such as user It can be seen that all AR data related to the scene can be stored in the corresponding augmented reality scene. When new data is added or data is updated, it can be synchronized to the augmented reality scene. Thus, after the client generates a virtual image and uploads it to the cloud, it can be directly stored in the corresponding augmented reality scene.

Regarding how to determine the augmented reality scene where the target object is located, several implementation manners are provided below.

(1) The user scans the real scene for the first time, that is, the cloud does not record any AR data about the real scene. In this case, in addition to scanning the target object, it is also necessary to scan the actual scene where the target object is located to generate map data. The map data can be a three-dimensional point cloud model reflecting the characteristics of a real scene, including information such as terrain, texture, and obstacles. The map data generated by the first terminal can adopt the SLAM (Simultaneous Localization And Mapping, simultaneous positioning and mapping) method. Specifically, after starting the camera, user A holds (or wears) the first terminal and moves in the real scene; the first terminal collects scene images and records pose information at the same time; after obtaining a certain number of frames of images, it can be initialized The map data of the scene usually only includes a small part of the scene, or is different from the actual scene; after that, every time a frame of image is collected, the map data can be updated and optimized according to the image, and the map points that are not in the map can be added. Or modify the position of the existing map point, etc.; until the entire scene is traversed, the complete map data is obtained. When SLAM collects scene images, a certain number of key frame images are usually selected from continuous frame images. This is a representative frame selected in order to reduce information redundancy in the modeling process, usually a certain number of frames per interval Choose a frame as the key frame, or extract the key frame image when the image content changes a lot. When creating map data, a collection of key frame images is usually packaged into the map data for subsequent use.

The map data can then be uploaded to the cloud, for example, it can be uploaded together with the virtual image of the target object, or it can be uploaded before the virtual image, so that an augmented reality scene corresponding to the real scene can be established on the cloud. Therefore, it is easy to determine that the target object belongs to the established augmented reality scene.

For example, user A uses his own kitchen as a real scene, scans the kitchen to create map data, scans the refrigerator in the kitchen, and edits virtual images. The client uploads map data and virtual images to the cloud to create a corresponding augmented reality scene. For example, the scene can be named "User A's Kitchen" and the virtual image is added to the augmented reality scene.

(2) If an augmented reality scene has been created before, the user can enter the augmented reality scene on the client before scanning the target object. For example, you can enter the scene number or scene name, send a request to the cloud, and after the cloud feedback allows, the client downloads the information of the augmented reality scene. Generally, an augmented reality scene includes key frame images of some or all objects. For example, when the kitchen is scanned for the first time, key frame images are recorded for more characteristic objects such as refrigerators and sinks in the kitchen. When the user scans the target object, the client can match the image of the target object with the key frame image. When the matching is successful, the target object is determined to be a recorded object, and the editing interface can be triggered at this time.

(3) If the client does not enter a certain augmented reality scene, and the user directly scans the target object and edits related information, when uploading a virtual image, you can also select at least one key frame image from the collected images of the target object, and Upload to the cloud. The cloud can match the key frame image with the key frame image in the stored augmented reality scene to determine which augmented reality scene the target object belongs to. Considering that the amount of data for matching calculation is large and mismatching is easy to occur (for example, the appearance of the refrigerator is similar, the refrigerator of user A may match the kitchen of user B), the first terminal is uploading virtual images and target objects When the key frame image is selected, GPS (Global Positioning System, Global Positioning System) and other positioning data can also be sent, so that the cloud can narrow the matching range based on the positioning data.

(4) Before each virtual image is generated, the user is required to scan the target object and the real scene, and send the key frame image, map data, and virtual image to the cloud. The cloud finds the corresponding augmented reality scene based on the map data, and if found, it will be directly retrieved Augmented reality scene, if not found, create a new augmented reality scene.

In actual applications, the above-mentioned different methods can be adopted or combined according to the situation, which is not limited in the present disclosure.

After the virtual image edited on the first terminal is added to the augmented reality scene, if a second terminal obtains the information of the augmented reality scene, the virtual image can be obtained synchronously, so that the virtual image can be displayed on the second terminal. For example, user A edits and generates a virtual note of the refrigerator and uploads it to the cloud; after user B downloads the augmented reality scene, the virtual note can be displayed while watching the refrigerator.

In an optional implementation manner, the prompt information of the virtual image can also be generated according to the position of the target object in the augmented reality scene, and when the virtual image is uploaded, the prompt information is uploaded to the augmented reality scene together. The prompt information is used to prompt the orientation of the virtual image, so that when the user is in different positions in the augmented reality scene, the virtual image can be found according to the prompt information. The prompt information can be generated at the same time as the virtual image, or can be generated later than the virtual image. The following provides two specific implementation methods on how to generate prompt information:

In one embodiment, the client first determines the position of the target object in the augmented reality scene, which is usually the position of the virtual image (most of the virtual images are displayed on the target object), and then can use this position as a reference to convert the augmented reality scene Divided into multiple areas, a guiding virtual indicator arrow is generated for each area to point to the position of the virtual image.

In another embodiment, the client first determines the position of the target object in the augmented reality scene, which is usually the position of the virtual image (most of the virtual images are displayed on the target object), and then can generate a virtual icon at this position. It is used to prompt the virtual image, and the virtual icon can be displayed when the user is far away from the position of the virtual image.

FIG. 4 shows a schematic flow of the virtual image realization method executed by the above-mentioned second terminal 120. The second terminal 120 may implement the following steps S410 and S420 through a client running an AR program:

Step S410: Obtain an augmented reality scene from the cloud. The augmented reality scene includes at least a virtual image of a target object, and the virtual image is uploaded to the augmented reality scene by the first terminal.

In step S420, relocation is performed based on the above-mentioned augmented reality scene, and when the relocation is successful, the above-mentioned virtual image or prompt information of the virtual image is displayed.

For example: suppose that user A uses the first terminal to edit and generate a virtual image and upload it to the cloud, and then user B uses the second terminal to download an augmented reality scene, then the virtual image is obtained; or the second terminal has downloaded the augmented reality scene After the first terminal uploads the virtual image, the cloud can send an update notification to the second terminal. After user B chooses to update, the new virtual image is obtained; or the first terminal and the second terminal are connected to the cloud at the same time, and the augmented reality is turned on In the scenario, after the first terminal generates and sends the virtual image, the cloud can be synchronized to the second terminal.

In an optional implementation manner, corresponding permissions may be set for acquiring the augmented reality scene. For example, when user A creates an augmented reality scene through the first terminal, set a password, and other users need to enter the correct password to download the augmented reality scene from the cloud; or when user A creates an augmented reality scene through the first terminal, set an access whitelist , Only users on the whitelist can download the augmented reality scene from the cloud, and so on.

After acquiring the augmented reality scene, relocation is required to determine the reality scene where the second terminal is located, that is, the augmented reality scene. The relocation process is generally as follows: the second terminal starts the camera, and user B holds (or wears) the second terminal and moves in the real scene; the second terminal collects images of the real scene and matches it with the map data in the augmented reality scene. When the matching is successful, it is determined that the relocation is successful, and the virtual image or its prompt message can be displayed at this time. Whether the virtual image is displayed or the prompt information is specifically related to the location of the user. Generally, when the user is close to the target object and the target object is within the field of view, a virtual image is displayed, otherwise, a prompt message is displayed to guide the user to move to another location where the virtual image can be seen.

It should be noted that the virtual image is generally displayed on the target object. If the first terminal edits the display position of the virtual image, the second terminal can also obtain the position when acquiring the augmented reality scene and display the virtual image at that position. .

In an optional implementation manner, referring to FIG. 5, step S420 may include the following steps S501 to S504:

Step S501: Perform relocation based on the augmented reality scene, and when the relocation is successful, determine the current pose of the second terminal in the augmented reality scene;

Step S502: Determine the current view area of the second terminal according to the foregoing current pose;

Step S503, when the display position of the virtual image is within the current view area, display the virtual image;

Step S504: When the display position of the virtual image is outside the current view area, display the prompt information of the virtual image.

The relocation process is also the process of aligning the camera coordinate system of the second terminal with the world coordinate system of the augmented reality scene. Through relocation, the current pose of the second terminal in the world coordinate system of the augmented reality scene can be determined; The current pose and the field of view of the second terminal (for example, 120 degrees) can determine the current field of view area to simulate the range of the area that can be captured by the camera of the second terminal, or the user is in the current pose and can see It can then determine whether the user can see the virtual image theoretically, that is, when the display position of the virtual image is within the current field of view, it is judged that the user can see the virtual image, then the virtual image is displayed, and the display position of the virtual image is When the current field of view is outside, it is determined that the user cannot see the virtual image, and a prompt message is displayed.

In an optional implementation manner, the relocation may also be performed based on the target object, so that when the relocation is successful, it can be directly determined that the user can see the virtual image, and therefore the virtual image is displayed. Specifically, the key frame image of the target object can be obtained from the augmented reality scene, and then the collected current image can be matched with the key frame image. When the second terminal is just aligned with the target object, the captured current image and key frame The image can be matched successfully, and then the virtual image is displayed.

In an optional implementation manner, when the prompt information of the virtual image is displayed, if the user is far away from the target object and the target object is no longer in the field of view, the user can follow the direction of the virtual image at the edge of the current interface. Display prompt information, such as virtual guide arrows, etc.; if the user is far away from the target object, but the target object is within the field of view, it can display the prompt information that displays the virtual image in the form of a virtual icon, such as displaying a virtual icon above the target object , And use an arrow to point to the target object to indicate that the location has a virtual image.

In an optional implementation manner, the user may be allowed to modify the virtual image, as shown in FIG. 6, which is implemented through the following steps S601 and S602:

In step S601, the user's modification information for the virtual image is obtained, and a new virtual image is obtained according to the modification information.

Step S602, upload the above-mentioned new virtual image to the augmented reality scene to update the virtual image stored on the cloud.

Among them, the user can modify any one or more aspects of the virtual image, such as modifying text content, replacing pictures, modifying animation effects, moving positions, and so on. After the second terminal modifies and obtains the new virtual image, it is uploaded to the cloud, and the cloud updates the original virtual image, which can overwrite the original virtual image, and can also save the new virtual image and the original virtual image at the same time. After that, other terminals can obtain new virtual images when downloading augmented reality scenes from the cloud.

In addition, the method process shown in FIG. 6 may also be executed by the first terminal or any other terminal. For example, any user who enters the augmented reality scene can modify the virtual image in it; or the user sets a password when editing and generating the virtual image, and other users can modify it after entering the correct password, and so on.

In summary, in this exemplary embodiment, the first terminal obtains the editing information for the target object input by the user, generates a virtual image of the target object according to the editing information, and uploads it to the augmented reality scene stored in the cloud; After acquiring the augmented reality scene, the second terminal can display the virtual image or its prompt information through relocation. On the one hand, it provides an interactive way that allows users to freely edit virtual content, and different virtual elements can be added based on the same real scene, thereby improving the diversified experience of augmented reality. On the other hand, by editing and displaying virtual images, it provides a channel for information transmission and sharing, which is highly practical and interesting. On the other hand, the data related to the augmented reality scene is stored in the cloud, thereby saving the storage space of the terminal and improving the operation efficiency, and the data is stored and read in units of the augmented reality scene, ensuring the privacy of information.

Exemplary embodiments of the present disclosure also provide a virtual image realization device, which may be installed in the above-mentioned first terminal 110. As shown in FIG. 7, the virtual image realization device 700 may include a processor 710 and a memory 720. Among them, the memory 720 stores the following program modules:

The obtaining module 721 is used to obtain edit information for the target object input by the user;

The generating module 722 is configured to generate a virtual image of the target object according to the above-mentioned editing information;

The upload module 723 is configured to upload the above-mentioned virtual image to an augmented reality scene, the augmented reality scene is stored in the cloud, so that the second terminal obtains the augmented reality scene from the cloud to display the virtual image;

The processor 710 is configured to execute the foregoing program modules.

In an optional implementation manner, the acquisition module 721 is further configured to trigger the display of an editing interface when a target object is scanned from a real scene, and the editing interface is used to input editing information.

In an optional implementation manner, the upload module 723 is also used to select at least one key frame image from the collected images of the target object when uploading the virtual image, and upload it to the augmented reality scene.

In an optional implementation manner, the generating module 722 is further configured to generate prompt information of the virtual image according to the position of the target object in the augmented reality scene, and the prompt information is used to prompt the orientation of the virtual image. The upload module 730 is also used to upload the prompt information to the augmented reality scene.

In an optional implementation manner, the generating module 722 is configured to generate a virtual indicating arrow pointing from the area to the virtual image for each area in the augmented reality scene.

In an optional implementation manner, the above-mentioned editing information may include any one or more of text, pictures, hand-drawn drawings, animation effects, video, and audio; in addition, the above-mentioned editing information may also include the display position of the virtual image. .

In an optional implementation manner, the generating module 722 is also used to scan the actual scene where the target object is located to generate map data. The upload module 723 is also used to upload map data to the cloud to establish an augmented reality scene corresponding to the real scene.

Exemplary embodiments of the present disclosure also provide another virtual image realization device, which can be set in the above-mentioned first terminal 110. As shown in FIG. 8, the virtual image realization device 800 may include:

The obtaining module 810 is used to obtain edit information for the target object input by the user;

The generating module 820 is configured to generate a virtual image of the target object according to the above-mentioned editing information;

The upload module 830 is configured to upload the above-mentioned virtual image to an augmented reality scene, which is stored in the cloud, so that the second terminal obtains the augmented reality scene from the cloud to display the virtual image.

In an optional implementation manner, the acquisition module 810 is further configured to trigger the display of an editing interface when a target object is scanned from a real scene, and the editing interface is used to input editing information.

In an optional implementation manner, the upload module 830 is further configured to select at least one key frame image from the collected images of the target object when uploading the virtual image, and upload it to the augmented reality scene.

In an optional implementation manner, the generating module 820 is further configured to generate prompt information of the virtual image according to the position of the target object in the augmented reality scene, and the prompt information is used to prompt the orientation of the virtual image. The upload module 730 is also used to upload the prompt information to the augmented reality scene.

In an optional implementation manner, the generating module 820 is configured to generate a virtual indicating arrow pointing from the area to the virtual image for each area in the augmented reality scene.

In an optional implementation manner, the generating module 820 is also used to scan the actual scene where the target object is located to generate map data. The upload module 830 is also used to upload map data to the cloud to establish an augmented reality scene corresponding to the real scene.

Exemplary embodiments of the present disclosure also provide a virtual image realization device, which may be installed in the above-mentioned second terminal 120. As shown in FIG. 9, the virtual image realization device 900 may include a processor 910 and a memory 920. Among them, the memory 920 stores the following program modules:

The acquiring module 921 is configured to acquire an augmented reality scene from the cloud, the augmented reality scene includes at least a virtual image of a target object, and the virtual image is uploaded to the augmented reality scene by the first terminal;

The relocation module 922 is used to perform relocation based on the augmented reality scene, and when the relocation is successful, display a virtual image or a prompt message of the virtual image;

The processor 910 is configured to execute the foregoing program modules.

In an optional implementation manner, the relocation module 922 is configured to:

Obtain the key frame image of the target object from the augmented reality scene;

Match the collected current image with the key frame image, and display a virtual image when the matching is successful.

Perform relocation based on the augmented reality scene, and when the relocation is successful, determine the current pose of the second terminal in the augmented reality scene;

Determine the current field of view area of the second terminal according to the current pose;

When the display position of the virtual image is within the current view area, display the virtual image;

When the display position of the virtual image is outside the current field of view, prompt information of the virtual image is displayed.

Match the collected current image with the map data of the augmented reality scene, and determine the relocation result according to the matching result.

Obtain the display position of the virtual image from the augmented reality scene; or determine the position of the target object as the display position of the virtual image.

In an optional implementation manner, the virtual image realization apparatus 900 may further include:

The modification module is used to obtain the user's modification information for the virtual image, obtain a new virtual image according to the modification information, and upload the new virtual image to the augmented reality scene to update the virtual image stored on the cloud.

Exemplary embodiments of the present disclosure also provide another virtual image realization device, which can be set in the above-mentioned second terminal 120. As shown in FIG. 10, the virtual image realization device 1000 may include:

The acquiring module 1010 is configured to acquire an augmented reality scene from the cloud, the augmented reality scene includes at least a virtual image of a target object, and the virtual image is uploaded to the augmented reality scene by the first terminal;

The relocation module 1020 is used to perform relocation based on the augmented reality scene, and when the relocation is successful, display a virtual image or a prompt message of the virtual image.

In an optional implementation manner, the relocation module 1020 is configured to:

In an optional implementation manner, the virtual image realization apparatus 1000 may further include:

The specific details of each part of the above-mentioned virtual image realization device 700, virtual image realization device 800, virtual image realization device 900 and virtual image realization device 1000 have been described in detail in the method part implementation. For undisclosed details, please refer to the method part. The content of the implementation mode will not be repeated here.

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to make the terminal device Perform the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification. The program product can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or device.

The program product can adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Type programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

The computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than a readable storage medium, and the readable medium may send, propagate, or transmit a program for use by or in combination with the instruction execution system, apparatus, or device.

The program code contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

The program code for performing the operations of the present disclosure can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural programming. Language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computing device, partly on the user's device, executed as an independent software package, partly on the user's computing device and partly executed on the remote computing device, or entirely on the remote computing device or server Executed on. In the case of a remote computing device, the remote computing device can be connected to a user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (for example, using Internet service providers). Business to connect via the Internet).

It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "Circuit", "Module" or "System". Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptive changes of the present disclosure. These variations, uses, or adaptive changes follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field that are not disclosed in the present disclosure. . The description and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

It should be understood that the present disclosure is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

A virtual image realization method applied to a first terminal, characterized in that the method includes:

Obtain the editing information for the target object input by the user;

Generating a virtual image of the target object according to the editing information;

The virtual image is uploaded to the augmented reality scene where the target object is located, and the augmented reality scene is stored in the cloud, so that the second terminal obtains the augmented reality scene from the cloud to display the virtual image.
The method according to claim 1, characterized in that, before obtaining the editing information, the method further comprises:

When the target object is scanned from the real scene, the display of the editing interface is triggered, and the editing interface is used to input editing information.
The method according to claim 2, wherein when uploading the virtual image, the method further comprises:

At least one key frame image is selected from the collected images of the target object and uploaded to the augmented reality scene.
The method according to claim 1, wherein the method further comprises:

Generating prompt information of the virtual image according to the position of the target object in the augmented reality scene, where the prompt information is used to prompt the orientation of the virtual image;

Upload the prompt information to the augmented reality scene.
The method according to claim 4, wherein the generating the prompt information of the virtual image according to the position of the target object in the augmented reality scene comprises:

For each area in the augmented reality scene, a virtual indicator arrow pointing from the area to the virtual image is generated.
The method according to claim 1, wherein the editing information includes any one or more of text, pictures, hand-drawn drawings, animation effects, video, and audio, and the editing information further includes the virtual image Display position.
The method according to claim 1, wherein the method further comprises:

Scanning the real scene where the target object is located to generate map data;

Upload the map data to the cloud to establish an augmented reality scene corresponding to the real scene.
A virtual image realization method applied to a second terminal, characterized in that the method includes:

Acquiring an augmented reality scene from the cloud, where the augmented reality scene includes at least a virtual image of a target object, and the virtual image is uploaded to the augmented reality scene by the first terminal;

Relocation is performed based on the augmented reality scene, and when the relocation is successful, the virtual image or prompt information of the virtual image is displayed.
The method according to claim 8, wherein the performing relocation based on the augmented reality scene, and when the relocation is successful, displaying the virtual image or the prompt information of the virtual image comprises:

Acquiring a key frame image of the target object from the augmented reality scene;

Match the collected current image with the key frame image, and when the matching is successful, display the virtual image.
The method according to claim 8, wherein the performing relocation based on the augmented reality scene, and when the relocation is successful, displaying the virtual image or the prompt information of the virtual image comprises:

Perform relocation based on the augmented reality scene, and when the relocation is successful, determine the current pose of the second terminal in the augmented reality scene;

Determining the current view area of the second terminal according to the current pose;

Displaying the virtual image when the display position of the virtual image is within the current view area;

When the display position of the virtual image is outside the current view area, display prompt information of the virtual image.
The method according to claim 10, wherein the relocation based on the augmented reality scene comprises:

Match the collected current image with the map data of the augmented reality scene, and determine the relocation result according to the matching result.
The method according to claim 8, wherein the method further comprises:

Acquiring the display position of the virtual image from the augmented reality scene; or

The position of the target object is determined as the display position of the virtual image.
The method according to claim 8, wherein the method further comprises:

Acquiring user modification information for the virtual image, and obtaining a new virtual image according to the modification information;

Uploading the new virtual image to the augmented reality scene to update the virtual image stored on the cloud.
A virtual image realization device is provided in a first terminal, and is characterized in that the device includes a processor, and the processor is configured to execute the following program modules stored in a memory:

The obtaining module is used to obtain the editing information for the target object input by the user;

A generating module, configured to generate a virtual image of the target object according to the editing information;

The upload module is configured to upload the virtual image to the augmented reality scene where the target object is located, and the augmented reality scene is stored in the cloud, so that the second terminal displays the augmented reality scene by acquiring the augmented reality scene from the cloud. The virtual image.
The device according to claim 14, wherein the generating module is further configured to generate prompt information of the virtual image according to the position of the target object in the augmented reality scene, and the prompt information is used To prompt the orientation of the virtual image;

The upload module is also used to upload the prompt information to the augmented reality scene.
A device for realizing a virtual image, which is provided in a second terminal, is characterized in that the device includes a processor, and the processor is configured to execute the following program modules stored in a memory:

An acquisition module, configured to acquire an augmented reality scene from the cloud, the augmented reality scene includes at least a virtual image of a target object, and the virtual image is uploaded to the augmented reality scene by the first terminal;

The relocation module is configured to perform relocation based on the augmented reality scene, and when the relocation is successful, display the virtual image or prompt information of the virtual image.
The device according to claim 16, wherein the relocation module is configured to:

Acquiring a key frame image of the target object from the augmented reality scene;

Match the collected current image with the key frame image, and when the matching is successful, display the virtual image.
The device according to claim 16, wherein the relocation module is configured to:

Perform relocation based on the augmented reality scene, and when the relocation is successful, determine the current pose of the second terminal in the augmented reality scene;

Determining the current view area of the second terminal according to the current pose;

Displaying the virtual image when the display position of the virtual image is within the current view area;

When the display position of the virtual image is outside the current view area, display prompt information of the virtual image.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 13 when the computer program is executed by a processor.
A terminal device, characterized in that it comprises:

processor;

A memory for storing executable instructions of the processor; and

monitor;

Wherein, the processor is configured to execute the method according to any one of claims 1 to 13 by executing the executable instructions.