JP2024512447A - Data generation method, device and electronic equipment - Google Patents

Abstract

A data generation method, apparatus, and electronic device are disclosed herein. The method includes the steps of acquiring first image data, the first image data being data representing a real environment in which a user is located, and acquiring category information and plane information of a target object. the target object is an object in the first image data, and the planar information includes information on the outer surface of the target object; and the step of obtaining the second image data, the step of obtaining the second image data. The image data is data including a virtual object, and the step of generating target image data by mixing the first image data and the second image data based on the category information and the plane information. , the target image data is data including a target object and a virtual object. [Selection diagram] Figure 1

Description

This application claims priority to the application titled "Data generation method, device and electronic device" filed on April 21, 2021, and the Chinese patent application number is "202110431972.6", The entire content of this Chinese patent application is incorporated by reference into this application.

TECHNICAL FIELD The present application relates to the field of mixed reality technology, and more particularly to data generation methods, devices, and electronic devices.

Currently, mixed reality (MR) technology has been widely applied in various fields such as scientific visualization, medical training, engineering design, remote office operation, and personal entertainment. In the scene generated by mixing environmental content and virtual content, users can interact with virtual objects, allowing users to have a deeper understanding of the enjoyment of some important data in the real environment.

However, mixed reality data generated by current electronic devices is always crude. For example, simply recognizing large surfaces in the real environment, such as floors, ceilings, walls, etc., and superimposing and arranging virtual objects based on such recognized information, can eliminate the lack of detail in the scene. This has the problem of affecting the user experience.

One of the objectives of the embodiments of the present application is to provide a new technical solution for generating mixed reality data so as to improve the user's enjoyment when using electronic devices.

According to a first aspect of the present application, a data generation method is provided, the method comprising:
obtaining first image data, the first image data being data representing a real environment in which the user is located;
a step of acquiring category information and plane information of a target object, the target object being an object in the first image data, and the plane information including information on an outer surface of the target object; step and
a step of acquiring second image data, the second image data being data including a virtual object;
generating target image data by mixing the first image data and the second image data based on the category information and the plane information, wherein the target image data and a step, which is data including the virtual object.

In some embodiments, the step of generating target image data by mixing the first image data and the second image data based on the category information and the plane information includes determining a relative positional relationship between the virtual object in the second image data and the target object in the first image data based on the plane information and the relative positional relationship; and obtaining the target image data by rendering to a predetermined position of the target object.

In some embodiments, the step of obtaining the category information and the planar information of the target object includes inputting the first image data to a target image segmentation model to obtain mask information of the target object. and obtaining the category information and the plane information based on the mask information.

In some examples, the step of obtaining the category information based on the mask information includes inputting the mask information into a target category recognition model to obtain the category information.

In some embodiments, the step of obtaining the plane information based on the mask information includes obtaining a target image block corresponding to the target object in the first image data based on the mask information. , obtaining target position information of key points of the target object in a world coordinate system based on the target image block, the key points including corner points of the target object; obtaining the plane information based on information and a predetermined plane fitting algorithm, the plane information including center point coordinates and plane normal vectors corresponding to each plane of the target object; including.

In some embodiments, the method is applied to an electronic device, and the step of obtaining the target location information of the key points of the target object in the world coordinate system based on the target image block includes detecting first position information of the key point in the first image data based on an image block; position and orientation information of the electronic device at a first time including the current time; and detecting first position information of the key point in the first image data; acquiring second position information in third image data acquired at a second time that is earlier than one time; and obtaining location information.

In some embodiments, the target image segmentation model and the target category recognition model include obtaining sample data, the sample data including sample objects in a predetermined scene; and jointly training an initial image segmentation model and an initial category recognition model based on the sample data to obtain the target image segmentation model and the target category recognition model.

In some embodiments, after obtaining the target image data, the method further includes displaying the target image data.

According to a second aspect of the present application, a data generation device is provided in the present application, the device comprising:
a first image data acquisition module used to acquire first image data, the first image data being data representing a real environment in which a user is located; ,
An information acquisition module used to acquire category information and plane information of a target object, wherein the target object is an object in the first image data, and the plane information is an object on an outer surface of the target object. an information acquisition module containing information about the
a second image data acquisition module used to acquire second image data, the second image data being data including a virtual object;
A target image data generation module used to generate target image data by mixing the first image data and the second image data based on the category information and the plane information, The target image data includes a target image data generation module that is data including the target object and the virtual object.

According to a third aspect of the present application, there is provided an electronic device, the electronic device comprising the device according to the second aspect of the present application, or
The electronic device includes a memory that stores executable instructions and a processor that operates the electronic device under control of the instructions to perform the method described in the first aspect of the present application.

Regarding the beneficial effects of the present application, according to the embodiments of the present application, an electronic device acquires first image data representing a real environment in which a user is located, and obtains plane information and category of a target object in the first image data. information, and then obtain second image data that includes a virtual object, thereby mixing the first image data and second image data based on the plane information and the category information to create a target object. It is possible to obtain target image data that simultaneously includes a virtual object and a virtual object. According to the method provided in this embodiment, by recognizing the outer surface information and category information of the target object, when an electronic device constructs mixed reality data, the category information and plane information of the target object are combined. Based on the virtual environment, it can be accurately combined with the virtual objects aggregated in the virtual environment, improving the definition of the built target image data, further improving the user experience and improving the user's enjoyment when using electronic devices. can be done.

Other features of the present application and its advantages will become apparent from the detailed description of exemplary embodiments of the present application with reference to the following drawings.

The drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
1 is a schematic flowchart of a data generation method provided in an embodiment of the present application. FIG. 1 is a principle block diagram of a data generation device provided in an embodiment of the present application. 1 is a schematic hardware configuration diagram of an electronic device provided in an embodiment of the present application.

Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative settings of members and steps, numerical formulas, and numerical values described in these examples do not limit the scope of the present application unless otherwise specified.

The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way used as any limitation on the present application and its application or use.

A detailed discussion of techniques, methods and devices known to those skilled in the art may not be provided, but where appropriate, said techniques, methods and devices should be considered as part of the specification. .

In all examples shown and discussed herein, any specific values should be construed as merely illustrative rather than limiting. Accordingly, other examples of example embodiments may have different values.

It should be noted that in the following drawings, like symbols and alphabets indicate similar things. Therefore, once something is defined in one drawing, there is no need to discuss it further in the next drawing.

Current electronic devices often only recognize large surfaces in a real environment and cannot recognize objects and object types in a real environment when generating mixed reality data. For example, after collecting image data in a real environment, an electronic device does not know that one surface in the image data corresponds to a table and the other surface corresponds to a chair. As a result, the mixed reality scene obtained by combining virtual contents based on the image data appears rough. Existing methods for generating mixed reality data lack precision, as relative positional relationships cannot be accurately determined, and virtual objects are simply superimposed and displayed at certain positions in the real image environment. There are issues with this being lacking and potentially impacting the user experience.

To solve the above problems, a data generation method is provided in an embodiment of the present application. Reference is made to FIG. 1, which is a schematic flowchart of a data generation method provided in an embodiment of the present application. By applying the method to an electronic device, the device can generate high-definition mixed reality data and display the data for viewing by a user, thereby improving the user experience.

Note that in this embodiment, the electronic device that implements the method can include, for example, a display device including a display screen and at least two image collecting devices for collecting real environment information. When implemented specifically, the image collection device is capable of capturing monochrome images with a collection range of around 153° x 120° x 167° (H x V x D), a resolution of 640 x 480 or more, and a frame rate of 30 Hz or more. Of course, cameras of other configurations may be used as needed, but the wider the collection area, the greater the optical distortion of the camera, which may affect the accuracy of the final data. There is. When specifically implemented, the electronic device may be a device such as a VR device, an AR device, or an MR device, for example.

As shown in FIG. 1, the method of the present example may include steps S1100 to S1400, as described in detail below.

In step S1100, first image data is obtained, the first image data representing a real environment in which a user is located.

Specifically, the first image data may be data that reflects the real environment in which the user is located, that is, the real physical environment. This image data may include various real objects in the real environment, such as sofas, dining tables, trees, buildings, cars, roads, etc., depending on the different scenes in which the user is located, for example.

In this embodiment, the first image data may be generated by collecting data in the real environment where the user is located, by at least two image collecting devices installed in the electronic device. Of course, in the case of specific implementation, the first image data may be generated by collecting data of the real environment in which the user is located using other equipment other than the electronic equipment, depending on actual needs. For example, the first image data may be collected and acquired by an image collecting device installed independently in the environment where the user is located, and the first image data may be acquired by establishing a connection with the electronic device. The first image data may be supplied to the electronic device, and the manner in which the first image data is obtained is not particularly limited in this embodiment.

In step S1200, category information and plane information of the target object are obtained, where the target object is the target object in the first image data, and the plane information includes information on the outer surface of the target object.

In this embodiment, the target object may be one or more objects in the first image data that correspond to real objects in the real environment. For example, the objects may correspond to objects such as tables, chairs, sofas, etc. in the real environment.

The plane information of the target object may be information about the outer surface of the target object, and specifically may be information representing attributes such as the position and dimensions of the outer surface of the target object. For example, the information is center coordinate data of an outer surface of the target object and a normal vector of the outer surface to simultaneously represent the position and dimensions of the outer surface.

The category information of the target object may be information indicating the object type to which the target object belongs. For example, if the target object is "sofa", the category information may be "furniture", and the category information may be information indicating the object type to which the target object belongs. ”. When specifically implemented, the category information of the target object can be set as necessary. For example, the information may be information about a major classification to which the object belongs, or information about a minor classification to which the object belongs. Further, the category information may be expressed using an identifier of the type to which the object belongs. For example, furniture may be represented by "0" and sofa may be represented by "1", but these are omitted here.

In one embodiment, the step of obtaining the category information and plane information of the target object includes inputting the first image data to a target image segmentation model and obtaining mask information of the target object; acquiring the category information and the plane information based on mask information.

In this embodiment, the above-described step of acquiring the category information based on the mask information includes inputting the mask information to a target category recognition model and acquiring the category information.

In the field of digital image processing, mask information may specifically be information for shielding an image (in whole or in part) awaiting processing to control the image processing area or processing process. good. In a specific implementation, the mask may be a two-dimensional matrix array or a multivalued image for extracting a region of interest to the user in the image waiting to be processed, that is, a region of interest to the user. For example, by multiplying the mask by the image to be processed, the image values of other areas of the image to be processed become 0, and the image values of the areas of interest to the user do not change.

Specifically, in this example, mask information of a target object is acquired using a target image segmentation model obtained by pre-training, and then, based on the mask information, target category recognition obtained by pre-training is performed. The category information of the target object is recognized by the model, and the plane information of the target object is calculated and obtained based on the mask information. Below, we will first explain how to train and obtain a target image segmentation model and a target category recognition model.

In this embodiment, the target image separation model is a model for separating an object from a carrier. For example, in order to perform a subsequent virtual-actual combination process using a target object, the target object is separated from its carrier image. do. In particular implementations, the target image segmentation model may be a convolutional neural network model, such as a model based on a Mask R-CNN network structure, and is not particularly limited here.

The target category recognition model is a model for recognizing the category to which an object corresponding to the mask information belongs based on input mask information. For example, if the target object is a sofa, the target category recognition model recognizes the category to which the object corresponding to the mask information belongs. By inputting it into the target category recognition model, it is possible to determine that the category is "furniture" and furthermore, it is possible to recognize it as "sofa." In a concrete implementation, the target category recognition model may also be a convolutional neural network model, the model structure of which will be omitted here.

In this embodiment, the target image segmentation model and the target category recognition model are created by obtaining sample data, which is data including sample objects in a predetermined scene, and based on the sample data, an initial image segmentation model and an initial jointly training a category recognition model to obtain the target image segmentation model and the target category recognition model.

When implemented specifically, environmental image data in different scenes can be acquired in advance as sample data. For example, environmental image data in 128 types of predetermined scenes can be acquired, and environmental image data in each environmental image data can be acquired in advance. By manually marking objects, we can obtain sample data for training the target image segmentation model and target category recognition model. After that, based on the sample data, the target image segmentation model and the target category recognition model are trained by jointly training the initial image segmentation model and the initial category recognition model corresponding to the target image segmentation model and the target category recognition model, respectively. can be obtained.

In one embodiment, the above-described step of jointly training an initial image segmentation model and an initial category recognition model based on the sample data to obtain the target image segmentation model and the target category recognition model includes inputting the sample mask information into the initial image segmentation model to obtain sample mask information for the sample object; inputting the sample mask information into the initial category recognition model to obtain sample category information for the sample object; , in the training process, adjusting parameters of the initial image segmentation model and the initial category recognition model to obtain the target image segmentation model and the target category recognition model that satisfy a predetermined convergence condition; including.

Specifically, after acquiring sample data, the sample mask information of the sample object is acquired by inputting the sample data into an initial image segmentation model. Then, the initial category recognition model is reused to process the sample mask information to obtain the sample category information of the sample object, and in the process of joint training, a loss function corresponding to the two models is designed, and the two models are By constantly adjusting parameters corresponding to the models, a target image segmentation model and a target category recognition model that satisfy a predetermined convergence condition are obtained. Among them, the predetermined convergence condition can, for example, prevent the error between the recognition results of the two models from exceeding a predetermined threshold, and the detailed process regarding model training is explained in detail in the conventional technology. Since there are many, I will omit it here.

The above has explained how to train and acquire the target image separation model and the target category recognition model, but when it is specifically implemented, the target object in the first image data is determined based on the target image separation model. In the process of obtaining the category information of the target object based on the mask information, it is also possible to obtain the plane information of the target object based on the mask information. Hereinafter, how to acquire the plane information will be explained in detail.

In one embodiment, the above-described step of obtaining the plane information based on the mask information includes obtaining a target image block corresponding to the target object in the first image data based on the mask information. , obtaining target position information in a world coordinate system of key points of the target object based on the target image block, wherein the key points include corner points of the target object; , obtaining the plane information including center point coordinates and plane normal vectors corresponding to each plane of the target object, based on the target position information and a predetermined plane fitting algorithm.

The target image block is an image block composed of pixels for forming a target object in the first image data.

Specifically, in order to accurately recognize the information on the outer surface of the target object and increase the definition of the target image data waiting to be acquired, in this embodiment, the target image block corresponding to the target object in the first image data is After the acquisition, the target position information of each key point, for example, a corner point constituting the target object, that is, the three-dimensional position coordinate of each key point in the real world coordinate system can be detected and acquired. Thereafter, a predetermined plane fitting algorithm can be reused to fit the information of each outer surface of the target object to obtain said plane information.

Note that the predetermined plane fitting algorithm may be, for example, a plane fitting algorithm based on the least squares method, or may be another algorithm, and is not particularly limited here.

In one embodiment, when acquiring target position information of a key point of the target object in a world coordinate system based on the target image block, the electronic device is configured to acquire target position information of the key point of the key point based on the target image block. detecting first position information in one image data, position and orientation information of the electronic device at a first time including the current time, and position and orientation information of the electronic device acquired at a second time earlier than the first time of the key point; 3 image data, and acquiring the target position information based on the first position information, the position/orientation information, and the second position information.

The first position information may be two-dimensional coordinate data of key points of the target object in the first image data. The position and orientation information of the electronic device can be calculated and acquired based on the system parameters of the image acquisition device included in the electronic device, and will not be described here.

The second position information may be image data of a key point of the target object collected at a historical point in time before the current point in time, that is, two-dimensional coordinate data in a historical image frame.

When implemented specifically, the position trajectory of the key point at the first time point can be predicted based on the second position information of the key point at the second time point, and the first position information can be corrected based on the position trajectory. do it like this. Finally, target position information in the world coordinate system of the key point, that is, three-dimensional coordinate data, can be obtained based on the first position information and the position and orientation information of the electronic device.

After step S1200, step S1300 is executed to obtain second image data that is data including a virtual object.

The virtual object may be an object that does not exist in the real environment where the user is located, that is, it may be virtual content, for example, it may be animals, plants, buildings, etc. in the virtual world, and is not particularly limited here.

Note that in this example, the first image data including the target object and the second image data including the virtual object may be two-dimensional data or three-dimensional data, and in this example, there are no particular limitations. Not done.

In step S1400, the first image data and the second image data are mixed based on the category information and the plane information to generate target image data that is data including the target object and the virtual object. do.

Specifically, through the steps described above, the plane information and category information of the target object in the first image data reflecting the real environment in which the user is located are acquired, and the second image data including the virtual object waiting to be mixed is acquired. After obtaining the target object in the real environment and the virtual object in the virtual environment, the target object in the first image data is divided and mixed with the virtual object in the second image data based on the plane information and the category information. Obtain target image data that simultaneously includes objects.

In one embodiment, the step of generating target image data by mixing the first image data and the second image data based on the category information and the plane information includes determining a relative positional relationship between the virtual object in the second image data and the target object in the first image data based on the plane information and the relative positional relationship; and obtaining the target image data by rendering to a predetermined position of the target object.

After obtaining target image data in which a target object and a virtual object are mixed through the above processing, the method further includes the step of displaying the target image data.

Specifically, in order to facilitate the user's interaction with the virtual object in the virtual environment based on the target object in the real environment, after acquiring the target image data, the electronic device displays the target image data on its display screen. Further, the user may further obtain interaction content for interacting with the virtual object based on the displayed target image data, for example, if the virtual object is a cat, the user may interact with the virtual object based on the displayed target image data. can interact with and save the corresponding interaction video.

In order to further improve the user's enjoyment when using the electronic device, the electronic device further includes a network module, and after connecting to the Internet through the network module, the electronic device can also store image data and/or video data, etc. It is possible to save the interaction data of the interaction between the user and the virtual object in the target image data, such as , and provide the interaction data to other users, for example, so that the user's friends can view it, and provide detailed processing instructions. I will omit the details here. Of course, the above is just an example of applying the method provided in this embodiment, and when specifically implemented, the method can be applied to wall stickers, social networks on networks, virtual remote offices, personal games, advertisements, etc. It can also be applied to the following scenes, so it will be omitted here.

As described above, according to the data generation method provided in this embodiment, an electronic device acquires first image data representing the real environment in which a user is located, and obtains plane information of a target object in the first image data. and category information, and then, by obtaining second image data including the virtual object, the first image data and the second image data are mixed based on the plane information and the category information, Target image data that simultaneously includes a target object and a virtual object can be obtained. The method provided in this embodiment allows the electronic device to construct mixed reality data by recognizing the outer surface information and category information of the target object, based on the category information and plane information of the target object. , can be accurately combined with virtual objects aggregated in the virtual environment, improve the definition of the constructed target image data, and further improve the user experience.

In this embodiment, a data generation device 2000 applicable to electronic equipment is further provided, as shown in FIG. 2, corresponding to the above method embodiment. Specifically, it may include a first image data acquisition module 2100, an information acquisition module 2200, a second image data acquisition module 2300, and a target image data generation module 2400.

The first image data acquisition module 2100 is used to acquire first image data, where the first image data is data representing a real environment in which a user is located.

The information acquisition module 2200 is used to acquire category information and plane information of a target object, where the target object is an object in the first image data, and the plane information is an object in the first image data. Contains information about the outer surface.

In one embodiment, when acquiring the category information and plane information of the target object, the information acquisition module 2200 inputs the first image data to a target image segmentation model and acquires mask information of the target object. and obtaining the category information and the plane information based on the mask information.

In one embodiment, when acquiring the category information based on the mask information, the information acquisition module 2200 is used to input the mask information into a target category recognition model and acquire the category information.

In one embodiment, when acquiring the plane information based on the mask information, the information acquisition module 2200 acquires a target image block corresponding to the target object in the first image data based on the mask information. obtaining target position information of key points of the target object in a world coordinate system based on the target image block; and obtaining the plane information including center point coordinates and plane normal vectors corresponding to each plane, where the key points include corner points of the target object.

In one embodiment, the apparatus 2000 is applied to an electronic device, and the information acquisition module 2200 is configured to acquire the target position information of key points of the target object in a world coordinate system based on the target image block. detecting first position information of the key point in the first image data based on an image block; position and orientation information of the electronic device at a first time including the current time; and first time of the key point. acquiring second position information in third image data acquired at a second time earlier than the target position information; and acquiring the target position information based on the first position information, the position/orientation information, and the second position information. used to obtain and.

The second image data acquisition module 2300 is used to acquire second image data, where the second image data is data including a virtual object.

The target image data generation module 2400 is used to generate target image data by mixing the first image data and the second image data based on the category information and the plane information, and includes: The target image data is data including the target object and the virtual object.

In one embodiment, the target image data generation module 2400 generates target image data by mixing the first image data and the second image data based on the category information and the plane information. , determining a relative positional relationship between the virtual object in the second image data and the target object in the first image data based on the category information; and based on the plane information and the relative positional relationship. , rendering the virtual object to a predetermined position of the target object to obtain the target image data.

In one embodiment, the apparatus 2000 further includes a display module used to display the target image data after acquiring the target image data.

In this embodiment, an electronic device that can include the data generation device 2000 according to any embodiment of the present application for implementing the data generation method of any embodiment of the present application is provided, corresponding to the embodiment of the method described above. Further provided.

As shown in FIG. 3, an electronic device 3000 includes a memory 3100 that stores executable instructions, and a processor 3200 that operates the electronic device under control of the instructions to execute the data generation method of any embodiment of the present application. and further includes.

Each module of the apparatus 2000 described above may be implemented by the instructions being executed by the processor 3200 to perform a method according to any embodiment of the present application.

In a specific implementation, the electronic device 3000 may include a display device including, for example, a display screen and at least two image collecting devices for collecting real environment information. When implemented specifically, the image collection device is capable of capturing monochrome images with a collection range of around 153° x 120° x 167° (H x V x D), a resolution of 640 x 480 or more, and a frame rate of 30 Hz or more. Of course, cameras of other configurations may be used as needed, but the wider the collection area, the greater the optical distortion of the camera, which may affect the accuracy of the final data. There is. When specifically implemented, the electronic device may be a device such as a VR device, an AR device, or an MR device, for example.

The present application may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions for causing a processor to implement various aspects of the present application.

A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, electrical storage, magnetic storage, optical storage, electromagnetic storage, semiconductor storage, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage media include portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory). ), static random access memory (SRAM), portable compressed disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, machine encoding device, e.g. Includes card or groove-in-channel boss structures, and any suitable combinations of the above. As used herein, computer-readable storage media can include radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated through waveguides or other transmission media (e.g., light pulses through a fiber optic cable), or is not interpreted as an instantaneous signal itself, such as an electrical signal transmitted over a wire.

The computer readable program instructions described herein can be downloaded to each computing/processing device from a computer readable storage medium or transmitted externally via a network such as the Internet, a local area network, a wide area network, and/or a wireless network. Can be downloaded to your computer or external storage. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage on a computer readable storage medium in each computing/processing device.

Computer program instructions for performing the operations of the present application may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or the like in one or more programming languages. It may be source code or object code written in any combination. The programming languages include object-oriented programming languages such as Smalltalk, C++, and general procedural programming languages such as the "C" language and similar programming languages. Computer-readable program instructions may be executed entirely on a user computer, partially executed on a user computer, executed as a separate software package, partially executed on a user computer, partially executed on a remote computer, or May be run entirely on a remote computer or server. As related to remote computers, the remote computer can be connected to the user computer through any type of network, including a local area network (LAN) or wide area network (WAN), or using an Internet service provider. Can be connected to external computers such as connecting via the Internet. In some embodiments, state information in computer readable program instructions may be utilized to customize electronic circuits such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs). The electronic circuitry can execute computer readable program instructions to implement various aspects of the present application.

Aspects of the present application are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to example embodiments of the present application. It is to be understood that each block in the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device. These instructions, when executed by a processor of a computer or other programmable data processing device, result in an apparatus capable of implementing the functions/acts set forth in one or more blocks in the flowcharts and/or block diagrams. . These computer-readable program instructions that cause a computer, programmable data processing apparatus, and/or other device to operate in a particular manner may be stored on a computer-readable storage medium. A computer readable medium having instructions stored thereon thereby includes an article of manufacture that includes instructions for implementing various aspects of the functions/operations set forth in one or more blocks in a flowchart and/or block diagram.

the computer-readable program instructions loaded into the computer, other programmable data processing apparatus, or other device to perform a sequence of operational steps on the computer, other programmable data processing apparatus, or other device; Instructions executed by a computer, other programmable data processing apparatus, or other device to produce a process that performs the functions/acts specified in one or more blocks in a flowchart and/or block diagram. do.

The flowchart and block diagrams in the drawings illustrate possible architecture, functionality, and operation of systems, methods, and computer program products in accordance with embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, program segment, or portion of instructions that includes one or more executable instructions for implementing the predetermined logical function. In alternative implementations, the functions depicted in the blocks may occur in a different order than that depicted in the drawings. For example, two consecutive blocks can actually be executed essentially in parallel, or they can even be executed in reverse order depending on the functionality involved. It should be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented with a dedicated hardware-based system that performs a predetermined function or operation, or a dedicated hardware-based system that performs a predetermined function or operation. It can be realized by a combination of hardware and computer instructions. It is known to those skilled in the art that hardware implementations, software implementations, and combination software and hardware implementations are equivalent.

Although each embodiment of the present application has been described above, the above description is illustrative, not exhaustive, and is not limited to each disclosed embodiment. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terminology used herein has been chosen to best explain the principles, practical application, or technical improvements in the marketplace of each embodiment, or to those skilled in the art, to understand each embodiment disclosed herein. The purpose is to make it easier to understand. The scope of the application is defined by the claims appended hereto.

Claims (10)

A data generation method,
obtaining first image data, the first image data being data representing a real environment in which the user is located;
obtaining category information and plane information of a target object, wherein the target object is an object in the first image data, and the plane information includes information on an outer surface of the target object; the above steps;
a step of acquiring second image data, the second image data being data including a virtual object;
generating target image data by mixing the first image data and the second image data based on the category information and the plane information, wherein the target image data The data generation method includes the steps of: data including the virtual object. The step of generating target image data by mixing the first image data and the second image data based on the category information and the plane information,
determining a relative positional relationship between the virtual object in the second image data and the target object in the first image data based on the category information;
The method according to claim 1 , further comprising: rendering the virtual object to a predetermined position of the target object based on the plane information and the relative positional relationship to obtain the target image data. The step of obtaining the category information and plane information of the target object includes:
inputting first image data into a target image segmentation model to obtain mask information of the target object;
The method of claim 1, comprising: obtaining the category information and the plane information based on the mask information. The step of obtaining the category information based on the mask information includes:
4. The method of claim 3, comprising inputting the mask information into a target category recognition model to obtain the category information. The step of acquiring the plane information based on the mask information,
obtaining a target image block corresponding to the target object in the first image data based on the mask information;
obtaining target position information of key points of the target object in a world coordinate system based on the target image block, the key points including corner points of the target object;
obtaining the plane information based on the target position information and a predetermined plane fitting algorithm, the plane information including center point coordinates and plane normal vectors corresponding to each plane of the target object; 4. The method of claim 3, comprising the steps of: 6. The method according to claim 5, wherein the method is applied to an electronic device, and the step of obtaining the target position information of the key points of the target object in the world coordinate system based on the target image block. teeth,
detecting first position information of the key point in the first image data based on the target image block;
acquiring position and orientation information of the electronic device at a first time including the current time, and second position information in third image data acquired at a second time prior to the first time of the key point; ,
6. The method of claim 5, comprising obtaining the target position information based on the first position information, the position and orientation information, and the second position information. The target image segmentation model and the target category recognition model are:
obtaining sample data, the sample data including sample objects in a predetermined scene;
5. The target image segmentation model and the target category recognition model are obtained by training an initial image segmentation model and an initial category recognition model based on the sample data. The method described. 2. The method of claim 1, wherein after acquiring the target image data, the method further comprises:
2. The method of claim 1, comprising displaying the target image data. A data generation device,
a first image data acquisition module used to acquire first image data, the first image data being data representing a real environment in which a user is located; ,
An information acquisition module used to acquire category information and plane information of a target object, wherein the target object is an object in the first image data, and the plane information is an object on an outer surface of the target object. an information acquisition module containing information of;
a second image data acquisition module used to acquire second image data, the second image data being data including a virtual object;
A target image data generation module used to generate target image data by mixing the first image data and the second image data based on the category information and the plane information, the A data generation device including a target image data generation module, wherein the target image data is data including the target object and the virtual object. An electronic device,
comprising an apparatus according to claim 9, or
An electronic device comprising a memory for storing executable instructions and a processor for operating the electronic device under control of the instructions to carry out the method according to any one of claims 1 to 8. .

Images