CN116912463A - 3D virtual image processing method, device, electronic device and readable storage medium - Google Patents

Abstract

The application discloses a 3D virtual image processing method, a device, electronic equipment and a readable storage medium, and relates to the technical field of computers. The 3D avatar processing method comprises the following steps: acquiring a first preview image comprising a target object; determining first information according to the first preview image, wherein the first information comprises at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, the orientation of the target object in the actual space, the number of the target objects and the gesture of the target object; determining interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information; the interaction information is used for realizing interaction between the 3D virtual image and a user, and the dynamic display information is used for generating a dynamic display picture of the 3D virtual image.

Description

3D virtual image processing method, device, electronic device and readable storage medium

Technical field

This application belongs to the field of computer technology, and specifically relates to a 3D virtual image processing method, device, electronic equipment and readable storage medium.

Background technique

With the development of animation, film and television, two-dimensional comics, animations, games, etc. are increasingly attractive to users, and collectible models such as hand-made models are an emotional extension of animation images. In recent years, the collection of physical toys such as hand models has become increasingly popular. However, the current gameplay of physical toys such as hand models is usually just as a decoration, which only plays a decorative display role. The display effect is single, and the interaction with users is not flexible enough.

Contents of the invention

Embodiments of the present application provide a 3D virtual image processing method, device, electronic device and readable storage medium, which can solve the problems in related technologies of single display effect of physical toys such as figures and inflexible interaction with users.

In the first aspect, embodiments of the present application provide a 3D virtual image processing method, including:

Obtaining a first preview image including the target object;

First information is determined according to the first preview image, and the first information includes at least one of the following: the first spatial position of the target object in the actual space, the orientation of the target object in the actual space, the The number and posture of the target object;

Determine the interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

Wherein, the interaction information is used to realize the interaction between the 3D virtual image and the user, and the dynamic display information is used to generate a dynamic display screen of the 3D virtual image.

In the second aspect, embodiments of the present application provide a 3D virtual image processing device, including:

An acquisition module, used to acquire the first preview image including the target object;

A first determination module, configured to determine first information based on the first preview image, where the first information includes at least one of the following: a first spatial position of the target object in the actual space, a first spatial position of the target object in the actual space, The orientation, the number of the target objects, and the posture of the target objects;

a second determination module, configured to determine the interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

Wherein, the interaction information is used to realize the interaction between the 3D virtual image and the user, and the dynamic display information is used to generate a dynamic display screen of the 3D virtual image.

In a third aspect, embodiments of the present application provide an electronic device, including a processor and a memory. The memory stores programs or instructions that can be run on the processor. When the program or instructions are executed by the processor, The steps of implementing the 3D virtual image processing method as described in the first aspect.

In the fourth aspect, embodiments of the present application provide a readable storage medium. Programs or instructions are stored on the readable storage medium. When the programs or instructions are executed by a processor, the 3D virtual image as described in the first aspect is realized. Processing method steps.

In the embodiment of the present application, after obtaining the first preview image of the target object, the first information is determined based on the first preview image. The first information includes the first spatial position and orientation of the target object in the actual space. , at least one of the number and posture of the target object, determine the interactive information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information, the interactive information can realize the interaction between the 3D virtual image and In the interaction between users, the dynamic display information is used to generate a dynamic display screen of the 3D virtual image. In this way, corresponding interactive information and/or dynamic display information can be set for target objects such as figures and toys, so that the target object can interact with the user based on the corresponding 3D virtual image, or can display the target The dynamic display screen of the object's 3D virtual image can effectively improve the interaction between the target object and the user, making the interaction between the target object and the user more flexible, and can also effectively enrich the display effect of the target object and effectively improve the target object. user experience.

Description of the drawings

Figure 1 is a flow chart of a 3D virtual image processing method provided by an embodiment of the present application;

Figure 2 is a schematic diagram of a spatial point cloud in an embodiment of the present application;

Figure 3 is a structural diagram of a 3D virtual image processing device provided by an embodiment of the present application;

Figure 4 is a structural diagram of another 3D virtual image processing device provided by an embodiment of the present application;

Figure 5 is a structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the scope of protection of this application.

The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It is to be understood that the figures so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in orders other than those illustrated or described herein, and that "first," "second," etc. are distinguished Objects are usually of one type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the related objects are in an "or" relationship.

The 3D virtual image processing method, device and electronic equipment provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios.

Please refer to FIG. 1 , which is a flow chart of a 3D virtual image processing method provided by an embodiment of the present application. The method is applied to electronic devices. As shown in Figure 1, the method includes the following steps:

Step 101: Obtain a first preview image including the target object.

It should be noted that the objects involved in the embodiments of the present application may include toys or ornaments such as figures and dolls, and may also include people or animals. For example, the objects may be some specific roles cosplayed by people. or image etc. Among them, the figure can also be called the first model or the first edition. It is a high-precision, small-size sculpture based on animation, film, television, and game characters. It was originally called a GARAGE KIT (GK) model. Figures are usually peripheral derivatives of animation, movies, games, etc. They are the materialization of virtual movie and game character images in real life. For example, the target object may refer to one or more figures, for example, it may be all figures in a certain series.

In this embodiment of the present application, the first preview image of the target object may be obtained through a camera. For example, the user can turn on the camera of the electronic device and aim the camera at the target object, and then the target object will be displayed in the preview interface of the camera, thereby obtaining the first preview image of the target object.

Step 102: Determine first information based on the first preview image. The first information includes at least one of the following: the first spatial position of the target object in the actual space, the orientation of the target object in the actual space, the Describe the number of target objects and the posture of the target object.

For example, after acquiring the first preview image of the target object, the electronic device may establish a spatial coordinate system using the camera of the electronic device as the coordinate origin, and then determine the target object based on the first preview image collected by the camera. The first information corresponding to the target object, such as the coordinate position of the target object in the spatial coordinate system, is used to determine the first spatial position of the target object based on the coordinate position. Wherein, the first spatial position may include the spatial coordinates and/or orientation of the target object in a spatial coordinate system.

It can be understood that information such as the number, posture, or orientation of the target object in the actual space may also be obtained based on the first preview image.

Step 103: Determine the interaction information and/or dynamic display information of the first 3D virtual image corresponding to the target object based on the first information.

Wherein, the interaction information is used to realize the interaction between the 3D virtual image and the user, and the dynamic display information is used to generate a dynamic display screen of the 3D virtual image.

Optionally, after determining the first spatial position of the target object in the actual space, the second spatial position of the first 3D virtual image corresponding to the target object in the virtual three-dimensional space may be determined based on the first spatial position. . The virtual three-dimensional space may be rendered according to the actual space, or may be a specified virtual three-dimensional space.

For example, after the electronic device determines the first spatial position of the target object in the actual space based on the first preview image collected by the camera, the electronic device may render a virtual three-dimensional space corresponding to the actual space based on the first preview image, and generate the virtual three-dimensional space according to the first preview image. The first spatial position of the target object in the actual space is determined, and the second spatial position of the 3D virtual image corresponding to the target object in the virtual three-dimensional space is determined.

Alternatively, the virtual three-dimensional space is a designated virtual three-dimensional space, and the electronic device may determine the second spatial position of the 3D virtual image corresponding to the target object in the virtual three-dimensional space based on the user's input operation. For example, after determining the first spatial position of the target object in the actual space, the electronic device may use the display interface of the electronic device as the designated virtual three-dimensional space, so that the user can intuitively know the location of the virtual three-dimensional space. Further, the user may place the 3D virtual image corresponding to the target object at a second spatial position in the virtual three-dimensional space. For example, the 3D virtual image may be displayed on a display interface of an electronic device. The electronic device may display the 3D virtual image based on the user's response to the target object. The drag operation of the 3D virtual image places the 3D virtual image in the second spatial position of the virtual three-dimensional space, thereby making the operation of the 3D virtual image more flexible and convenient. It should be noted that the 3D virtual image of the target object may be pre-constructed. Optionally, before step 103, the method may also include the following steps:

Obtaining multiple image frames obtained by shooting around the target object;

Extract the key points of each image frame, map the key points to the virtual three-dimensional space, and obtain the spatial orientation information of the key points in the virtual three-dimensional space;

Based on the spatial orientation information, a 3D virtual image of the target object in the virtual three-dimensional space is constructed.

It should be noted that in order to allow a target object (such as a figure) to appear in a virtual three-dimensional space as a 3D virtual image, it is first necessary to construct a three-dimensional space point cloud of the target object.

In the embodiment of the present application, the target object may be captured by a camera (such as a monocular camera) of an electronic device to obtain multiple image frames from different angles of the target object; further, the key points of each image frame may be extracted. For example, it can be based on Simultaneous Localization and Mapping (SLAM) technology to extract the key points of each image frame, and map these 2D key points to the virtual three-dimensional space to form points in the space. Then based on the key point matching between image frames, the spatial orientation information of different key points in the virtual three-dimensional space is obtained. Based on the spatial orientation information of these key points, a spatial point cloud is constructed, and then the target object is obtained. 3D virtual image in virtual three-dimensional space. As shown in Figure 2, the target object is a kettle. Based on the above method, the spatial point cloud of the kettle can be constructed, thereby obtaining the 3D virtual image of the kettle.

It should be noted that when there are multiple target objects, 3D virtual images corresponding to the multiple target objects in the same virtual three-dimensional space may be constructed. For example, taking the target object of a certain series of figures as an example, a series usually includes multiple figures, and the multiple figures have specific placement positions, that is, these figures have certain characteristics in the actual space. A specific spatial position; in this case, you can perform surround shooting for each figure to obtain multiple image frames of the figure, and then build a 3D virtual image of the figure based on the above method, based on the position of the figure in the actual space In the figure series, the position and orientation of the figure in the actual space are obtained, and the 3D virtual image of the figure is mapped to the specified virtual three-dimensional space according to the position and orientation in the actual space. Position; Based on this method, the corresponding 3D virtual images of multiple figures in the figure series are constructed, and the 3D virtual images corresponding to these figures can be determined according to the position and orientation of the corresponding figures in the actual space. The position and orientation of each 3D virtual image in the virtual three-dimensional space, that is, the position and orientation of the corresponding 3D virtual image of these figures in the virtual three-dimensional space, are consistent with the position and orientation of these figures in the actual space. For example, if a certain figure A is located in front and left of figure B in the actual space, then the 3D avatar corresponding to figure A is also located in front and left of the 3D avatar corresponding to figure B in the virtual three-dimensional space.

Optionally, after constructing 3D virtual images of different target objects, the electronic device may store these 3D virtual images in the cloud. Furthermore, the electronic device can obtain 3D virtual images corresponding to different target objects from the cloud.

In this embodiment of the present application, after determining the first information, the electronic device determines the interactive information and/or dynamic display information of the 3D virtual image based on the first information. Optionally, the interactive information and the dynamic display information Display information is related to the first information. For example, for the same 3D virtual image, the target object corresponding to the 3D virtual image may correspond to different interactive information and/or dynamic display information at different first spatial positions in the actual space, that is, one target object corresponds to the 3D virtual image. The virtual three-dimensional space may have multiple second spatial locations, and different interactive information and/or dynamic display information may be set for different second spatial locations.

Alternatively, according to different orientations of the target object in the actual space, the 3D virtual image corresponding to the target object may also have different interactive information and/or dynamic display information. For example, when the front of the target object is facing left, the dynamic display information corresponding to its 3D avatar is rotated to the left; when the front of the target object is facing right, the dynamic display information corresponding to its 3D avatar is rotated to the right. exhibit.

For another example, the interaction information and/or dynamic display information of the 3D virtual image may also be related to the number of target objects. Optionally, in the case where the first information includes the number of the target object, determining the interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information includes: :

According to the number of the target objects, interaction information and/or dynamic display information corresponding to the 3D virtual image of the target object and the number are determined.

For example, if it is recognized that the first preview image includes two target objects (for example, they can be two different figures, or the same two figures), the 3D virtual images corresponding to the two target objects are correspondingly generated. The dynamic display screen is screen 1; if it is recognized that the first preview image includes three target objects, the dynamic display screen generated based on the 3D virtual images corresponding to the three target objects is screen 2.

Furthermore, in the embodiments of the present application, with different numbers of target objects, the dynamic Zhang information and/or interactive information of the 3D virtual image corresponding to the target object will be different, which will be more helpful to attract users to collect more target objects. , providing users with more ways to play and effectively improving the interaction between target objects and users.

Optionally, the interaction information and/or dynamic display information of the 3D virtual image may also be related to the posture of the target object. For example, if it is recognized that the posture of the target object in the first preview image is a standing posture, the 3D virtual image corresponding to the target object can be dynamically displayed in a standing image; if it is recognized that the posture of the target object is a lying posture, the 3D virtual image corresponding to the target object can be dynamically displayed. The 3D virtual image corresponding to the target object can be dynamically displayed as a lying image, etc.

It should be noted that the first information may include the first spatial position of the target object in the actual space, the orientation of the target object in the actual space, the number of the target objects, and the posture of the target object. At least one item, and furthermore, the specific content of the first information in the embodiments of this application can also be other possible situations besides the above, and the interactive information and/or dynamic display information of the 3D virtual image can also be other possible situations, Not too many are listed here.

In the embodiment of the present application, the interaction information and/or dynamic display information of the corresponding 3D virtual image can be determined according to the first information corresponding to the target object, so that the target object can realize interaction with the user based on the corresponding 3D virtual image. Interaction, or a dynamic display screen that can display the 3D virtual image of the target object, thereby effectively improving the interaction between the target object and the user, and effectively improving the user experience of the target object.

Optionally, step 103 may specifically include:

Get the first input information;

When the second spatial position of the 3D virtual image in the virtual three-dimensional space is determined based on the first information, interaction information and/or dynamic display information of the 3D virtual image is determined based on the first input information.

The first input information may refer to relevant information input by the user, for example, it may be interactive information and/or dynamic display information input by the user. For example, after determining the second spatial position of the 3D virtual image corresponding to the target object in the virtual three-dimensional space, the electronic device may display an input box to obtain the user's first input information.

For example, the display interface of the electronic device is used as the designated virtual three-dimensional space. When the display interface of the electronic device displays a 3D virtual image corresponding to the target object, the display interface can also display an input box, and the user can enter the 3D virtual image in the display interface based on the 3D virtual image. display position (that is, the second spatial position), enter the corresponding interactive information and/or dynamic display information in the input box. In this way, the electronic device can determine the interactive information and/or dynamic display information of the 3D virtual image based on the input information input by the user.

Optionally, the 3D virtual image corresponding to the target object includes at least one second spatial position in the virtual three-dimensional space, and different second spatial positions correspond to different interaction information and/or different dynamic display information. For example, when the 3D virtual image is located at position A, the user can input the first interaction information and the first dynamic display information. When the 3D virtual image is located at the position B, the user can input the second interaction information and the second dynamic display information, and then also It is possible to determine the corresponding interactive information and/or dynamic display information of the 3D virtual image in different second space positions. In this way, the 3D virtual image of a certain target object will have different interactive information and/or dynamic display information according to its spatial position, thereby effectively enriching the interaction between the 3D virtual image and the user, and /Or make the dynamic display of 3D virtual images more diverse and rich, bringing a better interactive experience to users.

Optionally, when the number of the target objects is at least two, the 3D virtual images corresponding to the at least two target objects include at least one relative position relationship in the virtual three-dimensional space. Different relative position relationships Correspond to different interactive information and/or different dynamic display information. For example, take the target object of multiple figures of a certain series as an example. If the series of figures includes figure 1, figure 2 and figure 3, figure 1 can have two different second space positions. 2 also has two but different second spatial positions, then there can be four relative position relationships between figure 1 and figure 2. These four relative position relationships can respectively correspond to four different interactive information and/or four Different dynamic display information; accordingly, there can be at least one relative position relationship between figure 1 and figure 3, there can also be at least one relative position relationship between figure 2 and figure 3, and there can be at least one relative position relationship between figure 1 and figure 3, and Figure 1, Figure 2 and Figure 3 may also have at least one relative position relationship, and different relative position relationships may correspond to different interactive information and/or different dynamic display information. In this way, for different relative position relationships between multiple 3D virtual images, there will be different interactive information and/or dynamic display information, which can effectively enrich the interaction and/or dynamic display effects between the 3D virtual image and the user. , bringing a better interactive experience to users.

It should be noted that these interactive information and/or dynamic display information may be determined based on user input information.

In the embodiment of the present application, after obtaining the first preview image of the target object, the first information is determined based on the first preview image. The first information includes the first spatial position and orientation of the target object in the actual space. , at least one of the number and posture of the target object, determine the interactive information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information, the interactive information can realize the interaction between the 3D virtual image and In the interaction between users, the dynamic display information is used to generate a dynamic display screen of the 3D virtual image. In this way, corresponding interactive information and/or dynamic display information can be set for target objects such as figures and toys, so that the target object can interact with the user based on the corresponding 3D virtual image, or can display the target The dynamic display screen of the object's 3D virtual image can effectively enhance the interaction between the target object and the user, and effectively improve the user experience of the target object.

Optionally, in this embodiment of the present application, after step 104, the method may further include:

Generate and display a dynamic display screen of the 3D virtual image based on the dynamic display information; and/or,

When a user interaction instruction is obtained, a response instruction is generated based on the interaction information, so as to realize interaction between the 3D virtual image and the user through the response instruction.

In the embodiment of the present application, after determining the dynamic display information of the 3D virtual image corresponding to the target object, the electronic device can generate and display the dynamic display screen of the 3D virtual image based on the dynamic display information. For example, when the electronic device receives the user's input operation, such as the triggering operation of a specific button, or the user's voice input, the electronic device generates and displays a dynamic display screen of the 3D virtual image corresponding to the target object. . The dynamic display screen may mean that the 3D virtual image is displayed in a dynamic manner. For example, the 3D virtual image may rotate or move in a specific direction at a certain speed. Furthermore, by generating and displaying the dynamic display screen of the 3D virtual image, the target object can be dynamically displayed in the 3D virtual image, thereby effectively improving the interactivity and viewing of the target object, and bringing better results to users. usage experience.

Or, after determining the interaction information of the 3D virtual image corresponding to the target object, if an interaction instruction from the user is received, a response instruction for responding to the interaction instruction is generated according to the interaction information, so as to realize the relationship between the 3D virtual image and the user. interaction. Wherein, the interaction instructions include at least one of the following: touch interaction instructions, gesture interaction instructions, and voice interaction instructions.

For example, if the user's voice interaction instruction is received, such as "Hello", the electronic device can generate a corresponding response instruction based on the interaction information. For example, the response instruction is to control the 3D virtual image to rotate or jump, etc., so as to respond to the user. Respond to voice interaction instructions to achieve interaction with the user. For another example, if the user's gesture interaction instructions, such as waving, are obtained through the camera, the electronic device can generate corresponding response instructions based on the interactive information, such as controlling the 3D virtual image to wave (if the 3D avatar is a character), thereby realizing 3D Interaction between avatar and user. In this way, the interaction between the 3D virtual image and the user can be effectively improved, and a better interactive experience can be brought to the user.

It should be noted that for different interaction instructions, the electronic device can be pre-set to generate interaction information for corresponding response instructions. Then, when the interaction instructions are obtained, the electronic device can also determine which interaction information needs to be generated based on Corresponding response instructions to implement responses to interactive instructions. For example, for different gesture actions, such as waving, making fists, comparing hearts, etc., different interactive information can be set in advance, so that different response instructions can be generated based on different interactive information, such as interactive information corresponding to waving. It is used to control the 3D virtual image to generate response instructions for waving, and the interactive information corresponding to Bixin is used to control the 3D virtual image to generate response instructions for Bixin. In this way, response instructions that respond to different interaction instructions can be generated through different interaction information, so as to ensure that the 3D virtual image can achieve accurate and effective interaction with the user's different interaction instructions, so that the 3D virtual image can interact effectively with the user. can achieve richer and more diverse interactions.

In the embodiment of the present application, after determining the dynamic display information of the 3D virtual image corresponding to the target object, the user can scan the target object through the electronic device, so that the electronic device generates and displays the dynamic display screen.

For example, the user may scan the object to be identified through a camera of the electronic device. When the electronic device starts the camera, the preview image collected by the camera will be displayed. If the camera is pointed at the object to be identified, the object to be identified will also be displayed in the preview image.

It should be noted that the object to be identified may be a target object for which dynamic display information has been determined or set; in the embodiment of the present application, it may be a dynamic display of corresponding 3D virtual images established for multiple target objects based on the foregoing method. Information, the dynamic display information of the target object and the corresponding 3D virtual image may be stored locally on the electronic device or in the cloud. When the electronic device obtains an object to be recognized in the preview image based on the camera, it may compare the object to be recognized with the target object, or it may call the target object stored in the cloud for comparison. If the object to be recognized matches the target object for which D3 virtual image dynamic display information has been established, the 3D virtual image of the target object is used as the 3D virtual image of the object to be recognized.

For example, if the target object includes a figure of anime character A, the electronic device has determined the dynamic display information of the 3D virtual image of the figure. Afterwards, when the user scans the object to be identified through the electronic device camera and determines through comparison that the object to be identified is the figure of anime character A, the 3D virtual image corresponding to the figure and the corresponding dynamic display information are obtained. Based on the The dynamic display information of the 3D virtual image of the figure generates a dynamic display screen, and displays the dynamic display screen. In this way, objects such as figures can be scanned to generate and display a dynamic display of the 3D virtual image corresponding to the object, bringing a better interactive experience to users and helping to improve the user's collection and collection of figures and other objects. viewing interest, thereby promoting the consumption of these figures and other objects.

Optionally, the obtaining includes a first preview image of the target object, including:

When the first preview image is displayed, generating an identification frame in the first preview image;

Obtain the target object in the recognition frame of the first preview image.

In the embodiment of the present application, when the electronic device starts the camera and displays the first preview image, an identification frame may be generated in the first preview image, and the identification frame may be used to guide the user to adjust the collection orientation and angle of the camera. Thus, the target object is located within the recognition frame. Alternatively, the target object may be captured through the recognition frame so that the target object is located within the recognition frame, thereby better positioning the target object. The electronic device may determine the spatial position of the target object through the position of the recognition frame, and thereby determine which position of the target object is to be detected and recognized.

It should be noted that when there are multiple target objects, the identification frame may be generated for each target object respectively, or one identification frame may be generated for all target objects. For example, if there are multiple target objects, and these target objects all belong to the same series of figures, an identification frame can be generated, and then these figures of the same series can be positioned in an identification frame, and the figures of the same series can be positioned through the identification frame. The target detection model corresponding to the figure is used for identification and detection.

Optionally, the obtaining includes a first preview image of the target object, including:

When the first preview image is displayed, obtain the object to be recognized in the first preview image;

Obtain the object category to which the object to be identified belongs;

Determine a target detection model based on the object category;

When the target detection model recognizes that the object to be recognized matches the target object, the object to be recognized is used as the target object to obtain a first preview image including the target object, wherein different objects Categories correspond to different target detection models.

For example, assuming that the object to be recognized is a figure to be recognized, first the object category to which the figure to be recognized belongs is obtained, that is, which figure series it belongs to. Optionally, the object category to which the object belongs may be determined based on user input. Understandably, for a figure to be identified, the user can independently confirm which figure series the figure to be identified belongs to. The electronic device can display an object category selection box, and the user can select the corresponding figure in the selection box. series, so that the electronic device can determine the figure series to which the figure to be recognized belongs based on the user's input, that is, determine the object category to which the object to be recognized belongs.

Further, the electronic device determines a corresponding target detection model according to the object category. Different object categories correspond to different target detection models, and then detects whether the object to be identified matches the target object through the corresponding target detection model. For example, for different figure series, the electronic device can pre-train and store the target detection model corresponding to each figure series, and then for a figure under a certain figure series, the target corresponding to the figure series can be used detection model for identification. It should be noted that each target detection model can be trained using the relevant data of each figure under the corresponding figure series. For example, it can be collected by collecting image data of each figure at different angles, different orientations, different postures, etc. It is used as training data to train the target detection model, so that the trained target detection model can recognize the figures in the corresponding figure series, thereby improving the recognition accuracy of the figures to be recognized. Wherein, the training data of the target detection model also includes image data of the target object.

In the embodiment of the present application, by training corresponding target detection models for different object categories, the target detection model corresponding to the object category can be called to identify whether the object to be identified is the same as the target object based on the object category to which the object to be identified belongs. matching, which can effectively improve the recognition accuracy of the object to be identified.

Optionally, in the case where the first information includes the posture of the target object, the method further includes:

Obtaining the posture of the target object, the posture of the target object includes a standing posture and a lying posture;

Based on the dynamic display information of the 3D virtual image corresponding to the target object, a dynamic display picture matching the 3D virtual image with the standing posture or the flat posture is displayed.

It can be understood that the target object may have different postures, such as a flat posture or a cubic posture. Assuming that the target object is in a flat posture, after generating a dynamic display screen based on the dynamic display information of the 3D virtual image corresponding to the target object, a dynamic display screen matching the 3D virtual image and the flat posture may be displayed, for example The 3D virtual image is also dynamically displayed in a flat posture.

The posture of the target object can be determined based on the width and height of the target object. If the height is greater than the width, the target object can be considered to be in an upright posture. If the height is less than the width, the target object can be considered to be in a flat posture.

In the embodiment of the present application, by obtaining the posture of the target object, a dynamic display picture of the 3D virtual image matching the posture can be displayed, thereby effectively improving the richness and flexibility of the dynamic display picture of the 3D virtual image.

For better understanding, the following takes the target object and the object to be identified as figures as examples to specifically describe the 3D virtual image processing method provided by the embodiment of the present application. The 3D virtual image processing method includes the following steps:

Step 1. Pre-construct the 3D virtual image corresponding to each figure of each figure series (or figure type).

Among them, step 1 specifically includes the following steps:

Step 11. Construct a spatial point cloud of the figure.

Understandably, in order to allow the virtual image of the figure to appear in 3D space, it is first necessary to establish a three-dimensional space point cloud of the figure. For example, spatial SLAM technology can be used to surround the figure with a monocular camera, and then the key points of each captured image frame can be extracted based on SLAM technology, and these 2D key points can be mapped into a 3-dimensional space to form a space points in the frame, and then obtain the orientation of different key points in space based on the key point matching between frames, thereby constructing a spatial point cloud. As shown in Figure 2, the spatial point cloud of the kettle is constructed based on the above method.

Step 12. Construct a 3D virtual image of the figure.

Obtain the spatial coordinates and orientation of the figure. For example, you can place a pre-designed 3D virtual image of the figure at the specified spatial coordinates and render it to the specified coordinates in the space.

Step 13. Rendering of the 3D virtual image of the figure.

The user aims at the figure through the camera of the mobile client, and the client will automatically recognize the spatial coordinates of the figure and render a three-dimensional dynamic virtual image in front of the figure in the camera display interface.

Step 2. Place the 3D virtual image of each figure at the specified virtual space coordinates and orientation.

Each 3D virtual image of the figure can have more than one designated position, and there is more than one relative position relationship (or position range) between two or more 3D virtual figures of the figure.

Step 3. Based on the virtual space coordinate relationship and orientation relationship between each figure, set the interaction logic and corresponding dynamic display effect between the 3D virtual images of each figure.

Optionally, the first interaction logic and the first dynamic display effect are set when two or more 3D virtual figures of the figures are located within the first position range; when the two or more 3D virtual figures of the figures are located within the second position range, Set the second interaction logic and the second dynamic display effect; and by analogy, set different interaction logic and dynamic display effects between the 3D virtual images of the figures with different positional relationships.

Furthermore, different figure models are set for each figure in the same figure series. Different interaction logic and dynamic display effects are set between the 3D virtual images of different figure models.

In some embodiments, the above 3D virtual images may have the same interaction logic but different dynamic display effects, or may have different interaction logic and different dynamic display effects, etc.

Among them, the interaction logic includes but is not limited to: touch interaction, gesture interaction, voice interaction, etc.;

Dynamic display effects include but are not limited to: 3D avatar actions, 3D avatar special effects, etc. The dynamic display effects are rendered at the corresponding position of the figure in the camera display interface, such as the front or peripheral position.

Step 4. Take a surround shot of the figures collected by the user to obtain the figure type, spatial coordinates and orientation information between each figure; and based on the type of figure obtained through the shooting, call the pre-built 3D virtual image of the figure, and based on The actual space coordinates and orientation information between each figure obtained by shooting are placed to place the 3D virtual image of the figure at the corresponding virtual space coordinates and orientation.

The 3D virtual image of the figure, the spatial coordinate relationship and orientation relationship between each 3D virtual image, the interaction logic between each 3D virtual image, and the corresponding dynamic display effect can all be stored in the cloud; the user communicates with the 3D virtual image through the client The cloud performs communication connections and data interaction.

Users can see the figures appearing in the real space through the mobile phone client. The client can obtain the user's gestures through the mobile phone camera, and use AI capabilities to identify the user's gesture type, such as waving, making fists, comparing heart movements, etc. According to the recognized gesture movements, the 3D virtual image of the figure is allowed to perform corresponding movements to achieve an interactive effect.

For the same figure series, if the user collects more complete figures, the richer dynamic effects it can display; for example, if only a single figure is photographed, the display of the 3D virtual image and interaction corresponding to the single figure can be triggered. Action; if you shoot two or more figures, the 3D virtual image corresponding to the set of figures, the interactive actions of each figure, and the interactive actions between each figure can be triggered.

Step 5. Realize the interaction between the 3D virtual image of the figure and the user.

In some implementation scenarios, the augmented reality (AR) effect of the 3D virtual image of each figure can be unlocked based on the collection number of figures and the combination of different models.

For example, first, the user needs to select a series of figures to be recognized, and call the corresponding recognition model according to the selected series; secondly, the user is guided to identify the figures in the specified recognition box by generating a recognition frame on the camera preview interface. Under normal circumstances, according to the selected figure series, the recognition area will be automatically delimited in the camera preview interface (that is, a recognition frame will be generated), and the user needs to place the figure to be recognized in the recognition frame. Further, the target detection model corresponding to the series of figures is called to detect the figures within the recognition frame. This target detection model is specially trained using the data of the corresponding figure models. It can specifically identify each figure in the series and return the type and number of figures that appear in the recognition box. Assuming that there are currently eight figure models in this series of figures, the input of the target detection model is the image in the detection frame, and the output result is the outer rectangular frame of the figure in the detection frame, and its corresponding figure in the eight types of figures. In which category. According to the type and number of figures returned by the detection results, the 3D virtual image corresponding to each figure model is retrieved from the cloud, the spatial coordinates of the 3D virtual image of each figure model are obtained, and the above information is displayed on the corresponding spatial coordinates. 3D virtual image, generate AR special effects. If there are 3 figures currently recognized, the user can choose to watch the AR special effects achieved by the 3D avatars corresponding to the 3 figures, that is, the dynamic display effect of the 3D avatars corresponding to the 3 figures, and/or with The dynamic display effect of interaction between users, and/or the dynamic effect of interaction between three 3D virtual images; users can also choose to only watch the corresponding AR special effects of one of them. If the user collects all models of figures for the first time, the AR special effects achieved by the entire collection will be displayed, such as the dynamic display effect of the 3D virtual images corresponding to all the figure models. These 3D virtual images can generate dynamics based on preset interaction logic. Display screen; accordingly, when it is recognized that the user has collected all models of figures for the first time, corresponding achievement badges can be generated to enhance the user's interest in collecting.

In some implementation scenarios, different dynamic display screens can also be displayed according to the posture of the figures. For example, according to the standing or flat posture of the figure, the 3D virtual image of the standing or flat display is displayed accordingly to enrich the display effect of the 3D virtual image of the figure.

An embodiment of the present application also provides a 3D virtual image processing device. Please refer to Figure 3, which is a structural diagram of a 3D virtual image processing device provided by an embodiment of the present application. As shown in Figure 3, the 3D virtual image processing device 300 includes:

Acquisition module 301, used to acquire a first preview image including the target object;

The first determination module 302 is configured to determine first information according to the first preview image. The first information includes at least one of the following: the first spatial position of the target object in the actual space, the first spatial position of the target object in the actual space. The orientation of the space, the number of the target objects, and the posture of the target objects;

The second determination module 303 is configured to determine the interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

Wherein, the interaction information is used to realize the interaction between the 3D virtual image and the user, and the dynamic display information is used to generate a dynamic display screen of the 3D virtual image.

Optionally, please further refer to Figure 4. The device also includes an execution module 304 for:

Generate and display a dynamic display screen of the 3D virtual image based on the dynamic display information; and/or,

When a user interaction instruction is obtained, a response instruction is generated based on the interaction information, so as to realize interaction between the 3D virtual image and the user through the response instruction.

Optionally, the acquisition module 301 is also used to:

When the first preview image is displayed, generating an identification frame in the first preview image;

Obtain the target object in the recognition frame of the first preview image.

Optionally, the acquisition module 301 is also used to:

When the first preview image is displayed, obtain the object to be recognized in the first preview image;

Obtain the object category to which the object to be identified belongs;

Determine a target detection model based on the object category;

When the target detection model recognizes that the object to be recognized matches the target object, the object to be recognized is used as the target object to obtain a first preview image including the target object, wherein different objects Categories correspond to different target detection models.

Optionally, in the case where the first information includes the posture of the target object, the execution module is further configured to:

Obtaining the posture of the target object, the posture of the target object includes a standing posture and a lying posture;

Based on the dynamic display information of the 3D virtual image corresponding to the target object, a dynamic display picture matching the 3D virtual image with the standing posture or the flat posture is displayed.

Optionally, in the case where the first information includes the number of target objects, the second determination module 303 is also used to:

According to the number of the target objects, interaction information and/or dynamic display information corresponding to the 3D virtual image of the target object and the number are determined.

Optionally, the 3D virtual image corresponding to the target object includes at least one second spatial position in the virtual three-dimensional space, and different second spatial positions correspond to different interaction information and/or different dynamic display information; and/or ,

When the number of the target objects is at least two, the 3D virtual images corresponding to the at least two target objects include at least one relative position relationship in the virtual three-dimensional space, and different relative position relationships correspond to different interactions. information and/or different dynamic display information.

Optionally, the device further includes a building module for:

Obtaining multiple image frames obtained by shooting around the target object;

Extract the key points of each image frame, map the key points to the virtual three-dimensional space, and obtain the spatial orientation information of the key points in the virtual three-dimensional space;

Based on the spatial orientation information, a 3D virtual image of the target object in the virtual three-dimensional space is constructed.

Optionally, the second determination module 303 is also used to:

Get the first input information;

When the second spatial position of the 3D virtual image in the virtual three-dimensional space is determined based on the first information, interaction information and/or dynamic display information of the 3D virtual image is determined based on the first input information.

In the embodiment of the present application, the device can set corresponding interaction information and/or dynamic display information for target objects such as figures, toys, etc., so that the target object can interact with the user based on the corresponding 3D virtual image. Or a dynamic display screen that can display the 3D virtual image of the target object, which can effectively improve the interaction between the target object and the user, make the interaction between the target object and the user more flexible, and also effectively enrich the display effect of the target object. , effectively improve the user experience of the target object.

The 3D virtual image processing device 300 in the embodiment of the present application may be an electronic device or a component of the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than the terminal. For example, the electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a mobile Internet device (MID), or augmented reality (AR)/virtual reality (VR). ) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (personal digital assistant, PDA), etc., the embodiments of this application are not specifically limited.

The 3D virtual image processing device 300 in the embodiment of the present application may be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of this application.

The 3D virtual image processing device 300 provided by the embodiment of the present application can implement each process implemented by the method embodiment described in Figure 1 and can achieve the same technical effect. To avoid duplication, the details will not be described here.

An embodiment of the present application also provides an electronic device. Please refer to Figure 5. Figure 5 is a structural diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 5, the electronic device includes: a processor 400, a memory 420 and data stored on the memory 420 and capable of The program or instructions run on the processor 400, and the processor 400 is used to read the program or instructions in the memory 420; the electronic device also includes a bus interface and a transceiver 410.

Transceiver 410 for receiving and transmitting data under the control of processor 400.

In FIG. 5 , the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 400 and various circuits of the memory represented by memory 420 are linked together. The bus architecture can also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are all well known in the art and therefore will not be described further herein. The bus interface provides the interface. Transceiver 410 may be a plurality of elements, including a transmitter and a transceiver, providing a unit for communicating with various other devices over a transmission medium. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 can store data used by the processor 400 when performing operations.

Among them, the processor 400 is used to read the program or instructions in the memory 420 and perform the following steps:

Obtaining a first preview image including the target object;

First information is determined according to the first preview image, and the first information includes at least one of the following: the first spatial position of the target object in the actual space, the orientation of the target object in the actual space, the The number and posture of the target object;

Determine the interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

Wherein, the interaction information is used to realize the interaction between the 3D virtual image and the user, and the dynamic display information is used to generate a dynamic display screen of the 3D virtual image.

Optionally, the processor 400 is also used to read the program or instructions in the memory 420 and perform the following steps:

Generate and display a dynamic display screen of the 3D virtual image based on the dynamic display information; and/or,

When a user interaction instruction is obtained, a response instruction is generated based on the interaction information, so as to realize interaction between the 3D virtual image and the user through the response instruction.

Optionally, the processor 400 is also used to read the program or instructions in the memory 420 and perform the following steps:

When the first preview image is displayed, generating an identification frame in the first preview image;

Obtain the target object in the recognition frame of the first preview image.

Optionally, the processor 400 is also used to read the program or instructions in the memory 420 and perform the following steps:

When the first preview image is displayed, obtain the object to be recognized in the first preview image;

Obtain the object category to which the object to be identified belongs;

Determine a target detection model based on the object category;

When the target detection model recognizes that the object to be recognized matches the target object, the object to be recognized is used as the target object to obtain a first preview image including the target object, wherein different objects Categories correspond to different target detection models.

Optionally, in the case where the first information includes the posture of the target object, the processor 400 is also configured to read the program or instructions in the memory 420 and perform the following steps:

Obtaining the posture of the target object, the posture of the target object includes a standing posture and a lying posture;

Based on the dynamic display information of the 3D virtual image corresponding to the target object, a dynamic display picture matching the 3D virtual image with the standing posture or the flat posture is displayed.

Optionally, in the case where the first information includes the number of target objects, the processor 400 is also configured to read the program or instructions in the memory 420 and perform the following steps:

According to the number of the target objects, interaction information and/or dynamic display information corresponding to the 3D virtual image of the target object and the number are determined.

Optionally, the 3D virtual image corresponding to the target object includes at least one second spatial position in the virtual three-dimensional space, and different second spatial positions correspond to different interaction information and/or different dynamic display information; and/or ,

When the number of the target objects is at least two, the 3D virtual images corresponding to the at least two target objects include at least one relative position relationship in the virtual three-dimensional space, and different relative position relationships correspond to different interactions. information and/or different dynamic display information.

Optionally, the processor 400 is also used to read the program or instructions in the memory 420 and perform the following steps:

Obtaining multiple image frames obtained by shooting around the target object;

Extract the key points of each image frame, map the key points to the virtual three-dimensional space, and obtain the spatial orientation information of the key points in the virtual three-dimensional space;

Based on the spatial orientation information, a 3D virtual image of the target object in the virtual three-dimensional space is constructed.

Optionally, the processor 400 is also used to read the program or instructions in the memory 420 and perform the following steps:

Get the first input information;

When the second spatial position of the 3D virtual image in the virtual three-dimensional space is determined based on the first information, interaction information and/or dynamic display information of the 3D virtual image is determined based on the first input information.

In the embodiment of the present application, the electronic device can set corresponding interaction information and/or dynamic display information for target objects such as figures, toys, etc., so that the target object can interact with the user based on the corresponding 3D virtual image, or It is a dynamic display screen that can display the 3D virtual image of the target object, thereby effectively improving the interaction between the target object and the user, making the interaction between the target object and the user more flexible, and effectively enriching the display effect of the target object.

Embodiments of the present application also provide a readable storage medium. Programs or instructions are stored on the readable storage medium. When the program or instructions are executed by a processor, each process of the method embodiment described in Figure 1 is implemented, and can To achieve the same technical effect, to avoid repetition, we will not repeat them here.

Wherein, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes computer readable storage media, such as computer read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.

An embodiment of the present application further provides a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the method described in Figure 1. Each process in the example can achieve the same technical effect. To avoid repetition, we will not repeat it here.

It should be understood that the chips mentioned in the embodiments of this application may also be called system-on-chip, system-on-a-chip, system-on-a-chip or system-on-chip, etc.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions may be performed, for example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a computer software product that is essentially or contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM, disk , optical disk), including several instructions to cause a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Inspired by this application, many forms can be made without departing from the purpose of this application and the scope protected by the claims, all of which fall within the protection of this application.

Claims (12)

1. A 3D avatar processing method, comprising:

acquiring a first preview image comprising a target object;

determining first information according to the first preview image, wherein the first information comprises at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, the orientation of the target object in the actual space, the number of the target objects and the gesture of the target object;

determining interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

the interaction information is used for realizing interaction between the 3D virtual image and a user, and the dynamic display information is used for generating a dynamic display picture of the 3D virtual image.

2. The method according to claim 1, wherein after determining the interaction information and/or the dynamic presentation information of the 3D avatar corresponding to the target object based on the first information, the method further comprises:

generating and displaying a dynamic display picture of the 3D avatar based on the dynamic display information; and/or the number of the groups of groups,

and generating a response instruction based on the interaction information under the condition that the user interaction instruction is acquired, so as to realize the interaction between the 3D virtual image and the user through the response instruction.

3. The method of claim 1, wherein the acquiring a first preview image including the target object comprises:

generating an identification frame in a first preview image in a case where the first preview image is displayed;

a target object in the identification frame of the first preview image is acquired.

4. The method of claim 1, wherein the acquiring a first preview image including the target object comprises:

acquiring an object to be identified in a first preview image under the condition of displaying the first preview image;

obtaining an object category to which the object to be identified belongs;

determining a target detection model based on the object class;

and under the condition that the target detection model identifies that the object to be identified is matched with a target object, taking the object to be identified as the target object to acquire a first preview image comprising the target object, wherein different object categories correspond to different target detection models.

5. The method according to any one of claims 1-4, wherein, in case the first information comprises a pose of the target object, the method further comprises:

Acquiring the gesture of the target object, wherein the gesture of the target object comprises a standing gesture and a flat gesture;

and displaying a dynamic display picture of the 3D virtual image matched with the standing posture or the flat posture based on the dynamic display information of the 3D virtual image corresponding to the target object.

6. The method according to any one of claims 1-4, wherein, in case the first information includes the number of target objects, the determining interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object based on the first information includes:

and according to the number of the target objects, determining interaction information and/or dynamic display information of the 3D virtual images of the target objects corresponding to the number.

7. The method according to any one of claims 1-4, wherein the 3D avatar corresponding to the target object comprises at least one second spatial location in a virtual three-dimensional space, different ones of the second spatial locations corresponding to different interaction information and/or different dynamic presentation information; and/or the number of the groups of groups,

and under the condition that the number of the target objects is at least two, the 3D virtual images corresponding to the at least two target objects respectively comprise at least one relative position relation in the virtual three-dimensional space, and different relative position relations correspond to different interaction information and/or different dynamic display information.

8. The method according to any one of claims 1-4, wherein before determining the interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object based on the first information, the method further comprises:

acquiring a plurality of image frames obtained by shooting around the target object;

extracting key points of each image frame, mapping the key points into a virtual three-dimensional space, and acquiring space azimuth information of the key points in the virtual three-dimensional space;

and constructing a 3D virtual image of the target object in the virtual three-dimensional space based on the space orientation information.

9. The method according to any one of claims 1-4, wherein the determining interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object based on the first information comprises:

acquiring first input information;

and under the condition that the second space position of the 3D avatar in the virtual three-dimensional space is determined based on the first information, determining the interaction information and/or dynamic display information of the 3D avatar according to the first input information.

10. A 3D avatar processing apparatus, comprising:

The acquisition module is used for acquiring a first preview image comprising a target object;

the first determining module is used for determining first information according to the first preview image, and the first information comprises at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, the orientation of the target object in the actual space, the number of the target objects and the gesture of the target object;

the second determining module is used for determining interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

the interaction information is used for realizing interaction between the 3D virtual image and a user, and the dynamic display information is used for generating a dynamic display picture of the 3D virtual image.

11. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the 3D avatar processing method of any one of claims 1-9.

12. A readable storage medium, wherein a program or instructions are stored on the readable storage medium, which when executed by a processor, implement the steps of the 3D avatar processing method as claimed in any one of claims 1-9.