CN116912463A

CN116912463A - 3D avatar processing method, apparatus, electronic device, and readable storage medium

Info

Publication number: CN116912463A
Application number: CN202310879539.8A
Authority: CN
Inventors: 刘泽凡; 陈佳悦; 罗德海; 洪毅强; 胡良军; 张伟杰; 王�琦
Original assignee: Migu Cultural Technology Co Ltd; China Mobile Communications Group Co Ltd; MIGU Comic Co Ltd
Current assignee: Migu Cultural Technology Co Ltd; China Mobile Communications Group Co Ltd; MIGU Comic Co Ltd
Priority date: 2023-07-18
Filing date: 2023-07-18
Publication date: 2023-10-20

Abstract

The application discloses a 3D virtual image processing method, a device, electronic equipment and a readable storage medium, and relates to the technical field of computers. The 3D avatar processing method comprises the following steps: acquiring a first preview image comprising a target object; determining first information according to the first preview image, wherein the first information comprises at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, the orientation of the target object in the actual space, the number of the target objects and the gesture of the target object; determining interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information; the interaction information is used for realizing interaction between the 3D virtual image and a user, and the dynamic display information is used for generating a dynamic display picture of the 3D virtual image.

Description

3D avatar processing method, apparatus, electronic device, and readable storage medium

Technical Field

The application belongs to the technical field of computers, and particularly relates to a 3D virtual image processing method, a device, electronic equipment and a readable storage medium.

Background

With the development of cartoon videos and the like, the attractions of secondary cartoon pictures, cartoons, games and the like to users are suddenly advanced, and models with favorites such as hand models and the like are emotion extension to cartoon images. In recent years, collection of solid toys such as hand models has also become popular. However, the playing method of the entity toys such as the hand models is usually only used as a decoration, only plays a role in decoration and has single display effect, and the interactivity with users is not flexible.

Disclosure of Invention

The embodiment of the application provides a 3D virtual image processing method, a device, electronic equipment and a readable storage medium, which can solve the problems of single display effect, inflexible interaction with users and the like of entity toys in the related technology.

In a first aspect, an embodiment of the present application provides a 3D avatar processing method, including:

acquiring a first preview image comprising a target object;

determining first information according to the first preview image, wherein the first information comprises at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, the orientation of the target object in the actual space, the number of the target objects and the gesture of the target object;

Determining interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

the interaction information is used for realizing interaction between the 3D virtual image and a user, and the dynamic display information is used for generating a dynamic display picture of the 3D virtual image.

In a second aspect, an embodiment of the present application provides a 3D avatar processing apparatus, including:

the acquisition module is used for acquiring a first preview image comprising a target object;

the first determining module is used for determining first information according to the first preview image, and the first information comprises at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, the orientation of the target object in the actual space, the number of the target objects and the gesture of the target object;

the second determining module is used for determining interaction information and/or dynamic display information of the 3D virtual image corresponding to the target object based on the first information;

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing a program or instructions executable on the processor, the program or instructions implementing the steps of the 3D avatar processing method as described in the first aspect when executed by the processor.

In a fourth aspect, an embodiment of the present application provides a readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps of the 3D avatar processing method as set forth in the first aspect.

In the embodiment of the application, after a first preview image aiming at a target object is acquired, first information is determined according to the first preview image, the first information comprises at least one of a first spatial position, an orientation, the number and a gesture of the target object in an actual space, interaction information and/or dynamic display information of a 3D avatar corresponding to the target object are determined based on the first information, the interaction information can realize interaction between the 3D avatar and a user, and the dynamic display information is used for generating a dynamic display picture of the 3D avatar. In this way, corresponding interaction information and/or dynamic display information can be set for such target objects as a host, a toy and the like, so that the target object can realize interaction with a user based on the corresponding 3D virtual image, or a dynamic display picture of the 3D virtual image of the target object can be displayed, the interactivity between the target object and the user can be effectively improved, the interaction between the target object and the user is more flexible, the display effect of the target object can be effectively enriched, and the user experience of the target object is effectively improved.

Drawings

Fig. 1 is a flowchart of a 3D avatar processing method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of a spatial point cloud in an embodiment of the present application;

fig. 3 is a block diagram of a 3D avatar processing device according to an embodiment of the present application;

fig. 4 is a block diagram of another 3D avatar processing apparatus provided in an embodiment of the present application;

fig. 5 is a block diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions of the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which are obtained by a person skilled in the art based on the embodiments of the present application, fall within the scope of protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type, and are not limited to the number of objects, such as the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The method, the device, the electronic equipment and the like for processing the 3D virtual image provided by the embodiment of the application are described in detail through specific embodiments and application scenes thereof with reference to the accompanying drawings.

Referring to fig. 1, fig. 1 is a flowchart illustrating a 3D avatar processing method according to an embodiment of the present application, where the method is applied to an electronic device. As shown in fig. 1, the method comprises the steps of:

step 101, a first preview image including a target object is acquired.

It should be noted that, the object involved in the embodiment of the present application may be a toy or an ornament including a hand, a doll, etc., and may also include a person or an animal, etc., for example, the object may be a specific role or an image played by a person (cosplay). The sponsor can also be called as sponsor or first edition, and is a model of a high-precision small-size sculpture by taking cartoon film and television and game characters as prototypes, and a raw name suit module (GARAGE KIT, GK). The handholds are usually surrounding derivatives of cartoons, movies, games, etc., and are materialized in real life of the character image of the virtual movie game. Illustratively, the target object may refer to one or more handoffs, such as all handoffs of a certain series.

In the embodiment of the application, the first preview image aiming at the target object can be acquired through the camera. For example, the user may turn on the camera of the electronic device, align the camera with the target object, and then display the target object in the preview interface of the camera, so as to obtain a first preview image for the target object.

Step 102, determining first information according to the first preview image, wherein the first information comprises at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, setting the orientation of the target object in the actual space, setting the number of the target objects and setting the posture of the target object.

For example, after the electronic device acquires the first preview image for the target object, a spatial coordinate system may be established by using a camera of the electronic device as an origin of coordinates, and further, according to the first preview image acquired by the camera, first information corresponding to the target object, for example, a coordinate position of the target object in the spatial coordinate system, may be determined, and according to the coordinate position, the first spatial position of the target object may be determined. Wherein the first spatial position may be a spatial coordinate and/or an azimuth comprising the target object in a spatial coordinate system.

It is understood that the information such as the number, the gesture, or the orientation of the target object in the real space may be obtained according to the first preview image.

And step 103, determining interaction information and/or dynamic display information of the first 3D virtual image corresponding to the target object based on the first information.

Optionally, after determining the first spatial position of the target object in the real space, determining the second spatial position of the first 3D avatar corresponding to the target object in the virtual three-dimensional space based on the first spatial position may be performed. The virtual three-dimensional space can be rendered according to the actual space, or can be a designated virtual three-dimensional space.

For example, after determining a first spatial position of a target object in an actual space based on a first preview image acquired by a camera, the electronic device may render a virtual three-dimensional space corresponding to the actual space according to the first preview image, and determine a second spatial position of a 3D avatar corresponding to the target object in the virtual three-dimensional space according to the first spatial position of the target object in the actual space.

Alternatively, the virtual three-dimensional space is a designated virtual three-dimensional space, and the electronic device may determine a second spatial position of the 3D avatar corresponding to the target object in the virtual three-dimensional space based on an input operation of the user. For example, after determining the first spatial position of the target object in the actual space, the electronic device may use the display interface of the electronic device as the specified virtual three-dimensional space, so as to facilitate the user to intuitively know the location of the virtual three-dimensional space. Further, the user may place the 3D avatar corresponding to the target object in the second spatial position of the virtual three-dimensional space, for example, the 3D avatar is displayed on a display interface of the electronic device, and the electronic device may place the 3D avatar in the second spatial position of the virtual three-dimensional space based on a drag operation of the 3D avatar by the user, so that the operation of the 3D avatar is more flexible and convenient. It should be noted that the 3D avatar of the target object may be pre-constructed. Optionally, before the step 103, the method may further include the steps of:

acquiring a plurality of image frames obtained by shooting around the target object;

Extracting key points of each image frame, and mapping the key points into a virtual three-dimensional space to obtain space azimuth information of the key points in the virtual three-dimensional space;

and constructing a 3D virtual image of the target object in the virtual three-dimensional space based on the space orientation information.

It should be noted that, in order to enable a target object (e.g., a task) to appear in a virtual three-dimensional space as a 3D avatar, it is first necessary to construct a three-dimensional space point cloud of the target object.

In the embodiment of the application, the target object is shot around by a camera (such as a monocular camera) of the electronic equipment, so as to obtain a plurality of image frames of different angles of the target object; further, the extraction of the keypoints of each image frame may be, for example, based on simultaneous localization and mapping (Simultaneous Localization and Mapping, SLAM) techniques to extract the keypoints of each image frame and map the 2D keypoints into a virtual three-dimensional space to form points in space. And then, according to the key point matching between the image frames, obtaining the space azimuth information of different key points in the virtual three-dimensional space, and constructing a space point cloud based on the space azimuth information of the key points, so as to obtain the 3D virtual image of the target object in the virtual three-dimensional space. As shown in fig. 2, the target object is a kettle, and based on the above manner, a space point cloud of the kettle can be constructed, so that a 3D virtual image of the kettle is obtained.

In the case where there are a plurality of target objects, it is possible to construct 3D avatars corresponding to the plurality of target objects in the same virtual three-dimensional space. For example, taking a target object as a series of handholds as an example, a series of handholds generally includes a plurality of handholds, and a plurality of handholds have a specific placement position therebetween, that is, the handholds have a specific spatial position in the actual space; in this case, a plurality of image frames of each of the sponsors may be obtained by performing surrounding shooting, and then a 3D avatar of the sponsor may be constructed based on the above manner, and based on the placement position of the sponsor in the actual space, in the sponsor series, the position and orientation of the sponsor in the actual space may be obtained, and the 3D avatar of the sponsor may be mapped to the position in the designated virtual three-dimensional space according to the position and orientation in the actual space; based on such a manner, 3D avatars corresponding to the respective 3D avatars under the series of the hands are constructed, and the positions and orientations of the 3D avatars in the virtual three-dimensional space, that is, the positions and orientations of the 3D avatars corresponding to the respective hands in the virtual three-dimensional space, are determined according to the positions and orientations of the corresponding hands in the real space, so as to be consistent with the positions and orientations of the hands in the real space. For example, if a certain office a is located in front of the office B in the real space, the 3D avatar corresponding to the office a is also located in front of the 3D avatar corresponding to the office B in the virtual three-dimensional space.

Alternatively, after the 3D avatars of different target objects are built, the electronic device may store these 3D avatars to the cloud. Furthermore, the electronic device can obtain 3D virtual images corresponding to different target objects from the cloud.

In the embodiment of the application, after determining the first information, the electronic device determines the interactive information and/or the dynamic display information of the 3D avatar based on the first information, and optionally, the interactive information and the dynamic display information are related to the first information. For example, for the same 3D avatar, different first spatial positions of the target object corresponding to the 3D avatar in the real space may correspond to different interactive information and/or dynamic display information, that is, one target object corresponding to the 3D avatar may have multiple second spatial positions in the virtual three-dimensional space, and for different second spatial positions, different interactive information and/or dynamic display information may be set.

Or, according to different orientations of the target object in the actual space, the 3D avatar corresponding to the target object may also have different interactive information and/or dynamic display information. For example, in the case that the front surface of the target object faces to the left, the dynamic display information corresponding to the 3D avatar is displayed in a left rotating way; and under the condition that the front surface of the target object faces to the right, the dynamic display information corresponding to the 3D virtual image is displayed in a rightward rotating way.

As another example, the interactive information and/or dynamic presentation information of the 3D avatar may also be related to the number of target objects. Optionally, in the case that the first information includes the number of the target objects, the determining, based on the first information, interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object includes:

and according to the number of the target objects, determining interaction information and/or dynamic display information of the 3D virtual images of the target objects corresponding to the number.

For example, if it is recognized that the first preview image includes two target objects (for example, two different handoffs, or two same handoffs), a dynamic display screen generated based on the 3D avatar corresponding to each of the two target objects is screen 1; if the first preview image is identified to include three target objects, the dynamic display picture correspondingly generated based on the 3D avatars corresponding to the three target objects is picture 2.

Furthermore, in the embodiment of the application, the dynamic Zhang's information and/or the interactive information of the 3D virtual images corresponding to the target objects are different for different numbers of target objects, so that the method is more beneficial to attracting users to collect the target objects more, providing more playing methods for the users, and effectively improving the interactivity between the target objects and the users.

Optionally, the interactive information and/or dynamic presentation information of the 3D avatar may also be related to the pose of the target object. For example, if the gesture of the target object in the first preview image is recognized as a standing gesture, the 3D avatar corresponding to the target object may be dynamically displayed in a standing figure, if the gesture of the target object is recognized as a lying gesture, the 3D avatar corresponding to the target object may be dynamically displayed in a lying figure, and so on.

It should be noted that, the first information may include at least one of a first spatial position of the target object in the actual space, an orientation of the target object in the actual space, a number of the target objects, and a posture of the target object, and further, in an embodiment of the present application, specific content of the first information may be other possible situations besides the foregoing, and interaction information and/or dynamic display information of the 3D avatar may also be other possible situations, which are not listed herein too.

In the embodiment of the application, the interaction information and/or the dynamic display information of the corresponding 3D virtual image can be determined according to the first information corresponding to the target object, so that the target object can realize interaction with a user based on the corresponding 3D virtual image or can display the dynamic display picture of the 3D virtual image of the target object, thereby effectively improving the interactivity between the target object and the user experience of the target object.

Optionally, the step 103 may specifically include:

acquiring first input information;

and under the condition that the second space position of the 3D avatar in the virtual three-dimensional space is determined based on the first information, determining the interaction information and/or dynamic display information of the 3D avatar according to the first input information.

The first input information may refer to related information input by a user, for example, may be interactive information and/or dynamic presentation information input by the user. For example, after determining the second spatial position of the 3D avatar corresponding to the target object in the virtual three-dimensional space, the electronic device may display an input box to acquire the first input information of the user.

For example, with the display interface of the electronic device as the designated virtual three-dimensional space, when the display interface of the electronic device displays the 3D avatar corresponding to the target object, the display interface may further display an input box, and the user may input corresponding interaction information and/or dynamic presentation information in the input box according to the display position (i.e., the second spatial position) of the 3D avatar on the display interface. Thus, the electronic device can determine the interactive information and/or the dynamic presentation information of the 3D avatar according to the input information input by the user.

Optionally, the 3D avatar corresponding to the target object includes at least one second spatial position in the virtual three-dimensional space, and different second spatial positions correspond to different interaction information and/or different dynamic display information. For example, when the 3D avatar is located at the position a, the user may input the first interaction information and the first dynamic display information, and when the 3D avatar is located at the position B, the user may input the second interaction information and the second dynamic display information, so that the interaction information and/or the dynamic display information corresponding to each of the 3D avatar at different second spatial positions may be determined. Therefore, the 3D virtual image of a certain target object has different interaction information and/or dynamic display information according to the spatial position of the target object, so that the interaction between the 3D virtual image and a user can be effectively enriched, and/or the dynamic display picture of the 3D virtual image is more various and rich, and better interaction experience is brought to the user.

Optionally, in the case that the number of the target objects is at least two, the 3D avatar corresponding to each of the at least two target objects includes at least one relative positional relationship in the virtual three-dimensional space, and different relative positional relationships correspond to different interaction information and/or different dynamic display information. For example, taking a plurality of handoffs with a target object as a certain series as an example, if the series of handoffs includes handoffs 1, handoffs 2 and handoffs 3, and the handoffs 1 may have two different second spatial positions, and the handoffs 2 may also have two different second spatial positions, then the handoffs 1 and the handoffs 2 may have four relative positional relationships, and the four relative positional relationships may correspond to four different interaction information and/or four different dynamic presentation information, respectively; accordingly, at least one relative positional relationship may be provided between the hand 1 and the hand 3, at least one relative positional relationship may be provided between the hand 2 and the hand 3, at least one relative positional relationship may be provided between the hand 1, the hand 2 and the hand 3, and different relative positional relationships may correspond to different interaction information and/or different dynamic display information. Therefore, for different relative position relations among the 3D virtual images, different interaction information and/or dynamic display information are provided, so that interaction and/or dynamic display effects between the 3D virtual images and users can be effectively enriched, and better interaction experience is brought to the users.

It should be noted that the interactive information and/or the dynamic presentation information may be determined based on input information of the user.

In the embodiment of the application, after a first preview image aiming at a target object is acquired, first information is determined according to the first preview image, the first information comprises at least one of a first spatial position, an orientation, the number and a gesture of the target object in an actual space, interaction information and/or dynamic display information of a 3D avatar corresponding to the target object are determined based on the first information, the interaction information can realize interaction between the 3D avatar and a user, and the dynamic display information is used for generating a dynamic display picture of the 3D avatar. In this way, corresponding interaction information and/or dynamic display information can be set for such target objects as a host, a toy and the like, so that the target object can realize interaction with a user based on the corresponding 3D virtual image, or a dynamic display picture of the 3D virtual image of the target object can be displayed, the interactivity between the target object and the user can be effectively improved, and the user experience of the target object is effectively improved.

Optionally, in an embodiment of the present application, after the step 104, the method may further include:

generating and displaying a dynamic display picture of the 3D avatar based on the dynamic display information; and/or the number of the groups of groups,

and generating a response instruction based on the interaction information under the condition that the user interaction instruction is acquired, so as to realize the interaction between the 3D virtual image and the user through the response instruction.

In the embodiment of the application, after the dynamic display information of the 3D virtual image corresponding to the target object is determined, the electronic equipment can generate and display the dynamic display picture of the 3D virtual image based on the dynamic display information. For example, the electronic device may generate and display a dynamic display screen of the 3D avatar corresponding to the target object when receiving an input operation of the user, such as a trigger operation of a specific key, or a voice input of the user. The dynamic display screen may mean that the 3D avatar is displayed in a dynamic manner, for example, the 3D avatar may be rotated or moved in a specific direction at a certain speed, etc. Furthermore, by generating and displaying the dynamic display picture of the 3D virtual image, the target object can be dynamically displayed by the 3D virtual image, so that interactivity and ornamental value of the target object are effectively improved, and better use experience is brought to users.

Or after the interactive information of the 3D virtual image corresponding to the target object is determined, if an interactive instruction of the user is received, generating a response instruction for responding to the interactive instruction according to the interactive information so as to realize interaction between the 3D virtual image and the user. Wherein the interaction instruction comprises at least one of the following: touch interaction instruction, gesture interaction instruction and voice interaction instruction.

For example, if a voice interaction instruction of the user, such as "hello", is received, the electronic device may generate a corresponding response instruction based on the interaction information, where the response instruction is to control the 3D avatar to rotate or jump, so as to respond to the voice interaction instruction of the user, so as to implement interaction with the user. For another example, if a gesture interaction instruction of the user, such as a hand-in instruction, is obtained through the camera, the electronic device may generate a corresponding response instruction based on the interaction information, such as controlling the hand-in of the 3D avatar (if the 3D avatar is a character), so as to implement interaction between the 3D avatar and the user. Therefore, the interactivity between the 3D virtual image and the user can be effectively improved, and better interactive experience is brought to the user.

It should be noted that, for different interaction instructions, the electronic device may preset interaction information for generating a corresponding response instruction, and further, under the condition that the interaction instruction is acquired, the electronic device may determine which interaction information needs to be based on which interaction information is needed to generate the corresponding response instruction, so as to implement a response to the interaction instruction. For example, for different gesture actions, such as a hand, a fist, a heart, and the like, different interaction information may be preset, so that different response instructions are generated according to the different interaction information, for example, the interaction information corresponding to the hand is used for controlling the 3D avatar to generate the response instruction of the hand, and the interaction information corresponding to the heart is used for controlling the 3D avatar to generate the response instruction of the heart. In this way, the response instructions for responding to the different interaction instructions can be generated through the different interaction information, so that the 3D virtual image can accurately and effectively interact with the different interaction instructions of the user, and richer and more various interactions between the 3D virtual image and the user can be realized.

In the embodiment of the application, after the dynamic display information of the 3D virtual image corresponding to the target object is determined, the user can scan the target object through the electronic equipment so as to enable the electronic equipment to generate and display the dynamic display picture.

For example, the user may scan the object to be identified through a camera of the electronic device. And under the condition that the electronic equipment starts the camera, displaying a preview image acquired by the camera, and if the camera is aligned to the object to be identified, displaying the object to be identified in the preview image.

The object to be identified may be a target object for which dynamic display information has been determined or set; in the embodiment of the application, the dynamic display information of the 3D avatar corresponding to the target objects can be established based on the above manner, and the dynamic display information of the target objects and the corresponding 3D avatar can be stored locally or in the cloud of the electronic device. The electronic device may compare the object to be identified with the target object or call the target object stored in the cloud to perform the comparison when the object to be identified in the preview image is acquired based on the camera. And if the object to be identified is matched with the target object with the D3 avatar dynamic display information established, taking the 3D avatar of the target object as the 3D avatar of the object to be identified.

For example, if the target object includes a animation character A's hand, the electronic device has determined dynamic presentation information for the 3D avatar of the hand. And then, when the user scans the object to be identified through the camera of the electronic equipment and determines the object to be identified, namely the handwork of the cartoon character A through comparison, the 3D virtual image corresponding to the handwork and the corresponding dynamic display information are obtained, a dynamic display picture is generated based on the dynamic display information of the 3D virtual image of the handwork, and the dynamic display picture is displayed. Therefore, the dynamic display picture of the 3D virtual image corresponding to the object can be generated and displayed by scanning the object such as the handhold, better interactive experience is brought to the user, collection and viewing interests of the object such as the handhold by the user are improved, and consumption of the object such as the handhold is promoted.

Optionally, the acquiring a first preview image including the target object includes:

generating an identification frame in a first preview image in a case where the first preview image is displayed;

a target object in the identification frame of the first preview image is acquired.

In the embodiment of the application, when the electronic device starts the camera and displays the first preview image, an identification frame is generated in the first preview image, and the user is guided to adjust the acquisition azimuth and angle of the camera through the identification frame, so that the target object is positioned in the identification frame. Alternatively, the target object may be captured by the identification frame so that the target object is located within the identification frame, thereby better locating the target object. The electronic device may determine the spatial position of the target object by identifying the position of the frame, and further determine which position of the target object is detected and identified.

In the case where there are a plurality of target objects, a single recognition frame may be generated for each target object, or one recognition frame may be generated for all target objects. For example, if there are a plurality of target objects and all the target objects belong to the same series of handoffs, a recognition frame may be generated, and then the same series of handoffs may be positioned in a recognition frame, and recognition and detection may be performed by a target detection model corresponding to the series of handoffs.

acquiring an object to be identified in a first preview image under the condition of displaying the first preview image;

obtaining an object category to which the object to be identified belongs;

determining a target detection model based on the object class;

and under the condition that the target detection model identifies that the object to be identified is matched with a target object, taking the object to be identified as the target object to acquire a first preview image comprising the target object, wherein different object categories correspond to different target detection models.

Taking the object to be identified as the task to be identified as an example, the object category to which the task to be identified belongs, that is, which task series, is first obtained. Alternatively, the object class to which the user belongs may be determined according to the input of the user. It may be appreciated that, for the to-be-identified task, the user may autonomously confirm which task series the to-be-identified task belongs to, and the electronic device may display an object type selection box, in which the user may select a corresponding task series, so that the electronic device may determine, based on the input of the user, the task series to which the to-be-identified task belongs, that is, determine the object type to which the to-be-identified object belongs.

Further, the electronic device determines a corresponding target detection model according to the object category, and different object categories correspond to different target detection models, so that whether the object to be identified is matched with the target object or not is detected through the corresponding target detection model. For example, the electronic device may train and store the target detection model corresponding to each of the different task series in advance, and identify the task under a certain task series by the target detection model corresponding to the task series. The target detection models can be trained by using related data of each hand under the corresponding hand series, for example, image data under the conditions of different angles, different orientations, different postures and the like of each hand can be collected to serve as training data so as to train the target detection models, and further the trained target detection models can identify the hand under the corresponding hand series, so that identification accuracy of the hand to be identified is improved. Wherein, the training data of the target detection model comprises image data of the target object.

According to the method and the device for identifying the object to be identified, the corresponding target detection models are respectively trained for different object types, so that whether the object to be identified is matched with the target object or not can be identified based on the object type of the object to be identified, and the identification accuracy of the object to be identified can be effectively improved.

Optionally, in the case that the first information includes a pose of the target object, the method further includes:

acquiring the gesture of the target object, wherein the gesture of the target object comprises a standing gesture and a flat gesture;

and displaying a dynamic display picture of the 3D virtual image matched with the standing posture or the flat posture based on the dynamic display information of the 3D virtual image corresponding to the target object.

It will be appreciated that the target object may be of a different pose, for example a lay-flat pose or a cubic pose. Assuming that the target object is in a flat gesture, after generating a dynamic display picture based on the dynamic display information of the 3D avatar corresponding to the target object, the dynamic display picture may be a dynamic display picture for displaying that the 3D avatar matches the flat gesture, for example, the 3D avatar is dynamically displayed in the flat gesture.

The posture of the target object can be judged according to the width and the height of the target object, if the height is larger than the width, the target object can be considered to be in a standing posture, and if the height is smaller than the width, the target object can be considered to be in a flat posture.

According to the embodiment of the application, the gesture of the target object is obtained, so that the dynamic display picture of the 3D virtual image matched with the gesture can be displayed, and the richness and flexibility of the dynamic display picture of the 3D virtual image are effectively improved.

For better understanding, taking a target object and an object to be identified as a hand as an example, the 3D avatar processing method provided by the embodiment of the present application is specifically described, where the 3D avatar processing method includes the following steps:

and 1, pre-constructing 3D virtual images corresponding to the respective handoffs of each handoffs series (or handoffs type).

Wherein, the step 1 specifically comprises the following steps:

and 11, constructing a space point cloud for handholding.

It will be appreciated that in order to enable the presentation of a hosted avatar in 3D space, a hosted three-dimensional point cloud needs to be created first. For example, a spatial SLAM technique may be adopted, a monocular camera is used to capture around a camera, then key points of each captured frame of image frames are extracted based on the SLAM technique, the 2D key points are mapped into a 3D space to form points in the space, and then the orientations of different key points in the space are obtained according to the additional key point matching between frames, so as to construct a spatial point cloud. As shown in fig. 2, a space point cloud of the kettle is constructed based on the above manner.

And 12, constructing the hand 3D virtual image.

The spatial coordinates and the directions of the hand may be acquired, and for example, a hand 3D avatar designed in advance may be placed at the specified spatial coordinates and rendered to the specified coordinates in space.

And 13, carrying out rendering of the 3D virtual image.

The user aims at the hand-office through the camera of the mobile phone client, the client can automatically identify the space coordinates of the hand-office, and a three-dimensional dynamic virtual image is rendered in front of the hand-office in the camera display interface.

And 2, placing the 3D avatar of each hand on the appointed virtual space coordinates and directions.

Wherein each of the handheld 3D avatars may have more than one designated location, with more than one relative positional relationship (or range of locations) between the two or more handheld 3D avatars.

And 3, setting interaction logic and corresponding dynamic display effects between the 3D virtual images of the handholds according to the coordinate relationship and the azimuth relationship of the virtual space between the handholds.

Optionally, setting a first interaction logic and a first dynamic display effect when the two or more handheld 3D avatars are located within a first location range; setting a second interaction logic and a second dynamic display effect when more than two handful 3D virtual images are positioned in a second position range; and by analogy, different interaction logics and dynamic display effects are set between the 3D virtual images in different position relations.

Further, different hand models are set for respective hands of the same hand series. Different interaction logics and dynamic display effects are set between 3D virtual images of different handmade models.

In some embodiments, the 3D avatars may also have the same interaction logic, different only dynamic display effects, or different interaction logic, different dynamic display effects, and so on.

Wherein the interaction logic includes, but is not limited to: touch interactions, gesture interactions, voice interactions, etc.;

dynamic presentation effects include, but are not limited to: 3D avatar actions, 3D avatar special effects, etc., the dynamic presentation effect is rendered in a corresponding location of the hand in the camera display interface, such as a front or peripheral location.

Step 4, performing surrounding shooting on the handholds collected by the user, and acquiring the type of the handholds, the space coordinates among the handholds and the azimuth information; and calling a pre-constructed hand 3D virtual image according to the hand type obtained by shooting, and placing the hand 3D virtual image in the corresponding virtual space coordinates and azimuth according to the actual space coordinates and azimuth information among the hands obtained by shooting.

The 3D virtual images, the space coordinate relation and the azimuth relation among the 3D virtual images, the interaction logic among the 3D virtual images and the corresponding dynamic display effects can be all stored in the cloud; and the user is in communication connection and data interaction with the cloud through the client.

The user can see the hand in the real space through the mobile phone client, the client can acquire the gesture of the user through the mobile phone camera, identify the gesture type of the user through AI capability, such as hand-drawing, fist-making, heart-comparing and the like, and enable the 3D virtual image of the hand to swing out corresponding actions according to the identified gesture actions, so that the interaction effect is achieved.

For the same handhold series, if the handholds collected by the user are more complete, the dynamic effect which can be displayed by the user is more abundant; for example, if only a single hand is photographed, the 3D avatar and the interactive action corresponding to the single hand may be triggered to be displayed; if more than two handoffs are photographed, the 3D avatar corresponding to the set of handoffs, the interactions of the respective handoffs, and the interactions between the respective handoffs may be triggered.

And 5, realizing interaction between the 3D virtual image and the user.

In some implementations, the augmented reality (augmented reality, AR) effect of each of the handheld 3D avatars may be unlocked according to the number of collections of the handheld, a combination of different models.

Illustratively, first, a user needs to select a series of handholds to be identified, and a corresponding identification model is called according to the selected series; second, the user is guided to identify the handhold in the specified identification box by generating the identification box in the camera preview interface. Typically, according to the selected series of handoffs, a recognition area is automatically defined in the camera preview interface (i.e., a recognition frame is generated), and the user needs to place the handoffs to be recognized in the recognition frame. Further, the corresponding target detection model of the series of handoffs is invoked to detect handoffs within the identification frame. The target detection model is specially trained by using the data of the corresponding handy model, can specifically identify each handy in the series, and returns the handy type and the handy number appearing in the identification frame. Assuming that there are eight types of handpieces in the current handpieces series, the input of the target detection model is the image in the detection frame, and the output result is the outsourced rectangular frame of the handpieces in the detection frame, and which type of the eight types of handpieces corresponds to the outsourced rectangular frame. And calling the 3D virtual images of the cloud corresponding to each hand model according to the hand model and the number of the hands returned by the detection result, obtaining the space coordinates of the 3D virtual images of each hand model, displaying the 3D virtual images on the corresponding space coordinates, and generating the AR special effect. If 3 types of the currently identified handpieces exist, the user can select to watch the AR special effect achieved by the 3D virtual image corresponding to the 3 types of the handpieces, namely, the dynamic display effect of the 3D virtual image corresponding to the 3 types of the handpieces, and/or the dynamic display effect of the interaction with the user, and/or the interaction dynamic effect among the three types of the 3D virtual images; the user may also choose to view only one of the corresponding AR special effects. If the user collects all the models for the first time, displaying the AR special effects which are achieved through full collection, such as dynamic display effects of 3D virtual images corresponding to all the models, wherein dynamic display pictures can be generated between the 3D virtual images based on preset interaction logic; accordingly, when the situation that the user collects all types of the hands for the first time is identified, corresponding achievement badges can be generated, so that the collection interests of the user are improved.

In some implementation scenarios, different dynamic display frames can be displayed according to the placement gesture of the hand. For example, according to the standing or lying gesture of the user, the standing or lying 3D avatar is correspondingly displayed so as to enrich the display effect of the 3D avatar.

The embodiment of the application also provides a 3D virtual image processing device. Referring to fig. 3, fig. 3 is a block diagram of a 3D avatar processing apparatus according to an embodiment of the present application, and as shown in fig. 3, the 3D avatar processing apparatus 300 includes:

an acquiring module 301, configured to acquire a first preview image including a target object;

a first determining module 302, configured to determine first information according to the first preview image, where the first information includes at least one of the following: the method comprises the steps of setting a first space position of a target object in an actual space, the orientation of the target object in the actual space, the number of the target objects and the gesture of the target object;

a second determining module 303, configured to determine interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object based on the first information;

Optionally, referring to fig. 4, the apparatus further includes an execution module 304 configured to:

Optionally, the obtaining module 301 is further configured to:

obtaining an object category to which the object to be identified belongs;

determining a target detection model based on the object class;

Optionally, in the case that the first information includes a pose of the target object, the execution module is further configured to:

Optionally, in case the first information includes the number of target objects, the second determining module 303 is further configured to:

Optionally, the 3D avatar corresponding to the target object includes at least one second spatial position in the virtual three-dimensional space, and different second spatial positions correspond to different interaction information and/or different dynamic display information; and/or the number of the groups of groups,

and under the condition that the number of the target objects is at least two, the 3D virtual images corresponding to the at least two target objects respectively comprise at least one relative position relation in the virtual three-dimensional space, and different relative position relations correspond to different interaction information and/or different dynamic display information.

Optionally, the apparatus further comprises a construction module for:

extracting key points of each image frame, mapping the key points into a virtual three-dimensional space, and acquiring space azimuth information of the key points in the virtual three-dimensional space;

Optionally, the second determining module 303 is further configured to:

acquiring first input information;

In the embodiment of the application, the device can set corresponding interaction information and/or dynamic display information for such target objects as a host, a toy and the like, so that the target object can realize interaction with a user based on the corresponding 3D virtual image, or can display a dynamic display picture of the 3D virtual image of the target object, thereby effectively improving the interactivity between the target object and the user, enabling the interaction between the target object and the user to be more flexible, effectively enriching the display effect of the target object, and effectively improving the user experience of the target object.

The 3D avatar processing device 300 in the embodiment of the present application may be an electronic device, or may be a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. For example, the electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a mobile internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR)/Virtual Reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), or the like, and the embodiments of the present application are not limited in particular.

The 3D avatar processing device 300 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, and the embodiment of the present application is not limited specifically.

The 3D avatar processing device 300 provided in the embodiment of the present application can implement each process implemented by the method embodiment illustrated in fig. 1, and can achieve the same technical effects, and for avoiding repetition, a detailed description is omitted herein.

The embodiment of the application also provides electronic equipment. Referring to fig. 5, fig. 5 is a block diagram of an electronic device according to an embodiment of the present application, as shown in fig. 5, the electronic device includes: a processor 400, a memory 420 and a program or instructions stored on the memory 420 and executable on the processor 400, the processor 400 for reading the program or instructions in the memory 420; the electronic device also includes a bus interface and transceiver 410.

A transceiver 410 for receiving and transmitting data under the control of the processor 400.

Wherein in fig. 5, a bus architecture may comprise any number of interconnected buses and bridges, and in particular, one or more processors represented by processor 400 and various circuits of memory represented by memory 420, linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, power management circuits, etc., which are well known in the art and, therefore, will not be described further herein. The bus interface provides an interface. Transceiver 410 may be a number of elements, including a transmitter and a transceiver, providing a means for communicating with various other apparatus over a transmission medium. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 may store data used by the processor 400 in performing operations.

The processor 400 is configured to read a program or an instruction in the memory 420, and perform the following steps:

acquiring a first preview image comprising a target object;

Optionally, the processor 400 is further configured to read the program or the instruction in the memory 420, and perform the following steps:

obtaining an object category to which the object to be identified belongs;

determining a target detection model based on the object class;

Optionally, in the case that the first information includes the pose of the target object, the processor 400 is further configured to read a program or an instruction in the memory 420, and perform the following steps:

Optionally, in case the first information includes the number of the target objects, the processor 400 is further configured to read a program or an instruction in the memory 420, and execute the following steps:

acquiring first input information;

In the embodiment of the application, the electronic equipment can set corresponding interaction information and/or dynamic display information for such target objects as a host, a toy and the like, so that the target object can realize interaction with a user based on the corresponding 3D virtual image, or can display a dynamic display picture of the 3D virtual image of the target object, thereby effectively improving the interactivity between the target object and the user, enabling the interaction between the target object and the user to be more flexible, and effectively enriching the display effect of the target object.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the embodiment of the method described in fig. 1, and the same technical effects can be achieved, so that repetition is avoided, and no further description is given here.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium such as a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or instructions, so as to implement each process of the embodiment of the method described in fig. 1, and achieve the same technical effects, so that repetition is avoided, and no further description is given here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are to be protected by the present application.

Claims

1. A 3D avatar processing method, comprising:

acquiring a first preview image comprising a target object;

2. The method according to claim 1, wherein after determining the interaction information and/or the dynamic presentation information of the 3D avatar corresponding to the target object based on the first information, the method further comprises:

3. The method of claim 1, wherein the acquiring a first preview image including the target object comprises:

4. The method of claim 1, wherein the acquiring a first preview image including the target object comprises:

obtaining an object category to which the object to be identified belongs;

determining a target detection model based on the object class;

5. The method according to any one of claims 1-4, wherein, in case the first information comprises a pose of the target object, the method further comprises:

6. The method according to any one of claims 1-4, wherein, in case the first information includes the number of target objects, the determining interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object based on the first information includes:

7. The method according to any one of claims 1-4, wherein the 3D avatar corresponding to the target object comprises at least one second spatial location in a virtual three-dimensional space, different ones of the second spatial locations corresponding to different interaction information and/or different dynamic presentation information; and/or the number of the groups of groups,

8. The method according to any one of claims 1-4, wherein before determining the interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object based on the first information, the method further comprises:

9. The method according to any one of claims 1-4, wherein the determining interaction information and/or dynamic presentation information of the 3D avatar corresponding to the target object based on the first information comprises:

acquiring first input information;

10. A 3D avatar processing apparatus, comprising:

11. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the 3D avatar processing method of any one of claims 1-9.

12. A readable storage medium, wherein a program or instructions are stored on the readable storage medium, which when executed by a processor, implement the steps of the 3D avatar processing method as claimed in any one of claims 1-9.