WO2020098462A1

WO2020098462A1 - Ar virtual character drawing method and apparatus, mobile terminal and storage medium

Info

Publication number: WO2020098462A1
Application number: PCT/CN2019/112729
Authority: WO
Inventors: 朱克智; 王健; 严锋贵
Original assignee: Oppo广东移动通信有限公司
Priority date: 2018-11-16
Filing date: 2019-10-23
Publication date: 2020-05-22
Also published as: CN109597481A; CN109597481B

Abstract

An AR virtual character drawing method and apparatus, a mobile terminal and a storage medium. The method comprises: capturing a real three-dimensional scene picture by means of a camera, and building an augmented reality (AR) scene according to the real three-dimensional scene picture (101); acquiring at least one audio effect generated in the AR scene, and identifying whether there is a target audio effect in the at least one audio effect, wherein the target audio effect is generated from an audio generated by a virtual character that is not drawn in the AR scene (102); if so, acquiring the position of the camera in the AR scene, and determining an audio effect generation algorithm (103); determining, according to the audio generated by the virtual character, the target audio effect, the audio effect generation algorithm and the position of the camera in the AR scene, the position of the virtual character in the AR scene (104); and drawing the virtual character at the position of the virtual character in the AR scene (105). The method can improve the interaction effect of the virtual character in the AR scene.

Description

AR virtual character drawing method, device, mobile terminal and storage medium

Technical field

The present application relates to the field of audio technology, and in particular to an AR virtual character drawing method, device, mobile terminal, and storage medium.

Background technique

Augmented reality (AugmentedReality, AR) technology is a new technology that integrates real-world information and virtual world information "seamlessly". It is physical information that is difficult to experience in a certain time and space of the real world ( Visual information, sound, taste, touch, etc.) are superimposed after simulation, and the virtual information is applied to the real world, which is perceived by human senses, so as to achieve a sensory experience beyond reality. The real environment and virtual objects are superimposed on the same screen or space in real time. In the current AR scene, the virtual characters in the AR scene are synthesized according to special effects. The image and position of the virtual characters are determined according to a certain algorithm, and the interaction effect of the virtual characters in the AR scene is poor.

Summary of the invention

Embodiments of the present application provide an AR virtual character drawing method, device, mobile terminal, and storage medium, which can improve the interactive effect of virtual characters in an AR scene.

In a first aspect, an embodiment of the present application provides an AR virtual character drawing method, including:

Capture a real three-dimensional scene picture through a camera, and construct an augmented reality AR scene according to the real three-dimensional scene picture;

Acquiring at least one sound effect generated in the AR scene, and identifying whether a target sound effect exists in the at least one sound effect, and the target sound effect is generated by audio generated by an undrawn virtual character in the AR scene;

If it exists, obtain the position of the camera in the AR scene and determine the sound effect generation algorithm; according to the audio generated by the virtual character, the target sound effect, the sound effect generation algorithm and the camera in the AR scene The position of determines the position of the virtual character in the AR scene;

The virtual character is drawn at the position of the virtual character in the AR scene.

In a second aspect, an embodiment of the present application provides an AR virtual character drawing device, including:

Capture unit, used to capture the real three-dimensional scene picture through the camera;

A construction unit, configured to construct an augmented reality AR scene according to the real three-dimensional scene picture;

A first obtaining unit, configured to obtain at least one sound effect generated in the AR scene;

A recognition unit, configured to recognize whether a target sound effect exists in the at least one sound effect, and the target sound effect is generated by audio generated by an undrawn virtual character in the AR scene;

A second obtaining unit, configured to obtain the position of the camera in the AR scene when the recognition unit recognizes that the target sound effect exists in the at least one sound effect;

A determining unit, configured to determine a sound effect generation algorithm, and determine the virtual character in the AR according to the audio generated by the virtual character, the target sound effect, the sound effect generation algorithm, and the position of the camera in the AR scene Position in the scene;

The drawing unit is configured to draw the virtual character at the position of the virtual character in the AR scene.

In a third aspect, an embodiment of the present application provides a mobile terminal, including a processor and a memory, where the memory is used to store one or more programs, and the one or more programs are configured to be executed by the processor. The program includes instructions for performing the steps in the first aspect of the embodiments of the present application.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes the computer to execute the first embodiment of the present application. Part or all of the steps described in one aspect.

According to a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing the computer program, and the computer program is operable to cause the computer to execute as implemented in the present application Examples of some or all of the steps described in the first aspect. The computer program product may be a software installation package.

It can be seen that in the AR virtual character drawing method described in the embodiments of the present application, the mobile terminal captures a real three-dimensional scene picture through a camera, constructs an augmented reality AR scene based on the real three-dimensional scene picture; obtains at least one sound effect generated in the AR scene, Identify whether there is a target sound effect in at least one sound effect, the target sound effect is generated by the audio generated by the undrawn virtual character in the AR scene; if it exists, obtain the position of the camera in the AR scene, and determine the sound effect generation algorithm; according to the virtual character The position of the audio, the target sound effect, the sound effect generation algorithm and the camera in the AR scene determine the position of the virtual character in the AR scene; the virtual character is drawn at the position of the virtual character in the AR scene. In the embodiment of the present application, after the target sound effect is recognized, the exact position of the undrawn virtual character corresponding to the target sound effect in the AR scene can be reversed according to the sound effect generation algorithm, and can be drawn in the AR scene in the accurate position according to the sound effect of the virtual character Virtual characters, to improve the interactive effect of virtual characters in AR scenes.

BRIEF DESCRIPTION

In order to more clearly explain the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, without paying any creative work, other drawings can be obtained based on these drawings.

1 is a schematic flowchart of a method for drawing an AR virtual character disclosed in an embodiment of the present application;

2 is a schematic diagram of analog audio signal transmission disclosed in an embodiment of the present application;

3 is a schematic diagram of a principle for determining the position of a virtual character in an AR scene disclosed in an embodiment of the present application;

4 is a schematic flowchart of another method for drawing an AR virtual character disclosed in an embodiment of the present application;

5 is a schematic structural diagram of an AR virtual character drawing device disclosed in an embodiment of the present application;

6 is a schematic structural diagram of a mobile terminal disclosed in an embodiment of the present application;

7 is a schematic structural diagram of yet another mobile terminal disclosed in an embodiment of the present application.

detailed description

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of the present invention, but not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

The terms "first", "second", etc. in the description and claims of the present invention and the above drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally also includes Other steps or units inherent to these processes, methods, products, or equipment.

Reference herein to "embodiments" means that specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art understand explicitly and implicitly that the embodiments described herein can be combined with other embodiments.

The mobile terminals involved in the embodiments of the present application may include various handheld devices with wireless communication functions, in-vehicle devices, wearable devices, computing devices, or other processing devices connected to a wireless modem, and various forms of user equipment (User Equipment, UE), Mobile Station (MS), terminal equipment, etc. For convenience of description, the above-mentioned devices are collectively referred to as mobile terminals.

The following describes the embodiments of the present application in detail.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an AR virtual character drawing method disclosed in an embodiment of the present application. As shown in FIG. 1, the AR virtual character drawing method includes the following steps.

101. The mobile terminal captures a real three-dimensional scene picture through a camera, and constructs an augmented reality AR scene according to the real three-dimensional scene picture.

In the embodiment of the present application, the mobile terminal may include a camera, a display, and a speaker. Among them, the camera is used to capture real three-dimensional scene pictures in real time. The real three-dimensional scene pictures can be enclosed indoors or open outdoors. The display is used to display the AR picture corresponding to the AR scene. The speaker is used to output sound effects in the AR scene. Mobile terminals may include AR-enabled devices such as mobile phones and tablet computers, and may also include dedicated AR devices such as AR glasses and AR helmets.

Among them, AR scenes are built on the basis of real three-dimensional scenes. AR scenes can add multiple display controls on the basis of real three-dimensional scenes. The display controls can be used to call different virtual characters, as well as adjust the display effect of virtual characters, and adjust the position of virtual characters, and whether to use Turn on the 3D (3D) sound effect of the virtual character.

102. The mobile terminal obtains at least one sound effect generated in the AR scene, and identifies whether a target sound effect exists in the at least one sound effect, and the target sound effect is generated by audio generated by an undrawn virtual character in the AR scene.

In the embodiment of the present application, after the AR scene is constructed, the user can call an AR virtual character to display in the AR scene as needed. Since the drawing of the virtual character takes a certain amount of time, before the virtual character is drawn, the audio generated by the virtual character can come out before the image of the virtual character. If the 3D sound effect of the virtual character is turned on, the speaker of the mobile terminal will output the sound effect generated by the 3D sound effect generation algorithm. The user can also randomly call an AR virtual character to display in the AR scene, and the sound effect at this time is the sound effect of the randomly selected virtual character.

The AR scene includes not only the sound effects of virtual characters, but also the sound effects of background music, sound effects of virtual animals, and sound effects generated by virtual objects. The virtual character may be a virtual character in a game, a virtual character in a film and television work (such as anime), or a virtual character in a literary work.

Due to different types of avatars, the audio produced by them has different frequency characteristics. The virtual character can be used as an audio player, and the camera can be used as an audio receiver. The mobile terminal obtains at least one sound effect generated in the AR scene, which is at least one sound effect received on the basis of the camera as the angle of view. After receiving the sound effect, the mobile terminal can analyze the frequency characteristics of the sound effect to identify which sound effect is caused by Types of avatars generated by audio.

The target sound effect can be set in advance, which sound effect is set as the target sound effect that needs to be identified, and the target sound effect is identified in order to find the virtual character. The number of virtual characters may be one or more, and the number of virtual characters may be determined according to the AR scene.

In an AR scene, in a real three-dimensional scene, the audio player is playing voice / audio. For the audio receiving end, in addition to the direct sound signal directly passed by the audio playing end, the audio signal it receives also contains the reflected sound signal after various complex physical reflections. The reflected acoustic signal is delayed before the direct acoustic signal arrives, and its energy is attenuated due to physical reflection. And the AR scene is different, the delay of the reflected sound and the energy attenuation will be greatly different, resulting in the difference in the hearing of the audio receiving end. Therefore, for different AR scenes, different reverb sound effects algorithms can be used for sound effect processing.

as shown in picture 2. 2 is a schematic diagram of an audio signal analog transmission disclosed in an embodiment of the present application. The audio signal generated by the audio playing end in FIG. 2 can reach the audio receiving end through direct and reflective methods, thereby forming a reverberation effect at the audio receiving end. Two kinds of reflection paths are illustrated in FIG. 2, the first reflection path reaches the audio receiving end after two reflections, and the second reflection path reaches the audio receiving end after one reflection. Figure 2 is only an example of an audio signal transmission. The audio signal can be reflected by multiple reflection paths once, twice and more than twice to reach the audio receiving end. Different AR scenes have different reflection times and reflection paths. Whether the audio signal is direct or reflected, it will have a certain degree of attenuation. The attenuation coefficient is determined according to the distance of the path, the number of reflections, the transmission medium, and the material of the reflection point.

Optionally, in step 102, the mobile terminal identifying whether there is a target sound effect in at least one sound effect includes the following steps:

(11) The mobile terminal obtains the audio features generated by the virtual character;

(12) The mobile terminal recognizes whether there is a sound effect matching the above audio feature in at least one sound effect;

(13) If there is, the mobile terminal determines that the sound effect matching the audio feature among the at least one sound effect is the target sound effect.

In the embodiment of the present application, the audio characteristics include amplitude-frequency characteristics, that is, frequency characteristics and amplitude characteristics of audio. The audio generated by virtual characters generally has fixed frequency characteristics and amplitude characteristics. The frequency and amplitude vary within a certain amplitude, and the frequency and amplitude also have correlations, that is, the amplitude characteristics corresponding to different frequency points are not necessarily the same.

The mobile terminal recognizes whether there is a sound effect matching the above audio feature in at least one sound effect, specifically: the mobile terminal obtains the audio feature of each sound effect in the at least one sound effect, and calculates the audio feature of each sound effect and the audio feature generated by the virtual character Similarity, when there is a sound effect with a similarity greater than a preset similarity threshold in at least one sound effect, it is determined that a sound effect with a similarity greater than a preset similarity threshold in at least one sound effect is a target sound effect, and the above similarity is greater than Set the sound effect of the similarity threshold as the target sound effect.

The embodiment of the present application can identify whether the target sound effect exists in the at least one sound effect according to the similarity of the audio features. Since the recognition of the audio feature is relatively accurate, it can accurately identify whether the target sound effect exists in the at least one sound effect.

103. If it exists, the mobile terminal obtains the position of the camera in the AR scene, and determines the sound effect generation algorithm.

In the embodiment of the present application, the mobile terminal may determine the position of the camera in the AR scene according to the real three-dimensional scene image captured by the camera. Specifically, the mobile terminal can rotate the camera so that the camera can shoot a complete three-dimensional scene. A complete three-dimensional scene refers to a three-dimensional scene shot in 360 ° or 720 ° panorama. The mobile terminal determines the position of the camera in the AR scene according to the three-dimensional scene captured by the panorama.

The sound effect generation algorithm can be determined according to the scene in the real three-dimensional scene picture. For example, the sound effect generation algorithm corresponding to the indoor scene is different from the sound effect generation algorithm corresponding to the outdoor scene.

Optionally, in step 103, the mobile terminal determines the sound effect generation algorithm, specifically including the following steps:

The mobile terminal obtains the scene data corresponding to the AR scene, and obtains the type of the virtual character;

The mobile terminal determines the sound effect generation algorithm based on the scene data and the type of avatar.

In the embodiment of the present application, the sound effect generation algorithm is related to the scene data corresponding to the AR scene and the type of the virtual character. The scene data can include the geometric size of the real 3D scene constructing the AR scene (for example, the length, width, height, space volume, space length, width, and height of the building) and the material of the real 3D scene (for example, the floor in the building , Wall, ceiling materials), etc. The types of virtual characters may include virtual cartoon characters, virtual game characters, and the like.

The mobile terminal determines the sound effect generation algorithm based on the scene data and the type of virtual character, specifically:

The mobile terminal determines the sound effect algorithm model corresponding to the type of the virtual character according to the correspondence between the type and the sound effect algorithm model;

The mobile terminal determines the algorithm parameters of the sound effect algorithm model based on the scene data;

The sound effect generation algorithm is determined based on the sound effect algorithm model corresponding to the type of the virtual character and the algorithm parameters of the sound effect algorithm model.

For example, the sound effect algorithm model corresponding to the virtual cartoon character is different from the sound effect algorithm model corresponding to the virtual game character.

104. The mobile terminal determines the position of the virtual character in the AR scene according to the audio generated by the virtual character, the target sound effect, the sound effect generation algorithm, and the position of the camera in the AR scene.

In the embodiment of the present application, if the mobile terminal recognizes the target sound effect, the target sound effect is determined based on the sound effect generation algorithm, the audio generated by the virtual character, the position of the camera in the AR scene, and the position of the virtual character in the AR scene. If the mobile terminal determines the position of the audio generated by the virtual character, the target sound effect, and the camera in the AR scene, the position of the virtual character in the AR scene can be reversed.

Among them, the audio generated by the virtual character can be preset by the AR developer, the sound effect generation algorithm can be determined according to the scene data corresponding to the AR scene and the type of virtual character, the target sound effect can be directly obtained, and the position of the camera in the AR scene can be based on the panorama The three-dimensional scene taken is determined.

Optionally, if there is no target sound effect in the at least one sound effect, step 102 may be continued.

The following uses FIG. 3 as an example to explain the method for determining the position of a virtual character in an AR scene. FIG. 3 is a schematic diagram of a principle for determining the position of a virtual character in an AR scene disclosed in an embodiment of the present application. As shown in Figure 3, after the audio signal sent by the virtual person reaches the position of the camera through three paths, a reverb sound effect is formed at the position of the camera, the reverb sound effect P = S1 * R1 + S2 * R2 + S3 * R3, Where S1 is the attenuation coefficient of the first reflection path, S2 is the attenuation coefficient of the second reflection path, S3 is the attenuation coefficient of the third reflection path, R1 is the first initial audio signal transmitted along the first reflection path, and R2 is the edge The second initial audio signal transmitted by the second reflection path, R3, is the third initial audio signal transmitted along the direct path. The first reflection path passes through the first reflection surface, S1 is related to the material of the first reflection surface, the default propagation medium in the AR scene, and the path length of the first reflection path, the second reflection path passes through the second reflection surface, S2 and the second The material of the reflection surface, the default propagation medium in the AR scene, and the path length of the second reflection path are related. S3 is related to the default propagation medium and the length of the direct path in the AR scenario. R1, R2 and R3 are related to the spatial distribution of the sound field of the audio signal emitted by the virtual character in the real three-dimensional space. When the material of the first reflection surface and the default propagation medium in the AR scene are determined, the larger the path length of the first reflection path, the smaller S1; when the material of the second reflection surface and the default propagation medium in the AR scene When determined, the larger the path length of the second reflection path, the smaller S2; when the default propagation medium in the AR scene is determined, the larger the direct path length, the smaller S3.

In the case where the AR scene is determined, the spatial distribution of the sound field of the audio signal emitted by the virtual character in the real three-dimensional space is also determined, and the material of the first reflective surface and the second reflective surface are also determined, then R1 can be determined , The sizes of R2 and R3, and the default propagation medium in the AR scene can also be determined, leaving three variables, namely the path length of the first reflection path, the length of the second reflection path, and the length of the third reflection path. The target sound effect generated in the AR scene can be continuously obtained three times in a short time at the position of the camera to obtain three equations, the variables in the three equations are S1, S2 and S3, and the three equations are R1 and R2 , R3, and P are all determined and different (because the initial audio emitted by the virtual character changes in intensity and frequency distribution with time), you can solve S1, S2, and S3 through the ternary linear equations, and then according to S1 , S2 and S3 calculate the path length of the first reflection path, the length of the second reflection path, and the length of the third reflection path, and determine the position of the virtual character relative to the camera according to the lengths of the three paths. Since three sets of parameters are continuously acquired in a short time, the position of the virtual character relative to the camera is almost unchanged, and S1, S2, and S3 remain almost unchanged.

The above reverb sound effect algorithm (reverb sound effect P = S1 * R1 + S2 * R2 + S3 * R3) is just one possible example. According to different AR scenes and different sound effects produced by virtual characters, the reverb sound algorithm can also be This is achieved by other means, and will not be repeated here.

105. The mobile terminal draws the virtual character at the position of the virtual character in the AR scene.

In the embodiment of the present application, after determining the position of the virtual character in the AR scene in step 104, the mobile terminal may draw the image of the virtual character at the position of the virtual character in the AR scene. The display of the mobile terminal can display virtual characters in the AR scene. The mobile terminal can draw the virtual character according to the pre-set character model. The virtual character may have an animation effect.

In the embodiment of the present application, after the target sound effect is recognized, the exact position of the undrawn virtual character corresponding to the target sound effect in the AR scene can be reversed according to the sound effect generation algorithm, and the accurate effect of the virtual character sound effect in the AR scene Draw virtual characters at locations to improve the interactive effect of virtual characters in AR scenes.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of another method for drawing an AR virtual character disclosed in an embodiment of the present application. FIG. 4 is further optimized based on FIG. 1. As shown in FIG. 4, the AR virtual character drawing method includes the following steps.

401. The mobile terminal captures a real three-dimensional scene picture through a camera, and constructs an augmented reality AR scene according to the real three-dimensional scene picture.

402. The mobile terminal acquires at least one sound effect generated in the AR scene, and identifies whether a target sound effect exists in the at least one sound effect, and the target sound effect is generated by audio generated by an undrawn virtual character in the AR scene.

403. If it exists, the mobile terminal obtains the position of the camera in the AR scene, and determines the sound effect generation algorithm.

404. The mobile terminal determines the position of the virtual character in the AR scene according to the audio generated by the virtual character, the target sound effect, the sound effect generation algorithm, and the position of the camera in the AR scene.

405. The mobile terminal draws the virtual character at the position of the virtual character in the AR scene.

For specific implementation of steps 401 to 405 in the embodiments of the present application, reference may be made to steps 101 to 105 shown in FIG. 1, and details are not described herein again.

406. When the position of the virtual character in the AR scene changes and the AR scene does not change, the mobile terminal adjusts the sound effect corresponding to the virtual character according to the change in the position of the virtual character in the AR scene.

407. When the AR scene changes, the mobile terminal adjusts the sound effect corresponding to the virtual character according to the change in the AR scene.

In the embodiment of the present application, when the user moves with the mobile terminal, the real three-dimensional scene picture captured by the camera may change, and the corresponding AR scene may also change. For example, when a user holds a mobile terminal from one room to another, the AR scene will change. When the user moves with the mobile terminal, the position of the virtual character in the AR scene may also change. When the user clicks the display control in the AR scene to adjust the position of the avatar, the position of the avatar in the AR scene will change. When the user moves with the mobile terminal, the position of the virtual character in the AR scene may change.

When the position of the virtual character in the AR scene changes, the relative position relationship between the position of the virtual character in the AR scene and the camera will change, and the reverb sound effect of the audio generated by the virtual character propagating to the camera will also change. Therefore, the mobile terminal needs to adjust the sound effect corresponding to the virtual character according to the position change of the virtual character in the AR scene, and the interaction effect between the user and the virtual character in the AR scene can be increased through the change of the sound effect. Optionally, the user can send a voice interaction instruction, and the virtual character can move in the AR scene according to the voice interaction instruction, so that different interactive sound effects appear, increasing the interaction effect between the user and the virtual character in the AR scene.

When the AR scene changes, the position of the virtual character in the AR scene will inevitably change, and the corresponding parameters in the sound effect generation algorithm will also change accordingly, and the reverb sound effect of the audio generated by the virtual character propagating to the camera will also Changes occur, therefore, the mobile terminal needs to readjust the sound effect corresponding to the virtual character according to the change of the AR scene and the position of the virtual character in the new AR scene, and the interaction effect between the user and the virtual character in the AR scene can be increased through the change of the sound effect.

Optionally, in step 406, the mobile terminal adjusts the sound effect corresponding to the virtual character according to the position change of the virtual character in the AR scene, specifically including the following steps:

If the position of the virtual character in the AR scene changes from the first position to the second position, the mobile terminal re-determines the corresponding position of the virtual character according to the audio generated by the virtual character, the sound effect generation algorithm, the position of the camera in the AR scene, and the second position Sound effects.

In the embodiment of the present application, when the position of the avatar in the AR scene changes from the first position to the second position, the direct path and the reflection path of the audio generated by the avatar to the camera will change, so the parameters in the sound effect generation algorithm Corresponding changes will occur, and the reverb sound effect of the audio generated by the virtual character propagating to the camera will also change. The mobile terminal can recalculate the audio generated by the virtual character, the sound generation algorithm, the camera position in the AR scene, and the second position. Determine the sound effect corresponding to the avatar. The embodiment of the present application can adjust the sound effect corresponding to the virtual character in time when the position of the virtual character in the AR scene changes, and can increase the interaction effect between the user and the virtual character in the AR scene through the change of the sound effect. The user can change the sound effect of the virtual character in the AR scene by changing the position of the virtual character in the AR scene while the handheld mobile terminal is moving, thereby increasing the interaction effect between the user and the virtual character in the AR scene.

Optionally, in step 407, the mobile terminal adjusts the sound effect corresponding to the virtual character according to the change of the AR scene, which specifically includes the following steps:

(21) If the AR scene where the virtual character is located changes from the first AR scene to the second AR scene, the mobile terminal obtains the position of the virtual character in the second AR scene, and obtains the scene data corresponding to the second AR scene based on the second The scene data corresponding to the AR scene and the type of virtual character re-determine the new sound effect generation algorithm;

(22) The mobile terminal re-determines the sound effect corresponding to the virtual character according to the audio generated by the virtual character, a new sound effect generation algorithm, the position of the camera in the second AR scene, and the position of the virtual character in the second AR scene.

In the embodiment of the present application, when the virtual character changes from the first AR scenario to the second AR scenario in the AR scene, the parameters in the corresponding sound effect generation algorithm will also change accordingly, and the audio generated by the virtual character propagates to the camera The sound effect will also change, so the mobile terminal re-determines the correspondence of the virtual character according to the audio generated by the virtual character, the new sound effect generation algorithm, the position of the camera in the second AR scene, and the position of the virtual character in the second AR scene Sound effects. The embodiment of the present application can adjust the sound effect corresponding to the virtual character in time when the AR scene changes, and can increase the interaction effect between the user and the virtual character in the AR scene through the change of the sound effect.

Among them, the mobile terminal can analyze whether the AR scene where the mobile terminal is located changes according to the scene picture taken by the camera. Specifically, it can be analyzed whether the AR scene where the mobile terminal is located changes through elements in the scene picture (for example, buildings, plants, vehicles, roads, etc. in the scene picture). If the first AR scene is changed to the second AR scene, the mobile terminal may determine the position of the camera in the second AR scene according to the three-dimensional scene captured by the camera in a panoramic view.

For a specific implementation manner of the mobile terminal re-determining a new sound effect generation algorithm based on the scene data corresponding to the second AR scene and the type of virtual character, reference may be made to the description of step 103 in FIG. 1, and specific implementation of step (22) may be shown in FIG. The description of step 104 in 1 will not be repeated here. The above mainly introduces the solutions of the embodiments of the present application from the perspective of the execution process on the method side. It can be understood that, in order to realize the above-mentioned functions, the mobile terminal includes a hardware structure and / or a software module corresponding to each function. Those skilled in the art should easily realize that the present invention can be implemented in the form of hardware or a combination of hardware and computer software in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein. Whether a function is executed by hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

The embodiments of the present application may divide the functional unit of the mobile terminal according to the above method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The above integrated unit may be implemented in the form of hardware or software functional unit. It should be noted that the division of the units in the embodiments of the present application is schematic, and is only a division of logical functions. In actual implementation, there may be another division manner.

Please refer to FIG. 5, which is a schematic structural diagram of an AR virtual character drawing device disclosed in an embodiment of the present application. As shown in FIG. 5, the AR virtual character drawing device 500 includes a capturing unit 501, a construction unit 502, a first acquisition unit 503, a recognition unit 504, a second acquisition unit 505, a determination unit 506, and a drawing unit 507, where:

The capturing unit 501 is used to capture a real three-dimensional scene picture through a camera;

The construction unit 502 is used to construct an augmented reality AR scene according to the real three-dimensional scene picture;

The obtaining unit 503 is configured to obtain at least one sound effect generated in the AR scene;

The recognition unit 504 is configured to recognize whether a target sound effect exists in at least one sound effect, and the target sound effect is generated by audio generated by an undrawn virtual character in the AR scene;

The second obtaining unit 505 is configured to obtain the position of the camera in the AR scene when the recognition unit 504 recognizes that there is a target sound effect in at least one sound effect;

The determining unit 506 is used to determine a sound effect generation algorithm, and determine the position of the virtual character in the AR scene according to the audio generated by the virtual character, the target sound effect, the sound effect generation algorithm, and the position of the camera in the AR scene;

The drawing unit 507 is used to draw the virtual character at the position of the virtual character in the AR scene.

Optionally, the identification unit 504 identifies whether there is a target sound effect in at least one sound effect, specifically: acquiring audio features generated by the virtual character; identifying whether there is a sound effect matching the audio feature in the at least one sound effect; if it exists, determining at least A sound effect that matches an audio feature in a sound effect is a target sound effect.

Optionally, the determining unit 506 determines the sound effect generation algorithm, specifically: acquiring scene data corresponding to the AR scene; acquiring the type of virtual character; and determining the sound effect generation algorithm based on the scene data and the type of virtual character.

Optionally, the AR virtual character drawing apparatus 500 may further include an adjustment unit 508.

The adjusting unit 508 is configured to adjust the sound effect corresponding to the virtual character according to the position change of the virtual character in the AR scene when the position of the virtual character changes in the AR scene and the AR scene does not change;

The adjusting unit 508 is also used to adjust the sound effect corresponding to the virtual character according to the change of the AR scene when the AR scene changes.

Optionally, the adjusting unit 508 adjusts the sound effect corresponding to the virtual character according to the position change of the virtual character in the AR scene, specifically: if the position of the virtual character in the AR scene changes from the first position to the second position, according to the virtual character The generated audio, the sound effect generation algorithm, the position of the camera in the AR scene, and the second position re-determine the sound effect corresponding to the virtual character.

Optionally, the adjustment unit 508 adjusts the sound effect corresponding to the virtual character according to the change of the AR scene, specifically: if the AR scene where the virtual character is located changes from the first AR scene to the second AR scene, the virtual character is acquired in the second AR Position in the scene, obtain the scene data corresponding to the second AR scene, and re-determine a new sound effect generation algorithm based on the scene data corresponding to the second AR scene and the type of virtual character; based on the audio generated by the virtual character, the new sound effect generation algorithm, The position of the camera in the second AR scene and the position of the virtual character in the second AR scene re-determine the sound effect corresponding to the virtual character.

Optionally, the scene data corresponding to the AR scene includes spatial geometric parameters of the real three-dimensional scene and constituent material parameters of the real three-dimensional scene.

The capturing unit 501 may specifically be a camera in a mobile terminal, and the construction unit 502, first acquisition unit 503, recognition unit 504, second acquisition unit 505, determination unit 506, drawing unit 507, and adjustment unit 508 may specifically be mobile terminals Processor.

By implementing the AR virtual character drawing device shown in FIG. 5, after identifying the target sound effect, the exact position of the undrawn virtual character corresponding to the target sound effect in the AR scene can be reversed according to the sound effect generation algorithm, and can be based on the virtual The character's sound effect draws the virtual character at an accurate position, improving the interactive effect of the virtual character in the AR scene.

Please refer to FIG. 6, which is a schematic structural diagram of a mobile terminal disclosed in an embodiment of the present application. As shown in FIG. 6, the mobile terminal 600 includes a processor 601 and a memory 602, wherein the mobile terminal 600 may further include a bus 603, the processor 601 and the memory 602 may be connected to each other through the bus 603, and the bus 603 may be a peripheral component Peripheral Component (Interconnect, PCI) bus or Extended Industry Standard Architecture (EISA) bus, etc. The bus 603 can be divided into an address bus, a data bus, and a control bus. For ease of representation, only a thick line is used in FIG. 6, but it does not mean that there is only one bus or one type of bus. The mobile terminal 600 may further include an input and output device 604, and the input and output device 604 may include a display screen, such as a liquid crystal display screen. The memory 602 is used to store one or more programs containing instructions; the processor 601 is used to call the instructions stored in the memory 602 to perform some or all of the method steps in FIGS. 1 to 4 described above.

After implementing the mobile terminal shown in FIG. 6, after recognizing the target sound effect, the exact position of the undrawn virtual character corresponding to the target sound effect in the AR scene can be reversed according to the sound effect generation algorithm, and can be based on the sound effect of the virtual character in the AR scene Draw virtual characters at accurate locations to improve the interactive effect of virtual characters in AR scenes.

The embodiment of the present application also provides another mobile terminal. As shown in FIG. 7, for ease of description, only parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method of the embodiment of the present application section. The mobile terminal may be any terminal device including a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), an in-vehicle computer, etc. Taking the mobile terminal as a mobile phone as an example:

7 is a block diagram of a partial structure of a mobile phone related to a mobile terminal provided by an embodiment of the present application. 7, the mobile phone includes: a radio frequency (Radio Frequency) circuit 910, a memory 920, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a wireless fidelity (WiFi) module 970, a processor 980 , And power supply 990 and other components. Those skilled in the art may understand that the structure of the mobile phone shown in FIG. 7 does not constitute a limitation on the mobile phone, and may include more or less components than those shown in the figure, or a combination of certain components, or a different component arrangement.

The following describes each component of the mobile phone in detail with reference to FIG. 7:

The RF circuit 910 can be used for information reception and transmission. Generally, the RF circuit 910 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like. In addition, the RF circuit 910 can also communicate with other devices through a wireless communication network. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile (GSM), General Packet Radio Service (GPRS), and Code Division Multiple Access (Code Division) Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Message Service (SMS), etc.

The memory 920 may be used to store software programs and modules. The processor 980 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a storage program area and a storage data area, where the storage program area may store an operating system, application programs required for at least one function, and the like; the storage data area may store data created according to the use of a mobile phone and the like. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

The input unit 930 may be used to receive input numeric or character information, and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 930 may include a fingerprint recognition module 931 and other input devices 932. The fingerprint recognition module 931 can collect fingerprint data on the user. In addition to the fingerprint recognition module 931, the input unit 930 may also include other input devices 932. Specifically, other input devices 932 may include, but are not limited to, one or more of a touch screen, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, and so on.

The display unit 940 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 940 may include a display screen 941. Alternatively, the display screen 941 may be configured in the form of a liquid crystal display (Liquid Crystal) (LCD), an organic or inorganic light-emitting diode (Organic Light-Emitting Diode, OLED), or the like.

The mobile phone may further include at least one sensor 950, such as a light sensor, a motion sensor, a pressure sensor, a temperature sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor (also called a light sensor) and a proximity sensor, wherein the ambient light sensor may adjust the backlight brightness of the mobile phone according to the brightness of the ambient light, thereby adjusting the brightness of the display screen 941, and the proximity sensor may When the mobile phone moves to the ear, the display 941 and / or backlight is turned off. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when at rest, and can be used to identify mobile phone gesture applications (such as horizontal and vertical screen switching, magnetic force) Gesture attitude calibration), vibration recognition related functions (such as pedometer, tap), etc. As for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured on mobile phones, they will not be repeated here.

The audio circuit 960, the speaker 961, and the microphone 962 can provide an audio interface between the user and the mobile phone. The audio circuit 960 can transmit the converted electrical signal of the received audio data to the speaker 961, and the speaker 961 converts it into a sound signal to play; on the other hand, the microphone 962 converts the collected sound signal into an electrical signal, which is converted by the audio circuit 960 After receiving, it is converted into audio data, and then processed by the audio data playback processor 980, and then sent to, for example, another mobile phone through the RF circuit 910, or the audio data is played to the memory 920 for further processing.

WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 970. It provides users with wireless broadband Internet access. Although FIG. 7 shows the WiFi module 970, it can be understood that it is not a necessary component of a mobile phone, and can be omitted as needed without changing the essence of the invention.

The processor 980 is the control center of the mobile phone, connects various parts of the entire mobile phone with various interfaces and lines, executes or executes the software programs and / or modules stored in the memory 920, and calls the data stored in the memory 920 to execute Various functions and processing data of the mobile phone, so as to monitor the mobile phone as a whole. Optionally, the processor 980 may include one or more processing units; preferably, the processor 980 may integrate an application processor and a modem processor, where the application processor mainly processes an operating system, a user interface, and application programs, etc. The modem processor mainly handles wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 980.

The mobile phone also includes a power supply 990 (such as a battery) for powering various components. Preferably, the power supply can be logically connected to the processor 980 through a power management system, so as to realize functions such as charging, discharging, and power management through the power management system.

The mobile phone may further include a camera 9100. The camera 9100 is used to capture images and videos, and transmit the captured images and videos to the processor 980 for processing.

The mobile phone can also have a Bluetooth module, etc., which will not be repeated here.

In the foregoing embodiments shown in FIGS. 1 to 4, the method flow of each step may be implemented based on the structure of the mobile phone.

An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program causes the computer to execute any one of the AR virtual character drawing methods described in the above method embodiments Part or all of the steps.

An embodiment of the present application further provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing the computer program, the computer program is operable to cause the computer to perform any one of the methods described in the above embodiment Or all steps of a method for drawing an AR virtual character.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the sequence of actions described. Because according to the invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not detailed in an embodiment, you can refer to related descriptions in other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or may Integration into another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections through some interfaces, devices or units, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or software functional unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable memory. Based on such an understanding, the technical solution of the present invention essentially or part of the contribution to the existing technology or all or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a memory, Several instructions are included to enable a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned memory includes: U disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by a program instructing relevant hardware. The program may be stored in a computer-readable memory, and the memory may include: a flash disk , Read-Only Memory (English: Read-Only Memory, abbreviation: ROM), Random Access Device (English: Random Access Memory, abbreviation: RAM), magnetic disk or optical disk, etc.

The embodiments of the present application have been described in detail above, and specific examples have been used in this article to explain the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the method and core idea of the present invention; Those of ordinary skill in the art, according to the ideas of the present invention, may have changes in specific implementations and application scopes. In summary, the content of this specification should not be construed as limiting the present invention.

Claims

A method for drawing AR virtual characters, which includes:

Capture a real three-dimensional scene picture through a camera, and construct an augmented reality AR scene according to the real three-dimensional scene picture;

Acquiring at least one sound effect generated in the AR scene, and identifying whether a target sound effect exists in the at least one sound effect, and the target sound effect is generated by audio generated by an undrawn virtual character in the AR scene;

If it exists, obtain the position of the camera in the AR scene and determine the sound effect generation algorithm;

Determine the position of the virtual character in the AR scene according to the audio generated by the virtual character, the target sound effect, the sound effect generation algorithm, and the position of the camera in the AR scene;

The virtual character is drawn at the position of the virtual character in the AR scene.
The method according to claim 1, wherein the identifying whether a target sound effect exists in the at least one sound effect comprises:

Acquiring audio features generated by the virtual character;

Identify whether there is a sound effect matching the audio feature in the at least one sound effect;

If it exists, it is determined that the sound effect matching the audio feature in the at least one sound effect is the target sound effect.
The method according to claim 1, wherein the determining sound effect generating algorithm comprises:

Obtaining scene data corresponding to the AR scene, and obtaining the type of the virtual character;

The sound effect generation algorithm is determined based on the scene data and the type of the virtual character.
The method according to claim 3, wherein after the virtual character is drawn at the position of the virtual character in the AR scene, the method further comprises:

When the position of the virtual character in the AR scene changes and the AR scene does not change, adjust the sound effect corresponding to the virtual character according to the position change of the virtual character in the AR scene;

When the AR scene changes, adjust the sound effect corresponding to the virtual character according to the change in the AR scene.
The method according to claim 4, wherein the adjusting the sound effect corresponding to the virtual character according to the position change of the virtual character in the AR scene includes:

If the position of the virtual character in the AR scene changes from the first position to the second position, according to the audio generated by the virtual character, the sound effect generation algorithm, and the position of the camera in the AR scene 2. The second position re-determines the sound effect corresponding to the virtual character.
The method according to claim 4, wherein the adjusting the sound effect corresponding to the virtual character according to the change of the AR scene includes:

If the AR scene where the virtual character is located changes from the first AR scene to the second AR scene, obtain the position of the virtual character in the second AR scene, and obtain scene data corresponding to the second AR scene , Re-determining a new sound effect generation algorithm based on the scene data corresponding to the second AR scene and the type of the virtual character;

Re-determine the virtual based on the audio generated by the virtual character, the new sound effect generation algorithm, the position of the camera in the second AR scene, and the position of the virtual character in the second AR scene The sound effect corresponding to the character.
The method according to any one of claims 3 to 6, wherein the scene data corresponding to the AR scene includes spatial geometric parameters of the real three-dimensional scene and constituent material parameters of the real three-dimensional scene.
The method according to any one of claims 3 to 6, wherein the determining the sound effect generation algorithm based on the scene data and the type of the virtual character includes:

Determining a sound effect algorithm model corresponding to the type of the virtual character according to the correspondence between the type and the sound effect algorithm model;

Determine the algorithm parameters of the sound effect algorithm model based on the scene data;

The sound effect generation algorithm is determined based on the sound effect algorithm model corresponding to the type of the virtual character and the algorithm parameters of the sound effect algorithm model.
The method according to claim 1, wherein the method further comprises:

If the target sound effect does not exist in the at least one sound effect, the step of acquiring at least one sound effect generated in the AR scene is continued.
An AR virtual character drawing device, characterized in that it includes:

Capture unit, used to capture the real three-dimensional scene picture through the camera;

A construction unit, configured to construct an augmented reality AR scene according to the real three-dimensional scene picture;

A first obtaining unit, configured to obtain at least one sound effect generated in the AR scene;

A recognition unit, configured to recognize whether a target sound effect exists in the at least one sound effect, and the target sound effect is generated by audio generated by an undrawn virtual character in the AR scene;

A second obtaining unit, configured to obtain the position of the camera in the AR scene when the recognition unit recognizes that the target sound effect exists in the at least one sound effect;

A determining unit, configured to determine a sound effect generation algorithm, and determine the virtual character in the AR according to the audio generated by the virtual character, the target sound effect, the sound effect generation algorithm, and the position of the camera in the AR scene Position in the scene;

The drawing unit is configured to draw the virtual character at the position of the virtual character in the AR scene.
The apparatus according to claim 10, wherein the recognition unit recognizes whether a target sound effect exists in the at least one sound effect, specifically: acquiring an audio feature generated by the virtual character; identifying the at least one sound effect Whether there is a sound effect matching the audio feature in; if it exists, determining that the sound effect matching the audio feature in the at least one sound effect is a target sound effect.
The apparatus according to claim 10, wherein the determination unit determines the sound effect generation algorithm, specifically: acquiring scene data corresponding to the AR scene and acquiring the type of the virtual character;

The sound effect generation algorithm is determined based on the scene data and the type of the virtual character.
The device according to claim 12, wherein the device further comprises an adjustment unit;

The adjustment unit is configured to adjust the position of the virtual character in the AR scene when the position of the virtual character changes in the AR scene and the AR scene does not change Sound effects corresponding to virtual characters;

The adjustment unit is further configured to adjust the sound effect corresponding to the virtual character according to the change of the AR scene when the AR scene changes.
The apparatus according to claim 13, wherein the adjustment unit adjusts the sound effect corresponding to the virtual character according to the position change of the virtual character in the AR scene, specifically:

If the position of the virtual character in the AR scene changes from the first position to the second position, according to the audio generated by the virtual character, the sound effect generation algorithm, and the position of the camera in the AR scene 2. The second position re-determines the sound effect corresponding to the virtual character.
The apparatus according to claim 13, wherein the adjustment unit adjusts the sound effect corresponding to the virtual character according to the change of the AR scene, specifically:

If the AR scene where the virtual character is located changes from the first AR scene to the second AR scene, obtain the position of the virtual character in the second AR scene, and obtain scene data corresponding to the second AR scene , Based on the scene data corresponding to the second AR scene and the type of the virtual character to re-determine a new sound effect generation algorithm; according to the audio generated by the virtual character, the new sound effect generation algorithm, the camera in the The position in the second AR scene and the position of the virtual character in the second AR scene re-determine the sound effect corresponding to the virtual character.
The device according to any one of claims 12 to 15, wherein the scene data corresponding to the AR scene includes spatial geometric parameters of the real three-dimensional scene and constituent material parameters of the real three-dimensional scene.
The apparatus according to any one of claims 12 to 15, wherein the determining unit determines a sound effect generation algorithm based on the scene data and the type of the virtual character, specifically: according to the correspondence between the type and the sound effect algorithm model The relationship determines the sound effect algorithm model corresponding to the type of the virtual character; determines the algorithm parameters of the sound effect algorithm model based on the scene data; the sound effect algorithm model corresponding to the type of the virtual character and the algorithm of the sound effect algorithm model The parameters determine the sound effect generation algorithm.
The device according to claim 10, characterized in that

The first obtaining unit is further configured to obtain at least one sound effect generated in the AR scene when the recognition unit recognizes that the target sound effect does not exist in the at least one sound effect.
A mobile terminal is characterized by comprising a processor and a memory, the memory is used to store one or more programs, the one or more programs are configured to be executed by the processor, the program includes The method according to any one of claims 1-9 is performed.
A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program for electronic data exchange, wherein the computer program causes a computer to execute the computer program according to any one of claims 1-9 method.